All the popular nx cache libraries, including the Nx Cloud's one, seem to be using the npm tar package to tar/untar archives using Node.js. While this increases portability, I came to realize that this can be 12x slower than using system calls to GNU tar, and performance can be massively be improved by compressing using pigz - a multithreaded gzip alternative.
At the same time, most packages don't seem to be using the best practices for fetching large files from S3 using the AWS SDK resulting in slower uploads and downloads.
This package requires GNU tar and pigz for creating tar.gz archives using multithreading. These tools are already pre-installed in Github Actions' Linux runners.
This limits where this nx cache runner can work, but it should massively improve performance; enabling people to cache things like node_modules if need be in CI.
This package is just a combination of nx-remotecache-custom and @pellegrims/nx-remotecache-s3 with the following changes:
- Utilize best practices for uploading and downloading large files to and from Amazon S3
- Utilize GNU tar+pigz instead of npm tar