Changelog

ed0556d Brew formula update for s5cmd version v1.3.0
e81b978 add cache-control and expires options (#318)
fb03bce all: enable -race flag for tests (#336)
5a611da command/cp&rm: add --raw flag (#315)
fbd99c5 command/cp: add force-glacier-transfer flag (#312)
b4c91bd command/cp: allow overriding bucket region (#270)
d80fc02 command/expand: fix data race for concurrent writes (#330)
68f797b command/mb: fix incorrect usages in help output (#316)
d5e5ea3 command/mv: fix option usage (#338)
babc8d5 command/run: fix concurrent writes to the flags list (#335)
604f03a command: add --exclude flag (#334)
9defce6 command: append object error to multierror to return non-zero exit code error (#328)
5495dd4 command: implement 'remove bucket' feature (#310)
c2a9d31 command: implement 'select' command (#300)
02f63b3 makefile: fix check-fmt target (#327)
89baee6 readme: add 'windows' to the list of supported operating systems (#323)
c73528f readme: add steps for installation using macports (#311)
4130a7e storage/fs: change order of precedence for URL expansion (#322)
5ba7ee0 storage/s3: do not retry on expired token (#337)

Changelog

Features

• Added global --no-sign-request flag. API requests won't be signed and credentials won't be used if this option is provided. It is useful for accessing public buckets. (#285)

Improvements

• If retryable errors are received during command execution, users now can see what's happening under the hood. (#261)
• Update documentation about the AWS_PROFILE environment variable. (#275) @davebiffuk

Bugfixes

• Fixed a bug where write-bit was required to upload a file. (#258)
• Fixed a bug where object could not be found if S3 key contains certain special characters. (#279) @khacminh
• s5cmd exits with code 1 if given command is not found. It was 0 before. (#295)

Improvements

• Statically link s5cmd in Docker image (#250)

Bugfixes

• Fixed a bug where HeadBucket request fails during region information retrieval. (#251, #252)

With this release, s5cmd automatically determines region information of destination buckets.

Features

• Added global --dry-run option. It displays which command(s) will be executed without actually having a side effect. (#90)
• Added --stat option for s5cmd and it displays program execution statistics before the end of the program output. (#148)
• Added cross-region transfer support. Bucket regions are inferred, thus, supporting cross-region transfers and multiple regions in batch mode. (#155)

Bugfixes

• Fixed incorrect MIME type inference for cp, give priority to file extension for type inference. (#214)
• Fixed error reporting issue, where some errors from the ls operation were not printed.

Improvements

• Requests to different buckets not allowed in rm batch operation, i.e., throw an error.
• AWS S3 RequestTimeTooSkewed request error was not retryable before, it is now. (205)
• For some operations errors were printed at the end of the program execution. Now, errors are displayed immediately after being detected. (#136)
• From now on, docker images will be published on Docker Hub. (#238)
• Changed misleading 'mirroring' examples in the help text of cp. (#213)

With this release, Windows is supported.

Breaking changes

• Dropped storage class short codes display from default behaviour of ls operation. Instead, use -s flag with ls to see full names of the storage classes when listing objects.

Features

• Added Server-side Encryption (SSE) support for mv/cp operations. It uses customer master keys (CMKs) managed by AWS Key Management Service. (#18)
• Added an -s flag to show full form of storage class when listing objects. (#165)
• Add access control lists (ACLs) support to enable managing access to buckets and objects. (#26)

Bugfixes

• Fixed infinite repetition issue on mv/cp operations which would occur if the destination matched the source wildcard. (#168)
• Fixed windows filepath issue, where backslashes should be treated as the path delimiter. (#178)
• All tests pass on windows, by converting and treating file paths to UNIX filepath format.
• Fixed a transfer issue where the object path contains particular regex metacharacters. (#111) @brendan-matroid
• Correctly parse object paths that contain whitespaces in run-mode. (#111) @brendan-matroid

Improvements

• Retry when connection closed by S3 unexpectedly. (#189) @eminugurkenar

This is a major release with many breaking changes.

Breaking changes

• Dropped get command. Users could get the same effect with s5cmd cp <src> ..
• Dropped nested command support.
• Dropped ! command. It was used to execute shell commands and was used in
  conjunction with nested commands.
• s5cmd -f and s5cmd -f - usage has changed to s5cmd run. run command
  accepts a file. If not provided, it'll listen for commands from stdin.
• Exit code for errors was 127. It is 1 now.
• Dropped exit command. It was used to change the shell exit code and usually
  a part of the nested command usage.
• Dropped local->local copy and move support. (#118)
• All error messages are sent to stderr now.
• -version flag is changed to version command.
• Dropped batch-rm command. It was not listed in the help output. Now that we
  support variadic arguments, users can remove multiple objects by providing wildcards or multiple arguments to s5cmd rm command. (#106)
• Virtual host style bucket name resolving is enabled by default for S3 and GCS. If you provide a custom endpoint via --endpoint-url flag (other than GCS and S3 transfer acceleration), s5cmd will fall back to the path-style. (#92)
• Listing a non-existent object will return exit code 1, instead of 0. (#23)
• -ds, -dw, -us and -uw global flags are no longer available. Multipart concurrency and part size flags are now part of the cp/mv command. New replacement flags are --concurrency | -c and --part-size | -p. (#110)
• s5cmd cp command follows symbolic links by default (only when uploading to s3 from local filesystem). Use --no-follow-symlinks flag to disable this feature. (#17)
• Dropped -parents flag from copy command. Copy behaviour has changed to preserve the directory hierarchy as a default. Optional -flatten flag is added to flatten directory structure. (#107)
• Dropped -vv verbosity flag. --log flag is introduced.

Features

• Added mb command to make buckets. (#25)
• Added --json flag for JSON logging. (#22)
• Added S3 transfer acceleration support. (#40)
• Added Google Cloud Storage support. (#81)
• Added cat command to print remote object contents to stdout (#20)

Bugfixes

• Correctly set Content-Type of a file on upload operations. (#33)
• Fixed a bug where workers are unable to consume job if there are too many
  outstanding wildcard expansion requests. (#12, #58)

Improvements

• Pre-compiled binaries are provided on releases page. (#21)
• AWS Go SDK is updated to support IAM role for service accounts. (#32)
• For copy/move operations, s5cmd now creates destination directory if missing.
• Increase the soft limit of open files to 1000 and exits immediately when it encounters too many open files error. (#52)

Add Go modules support