Skip to content

feat: add functions to create pre-signed URLs for S3#3

Merged
keynslug merged 32 commits into
emqx:masterfrom
keynslug:ft/sigv4-presigned-urls
Apr 7, 2023
Merged

feat: add functions to create pre-signed URLs for S3#3
keynslug merged 32 commits into
emqx:masterfrom
keynslug:ft/sigv4-presigned-urls

Conversation

@keynslug

@keynslug keynslug commented Apr 7, 2023

Copy link
Copy Markdown

tommysrn and others added 30 commits August 9, 2022 23:43
erlcloud_s3:make_presigned_v4_url.
See erlcloud#562

Signed-off-by: Lincoln Baker <lbaker@chef.io>
Signed-off-by: Lincoln Baker <lbaker@chef.io>
While processing a sigv4 presigned url request, Bookshelf needs to
be able to determine whether the presigned url is valid, appropriately
signed, etc.

The only way I know to do this is to create a new presigned url
using whatever inputs the original presigned url was created with, and
compare the two presigned urls.  In order to do this, I have to use the
same host header and date the original presigned url was created with.
But it wasn't possible to pass in a host header and date, thus this commit.

Signed-off-by: Lincoln Baker <lbaker@chef.io>

code cleanup src/erlcloud_s3.erl

Signed-off-by: Lincoln Baker <lbaker@chef.io>
Signed-off-by: Lincoln Baker <lbaker@chef.io>
Signed-off-by: Lincoln Baker <lbaker@chef.io>
1) Host headers are treated inconsistently between various componenents
(erlcloud, mini_s3, bookshelf, etc).

2) The generated host header was incorrect in certain cases, eg the host
header generated from http://127.0.0.1:4321 left off the port and resulted
in Host: 127.0.0.1.  However host header specs say that in the case of a
missing port the assumed port will be based on the scheme - 80 for http, 443
for https.  So the port should be added in this case, as it is 4321.

https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html

Signed-off-by: Lincoln Baker <lbaker@chef.io>
This functionality was added to support creation of expiration windows
needed in oc_erchef and other components.

Signed-off-by: Lincoln Baker <lbaker@chef.io>
Signed-off-by: Lincoln Baker <lbaker@chef.io>
Signed-off-by: Lincoln Baker <lbaker@chef.io>
Signed-off-by: Lincoln Baker <lbaker@chef.io>
Signed-off-by: Lincoln Baker <lbaker@chef.io>
Fix license_scout error in Bookshelf.

Signed-off-by: Lincoln Baker <lbaker@chef.io>
===> Compiling erlcloud
===> Compiling src/erlcloud_aws.erl failed
src/erlcloud_aws.erl:1152:118: type headers() undefined

Signed-off-by: Lincoln Baker <lbaker@chef.io>
Signed-off-by: Lincoln Baker <lbaker@chef.io>
===> Compiling src/erlcloud_s3.erl failed
src/erlcloud_s3.erl:1141:22: variable 'SignedHeaders' is unused

Signed-off-by: Lincoln Baker <lbaker@chef.io>
===> Compiling src/erlcloud_aws.erl failed
src/erlcloud_aws.erl:1170:40: variable 'Date' exported from 'case' (line 1155, column 9)

Signed-off-by: Lincoln Baker <lbaker@chef.io>
Signed-off-by: Lincoln Baker <lbaker@chef.io>
Signed-off-by: Lincoln Baker <lbaker@chef.io>
Some server configurations (uppercase letters in FQDNs) can cause
sigv4 signature calculations using uppercase letters in the 'value' of the
host header key-value pair (example: "host: Whatever" vs "host: whatever").
Bookshelf or S3 will then fail on signature verification because the
host header key must be in lowercase for sigv4.

Nginx was found to be one of the culprits causing the issue, but
changing the nginx configuration is opening more of a can of worms than
fixing the issue here.

Signed-off-by: Lincoln Baker <lbaker@chef.io>
Extension of AWS Secrets Manager module
@keynslug keynslug requested a review from savonarola April 7, 2023 13:29
@keynslug keynslug merged commit a9700bf into emqx:master Apr 7, 2023
@keynslug keynslug deleted the ft/sigv4-presigned-urls branch April 7, 2023 18:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants