Skip to content

"s3cmd rb --recursive" fails to delete objects named with whitespace #976

New issue
New issue
Closed
Closed
"s3cmd rb --recursive" fails to delete objects named with whitespace#976
Labels
to-be-fixed
Milestone
2.4.0
@jamshid

Description

@jamshid
jamshid
opened on May 30, 2018

I believe this is an s3cmd bug in handling of files named with a space or newline. That's not a common character in object names but it's seen with test code.

I have a test that creates an S3 object like this

 touch '/tmp/ '
 # btw this PUT might fail with 403 when using V4 signatures?
 s3cmd -d -c ~/.s3cfg-s3 put '/tmp/ ' s3://spacetest1/

You can see the object named by a space character <Key> </Key> returned in the listing but the --recursive delete is not getting rid of this object.

Maybe s3cmd's xml parser is skipping keys that are just whitespace (also check objects named '\n').

Maybe s3cmd isn't encoding that key name properly on (multi) DELETE.

$ s3cmd -d -c ~/.s3cfg-s3 rb --recursive s3://spacetest1/
DEBUG: s3cmd version 2.0.1
DEBUG: ConfigParser: Reading file '/Users/jamshid/.s3cfg-s3'
DEBUG: ConfigParser: access_key->AK...17_chars...Q
DEBUG: ConfigParser: bucket_location->US
DEBUG: ConfigParser: cloudfront_host->cloudfront.amazonaws.com
DEBUG: ConfigParser: cloudfront_resource->/2010-07-15/distribution
DEBUG: ConfigParser: default_mime_type->binary/octet-stream
DEBUG: ConfigParser: delete_removed->False
DEBUG: ConfigParser: dry_run->False
DEBUG: ConfigParser: encoding->US-ASCII
DEBUG: ConfigParser: encrypt->False
DEBUG: ConfigParser: follow_symlinks->False
DEBUG: ConfigParser: force->False
DEBUG: ConfigParser: get_continue->False
DEBUG: ConfigParser: gpg_command->/usr/local/bin/gpg
DEBUG: ConfigParser: gpg_decrypt->%(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
DEBUG: ConfigParser: gpg_encrypt->%(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
DEBUG: ConfigParser: gpg_passphrase->...-3_chars...
DEBUG: ConfigParser: guess_mime_type->True
DEBUG: ConfigParser: host_base->s3.amazonaws.com:443
DEBUG: ConfigParser: host_bucket->%(bucket)s.s3.amazonaws.com:443
DEBUG: ConfigParser: human_readable_sizes->False
DEBUG: ConfigParser: list_md5->False
DEBUG: ConfigParser: log_target_prefix->
DEBUG: ConfigParser: preserve_attrs->True
DEBUG: ConfigParser: progress_meter->True
DEBUG: ConfigParser: proxy_host->
DEBUG: ConfigParser: proxy_port->0
DEBUG: ConfigParser: recursive->False
DEBUG: ConfigParser: recv_chunk->4096
DEBUG: ConfigParser: reduced_redundancy->False
DEBUG: ConfigParser: secret_key->gB...37_chars...8
DEBUG: ConfigParser: send_chunk->4096
DEBUG: ConfigParser: simpledb_host->sdb.amazonaws.com
DEBUG: ConfigParser: skip_existing->False
DEBUG: ConfigParser: socket_timeout->300
DEBUG: ConfigParser: urlencoding_mode->normal
DEBUG: ConfigParser: use_https->True
DEBUG: ConfigParser: verbosity->WARNING
DEBUG: ConfigParser: signature_v2->True
DEBUG: ConfigParser: check_ssl_certificate->False
DEBUG: Updating Config.Config cache_file -> 
DEBUG: Updating Config.Config follow_symlinks -> False
DEBUG: Updating Config.Config recursive -> True
DEBUG: Updating Config.Config verbosity -> 10
DEBUG: Unicodising 'rb' using UTF-8
DEBUG: Unicodising 's3://spacetest1/' using UTF-8
DEBUG: Command: rb
DEBUG: CreateRequest: resource[uri]=/
DEBUG: Using signature v2
DEBUG: SignHeaders: u'DELETE\n\n\n\nx-amz-date:Wed, 30 May 2018 23:48:01 +0000\n/spacetest1/'
DEBUG: Processing request, please wait...
DEBUG: get_hostname(spacetest1): spacetest1.s3.amazonaws.com:443
DEBUG: ConnMan.get(): creating new connection: https://spacetest1.s3.amazonaws.com:443
DEBUG: Using ca_certs_file None
DEBUG: Disabling SSL certificate checking
DEBUG: httplib.HTTPSConnection() has only context
DEBUG: non-proxied HTTPSConnection(spacetest1.s3.amazonaws.com, 443)
DEBUG: format_uri(): /
DEBUG: Sending request method_string='DELETE', uri=u'/', headers={'Authorization': u'AWS AKIAJV3VAEW6OFKXWMLQ:IjoSvlPqn79IfNv0XRrhkESMAG0=', 'x-amz-date': 'Wed, 30 May 2018 23:48:01 +0000'}, body=(0 bytes)
DEBUG: ConnMan.put(): connection put back to pool (https://spacetest1.s3.amazonaws.com:443#1)
DEBUG: Response:
{'data': '<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>BucketNotEmpty</Code><Message>The bucket you tried to delete is not empty</Message><BucketName>spacetest1</BucketName><RequestId>FA045631C650C33B</RequestId><HostId>oORPykd8PJVxMdJxgKm3Gg1zZtdkc+aZsbEsD4X/XQ64oy0ll6YZgqQItaLWTjjiGDk2EdCxM8g=</HostId></Error>',
 'headers': {'content-type': 'application/xml',
             'date': 'Wed, 30 May 2018 23:48:00 GMT',
             'server': 'AmazonS3',
             'transfer-encoding': 'chunked',
             'x-amz-id-2': 'oORPykd8PJVxMdJxgKm3Gg1zZtdkc+aZsbEsD4X/XQ64oy0ll6YZgqQItaLWTjjiGDk2EdCxM8g=',
             'x-amz-request-id': 'FA045631C650C33B'},
 'reason': 'Conflict',
 'status': 409}
DEBUG: S3Error: 409 (Conflict)
DEBUG: HttpHeader: x-amz-id-2: oORPykd8PJVxMdJxgKm3Gg1zZtdkc+aZsbEsD4X/XQ64oy0ll6YZgqQItaLWTjjiGDk2EdCxM8g=
DEBUG: HttpHeader: server: AmazonS3
DEBUG: HttpHeader: transfer-encoding: chunked
DEBUG: HttpHeader: x-amz-request-id: FA045631C650C33B
DEBUG: HttpHeader: date: Wed, 30 May 2018 23:48:00 GMT
DEBUG: HttpHeader: content-type: application/xml
DEBUG: ErrorXML: Code: 'BucketNotEmpty'
DEBUG: ErrorXML: Message: 'The bucket you tried to delete is not empty'
DEBUG: ErrorXML: BucketName: 'spacetest1'
DEBUG: ErrorXML: RequestId: 'FA045631C650C33B'
DEBUG: ErrorXML: HostId: 'oORPykd8PJVxMdJxgKm3Gg1zZtdkc+aZsbEsD4X/XQ64oy0ll6YZgqQItaLWTjjiGDk2EdCxM8g='
WARNING: Bucket is not empty. Removing all the objects from it first. This may take some time...
INFO: Retrieving list of remote files for s3://spacetest1/ ...
DEBUG: CreateRequest: resource[uri]=/
DEBUG: Using signature v2
DEBUG: SignHeaders: u'GET\n\n\n\nx-amz-date:Wed, 30 May 2018 23:48:01 +0000\n/spacetest1/'
DEBUG: Processing request, please wait...
DEBUG: get_hostname(spacetest1): spacetest1.s3.amazonaws.com:443
DEBUG: ConnMan.get(): re-using connection: https://spacetest1.s3.amazonaws.com:443#1
DEBUG: format_uri(): /
DEBUG: Sending request method_string='GET', uri=u'/', headers={'Authorization': u'AWS AKIAJV3VAEW6OFKXWMLQ:Zg9SlLyGI0rzLVXel/vqa8JiHvE=', 'x-amz-date': 'Wed, 30 May 2018 23:48:01 +0000'}, body=(0 bytes)
DEBUG: ConnMan.put(): connection put back to pool (https://spacetest1.s3.amazonaws.com:443#2)
DEBUG: Response:
{'data': '<?xml version="1.0" encoding="UTF-8"?>\n<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>spacetest1</Name><Prefix></Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated><Contents><Key> </Key><LastModified>2018-05-30T23:47:38.000Z</LastModified><ETag>&quot;d41d8cd98f00b204e9800998ecf8427e&quot;</ETag><Size>0</Size><Owner><ID>acdd0551e36e3cad005683fdcba346c2f98cdbf75eaadd884d7d3ed342609b71</ID><DisplayName>jamshid</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents><Contents><Key>x</Key><LastModified>2018-05-30T23:47:30.000Z</LastModified><ETag>&quot;d41d8cd98f00b204e9800998ecf8427e&quot;</ETag><Size>0</Size><Owner><ID>acdd0551e36e3cad005683fdcba346c2f98cdbf75eaadd884d7d3ed342609b71</ID><DisplayName>jamshid</DisplayName></Owner><StorageClass>STANDARD</StorageClass></Contents></ListBucketResult>',
 'headers': {'content-type': 'application/xml',
             'date': 'Wed, 30 May 2018 23:48:02 GMT',
             'server': 'AmazonS3',
             'transfer-encoding': 'chunked',
             'x-amz-bucket-region': 'us-east-1',
             'x-amz-id-2': 'CaiA7jm/bC4l7gwZMkLW3mpPg9v+lXQaVTr1fARqwmtW2awjLVuAJL5Jp3w8PgNtjl0ZyIJlLtM=',
             'x-amz-request-id': 'CFD3E44CD908D355'},
 'reason': 'OK',
 'status': 200}
WARNING: Empty object name on S3 found, ignoring.
DEBUG: Applying --exclude/--include
DEBUG: CHECK: x
DEBUG: PASS: u'x'
DEBUG: Batch delete 1, remaining 0
DEBUG: CreateRequest: resource[uri]=/
DEBUG: Using signature v2
DEBUG: SignHeaders: u'POST\nbbWx3PWpgOsdjIjq4SeQKg==\napplication/xml\n\nx-amz-date:Wed, 30 May 2018 23:48:01 +0000\n/spacetest1/?delete'
DEBUG: Processing request, please wait...
DEBUG: get_hostname(spacetest1): spacetest1.s3.amazonaws.com:443
DEBUG: ConnMan.get(): re-using connection: https://spacetest1.s3.amazonaws.com:443#2
DEBUG: format_uri(): /?delete
DEBUG: Sending request method_string='POST', uri=u'/?delete', headers={'content-type': 'application/xml', 'Authorization': u'AWS AKIAJV3VAEW6OFKXWMLQ:UJ+ZOWqtn+3fHtJlkzC9CH58jjE=', 'x-amz-date': 'Wed, 30 May 2018 23:48:01 +0000', 'content-md5': u'bbWx3PWpgOsdjIjq4SeQKg=='}, body=(84 bytes)
DEBUG: ConnMan.put(): connection put back to pool (https://spacetest1.s3.amazonaws.com:443#3)
DEBUG: Response:
{'data': '<?xml version="1.0" encoding="UTF-8"?>\n<DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Deleted><Key>x</Key></Deleted></DeleteResult>',
 'headers': {'connection': 'close',
             'content-type': 'application/xml',
             'date': 'Wed, 30 May 2018 23:48:02 GMT',
             'server': 'AmazonS3',
             'transfer-encoding': 'chunked',
             'x-amz-id-2': 'rLb9Tvb6/8tm4R14aIqXkDmlBmoe+LyAPOhmW4nWbRILy1qqNP40ll6gkmcshZ8+/1vY2kGrHWw=',
             'x-amz-request-id': '4D9D6C510AEB4D0F'},
 'reason': 'OK',
 'status': 200}
delete: 's3://spacetest1/x'
DEBUG: CreateRequest: resource[uri]=/
DEBUG: Using signature v2
DEBUG: SignHeaders: u'DELETE\n\n\n\nx-amz-date:Wed, 30 May 2018 23:48:02 +0000\n/spacetest1/'
DEBUG: Processing request, please wait...
DEBUG: get_hostname(spacetest1): spacetest1.s3.amazonaws.com:443
DEBUG: ConnMan.get(): re-using connection: https://spacetest1.s3.amazonaws.com:443#3
DEBUG: format_uri(): /
DEBUG: Sending request method_string='DELETE', uri=u'/', headers={'Authorization': u'AWS AKIAJV3VAEW6OFKXWMLQ:skY2LFONBnNyFRqOZYtOBgqP8SQ=', 'x-amz-date': 'Wed, 30 May 2018 23:48:02 +0000'}, body=(0 bytes)
DEBUG: ConnMan.put(): connection put back to pool (https://spacetest1.s3.amazonaws.com:443#4)
DEBUG: Response:
{'data': '<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>BucketNotEmpty</Code><Message>The bucket you tried to delete is not empty</Message><BucketName>spacetest1</BucketName><RequestId>B5EAD536D27A49BA</RequestId><HostId>s6cu4DtnkJIAABhD0YYjZlcsmzsUd0yLsCoyOiWrKUidJmrdZZWtUdNbzGQSgrTHLbveH2dofEs=</HostId></Error>',
 'headers': {'content-type': 'application/xml',
             'date': 'Wed, 30 May 2018 23:48:02 GMT',
             'server': 'AmazonS3',
             'transfer-encoding': 'chunked',
             'x-amz-id-2': 's6cu4DtnkJIAABhD0YYjZlcsmzsUd0yLsCoyOiWrKUidJmrdZZWtUdNbzGQSgrTHLbveH2dofEs=',
             'x-amz-request-id': 'B5EAD536D27A49BA'},
 'reason': 'Conflict',
 'status': 409}
DEBUG: S3Error: 409 (Conflict)
DEBUG: HttpHeader: x-amz-id-2: s6cu4DtnkJIAABhD0YYjZlcsmzsUd0yLsCoyOiWrKUidJmrdZZWtUdNbzGQSgrTHLbveH2dofEs=
DEBUG: HttpHeader: server: AmazonS3
DEBUG: HttpHeader: transfer-encoding: chunked
DEBUG: HttpHeader: x-amz-request-id: B5EAD536D27A49BA
DEBUG: HttpHeader: date: Wed, 30 May 2018 23:48:02 GMT
DEBUG: HttpHeader: content-type: application/xml
DEBUG: ErrorXML: Code: 'BucketNotEmpty'
DEBUG: ErrorXML: Message: 'The bucket you tried to delete is not empty'
DEBUG: ErrorXML: BucketName: 'spacetest1'
DEBUG: ErrorXML: RequestId: 'B5EAD536D27A49BA'
DEBUG: ErrorXML: HostId: 's6cu4DtnkJIAABhD0YYjZlcsmzsUd0yLsCoyOiWrKUidJmrdZZWtUdNbzGQSgrTHLbveH2dofEs='
ERROR: S3 error: 409 (BucketNotEmpty): The bucket you tried to delete is not empty

Metadata

Metadata

Assignees

No one assigned

    Labels

    to-be-fixed

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions