Sync with --cache-file fails when file is removed (KeyError in FileLists.fetch_local_list()) #377
Description
This sounds similar to #194 and http://sourceforge.net/p/s3tools/mailman/message/32305453/. I looked into the issue a little bit, and I think this has to do with the fact that I'm trying to sync a directory that is periodically removing files, and there are enough large files in the directory that the initial MD5 calculations takes a couple hours. The fix in the aforementioned issues has been to wrap the calls with a try/except on KeyError. I could submit a patch doing the same here, but I'm not sure that doing so is a proper fix.
This was tested against HEAD as of this writing (revision 76eb785). Here is the exception:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
An unexpected error has occurred.
Please try reproducing the error using
the latest s3cmd code from the git master
branch found at:
https://github.com/s3tools/s3cmd
If the error persists, please report the
following lines (removing any private
info as necessary) to:
s3tools-bugs@lists.sourceforge.net
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Invoked as: ./s3cmd --cache-file=/path/to/cache-file -v sync /path/to/sync/ s3://path/to/backup/
Problem: KeyError: 'dev'
S3cmd: 1.5.0-rc1
python: 2.7.3 (default, Apr 10 2013, 06:20:15)
[GCC 4.6.3]
environment LANG=en_US.UTF-8
Traceback (most recent call last):
File "./s3cmd", line 2521, in <module>
rc = main()
File "./s3cmd", line 2439, in main
rc = cmd_func(args)
File "./s3cmd", line 1470, in cmd_sync
return cmd_sync_local2remote(args)
File "./s3cmd", line 1440, in cmd_sync_local2remote
local_list, single_file_local, src_exclude_list = fetch_local_list(args[:-1], is_src = True, recursive = True)
File "/home/user/s3cmd/S3/FileLists.py", line 321, in fetch_local_list
_maintain_cache(cache, local_list)
File "/home/user/s3cmd/S3/FileLists.py", line 277, in _maintain_cache
cache.unmark_for_purge(local_list[i]['dev'], local_list[i]['inode'], local_list[i]['mtime'], local_list[i]['size'])
KeyError: 'dev'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
An unexpected error has occurred.
Please try reproducing the error using
the latest s3cmd code from the git master
branch found at:
https://github.com/s3tools/s3cmd
If the error persists, please report the
above lines (removing any private
info as necessary) to:
s3tools-bugs@lists.sourceforge.net
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!