This Java programme uploads large files with scp and ssh over unreliable internet connections. It does this by splitting the large file into small parts and uploading them in parallel over multiple scp connections. When an upload stalls, it is repeated for that part.
SSHFileUploader path_to_configuration file_to_upload path_on_remote_server
java -jar sshfileuploader.jar /home/jack/tools/sshfileuploader.properties ./photos/backup.zip /mnt/raid/backups/2013-06-13
Example configuration file:
INTERVAL=1000
PAUSE_AFTER_NEW_PROCSS=500
MAXPROCESSES=2
Maximum size of file part to upload in kilobytes. Rule of thumb: the more reliable the connection the larger the value you can pick.
PARTSIZE_KB=64
MAX_FAILURES=20
scp=/usr/bin/scp
ssh=/usr/bin/ssh
userName=sshuser
server=ssh.example.com
tmpDir=/tmp
logging=normal
sshArgs=-o ConnectTimeout=30
Java 1.6 scp ssh
Q: Can I specify the password to use for ssh ? A: No. SSH must be set up to allow non-interactive login (i.e. keyfile)
Q: What is the best setting for MAXPROCESSES and PARTSIZE_KB ? A: That depends on your connection. Unreliable connections will require smaller PARTSIZE_KBs, a slow ssh handshake will benefit from larger MAXPROCESSES. The best thing is to try it out.
Q: It hangs A: Assuming that your network connection to the remote server is not completely dead, make sure you have disabled ControlMaster in ssh_config. If enabled, it will simulate multiple ssh connections over the same TCP connection which defies the purpose of this programme.
Q: Merging doesn't seem to work A: You need a bash on the remote server
Author: George Georgovassilis https://github.com/ggeorgovassilis/ssh-file-uploader