The easiest optimization is to use blowfish.
Edit /etc/ssh/ssh_config.
Change
# Cipher 3des # Ciphers aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc
to
Compression yes Cipher blowfish Ciphers blowfish-cbc,aes128-cbc,3des-cbc,cast128-cbc,arcfour,aes192-cbc,aes256-cbc
Verify by adding -v to the SSH command.
Much much better with HPN-SSH.
See here for some directions on installation. I haven't yet made my own. (dead link)
See https://launchpad.net/~w-rouesnel/+archive/openssh-hpn also.
For long distance transfers, see http://sourceforge.net/projects/hscp/.
Even if you don't use NoneEnabled/NoneSwitch, you can double the speed of the transfer because ssh won't be bottle necked by a single core. Using NoneEnabled/NoneSwitch, you can about the same speed, but with less CPU consumption.
$ time rsync -e \ "ssh -v -o NoneEnabled=yes -o NoneSwitch=yes" \ --progress --inplace --recursive \ hostname:/home/username/somedir local_dir
Updated link