[BBLISA] Large file transfer in 65ms latency

[John Stoffel]

>>>>> "Edward" == Edward Ned Harvey <bblisa4 at nedharvey.com> writes:

>> From: bblisa-bounces at bblisa.org [mailto:bblisa-bounces at bblisa.org] On
>> Behalf Of Tika Mahata
>> 
>> I am looking for the solution to transfer  large oracle database files
Edward> about 2TB
>> between two datacenters which have latency of 65ms with 1gpbs p2p link.
>> What are the options to reduce the transfer time? I have Linux at both
Edward> ends.
>> Any software or protocol or tuning the OS parameters?

Edward> The latency should be irrelevant for a continuous stream, but
Edward> I find transfers over sftp for some reason are susceptible to
Edward> the latency.

Absolutely not.  Read up on the Bandwidth Delay product.  If you crank
up your TCP window size on both ends, you can mitigate this, but TCP
was never designed with large latencies and large bandwidth at the
core algorithm levels.  

Edward> As long as you can use any other protocol - mbuffer, ftp,
Edward> http, cifs/samba, nfs ...  Presuming your connection is
Edward> secured by some other means (such as vpn).  Then even a single
Edward> stream continuous transfer should saturate the link.

Nope, once TCP gets to a certain number of outstanding packets, it
pauses and waits for the acknowledgements to come back.  So if you
look at the traffic, you'll see a classic saw tooth performance
graph.  

Edward> Also, you mentioned the data is oracle database.  This should
Edward> be irrelevant, as long as you stop the db server first.  Then
Edward> it's just a simple file transfer.

Edward> Another thing.  Being a database, it's probably compressible
Edward> (depending on the data inside there).  But being a 1Gbit link,
Edward> you'll probably slow yourself down by running through gzip.
Edward> Even gzip --fast isn't going to be fast enough.  You might
Edward> consider pigz --fast, or lzop.  These should gain you some
Edward> compression acceleration.

It acutally might be smarter to just ship the redo logs over there
instead too, but I'm not any type of Oracle expert.

John