Network Speed Discrepency?

I don't know what the issue is here...

Here's my setup.

I have a reasonably fast Desktop PC sitting on three T1's (twisted into a single line? or however they are joined) which normally gives us about 4Mbps up and down. The desktop has a static IP, and we're currently hosting it in midtown Manhattan.

We just built a dedicated server box at a hosting company in Brooklyn, which comprises of a much faster machine sitting on a 10Mbps unmetered and reportedly unthrottled line (has a dedicated IP as well). The server is running Windows Server 2008.

So on one side of the equation I should have 4Mbps up, and on the other I should have 10Mbps down, yet when I transfer a large file (2Gb) I'm getting speeds around 200Kbps!! Wtf?!?

Here is a traceroute from here to there:

traceroute to 66.109.xxx.xxx (66.109.xxx.xxx), 64 hops max, 52 byte packets
 1  66.9.31.1 (66.9.31.1)  1.459 ms  0.744 ms  0.670 ms
 2  66.9.212.18 (66.9.212.18)  44.353 ms  15.398 ms  11.758 ms
 3  ip-160-79-127-229.autorev.intellispace.net (160.79.127.229)  14.341 ms  10.849 ms      8.220 ms
 4  gigabitethernet6-24.ar4.nyc1.gblx.net (207.136.166.53)  7.620 ms  33.029 ms  33.442 ms
 5  64.209.99.6 (64.209.99.6)  23.604 ms  31.267 ms  19.221 ms
 6  galaxy-visions.nyc2.webair.net (69.42.90.156)  6.183 ms  4.622 ms  13.551 ms
 7  *^C

(Step 7 is my server, as the traceroute is blocked by the firewall).

I've ran speedtests on both machines (speedtest.net) and the numbers roughly match what I should be getting (what I listed earlier).

The strange thing is that when I start my transfer of a 2Gb file, the transfer speeds for the first 1Mb are around 1Mbps, then it falls steadily after that until it flattens out at anywhere between 200-300kbps. I don't understand why I'm getting such slow speeds from these two very high speed connections!!

Any thoughts?

From serverfault neezer

• This is by no means a complete answer, but what is the MTU on the packets, and does the switching/routing equipment in between support that MTU if it's not of a standard size (for example jumboframes).

  Also, can you see if packets are being dropped?

  Are there other machines at each end you can do a similar testing with to try and rule out the problem being one of the hosts themselves?

  Finally, if you do a speed test between each end point, and a third-party node, do you find anything interesting?

  If you do some ping tests (set the no-fragment flag, and try various packet sizes), it may help to have tcpdump/wireshark running - so you can see any messages being sent back from intermediate routers which may tell you why the connection speed slows down to a grind.

  neezer : How would I find the MTU on the packets? I believe our routing equipment is capable of jumboframes (don't know off the top of my head, though). How do I see if packets are being dropped? Other machines on the 4Mbps network have similar speeds uploading to our dedicated box, but I don't have other machines on the Brooklyn side to test with. Not sure what "anything interesting" would entail in your last question; the traceroute and the speed test (@ speedtest.net) don't seem to say anything interesting that I can see.

  Xerxes : Well you can use ping and set the MTU yourself to begin with, on Windows I think you'd do something like "ping -f -l 1472", and if there are no dropped packets, then go higher. 1472 + 28 overhead makes up the 1500 is the traditional size, but it's a hangover from long ago, and you should be able to go higher than that. The -f you'll need to double check - it's the option that says "don't fragment the packets".

  Xerxes : Regarding "anything interesting" - I'm trying to see out of all the permutations of 2 hosts, if any have a considerably improved throughput than others, say node A is one end, B the other, C a machine in A's subnet, D a machine on B's subnet, and E an external host, if you try various pairs from A-E, do you see some that are faster than others, and that may help in troubleshooting this further.

  Xerxes : (host `E' in the above comment would be speedtest.net)

  Xerxes : Also, what protocol are you using to transfer the file with? Is there another protocol you can use instead?
  From Xerxes