Jumbo frames, QoS and other networky things

[cat6man]

hi there everyone.......still here and enjoying my sagetv.........just upgraded my media drive from 500GB to 2TB but everything is pretty stable.

while upgrading my drives, i've gotten rid of my one NAS that didn't have a gigabit interface and verified that all my routers & switches & NAS support gigabit LAN and jumbo frames.

i know the hd300 and hd200 don't support gigabit interface. what i'd like to know is if i can set the rest of my network to support jumbo frames and would this hurt my video streaming (or music for that matter, since I stream to logitech touch for my music system).

i'd like to get highest possible throughput copying files from my pc to/from my NAS and also between NAS (for backups). jumbo frames should help, but i won't do it if it messes up my sage system.

is there a way to set jumbo frames for communications between element A and element B and separately set things for normal sized frames between element A and element S?

any other suggestions on network settings for optimizing performance of latency sensitive streaming (e.g. music and video) and protecting it from large file transfers and torrents?

while i'm replacing all these old 100-250GB drives with 2TB drives (see signature), i might as well improve my networking skills.

thanks

[routerunner]

Originally Posted by cat6man

is there a way to set jumbo frames for communications between element A and element B and separately set things for normal sized frames between element A and element S?

any other suggestions on network settings for optimizing performance of latency sensitive streaming (e.g. music and video) and protecting it from large file transfers and torrents?

If your server has two gigabit ethernet controllers you can split your network so that all your NAS devices can go to the same switch to one of your server's ethernet and that could have jumbo frames, then connect your extenders together to a switch without jumbo frames to the server's second ethernet controller, this way is the server that manages the traffic, but you also need to configure your Sage UNC directory paths to point to the static address of the second gigabit controller, no hostname.

Mixed jumbo ethernet environment are not a good idea as it all depends how good is your switch on handling this, also there isn't an official jumbo frame packet size, it could vary between appliances.

If your switches are managed you can do other cool things with VLAN tagging as well...

Eddy

[drewg]

As long as you are speaking TCP, and have the switches' MTU set as large or larger than the MTU on your computers & NAS, you should be OK. This is because TCP negotiates the max seg size (MSS) when a connection is established. So if you have computers A & B, and A has an MTU of 9216 and B has an MTU of 9000, they will negotiate an MTU of 9000.

There are 3 problems that you can get into with jumbo frames

1) Not using TCP -- this bypasses the negotiation above. This happens more than you think (nameservers use UDP, for example).

2) Having a switch with a *SMALLER* MTU than 2 computers trying to talk across it. Assume you have a switch with a 4000 byte MTU in the example above. A & B will not notice that, and will try to to talk at 9000 bytes. What will happen is that every time the connection ramps above 4000b packets, they will be dropped, and re-sent. So you'll see bandwidth in the 100Kb/s range, and connections stalling.

3) Encountering a bug in some vendor's jumbo frame support. It is a very uncommonly used feature, and does not get a lot of Q/A.

With optimizations like TSO (transmit side) and LRO/GRO (rx side), jumbo frames are becoming largely unnecessary even on 10GbE. A single 10Gb/s benchmark on today's hardware uses very little CPU; I've setup demos with 8 ports of 10GbE, all running at full speed, which leave more than 50% idle time.

So...I'd suggest avoiding the hassle & just sticking to 1500.

Drew

[drewg]

Originally Posted by cat6man

any other suggestions on network settings for optimizing performance of latency sensitive streaming (e.g. music and video) and protecting it from large file transfers and torrents?

BTW, if you intend to use QoS, pay attention to your internet download speed. Comcast doubled our download speed to 50Mb/s, and with QoS on, my ~2 year old Netgear WNDR3300 router was maxed out and could not handle more than ~35Mb/s. I'm shopping for a replacement. In the meantime, I've turned off QoS..

Drew

[routerunner]

Originally Posted by drewg

As long as you are speaking TCP, and have the switches' MTU set as large or larger than the MTU on your computers & NAS, you should be OK. This is because TCP negotiates the max seg size (MSS) when a connection is established. So if you have computers A & B, and A has an MTU of 9216 and B has an MTU of 9000, they will negotiate an MTU of 9000.

There are 3 problems that you can get into with jumbo frames

1) Not using TCP -- this bypasses the negotiation above. This happens more than you think (nameservers use UDP, for example).

2) Having a switch with a *SMALLER* MTU than 2 computers trying to talk across it. Assume you have a switch with a 4000 byte MTU in the example above. A & B will not notice that, and will try to to talk at 9000 bytes. What will happen is that every time the connection ramps above 4000b packets, they will be dropped, and re-sent. So you'll see bandwidth in the 100Kb/s range, and connections stalling.

3) Encountering a bug in some vendor's jumbo frame support. It is a very uncommonly used feature, and does not get a lot of Q/A.

With optimizations like TSO (transmit side) and LRO/GRO (rx side), jumbo frames are becoming largely unnecessary even on 10GbE. A single 10Gb/s benchmark on today's hardware uses very little CPU; I've setup demos with 8 ports of 10GbE, all running at full speed, which leave more than 50% idle time.

So...I'd suggest avoiding the hassle & just sticking to 1500.

Drew

He's the man

Also, if your server runs Windows Server 2008 (Drew correct me if I'm wrong) and Windows 7 onwards you have the benefit of the variable TCP receive window size which helps in large file transfer.

Eddy

[drewg]

Originally Posted by routerunner

He's the man

Also, if your server runs Windows Server 2008 (Drew correct me if I'm wrong) and Windows 7 onwards you have the benefit of the variable TCP receive window size which helps in large file transfer.

Eddy

You're talking about RFC1323 window scaling (eg, scaling TCP windows beyond 64KB). I'm a *nix guy, but from a quick google, it looks like this has been available forever, but was enabled by default in Vista & Server 2K8. But it can be enabled via a registry change in XP

Drew

[routerunner]

Originally Posted by drewg

You're talking about RFC1323 window scaling (eg, scaling TCP windows beyond 64KB). I'm a *nix guy, but from a quick google, it looks like this has been available forever, but was enabled by default in Vista & Server 2K8. But it can be enabled via a registry change in XP

Not really, I know about the window scaling, but what I'm talking about is:

RWIN (Receive Window)
The TCP Receive Window can be thought of as the main data gate keeper to your computer. It sets the limits on the amount of data that can be received before it must send an acknowledgement and broadcasts that to the sender. If the number is too low you will tie up bandwidth with frequent and unnecessary acknowledgements. Too high of an RWIN will create a slowdown if any packets need to be retransmitted.

Now, this depends on the network latency and up to XP it was fixed, but from Vista and WS2800 this has been made dynamic and apparently it helps to improve data transfer a lot in particular on Gigabit network...I know that Linux has a similar concept, but is called differently...

Eddy

[drewg]

Originally Posted by routerunner

RWIN (Receive Window)

<...>
Vista and WS2800 this has been made dynamic and apparently it helps to improve data transfer a lot in particular on Gigabit network...I know that Linux has a similar concept, but is called differently...

Eddy

Yes, that's important too. Linux sets this on a bunch of factors, mainly net.core.rmem_max and net.ipv4.tcp_rmem and net.ipv4.tcp_moderate_rcvbuf (to auto-tune it).
Typically in linux, it is enough for 10Gb/s without touching anything

BSD/OSX call this stuff net.inet.tcp.recvspace and kern.ipc.maxsockbuf. FreeBSD will auto-tune it based on net.inet.tcp.recvbuf_auto and net.inet.tcp.recvbuf_inc/net.inet.tcp.recvbuf_max

Now that you mention it, I remember having to tune Win7 Rwin settings to get decent performance from my Win7 VM when I was playing with 7MC.

Drew