NAS - direct record/playback or storage?

[jlindborg]

Just looking to get some advice from folks who have probably been down this road before me. Short version – I coughed up for a ReadyNAS NV+ last week and populated it with 4 500 gig drives (boo-ya!) – did a bunch of testing with it on my gig network with various cards and with/without jumbo packets enabled and the results were generally good (around 10 M/s write, 22 read). However with 3 recordings going on and 2 playbacks going on all hitting the NAS directly, it was sweating a bit and playback could jerk now and again (turned out to be a problem with the recording, actually, not the playback). So recording directly to the NAS doesn’t seem to be a solid option unless I’ve missed something with my hardware setup which I don’t think I have but I’m open to suggestion.

So – looking for some advice. I plan to do a couple things:

1. Change everything over to UNC instead of mapped drives – it’s my understanding that the clients will stream directly from the source in this case instead of having to hook-and-ladder through the sage server itself. Not specific to my problem but seems like a good plan anyway. Is this a true statement? I have a nice UPS capable of running my entire rack, router/switch included for a while and everything shuts down together when power goes out so UNC issues with name resolution don’t come into play here – I can use the MS loopback trick if I’m worried about it, but I’m not.

2. Setup Sage to do _all_ recordings to a local drive on the sage server – the NAS volume is there as a recording source but set to not allow further recordings. Then once a night I’ll have a script runs that net stops the sage service and moves all recordings to the NAS and starts the service again. This solves several problems, among them fragmentation issues when recording directly to the NAS since it does not allow for formatting with 64 K chunks. As long as I’m not recording more than 300 gig worth of TV in a day, this should be OK.

The one issue is the app I wrote to do #2 doesn’t have the ability to tell if sage is currently recording something or not – I can assume that 4am to 6am is clear and let it fly but it’d be handy to figure out an easy way to see “are you recording anything right now”? Presumably this would have to be done via the web interface somehow… Sure would be nice if there were a “move this file” option one could call instead of having to do this shut-down-copy-restart hack – from the threads I’m not the only one that’s wishing for this. Even if the DB sage used were a nice mySQL db I could hack on my own I’d be a happy camper, but hey – you can’t have everything.

Anyone else actually doing heavy duty record/playback directly to a NAS having luck or is everyone going for more of the “archived storage” model?

[GKusnick]

Originally Posted by jlindborg

The one issue is the app I wrote to do #2 doesn’t have the ability to tell if sage is currently recording something or not – I can assume that 4am to 6am is clear and let it fly but it’d be handy to figure out an easy way to see “are you recording anything right now”? Presumably this would have to be done via the web interface somehow…

"Right now" is not the right question. What you want to know is whether anything is scheduled to record in the next X minutes that it will take to stop the service, move several gigabytes of data, and start it back up again.

Fortunately, detailed information about the recording schedule is available through the SageTV API. It shouldn't be too hard to write a Java plugin for Sage that queries the schedule periodically (say once an hour) and writes it out as a text or XML file. Then your shutdown-and-copy script can read that file to know what's going to be happening in the near future.

I'm sure such a plugin would be of interest to other users as well, since this is not the first time this issue has come up.

[jlindborg]

I'm not a Java guy myself - just give me a command line app that kicks out some text and I'm off to the races - or a web interface that can splat some XML back at me or something.

yeah, lots of folks would be interested in this - but a "hey, move this file from here to here" interface would be even better since it would preclude the need to go through the shutdown process at all.

[Diginerd]

Heyas,

I was thinking about this exact same problem. Right now I have a batch file which quits sage, moves filesfrom a very fast, but not massive (140GB) scsi raid over to a pair of IDE drives striped as RAID 0 and then restarts Sage.

This does indeed solve file fragmentation (2 x interleaved HD streams creates a lot of fragments ), but suffers from killing anything sage happens to be recording, running out of space on the big raid isn't handled well (actually at all!).

I started think that dirmon might be a way of doing this without shutting down sage it can run any program, not just comskip, so have that copy files @Not in use@ over the course of the day might be better than all at once at night..

Not had chance to play with that yet, but it is a pretty cool direction I think if we can get it to work.

[jlindborg]

The problem there is that sage will get confused on the location of the file unless you shut it down when moving it (or at least restart it afterwards) - it seems to cache the location of the file and does not go looking for it again until startup. I tested this back with 5 and I don't think it's changed in 6 although I can test it again when I get some time. Short story is I don't think Sage likes it when you move files around from under it while it's running.

[Forged1]

Originally Posted by jlindborg

Anyone else actually doing heavy duty record/playback directly to a NAS having luck or is everyone going for more of the “archived storage” model?

Hi,

I have the same NAS and found that disabling the UPnP and Bonjour services on it helped tremendously. My problem was a little different though. I was seeing OTA HD breakups in Live TV and pauses in xvid playback every 30 minutes or so.

Also, if you want to increase read/write times a little further, the Infrant forums say installing 1GB of RAM in the NV+ will increase overall throughput by 12%. My 1GB chip is on order, so I can report back on that one later...

Hope that helps.

[jlindborg]

Yeah, I disabled everything I'm not using and it has 1 gig of RAM in there already - I also have it on a UPS with all the journaling disabled which helps a little.

I have a new NIC I'll be putting in my server that properly supports Jumbo packets (I'm told the one I have in there now does not which is why I'm not seeing improvement when turning on jumbo support - it actually gets slightly slower).

It's right on the edge of just doing it directly but the worst case scenario does still give it some problems - perhaps proper jumbo packet support will put it over the edge. They claim transfer rate numbers somewhat higher than I'm getting so I have some hope this will do it for me.

[Forged1]

Hi,

Jumbo packets never really worked out for me either. And, I've tried with various computers and NIC's. I gave up on seeing any performance enhancements from that.

For me, IOMeter gets about 16 MBs write and 26 MBs read without jumbo packets, and all journaling, UPnP / media streaming stuff disabled. I tend to see those numbers in real world file copies, too.

Have you tried different patch cables? Perhaps shielded Cat5e or 6 would give you a boost.

It sound like you've tried just about everything. I'm probably not giving you any new info here, sorry.

[gplasky]

Originally Posted by jlindborg

I have a new NIC I'll be putting in my server that properly supports Jumbo packets (I'm told the one I have in there now does not which is why I'm not seeing improvement when turning on jumbo support - it actually gets slightly slower).

Also verify that the switch and/or router with switch ports supports the jumbo packets. Very few consumer level switches do. Last time I looked I could only find some from SMC. Netgear has one I believe too. But you usually have to move up a level or buy one with more ports to get that support.

Gerry

[jlindborg]

I've been using a Netgear GS108 for testing (my production switch is a Dell 2616 which does not do jumbo packets - gotta step up to the 2716 for that). The Netgear specs claim to have full support for it and it's on the hardware compatibility list from Infrant so I'm assuming I'm good there for testing purposes.

It _could_ be cabling - I have a rats nest of cables in my wiring closet and I couldn't tell you for sure which ones were rated how - I'll go dig out some qualified patch cables and make sure everythig I'm testing between has good cables and try the throughput tests again tonight after work.

[gplasky]

That was going to be my next suggestion. You definitely want to make sure the cables are at least Cat 5E or Cat 6. Test with manufactured ends if you're crimping them yourself.

Gerry

[Diginerd]

Couple of realy stupid and for the network literate obvious (so sorry if you know this already..).

1. Are all the nics & switch ports at Auto/Auto setting? Don't hardcode a nic to 100/Full and have the switch be left on auto (As unmanaged switches generally are). That will cause your network to set to 100 half on the switch side, at that point you have a duplex mismatch and throughput will suffer really badly. Same is true for 10Mbit and 1000Mbit interfaces..

Only hardcode speed and duplex if both sides are nailed.

2. Throughput <= ~0.7 * MSS / (rtt * sqrt(packet_loss))

MSS = Max segment size (ie MTU minus TCP headers)
rtt = Round trip latency, on a lan this should be sub millisecond
packet loss = as a percentage

So, if you have poor cabling it significantly affects throughput, so does higher latency (Think WAn links, but outside the scope of this). Also at first sight increasing MSS by using jumbo frames will increase throughput too.

The problem comes that as you increase packet size you increase jitter, which is the variability of the timing of delivery. for transfers such as ftp this normally isn't a concern, but for realtime processes such as streaming HD Video it can become a major concern. Espectially if you couple it with faulty cabling as TCP will keep needing to readjust its window size when it drops packets. This can severly degrade performance again.

The HD Homerun uses UDP and smallish packets to stream. If it dops the odd one the stream keeps going regardless.

So to conclude:-

I wouldn't beat yourself up over jumbo packets too much, the payoff is not as great as you'd first think. Infact is can make things WORSE in streaming applicaitons. You should be more concerned about jitter and lack of drops

Make sure you have decent cabling. You can use a cable tester or simply run some PerfMon tests to spot bottlenecks.

Which leads into doublechecking speed and duplex settings everywhere, just one wrong can make a major mess...

Disk fragmentation (not even a network issue!) has a serious impact on throughput at these kinds of speads, especially if there are multiple read/writes going on. RAID 0 will help you a little here, but at the expense ofdoubling or quadrupling your risk of HD failure.. Only use if you don't care about the data or have a backup! RAID 5 looks promising, but if you throw a drive you are reduced to 25% of initial performance until the array is rebuilt.

Finally windows SMB is one of the most G-d awful, inefficient and all around crappy transmission protocols. Unfortunately we're stuck with it..

Yes, I'm a real network weenie who does this stuff for a living, and I'm happy to chat to anyone about this kind of thing as for many it's beyond a black art..

[Diginerd]

Oh, one more thing about cabling.. Don't run UTP (Unsheilded Twisted Pair, aka Cat5, Cat6 etc) cables in parallel with powercables. They're unshielded and twisted, which makes them really good inductors for picking up electrical noise and corrupting your networking signal.

If they need to be near each other cross them at 90 degrees and the induced current will be reduced to close to zero.

Also as induction is subject to the inverse square law if you double the distance between the cables the effect will be 1/4 of what it was.

All this is true for audio cables too, my other (old) dayjob

[hotwire]

Infrant said that they would look into the possiblilty of supporting 64k bock sizes if there were enough requests for it. Maybe if enough SageTV users asked for it, it might happen. What do you guys think?

[jlindborg]

I sent them a request along these lines - it'd be nice to have...

Following up on the bandwidth quest - replaced all the cables with the shortest production CAT5e cables (5 ft mostly) and updated the NIC in my server, put the 3 most important connections on the Netgear 108 (sage server, NAS, primary living room client) and retested - the read speed went up a bit (to around 26 or a little better) but the write speed jumped up quite a bit - hovering right around 19. Much better. Jumbo packets didn't help but I may have tested to quickly after the NAS restart - it seemed to be busy doing something after it came up, but either way I know my clients don't support this and it's kind of pointless so I left it off.

But all that said, I think I'm still going to record to my local drive and shuffle it off to the NAS every morning if nothing else than to avoid a critical problem with the connection to the NAS and keep fragmentation to a minimum. For now I'll just blindly shut down at 4am and move stuff and then bring it backup - I don't think I've ever recorded anything at 4am before.

[Lucas]

The READYNAS box may have it's limits too so you probably won't go much higher....

[dwk]

I'd really be interested in the script used to shutdown sage, move recordings to a different storage device, and then restart sage if someone has already gone through the work. If not, I'll be glad to whip one up in perl, but duplicating effort is not high on my priority list

Thanks,

[valnar]

Has anyone looked into this? Or more specifically, can I put in a request for Sage to add this capability to their product?

NAS boxes will become more prevalent so having a built-in "nightly database maintenance routine" would be very useful, especially since we all need Sage to be aware of the moved files. I only want to record locally too for reliability. But even better would be if a Sage client could stream directly from the NAS box without going through the Sage Server.

-Robert