SageTV in a school system

[JBurgin]

Howdy guys,

I work for a school system that has decided to give SageTV a try in a brand new school we're opening up. Instead of the classic TV distribution system, we've decided to build up a machine with tuner cards and feed Comcast cable to the classrooms over the network.

We are running into some problems, and I'd just like some feedback on our setup and see if maybe anybody out there could provide some insight.


Our setup:

One server, 3 Hauppage dual tuner cards (the model number escapes me now)

4x400GB hard drives in a Raid 5 array connected to a PCI Adaptec 9500S Sata Card

3 Ghz Processor
1GB RAM

We have purchased close to 60 client licenses, as we plan on providing it to every classroom in the school, on one computer that is fed to a smart TV.

Since we've gotten this project up to speed and have deployed everything (on test schools), we've run into problems. Mainly stuttering. Our main test bed is a lab of 25 or so computers. We do everything on the local LAN here at the school, but the server has a 1Gb network connection running to it. We fire up as many clients as we can (which is nearly every machine) and tune to the same channel. Performance is "decent" (lowest quality setting on the encoder....well..not lowest....but "Good" DVD quality)with occasional stuttering and lag on a handful of machines (they never stay together) However, when we start to change channels and take advantage of the 6 channel option....that's when things start to go downhill. We have stuttering, freezing, hangups, no response to keyboard commands.

If a client is unresponsive, we try to close the program and then reopen it to connect to the server. When we do this, the client will sit there for about a minute and then state that it cannot find the server. Even though several other clients may be connected and running TV just fine.

I realise that there is a chokepoint somewhere. We first thought it was the network, which is why we put it on a gig line. Althought network utilization went from 99% to 20%, performance was still the same.

My question now is, is the chokepoint at the Tuner Cards? or the hard drives? I know that standard PCI slots do not have an extremely high transfer rate, so I'm beginning to wonder if maybe the fact that I'm trying to feed 6 feeds at 1.0GB/hr through a slot that was only meant to handle 1.33MB/s is beginning to cause problems.

I'm really looking for some advice here, as we're getting close to crunch time (school starts August 7th) and I'd like to have it up and running the first day (so would my boss )

I have read into this "network encoding" feature and am also wondering if that would help out as well. My theory is that if we build a server that ONLY does encoding, then build a server that handles storage and scheduling...would that take our load off and allow the clients better performance?

Also, has it ever been tested exactly HOW MANY clients can connect to a SageTV server before problems start rising up?


Any and all help is appreciated,

Jack Burgin
Effingham County Board of Education
jburgin@effingham.k12.ga.us

[JUC]

First off--WOW that is ambitious! I'm thinking the decoding of the video might be the problem. What type of video cards are you using on the PC's? How will this work with the 'smart TV'. Next, what type of video decoder are you using? You provided great info in your post so hopefully these answers will help. Also-if you can get this going this is going to be a great test for a large installation of sage!!
JUC

[Mark SS]

First off, I'd suggest getting in touch with Sage directly rather than the forums. This is an ambitious project but potentially a good case study (and earner) for Sage and as such I'm sure they'll fall over themselves to get the installation right.

Whats happening on the server when you see problems? Whats the CPU/memory/paging situation? What are the read/write rates and % disk time for the raid array?

My money would be on the disk subsystem. Your trying to push 25 streams out, each of which will be seeking from disk at different locations and potentially thrashing the hell out of them. A raid solution with a large cache might help but I wouldn't spend anything until you've identified the problem.

Is that RAID card a hardware solution or does it tax CPU? Couldn't find it with a quick google. If the performance stats point to a disk issue you could always switch to a striped array.

[Humanzee]

my vote is for not enough pci bus bandwidth. If you could find a PCIe raid card that might help. This article says that a PCIe x8 raid card can provide 15 times the bandwidth of a standard 32bit pci slot.

http://www.gamepc.com/labs/view_cont...ookie%5Ftest=1

[JBurgin]

Thanks for all your replies so far. I'll try to answer your questions as detailed as I can.

As far as CPU/Memory usage goes, it doesn't seem to be a factor. The CPU never peaks above 35% (on the server side) and the memory is more than enough it seems.

The RAID card is a hardware solution, not the onboard SATA ports that come on the motherboard. They interface into a PCI bus, which I think might be one of our problems. The card CAN do PCI-X as well (it's a dual interface card). Our main issue now is that the motherboard we're using only has PCI slots. We may end up upgrading to a server style motherboard before this is done.

To answer JUC's questions, the decoders are built into the Tuner cards. The TV cards are Hauppage WINTV PVR PCI II tuner cards (that's what device manager calls em ) How they interface with the 'smart tv' is just like hooking it up to a projector. The TV's have a VGA input on the back (and front ) of the TV's, so you can easily interface a computer with the TV no problem at all.


Now that I've answered questions from the press ( ), let me update you on the changes we've made since yesterday. We decided to blow out the server (start over) and redo things. We've added an 80GB hard drive to the onboard sata connection, this will act as the OS drive as well as the Sage Server program drive. We've redone the raid to a RAID10 array with a 256kb stripe (the largest it offered). This has cut our capacity from 1+TB to 745 GB. A little disheartening, but we feel it's necessary. According to some reports, Raid10's sequential write and read rates are way faster than RAID5. I just got finished installing all my software (again) and am now about to take it out to the school again and test.

Thanks for all your help, if we cannot see an increase in performance, we'll definitely be doing a conference call with SAGE themselves. After reading the XXL Sage forum post, I thought you guys might like to hear what we're doing down here.

Have a good day

Jack Burgin
Effingham County Board of Education
jburgin@effingham.k12.ga.us

[blade]

Originally Posted by JBurgin

To answer JUC's questions, the decoders are built into the Tuner cards. The TV cards are Hauppage WINTV PVR PCI II tuner cards (that's what device manager calls em ) How they interface with the 'smart tv' is just like hooking it up to a projector. The TV's have a VGA input on the back (and front ) of the TV's, so you can easily interface a computer with the TV no problem at all.

The decoders aren't built into the tuner cards, the encoders are. JUC was asking about the decoders that you're using on the client machines that are playing back the video. Playing back video can be taxing on the client which is why the video card and decoder you're using can be important.

From the sound of it you're probably correct that the problem lies in the server.

[SHS]

JBurgin you really don't have enough bandwidth to go round to push 25 streams output on one server and do other thing like change channels.

Since we've gotten this project up to speed and have deployed everything (on test schools), we've run into problems. Mainly stuttering. Our main test bed is a lab of 25 or so computers. We do everything on the local LAN here at the school, but the server has a 1Gb network connection running to it. We fire up as many clients as we can (which is nearly every machine) and tune to the same channel. Performance is "decent" (lowest quality setting on the encoder....well..not lowest....but "Good" DVD quality)with occasional stuttering and lag on a handful of machines (they never stay together) However, when we start to change channels and take advantage of the 6 channel option....that's when things start to go downhill. We have stuttering, freezing, hangups, no response to keyboard commands.

This one easy it becuase of the fact that your runing one RAID setup writing one file and pushing 25 streams output which make it very hard to start 2nd file and a new 2nd streams output and one possable way a round this is Multi-HardDrive non RAID another way decrease and less lag is load balancing a cross Multi-Server.

Oh what your try to do you need look at to Solid State Disk and Fibre Channel connections which give you GB per second.

[JBurgin]

Originally Posted by SHS

JBurgin you really don't have enough bandwidth to go round to push 25 streams output on one server and do other thing like change channels.


This one easy it becuase of the fact that your runing one RAID setup writing one file and pushing 25 streams output which make it very hard to start 2nd file and a new 2nd streams output and one possable way a round this is Multi-HardDrive non RAID another way decrease and less lag is load balancing a cross Multi-Server.

This has become our realization. I did get my RAID 10 setup and running and performance WAS better...but nowhere near good enough. The disk write rates are OUTSTANDINGLY high when you're pulling 6 different channels at once with 25 clients each watching different sections. But I will say this...it did work. Just not perfectly. The occasional stutter and just a slight noticeable glitch in the audio. I was actually impressed with it. Especially considering that I had put the server back on a 100Mb connection, and not the gig I once had.


Our next plan is to try this:

Remove the RAID10 array (putting the OS on a separate HD has allowed me to blow out RAID's without spending 3 hours getting the OS reinstalled )

reconfigure all 4 drives as individual drives. Configure SAGE and the tuners to record to only certain drives. IE: Tuner #1 goes to it's own hard drive, etc etc. We'll use drive #4 for Imported Videos and whatnot.

Hopefully that might increase some performance. We shall see, the drives are formatted now. All I have to do is reconfigure the sage.properties file.

Thanks for all your help so far, guys.

To blade: my apologies for the misinformation. I got enough terms in my head that encoders and decoders get cross sometimes. As far as what kind the clients have, I am unsure at this moment. I have not installed any by myself. I would assume they're the defaults in a windows system.

[blade]

Originally Posted by JBurgin

To blade: my apologies for the misinformation. I got enough terms in my head that encoders and decoders get cross sometimes. As far as what kind the clients have, I am unsure at this moment. I have not installed any by myself. I would assume they're the defaults in a windows system.

No problem, just wanted to make sure you had it all straight. Most likely they're using the Sage decoder. Obviously your problem is the server's bandwith, so not much of a reason to worry about how the clients are setup.

Do you have the video drives formated to 64k clusters? If not you definetly need to do that.

[JBurgin]

Oh yeah! Our first iteration of the server didn't have that feature, but since I have realized how much video we're drawing in at one given time, I've realized how necessary this is.

[SHS]

64k clusters is not a must blade.

[ToxMox]

I'm also curious if the property:
default_pause_buffer_size=16777216
if increasing the value would help. I'm not sure if you would do it to the clients' or the server's properties but heck you could try doubling it everywhere and see if it helps.
64k clusters is not a must but it is recommended and could help your situation.

[JBurgin]

what exactly does that option do?

[Mark SS]

Stripe array would be far better than going back to individual drives.

On the clients you can try increasing a couple of registry values to increase the size of the buffers:

Frey Technologies/Common/DSFilters/MpegDeMux/Buffersize and NumBuffers

Can't remember what the defaults are but switching to 131072 and 1024 respectively resolved juddering over WLAN.

[blade]

Originally Posted by SHS

64k clusters is not a must blade.

It may not be a must, but I've always heard that it reduces fragmentation and can help with stuttering. If this is actually the case then why not do it now before putting it into service? I would think you'd want to do everything possible to improve performance and reduce future maintenance for a system of this scale.

Or is there some downside to it that makes you think he shouldn't do it?

[SHS]

Mark SS that not case here.

If I recall rigth I think the default_pause_buffer_size is for memory meaning store 16MB temp MPEG video before next buffer request.

JBurgin I still recommended you just cutting back the number of clients like down to 16 clients and just add more sage server.
The nice about the sage clients is you can tell it to use what ever server you want to use by tell it auto conn as long you setup each server on it own IP.
Not be going the above way you lot less headack with one system going down

[Mark SS]

Originally Posted by SHS

Mark SS that not case here.

If I recall rigth I think the default_pause_buffer_size is for memory meaning store 16MB temp MPEG video before next buffer request.

I was the case for myself and several others. The name default_pause_buffer_size sounds to me like its the size of the mpeg buffer when you press pause.

[SHS]

Mark SS for a small number clients that ok but we talk 25 out going clients which using far more resources then the system harddrive that can even handle.

[ToxMox]

Well Jack says he is using a quality that records around 1GB/hr per tuner. So the drive system needs to be able to handle 6GB/hr for the tuners plus 25GB/hr for the clients for a total of 31GB/hr. Divide that by 3600 and you get 8.61MB/second that the drive system would need to be able to randomly seek.
Double that to around 17MB/s if you want to use a better quality. I would imagine that using 64K clusters, using the drives individually per tuner and using large buffers all around would most likely handle his problem as long as not all the clients were viewing the same recording or happened to all be viewing different shows that happen to reside on the same drive.
It would be nice if one could assign a RAM cache for Sage to use.

I can't remember but is there a way to make the clients read the files directly from the recording folders rather than getting them streamed through the Sage service? I seem to remember something about UNC paths that could help out.

I would imagine there is some raid out there that could handle the throughput. Could take the seperate drive theory further and store to seperate RAIDs.

Have you contacted Sage support yet? I'm curious what they have to say regarding this.

[Opus4]

Originally Posted by ToxMox

I can't remember but is there a way to make the clients read the files directly from the recording folders rather than getting them streamed through the Sage service? I seem to remember something about UNC paths that could help out.

Yes, if the client can see the video file via something like a UNC path, then it will play it by reading the file itself.

- Andy