server crashes

[salsbst]

Lately we've been experiencing the following:

When watching a (pre-recorded) show on a Windows SageTV client, the show freezes on a frame, and I must forcibly end the process. Meanwhile, and this is the more frustrating part, the server is found to be dead -- I can ping it but not browse for its files over the network, and it has ceased putting out video to the computer monitor.

Here is a log from the server as things went bad most recently. This seems to be happening since moving to 6.4.8.

Tue 10/14 6:01:56.457 Seeker awoken
Tue 10/14 6:01:56.457 MemStats: Used=219MB Total=266MB Max=266MB
Tue 10/14 6:01:56.458 MARK 1 currRecord=A[4473274,4473256,"Whose Wedding Is It Anyway?",18511@1014.06:00,60] enc=Cable Box via Firewire on 127.0.0.1:6969 clients=[] ir=true
Tue 10/14 6:01:56.459 Seeker in AUTOMATIC mode nextRecord=A[4472611,4472580,"Ancient Apocalypse",16615@1014.07:00,60] nextTTA=3483542
Tue 10/14 6:01:56.459 newRecord=A[4473274,4473256,"Whose Wedding Is It Anyway?",18511@1014.06:00,60]
Tue 10/14 6:01:56.460 Keeping record just the way it is.
Tue 10/14 6:01:56.460 MARK 1 currRecord=null enc=Hauppauge WinTV PVR PCI II Capture clients=[] ir=true
Tue 10/14 6:01:56.460 defaultRecord=A[4474978,4474951,"Champions of the Wild",57394@1014.06:00,30]
Tue 10/14 6:01:56.461 Seeker in AUTOMATIC mode nextRecord=A[4471802,4471776,"Winston Churchill: The Lion's Roar",14771@1014.06:30,30] nextTTA=1683542
Tue 10/14 6:01:56.461 newRecord=null
Tue 10/14 6:01:56.461 NOTHING TO RECORD FOR NOW...
Tue 10/14 6:01:56.462 MARK 1 currRecord=null enc=Hauppauge WinTV PVR PCI II Capture #2 clients=[] ir=true
Tue 10/14 6:01:56.462 defaultRecord=A[4482369,2790762,"magicJack.com",51529@1014.06:00,30]
Tue 10/14 6:01:56.462 Seeker in AUTOMATIC mode nextRecord=A[4473123,4473095,"The Baby Lab",18327@1014.09:00,60] nextTTA=10683542
Tue 10/14 6:01:56.463 newRecord=null
Tue 10/14 6:01:56.463 NOTHING TO RECORD FOR NOW...
Tue 10/14 6:01:56.463 MARK 1 currRecord=null enc=Silicondust HDHomeRun Tuner 10100C7D-0 clients=[] ir=true
Tue 10/14 6:01:56.464 defaultRecord=A[4475783,73377,"News 4 Today",19578@1014.05:00,120]
Tue 10/14 6:01:56.464 Seeker in AUTOMATIC mode nextRecord=A[4539747,4539691,"Oprah Winfrey",19579@1014.16:00,60] nextTTA=35883542
Tue 10/14 6:01:56.464 newRecord=null
Tue 10/14 6:01:56.465 NOTHING TO RECORD FOR NOW...
Tue 10/14 6:01:56.466 RootFile=\\hindemith\sage_1tb_b fstype=null fileLength=49042432
Tue 10/14 6:01:56.467 VF processing job null nPlayin=false
Tue 10/14 6:01:56.467 Saving properties file to C:\Program Files\SageTV\SageTV\Sage.properties
Tue 10/14 6:01:56.468 isRec=false rd=3598394 base=2297466
Tue 10/14 6:01:56.469 VF thread is now waiting for 0:00:00.000
Tue 10/14 6:01:56.469 VF processing job null nPlayin=false
Tue 10/14 6:01:56.470 VF thread is now waiting for 0:00:00.000
Tue 10/14 6:01:56.585 Checking video directories for new files
Tue 10/14 6:01:56.610 Diskspace checking is running
Tue 10/14 6:01:56.610 Verifying existence of all TV media files in database fixDurs=false avoidArchive=true
Tue 10/14 6:01:57.943 Enforcing keep at most limits for the files...
Tue 10/14 6:01:58.963 needFreeSize = 0.4941 GB currFreeSize=6.325935104 GB
Tue 10/14 6:01:58.963 Seeker waiting for 23 mins.
Tue 10/14 6:01:58.994 Error in MediaServerConnection of :java.io.IOException: An existing connection was forcibly closed by the remote host
Tue 10/14 6:01:58.995 Error in MediaServerConnection of :java.io.IOException: An existing connection was forcibly closed by the remote host
Tue 10/14 6:02:00.088 Seeker.finishWatch(/192.168.6.22:3540@@SAGETV_PROCESS_LOCAL_UI)
Tue 10/14 6:04:16.779 EventThread-00085c53a270 Hang Detected - hang time = 76921
Tue 10/14 6:05:45.871 Screen Saver Timeout expired....tv=false playin=false fs=true sleepy=false currSS=true
Tue 10/14 6:05:45.872 Screen Saver Timeout expired....tv=false playin=false fs=true sleepy=false currSS=true
Tue 10/14 6:08:25.823 EventThread-00085c53c682 Hang Detected - hang time = 166105
Tue 10/14 6:08:25.824 Ministry is waiting for 180 sec
Tue 10/14 6:08:31.981 Error with c/s comm:java.net.SocketException: Connection reset by peer: socket write error
Tue 10/14 6:08:31.981 NetworkManager CommunicationFailure : /192.168.6.22:3540 type=2
Tue 10/14 6:11:11.987 NetworkClient fullCleanup /192.168.6.22:3540
Tue 10/14 6:11:11.988 Cleaning up c/s connection
Tue 10/14 6:11:11.988 Cleaning up c/s connection
Tue 10/14 6:11:18.135 Screen Saver Timeout expired....tv=false playin=false fs=true sleepy=false currSS=true
Tue 10/14 6:11:18.135 SageTV received connection from:Socket[addr=/192.168.6.22,port=4610,localport=42024]

[gplasky]

I had similar issues. The way I fixed it was copy the wiz.bin, uninstall Sage, delete the directory and do a clean isntall. You'll need to reconfigure your tuners but after I was done it worked perfect. Then I dropped in the wiz.bin and everything continued to work fine. Sometimes I think it happens when you have a .properties file from way back.

Gerry

[salsbst]

Gerry, thanks for your response.

I finally bit the bullet and followed your advice after the crashes became very frequent. At this point, we can't use SageTV at all. It manages to record stably (not sure if that's a word, but I'm sure it makes sense to this crowd). But when we try to watch a recorded show, the server crashes. It usually takes 10 seconds to 2 minutes for the crash, but it is rather predictable and happens eventually one way or the other.

I uninstalled and completely reinstalled SageTV on the server -- given the complication of our channel setup, this took nearly a day. I figured this would do the trick, but alas, no.

Then, I looked into my hard disks. I found some bad blocks on two of our three recording drives, fixed them, and still no joy. The files with the previously bad blocks play back fine outside of SageTv.

I am preparing to wipe my server clean with a format of the C drive (not one of the recording drives). I don't know if this is a Sage problem, a Java problem, or something with one of my drives or hardware devices. All I know is that I'm dead in the water right now, and quite frustrated.

The odd thing is that it doesn't matter which client we use -- we currently have one Windows client and two HD-100s. All of them seem to exhibit the same behavior -- SageTV server records fine (1xfirewire, 2xHauppague500, 2xHDHomeRun, but playback from any of the clients eventually causes the server to crash.

Why would the server crash when serving a file to any/all clients? Aside from the bad blocks (fixed), what would cause such instability. The machine seems fine otherwise, but using either our Windows or HD100 clients seems to be a kiss of death.

Any ideas would be welcome. After many happy years with SageTV, my wife is nearing her limit on this project, so it's a sad time. I know the technology is great, and we've had a lot of enjoyment with it, but at some point she will give out (and of course, I will follow).

Here's a more direct question (as opposed to the above rambling) -- what is the architecture of the playback operation? If a server is otherwise sound -- can serve files and doesn't crash otherwise -- what would cause it to be so unstable during playback of a SageTV recording from a client? Especially when two of the clients are HD100s and the other is a Windows client with a long term history of successfully negotiating the waters of difficult recordings? BTW, these problems are not specific to a particular encoder -- it can happen on a firewire recording, a Hauppauge recording and/or an HDHomeRun recording. The files themselves are sound. I can play them back on my workstation with no problem. (this machine is normally not part of the SageTV network, but can access the files for testing).

Confused, frustrated, hopeful,
-salsbst

P.S. In an effort to get current, I'm now on the latest beta (6.5.2). I can no longer tell whether my issue is with the beta or something that is a longer standing problem with SageTV, but problems most certainly started prior to beginning to test 6.5.

[gplasky]

Maybe it is a NIC going/gone bad. I started to have playback issues on my WHS for a short time. Then it just stopped communicating. I tried rebooting and was getting errors when it came up. I finally disabled the onboard NIC and installed a NIC card. After a couple of reboots and remembering to reset the MAC address in the router so the NIC woiuld get its previous address everything worked fine. Maybe you have a NIC going bad.

Gerry

[JDJinTX]

I'm also having problems, except I often lose the ability to communicate with the server even before videos are played; we get a black screen when we power up an HD100 without ever connecting to the server. This does not always affect all four HD100s, as some will work when others cannot communicate. The PC never needs a reboot; simply stopping & restarting SageTV does the trick.

If salsbst is simply restarting the software as a quick fix (rather than rebooting the machine), wouldn't this rule out the NIC card? If this does not eliminate the NIC as a source of the trouble, maybe I should try that as well.

jdj

[gplasky]

If you have a spare NIC it shouldn't take long. Make sure the new NIC gets the same address your old NIC has now or your clients won't connect. I think I stopped and started Sage a few times and got it to work for a mniute. I guess it would depend on what is flaking out on the NIC.

Gerry

[mickp]

Take a look in the windows event log. The bad blocks that you saw are unlikely to actually be fixed imho. What appears in the system and application logs during a crash/hang? In my experience, disk errors are likely to cause the symptoms that you describe.

Good luck.

Mick.

[salsbst]

Thanks, everyone for playing the "help Stuart fix his server" game!

Gerry -- what makes you suspect the network card?

mickp -- I did have both "bad block"/eventID 7 errors in the sytem log, as well as "The device, [my SATA adapter driver], did not respond within the timeout period"/eventID 9 errors. I haven't seen them since running chkdsk on the drives, but the problem persists. There are no events in any of the logs which correspond to the system crashes. When the server crashes, the first sign on the server(other than whichever client we are watching TV on freezing) is that the video is not displayed -- i.e., when I "kvm" over to the server, my monitor tells me there is no signal. IIRC, I can still ping the machine at this point, but the server console isn't accessible and I can't termserve into it either. I'll double check that, though, as my memory may be hazy on that after a long Thanksgiving-related drive.

All -- anyone know a general purpose stress test (that would test not only RAM and CPU but other standard hardware such as NIC, video, firewire, etc.)? I'm leaning toward something else being the problem. So far, the only thing that I can do to crash the machine is try to play back a show from any of our clients. (This is probably why Gerry is leaning toward the NIC, and perhaps he's right, but I'd like to find some evidence, as PCI slots are scarce in this machine so testing that theory would be complicated).

[salsbst]

Waiting for my wife to stop watching TV (she's watching live/no Sage) so I can do some more testing...

... in the meantime, I have a question to pose for anyone who can answer....

What does the server need to do (what does the server do) to serve a file for playback by a client? Is the server streaming bytes to the client? Or is the client reading a file from a network share? Or is it something more intricate and difficult to explain?

I ask because my next test is to play the media files through Windows Media Player on the one Windows client -- if the files play cleanly with no crashes, can I narrow down the problem to Java and/or SageTV? Or is this not a reasonable conclusion to draw?

[Wheemer]

with unc paths it's streamed outside of sage's control

[gplasky]

Originally Posted by salsbst

Thanks, everyone for playing the "help Stuart fix his server" game!

Gerry -- what makes you suspect the network card?

I saw similar symptoms on my WHS server momentarilly until the NIC finished dying and nothing could connect nor could I get out to the Internet.

Gerry

[wado1971]

FWIW - Mid-year I experienced the same issue your describing so I feel your pain. Also had Timeout/eventId errors so I looked at the disks first.

My first fix was to set RAID to IDE in the bios - based on others problems with similar MB's. At the same time I switched around my SATA plugs in a random attempt to fix something. Seemed to work ok.... (NOT!)

Issue came back about a month later with only occasional lockups (1/wk) but I'd loose a recording drive about 2-3 times a week. Pushing hard on the SATA cable for the dropped recording drive would reconnect it so it got me to thinking about the lockup also being a cable.

Long story short. Purchased all new SATA cables and took my time laying the cables. No lockups for over 3 months!!

[taylormadearmy]

Originally Posted by salsbst

All -- anyone know a general purpose stress test (that would test not only RAM and CPU but other standard hardware such as NIC, video, firewire, etc.)? I'm leaning toward something else being the problem. So far, the only thing that I can do to crash the machine is try to play back a show from any of our clients. (This is probably why Gerry is leaning toward the NIC, and perhaps he's right, but I'd like to find some evidence, as PCI slots are scarce in this machine so testing that theory would be complicated).

http://www.hiren.info/pages/bootcd is a great tool.

I have learnt from bitter experience that the first thing worth *testing* is your RAM. Last year on my MCE machine (in the bad ol' days before I upgraded to Sage ) I had weird problems when recording. The only other symptom was that I could not open Windows Event Viewer. A couple or re-installs later and after replacing many component, including my RAM I still had the same issue. I eventually ran a MemTest (available on Hiren's boot CD) that showed me that not only was my original memory duff, so was the replacement stick I had bought (good luck there hey!). Getting a new memory stick in place and all worked again...

The moral of the story for me was to run memtest on all machines for me and my family. Everyone has far more stable machines now as it also showed issues on another couple of machines.

Of course - it may not be your memory - but broken/unreliable memory can cause pretty much any symptom and is such an easy thing to rule out I recommend it as a first port of call to everyone.

Another useful tool that can be used within Windows to check system stability is http://www.passmark.com/products/bit.htm

T

[Flips]

Were you ever able to fix your issue? I was struggling with a very similar sounding issue myself, which I finally traced back to zonealarm. I tried disabling as one of the first things, but I had to actually uninstall it.

I was watching movies on a SageTV server that were shared via a windows share on another computer. The computer with the movies was a newer windows install. While watching a video from that computer the Sage computer would lock hard just like what what you explained. It would also lock up if I would go to the media library since I default to folder view on that computer.

It would play the file fine for a while sometimes 45 minutes, and then lock up. Samething with windows media player. Also sometimes it would lock up if I would copy a file. I also saw these entries in the event log of the computer with the video file:
30000 milliseconds timeout from ShellHWDetection
30000 milliseconds timeout from dmserver
30000 milliseconds timeout from TrkWks.

I found this page which suggested zonealarm:
http://www.ocforums.com/showthread.php?t=571773

Not sure if you have the same issue, but since it sounds similar I thought I would give it a shot.

flips

[salsbst]

Update:

I think I repaired the bad blocks on my SageTV recording drives.

I also believe I've tried every solution in the book to fix potential network problems (new NIC, different ethernet cable, different ports on switch).

All to no avail.

At this point, I'm beginning to suspect the PCI SATA card. It seems I can stably copy a large file from my C drive (which is not serviced by the PCI SATA card) to another server, but when attempting to copy a large file from one of the drives that *is* serviced by the PCI SATA card, the machine gets cranky.

So, next step is to replace that card. Anyone have a good experience with a PCI (not express, just plain old PCI) SATA card? The one I have is from BestBuy. Normally, I shop at NewEgg, but I was in a hurry when I bought this card. Needs to work (properly, completely) in Windows Server 2003....

Thanks for all of your help. This problem is pernicious and driving us crazy. I just wish I could get something useful from the eventlog, but whatever this is, it is at a layer under where the eventlog can record.

[Flips]

I can't give you any recommendations, typically for something like that I would just go to newegg and get one with good reviews.

One of the places I was going to go next for my issue was looking at the SATA settings, to make sure the drive speed matched the card speed. Have you tried lowering the speed setting or the compatibility setting on the card? I'm assuming some sort of bios that you can access during boot up.

Although on the bright side you could take the card back to Best Buy and say it's broken, if you got it in the last 30 days.

flips

[salsbst]

This SATA card worked well for at least year, so no opportunity to return it, but thanks for the idea.

I'll look into the drive/transport speed issue, and thanks for that idea, too. But since everything worked for so long before problems cropped up, I suspect that either a hardware failure or a dreaded Windows Update incompatibility is the source.

[HelenWeathers]

Have you tried playing the video(s) on the SageTV server itself?
Have you tried playing the video(s) on the SageTVClient using media player or some other software?
What results do you get?

That's where I would start my troubleshooting.

[dcardellini]

I'll second the memory problem.

I have found that Prime95 is the best benchmark for testing system stability.

My current server is marginally stable...it runs 24/7, and I may see one crash every 2-3 weeks. I can get it to crash quickly by transcoding for SageTV placeshifter though....anything that puts a load on it.

My board will pass Memtest indefinitely, but Prime95 will show errors within 10minutes.

Never got a solution....changed everything, memory five times, cpu, motherboard change (3 times), different Windows XP disks, hard drives.

Board is a uATX ga-ma69gm-s2h. Will be changing this MB out shortly.

[jlmdxtv]

A rule of thumb for me:

A hard drive showing *any* signs of failure gets tossed - because if it's showing me problems, it likely has dozens more that it "corrected" behind the scenes with retries.

john