Halt Detected in Recording - repaired

[Skiier__Dude]

Hey all,

Over the past 2 weeks my system has become completely unstable. I'm experiencing "micro-failures" in both my HD-PVR and HDHR but I don't think these failures are hardware related, I believe it's a Sage software issue given that these failures are happening on two different devices with two completely different loads of independent firmware.

During ALL recordings now (from either device) I will see anywhere from 10 to 50+ error messages. Live TV is now completely unwatchable. During a recording, the errors will result in about 10-60 seconds of a show being clipped while the recording device is reset by Sage.

Error message as follows:
Error - Halt Detected in Recording
The device input "Hauppauge HD PVR Capture Device Component+SPDIF" stopped producing data while recording... Sage will attempt to reset the device.

The same error message happens with the HDHR during recordings with that device as well. Exactly the same message (sub in the HDHR device name) with the same clipping of time from the show.

A restart of Sage or the recording hardware had zero impact, errors would immediately start up as soon as a recording started (usually about every 5 minutes).

My system as follows:

Running Sage on a headless standalone server, playback on HD-100 and HD-200s.

Server: WHS, AMD Athlon 5200+ Dual core, 2Gb RAM, 2 x 500Mb, 1 x 1Tb Sata drives

Sage: V6.6.2.218

Java 1.6.0

Java Heap size 768 Mb

For Plug-ins, I have been running Jetty with SRE and the Mobile Webserver. SageMC on 3 extenders. Running Comskip via Comskip Monitor.

The server itself shows no ill effect while these errors are happening. CPU idles along at about 20-30% during a recording or about 80% if Comskip kicks in (normal load). Lots of memory still available.

.
.
.

So here is what I did to fix. On the weekend, the corruption got so bad, live or recorded shows were completely unwatchable. I had put a request in for support, but could not wait to troubleshoot as we rely on Sage now for distribution of TV across the house.

I completely blew away my install of Sage. Backed up my Sage.properties and Wiz.bin files, uninstalled the application via the Control Panel in WHS and then renamed the Sage directory so that I was starting with a fresh install. I did not touch either the HD-PVR or HDHR installations.

Re-installed Sage and recovered the Wiz.bin file into it's directory and let Sage rebuild the Sage.properties files (I had to build the channel lineups again). Other than Comskip, I am now running Sage without any plug-ins.

So far, 72 hours has passed and I am no longer seeing any of the "Halt Detected..." errors and recordings are completely stable. Not sure what/how corrupted my Sage install (it's been running on WHS for about 1 year) but something happened.

Any takers for suggestions on what might have happened???

[gplasky]

Well for one thing on WHS you install and uninstall through the WHS console. You do not uninstall through the control panel or install the application using the .msi. I'm not sure if you actually menat you used the control panel to uninstall and I'm just asking if you installed it running the .msi. All WHS add-ins (like SageTV) get coiped to the Add-in directory under Software and installed/uninstalled through the WHS console. If it wasn't done that way that would be part of your problem.

Gerry

[Skiier__Dude]

No, in Un-installed it via the Control Panel. The un-install seemed to work as it completely removed the package from the console.

I Installed it by downloading the .msi and just double clicking on it and it did put it into the WHS console. It did work for the better part of a full year.

[gplasky]

How to install or remove an Add-in for Windows Home Server

Strange things have been reported to the WHS team and Microsoft support when a WHS app is not installed thru the console. It varies from not being reported as installed, can't be uninstalled, to just strange things malfunctioning with the app. And it always doesn't happen right away. It has reported taking months for symptoms to show. Not saying that was your issue but there was a reason an add-in was designed to be installed thru the WHS console. Just as Microsoft only recommends using the console for share folders, permissions, creating users, etc. Not that you can't do it thru File Explorer, Computer Manager, Disk Administrator etc. but they just guarantee that it may not continue to work for you. My recommendation would be to reinstall it properly but that is up to you. And good luck further on down the road with this install. Hope it's better than your last one.

Gerry

[Skiier__Dude]

Hi Big Gerr,

Thanks for the reply. In fact, when I go into the WHS Console and look at the Add-Ins section, Sage does NOT appear. The Sage service appears in the console and I can start and stop it from there.

I'm going to do as you have advised and get rid of my installation again and see what happens. Like you say, may or may not be the smoking gun, but worth the time doing it right.

[gplasky]

Follow the instructions in the link and it will take you probably 10 minutes. Uninstall from the control panel. Drop the .msi in the add-in folder. Go to the WHS console and install it. (Copy the folder somewhere else as a backup until after you're done.)

Gerry

[pjpjpjpj]

Skiier_Dude,

I am seeing the same behavior on one of my two HDHRs, since upgrading to the latest Sage release (see sig below). I'm not on WHS, just plain old XP. Nothing changed about my server or usage in the past 6 months (during which I have had little or no problems) until updating the software a few weeks ago. I have all the same plug-ins that I always had. Not even using comskip.

This all sorta came to a head last night when I discovered some other strange behavior (for which I posted a separate thread here... check that out, though it is not about this issue).

I have been seeing the same error messages and having failed or lost recordings as well. I thought that maybe I had always had those losses all along, and just didn't know about them until the error message center (new to the latest Sage release) was added. Then when I saw the choppy recording myself, I feared I had a dying HDHR, or switch port, or cable, or something... but the timing of this occurrence (along with my software update) seems too coincidental. Your post makes me now think that it's a glitch in the Sage software.

[gplasky]

What version of the HDHomerun software\firmware are you running? If not the latest then get the latest. Update and I would recommend removing all your HDHR sources and recreate following the instructions on this forum. Remember to scan for all the tuners in the Silicondust software. And then to scan for channels on all the tuners. Since you have the latest Sage software it only makes sense to have the latest from Silicondust because both Sage and Silicondust work closely together.

Gerry

[pjpjpjpj]

Originally Posted by gplasky

What version of the HDHomerun software\firmware are you running? If not the latest then get the latest. Update and I would recommend removing all your HDHR sources and recreate following the instructions on this forum. Remember to scan for all the tuners in the Silicondust software. And then to scan for channels on all the tuners. Since you have the latest Sage software it only makes sense to have the latest from Silicondust because both Sage and Silicondust work closely together.

Gerry

I thought we were not supposed to scan the HDHRs with the SiliconDust software anymore. I thought SageTV took care of that. Did they change the rules back again?

You might be right - it could be the HDHR - and I definitely should update my HDHR software/firmware. But the issues I'm seeing are only affecting one of my two HDHRs. That sounds fishy.

I'll be honest - I discovered my issues (that I mentioned in that other thread I started) late last night as I went to bed, so I have not done anything about it yet. My first step is going to be to reboot the server... and I have a suspicious feeling that the good old fashioned reboot might fix everything.

[Skiier__Dude]

On your HDHRs are you recording OTA or Cable? I find that with OTA, sometimes I see a signal drop out for stations that are a long distance away and it results in the same error message but instead of having a RED error warning, it generates a Yellow one.

[gplasky]

Originally Posted by pjpjpjpj

I thought we were not supposed to scan the HDHRs with the SiliconDust software anymore. I thought SageTV took care of that. Did they change the rules back again?

You might be right - it could be the HDHR - and I definitely should update my HDHR software/firmware. But the issues I'm seeing are only affecting one of my two HDHRs. That sounds fishy.

I'll be honest - I discovered my issues (that I mentioned in that other thread I started) late last night as I went to bed, so I have not done anything about it yet. My first step is going to be to reboot the server... and I have a suspicious feeling that the good old fashioned reboot might fix everything.

HDHR OTA only requires SageTV to scan for the channels. HDHR QAM from cable requires scanning all the tuners with the Silicondust software to create .scn and .frq files. Then do a channel scan for all tuners in Sage.

Gerry

[pjpjpjpj]

All OTA - see my sig...

[waynedunham]

Originally Posted by Skiier__Dude

During ALL recordings now (from either device) I will see anywhere from 10 to 50+ error messages. Live TV is now completely unwatchable. During a recording, the errors will result in about 10-60 seconds of a show being clipped while the recording device is reset by Sage.

Error message as follows:
Error - Halt Detected in Recording
The device input "Hauppauge HD PVR Capture Device Component+SPDIF" stopped producing data while recording... Sage will attempt to reset the device.

The same error message happens with the HDHR during recordings with that device as well. Exactly the same message (sub in the HDHR device name) with the same clipping of time from the show.

I didn't see you specifically mention how your getting your signal (OTA, Cable, Satellite, etc) but since I see you're using an HD-PVR I'm going to guess cable since that's what I use and had very similar issues.

I have Comcast cable and the signal issues (dropouts, 50+ halt detected messages, etc) were getting worse and worse. I called Comcast to come out and check my line since I'd had signal issues before. One time it was my line coming from the pole and another time it was a problem on their line with their amps.
A day or so before they showed up to check it out everything returned to normal. Come to find out on the day everything returned to normal they had come out for a problem at my neighbors house and found their amps out of adjustment (AGAIN!).

Everything was fine for a couple weeks and then the problem returned even worse than before. This time it was my amp/splitter in the basement. Only outlets in the house going through that amp/splitter had troubles (both with STB's hooked to just the tv or SageTv). I have 1 outlet running straight from where the cable splits as it comes into the house and that line was fine.

I wonder if your cableco (again assuming you're using cable) had issues similar to mine and they fixed it at the same time you "fixed" your system.

Just curious if you dragged the STB off your SageTV system and hooked it directly to a TV to see if the issue was happening without SageTV in the loop? With my issue the glitches happened with just the STB hooked to the TV so I knew it wasn't a SageTv issue.

[Skiier__Dude]

So, I thought I had this problem licked, but two weeks later and I am right back at the beginning again.

To recap -- I completely removed Sage via the Control Panel and then completely removed all directories and backed them up. I then did a brand new Sage install the proper WHS way as above (.msi as an Add-On).

As the week went on, I added in the Comskip playback STVi, later the Jetty server along with SRE and the mobile web interface.

Things worked fine without any errors until about 2 days ago and I started to slowly see the Halt Detected errors. Tonight -- I'm seeing a count in the low-teens. Some weird behaviour as well...

Kids are watching a show that was recorded several months back on an HD-200 upstairs. I go to try to watch a show that is about 50% completed recording on an HD-100 extender. The interface is very very slow on the HD-100 -- playback of the recording-in-progress eventually starts, plays for a few seconds, pauses for 20 seconds, plays about 10 seconds, pauses etc... When it pauses, the green data lights don't illuminate (like no data is being sent). Eventually, I power cycle the extender (video is still playing upstairs). When the extender starts -- it tells me there are no Sage Servers on the network (video is still playing upstairs however) -- log into the WHS machine -- Sage is running. Have the extender search for the server about a half dozen times -- it eventually picks up the server and starts. Go to watch the video again -- all is good for about 2 minutes, then playback pauses and the extender no longer responds to commands -- all of a sudden, extender goes black and both extenders no longer play video -- WHS is still running, but only a reboot of the server restores Sage once again.

So the only odd thing in my install is the Jetty/SRE/Mobile Web installs. I will remove these services completely tonight and see what happens. Given that it looks like the Sage server stops responding (stops sending data to the STB, does not appear on the network, and maybe stops working with the HDPVR etc...) I am wondering if Jetty or one of the Jetty tools is creating this issue as both were installed the last go-round and this problem only appeared after the install of these tools....

[gplasky]

You may also want to consider you may have a hardware piece starting to go. Maybe something heat related. It could be a NIC, a switch port or switch, a cable, etc. You may want to enable the Sage debug log and run diagnostics on your PC server.

Gerry

[Skiier__Dude]

I don't disagree that it could be a HW failure, but it seems that all signs point to something in software here. I reinstall Sage (don't touch any hardware) and things work perfectly for 2 weeks. Slowly add back in some customizations and the errors slowly work their way back into my system and then rapidly build to the point where the system is more-or-less unusable.

Not saying I disagree, but I see the Halt Detected error messages from two different models of tuner (HD-PVR and HDHR) connected in two different ways.

I am running debug logging now to see what sort of messages spit out.

[Mitch G]

I started getting these errors this past Thursday night for stuff recorded on my HDHR (clear QAM cable).
(Thank goodness for the Playon-Hulu plugin since it's letting me watch all the shows we didn't record. )

I started thinking I was experiencing "HDHR bad power supply part II."
But, I decided to first drop the HDHR video source and do a complete rescan on the HDHR and then re-add the video source into Sage.
Since then everything's been solid.

I see you completely reinstalled Sage, but did you start from scratch with your HDHR - including downloading the latest firmware? If not, it might be worth a try.


Mitch

[gplasky]

Originally Posted by Skiier__Dude

I don't disagree that it could be a HW failure, but it seems that all signs point to something in software here. I reinstall Sage (don't touch any hardware) and things work perfectly for 2 weeks. Slowly add back in some customizations and the errors slowly work their way back into my system and then rapidly build to the point where the system is more-or-less unusable.

Not saying I disagree, but I see the Halt Detected error messages from two different models of tuner (HD-PVR and HDHR) connected in two different ways.

I am running debug logging now to see what sort of messages spit out.

You've reinstalled how many times now-3? They all started out good but went down the crapper soon after. It's time to start looking somewhere else. Unless you've reinstalled it incorrectly or not from a complete clean install 3 times.

Gerry

[martin]

The halt error was happening more frequently on my system until I cleared out my Video Recording directories. I have four 250GB recording drives, but I had so many recordings I didn't want to delete that Sage was left with only one drive to work with, and it became badly fragmented. Three of the drives had zero space available, so my scheduled weekly defrag with Diskkeeper failed.

This weekend I just finished expanding my linux file server to a capacity of 30 disks to give me more archive space. After I moved 400GB of recordings off to a Media Centre Import directory located on my file server and defragged the rest of the recordings, it has been running smooth for a few days now.

I noticed before that Sage will record reliably for days, but as soon as I stress the server by playing back a recording on my HD200 I got the halt error immediately. It seems the HD-PVRs are sensitive to I/O waits, almost like they have small buffers.

Anyone else observe this?

[Skiier__Dude]

Well interesting that you mention that. I've opened a Case with Sage support and the tech got back to me and told me to switch my recording directory from UNC paths back to direct drive paths. I've always been recording to a direct attached storage device but have been using UNC paths from the days when I used to run PC clients and wanted Comskip to work. Anyhow, he indicated that it looks like my network is busy or there is something interrupting the I/O so I have switched the recording directories to the local path.

Last night (before making this switch), I got a System Message indicating that Sage does not have enough storage space available (although there is over 1Tb of free space). This is the first time I have seen this error message.

I don't have any drive errors reported and everything looks to be happy. Wondering if my drives need to be defragmented or if there is something else going on? My setup uses WHS and I have read that I should not use the standard Windows Defragment tool, and that there is even some trouble with off the shelf defrag tools. Anyone have any experience with Defragmenting WHS storage drives and the best Disk tools to use if any?