HDHomeRun Prime - Preventing Tuner-Induced Halts in Recording or Lost Recordings

[jgsouthard]

Are you getting occasional "Halt Detected in Recording" errors where the recording is completely lost, SageTV can't even see the tuner, and you have to power cycle the tuner to get things restarted? Do these seem to occur more frequently when the HDHRP has been idle for several hours?

A lot of things can cause halts in recording, and I've been on a months-long quest to get rid of them. I FINALLY got rid of all of the intermittent halt errors I was seeing by rigorously adopting every networking tweak I found on this forum, and moved from PrimeNetEncoder back to SageDCT as my network encoder.

But even then, I was left with these occasional lost recordings where one of my HDHomeRun Primes would disappear from the network. Not just disappear from SageTV, but actually drop from the network such that it no longer appeared in File Explorer Network Devices, not longer responded to pings, and couldn't be accessed by HDHomeRun View. I initially thought my HDHRP was bad so I bought a new one, only to find that they BOTH did this occasionally.

More frequent use seemed to reduce the probability of this happening, but when it did happen the only remedy was a hard power cycle (unplug, plug back in).

So I set out to develop a Task Scheduled batch routine that would keep my HDHRP's working, and in the month or so that I've had this working I have not had a single halt error or lost recording. It's working well enough that I thought it might be worth sharing.

Special thanks to nyplayer for heading me in this direction, and for long ago sharing a batch file that would do a soft restart of SageDCT, SageTV, or the PC itself when SageTV was not recording. I've expanded that to focus on the HDHomeRun Prime tuners directly.

I have TWO batch files running via Task Scheduler:

1. RestartTuners.bat - This runs once per hour at 10 minutes before the top of the hour. For each HDHRP, it checks to see if the HDHRP is accessible on the network. If it is not, then it does a hard power cycle of the HDHRP by switching power off and then back on using a Belkin WeMo switch (WiFi controlled). If the HDHRP is accessible on the network, then it checks to see if any tuner is currently being used (this check is not tied to SageTV -- it will also check for use by HDHomeRun View or any other application using the tuner). If none of the tuners are being used, then it will do a soft restart of the HDHRP. If a tuner is in use it does nothing with that HDHRP until the next hourly check.

2. RestartPC.bat - This runs once per day at 1:40 am. It checks to see if SageTV is recording or in use (playback or live TV), and if it is not recording then it reboots the PC. If SageTV is currently in use, then it waits for 30 minutes and checks again, continuing to check and wait until it finds a time when SageTV is free and does the reboot. It will keep trying for up to 24 hours, after which point it dies and restarts for the next day. This pretty much guarantees that my SageTV PC gets rebooted once per day, almost always on the first or second check during the middle of the night.

The Belkin WeMo switches that allow hard power cycling of the HDHRP's may not be essential, since once-per-hour soft restarts of each HDHRP seem to have been sufficient to end most of the lock ups. Cygwin and Curl are also needed for the wemo_control.sh script, since I have not yet found a Windows-based program for command line control of WeMo devices.

The attached ZIP file contains copies of each of my two scripts, examples of the log files they produce, and the two supporting commands needed.

[Tiki]

I haven't had problems with Halt Detected in Recording errors using the Primes (or with them disappearing from the network). However, I've been noticing more and more frequently where one of my Primes will start showing all channels as "unsubscribed". When this happens:

• Sage makes a blank recording
• it records the whole show and reports no errors
• the file size is normal
• if you try to play it back in Sage using the PC it will hang for about a minute and then give a playback error
• if you try to play it back using a Sage extender, it will hang and need to restart the extender
• if you try to play it back in WMC or VLC it just plays a black screen with no sound
• if you highlight the recording in one of the Malore menus (either the Recording Schedule or the current recordings) while it is still recording, it will hang the Sage UI and must use the task manager to stop it.

This is with SageDCT.

If I check the status log in the Prime, I will see it says it tried to tune a channel that was unsubscribed. Unplugging and re-plugging the power from the Prime will fix the problem - it also clears the error log on the Prime, so can't show the log.

It's happened 3 or 4 times on each of my two primes over the last 6 months or so, but doesn't happen at the same time on both of them.

I noticed that around the same time that this started happening, I started getting another problem with the Primes (I have Comcast, and I suspect this is the same problem others have been reporting - I don't use the FFMPEG capture mode). The symptoms are very similar, but in this case the Prime is fine and the recordings it makes can be played back in WMC or VLC with no problems, but they will make the extender hang (it shows "no signal" and a spinning circle). From a PC, it hangs for a minute or so and then gives a Playback error. Selecting the recording from one of the Malore menus while it is still recording will hang the UI and need to force quit to recover.

I will probably move to the FFMPEG version of SageDCT or give OpenDCT a try to fix this second problem, but it's not going to help with the issue above where the Prime thinks its not subscribed to any channels. The question is, is this caused by a problem in the Prime or is it something Comcast is doing? I had been thinking it was Comcast, but after hearing your story, I'm starting to wonder if the Primes couldn't be at fault.

[jgsouthard]

Originally Posted by Tiki

I noticed that around the same time that this started happening, I started getting another problem with the Primes (I have Comcast, and I suspect this is the same problem others have been reporting - I don't use the FFMPEG capture mode). The symptoms are very similar, but in this case the Prime is fine and the recordings it makes can be played back in WMC or VLC with no problems, but they will make the extender hang (it shows "no signal" and a spinning circle). From a PC, it hangs for a minute or so and then gives a Playback error. Selecting the recording from one of the Malore menus while it is still recording will hang the UI and need to force quit to recover.

I will probably move to the FFMPEG version of SageDCT or give OpenDCT a try to fix this second problem, but it's not going to help with the issue above where the Prime thinks its not subscribed to any channels. The question is, is this caused by a problem in the Prime or is it something Comcast is doing? I had been thinking it was Comcast, but after hearing your story, I'm starting to wonder if the Primes couldn't be at fault.

If you are using the DUMP capture method in SageDCT with Comcast, you may need to set encode_digital_tv_as_program_stream=true in Sage.properties, and also mmc/encoders/<number>/100/0/encode_digital_tv_as_program_stream=true in the encoder section for each tuner. This will cause SageTV to capture the stream as an MPG file, which should play fine from both SageTV and extenders.

Otherwise, the TS file that gets created may at times be incompatible with the old firmware in the HD300 and other SageTV media extenders.

I know that both the SageDCT developer and other folks on the forum have recommended using FFMPEG capture instead (in which case you need to set encode_digital_tv_as_program_stream back to false in each case where it occurs), but I get consistent pixelation and skips when I use FFMPEG.

[KryptoNyte]

Originally Posted by jgsouthard

Are you getting occasional "Halt Detected in Recording" errors where the recording is completely lost, SageTV can't even see the tuner, and you have to power cycle the tuner to get things restarted? Do these seem to occur more frequently when the HDHRP has been idle for several hours?

Yes, I have seen this exact problem. Most frequently in the very early hours of the morning, which seems to agree with your assessment of idle time. And yes, the tuner completely disappears and it is only resolved with a hard power cycle (pulling the plug). After questioning Silicondust on this, and flipping on the logging, they indicate it is not a problem with the device - unfortunately they are never willing to share the log. I would say this happens about one to two times a week, and then of course all subsequent recordings that can't find the missing tuner also fail.

Thanks for sharing your solution, I'll definitely be checking it out. Because I have to maintain this server remotely, the Wemo switch should come in handy.

What firmware is the offending Prime on?

[jgsouthard]

Originally Posted by KryptoNyte

Yes, I have seen this exact problem. Most frequently in the very early hours of the morning, which seems to agree with your assessment of idle time. And yes, the tuner completely disappears and it is only resolved with a hard power cycle (pulling the plug). After questioning Silicondust on this, and flipping on the logging, they indicate it is not a problem with the device - unfortunately they are never willing to share the log. I would say this happens about one to two times a week, and then of course all subsequent recordings that can't find the missing tuner also fail.

Thanks for sharing your solution, I'll definitely be checking it out. Because I have to maintain this server remotely, the Wemo switch should come in handy.

What firmware is the offending Prime on?

My HDHRP's are running 20150826 firmware, which I believe is still the latest.

You might try just running the RestartTuners.bat script without buying WeMo switches or installing Cygwin (just comment out the cygwin64 commands), and see if that resolves the issue. It will still restart the tuners hourly, and it will log to the log file any cases where it can't see the tuner on the network (of course, the tuner will stay offline until manually power cycled).

In my case, it appears so far that just "touching" the tuners hourly (with a soft reset if they aren't in use) has been enough to keep them visible on the network. Although I've tested the WeMo setup and it is working correctly, and it gives me some extra peace of mind that I won't lose a recording, in three weeks of constant use I haven't seen a single new incident of a tuner needing to be power cycled.

By the way, I do believe that this is a tuner problem, probably with the NIC circuitry on the Ethernet port. The tuners otherwise seem to be up and running (all status lights normal), just not visible to the network. My experience is that when devices lose network connection in a way that requires a full power cycle (not just a soft restart), but are otherwise functioning normally, the issue is almost always a hung network interface.

[weeber]

I'm going to have to check this out. I've had my HDHR Prime drop off the network several times where only a hard reboot would fix it. It's not a regular thing, and haven't had a chance to dig into it, but I'm interested in trying this fix.

[Tiki]

Originally Posted by jgsouthard

By the way, I do believe that this is a tuner problem, probably with the NIC circuitry on the Ethernet port. The tuners otherwise seem to be up and running (all status lights normal), just not visible to the network. My experience is that when devices lose network connection in a way that requires a full power cycle (not just a soft restart), but are otherwise functioning normally, the issue is almost always a hung network interface.

Just a thought... is there any chance that this is due to the Ethernet switch you are using? I know many switches now are advertised as "green" where they put individual ports into a low-power mode if they detect they are not in use. Could the switch be putting the port to sleep and not waking up fast enough when there is activity?

[jgsouthard]

Originally Posted by Tiki

Just a thought... is there any chance that this is due to the Ethernet switch you are using? I know many switches now are advertised as "green" where they put individual ports into a low-power mode if they detect they are not in use. Could the switch be putting the port to sleep and not waking up fast enough when there is activity?

Thanks - actually that's a interesting idea that's worth pursuing. The switch between the tuners and the rest of the network IS a "green" switch (TRENDNet Green TEG-S80g). It will be both embarrassing and highly irritating if that's all it turns out to be, given the many 10's of hours that I've spent trying to resolve the issues.

I've run hours of network packet loss testing to make sure I didn't have a problem with the switch, and I replaced the NIC in the SageTV PC with a high-quality Intel NIC. But of course, none of that would adequately test for a timing race condition regarding awakening from sleep.

But I'm not sure that's what it is, since when one of the tuners drops from the network, no amount of pinging or other network activity (i.e. attempts to access the tuner) will bring it back to life. So it's not just slow to wake up -- it won't wake up at all. That could I suppose be an indication that the tuner itself has to hit the port with activity (i.e. activity has to come from the tuner side of the port); but if so that would be a ridiculous design limitation of the switch, since you'd never be able to awaken any sleeping device connected to the switch on a sleeping port.

Still, it's worth a few tests with another switch....

[CptKirk]

Batch file for the Prime reset (RestartTuners.bat) is hardcoded with your ID's.

for %%a in (1319784E,132277E9) do (

if %%a==1319784E set WeMoIP=192.168.1.200
if %%a==132277E9 set WeMoIP=192.168.1.201

To use on your own system replaced 1319784E,132277E9 with your own ID's.

Might be nice to add a comment in the batch file for other users.

[jgsouthard]

Originally Posted by CptKirk

Batch file for the Prime reset (RestartTuners.bat) is hardcoded with your ID's.

for %%a in (1319784E,132277E9) do (

if %%a==1319784E set WeMoIP=192.168.1.200
if %%a==132277E9 set WeMoIP=192.168.1.201

To use on your own system replaced 1319784E,132277E9 with your own ID's.

Might be nice to add a comment in the batch file for other users.

Thanks for the reminder. I intended to mention that and forgot.

If you are using WeMo switches for power control, you'll also need to set Static IP Mapping to the WeMo MAC address(es). You can't actually set the Belkin WeMo switches to a Static IP (they are DHCP only), and the script for controlling the switches can only talk to the switches by IP address; so to get things to work reliably you'll need to make sure your router always assigns the Wemo's the same IP address.

[jgsouthard]

A quick followup to my original post. Although my batch programs resolved the tuner problems I was having where a tuner would occasionally drop off the network, I continued to have problems with SageDCT using either FFMPEG capture (pixelation/skips) or DUMP capture (Comcast recordings that wouldn't play in SageTV but would play in VLC). A couple of days ago I moved to PrimeNetEncoder 2.0.0, but I found that it causes frequent No Signal / Halt errors with using 6 tuners in Windows 10 (two HDHomeRun Primes), but seemed to work OK with only 3 tuners (one HDHomeRun Prime).

So last night I switch to OpenDCT 0.4.14 beta. I'm very impressed with what EnterNoEscape and spencerw have accomplished in so short a time. Although I still have a few glitches, I think there is a future there and I'm going to keep playing with the properties.

But I'm finding that my RestartTuners.bat batch routine (to reset the tuners if not in use) is NOT working consistently with OpenDCT. I'm using an "hdhomerun_config.exe <HDHRP ID> get /tunerX/lockkey" command to check if the tuner is in use ("none" is the result if NOT in use, so I look for "none" for each of the three tuners in the HDHRP). But for reasons I do not yet understand, occasionally when OpenDCT is using a tuner it will return "none" even when the tuner is recording (not always, but sometimes).

So the result is that every few recordings will show a "Halt detected in recording error" at the exact moment that the batch file kicks off (which for me is at 50 minutes after the hour, each hour), and I will lose about 1-2 minutes of the recording while the tuner reboots and the recording restarts.

So until I figure this out, I do not recommend using RestartTuner.bat with OpenDCT, but only with SageDCT or PrimeNetEncoder. I either need to set the parameters in opendct.properties differently (related to locking the tuner) or I need a different test to check if the tuner is in use.

I don't have time to further research tonight, but I will work on this as I have time.

[EnterNoEscape]

OpenDCT should be locking the encoder every time. The first step before it even tries to tune the channel is to lock the encoder. The time that you saw it wasn't locked I think was when it was having issues with Prime because it couldn't reach it. The end result was that it tried to tune the channel anyway which resulted in the tuner being used, but not being locked.

[jgsouthard]

I've now moved to OpenDCT 0.4.16 Beta, and after 6-10 recordings today I haven't seen any more instances of RestartTuner.bat causing a recording glitch by rebooting a tuner that was being used for a recording. So at least for now I'm not going to pursue any changes to the batch file and instead keep using it as is.

[KryptoNyte]

I'd be curious to see if you can't just stop the scripts entirely with the use of OpenDCT and see if the halts still occur.

[jgsouthard]

Originally Posted by KryptoNyte

I'd be curious to see if you can't just stop the scripts entirely with the use of OpenDCT and see if the halts still occur.

Yep, me too. I may give it a try once I get OpenDCT stabilized and working solidly for a week or two. Otherwise if I get errors I won't know for sure where they are coming from.

[KryptoNyte]

I have noticed so far after about 3 days of clean use that there have been no halted recordings with OpenDCT, where I was seeing one to two halts a week with SageDCT - when the HDHR Prime Tuner disappears from the network.

Initially upon using OpenDCT, I was seeing a number of halted recordings, and then came to the conclusion that these were occurring on stations that required the tuning adapter(s), which I had removed on the switch to OpenDCT thinking they weren't needed in this smaller town (Charter). Turns out there are about 15 to 20 stations out of a 150 or so that are using SDV.

Any updates from other folks that were experiencing the halts prior to the use of OpenDCT? Still seeing the halts (missing tuner) once in a while? Still using the keep-alive script for the Prime?

One last question, has anyone ever had the Wemo switch power itself off once in a while for no apparent reason?

[jgsouthard]

Originally Posted by KryptoNyte

I have noticed so far after about 3 days of clean use that there have been no halted recordings with OpenDCT, where I was seeing one to two halts a week with SageDCT - when the HDHR Prime Tuner disappears from the network.

Initially upon using OpenDCT, I was seeing a number of halted recordings, and then came to the conclusion that these were occurring on stations that required the tuning adapter(s), which I had removed on the switch to OpenDCT thinking they weren't needed in this smaller town (Charter). Turns out there are about 15 to 20 stations out of a 150 or so that are using SDV.

Any updates from other folks that were experiencing the halts prior to the use of OpenDCT? Still seeing the halts (missing tuner) once in a while? Still using the keep-alive script for the Prime?

One last question, has anyone ever had the Wemo switch power itself off once in a while for no apparent reason?

I'm up to about 3 weeks of totally clean recordings on 6 tuners (2 HDHomeRun Primes) using the latest beta of OpenDCT. I'm very impressed. Installation and setup is also far easier with OpenDCT.

While both SageDCT and PrimeNetEncoder were valuable assets that have kept us going on SageTV with the newer tuners, I have recently not been able to get totally clean recordings with either of them. I got clean recordings with SageDCT for 2-3 years using the Dump capture method, but when Comcast started screwing around with file formats and I lost the ability to watch some recordings through SageTV using Dump, that's when things went south. I got halts with PNE and pixelation and skips with SageTV using FFMPEG capture.

I'm still running the keep-alive scripts with my Primes as extra insurance, but I don't really think they are necessary. I watch the logs for any indication that one of the tuners was not visible on the network and had to be power cycled, but it hasn't happened since I set the script up, even though it was happening once every day or two before that. But I don't know whether resetting the tuners every hour or so is preventing the network drops, or whether something about the change in network encoder contributed to that. But since it's working now, I've been reluctant to shut the scripts down and risk losing recordings.

Regarding the WeMo switches -- one of the biggest (and irritating) limitations of those switches is that if they lose power (e.g. get unplugged, power outage, etc.), they will come back up in the OFF state. So if a WeMo occasionally turns off, it may be seeing a momentary power glitch. I wish they had a setting to allow you to set the power-on state like you can with some PC BIOS; i.e. On, Off, or last state.

One of the reasons I have continued running the script is that it will set the WeMo's back to ON state if a power glitch caused them to turn off. By running the script once per hour, that should insure that I won't lose more than an hour of recordings if a WeMo turned itself off. I also have them plugged into a UPS to help minimize that.

[KryptoNyte]

Understood. Thanks for the update. On a sidenote, I have this SageTV server on an uninterruptible power supply, so when the Wemo showed up last Friday morning as "off," I was surprised.

[ahecht]

I know I'm reviving a long dead thread, but I updated the RestartTuners script to support devices other than just that particular model of HDHomeRun Prime. It should support any model beginning with HDHR with up to 6 tuners. I also simplified it so the device IDs and IP addresses are set up at the top, added a flag for enabling and disabling WeMo support, and improved logging for better debugging.

@ECHO OFF >NUL
setlocal ENABLEDELAYEDEXPANSION

::Set WeMo=True to enable WeMo control
set WeMo=False
::Set to the device ID of your first HDHomeRun Device
set HDHROne=12345678
::Set to the device ID of your second HDHomeRun Device
set HDHRTwo=23456789
::Set to the IP Address of the WeMo connected to your first HDHomeRun Device
set WeMoOneIP=192.168.1.200
::Set to the IP Address of the WeMo connected to your second HDHomeRun Device
set WeMoTwoIP=192.168.1.201
:: You should not have to edit anything below this line unless cygwin or the Silicondust software are installed in non-standard locations

set PATH=C:\cygwin64\bin;%PATH%
CD /d "%~dp0"

echo ................................................................... >> "%~dp0RestartTuners.log"

for /F "tokens=2" %%i in ('date /t') do set mydate=%%i
set mytime=%time%
echo %mydate%:%mytime% >> "%~dp0RestartTuners.log"

set TunerReset=

:: Check each HDHomeRun in turn
for %%a in (%HDHROne%,%HDHRTwo%) do (

	if %%a==%HDHROne% set WeMoIP=%WeMoOneIP%
	if %%a==%HDHRTwo% set WeMoIP=%WeMoTwoIP%
	set HDHROnNetwork=
	"C:\Program Files\Silicondust\HDHomeRun\hdhomerun_config.exe" %%a get /sys/hwmodel 1> "%~dp0TempFile.txt" 2>&1
	set /P HDHROnNetwork= < "%~dp0TempFile.txt"
	del "%~dp0TempFile.txt"

	if not !HDHROnNetwork:~0^,4!==HDHR (
		echo HDHomeRun %%a is not seen on the network ^(!HDHROnNetwork!^)>> "%~dp0RestartTuners.log"
		if %WeMo%==True (
			echo Power Cycling HDHomeRun %%a >> "%~dp0RestartTuners.log"
			c:\cygwin64\bin\sh.exe wemo_control.sh !WeMoIP! OFF 1>nul 2>&1
			c:\cygwin64\bin\sh.exe wemo_control.sh !WeMoIP! ON 1>nul 2>&1
		) else (
			echo WeMo control disabled, skipping power cycle >> "%~dp0RestartTuners.log"
		)
	) else (
		echo Found !HDHROnNetwork! %%a on network >> "%~dp0RestartTuners.log"
		set HDHRInUse=

		:: Check each Tuner on this HDHomeRun in turn
		for %%i in (0,1,2,3,4,5) do (
			set TunerInUse=
			"C:\Program Files\Silicondust\HDHomeRun\hdhomerun_config.exe" %%a get /tuner%%i/lockkey > "%~dp0TempFile.txt"
			set /P TunerInUse= < "%~dp0TempFile.txt"
			del "%~dp0TempFile.txt"
			if not "!TunerInUse!"=="none" (
				if not "!TunerInUse!"=="ERROR: unknown getset variable" (
					set HDHRInUse=!HDHRInUse!!TunerInUse!;
				)
			)
		)
	
		if "!HDHRInUse!"=="" (
			echo !HDHROnNetwork! %%a is not being used - Restarting >> "%~dp0RestartTuners.log"
			"C:\Program Files\Silicondust\HDHomeRun\hdhomerun_config.exe" %%a set /sys/restart self 1>> "%~dp0RestartTuners.log" 2>&1
		) else (
			echo !HDHROnNetwork! %%a is currently in use ^(!HDHRInUse!^) and will not be reset at this time >> "%~dp0RestartTuners.log"
		)	
	)
)

exit

[Bob F]

I have been having infrequent "Halt detected in recording" failures once in every month or less for some time. I discovered some time ago that stopping and restarting the SageTV OpenDCT service immediately corrects the problem. It just happened again, and after resetting it again, I checked the properties for that service and discovered the "Recovery" tab which offers actions that can be taken after failures. I set it to reset the service after multiple failures, hoping that this will solve the problem. We will see.

I would appreciate opinions on whether this was a good decision or if you see a better solution.

Using HDHR Prime tuner with SageTV