SageTV hang. Seems like prioritizing write streams over read stream will fix?

[easperhe]

All,

SageTV V6.6 will hang sometimes when I skip in one recording while recording that program or other programs. I've been using SageTV 6.x for a couple years now with two firewire tuners (comcast STBs) and one clear QAM turner for HD recordings and playback.

Quick Summary (see more detail section below if needed)
------------------
I believe the hang is due to the SATA II drive bandwidth (WD Green drive: WD6400AACS or WD black drive: WD7501AALS) being saturated from the skipping ahead in the recorded program. The high bandwidth read stream is blocking the write stream long enough to lead to an overflow condition in the write stream/thread(s). Resource manager shows a dramatic increase in IO bandwidth (from ~10MB/s to ~70-90MB/s) when skipping through the recorded show. This gets very close and likely bumps into the sustained data transfer rate of ~100MB/s for these drives.

Since Vista/W7 allows for apps to prioritize IO, this SageTV hang issue would likely be solved (or greatly reduced the likelyhood of) if SageTV would prioritize the write stream/threads over the read stream/threads.

Questions
------------
1) This looks like the issue given the dramatic increase in disk bandwidth when skipping ahead in a show and that the hang always occurs on a skip. Does this make sense to SageTV experts or is there another explanation?

2) If this does make sense, can SageTV (or some 3rd party app) increase the priority of the write streams/threads?

3) I just upgraded to SageTV V7. Any reason to believe this issue is solved there? Resource manager shows all SageTV read/write threads with "normal" IO priority. So, I wouldn't think anything would change in V7.


More detail
-------------
When I run resource manager and have all 3 HD tuners recording college football games and playing back a recorded football game, I see that each IO stream (3 write and 1 read) takes ~2.4 MB/s of disk bandwidth for a total of ~10 MB/s. This is well within the ~100MB/s sustained data transfer rate that I see is typical for SATA II drives. But if I start to skip ahead in the recorded show, I see the read IO stream bandwidth skyrocket to between 70-90MB/s. Along with the other 3 write IO streams, this is getting close to (and probaby bumping up against at times) the max SATA II sustained data transfer rate of ~100MB/s. So it seems that the skipping through the HD recording is saturating the SATA II disk IO so that one of the write streams overflows and losses data and causes something to barf and hang SageTV. This makes sense as I don't think I've ever seen SageTV hang/crash like this when I wasn't skipping through a show WHILE recording a show.

Since Vista/W7 allow for apps to prioritize IO, this issue would be likely solved (or greatly reduced the likelyhood of) if SageTV would prioritize the write stream/threads over the read stream/threads.

[Savage1701]

ea:

Here's my experience, FWIW, in regards to each of your three questions.

1. I can reproduce that under your same scenario, using an HD200 or clients on Vista or Win 7. I minimize it by using a RAID controller with cache onboard to help buffer those events, I treat most of my disks as "Single Disks" under RAID except for a 3 in RAID 5 that I want for preserving shows. Obvious questions - Are you using NCQ if available? Are you enabling disc caching from the disc's "Hardware" properties tabs? I also use SuperSpeed's SuperCache RAM caching software. You can get a 14-day free trial. It lets pages sit "dirty" in RAM for a drive, but you can use some system RAM to make a "fake" RAID-type buffer or augment the one you have and increase I/O.

2. Please see above. I think I packed this question into the above response. I'm guessing the green drives are going to have the worst I/O capability. I have also used Disktrix software to defrag the video drives, but ONLY when not recording, say, late at night. Keeping the files sequential and at the outermost parts of the platters helps. Also, format your drives as 64KB blocks for video. It helps. Most WD Green drives have poor I/O. Also, if using XP or Vista, and your drives are Advanced Format, make sure you have them set to be compatible with XP/Vista's 512-byte sector limit. Only Win 7 can handle 4K sectors, not to be confused with NTFS formatting choices. And even if they are set to be compatible, many seem to think they still are not as competent as regular 512-byte sector drives since they have to translate "on the fly" for XP/Vista.

3. I can tell you the opposite - I just upgraded the FW in my HD200 which rarely skipped until I just did the upgrade. I run Sage Server 6.X, and missed the fine print that suggested it only be done for Sage 7.X server. It is now sluggish and far more prone to "halts" and such, and has problems playing DVD's that played fine on the last FW version. Nothing else in my system changed and I don't use my server for anything else but video. I also don't allow any Windows updates unless I have checked them, and none of my other hardware in the signal chain changed. Intersetingly, my 2 Sage 6.X clients, one under Vista and the other under Win 7, are not sluggish or prone to skipping or prone to problems with DVD playback (example: Iron Man 2 no longer defaults to ANY soundtrack on my HD200 and has to have the first scene manually selected to play to have the movie have sound. Never happened until I upgraded the FW on the HD200). Point being, can I prove a cause/effect on my FW update and HD200? No. But the correlation is about a perfect 1.0. And it's one of those things you can't control.

[blade]

I have no clue what's causing your problem, but I would doubt it's a bandwidth issue because I've used more tuners with much slower drives and have never experienced the problem. Currently I have 4 QAM tuners, 1 HD-PVR and 2 Analog SD tuners and don't have any problems with Sage pausing when I skip forward while watching a show that is in the process of being captured.

Most nights I have at least 4 HD tuners in use while I'm watching and skipping forward. I'm using WD Green drives and currently all streams are being captured to 1 of the drives because it's new and has the most free space. Before I got the green drives I had some older SATA II drives that were much slower than the green drives and they never had problems either.

Improvements were made to v7 when it comes to writing streams. The recommendation to format using 64K blocks was even removed. Also there is a setting to optimize for performance instead of space where Sage will attempt to spread recordings across multiple drives instead of to the one with the most free space.

[easperhe]

Thanks to Savage1701 and blade for the feedback.

Savage1701 - I'll definitely check if NCQ and caching is turned on. I understood that recent drives use NCQ and caching. But I just expected that these performance enhancements would be enabled by default. And, I've never looked into using RAID. And, I look into some of your other suggestions if NCQ/caching are already turned on.

blade - interesting feedback. Potentially good news that a system with more HD tuners than my own is not seeing SageTV hang when skipping forward. A few things to note.
* I have firewire tuners (comcast STB) and the associated driver developed by ExDeus. It possible that STB driver's buffering is not as robust as consumer QAM tuner solutions. If that's the case, my system may be more likely to fail under IO bandwidth scenarios. BTW - really hoping SageTV will work with Ceton InfiniTV 4 (or the like) in the near future.
* I will also try the optimize for bandwidth. But given your experience, this doesn't appear to be necessary.

However, I still believe it would be better if SageTV application prioritized write/record streams over read/playback streams.
* A recording stream that cannot get to disk in time will be overwritten, i.e. data will be lost. This leads to corrupted recordings (at best) or a hung system (at worst).
* A playback stream that gets delayed will only delay the displaying of the video. A slight performance hit which is much better than bad recordings or a hung system.

Do you (or others) disagree? If not, I'll file an enhancement request. It may not be needed for some systems (like blade's) but I don't see how it would hurt anything and might help others sytems (like mine).

[easperhe]

All,

Wanted to provide followup hard drive info after investigating this more.

* On my drives, the HD cache and NCQ are not controllable via device manager. So, these are always being used.
* What is controllable via device manager is whether to enable/disable disk write caching. When enabled, the disk cache operates as a "write-back" cache. When disabled, the disk cache operates as a "write-through" cache. write-back operation is somewhat better for performance but has a risk that data could be lost if the power goes out (since some of the data in the disk cache has NOT yet been written to the drive)

* I did change my SageTV option to optimize for bandwidth (so that multiple recordings are sent to separate hard drives).

In the end, I still think prioritizing write streams over read streams in SageTV application would be a good thing to do. Optimizing for bandwidth may help reduce the probability of this failure. But, I don't think it's a full proof fix. I'll file an enhancement request.