Sage Server with multiple Hard Drives: Write Delay Failed

[Steve52]

I have a SageTV server with multiple 320GB hard drives and 3 ATSC tuners. When I am recording from two or three tuners at one time and also viewing on clients I am getting windows Write Delay Failed Errors on the server and it crashes. Anyone else experience this problem. Fix?

[stanger89]

Sounds like you've got a drive going bad.

[NEOSG]

Originally Posted by Steve52

I have a SageTV server with multiple 320GB hard drives and 3 ATSC tuners. When I am recording from two or three tuners at one time and also viewing on clients I am getting windows Write Delay Failed Errors on the server and it crashes. Anyone else experience this problem. Fix?

Try forcing each tuner to use a different drive.

Here is a sample line from one of my encoders, specifically, the one labeled -1327961488/

mmc/encoders/-1327961488/forced_video_storage_path_prefix=\\\\IPADDRESS\\SHARE

Here is another example, using a drive letter path, specifically for encoder labeled 12346

mmc/encoders/12346/forced_video_storage_path_prefix=E\:\\


By using a different drive for each of my tuners, I have been able to record as many streams at the same time as I want, and never have an issue. During the early round of the hockey playoffs, and a night when new episodes of prime time shows were one, I think I had 6 or 7 things recording at one time. at least 4 of those were HD if not more

[NEOSG]

Originally Posted by stanger89

Sounds like you've got a drive going bad.

oh yeah, that too

[Steve52]

Originally Posted by stanger89

Sounds like you've got a drive going bad.

Maybe but my drives are all Western Digital, and I downloaded and ran their diagnostics program. It said all drives were fine.

An internet search turned up a lot on this. Microsoft gives several reasons most don't apply to me, but it seems there can be a problem on some systems with the page file size. May be something else of course, but this page file size problem crops up when you have multiple large volume drives in a windows system. I just wondered if anyone else had encountered this?

[doc]

I've only seen the delayed write fail error on network drives when the network was a bit dodgy - mainly duplex mismatch errors, when a drive is on its way out like already stated, or when a program is hogging the cpu, stopping other programs from working properly.

Have you checked the windows event logs for disk errors?

[Steve52]

Originally Posted by doc

I've only seen the delayed write fail error on network drives when the network was a bit dodgy - mainly duplex mismatch errors, when a drive is on its way out like already stated, or when a program is hogging the cpu, stopping other programs from working properly.

Have you checked the windows event logs for disk errors?

No, I will check the event log tonight.

I have never seen the CPU usage go very high actually it has always very low at the times I have checked.

My hard drives are all very new, and like I said showed nothing wrong when checked with the Western Digital diagnostics tool.

[flavius]

I have lost two WD drives recently, one of them brand new. No more WD for me.-

[briands]

Also might make sure the drives are all in UDMA. Windows likes to dump back to PIO if it detects too many drive errors.

[src666]

The manufacturer's diagnostics are actually pretty useless, unless the drive has already failed. It's a bit of an investment, but SpinRite (http://www.grc.com/intro.htm) is just about the best $89 you will ever spend. It does _serious_ drive diagnostics and maintenance.

As for "I will never by [x brand] again", you will find plenty of very unhappy people no matter what brand you look at. Maxtor/WD/Seagate/Hitach - all of them have had good drives and bad drives, as well as just the run of the mill drives.

Personally, I lose at least 2 drives every year. It's frustrating as all getout, but I've become used to it. This is in a house with 6 active systems and more than 14 drives running 24/7. Plan for failure - it will happen.

[stevech]

Sometimes I think that WD and the rest buy batches on the Asian spot markets. HP and Dell do.

The quality is inconsistent unlike days gone by.

Wish there were a "Lexus" of SATA hard disks- pay a bit more and be assured of quality.

[Thraxius]

Well the error that you are talking about was happening on a machine I was using for other things. What I did was turn off the write cache on the drive. I did not see a performance hit on the drive but it did become more active. (IE. Noisy)

[BobPhoenix]

I just had 3 Seagate's go at the same time in my Raid5 array of 8 drives - those three are just 3 months old others about a year. Up until now I've had good luck with Maxtor's. Going to try Seagates server drives as replacements since the Maxtor drives I had were not rated for 24/7 operation. I guess you could say I like to live dangerously. Luckily I still had last year's backup of the recorded content on another raid array so I only lost recordings since about Nov or Dec. Just wished I had copied the new files to the backup. Oh well it's just TV

[Opus4]

I had an old system that was regularly eating Maxtor drives. When I built new systems, I made sure there was a fan blowing directly on each HD cage & I haven't had a drive fail since, for almost the past 3 years I think. Well, at least not until I just said they were all working... I probably just ruined the record by saying that.

- Andy

[stanger89]

The only drives I've had go out on me are those that were in my server without (apparently) adequate cooling.

[BobPhoenix]

That could be the case but they were in this case - earlier version however. With a blow hole in the top and a side hole drilled to cool the processor. Also when I had the twelve hdd slots full in the bottom I put the overflow drives in these. Which have an 80mm fan to cool the 3 drives each. It could be the difference in drives 500gb which failed verses the 250gb which are running in another of these cases with 21 drives and no extra cooling holes - just fine I suppose. Personally I think I just have a bad drive or possibly a bad cable. The 9500S-12 card reported that each drive had a power on(down?) reset which threw it out of the array and lost the data to me. I thought about changing ports on the card to clear the error to possibly get back enough drives to complete the raid 5 array. I've done that before with the High-point that I replaced with the 3ware card and it worked. I just thought it wasn't worth the effort this time since I had 75% of the videos backed up anyway. As soon as I get the data restored I'll start on re ripping my DVD's which took up a majority of the space on the array again but I'm going to split my drives into multiple arrays instead of using the auto-carving function of the 3ware cards. I'm using the 5 drives that I've had for about a year and didn't "die" as that array. I'm also going to get some different cables and see if that will help. Note I don't expect to find the drives permanently gone. I figure when I hook them up to a different controller or maybe even to different ports on the same controller I will get them back. I've already got 3 drives for off line storage now - that's where I'm going to put the backups from now on - just hadn't transferred the videos to them yet. I just use this and remove the drive to a shelf for my backups - so it gets much less use and I figure should last much longer. I still believe in Seagate (it wasn't Maxtor drives that went bad) drives it's just that these are not designed to be 24/7 drives so probably will fail sooner. I just wanted to point out that it isn't only Western Digital that has problems. I have 12 250gb Western Digital drives on another 9500s-12 and they are working just fine - also in the same case and the same extra holes for additional air.

BobP.

[Steve52]

Originally Posted by doc

I've only seen the delayed write fail error on network drives when the network was a bit dodgy - mainly duplex mismatch errors, when a drive is on its way out like already stated, or when a program is hogging the cpu, stopping other programs from working properly.

Have you checked the windows event logs for disk errors?

The windows event log for the system showed lots of sourceISK errors. It also showed event:11.
An internet search indicates this could be a lot of things, but certainly there is some kind of disk writing problem. It seems however to only crop up when recording from three sources at the same time with Sage. Turning off cache writing did not seem to stop the problem.
I am in the process of testing the drives by copying the entire drive to another blank drive I know is good from another computer. I already ran diagnostics which showed no error.

[Polypro]

WD...cuz 4 Pin Molex connectors for power blow the doors off SATA Power connectors (I had to tape the wire to the bottom of a Maxtor DiamondMax 10 drive, putting downward pressure on the SATA power connector to get it to power up).

I'd just replace the drive, its easier when it's on your terms (When no recordings are scheduled)!

P

[Steve52]

When I started this thread I didn't know if this was a hardware or software problem so I posted it in the general discussion section. It got moved to the hardware section when everyone started pointing to hard drives as the possible problem. Well after a week of testing I now know what the problem was. I have 4 hard drives in this server. None of them are bad. The problem is a faulty SATA port/channel on the motherboard. All the other ports work fine, just SATA port 1 is bad, and to make the problem even harder to find you can read thru the port without problems you just cannot write reliably. Small files will write without problem, extended hard drive diagnostics will run and report no errors, but if you try and copy 200 to 300 GB of video files it will crash. So when Sage was trying to write a video file to a hard drive connected to SATA1 it would crash the computer. Just thought I would end this thread with what the problem really was. Might help someone in the future.

[Polypro]

Had that happen on a K8N Neo2 Platinum. Glad you solved it.

P