Some questions about RAID

[jbarr]

I have an ASUS motherboard that has 5 SATA connectors and hardware RAID capabililty. I currently have one 80GB IDE "OS" drive connected, and one 320GB SATA "recordings" drive connected. I would like to add 2 more 320GB SATA drives and convert the 3 SATA drives to a RAID 5 array (leaving the IDE drive untouched.)

What's the best approach to do this without affecting the current content? I don't want to lose any of the existing content on the 320GB drive, but I want that drive (and the contained data) to ultimately be included in the RAID 5 array.

Next, once I have the three SATA drives successfully configured as a RAID 5 array, what are the implications of later adding a 4th or 5th drive? Can I just seamlessly integrate them into the existing array without risking losing data?

Finally, what about Windows drivers? Do I need to be concerned if I re-install Windows?

I know these questions are outside the scope of SageTV, but I hope that some fellow SageTV gurus have encountered this...

Thanks!

Update: I just realized that I cannot have a RAID 5 array with just 2 drives, so to do it right, I guess I would need a total of 4 drives (one to temporarily hold the existing data), create the RAID 5 array on three drives, move the data from the fourth to the array, and then integrate the 4th drive into the array. Hmmm.

[stanger89]

Originally Posted by jbarr

I have an ASUS motherboard that has 5 SATA connectors and hardware RAID capabililty. I currently have one 80GB IDE "OS" drive connected, and one 320GB SATA "recordings" drive connected. I would like to add 2 more 320GB SATA drives and convert the 3 SATA drives to a RAID 5 array (leaving the IDE drive untouched.)

One thing to clarify, RAID on desktop motherboards is (AFAIK) never hardware RAID, it's either firmware of driver RAID. Basically it's a standard HDD controller with a special bios.

Constrast that with true HW RAID which has a dedicated RAID processor.

What's the best approach to do this without affecting the current content? I don't want to lose any of the existing content on the 320GB drive, but I want that drive (and the contained data) to ultimately be included in the RAID 5 array.

There are a couple ways depending on your controller's capabilities. The safest is to build a new RAID array and copy the data to it.

But if your controller supports RAID Level Migration and Online Capacity Expansion, you should be able to install the drives, and first convert the single drive into a 2-drive RAID-1 Array, (using one of the new drives) and second, Migrate the RAID-1 array to a 3-drive RAID-5 array.

Next, once I have the three SATA drives successfully configured as a RAID 5 array, what are the implications of later adding a 4th or 5th drive? Can I just seamlessly integrate them into the existing array without risking losing data?

Only if the controller supports Online Capacity Expansion.

Finally, what about Windows drivers? Do I need to be concerned if I re-install Windows?

Not for true HW RAID, probably not for driver RAID like this either.

[rickgillyon]

Originally Posted by stanger89

One thing to clarify, RAID on desktop motherboards is (AFAIK) never hardware RAID, it's either firmware of driver RAID. Basically it's a standard HDD controller with a special bios.

Is this correct? I know that on all my desktop boards there is a dedicated chip for the RAID (e.g. Silicon Image, ITE, NVRaid).

[stanger89]

None of those are real/full hardware chips, Intel, 3ware, LSI are the only ones I can think of that are real hardware RAID chips (Areca uses Intel processors IIRC).

Price is a very easy way to tell if it's a full hardware RAID controller or not.

[stevech]

Isn't it true that current motherboards have true hardware RAID-0 and RAID-1 built-in?

From all I've read, for safety backup of hundreds of GB of mpeg files as produced by Sage, the best way to go is RAID-1, two-drive mirroring. RAID5 on affordable disks has a fairly low write speed due to the striping.

And with 500GB drives at $120 ea now, a two-drive RAID-1 at least protects from losing all the video on a failed drive. If I outgrew 500GB, I guess I'd add another pair in RAID-1. We average every day a 300GB total of videos, and we don't archive movies. Can't imaging having a 500GB backlog, unless we start storing movies on hard disk. (just not many movies we care to keep).

[stanger89]

Originally Posted by stevech

Isn't it true that current motherboards have true hardware RAID-0 and RAID-1 built-in?

That's a little trickier determination since with RAID-0 and RAID-1 there isn't really anything to do (no calculations) so, especially for small numbers of drives, it doesn't really matter, but I guess I'd say no, they're just normal controllers with special firmware/drivers.

From all I've read, for safety backup of hundreds of GB of mpeg files as produced by Sage, the best way to go is RAID-1, two-drive mirroring.

First things first, RAID is not backup, RAID is fault tolerance, it only protects you from hard disk failure. As for the best way, it really depends. If you've only got 1-2 drives of media you want fault tolerant, then RAID-1 on a cheap controller (or software RAID-1 even) is a very viable option.

However once you start getting into 3 drives of data, the costs of the extra HDDs outweighs the cost of a good RAID controller.

RAID5 on affordable disks has a fairly low write speed due to the striping.

True and myth at the same time. An n-drive RAID-5 array is slower at writes than an n-drive RAID-0 array. But on a good controller (Areca/3ware) an n-drive RAID-5 array is about as fast as an (n-1)-drive RAID-0 array for writes.

[Osoksniper]

I use raid a lot, there are several options however, to convert to raid 5 you really need to go in steps...first to raid 1 and then on to raid 5. The best raid controller for sata drives on the market, is probably the oldest one too...ide that is, is manufactured by Promise...it has true 3gb/s across mutiply drives and actually sees an increase in speed over the usual single drive configuration...you have an 80 gig drive right?

Here is one possibility and I did it for someone recently...take the 80 gig drive and go to raid 1 with one of the three 320 gig drives...most raid 1 controllers will equalize the two drive sizes to the smaller of the two...they will mirror...once they are done...take out the small 80 gig...put in another of the 320 drives...let them mirror again to raid 1...now you will have 320 gig and a raid 1 array in tact...then you can put in the 3rd 320 gig drive and convert the raid array from raid 1 to raid 5...most controllers will allow this although, this will take several hours....all the calculations needed internally to do the redundancy checks and write the data take a while...however, when done...it all worked for me...

[stanger89]

Originally Posted by Osoksniper

I use raid a lot, there are several options however, to convert to raid 5 you really need to go in steps...first to raid 1 and then on to raid 5. The best raid controller for sata drives on the market, is probably the oldest one too...ide that is, is manufactured by Promise...it has true 3gb/s across mutiply drives and actually sees an increase in speed over the usual single drive configuration...you have an 80 gig drive right?

Not really, the best controller on the market at the moment, is probably the Areca, which one specifically depends on how many drives you need. The Areca cards scale linearly with drives, and offer (n-1)x performance (the theoretical max for RAID-5.

[chrisc16]

Originally Posted by stanger89

Not really, the best controller on the market at the moment, is probably the Areca, which one specifically depends on how many drives you need. The Areca cards scale linearly with drives, and offer (n-1)x performance (the theoretical max for RAID-5.

Stanger,

Would the Areca ARC-1210 be one of the good controllers? I was thinking of getting one of these, along with my 4x320GB SATA drives, for a 900GB RAID5 recording drive. There have been a few occasions when I've been recording 2 simultaneous HD shows, while trying to watch a 3rd recorded HD show, and my drives weren't able to keep up. This was a rare occasion, though, when all 3 shows were on one drive.

-Chris

[stanger89]

Originally Posted by chrisc16

Stanger,

Would the Areca ARC-1210 be one of the good controllers?

Yeah, I think any of theirs are good. In fact my friend got one of the larger ones recently, and is very happy.

I was thinking of getting one of these, along with my 4x320GB SATA drives, for a 900GB RAID5 recording drive. There have been a few occasions when I've been recording 2 simultaneous HD shows, while trying to watch a 3rd recorded HD show, and my drives weren't able to keep up. This was a rare occasion, though, when all 3 shows were on one drive.

-Chris

I don't know if a RAID array will really help that. I'm rather surprised one HDD can't keep up with 3 HD streams, they're really not that demanding.

[chrisc16]

Originally Posted by stanger89

I don't know if a RAID array will really help that. I'm rather surprised one HDD can't keep up with 3 HD streams, they're really not that demanding.

Hm, well maybe my drive is too fragmented, and the heads were bouncing all over the disk. I've never had much luck defragging my 64K drives. Or maybe my nForce4 sata controller is a piece of junk. But I've definitely had disk-throughput related problems ever since I started recording HD files. Sometimes Sage even has to break the files into two segments. I guess I was kinda hoping that throwing $300 at the problem by moving to RAID would eliminate disk/controller speed as a potential culprit.

-Chris

[nielm]

My thoughts on Raid vary from day to day...

Raid 0 (striping) No thanks.
Fast, but no data integrity. MTBF is less than half that of a single drive (because a controller/MB failure would also kill your data)
I dont feel like losing several hundred Gb of data, even if it is 'only' recorded TV or ripped CDs/DVDs..

Raid 1 (Mirroring)
ok, Slow write, fast read, and at least my data is secure from single disk failure... but I am using twice as many disks! What a waste of space!... A simple automatic backup script would be just as good (perhaps better as inadvertant deletions could be recovered, and I could only backup things I really want to keep)

Raid 5 (Striped/mirrored 3disks gives 2disks worth of data)
hmmm... Not sure about this one. Ok so my data is safe from a single disk failure, but what about a controller/MB failure? Can I unplug the disks and use them in another machine? Unlikely (as "RAID data configuration is almost always proprietary")...

(Note I have un-Raid-1'd my machine by simply unplugging the disks and plugging them in non-RAID ports. They got recognised as 2 normal disks).

So I end up with a backup script, but still think from time to time about going RAID again!

BTW, a second way to tell if MB RAID is full HW or SW is if it lists CPU/Memory requirements, which pretty much means that its done in software... (My SI 3114 says that I need 1GB of RAM to do RAID 5, but does not list any RAM requirements for 0/1/0+1)

[wvpolekat]

Some very good and knowledgeable comments here.

The most important one is that RAID is NOT a backup strategy. It is fault tolerance.

A mirrored set of drives does not mean your data is safe. Only protected from hardware failure.

I put alot of thought into this over the last week or so as I thought about storage for my box.

Here is the conclusion I came to for my situation:

I will not use any RAID at all. I do not need more than 300 GB of continuous space, so RAID 0 or 5 doesn't buy me much of anything.

As nielm pointed out, RAID 1 can be seen as a waste of space.

What I am going to do is to just run my drives in normal mode, determine what content needs to be backed up and just script backing that up nightly. I do not see the need to make backups of ripped DVDs as I am willing to re-rip them if needed at this point. The time may come when I will back them up as well.

RAID can introduce a whole host of possible problems on consumer hardware. Unless you need LOTS of continuous space, the risks probably don't outweigh the benefits.

If you are willing to spend the money on a real/proper RAID controller, the situation changes significantly. The HP/Compaq SCSI RAID controllers I am used to dealing with at work are a good example of this. You can easily change out any part of it with no data loss or worries about if your array will be seen. But, they cost as much as most people's Sage server, not to mention the drives.

Now, on performance.... I can saturate a 100 mb network with a regular old ATA 100/133 drive without much trouble. IMO, the smarter approach to I/O performance for a situation like this would be to put your data on drives in such a way that you use as many spindles as possible. Meaning, don't put your DVDs on the same physical drive as your recordings, move your recordings to another drive after they are complete, things like that. As cheap as large drives are, this is pretty reasonable to do.

[stevech]

wouldn't RAID-1 do the mirroring down in the controller and thus be a lot faster than an application doing backup dupes - all that data has to percolate twice in the driver pile, to the application buffers, to be written to a different disk, and two trips through the file system.

Do I correctly assume that cheap RAID-1 controllers (even the ones built-in) take the same data from the operating system and write it to two drives, transparently to the file system and applications? If not, what's the point of claiming the hardware is RAID-1?

[wvpolekat]

The problem(s) with a mirror for me are

1) It's inefficient use of space. It takes 1TB to have 500GB of storage. This is ok if you are running near capacity because a backup will take just as much space, which is effectively the whole drive.

2) It's not protecting you against anything other than hardware failure. You are still just as vulnerable to accidental deletion and such. As my wife about the great MP3 disaster of 03.....

My decision was based on what impact a hardware failure would have on the usability of the box. Worst case, I could flip over to using shares to another machine or transplant the drive until I got a replacement.

So, in case of hardware failure, my Sage box may stop recording until I catch it and I am willing to trade that for the ability to recover something that is corrupted or accidentially deleted.

[stanger89]

Originally Posted by wvpolekat

RAID can introduce a whole host of possible problems on consumer hardware. Unless you need LOTS of continuous space, the risks probably don't outweigh the benefits.

If you are willing to spend the money on a real/proper RAID controller, the situation changes significantly. The HP/Compaq SCSI RAID controllers I am used to dealing with at work are a good example of this. You can easily change out any part of it with no data loss or worries about if your array will be seen. But, they cost as much as most people's Sage server, not to mention the drives.

Agreed, but you don't need to go SCSI to get "real/proper" RAID controllers. 3ware, Areca, LSI all make excellent SATA (even IDE), full-hardware RAID controllers.

[wvpolekat]

Oh, SCSI is not a requirement, it's just a reference to what I am used to. Even that is going away now, we just boot off SAN with no local drives allowed in the datacenter anymore.

[stanger89]

It was more a comment on cost than anything. SCSI equipment is very expensive (I think it still is at least). Good SATA/IDE RAID controller's aren't cheap, but they aren't outrageous either.

[stevech]

Originally Posted by wvpolekat

The problem(s) with a mirror for me are

1) It's inefficient use of space. It takes 1TB to have 500GB of storage. This is ok if you are running near capacity because a backup will take just as much space, which is effectively the whole drive.

2) It's not protecting you against anything other than hardware failure. You are still just as vulnerable to accidental deletion and such. As my wife about the great MP3 disaster of 03.....

My decision was based on what impact a hardware failure would have on the usability of the box. Worst case, I could flip over to using shares to another machine or transplant the drive until I got a replacement.

So, in case of hardware failure, my Sage box may stop recording until I catch it and I am willing to trade that for the ability to recover something that is corrupted or accidentially deleted.

But with 500GB for $120 now, I don't care. A drive failure is inevitable and I'm not protected.
Copying big mpeg files at the application level seems way too slow, esp. if you have to use a LAN, or worse, like me, as for now the SageTV server connects by WiFi at 54Mbps = 20Mbps at the IP layer = l-o-n-g transfer times. To precude an Oh-oops delete, maybe we have to do these with an application program, as a special case, for those especially valuable files. Lastly, I am unconvinced that Sage will avoid goobering up (corrupting) the wiz.bin during and after a drive failure. Indeed, so far for me, Windows usually blue-screens on the failure of a non-boot drive. I keep the last 14 copies of wiz.bin now, having been burned twice (now using SecondCopy to automate this).