Massive Storage Solution(s) -- The search for a fast/expandable/HUGE storage solution

[taylork]

Hi all. I've been lurking/reading threads for a while and thought it time to start a new thread as I haven't found a definitive discussion that solves a problem that we will all run into at some point in time -- what I'm wanting to do is...

To create a pseudo step-by-step guide for the creation of a network/server-based massive storage solution. A box that provides:

1. Huge storage capabilities (8TB+ ... w/ 20TB+ being doable)
2. Redundancy
3. Ease of management and setup (Not something that will be easy, but if we get enough feedback and people contributing, this is doable).
4. Strong read/write performance (the ability to saturate a GigE link)
5. Ability to increase your storage with as little pain as possible
6. Low cost.
7. ?? (Saved for something I've missed that someone will certainly point out)

Now, how to tackle such a beast...

For starters, I'm thinking a Linux-based software-RAID box using SATAII drives and port multipliers (PMP).

The various Linux-smart users in this thread are discussing a solution that looks solid:
http://forums.sagetv.com/forums/showthread.php?t=25709

Why Linux/software-RAID/SATAII/PMP's? Good question:

1. Linux is free.
2. It does RAID 5, which provides redundancy.
3. Software RAID 5 in Linux is FAST (go read mikesm's performance numbers in the thread above -- it blows Windows software-based RAID away and gives hardware solutions a run).
4. Linux software RAID is flexible (grow arrays, move them, etc).
5. AHCI SATA ports, coupled with PMP's give the potential for tons (TONS) of growth -- every SATA port on your motherboard can connect to a PMP, which provides connections to 5 drives. So, 4 motherboard SATA ports = support for 20 drives. 6 SATA ports = 30 drives, etc.
5. PCIe ports allow for further expansion via eSATA controller cards that are PMP compatible... so for every eSATA port, you can get another 5 drives.

Obviously there are caveats here, but just to wet your whistle, just look at the potential of this motherboard I have sitting here, a GIGABYTE GA-EP35C-DS3R:
http://www.newegg.com/Product/Produc...82E16813128082

8 onboard SATA ports x 5 drives each (via the use of PMP's) = 40 drives
+ 4 PCIe ports that would support controllers w/ 4 eSATA ports each x 5 drives per eSATA port = 20 drives per PCIe port
= Support for 120 drives

Now obviously this is theoretical and there are caveats to this, but I think the point is made.

Potential Advantages/Issues:

1. Linux: I (like most here, I suspect) am a Windows user (not by choice... my god how I loathe Vista)... but Windows is what I know. (Un)fortunately, Windows, as far as I know, doesn't offer a viable solution for us for this storage issue because it would require the purchase of an expensive RAID controller, which removes the expansion possibilities offered by port multipliers (I haven't found a PMP compatible hardware-based RAID controller that would suit the needs of this project).
2. Cost: The good thing about this solution is that you can start small -- 3 drives in a box, and add to it as needed. And as you grow your storage, you can grow your network, etc. as needed.
3. More than just a NAS: I would prefer something that can do more than just act as a file server (ie., assisting w/ commercial skipping processing, etc.) -- but obviously each user could tailor their solution.
4. ....??

Ok, I'll stop here so we can get the discussion going... because I suspect that this is a hot issue w/ many users.

[mikesm]

I'm not sure why you want to start another thread when the old one is still the best for info, but in any case, that gigabyte P35 motherboard is not what you want to build a software based NAS on. While the ICH9R southbridge may support PMP's, it does not support FIS switching, which kills performance for port multipliers.

As I think I posted in the other thread, the type you want (and the one I plan on upgrading to) is the Intel P45 series, which comes with the new ICH10R southbridge. This supports FIS switching for PMP's, and Tejun's libsata patches which are now in the mainline kernel should recognize them and use FIS switching. This configuration should not have the 120 MB/s limitations the SI3132 based controllers seem to have, even though they do support FIS switching.

The other thing is making sure you have an Intel or Marvell based gigabit NIC onboard, as they are much better for NAS functions that the realtek NIC's found on many boards (including the P35 board you mentioned). You can always burn a PCI-E slot on a discrete NIC though.

[jvl711]

I have spent some time thinking about this exact problem over the past week. I personally think that direct attached storage to the Sage Server is the best route to go.

I do not see a reason to use raid 5 for the recording drives. I see these drives as more scratch pads. If there was a failure I personally would only loose a handfull of recordings. The more single drives for recording the better. Raid would most likely just complicate this.

Library content is a different story. This could benift from redundancy. I think a mirror would be sufficient if there is drive space to spare, because drive performance is not a big issue here. The number of streams coming from these drives would most likely not be the same as the recording drives.

Here is one of my proposed sollution:

I have been looking into getting a 4 port eSata card that supports 0, 1 and 1+0 in hardware. Here is a card that I think meets this need. $89

http://www.newegg.com/product/produc...82E16816318004

Then I was looking into finding an afforable external enclosure. $223

http://www.newegg.com/product/produc...82E16816855007

This gives me a 4 drive sollution with the ability to add upto 12 more drives, at full attached storage speed for under $400.

[stanger89]

Here's a question, is there anything under linux that works along the lines of unRAID? Mainly redundancy without striping?

[Jabroni]

I think a USB2 enclosure solution is more douable and less trouble making.. i think theres like 4 hdd raid cases that are usb2.. maybe having many of those hooked to your sage box...

[mikesm]

Originally Posted by stanger89

Here's a question, is there anything under linux that works along the lines of unRAID? Mainly redundancy without striping?

Freenas is BSD based, but similar to unraid. Openfiler is more appliance oriented and linux based. It supports raid, but of course if you want to make 20 seperate volumes and export them all, that will work too.

You can always build a config with linux that does not use software raid, and just configure it as a bunch of separate volumes. Just because all that horsepower is there, it doesn't mean to have to use it.

[mikesm]

Originally Posted by Jabroni

I think a USB2 enclosure solution is more douable and less trouble making.. i think theres like 4 hdd raid cases that are usb2.. maybe having many of those hooked to your sage box...

You can also get eSATA enclosures (they typically come with a PMP in them) that you could plug into a cheap SI3132 based controller, and each disk will show up in windows as a discrete volume. They are way faster than USB, and work pretty well. Under windows I would stay away from software raid though.

[taylork]

As I PM'd to mikesm: The main reason for a new thread was to provide a central place to focus on a solution of this nature -- generating as much step-by-step info as possible for those of us light on Linux/hardware requirement knowledge, as I'm certain there are a ton of us.

For example, I had seen port multiplier Command-based vs. FIS-based Host support discussed here:
http://www.sata-io.org/portmultiplier.asp

But wasn't sure how to determine which chipset(s) supported FIS. Intel P35 doesn't, but P45 does (thanks for the heads-up, Mike!). Apparently ICH10 support is required for FIS switching...?

Are there any boards that are currently available that handle FIS switching?

jvl711: I've also pondered whether I should keep things simple for direct Sage recording (i.e, just use a few standard hard drives for recording and moving things over to a RAID set for permanent storage). But then again, if you're going through the trouble of building a tanker for storage, why not record directly to it if possible? (And obviously this certainly depends upon other use... if you've got folks constantly hitting the server for files, it might not be feasible for it to keep up w/ stream-recording too.)

Also, that Addonics card is software raid, so your performance probably wouldn't be capable of direct stream-recording.

[stanger89]

Originally Posted by mikesm

Freenas is BSD based, but similar to unraid. Openfiler is more appliance oriented and linux based. It supports raid, but of course if you want to make 20 seperate volumes and export them all, that will work too.

That's the thing, I don't want just a bare NAS, that's the reason I won't consider unRAID, it's too limited (can you run SageTV, or Squeezecenter on an unRAID?). But I like the unRAID theory of basically separate discs with redundancy. I'm way out of the loop on Linux, I know LVM will allow you to build a single volume over multiple discs, is there a way to add redundancy/parity calculation to that.

If you go with LVM, will only used disks be spun or does the whole volume get spun?

I really just can't decide which way to go for my server redux, at the moment I keep wavering between just drives, and full-up 3ware 9550SXU-8 RAID-5.

[mikesm]

Originally Posted by stanger89

That's the thing, I don't want just a bare NAS, that's the reason I won't consider unRAID, it's too limited (can you run SageTV, or Squeezecenter on an unRAID?). But I like the unRAID theory of basically separate discs with redundancy. I'm way out of the loop on Linux, I know LVM will allow you to build a single volume over multiple discs, is there a way to add redundancy/parity calculation to that.

If you go with LVM, will only used disks be spun or does the whole volume get spun?

I really just can't decide which way to go for my server redux, at the moment I keep wavering between just drives, and full-up 3ware 9550SXU-8 RAID-5.

If you aren't used to working in Linux, I would recommend going with a hardware controller in windows. It's just not worth the time for most people. It's not that hard to deal with, but I don't think it's worth the time. The hardware based controllers have their issues and cost a lot more too, but I think it's a better bet for windows only folks.

Unriad does this funky kind of copying for redundancy. Personally, I don't understand why it's better than RAID1, but you can certainly run things like rsync or other script based tools to sync files across various volumes if that's what you are looking for.

LVM has striping available that's separate from RAID striping, but not redundancy as typically found in RAID5 or RAID6, or at the filesystem level with Solaris' ZFS.

What's the issue with RAID5 if you want redundancy? It's less work to administer than syncing things in linux, but I must be missing something...

[bcjenkins]

The unRaid distro can be adapted. Also, I understand people are running his stuff on other distros.

B

[jvl711]

Originally Posted by taylork

As I PM'd to mikesm: The main reason for a new thread was to provide a central place to focus on a solution of this nature -- generating as much step-by-step info as possible for those of us light on Linux/hardware requirement knowledge, as I'm certain there are a ton of us.

For example, I had seen port multiplier Command-based vs. FIS-based Host support discussed here:
http://www.sata-io.org/portmultiplier.asp

But wasn't sure how to determine which chipset(s) supported FIS. Intel P35 doesn't, but P45 does (thanks for the heads-up, Mike!). Apparently ICH10 support is required for FIS switching...?

Are there any boards that are currently available that handle FIS switching?

jvl711: I've also pondered whether I should keep things simple for direct Sage recording (i.e, just use a few standard hard drives for recording and moving things over to a RAID set for permanent storage). But then again, if you're going through the trouble of building a tanker for storage, why not record directly to it if possible? (And obviously this certainly depends upon other use... if you've got folks constantly hitting the server for files, it might not be feasible for it to keep up w/ stream-recording too.)

Also, that Addonics card is software raid, so your performance probably wouldn't be capable of direct stream-recording.

I was suggesting using the tanker of a disk subsystem for recording, just not to put those drives in a raid set. I would leave them as JBODS.

I agree about the Addonics software raid. That is why I was suggesting not using raid 5. Raid 5 has parity calculations were raid 1 does not. Neither does 1 + 0. Both of these would perform pretty well with a software raid card.

I was also trying to say if you really want raid 5 for library content a software raid would not be a huge downfall. The amount of bandwidth needed for library content is much less than recording drives.

[PGPfan]

Originally Posted by mikesm

Unriad does this funky kind of copying for redundancy. Personally, I don't understand why it's better than RAID1, but you can certainly run things like rsync or other script based tools to sync files across various volumes if that's what you are looking for.

FWIW, unRaid doesn't do any funky copying for redundancy. What it does do (or perhaps more accurately 'doesn't' do) is stripe data across multiple disks. It places a separate file system on each individual hard drive (NO striping) which allows it to be expanded easily, and use ANY size drive. It also implements real parity to allow easy recovery from a failed drive. Unlike other raid implementations, if 2 drives happen to fail with unRaid, you only lose data from those drives - other raid systems, you'd lose ALL the data on ALL the drives.

The funky file copy part you describe is merely how the write algorithym determines which disk to write the files to at any given time.

Hope that clears it up a little.

[stanger89]

Originally Posted by mikesm

If you aren't used to working in Linux, I would recommend going with a hardware controller in windows. It's just not worth the time for most people. It's not that hard to deal with, but I don't think it's worth the time. The hardware based controllers have their issues and cost a lot more too, but I think it's a better bet for windows only folks.

Well I've run linux before, I've always had a "soft spot" for linux really. Also I've been pondering converting anyway.

Unriad does this funky kind of copying for redundancy. Personally, I don't understand why it's better than RAID1, but you can certainly run things like rsync or other script based tools to sync files across various volumes if that's what you are looking for.

The thing that's better about unRAID is it's really parity (or equivalent to it) vs duplication, you only lose one drive to redundancy vs WHS, rsync, or RAID-1 where you lose 50% of your capacity to redundancy.

LVM has striping available that's separate from RAID striping, but not redundancy as typically found in RAID5 or RAID6, or at the filesystem level with Solaris' ZFS.

What's the issue with RAID5 if you want redundancy? It's less work to administer than syncing things in linux, but I must be missing something...

Cost to a degree, but I'd really like to have a solution that doesn't need all the disks to run.

[tmiranda]

I personally see no reason to go through all the expense and complications of fancy RAID solutions to store recorded TV shows. My theory is to use something relatively cheap and if a disk fails, so be it, it's only TV. (I do backup things that I can't replace )

I have had very good luck using NASLite (www.serverelements.com). It has limitations but it is cheap, fast, easy and reliable. If I want to expand my Sage storage I just buy another drive or replace an existing drive with a larger drive.

NASLite does support RAID, but only if you are using a hardware RAID card.

[jerryt]

I have the 7th criteria
"A box that provides;
7. Ability to spin down the drives when not in use. (maid)."

Running all my drives 24/7 for the few minutes a day that the drives are accessed is crazy. I understand Windows server 2008 will spindown the drives with port multipliers (software raids) and also some Highpoint cards (hardware raids) will allow drive spin down.

I like (price) the Highpoint 2314 (4 esata external ports) PCIe card but highpoint has not responded to my emailed question about it supporting spindown. The 3000 series cards are listed on their website as supporting "maid" which is spindown. So 2314 may be out as an option for spindown.

[stanger89]

Originally Posted by tmiranda

I personally see no reason to go through all the expense and complications of fancy RAID solutions to store recorded TV shows. My theory is to use something relatively cheap and if a disk fails, so be it, it's only TV. (I do backup things that I can't replace )

I don't have my recordings on a RAID volume, but I do have my DVDs on one. Reason being I don't want to have to re-rip hundreds of DVDs when a drive fails.

I have had very good luck using NASLite (www.serverelements.com). It has limitations but it is cheap, fast, easy and reliable. If I want to expand my Sage storage I just buy another drive or replace an existing drive with a larger drive.

Problem with the NAS solutions is that they require another PC, in addition to the SageTV server, I see no reason to run two machines 24/7.

[PGPfan]

Originally Posted by jerryt

I have the 7th criteria
"A box that provides;
7. Ability to spin down the drives when not in use. (maid)."

Running all my drives 24/7 for the few minutes a day that the drives are accessed is crazy. I understand Windows server 2008 will spindown the drives with port multipliers (software raids) and also some Highpoint cards (hardware raids) will allow drive spin down.

I like (price) the Highpoint 2314 (4 esata external ports) PCIe card but highpoint has not responded to my emailed question about it supporting spindown. The 3000 series cards are listed on their website as supporting "maid" which is spindown. So 2314 may be out as an option for spindown.

unRaid provides this as well, spinning up only the disk that is needed at that time.

-PGPfan

[mikesm]

Originally Posted by jerryt

I have the 7th criteria
"A box that provides;
7. Ability to spin down the drives when not in use. (maid)."

Running all my drives 24/7 for the few minutes a day that the drives are accessed is crazy. I understand Windows server 2008 will spindown the drives with port multipliers (software raids) and also some Highpoint cards (hardware raids) will allow drive spin down.

I like (price) the Highpoint 2314 (4 esata external ports) PCIe card but highpoint has not responded to my emailed question about it supporting spindown. The 3000 series cards are listed on their website as supporting "maid" which is spindown. So 2314 may be out as an option for spindown.

Well, that's great if your use model for storage supports that style of operation. My sage system tends to record things a lot of the time, and plus I have a sonos system that indexes the library periodically, as well as a backup system that uploads content for some of the media offsite (I don't want that running in primetime since it congests my cable modem upstream).

I'm curious that if unraid supports drive spindown how much of the time the drives are actually spun down. Also, I didn't think windows liked waiting very long for a file once a request was made, but maybe that's been fixed in recent versions.

Drive spin up /spin down is also harder on the drive from a lifetime perspective than continuous operation, but that may not be a practical issue with how we long we use the drives. My corporate data center experience makes me very nervous about this kind of operation, but it may be unfounded for light duty NAS use.

How much money would you save by having the NAS do this kind of thing?

[mikesm]

Originally Posted by stanger89

Well I've run linux before, I've always had a "soft spot" for linux really. Also I've been pondering converting anyway.



The thing that's better about unRAID is it's really parity (or equivalent to it) vs duplication, you only lose one drive to redundancy vs WHS, rsync, or RAID-1 where you lose 50% of your capacity to redundancy.



Cost to a degree, but I'd really like to have a solution that doesn't need all the disks to run.

Ok, so then the real issue is that you want some level of redundancy without having the whole raid5 array spun up for access?

One thing I like about having the NAS up and running 24x7 is that I can run other things on it, which is also why personally, the appliance type solutions are less interesting to me.

After my next upgrade, I'll have a Q6600 in the core of the NAS, plus 4 GB of RAM, I plan to run a vmware instance of XP to run Sage Videotools compression and showanalyzer right on the NAS where all the content is stored. What's funny is that I just got a recent version of Norton Ghost, and one of the options it has is to build a backup snapshot in a vmware snapshot format, so you can literally backup a windows system, and reconstitute it as a virtual machine under linux... Very cool.

This is not quite a NAS application, but I like the ability to run heavily I/O bound tasks right on the NAS.