HUGE headaches with FlexRAID's tRAID

[Skirge01]

Originally Posted by stanger89

But that brings me back to one thing I like about the "commercial" solutions, they seem to have much better status/alerting solutions than the roll-your-own (including unRAID). I remember my ReadyNAS would email me if it detected a problem (like power loss).

I've noticed the same thing. The down side is that you pay a huge premium for that and are limited to the hardware (e.g. disk expansion, memory, CPU). The cost of a feature-filled 8 bay NAS is ridiculous. But, between the RAID software, HD Sentinel, and Powerchute, you get the most important features the commercial NAS offers at a (sometimes significantly) lower price.

[panteragstk]

I'm still exploring a pseudo RAID 10 setup, but I think I'm going to stick with a Drive Bender setup with duplication enabled. Performance is single drive, but no duplication performance issues. Once I get enough drives I'm going to enable this option. I'll be able to read all data since all drives are NTFS. I don't want to deal with controller issues, software RAID or any of the other potential issues. The only way all data is gone is if all the drives die at the same time. Not likely to happen.

I'm still looking at the FlexRAID or unRAID route, but I want to avoid another box or having to use a VM of some kind.

[Skybolt]

Originally Posted by stanger89

It seems what you're calling redundancy, I'd call duplication. RAID-5 is redundant, it can survive a failure. Non redundant storage would be JBOD, spanning or RAID-0.

"Duplication" is NOT a term used when refering to computer systems, disaster recovery sites, when there are two completly redundant servers at two very differant sites one in most cases is underground. After 20yrs in the IT field I have never heard anyone use anything else but redunentcy. So I guess we have to agree to disagree with those terms.

But it is, with a single HDD, if it dies, everything is gone. With RAID-5 two have to fail for you to lose your data, with RAID-6 three must fail

Barring the terminology we are pretty much in agreement, except when I did lose three drives, which is why at some point I will go back to a RAID5 mirrored, but thats a ways off. I want to wait for larger drive capacity.

[Skybolt]

Originally Posted by panteragstk

I'm still exploring a pseudo RAID 10 setup, but I think I'm going to stick with a Drive Bender setup with duplication enabled. Performance is single drive, but no duplication performance issues. Once I get enough drives I'm going to enable this option. I'll be able to read all data since all drives are NTFS. I don't want to deal with controller issues, software RAID or any of the other potential issues. The only way all data is gone is if all the drives die at the same time. Not likely to happen.

Yes, RAID 10 is the safest way to go, which is what I am currently running in pairs, but to do this twith two RAID5 arrays would be awesome. I have never explored Drive Bender, do you currentnly use this now?

[stanger89]

Originally Posted by Skybolt

"Duplication" is NOT a term used when refering to computer systems, disaster recovery sites, when there are two completly redundant servers at two very differant sites one in most cases is underground. After 20yrs in the IT field I have never heard anyone use anything else but redunentcy. So I guess we have to agree to disagree with those terms.

Fair enough, though duplication is a form of redunancy. My point was, RAID (other than RAID-0) is redundant (ie "Redundant Array of Inexpensive Disks"). RAID levels 2+ provide that redundancy via parity, RAID 1 does it via mirroring (duplication). Again, the point is, you don't need a bit-for-bit copy of the data, a mirror, to be redundant. RAID levels 1+ are all redundant (more data stored than is necessary, and can be recovered in the event of a drive failure) and fault tolerant:
https://blogs.oracle.com/Doc/entry/r...fault_tolerant

Barring the terminology we are pretty much in agreement, except when I did lose three drives, which is why at some point I will go back to a RAID5 mirrored, but thats a ways off. I want to wait for larger drive capacity.

If I was going to go that far, I would not have them mirrored. If I were going to go to the expense of duplicating all of my data (having nearly 20 disks) I would set up some sort of real backup system. One live system with redundancy and then perhaps something as simple as backing the whole thing up to separate discs. I mean my media server isn't changed that often, I could just run an incrimental backup on my media whenever I add a new disk. Then if I ever did lose more drives than my redundant system could handle I could recover.

This would actually be safer IMO than running a mirrored system since it would protect me (if the backup is offline and unplugged when not needed) from things like natural disasters and viruses too.

Of course this really all goes back to just how much are you willing to spend to protect some work (because ripped media is already recoverable). My current unRAID is 20TB (according to Windows) of "live" storage (not counting parity or cache drives), 16.8 of it used. If I were to buy (just disks) today, to build something that would hold that:

No redundancy: 5*4TB @ $150 = $750
Single parity (RAID-5, 4, unRAID, etc): 6*4TB @ $150 = $900 (+20%)
Dual parity (RAID-6, etc): 7*4TB @ $150 = $1050 (+40%)
Mirroring (RAID-10, duplication, etc): 10*4TB @ $150 = $1500 (+100%)
Pairty+Mirroring (RAID-51, etc): 12*4TB @ $150 = $1800 (+140%)

When I sit down and look at the likelihood of failure, 20% extra for parity is a no brainer, 40% extra is not bad. But double or more, for something that isn't irreplaceable, that's well over the line IMO.

But back to your fan failure that caused (presumably) overheating and drive failures, that's one nice thing about something prebuilt, they will alert you to such a failure (fan failure or drives overheating) so you can shut them down to prevent failure until you can address the issue.

I mean just take a step back, for about the difference in price between just parity and a RAID-51 setup, you can get a Synology 1813+ that will just work and (I assume) give you health alerts out of the box. I guess price is moot* since it's about the same either way, but it would cost less to run, and have more features and potentially more reliability than some roll your own system.

*Though in the future when you want to upgrade, the incremental cost to expand a parity setup is half that of a mirrored setup.

FWIW, I'm not trying to convince anyone to change what they're doing, these are just the thoughts that run through my head when looking at this stuff.

[Slugger]

I apologize in advance for the massive hijack here, but feel like there's a tiny bit of relevance to this thread, which I have been following, so here it goes...

I built my unRAID system 3.5-4 years ago, over-spec'd the server (big time, too embarrassed to even list the specs ) and started with 3x1TB drives. This let me use the free version while I figured out just how fast I was going to grow. Anticipated steady growth, well it turns out the usage pattern in my house doesn't need all the storage I anticipated. 4 years later still got ~1TB free on the unRAID box, which is only used for iTunes, photos, and archiving content from Sage, which is first recorded locally on the Sage server. We're a watch it and delete house, which I already knew beforehand, but my policy is to never delete a recorded movie - archive them all! I also have scripts to archive any recorded tv that is older than 12 months or archive things if the recording drive on the Sage box falls below 10% free space. Well in 4 years, I can count the number of times the <10% rule was triggered on one hand (and still have some free fingers) and that was with just 750GB on the recording drive -- again, we're a watch it and delete it house.

This past summer it was time to retire the original Sage server: Q6600, WinXP, 2x750GB RAID 1 recording drives. After much thought, I went with a single 2TB recording drive with no redundancy. With a 2TB recording drive, I'll be shocked if that drive ever falls below somewhere around 800GB free space. After 7+ years of Sage, I've come to the conclusion that it's "just tv" and instead of RAID 1 on the recording drive, I simply generate a report of what's on the drive every day and in case of disaster the new recovery plan is to skim over the list and anything that "has to be recovered", which won't be much I don't think, well it'll be recovered thru BR/DVD/iTunes/etc. purchases.

That just left the iTunes, photos, and archive of movies on the NAS... well we've been a subscriber to Netflix for sometime now and after some research, I've concluded enough of the content (but definitely not all of it) is available on Netflix that Netflix is now this house's "backup" solution. Not to mention all the other content it has. Again, those things that can't be found on Netflix, well I've got the BR somewhere or if it was recorded from movie channels/PPV/etc. and someone decides it "just has to be recovered" then the BR is purchased when needed.

So the next/final step for me will be to slap another 2TB drive in the Sage server and copy what's on the NAS to it and to an external drive, which will replace the NAS that I so desperately want to take offline. I can't justify keeping that box online 24/7 anymore so the plan is to decommission it and sell the box.

I guess my point here is just to show how I'm redefining "the line" for my storage needs. It went from "the sky's the limit" 4 years ago to now "don't need it, don't want it, why am I running it". Netflix is a big reason I'm willing to layoff on the redundancy efforts + the fact that we're a watch and delete house when it comes to tv. To me, it just seems way cheaper to buy whatever tv BR/DVDs are needed to recover in case of disaster instead of trying to build up arrays of disks. And it may even turn out that whatever tv I may lose will be available on Netflix by the time I lose it, which is even better. $8/mth for Netflix seems like a much more sustainable approach than the hundreds or thousands of dollars in disks needed to keep up the redundancy, which inevitably fails you at the most inopportune time anyway. Just something else to think about...

[Skybolt]

Originally Posted by stanger89

Fair enough, though duplication is a form of redunancy. My point was, RAID (other than RAID-0) is redundant (ie "Redundant Array of Inexpensive Disks"). RAID levels 2+ provide that redundancy via parity, RAID 1 does it via mirroring (duplication). Again, the point is, you don't need a bit-for-bit copy of the data, a mirror, to be redundant. RAID levels 1+ are all redundant (more data stored than is necessary, and can be recovered in the event of a drive failure) and fault tolerant:
https://blogs.oracle.com/Doc/entry/r...fault_tolerant

nice referance to a spare tire.

If I was going to go that far, I would not have them mirrored. If I were going to go to the expense of duplicating all of my data (having nearly 20 disks) I would set up some sort of real backup system. One live system with redundancy and then perhaps something as simple as backing the whole thing up to separate discs. I mean my media server isn't changed that often, I could just run an incrimental backup on my media whenever I add a new disk. Then if I ever did lose more drives than my redundant system could handle I could recover.

This would actually be safer IMO than running a mirrored system since it would protect me (if the backup is offline and unplugged when not needed) from things like natural disasters and viruses too.

Well, thats why I pull the mirrors after they have finished replicating. Which should cover the above point. For all of my important data, I rotate in a third disk so I am completely safe from almost anything. Movies and the like are not done like that.

When I sit down and look at the likelihood of failure, 20% extra for parity is a no brainer, 40% extra is not bad. But double or more, for something that isn't irreplaceable, that's well over the line IMO.

I guess that all depends on how much your time costs and what you're not willing to re-do.

But back to your fan failure that caused (presumably) overheating and drive failures, that's one nice thing about something prebuilt, they will alert you to such a failure (fan failure or drives overheating) so you can shut them down to prevent failure until you can address the issue.

Oh, my system e-mails me on everything, this just all happend way to fast, not to mention I was away on my boat when this all took place, and I don't realy pay attention to email when I am away on that. The worst part is this totally could have been prevented had I checked my phone, but I turn off notifications when I am on vaca. I could have shut the server off remotely ...

I guess price is moot* since it's about the same either way, but it would cost less to run, and have more features and potentially more reliability than some roll your own system.

Supermicro has some pretty nice tools for dealing with this, rolling your own is not the issue. For that matter my Rocket Raid card has software for notifications as well.

*Though in the future when you want to upgrade, the incremental cost to expand a parity setup is half that of a mirrored setup.

FWIW, I'm not trying to convince anyone to change what they're doing, these are just the thoughts that run through my head when looking at this stuff.

I have to totally agree with you here, I am not saying this is the way to go either. I guess my only point in THIS discussion was that you're never totally safe from losing your data no mater how much prep you do. I went on vacation knowing that two hot spares were more then enough to protect my invested time. Then over reacted by mirroring everything.

[stanger89]

Originally Posted by Skybolt

I guess that all depends on how much your time costs and what you're not willing to re-do.

It's more of a cost/trouble vs probability of event. The likelihood of having two drives fail "at the same time" (ie close enough to not allow a rebuild to occur), is too remote in my estimation to justify having almost $1000 of extra disks around and managing them.

Oh, my system e-mails me on everything, this just all happend way to fast, not to mention I was away on my boat when this all took place, and I don't realy pay attention to email when I am away on that. The worst part is this totally could have been prevented had I checked my phone, but I turn off notifications when I am on vaca. I could have shut the server off remotely ...

Well that sucks. Wild curiosity, why didn't you just shut down the storage when you were away? If you weren't going be accessing it.

Supermicro has some pretty nice tools for dealing with this, rolling your own is not the issue. For that matter my Rocket Raid card has software for notifications as well.

FWIW, I sort of put solutions like that outside of the roll your own category. I mean if I spend a few hundred on a raid card I expect integrated management/alerting tools, same if you go with an off the shelf system from Supermicro.

By "roll your own" I was thinking more like FlexRAID/tRAID/SnapRAID/unRAID/etc, where you find something to manage the storage but leave it to you to come up with email alerts or auto shutdown or the like.

I have to totally agree with you here, I am not saying this is the way to go either. I guess my only point in THIS discussion was that you're never totally safe from losing your data no mater how much prep you do. I went on vacation knowing that two hot spares were more then enough to protect my invested time. Then over reacted by mirroring everything.

Well I almost went out and bought a Synology when I hit a roadbump with my unRAID. Actually my current laptop has ticked me off with it's upgradability (Lenovo is terrible, never buying one again) so I ordered a Dell M3800, so how's that for overkill

And as far as data safety, all my personal data (ie stuff I can't just rerip) is on Crashplan as well.

[Skybolt]

Originally Posted by stanger89

It's more of a cost/trouble vs probability of event. The likelihood of having two drives fail "at the same time" (ie close enough to not allow a rebuild to occur), is too remote in my estimation to justify having almost $1000 of extra disks around and managing them.

Well at the time, I would have to agree with you, because I knew I would never have two drives fail at the same time - let alone three. That's just not going to happen.

Well that sucks. Wild curiosity, why didn't you just shut down the storage when you were away? If you weren't going be accessing it.

Why would I? Do you? It was and now is again "out of sight out of mind", and then there's the old "that will never happen to me" and then the "what could possibly happen, I have two hot spares for an eight disk array".

I guess that point is totaly from hind sight but, even now I would never shut my server down if I leave for an extended period of time. I mean, battery backup takes care of things in a power failure, drives spin down (now) when not in use, no real reason to. Until the next event I guess.

FWIW, I sort of put solutions like that outside of the roll your own category. I mean if I spend a few hundred on a raid card I expect integrated management/alerting tools, same if you go with an off the shelf system from Supermicro.

Yeah, I guess comercial products are "off the shelf" solutions, even if you build it yourself. But I would definetly "do it again". There is a peace on mind knowing your using a proven solution. I don't think I would ever go with a Flexraid or even your unRaid system, although I almost did.

Well I almost went out and bought a Synology when I hit a roadbump with my unRAID. Actually my current laptop has ticked me off with it's upgradability (Lenovo is terrible, never buying one again) so I ordered a Dell M3800, so how's that for overkill

Something like the DS2413+ IS worth the money, but then your spending more then I did when I re-built everything. But then again $1600 is what got me my server with the first 6T of storage.

I guess it's like me saving money by not buy something like your Anthem and going with Onkyo instead, although your Anthem may be a bit more tangible, for me my time is and knowing that when I want to sit down and marvel at all of the fanart etc, that it's going to work.

All in all hot spares should be enough if you stick with quality drives like Seagate enterprise class drives. They have the best track record of them all. IMO.

[stanger89]

Originally Posted by Skybolt

Why would I? Do you? It was and now is again "out of sight out of mind", and then there's the old "that will never happen to me" and then the "what could possibly happen, I have two hot spares for an eight disk array".

If I wasn't going to use it, yes I probably would turn it off. I usually leave it on because I do on occasion access it (stream music from it, etc). I mean it's sort of like the water or the water heater, I shut off my water (nothing should go wrong, but why chance it) and I turn off my water heater (no point heating water when nobody's around), turn the heat up/down/off as appropriate. I turn off the power strip to my HT equipment, unplug my projector and my TV, etc...

I mean no, I don't usually turn off my NAS when I leave but I want it there to access it, maybe you were thinking the same thing, but if I knew I was not going to access it, no point leaving it on for something to happen, nor to waste power.

Yeah, I guess comercial products are "off the shelf" solutions, even if you build it yourself. But I would definetly "do it again". There is a peace on mind knowing your using a proven solution. I don't think I would ever go with a Flexraid or even your unRaid system, although I almost did.

I mean I really like the idea of unRAID, that's why I bought it, and it does what it's supposed to do well. But having lived with it for quite a while now, I miss the build in health monitoring of my ReadyNAS. FlexRAID, yeah, I don't think I'll touch it, the developer has turned me off too many times (disappearances, bad attitude, etc), and for some reason I just don't want my NAS running Windows, or you know what it is more, I really like having my NAS separate from my other devices so it's more stable and independent. Windows would be fine, but rather expensive for such a use.

Something like the DS2413+ IS worth the money, but then your spending more then I did when I re-built everything. But then again $1600 is what got me my server with the first 6T of storage.

I guess it's like me saving money by not buy something like your Anthem and going with Onkyo instead, although your Anthem may be a bit more tangible, for me my time is and knowing that when I want to sit down and marvel at all of the fanart etc, that it's going to work.

You've got me curious, what's the connection between Anthem/Onkyo and Fanart?

[Skybolt]

Originally Posted by stanger89

If I wasn't going to use it, yes I probably would turn it off. I usually leave it on because I do on occasion access it (stream music from it, etc). I mean it's sort of like the water or the water heater, I shut off my water (nothing should go wrong, but why chance it) and I turn off my water heater (no point heating water when nobody's around), turn the heat up/down/off as appropriate. I turn off the power strip to my HT equipment, unplug my projector and my TV, etc...

I can only say wow to that, I mean I never ever thought of shutting those things off. I look at a server as something that should always be running for the inevitable "just in case" as I access it remotely quite often. My HT is running off of a large battery backup, I guess I could just flip the switch, but I never even thought about it.

You've got me curious, what's the connection between Anthem/Onkyo and Fanart?

LOL, yeah if I typed every sentence that ran through my head, this would be more of a war and peace read then it already is. My referance to Anthem and Onkyo was about the cost of HD's etc. and in the end things just working, I know that when I sit down to watch a movie the shares are going to be up and running with all of the fanart in tact.

Onkyo vs Anthem was equal to NAS (unRaid etc.) vs Mirroring. Maybe a bad analogy, but both get the job done with very little noticable differences except how much they cost you.

[panteragstk]

Funny you guys should mention the Synology stuff. I think I'm at the point to where if I go the NAS route (opposed to Windows server with Drive bender) I'll just get something from Synology.

My boss has a Drobo and swears by them, but they are more feature limited than the Synology units of the same size.

I do like the drive re-ordering feature (doesn't matter what order your drives are in) that the Synology doesn't have, but that isn't an issue if you keep them numbered properly. Plus the Synology units have more expandability. I don't want to have multiple Drobos or Synology NAS units and have tons of shared folders again. I like having one huge drive that Drive Bender gets me.

I'm thinking about getting a DS1813+. Can expand to 18 drives. Plenty for me.

[Skybolt]

Originally Posted by panteragstk

Funny you guys should mention the Synology stuff. I think I'm at the point to where if I go the NAS route (opposed to Windows server with Drive bender) I'll just get something from Synology.

I can understand wanting to go with a Synology device, but why Drive Bender? I mean isn't it just a drive pool, little more a JBOD, no redundency, quite differant from Synology's RAID implimentation. Unless your just going to make a drive pool with the nas device.

[panteragstk]

Originally Posted by Skybolt

I can understand wanting to go with a Synology device, but why Drive Bender? I mean isn't it just a drive pool, little more a JBOD, no redundency, quite differant from Synology's RAID implimentation. Unless your just going to make a drive pool with the nas device.

Drive bender is just used to pool the drives. I have duplication set up on one of my drives that has important info on it.

I want more than a drive pool and as nice as the Synology devices are, they are pretty much the same as going with RAID 5 or 6. From what I've been reading the Drobo's are quite different in the way they use RAID.

No waiting for rebuilding after a drive failure. No downtime when a drive does fail, unit keeps on working.

My boss said that he was in the middle of watching a movie stored on his and a drive failed. The only reason he knew the drive failed was that the Drobo sent him an email. He popped in a new drive and that was that.

He even had a Drobo unit fail (PSU died) and he put his drives in the new one and it worked without issue. He didn't even have to put the drives in the correct order.

That sounds pretty good to me. Not as good about expandability as the synology units, but the way they do RAID is pretty fantastic. I haven't found anything that has the same features.

[Skybolt]

Originally Posted by panteragstk

... I have duplication set up on one of my drives that has important info on it.

So it duplicates the file you manually tag and keeps them duplicated in the drive pool or another specified place?

He even had a Drobo unit fail (PSU died) and he put his drives in the new one and it worked without issue. He didn't even have to put the drives in the correct order.

That sounds pretty good to me. Not as good about expandability as the synology units, but the way they do RAID is pretty fantastic. I haven't found anything that has the same features.

Yeah, I would have to agree, that sounds like the way to go, if you want just want space.

[panteragstk]

Originally Posted by Skybolt

So it duplicates the file you manually tag and keeps them duplicated in the drive pool or another specified place?

It is drive based and folder based. Pretty cool feature actually. You can choose to duplicate folders on a drive or the whole drive. No overhead since it just writes the file to two drives at the same time. If one dies, no problem. All the files are stored and any windows PC can access them.

I want to have a better level of protection so a NAS for me is the easiest way to do it. I don't really trust the software RAID that is available.

Again, there is no perfect solution.

[Gustovier]

Originally Posted by stanger89

You guys are making me glad I run unRAID

[panteragstk]

So, this topic has me looking at this yet again.

While the drobo advice I talked about above seems like they have a good product, the fact that the drives can't be read by anything else (plus the cost) really turns me away.

I looked at FreeNAS with ZFS and that looks great...except for the hardware requirements...and the fact that it has all/most of the limitations of RAID. The expansion issue is what has me looking elsewhere.

tRAID and FlexRAID look decent, but they have some limitations as well, but the ability to read information in case of a failure looks nice, plus I don't have to run an extra box or VM.

I keep coming back to unRAID. Yes, it will make me have another box, but there is a virtual box plugin that would allow me to virtualize my current 2012 essentials server. I could still have just one box. I really would rather run UnRAID as the main OS instead of 2012.

So, UnRAID will do everything I need and they now have up to 24 drives (too many, but nice to have the ability to have that much expansion) and they will or currently do have the ability to use two parity drives. Sounds like they are going the direction I want.

My requirements for a storage solution are:

1. Able to expand on the fly and quickly be able to use my data. No downtime preferred.
2. Able to replace a small drive with a larger one with little to no downtime.
3. Able to replace a bad drive with little to no downtime.
4. Able to recover from a multiple drive failure.
5. Able to read files from individual drives should more than two drives fail.
6. Able to transfer to alternate hardware should something major go wrong.

So, UnRAID fits most if not all of these things. tRAID should, but it is a bit odd in the way that it works. Especially after IVB's issues. Not as mature as UnRAID, but a similar product.

FreeNAS looks nice, but is a bit limited in too many ways.

If anyone can think of any more let me know.

[Skybolt]

Well here is one way to survive as far as the case case goes anyway Norco. This is going to be my next upgrade to my server. I still can't let go of 2008R2 as my main OS.

[panteragstk]

Originally Posted by Skybolt

Well here is one way to survive as far as the case case goes anyway Norco. This is going to be my next upgrade to my server. I still can't let go of 2008R2 as my main OS.

Yep. That or the 24 drive version. I think 20 is plenty though.