Snapshot RAID with SageTV

[Tiki]

After a couple of recent drive failures in my Sage rig, I've decided to rethink my data protection strategy. In the past I thought it was not worth doing any sort of backup on my Sage recordings, but I've learned that it can be very frustrating when you loose a few TB of recordings.

I still don't think full backups (for this type of data) are worthwhile, but some sort of parity based RAID makes sense. I would rather not shell out a ton of cash for a hardware appliance (Synology, QNAP, drobo, etc.). And I would rather not build a separate dedicated server running a software based raid (unRaid, etc.), because I don't have suitable hardware just laying around, don't want to make a large investment to buy all new hardware, and am concerned about throughput using an external unRaid server connected via Ethernet as a storage device for Sage (especially when I have multiple Ethernet-connected tuners and extenders).

I also briefly considered virtualization (running UnRaid and Sage in separate VM's on the same hardware on top of XEN or ESXI, or something), but I don't think my current Sage box is up to the task (CPU and Mobo need to support virtualization, need lots of RAM, etc.).

That brings me to look at snapshot based software raid (FlexRAID-F, or SnapRAID). This would only require adding one large hard drive to store parity information. I could set it to run snapshots daily, and then if I had a single drive failure I would (in theory) only lose changes made on the failed disk since the last snapshot (one day's recordings at most).

Is anyone using FlexRAID-F with Sage? Are you letting Sage write directly to the FlexRAID array, or doing some manual move process?

My main concern is that any files I add (new recordings) or delete (shows I manually delete through Sage after watching) between snapshots would affect the parity calculations and may make the restore fail.

FlexRAID claims that it can handle file additions (you would just lose any files that were added to a failed data drive since the last snapshot). Is this true? Wouldn't new files added to the non-failed drives throw off the parity calculation? Or is FlexRAID smart enough to recognize the added files and alter its parity calculation somehow?

I also see that FlexRAID implements its own custom recycle bin so it can temporarily hold any deleted files until the next snapshot. From what I have read, this is only available if you use the drive pooling feature (not sure why that is - I would rather not pool my drives). My question is - would this help with files deleted by Sage? I wonder because Sage normally bypasses the Windows recycle bin when deleting files. Would it bypass the FlexRAID recycle bin too?

If I do pool my drives, how much of a performance hit would it be?

Finally, how long do the snapshot updates take to run? It sounds like the initial parity generation can take more than 24 hours - hopefully the updates are much faster. Do I have to keep Sage from recording or deleting files during this whole time? What's the best way to schedule the updates so they don't interfere with Sage usage?

Sorry for the long post, and I know there have been lots of posts about RAID, but although I've read a ton, I haven't found the answers to my questions.

[NetworkGuy]

I was using Windows Home Server to backup all of the PC's in the house. When I started to have hardware and with the next release of WHS removing drive pooling, I decided to invest in a Synology Raid. Yes, it was expensive, but since I reused the drives from WHS, I was able to keep the cost down.

As I start to have problems with a drive, I replace it with a larger drive and gain more space.

I am using SyncBack Free to do a nightly backup of my SageTV recordings.

It was worth the investment.

John

[Skirge01]

Take a look at my post here and let me know if you need more information. What you want is FlexRAID's tRAID software, not RAID-F. tRAID is real-time parity protection, so you don't have any of the concerns you mentioned.

[nyplayer]

Has anybody here tried Windows 8.1 Storage Spaces?

[Tiki]

Originally Posted by Skirge01

Take a look at my post here and let me know if you need more information. What you want is FlexRAID's tRAID software, not RAID-F. tRAID is real-time parity protection, so you don't have any of the concerns you mentioned.

I was a little nervous about tRAID, because it's pretty new and I saw that thread and the OP had lots of problems.

I'm not sure I fully understand your post... Are you saying that you are still putting your Sage recording directories in the tRAID array, but you are writing to them directly, rather than writing to the "pool" and letting tRAID decide which disk they end up on?

[Skirge01]

Originally Posted by Tiki

I was a little nervous about tRAID, because it's pretty new and I saw that thread and the OP had lots of problems.

Obviously, everyone's system is different, so that's just one example. I had no such issue with getting the service to start.

I'm not sure I fully understand your post... Are you saying that you are still putting your Sage recording directories in the tRAID array, but you are writing to them directly, rather than writing to the "pool" and letting tRAID decide which disk they end up on?

I think you got it; however, let me try to explain it further. I have no doubt you got some/most of this, but I want to be thorough. (Sorry for the length.) Also, keep in mind that I'm speaking at a very high level, since I really don't know what tRAID is doing behind the scenes.

With tRAID, you have the option of having a storage pool and you have the option for parity protection. Additionally, the drives you put into either of these have their data completely untouched, meaning that you can remove a drive from tRAID, place it into any other computer and access the data. What tRAID does for pooling is present all of these drives to you under a single folder. If you opt for parity protection, tRAID will calculate it from your drives and store it on your parity drive (PPU, in tRAID speak).

Now, because tRAID (aka Transparent RAID) doesn't touch the data on your data drives (DRUs), it also allows you to access the drives outside of the pool, if you choose and set it up like I described in my other post. Using this method, you have the storage pool, parity protection, and access directly to each individual drive.

What my instructions in the other post were ensuring is that when you "moved" your recordings (as far as Sage is concerned) from the storage pool to individual disks, it was able to see that the old directory from the storage pool was now empty and that the files had "moved" to the new folder outside the pool. This prevents Sage from creating duplicates by keeping both the old location and the new.

So, I have Sage writing to UNC folders I've added to the individual drives, of which the drives themselves are still part of the pool and have parity protection under tRAID.

To illustrate this, here's my setup which includes 7 data drives and 1 parity drive): tRAID is configured to provide me with an array consisting of a storage pool which has parity protection. On my server, this array is locally listed as my V: drive. It is also shared as a UNC path on the server (named \\W7-Server\).

I have the following folders in that V: drive (pooling all 7 data--aka DRU--drives together and providing parity protection):

V:\Recordings (DVR)
V:\Photos
V:\Music
V:\Videos

They're shared as:

\\W7-Server\Recordings (DVR)
\\W7-Server\Photos
\\W7-Server\Music
\\W7-Server\Videos

Sage itself does not have access to any of the above folders, but all other users/computers on my network have access to them.

Using Windows' Disk Management, each of my DRUs which make up the tRAID array also have direct UNC mappings to an empty folder on my C: drive under C:\Shares, listed as:

C:\Shares\4_1
C:\Shares\4_2
C:\Shares\4_3
C:\Shares\4_4
C:\Shares\5_2
C:\Shares\5_3
C:\Shares\5_4

The UNC mappings are:

\\W7-Server\4_1
\\W7-Server\4_2
\\W7-Server\4_3
\\W7-Server\4_4
\\W7-Server\5_2
\\W7-Server\5_3
\\W7-Server\5_4

Combined, the above shares contain the exact same data as the V: drive, folders and all:

\\W7-Server\4_1\Recordings (DVR)
\\W7-Server\4_1\Photos
\\W7-Server\4_1\Music
\\W7-Server\4_1\Videos
\\W7-Server\4_2\Recordings (DVR)
\\W7-Server\4_2\Photos
...etc

These are the shares that Sage has access to. When Sage writes to them, V: is updated, which causes an update to the parity.

[brianrent]

Originally Posted by nyplayer

Has anybody here tried Windows 8.1 Storage Spaces?

I currently use it and it seems to be fairly easy to use. I just had to transfer all of my recordings to another drive as it MUST format both drives to create the drive initially. Haven't had a failure yet but seems simple to replace a drive and it automatically copies from the remaining drive. So far so good anyway..

[toony]

Has anyone else tried Drive Bender? I've been using it for a couple years now ever since I dropped WHS. I used Flexraid briefly but it wasn't the drive pool replacement I wanted. DB is ver stable and works well. I use it as JBOD. It has file duplication if you want. Swapping drives is fairly easy. I have about 12TB on it spread over about 8 drives. I use it for my media server and have never noticed any performance issues.

[panteragstk]

I use drive bender on server 2012 and UnRAID for all the read only data. Drive bender is used for duplication on the 2012 server for the important stuff. Works well.