|
Hardware Support Discussions related to using various hardware setups with SageTV products. Anything relating to capture cards, remotes, infrared receivers/transmitters, system compatibility or other hardware related problems or suggestions should be posted here. |
|
Thread Tools | Search this Thread | Display Modes |
#1
|
|||
|
|||
My Improved SageTV + Home PC + unRaid Server + Backup Plan. Suggestions Requested…
[This got a bit longer than I intended, but there were lots of details. The basic problem is: how do I store & backup 4-5TB of data at an acceptable risk level? And how have others handled this?]
I’m trying to come up with a better / more secure way to store my SageTV + Home PC data. Most of my data will be media files that either come from SageTV recordings [< 1TB], backups of often watched DVDs [1-2 TB], or home videos [1-2TB]. I currently have a 5 year old XP Pro mini-tower PC that is the home PC / SageTV server / storage location all in one. (Backup-wise I’m currently mirroring the recordings to an external hard drive and using CrashPlan to backup the small non-replaceable files on my computer.) I want to get a new home PC and at the same time separate the home PC / SageTV / Storage into 3 machines. (Honestly I wish I could do it on 1 machine to simplify things, but I don’t see an easy / non-pricey (<$2k) way with this much data... Multiple machines allows me to reboot without messing up live TV, sharing data between machines, etc...) I basically have 4 types of files all of which are on my current PC:
What I’d like to do is come up with a storage mechanism that is automated and secures each type of file appropriately thru backups / mirroring / etc. (It doesn’t have to be simple to setup, just simple to maintain. i.e. I don't want to have to 'remember' to back it up.) Here is my plan. I’m open for suggestions. This has went thru a few iterations, but I haven’t bought anything new yet so I can completely change it. 1) SageTV PC:
2) New PC
3) unRAID Storage
My biggest concern in the whole scheme is regarding the ‘Large Non-replicable’ files. I originally considered just buying external USB hard drives to sync to every week instead of unRaid, but I didn’t see how that would be any more secure than separate internal unRaid drives. (Other then I could put them offsite, but in practice I never would… i.e. Not simple to maintain.) It would be rare that additional ‘Large Non-replaceable’ files would be created so I could burn them to a DVD as they are created. (They would mostly likely be created only when I fill up a SD Card of video or pull another HD movie from MiniDV tape.) Another option was to just to put the backup drives in the New PC instead of the unRaid. (No raid in the New PC.) Then the ‘Large Non-replaceable’ files would be on the New PC and backed up weekly to the unRaid. In this case the unRaid simply becomes a backup location and storage for ‘Painfully replaceable’ data. (I could still mirror just the drives that contained the ‘Painfully replaceable’ data to make it a bit more secure. Or just keep the master copy of the ‘Painfully replicable’ data the New PC and replicate to the unRaid, then no internal unRaid mirroring would be needed.) Seems kinda wasteful though to have an unRaid just for backup and not be the master location of shared files… (I guess I could reverse it and mirror the unRaid to the New PC.) Another option: try to get a fast enough upload connection so I could backup the “Large non-replacable’ files to CrashPlan Online. Or maybe upload them to a friends house as a new CrashPlan backup set. Too many options. I know others are backing up 10TB+ unRaids somehow, so there must be a good way others have figured out. Thoughts? Better / Simpler Ideas? Thanks! Last edited by rdellar17; 02-20-2011 at 01:31 AM. |
#2
|
||||
|
||||
You have around 4 TB to backup, I say just get a couple of 2TB green internal drives and use a backup software to schedule regular backup. With green drives so inexpensive now, this seems to be the cheapest and easiest solution. And you can still go with online backup on top of that if you really need an offsite backup plan.
It's not bad idea to upgrade that PC though, a 5 year old P4 system seems like it should be retired from server duties. And I wouldn't use it for an UnRaid system either, it doesn't seem like a good idea to use old hardware for UnRaid.
__________________
Mayamaniac - SageTV 7.1.9 Server. Win7 32bit in VMWare Fusion. HDHR (FiOS Coax). HDHR Prime 3 Tuners (FiOS Cable Card). Gemstone theme. - SageTV HD300 - HDMI 1080p Samsung 75" LED. |
#3
|
||||
|
||||
It's good that you are thinking about how to recover your system before the next disaster occurs, instead of after the next disaster! I don't see anything about disk imaging. If there are any changes to your Windows computer that you need to reverse or the system fails, including a boot failure, you need to have an image to recover the system to a point in time when the system used to work perfectly. You should be doing periodic disk imaging of each Windows computer. Imaging does not work with WHS, but you don't have WHS. Others that decide to use WHS may face a painful and time consuming rebuild process. I use both Acronis and Ghost to periodically image my Windows XP SageTV computer. The image files are stored on a secondary drive and are copied to my unRAID server. Do not use DVDs to store image files or any other files. Long term DVD storage is not reliable. I have had several failed recoveries in the past based on duplicate DVDs and CDs. So even duplicating the optical media can result in failed recoveries. Use USB hard drives or USB flash drives for portable media instead of optical media.
I've never heard of CrashPlan on-line backup before. I did not check out the cost, but I did see that they have unlimited storage. You probably want your files encrypted on CrashPlan too. You don't have any control over CrashPlan. If CrashPlan goes out of business, when you hear about it, it will be too late. To protect your data from that possibility, you should back up the same files to a USB hard drive and keep that USB drive off site. You could have two USB portable drives, swapping them periodically to update your offsite backups. Another way would be to backup to a neighbor or relative's computer on-line. I signed up for a year of Acronis on-line backup, which is limited space. I've been disappointed with the Acronis on-line backup since to transfer about 60 gigs might take 3 - 4 days, which is 72 to 96 hours. Although after the initial data transfer, the changes take much less time. If you had an out-building, that might be a place you could use for off-site storage since you could run a Eithernet cable there and power the off-site computer. Although this off-site location wouldn't be very far away, but might protect your system from fire. If you try to backup the big files to another computer across the Internet, that might put your data transfers the over monthly Internet provider's file transfer limits. You might also check out a program called SyncBack. There is a free version and two relatively cheap pay versions. SyncBack will automatically synchronize folders to another computer. Backing up the unRAID server can be expensive if the unRAID server is big. If you keep the unRAID server down to 4 - 5 TB, then it could be backed up to 2 - 3 drives on another computer. Maybe you could setup a RAID 5 array on one of your computers to backup the unRAID server. SyncBack could do that synchronization for free. I recommend backing up your smaller, critical files, such as photos to an on-line backup company and to a portable USB drive that is normally stored off-site. Then deploy periodic SageTV computer disk imaging. Make sure you have a second copy of the image files on another drive or system. Do not depend on optical media for long term backups. If you use optical media for short term backups, use Verbatim brand, not a cheaper off-brand. Then build up an unRAID server to replicate your more important video files. If you need more redundancy, then add another USB hard portable hard drive for your critical files or a different on-line backup service. If you need a third copy of your important video files, replicate them onto a separate RAID 5 array on one of your other computers. Dave |
#4
|
|||
|
|||
When done I'll probably have close to 6 TB, but I could put that easily in the new machine. Every time I get a new 2 TB worth of data I could just get another external drive to backup to... That would be pretty simple.
|
#5
|
|||||
|
|||||
Replying to some of the comments and had a few questions. Thanks Dave for the suggestions!!!
Quote:
Good point it is painful to backup to DVD anyway. (I mostly use the Verbatium DVD+R AZO disks which I've had good luck with.) Quote:
One of the best things about Crashplan is that it will automate the backup to local attached USB drives, local network storage, Online or to a friend / relatives house. You can setup a PC at a friend/relatives house and as long as the PC has an internet connection and the CrashPlan software you can backup to it. It even works on Linux/Mac. And there is quite a long discussion on the unRaid forums about using CrashPlan to backup to unRaid. Since it works locally even if the company went out of business you could still do restores using the software from the local hard drive you backed up to. (i.e. No internet connection needed.) The backup dir is simple also and looks identical on every destination. (You can even copy the directories around to other devices and 'reconnect' to them for a restore. Or seed a remote backup with a backup from a USB drive.) Everything is encrypted (448bit Blowfish) and it will smartly compress / dedupe data also... (i.e. Doesn't compress non-compressable files. And if you move a file to a new directory it won't back it up again, just change the file pointer.) CrashPlan also self validates the data on whatever schedule you want to check for corruption. (Lots of other features too...) I need to investigate further on how the backup sets work. Originally I was going to use CrashPlan to backup to my USB drive, but on the old version you could only have one backup set. What that meant is you had to backup all the data to ALL destinations. (So it had to go Online, USB hard drive, etc.) I really just wanted to backup some of my 'Painfully replaceable' data to USB only and not online. (Also I don't need more than one revision on these huge files, maybe remove from backup if deleted over a month.) The new version, 3.X, supposedly adds all these features. They claim some folks get 50 Mbps upload speed to CrashPlan online. But I think that is on a really really good day and highly dependent on the internet interconnects between your ISP and CrashPlans. Quote:
Quote:
Quote:
So files would be stored this way:
I guess my main thing to figure out now is if I should go unRaid or get multiple external USB drives. (I need to map out how much storage I'll need to determine this probably.) I also need to determine if I should create a server at my friend / relatives house or maybe go with local Raid5 for the 'Large non-replaceable' files. (Since the 'Large non-replaceable' files are under 2gb I could put a PC over there without much issue.) I have 2 older PCs one of which I could use for the unRaid and the other could go to my relatives house as an offsite backup. (I'd have to verify the older PC is fast enough for unRaid, but it meets the minimum specs.) I'm also leaning towards just combining my New PC with my Sage TV machine just to simplify things. There is quite a bit of downtime on the TV where I can reboot if need or run intensive processes. Getting closer. Any Additional Thoughts? Thanks! Last edited by rdellar17; 02-21-2011 at 12:10 PM. |
#6
|
|||
|
|||
My thoughts on everything mentioned so far:
|
#7
|
||||
|
||||
Quote:
I fixed my signature. I technically don't have the unRAID server working right now either. I am in the process of rebuilding it with a different system board. I need to replace the Gigabyte system board that has HPA with a Supermicro system board. I should have it built up and have the data moved back to it in a couple weeks. This will free up the Gigabyte with the i-7 950 3.06 gig quad-core for my new Linux SageTV computer. Hopefully I will be able to migrate from the Windows XP computer to the Linux computer in a few months. The old Windows XP SageTV computer would then be used for something else. Dave |
#8
|
|||
|
|||
Quote:
What is kinda depressing about this simple backup process is knowing what you can do if you have lots of money. Some dev folks I know have multiple fancy rack mounted NetApp RAID6 devices (each device with dual parity drives and hot drives ready to replace any that goes out.) Then each of these RAID6 devices writes snapshots to another nightly. We are talking lots of TB here... All in less than a rack and what they say is in a location that is like a bunker. And that is the storage just for the non-critical stuff. |
#9
|
|||
|
|||
Quote:
Maybe I can get my inlaws to leave a PC running all the time at their place. Then I would have local unRaid and a remote backup. All done thru CrashPlan. (Only thing I wish that CrashPlan didn't do was remove your backup files if you UNCHECK that you want something backed up at the next maintenance window. Maybe I'll open a feature request a feature for them not to do this. Or only after X days. I guess that is why the options are secured by password though.) EDIT: I just realized it would be dumb for me to setup a server at my inlaws that was running CrashPlan to host my backups. Why not just backup directly to the cloud since there is unlimited storage there and the speeds would be the same. (It would be limited by my upload speed either way.) I think I trust CrashPlan Online to not mess up my data more than a box sitting at my inlaws that might get randomly unplugged... (It will still be local on the unRaid either way.) The ONLY benefit I see of having a box at my in-laws is that I could seed it faster AND get the data back faster. (And I could do something cool (i.e. not worth it) like run CrashPlan on the unRaid O/S so it would wait listen for my box to backup to it. That would allow me to keep adding drives over there. Again not worth it.) Last edited by rdellar17; 02-21-2011 at 05:59 PM. |
#10
|
|||
|
|||
The thing that has stopped me from moving to Crashplan (I'm currently using Spideroak) is the lack of offficial support for backing up network drives - with my setup this is necessary. Are you using the unofficial method or do you have no need to backup network drives?
Thanks! T |
#11
|
|||
|
|||
Quote:
I think the reason they don't want to support it, at least according to a support post by the company owner, is because there isn't a super stable way to handle network disconnects, network failures, etc since it all relies on windows doing the right thing. They said they could do it, but that it isn't in their top 10 list of requests. (They trust their servers and USB drives since they fail in a very specific/standard way.) See this post for more details: https://crashplan.zendesk.com/entrie...tached-storage (Matthew Dornquast's posted a few times on Aug 3, 2010 with a semi-official reply.) There are three methods... All unsupported by official crashplan support for the reasons listed above.
As for me at this moment I don't need it. I only have a single usb 2 gb external drive. I use SyncBack to mirror over to it. However at some point this will become an issue with the new setup. If I go unRAID I'll go with the method that runs CrashPlan on the unRAID server most likely. (Or if I chicken out I'll go with the casey-tech option. ) This would be way easier if unRAID ever releases 5.x. since it will probably just have a plugin to do it. I could stick with syncBack (it has worked well for years), but it is harder to tell if your source files were corrupted/deleted or something else. With CrashPlan you can see all that and control when they are removed, if ever. (I'm only trusting SyncBack with my 'Painfully replaceable' files now. For the 'non-replaceable' files I need something that keeps old versions/check integrity/etc. I know SyncBack does some of this, but it is limited since it has to deal with the constraints of readable files on disk.) Last edited by rdellar17; 02-22-2011 at 09:30 AM. |
#12
|
|||
|
|||
Wow - thanks for the detailed response... I do keep chickening out as it is unsupported, which is a shame as crashplan does seem ideal in every other way... Maybe I'll reconsider soon when my Spideroak subscription expires..
|
#13
|
|||
|
|||
Quote:
Mine is renewed thru 2012 so the coupon is of no use to me... But maybe someone else is interested. From the FAQ LINK: (Talks more about what Mozy is doing...) Quote:
Last edited by rdellar17; 02-22-2011 at 12:47 PM. |
Currently Active Users Viewing This Thread: 1 (0 members and 1 guests) | |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
New Server - SageTV+unRAID under ESXi 4.1 | stanger89 | Hardware Support | 211 | 01-07-2012 10:38 PM |
Requested video was not found on the server | bastafidli | SageTV Linux | 11 | 04-22-2010 10:23 PM |
Backup and recovery strategies for your SageTV server | michaeldjcox | The SageTV Community | 2 | 01-07-2010 09:28 AM |
How Does SageTV Media Server for Windows Home Server work? | PhillTheChill | SageTV Software | 7 | 11-16-2007 01:46 PM |
SageTV Announces Support for AMD LIVE!(TM) Home Cinema and Home Media Server | Narflex | Announcements | 0 | 01-10-2007 09:09 AM |