SageTV Server Linux with Comskip for Docker (and unRAID) Support Thread

[BobPhoenix]

Originally Posted by Taddeusz

For me it would be nice to move everything to a single system but would mean a lot of time I really don't have with 2 wee ones.

I currently have 2 "server" systems. One is running vmware esxi, dual quad core 2.66Ghz Xeon LGA 771 system maxed out at 24GB RAM. Plenty fast but really kind of weak in the disk department. Running 2 Windows vm's, 3 Ubuntu vm's, and a single OS X vm.

My SageTV server is on a separate quad core machine with an 8 port SAS card with SATA drives.

I've toyed with the idea of moving SageTV to a vmware vm and doing drive pass-through. However, I learned that feature requires SCSI or SAS drives.

So, I'm really kind of stuck with this really kind of old but still fast enough Xeon system that runs mostly domain and other network stuff. Really kind of underutilized. Actually, with a family all my stuff is old but fast enough. Really have to scrape by with what I have and hope nothing too major fails.

Wish I could strip it all down and start over from scratch but I just don't have the time.

Enough with my sob story. How difficult is it to migrate NTFS drives to unRAID?

I just retired the last of my ESXi servers in favor of unRAID native. But I had an unRAID VM and a SageTV Windows VM on each ESXi server for years. For a while I used RDM to pass through individual drives to the Windows VM. It was SATA drives and did not require SCSI or SAS drives. What I didn't like was not being able to get drive temperatures or to spin them down so switch to passing a card through to the Windows VM but it worked well and as I said didn't require SCSI or SAS drives. My guide was the Atlas thread on the unRAID forums http://lime-technology.com/forum/ind...?topic=14695.0 Specifically the RDM info in this post http://lime-technology.com/forum/ind...8466#msg138466

[stuckless]

Originally Posted by Taddeusz

For me it would be nice to move everything to a single system but would mean a lot of time I really don't have with 2 wee ones.

Wish I could strip it all down and start over from scratch but I just don't have the time.

I hear you... I bought my hardware and drives and they were sitting in boxes for a couple months before I got time to actually set this up. I've been wanting to do this for a couple years now, but finding the time has been hard.

Originally Posted by Taddeusz

Of course. My issue is that I have almost 7.5TB of music & video across 4 1TB and 3 2TB drives. I could probably scrape up enough to buy a single 2TB drive to migrate data over but the process would be time consuming and cause downtime for the current SageTV server.

I might also be able to do migration in "blocks". Do the imported video & music first separately. Then do the recordings only after things are set up correctly.

You could start with a pair of 3tb drives, which gives you 3TB of storage... then migrate a 2 tb drive, and then add that drive into the array, and repeat until done... it'll take some time for sure (This is how I setup my server, actually... I shut it down on a friday night, and was back up and running on the sunday)

[stanger89]

You don't actually have to go all at once, you can do it sort of piecemeal, that's what I did.

I built an unRAID server when I filled up my ReadyNAS. I of course left my standalone SageTV running. Then when unRAID added Virtualization. First I just moved some of my "low risk" stuff, Squeezebox Server, Crashplan, etc. Then I setup a Windows 7 VM and installed SageTV. For a while I ran it parallel with my standalone system, until I was satisfied with the stability. At that point I finally moved my recordings over and decommissioned my standalone server.

I'm doing the same thing now with the SageTV docker, I've got it and my VM running, and once I get some more stuff in place, I'll probably switch everything over to the docker.

[Fuzzy]

Originally Posted by stuckless

I've recently become a fan of unRAID, myself I've run a number of different NAS solutions over the years, but, nothing compares to unRAID in terms of each of setup and maintenance, and not to mention the rich community that has created 100s of docker containers and plugins.

It's free to try (for 30 days)... and for $59 you can support up to 6 drives which fine for most people... or get support for 12 devices for $89.

Honestly, since I've had a SageTV server for so long, I've never found a need for a NAS (shares on the sagetv server have always met all my shared storage needs). For me, what is appealing to UnRaid at THIS point is the easy virtualization - so in my next rebuild (probably in a couple weeks while the rest of the family is gone) I'll be moving to UnRaid, with Sage running in the docker (as well as quite a few other little containerized tasks I'm sure).

[stuckless]

Originally Posted by Fuzzy

Honestly, since I've had a SageTV server for so long, I've never found a need for a NAS (shares on the sagetv server have always met all my shared storage needs). For me, what is appealing to UnRaid at THIS point is the easy virtualization - so in my next rebuild (probably in a couple weeks while the rest of the family is gone) I'll be moving to UnRaid, with Sage running in the docker (as well as quite a few other little containerized tasks I'm sure).

For me the NAS is not really about sharing storage, but rather the ease of adding storage and the fault tolerance. I ran a "raided" linux setup a couples years ago... lost a drive and then I lost everything I fortunately had a backup of my important documents but I lost all my media I later moved to another setup where I stored stuff on separate drives... and I had a 2TB drive fail and I lost my media again So this time I'm banking on unRAID as a fault tolerant solution. The fact that it's also a great virtualization server is bonus.

[Fuzzy]

Originally Posted by stuckless

For me the NAS is not really about sharing storage, but rather the ease of adding storage and the fault tolerance. I ran a "raided" linux setup a couples years ago... lost a drive and then I lost everything I fortunately had a backup of my important documents but I lost all my media I later moved to another setup where I stored stuff on separate drives... and I had a 2TB drive fail and I lost my media again So this time I'm banking on unRAID as a fault tolerant solution. The fact that it's also a great virtualization server is bonus.

Redundant/Parity storage in a single system provides no data safety in my mind - only continued access to a single drive failure. Remote backup is the only 'safe' way to store important information.

RAID has never been about keeping data safe - it's about keeping data readily available.

[Taddeusz]

Originally Posted by Fuzzy

Redundant/Parity storage in a single system provides no data safety in my mind - only continued access to a single drive failure. Remote backup is the only 'safe' way to store important information.

RAID has never been about keeping data safe - it's about keeping data readily available.

Correct, RAID != backup.

[m1k3g]

Originally Posted by stuckless

I've recently become a fan of unRAID, myself I've run a number of different NAS solutions over the years, but, nothing compares to unRAID in terms of each of setup and maintenance, and not to mention the rich community that has created 100s of docker containers and plugins.

It's free to try (for 30 days)... and for $59 you can support up to 6 drives which fine for most people... or get support for 12 devices for $89.

I tried using unraid years ago but for some reason could never get it working correctly. Then I tried Freenas and a couple of others. I settled on a Synology device out of convenience and expediency, as it's (nearly) plug & play. But the thing I realized over the past few years is that it's woefully underpowered and can't really be upgraded, and more powerful Synology devices seem to be horribly expensive.

I prefer to use open source whenever possible, since our house is already littered with obsolete, closed source devices. Xpenology is the open source version of Synology's OS and boots from a USB drive. I built a nice small footprint box with an i5 and 4tb drive and have migrated everything to that, except Sage. There is a version of TVHeadend for Synology but it doesn't seem to perform any where near as well as Sage (plus I've been using Sage for many years). Eventually I'll get everything on the one box. If not, perhaps I'll revisit unraid.

[stuckless]

Originally Posted by Fuzzy

Redundant/Parity storage in a single system provides no data safety in my mind - only continued access to a single drive failure. Remote backup is the only 'safe' way to store important information.

RAID has never been about keeping data safe - it's about keeping data readily available.

Originally Posted by Taddeusz

Correct, RAID != backup.

Agreed.. which is why I never lose anything of real importance... just time... ie, all my personal documents and pictures are fully backed up, externally, so I never worry about losing them. That being said, one of my primary reason to moving to unRAID was so that if I had a single drive failure, there would be enough fault tolerance in the system that I'd not lose everything simply because of a single drive. I don't back up TB of recordings and ripped media, so if I l do lose them, so be it. (unRAID now provides the ability for 2 parity drives, so, in theory you can provide fault tolerance for 2 drives going at the same time, but I'm not there yet)

The other reason for unRAID was to dynamically add storage. Over the past few years I've been having to go in and mass delete recordings, etc, to keep making room (a couple times I didn't have to worry about, since drive failures took care of that for me ). I bought new drives, but then I have stuff all over the place, and it was a mess. Basically unRAID tidies that up for me, allowing me to setup shares that span multiple drives, etc.

I can probably do everything that unRAID does today, for free, in linux using md, lvm, docker and kvm, etc... but the ease in which I can do it in unRAID is worth the $89.

[Fuzzy]

Pooled storage is something I've never liked. Sage handles multiple storage locations so well, I can't see a reason to pool storage. Performance is so much better when sage can spread recordings across the different recording drives than writing it all to one pooled location. Adding storage? I add a new drive, and add a new Recording folder, a KidsMovies folder, and a NonKidsMovies folder. Recording folders are all set to leave 60GB free, which always leaves plenty of room on my drives to drop in a BluRay rip, and let sage work around it on the recording side. Having more than one KidsMovies folder on different drives lets me do things like ReMux from one to another, which ends up running the process 5-10 times faster than doing it on the same drive.

[stuckless]

Originally Posted by Fuzzy

Pooled storage is something I've never liked. Sage handles multiple storage locations so well, I can't see a reason to pool storage. Performance is so much better when sage can spread recordings across the different recording drives than writing it all to one pooled location. Adding storage? I add a new drive, and add a new Recording folder, a KidsMovies folder, and a NonKidsMovies folder. Recording folders are all set to leave 60GB free, which always leaves plenty of room on my drives to drop in a BluRay rip, and let sage work around it on the recording side. Having more than one KidsMovies folder on different drives lets me do things like ReMux from one to another, which ends up running the process 5-10 times faster than doing it on the same drive.

Just one more reason that unRAID is great... It works for you, with wanting multiple drives that are discretely separate... and it works for me where I just want to a "recordings" area that might use 1 or more drives, and when it fills up, I just just add another drive and add it to the share.

[Taddeusz]

Originally Posted by stuckless

Just one more reason that unRAID is great... It works for you, with wanting multiple drives that are discretely separate... and it works for me where I just want to a "recordings" area that might use 1 or more drives, and when it fills up, I just just add another drive and add it to the share.

Do you have your recordings go to it's own pool of drives or is it part of a single pool?

[stuckless]

Originally Posted by Taddeusz

Do you have your recordings go to it's own pool of drives or is it part of a single pool?

I have multiple "shares" such as "recordings" and "movies", etc, but they all share the same pool of drives. The exception is that the recordings are initially written to a cache drive (configured in the share), and then later moved to the "recordings" share... although that is transparent to me... ie, if I check the "recordings" share while a recording is happening I'll see the recording, even though it's initially being recorded to cache.

I'm not sure I really need the recordings going to cache first... I might end up changing that.

With unraid you can configure that "recordings" share can only use drives A and B, and that "movies" can only use drive C, etc... I just opted to give everything the max amount space, and then organize everything using shares and subfolders.

[stanger89]

Originally Posted by Fuzzy

Pooled storage is something I've never liked. Sage handles multiple storage locations so well, I can't see a reason to pool storage. Performance is so much better when sage can spread recordings across the different recording drives than writing it all to one pooled location. Adding storage?

For recordings sure, that's easy. For imported media, I don't want to have to know or care about how much space is available on which drives.

I add a new drive, and add a new Recording folder, a KidsMovies folder, and a NonKidsMovies folder. Recording folders are all set to leave 60GB free, which always leaves plenty of room on my drives to drop in a BluRay rip, and let sage work around it on the recording side. Having more than one KidsMovies folder on different drives lets me do things like ReMux from one to another, which ends up running the process 5-10 times faster than doing it on the same drive.

But then you have to manually manage which drives stuff goes on. When I rip a disc I don't want to have to think about which disc to put it on. If I need to access my rips outside of a front end, I don't want to have to search through 14 drives to figure which "NonKidsMovies" folder it's in.

Pools remove all that headache from the user.

[Taddeusz]

Looks like I need to invest in another IBM m1015 SAS card.

[Fuzzy]

Originally Posted by stanger89

For recordings sure, that's easy. For imported media, I don't want to have to know or care about how much space is available on which drives.




But then you have to manually manage which drives stuff goes on. When I rip a disc I don't want to have to think about which disc to put it on. If I need to access my rips outside of a front end, I don't want to have to search through 14 drives to figure which "NonKidsMovies" folder it's in.

Pools remove all that headache from the user.

meh.. I type faster than I can browse through a folder anyway. That's what searches are for.

[stanger89]

Windows search is pretty much crap, and searching across 12+ drives isn't really ideal.

[Taddeusz]

Ok, I'm trying to get this set up. I've both installed the Community Applications plugin and tried adding it that way as well as adding the github path to the Docker templates to my list of repositories. No matter what I do nothing happens when I click the "Create" button at the bottom. No errors seem to appear or anything. It just does nothing. What am I doing wrong?

[stuckless]

Originally Posted by Taddeusz

Ok, I'm trying to get this set up. I've both installed the Community Applications plugin and tried adding it that way as well as adding the github path to the Docker templates to my list of repositories. No matter what I do nothing happens when I click the "Create" button at the bottom. No errors seem to appear or anything. It just does nothing. What am I doing wrong?

What version of unRAID are you using... I;m running the 6.2-beta23. I've never tested it with anything less than that.

[Taddeusz]

Originally Posted by stuckless

What version of unRAID are you using... I;m running the 6.2-beta23. I've never tested it with anything less than that.

Well, not sure what was going on but I tried it again when I got home. This time from my iPad. It appears to have completed successfully. Weird.