Is unRAID/Docker the right OS for me?

[wayner]

Sorry for the long post, but lots of questions...

I have been playing with SageTV on unRAID/docker and I really like it, I just want to make sure that my expectations are reasonable and that I am not making a mistake using unRAID as my Sage server as opposed to a regular Linux OS like Ubuntu or Windows.

My current setup is that I have a Win7 machine dedicated to Sage. It also runs Plex, PlayON and AirVideo HD server but those are used rarely, often to facilitate playback on iOS devices or for remote playback. I also have a WHS 2011 server that backs up all PCs on my LAN and also has a SageTV license. It is my backup – if there is an issue with my main server I can easily switch an extender to the backup server. The backup server does not have any tuners other than the HDHR OTA tuner that is on the LAN.

I use SyncToy to sync all files between the Sage server and the WHS server – with the exception of recorded TV. Any TV files that I plan to keep I archive to a Videos area that does get synced. These files are also synced to the cloud using Backblaze – I have 5.2TB on Backblaze which took months. I also have many of these files synced to an external HD.

I have eight TVs in my house (nine in summer with the pool cabana) that all have extenders – 2XHD300 plus a bunch of HD200s. But we rarely run more than 3-4 simultaneously – plus the server UI instance.

My main motivation for change is the limitation of 32 bit JVM. After several days I will start running out of JVM and eventually SageTV will die or the PC itself may crash. Recently I have been running some of the extenders off of the backup server and that makes the system much more stable. Being able to use 2GB or more for JVM heap size will be awesome if it means that I never have JVM heap issues. I may even leave the extenders on permanently so that you can start watching as soon as the TV powers on.

My current thinking is as follows – build an unRAID system. On this system I would install, via Docker, SageTV, Plex, AirVideoHD, an Ubuntu VM and a WHS2011 VM to continue to backup my Windows PCs. I would also install SageTV v9 on another system and keep the media folders in sync (other than TV) so this would allow me to continue to have a hot backup that can be used in one minute with an extender reboot. (I actually have a third SageTV install up and running on a PC that is also my Occulus Rift workstation)

1. Is SageTV on unRAID/Docker suitable for me in a system that will be driving a lot of extenders running a demanding UI like Gemstone?
2. Are there any performance downsides to running on unRAID as opposed to the alternatives?
3. How much does CPU speed matter for the unRAID CPU? It will be running some stuff like Comskip and perhaps Handbrake if I can figure out how to do this on unRAID.
4. If I want to be able to hold 8TB of content how big of a parity drive will I need? Should I buy a big 8TB drive to use for parity?
5. Am I better off using more smaller drives or just a couple of big (8TB) drives? I have a bunch of drives from my current Sage server that I could move to this system - 3x2TB plus a 2TB drive.
6. How much memory should I put in this system? I could see using as much as 3-4GB for the JVM heap since currently my experience is that you need at least 250MB per UI. Does 16GB make sense or is that overkill?
7. Any options for backing up Windows PCs other than running WHS2011 in a VM?
8. If I want a web server on the Docker would I use a docker version of Apache2 or should I just run Apache2 under Ubuntu?
9. How would I configure hard drives on unRAID to maximize performance using an SSD as a cache drive?
10. Anything else to keep in mind?

[BobPhoenix]

Originally Posted by wayner

1. Is SageTV on unRAID/Docker suitable for me in a system that will be driving a lot of extenders running a demanding UI like Gemstone?

It should be fine for lots of extenders. I do only run 2 so don't have experience with more however.

2. Are there any performance downsides to running on unRAID as opposed to the alternatives?

I only see advantages with unRAID myself.

3. How much does CPU speed matter for the unRAID CPU? It will be running some stuff like Comskip and perhaps Handbrake if I can figure out how to do this on unRAID.

unRAID itself does not need much CPU unless you go to Dual Parity and even with Dual Parity it isn't much. It is only the VMs and Dockers that could require more and the same power required on Windows for the App would be required with unRAID VM or Docker.

4. If I want to be able to hold 8TB of content how big of a parity drive will I need? Should I buy a big 8TB drive to use for parity

The bigger the better in my opinion. But I have close to 200TB+ of space on my unRAID servers most don't keep that much.

5. Am I better off using more smaller drives or just a couple of big (8TB) drives? I have a bunch of drives from my current Sage server that I could move to this system - 3x2TB plus a 2TB drive.

I would personally setup with new drives that you test thoroughly first. My SageTV drives were not usable because when I tried I had lots of faults. I didn't test the drives out much before I added them to my Windows SageTV servers. With a NAS you want drives that are as reliable as possible if you want long term storage like I do.

6. How much memory should I put in this system? I could see using as much as 3-4GB for the JVM heap since currently my experience is that you need at least 250MB per UI. Does 16GB make sense or is that overkill?

I have 128GB in my VM Server unRAID instance. If you are going to use VMs I would think 16GB as the minimum if I was going to be running very many VM. unRAID itself should probably have 4GB left free for overhead when running VMs maybe only 2GB if only running dockers.

7. Any options for backing up Windows PCs other than running WHS2011 in a VM?

I use Acronis True Image on my Windows PCs as well as WHSv1 for backups. I want 2 backup sources.

8. If I want a web server on the Docker would I use a docker version of Apache2 or should I just run Apache2 under Ubuntu?

Can't help with this don't run a web server other than the SageTV web server. And for that I just installed the SageTV plugin.

9. How would I configure hard drives on unRAID to maximize performance using an SSD as a cache drive?

Not really sure what you mean here. There is nothing in unRAID to temporarily cache files on the cache drive for frequently used files. The cache drive is used to cache files written to the array so that you can write faster to the array. It stores the files on the cache drive to be copied later to the array at the slower speeds. 3:40am is the default time that unRAID uses to start the move from cache drive to array. You can setup folders that exist only on the cache drive for always faster access but that is something that you do yourself not something that unRAID does with frequently accessed files.

10. Anything else to keep in mind?

If I think of any I will edit this post or add a new one.

[wayner]

Originally Posted by BobPhoenix

Not really sure what you mean here. There is nothing in unRAID to temporarily cache files on the cache drive for frequently used files. The cache drive is used to cache files written to the array so that you can write faster to the array. It stores the files on the cache drive to be copied later to the array at the slower speeds. 3:40am is the default time that unRAID uses to start the move from cache drive to array. You can setup folders that exist only on the cache drive for always faster access but that is something that you do yourself not something that unRAID does with frequently accessed files.

Thanks for all your answers.

That question that you don't understand relates to the following - when I changed my SageTV system drive from a mechanical hard drive to an SSD a few years I saw a dramatic pickup in performance - there are threads on this from a few years ago. Presumably this is because stuff like metadata, file lists, etc are quicker to access on an SSD than on a mechanical drive. How do you ensure that you keep the stuff that is accessed most frequently on an SSD rather than having unRAID hit a slower hard drive for this info? Especially if unRAID is having to go through some sort of a virtualization layer to hit the hardware.

[BobPhoenix]

Originally Posted by wayner

Thanks for all your answers.

That question that you don't understand relates to the following - when I changed my SageTV system drive from a mechanical hard drive to an SSD a few years I saw a dramatic pickup in performance - there are threads on this from a few years ago. Presumably this is because stuff like metadata, file lists, etc are quicker to access on an SSD than on a mechanical drive. How do you ensure that you keep the stuff that is accessed most frequently on an SSD rather than having unRAID hit a slower hard drive for this info? Especially if unRAID is having to go through some sort of a virtualization layer to hit the hardware.

If it is info that is stored in the SageTV directory itself then it would be on the faster drive (SSD cache drive for me). If it is info on the recording or import drives then they would be slower access. So for that type of data it should be just as fast as on Windows install.

[wayner]

So then I guess you want your appdata share on the cache drive. How do you do that?

[BobPhoenix]

Originally Posted by wayner

So then I guess you want your appdata share on the cache drive. How do you do that?

I forced it to that location in setting in unRAID. In settings turn on Advanced view and change "Default appdata storage location:" to what you want it to be mine is: "/mnt/cache/appdata/". I also went against unRAID advice and set it up to be Cache ONLY. I didn't ever want it to end up on the array at the expense of possible down time due to cache drive failures. That is why I setup a cache drive pool so that at least a drive failure is protected with the RAID 1 protection of the cache pool. That doesn't affect file system errors so if you are really wanting to protect it you can use the Community Applications plugin backup feature. I don't because I tend to record things all hours of the day and don't have a consistent window for backups. The backup feature in Community Applications stops dockers before doing the backup and I didn't want that happening while a recording is going on. I need to setup something in SageTV and unRAID that would fire off a backup when a recording window opens up. Just don't have time to even research if that is even possible let alone implement it.

[graywolf]

4. If I want to be able to hold 8TB of content how big of a parity drive will I need? Should I buy a big 8TB drive to use for parity

The parity drive has to be the largest (or same as largest) drive compared to the data drives.

So let's say your largest data drive is 4TB, then your parity drive has to be 4tb or larger. And you can only add 4TB or lower data drives until you get a larger parity drive.

So with a 4TB parity drive, and lets say 1x4TB and 1x2TB and 2x1TB drives, you would have 8TB capacity

[Fuzzy]

unRAID also does use free memory for caching data, just like any other modern OS does. For instance, my unRAID server currently has 16GB. a look right now shows that I'm only actually 'using' 6.5GB of that, and the other 9.5GB of that is being used for cache. Another 'quickness improvement' I've made was to install the Dynamix Cache Dirs plugin, which keeps the entire file system directory in memory, so when something needs just a list of files (something that seems to happen quite a lot), it is pulled very quickly, and doesn't need to spin up drives to do so.

[Fuzzy]

Regarding backup, I have seen a lot of discussion on the unRAID forums about using Crashplan. There is a docker for crashplan in the Community Applications, and I did install it just to get a first look, and it looks very easy to deal with. A free account can backup to another computer, including the ability to backup to a friends computer as well. Crashplan automatically detects other computers you've logged into, making it very easy to install it on your desktop, and set your unraid server as a backup destination (and vice versa, for things like your sagetv server folder on the tower to target your desktop).

[wayner]

I need something that allows you to do a bare metal restore and it doesn't appear that Crashplan allows you to do that.

[Fuzzy]

Originally Posted by wayner

I need something that allows you to do a bare metal restore and it doesn't appear that Crashplan allows you to do that.

True. Crashplan is about multisite backup of valuable data, not rapid restoration of a specific system. To me, those are completely different things, and honestly, I tend to treat system and data differently. 'Bare metal system restoration' is really only a time saver. It's the documents and such that are important to me.

[stanger89]

Originally Posted by wayner

I need something that allows you to do a bare metal restore and it doesn't appear that Crashplan allows you to do that.

You can always use Macrium Reflect (which is free) for that. But I can't see you needing to do that a lot.

I still don't really understand what's so special about WHS's backup, but then again I've never used it.

[wayner]

It is not something you do a lot but boy is it useful when you need it. I have used it several times for my Sage server, once when switching from a mechanical drive to an SSD and again when the original SSD failed. I have about 5-6 PCs in the house so it is something that I end up doing at least once per year.

edit - It appears that Win 8 and above have this functionality built in and you can do the backup to a network drive - so that is one option. But I don't know if it is exactly the same as what you have in WHS.

[Fuzzy]

Originally Posted by wayner

It is not something you do a lot but boy is it useful when you need it. I have used it several times for my Sage server, once when switching from a mechanical drive to an SSD and again when the original SSD failed. I have about 5-6 PCs in the house so it is something that I end up doing at least once per year.

I have always used those instances to get myself a 'clean' install, so I can get rid of the bloat that I end up with after a couple years of operating a computer (need a quick tool for a one-time thing, so it's downloaded, installed, used once, and forgotten about). I dont' think I've ever felt the need for a bare-metal restore of any system since the old winxp days (and that was only because it took so long to install in those days that a bare metal restore was actually faster).

[wayner]

That isn't always the case - even a fresh Win install and then running all updates which will likely require multiple reboots will likely take an hour. And even taking ninite it takes a while to install all of the software that you want and then you always forget a few things as well. And then there may be stuff where you can't find license keys, etc.

It was especially useful for restoring my SageTV server. I tend to keep an extra SSD around so if the system is down I can switch the extenders to my backup server, do a restore and be up and running usually within an hour or so.

[stuckless]

FYI.... The "Community Applications" pluigin (hopefully you have it installed) adds a simple backup/restore to the "Settings" / "Community Applications" tab/section.

This is not a full blown backup, but since unRAID provides some fault tolerance in drive failure, this basically schedules a back to backup stuff in the "appdata" share. It also backs up the USB drive as well. This is really just a superficial backup in the event that you need to quickly restore the appdata based on some other accidental event.

The docker.img never needs to be backuped up... it is likely easier to just recreate the docker images using the "my-xxx" templates... I did this recentlly because of a corrupted docker.img and it took a few minutes to completely restore about 4 docker isntances.

[wayner]

Understood that once I move Sage to unRAID then I won't have to worry about backing up my Sage install. However I may want to occasionally back up the Sage folder in case I want to roll back to an old sage.properties or wiz.bin for some reason - that has happened in the past. I know these are both backed up automatically by Sage but I like to keep a few more versions around in case you need to go back to a file from a week ago for some reason.

[KeithAbbott]

To add another unRAID question into the mix, would it be best to set up a user share as the SageTV recording directory, and a separate user share as the SageTV media import directory? Or is there a better way to do that?

[Fuzzy]

Originally Posted by KeithAbbott

To add another unRAID question into the mix, would it be best to set up a user share as the SageTV recording directory, and a separate user share as the SageTV media import directory? Or is there a better way to do that?

The SageTV docker container already sets up shares as needed by Sage. Simplest to just lump things into them.

[panteragstk]

Originally Posted by wayner

Understood that once I move Sage to unRAID then I won't have to worry about backing up my Sage install. However I may want to occasionally back up the Sage folder in case I want to roll back to an old sage.properties or wiz.bin for some reason - that has happened in the past. I know these are both backed up automatically by Sage but I like to keep a few more versions around in case you need to go back to a file from a week ago for some reason.

You can use second copy for this. Simple to set up, but you may have to use another non-unraid machine to do it. I used it on my windows server with great results.