Advice for moving to unRaid/Sage/OpenDCT production

[MacDaddy]

Hello all the experts out there. I've been kicking the tires on the unraid/docker/sagetv/opendct adventure and I really like it. I'm starting to get my mind wrapped around it and know I've made some rookie mistakes as I climb the learning curve. I'm ready to move this to my production system and intend to blow everything away and do a fresh install. I would appreciate any advice on the following tactics.

Current platform: unRaid 9.3.3 on Supermicro X10SL7-F with Xeon E3-1276 @ 3.6GHz and 32GB memory plus 2x1Gb/s NICs. There are 5x4TB data, 1x4TB parity and 1 500GB SSD cache drive.

My first time out I installed the sagetv docker before the opendct docker. I realized the error of my ways.

On my old system, I had 4 HDHR OTAs and 3 HD-PVRs/Uverse as tuners. The old server had 2 NICs with one connected to the backbone network with the rest of the equipment (router/gateway/DHCP on this segment) and all the HDHRs were attached via a gigabit switch to the second NIC (no DHCP, would use autoconfig). The intent was to isolate the traffic.

When I originally set up the docker, I knew to put the image on the cache drive. It was easy to select it from the list on setup. However the /appdata was a little more confusing. There was not a cache option in the listing. I created an /appdata on the cache drive and manually placed it in the setup option box, but it showed in the mappings as /mnt/user/appdata. When I look at the /mnt/user/appdata, it is on cache and disk4. There are only two directories - opendct and sagetv. The sagetv directory shows all on cache. The opendct directory shows cache and disk4.
-Do I need to force /appdata on cache drive or should I use "prefer" on the share settings to let unRaid get it to cache?
-Is there any reason why opendct would need to put things on disk4? (might have been my fault at installation)
-Any reason to expand beyond the 10GB image size at this point?
-Any reason to increase JAVA memory over 1024MB?
-is it worth adding a second 500GB cache drive for fault tolerance?

In my current test setup I have one HDHR OTA and a new HDHR Prime on the same network with the unRaid server. The Prime is controlled by OpenDCT and the OTA is direct to Sage. I plan to move these over to the extra NIC, have them autoconfig IP addresses and add the 3 remaining HDHR OTA to the mix.
-Are there any issues getting OpenDCT to see them?
-Is there any advantage to using OpenDCT to control the HDHR OTAs?

The Sage server was super easy to deploy. I went the java8 route. I added the plugins for - BMT, CMT, Gemstone, SageTCPServer, Web Interface, UPnP and Youtube. I haven't enabled fanart yet.
-I have 11GB of existing fanart. Should that be on cache?
-There are existing samba shares for movies and series. What's best practice to attach these back to unRaid sage?
-I'm assuming I want to get the fanart folder recognized by sage before reconnecting the other directories. Will BMT get new fanart if the central directory is not seen?


Thanks for any advice. I'm really excited to enter the V9 world and breath new life in my Sage installation.

[stuckless]

I don't know if there is any right or wrong ways to configure unRAID... In my setup I actually don't have appdata on a cache share, mainly because I have a fairly small cache drive (ssd ~128G).

I think when you have a share set to cache only (which I would do for appdata) it should not show up on any other disk. If you see that, it's likely because at some point the share was set to be non-cache or prefer cache, etc. There is a plugin in unraid that can go through and find those kinds of things, and clean up the other files, etc. I'm not at home, so I forget what it's called, but it might be something like "Fix common problems" or "Clean..."

I use my cache drive for the docker.img (which I set to 50g) and for immediate recordings. My sagemedia share (which docker image records to) is a cache share and the recordings get moved to the main array later. I do this because the IO throughput on the ssd would be better than a standard disk in the array. In this way, multiple recordings happening at the same time, plus running comskip, hasn't resulting in any disk IO issues in my setup.

As I mentioned I set my docker.img to 50G. I did this because depending on how many docker instances you run, 10G can fill up pretty quickly, especially if they are not configured to write logs to outside the container, etc. I've had docker crash because the docker.img filled up I run a number of docker containers (Plex, Emby, etc), so I would make the docker.img bigger than the 10G default.

As for the fault tolerance of the cache drive... If you are storing appdata as cacheonly, then you will want some sort of fault tolerance... either a built into the array itself (second cache drive), or, you will want to run a backup utility to backup appdate to the main array (I think the Community Applications plugin has a backup tool that can run every night to do this).

The other thing you might want to do.. is backup the usb flash drive OUTSIDE of unRAID. If that USB drive dies and you don't have a backup, then getting your config back can be a tedious process.

[MacDaddy]

Thanks for the advice. I'll certainly start with a larger image size this next time. I really need to take the time to go through all the community applications. It's really impressive how many things are there. I wonder how many beers I would have to buy Fuzzy to convince him to place OpenDCT in the community applications pool:-)

[dstanley]

Originally Posted by MacDaddy

-is it worth adding a second 500GB cache drive for fault tolerance?

-There are existing samba shares for movies and series. What's best practice to attach these back to unRaid sage?

I think you should be fine with the one cache drive - go to SETTINGS - SCHEDULER and set the MOVER settings in there - this will move files OFF the cache drive and into the array for the 'PREFER' shares. I have mine set HOURLY and believe that should not create any problems.

You need some plugins to help with the Samba shares -
UNASSIGNED DEVICES: This lets you connect to Samba shares as well as any disks that are not part of the array. If you need the Samba shares from within SageTV then you need to assign them in the Docker settings so that Sage can see/use them.
KRUSADER: This is a great two-paned file manager that lets you easily copy and move files - either from the array/array or unassigned devices/array.

There are many other wonderful plugins that you will discover and use - this is my first venture into something like this Unraid and Docker and I am just loving it!

Look for an watch many of the great videos by SPACEINVADER ONE on youtube. He has helped me immensely with understanding the concept of array, cache, Krusader, etc.
https://www.youtube.com/channel/UCZD...N0WeAPvMqTOrtA

Keep us informed of your progress!

Dwight

[MacDaddy]

Thanks for the tip on SpaceinvaderOne. Very useful. He's made a number of great videos. I'll need to take a little more time to view them all.

I think I'm getting the hang of the "prefer" option. In addition to a few more of the community apps I should be able to clean up some of the non-cache mistakes I made the first time.

I still need to research what happens if I put the HDHR devices on a dedicated NIC with autoconfig.

[MacDaddy]

I'm up and running for 24hrs now with no issues. I was a little intimidated with the docker methodology. My memories of the early days of getting Sage to work seemed to require magic and sacrifices to the gods of USB. If I had realized how approachable it was then I would have made the jump to V9 much earlier.

Currently running the docker/sage/opendct approach with 8 HDHR OTA tuners, 3 HDHR Prime tuners and 3 HD-PVR/Uverse/iTach tuners. I have the Prime on Comcast and the Uverse of course with ATT. Given how great the Prime has been, I will most likely decommission Uverse at the end of the month.

Here's the general laundry list of I what I did to get here.
- While the HDHRs were on my general LAN I used my windows machine with the latest HDHomeRun Setup to make sure that everything was on the latest firmware. Then I powered them down.
- I placed all the HDHR devices behind an old GS108 switch. The switch was connected to the second NIC on my unRaid server.
- I enabled the second NIC. Bridging was not enabled. IP assignment was "Automatic". It complained about lack of DHCP but eventually settled on an autoconfig address.
- I went to the Scheduler settings and set up Mover to run hourly.
- I powered up all the HDHRs. I could see the indicator lights on the HDHR OTA devices blinking (not connected).
- I enabled a docker image with 10GB size on cache. I will grow this if/when I load more containers.
- The appdata share was set up with cache "prefer". I believe this will do everything to keep the appdata on cache, but allow it to move to the array if space exceeded. Then it will come back as space is available. However it means it is not protected.
- I loaded the CAdvisor container first. I wanted to have insights in to the resources consumed as I brought on more containers. Load from the Apps page with full defaults.
- I loaded the CA Backup/Restore Appdata plugin. For the second option I chose to backup to a user share. I also selected to have a daily backup, place the backups in dated subfolders, and delete backups after 14 days.
- I loaded OpenDCT using the crazifuzzy-opendct container. This is not on the Community Applications page. You first need to put "https://github.com/crazifuzzy/opendct-docker" in the template repositories area. Then you can load the container and use the defaults offered. Remember this has to be started before Sage. When it is loaded and running, I saw the HDHR indicator lights go solid.
- I loaded SageTV in the "stuckless-sagetv-server-java8" container next. I increased the Java memory option to 1024MB. While I have a Sage license key, I did not use it because I intended to use SchedulesDirect for EPG. All other fields were left at defaults.
- I stopped the server. I copied my custom GENTUNER script in the SageTV server share. I started the server.
- I loaded SagePlaceshifter on my windows laptop. I then connected to the new Sage server and started the standard setup wizard.
- I did not set up any additional media directories on this first pass. I went to the Setup Video Sources and began adding the different sources. There will be a set of sources that end with ":9000". These are the sources that can be controlled by OpenDCT. While it is possible for SageTV to control the HDHR OTA devices natively, I chose to have all the HDHR device under OpenDCT control.
- These will set up just like a conventional source. You will be prompted to perform full channel scan for available channels. I did the channel scan for the first OTA source and declined the rest. The first source triggered a download of the channels for "Local Over The Air Broadcast - 77096". I went through the lineup and enable the channels I wanted. You can use the channel preview here to confirm if needed. I used this same lineup for all the remaining OTA sources and declined channel scans.
- The HDHR Prime was similar. I used the "Comcast - Digital - Houston" lineup. There are three tuners to set up. I believe it also asked for a channel scan, but I'm a little fuzzy here. Once the lineup is downloaded you can enable the channels you want. This one can also preview a channel if needed.
- The HD-PVR was set up to use the GENTUNER with my custom developed script for the Global Cache iTach IR blaster.
- I closed Placeshifter and went to continue on the HD200.
- My media libraries are already on samba shares on unRaid. I'm not certain if this is best practice, but I set up /unraid/DVD and /unraid/TV as media center import libraries. It appears that you get the unRaid presented file system without the samba overhead.
- I had an existing central fanart directory from my previous Sage instance. I changed that to "use cache disk - only" and it is ready for fanart processing. I need to think on leaving this one unprotected.

At this point I loaded Gemstone and I've been playing with that. I'll share that adventure on another day.

[dstanley]

There is a pretty neat docker called "CA Docker Autostart Manager" that will let you start certain dockers before others if that is important to getting your system restarted correctly.

Awesome Community App for you to check out!
http://lime-technology.com/forum/ind...?topic=53731.0

Dwight

[MacDaddy]

Excellent idea. It's important to start DCT before Sage.

Thanks for the info.

[jbuszkie]

This is a very informative thread! I'm planning on doing this very thing in the near future! I need to upgrade my Unraid box first! (core2 Duo!)
Bookmarked!

[Fuzzy]

A few notes:
With SageTV v9, it's really only important for openDCT to be running before Sage when initially detecting tuners. This is because Sage will only look for new network tuners when it's service is starting up. However, once a tuner's mmc entries are in the sage.properties file, that has already been completed. On subsequent starts, if sage doesn't see the tuners that it knows are out there, it will continue to check for them every couple seconds util it does see them. For this reason, the staggered start of the dockers is not really all that necessary. If you do add new tuners to openDCT, all it takes is a quick restart of the sagetv docker and on it's startup, it will pick up anything new that openDCT has to offer.

Also, while I understand the desire to separate the HDHR traffic off of the main network, it is probably better to have some sort of DHCP on the secondary network. Auto-configured IP addresses can cause problems, and slow startup of each device quite a bit. DHCP, especially with static leases, provides a much more stable network environment, as well as keeping your HDHR's on the same address each time, for easier access to the built in web pages if needed for troubleshooting. This could be accomplished by bridging the NIC's on the unRAID server, essentially creating a virtual switch between your two segments. Traffic will still be isoalted between the two, as only the unraid server will be talking with the HDHR's most the time, but everything will reside on the same subnet, so anything CAN be accessed fromeither side, including the DHCP server. Now, the downside of bridging through the unraid server, is the bridge has to be up for the HDHR's to find the DHCP, so powerup order begins to matter. The other option is to use an old router laying around to serve up the dhcp on the secondary network. Simple, cheap and effective, and would likely eliminate issues of the server being down when the HDHR's boot up.
Honestly, though - I don't think the secondary network is even required at this point. A single gigabit connection to your server is likely plenty for even a large number of tuners. Even 11 full bandwidth OTA streams maxes out at about 200 Mb/sec. Even if all 11 were being watched live, that's 200Mb/sec going into the unRAID server, and then 200Mb/sec going back out to the clients. This should work just fine, and I'm guessing this isn't a common worst-case scenario.