SageTV Community  

Go Back   SageTV Community > SageTV Products > SageTV Linux > SageTV for unRAID/Docker

Notices

SageTV for unRAID/Docker Discussion related to SageTV for unRAID/Docker. Questions, issues, problems, suggestions, etc. relating to SageTV for unRAID/Docker should be posted here.

Reply
 
Thread Tools Search this Thread Display Modes
  #1  
Old 01-06-2017, 10:11 AM
wayner wayner is offline
SageTVaholic
 
Join Date: Jan 2008
Location: Toronto, ON
Posts: 7,373
Moving files from Windows to unRAID

I guess under unRAID I will be using a different FS so I can't just plug in my old data drives. What is the process to transfer? Just copy files over to the new unRAID drives? Assuming that I have .properties files set up then metadata should be good, correct? These files will be a combination of prior SageTV recordings and DVD-rips - plus music and photos.
__________________
New Server - Sage9 on unRAID 2xHD-PVR, HDHR for OTA
Old Server - Sage7 on Win7Pro-i660CPU with 4.6TB, HD-PVR, HDHR OTA, HVR-1850 OTA
Clients - 2xHD-300, 8xHD-200 Extenders, Client+2xPlaceshifter and a WHS which acts as a backup Sage server
Reply With Quote
  #2  
Old 01-06-2017, 10:27 AM
stuckless's Avatar
stuckless stuckless is offline
SageTVaholic
 
Join Date: Oct 2007
Location: London, Ontario, Canada
Posts: 9,711
There are ways, in unRAID to add drives that are NOT a part of the storage pool... but, I don't recommend doing it, since, those drives will not benefit from the parity disk protection.

In setting up unRAID, you will likely have a large parity drive and a number of other drives, and I recommend creating "shares" and letting unRAID decide how to distribute the data across the disks... that way, as things fill up, you can just add more drives and unRAID can just start using those drives.

When I did mine... I started with a 3tb partity disk and a single 3TB drive. I then setup my shares for "media" and I copied a disk from my old setup into unRAID. Once that was done, I then added the drive to unRAID storage pool, and I got the next drive and did the same, until I was done.... It took some time to copy over all the media.
Reply With Quote
  #3  
Old 01-06-2017, 10:30 AM
wayner wayner is offline
SageTVaholic
 
Join Date: Jan 2008
Location: Toronto, ON
Posts: 7,373
I know that unRAID uses a USB drive for the OS, but does it make sense to use a SSD for the partitions where SageTV and other applications are stored? I know that moving to an SSD years ago on Win dramatically improved performance.
__________________
New Server - Sage9 on unRAID 2xHD-PVR, HDHR for OTA
Old Server - Sage7 on Win7Pro-i660CPU with 4.6TB, HD-PVR, HDHR OTA, HVR-1850 OTA
Clients - 2xHD-300, 8xHD-200 Extenders, Client+2xPlaceshifter and a WHS which acts as a backup Sage server
Reply With Quote
  #4  
Old 01-06-2017, 10:41 AM
stuckless's Avatar
stuckless stuckless is offline
SageTVaholic
 
Join Date: Oct 2007
Location: London, Ontario, Canada
Posts: 9,711
Quote:
Originally Posted by wayner View Post
I know that unRAID uses a USB drive for the OS, but does it make sense to use a SSD for the partitions where SageTV and other applications are stored? I know that moving to an SSD years ago on Win dramatically improved performance.
so unRAID allows this via it's "shares". When you create a "share" you can say which drives it can use, and whether or not it will use a "cache" drive, and whether or not it will ONLY use a cache drive. A Cache drive is generally 1 or more SSD drives.

So, if you wanted all your installed apps to be on SSD, you an just creats an "appdata" or "apps" share and then set it to be cache only, and then all the data for those apps would be on SDD.

For Docker, my docker images are all only SSD, but, docker images don't store their configurations in the image (or else you would lose it when docker images get updated), so typically when you are configuring a docker image (like sagetv), you have to tell it where on the unRAID system it can store it's configuration/appication files, and that share can also be a cache share, if you want.
Reply With Quote
  #5  
Old 01-06-2017, 03:32 PM
SomeWhatLost's Avatar
SomeWhatLost SomeWhatLost is offline
Sage Expert
 
Join Date: Jan 2009
Location: earth
Posts: 531
just to add to the confusion...
there is also a plugin for unRAID called unassigned drives (or something to that effect...)
it allows you to install a drive(s) that unRAID can see/use/etc but are not assigned to either the drive pool or the cache pool...
as I saw no reason to have my recordings "protected" by parity, I have my 2 recording drives "unassigned"... works well for me... anyway, just tossing out more options for you
__________________
NOTE: As one wise professional something once stated, I am ignorant & childish, with a mindset comparable to 9/11 troofers and wackjob conspiracy theorists. so don't take anything I say as advice...
Reply With Quote
  #6  
Old 01-06-2017, 03:50 PM
Taddeusz Taddeusz is offline
SageTVaholic
 
Join Date: Nov 2004
Location: Yukon, OK
Posts: 3,919
Quote:
Originally Posted by SomeWhatLost View Post
just to add to the confusion...
there is also a plugin for unRAID called unassigned drives (or something to that effect...)
it allows you to install a drive(s) that unRAID can see/use/etc but are not assigned to either the drive pool or the cache pool...
as I saw no reason to have my recordings "protected" by parity, I have my 2 recording drives "unassigned"... works well for me... anyway, just tossing out more options for you
I used the Unassigned Devices plugin to move my files from NTFS formatted drives to unRAID. It was a long process for me because the drives that I was moving files from I was also planning to use in the array. Mainly because after copying the files over I had to clear the drives for use in the array. It takes a couple hours to clear the drives before they can be added.
__________________
Server: i5 8400, ASUS Prime H370M-Plus/CSM, 16GB RAM, 15TB drive array + 500GB cache, 2 HDHR's, SageTV 9, unRAID 6.6.3
Client 1: HD300 (latest FW), HDMI to an Insignia 65" 1080p LCD and optical SPDIF to a Sony Receiver
Client 2: HD200 (latest FW), HDMI to an Insignia NS-LCD42HD-09 1080p LCD
Reply With Quote
  #7  
Old 01-06-2017, 05:01 PM
KarylFStein KarylFStein is offline
Sage Fanatic
 
Join Date: Apr 2006
Location: Westland, Michigan, USA
Posts: 999
Quote:
Originally Posted by stuckless View Post
so unRAID allows this via it's "shares". When you create a "share" you can say which drives it can use, and whether or not it will use a "cache" drive, and whether or not it will ONLY use a cache drive. A Cache drive is generally 1 or more SSD drives.

So, if you wanted all your installed apps to be on SSD, you an just creats an "appdata" or "apps" share and then set it to be cache only, and then all the data for those apps would be on SDD.

For Docker, my docker images are all only SSD, but, docker images don't store their configurations in the image (or else you would lose it when docker images get updated), so typically when you are configuring a docker image (like sagetv), you have to tell it where on the unRAID system it can store it's configuration/appication files, and that share can also be a cache share, if you want.
It seems like unRAID defaults seem to push VMs and docker data to the cache. I didn't do any special configuration and those things are on my cache SSDs. I guess if you don't set cache only then if you have space issues on the cache it'll move things to the pool?
__________________
Home Network: https://karylstein.com/technology.html
Reply With Quote
  #8  
Old 01-06-2017, 05:25 PM
stuckless's Avatar
stuckless stuckless is offline
SageTVaholic
 
Join Date: Oct 2007
Location: London, Ontario, Canada
Posts: 9,711
Quote:
Originally Posted by KarylFStein View Post
It seems like unRAID defaults seem to push VMs and docker data to the cache. I didn't do any special configuration and those things are on my cache SSDs. I guess if you don't set cache only then if you have space issues on the cache it'll move things to the pool?
If you don't set cache only, then it means all writes happen in the cache, and then transparently, unRAID will move it from the cache to the pool "later" (not sure when later is).
Reply With Quote
  #9  
Old 01-06-2017, 06:14 PM
EnterNoEscape's Avatar
EnterNoEscape EnterNoEscape is offline
SageTVaholic
 
Join Date: Jun 2010
Location: Harrisburg, PA
Posts: 2,657
Quote:
Originally Posted by stuckless View Post
If you don't set cache only, then it means all writes happen in the cache, and then transparently, unRAID will move it from the cache to the pool "later" (not sure when later is).
There's help on the page itself that tells you what they all do. Just click on Use cache disk:
__________________
SageTV v9 Server: ASRock Z97 Extreme4, Intel i7-4790K @ 4.4Ghz, 32GB RAM, 6x 3TB 7200rpm HD, 2x 5TB 7200rpm HD, 2x 6TB 7200rpm HD, 4x 256GB SSD, 4x 500GB SSD, unRAID Pro 6.7.2 (Dual Parity + SSD Cache).
Capture: 1x Ceton InfiniTV 4 (ClearQAM), 2x Ceton InfiniTV 6, 1x BM1000-HDMI, 1x BM3500-HDMI.

Clients: 1x HD300 (Living Room), 1x HD200 (Master Bedroom).
Software: OpenDCT :: WMC Live TV Tuner :: Schedules Direct EPG
Reply With Quote
  #10  
Old 01-06-2017, 07:14 PM
KarylFStein KarylFStein is offline
Sage Fanatic
 
Join Date: Apr 2006
Location: Westland, Michigan, USA
Posts: 999
Just looked and the default for system/domains/appdata is Prefer. It writes to the cache disk unless full in which case it'll write to the array. BUT if space clears up it'll move back to the cache on the next mover invocation. Seemed sensable to me and keeps VMs / Docker stuff on the SSD as long as there is space. Only will fail on write if the cache disk is full. Anyway, all this was automatically done for me, but I also started with an SSD cache drive, a single parity drive and a single data drive, (since expanded to 2 cache, 1 parity and 3 data). If you don't start with a cache drive maybe it requires manual tweaking after.

Unassigned devices also lets you mount network shares. I did some testing of copying data from my Windows server that way. Faster to just mount locally, but can be used to get fairly static things moved over without causing an outage.
__________________
Home Network: https://karylstein.com/technology.html
Reply With Quote
  #11  
Old 01-07-2017, 05:09 AM
stuckless's Avatar
stuckless stuckless is offline
SageTVaholic
 
Join Date: Oct 2007
Location: London, Ontario, Canada
Posts: 9,711
Quote:
Originally Posted by KarylFStein View Post
Just looked and the default for system/domains/appdata is Prefer. It writes to the cache disk unless full in which case it'll write to the array. BUT if space clears up it'll move back to the cache on the next mover invocation. Seemed sensable to me and keeps VMs / Docker stuff on the SSD as long as there is space. Only will fail on write if the cache disk is full. Anyway, all this was automatically done for me, but I also started with an SSD cache drive, a single parity drive and a single data drive, (since expanded to 2 cache, 1 parity and 3 data). If you don't start with a cache drive maybe it requires manual tweaking after.

Unassigned devices also lets you mount network shares. I did some testing of copying data from my Windows server that way. Faster to just mount locally, but can be used to get fairly static things moved over without causing an outage.
I didn't start out with a cache drive, so likely, that is why, for me, I had to manaully tweak this stuff later.
Reply With Quote
  #12  
Old 01-08-2017, 01:38 AM
Fuzzy's Avatar
Fuzzy Fuzzy is offline
SageTVaholic
 
Join Date: Sep 2005
Location: Jurupa Valley, CA
Posts: 9,957
Regarding the moving of data into unRaid, I did the same as Taddeusz above. I went ahead and added all the drives of concern to unRAID, as well as a single new drive. I added only the new drive to the pool (didn't have a parity drive yet). I then installed the SageTV Docker, and let it create the recordings and import folders. I then installed the Unassigned Devices plugin, which allowed me to browse all the attached NTFS drives from my old windows server. I picked the largest NTFS, and moved the files to the appropriate folders in the pool, then formatted/cleared the drive, and added it to the pool. Then went on to the next largest NTFS, and so on. Did this a little at a time (I actually configured Sage to look at those unassigned drives as import folders, so sage still had read access to those recordings during this process). Eventually, I had all the drives in the pool, and all the sage content in the correct folders. I added a parity drive a couple days later after I received the new one (my previous 'new' drive was 3TB, and the parity has to be at least as big as the biggest singular drive in the array), and one installed, manually started a parity check to populate the parity drive.
__________________
Buy Fuzzy a beer! (Fuzzy likes beer)

unRAID Server: i7-6700, 32GB RAM, Dual 128GB SSD cache and 13TB pool, with SageTVv9, openDCT, Logitech Media Server and Plex Media Server each in Dockers.
Sources: HRHR Prime with Charter CableCard. HDHR-US for OTA.
Primary Client: HD-300 through XBoxOne in Living Room, Samsung HLT-6189S
Other Clients: Mi Box in Master Bedroom, HD-200 in kids room
Reply With Quote
  #13  
Old 01-21-2017, 01:12 PM
wayner wayner is offline
SageTVaholic
 
Join Date: Jan 2008
Location: Toronto, ON
Posts: 7,373
Is there any reason not to just copy over the network from my old server to my new unRAID server. It will be much slower but if you aren't in a rush that isn't a big deal - just let it run overnight. I have about 7TB of data to move over.
__________________
New Server - Sage9 on unRAID 2xHD-PVR, HDHR for OTA
Old Server - Sage7 on Win7Pro-i660CPU with 4.6TB, HD-PVR, HDHR OTA, HVR-1850 OTA
Clients - 2xHD-300, 8xHD-200 Extenders, Client+2xPlaceshifter and a WHS which acts as a backup Sage server
Reply With Quote
  #14  
Old 01-21-2017, 01:28 PM
Fuzzy's Avatar
Fuzzy Fuzzy is offline
SageTVaholic
 
Join Date: Sep 2005
Location: Jurupa Valley, CA
Posts: 9,957
Quote:
Originally Posted by wayner View Post
Is there any reason not to just copy over the network from my old server to my new unRAID server. It will be much slower but if you aren't in a rush that isn't a big deal - just let it run overnight. I have about 7TB of data to move over.
That's fine, if you have twice as much hard drive as you have content. I was reusing the drives from my old server in my unRAID server.

For information, copying a single file from my desktop to my unraid tower over gigabit runs at about 70MB/sec. So your 7TB of data would take around 30 hours.
__________________
Buy Fuzzy a beer! (Fuzzy likes beer)

unRAID Server: i7-6700, 32GB RAM, Dual 128GB SSD cache and 13TB pool, with SageTVv9, openDCT, Logitech Media Server and Plex Media Server each in Dockers.
Sources: HRHR Prime with Charter CableCard. HDHR-US for OTA.
Primary Client: HD-300 through XBoxOne in Living Room, Samsung HLT-6189S
Other Clients: Mi Box in Master Bedroom, HD-200 in kids room

Last edited by Fuzzy; 01-21-2017 at 01:41 PM.
Reply With Quote
  #15  
Old 01-23-2017, 08:40 PM
wayner wayner is offline
SageTVaholic
 
Join Date: Jan 2008
Location: Toronto, ON
Posts: 7,373
@Fuzzy - do you know what the speed was when you moved your files over using this methodology? When using the File Explorer in Win 10 my transfers start at about 110MBps, which uses pretty much the full network bandwidth of 1Gbps but then it slows down to about 10MBps. But I wonder what is the limiting factor, is it the parity drive? Note that my parity drive is still building right now. I think you can also stop the array while you are transferring files.
__________________
New Server - Sage9 on unRAID 2xHD-PVR, HDHR for OTA
Old Server - Sage7 on Win7Pro-i660CPU with 4.6TB, HD-PVR, HDHR OTA, HVR-1850 OTA
Clients - 2xHD-300, 8xHD-200 Extenders, Client+2xPlaceshifter and a WHS which acts as a backup Sage server
Reply With Quote
  #16  
Old 01-23-2017, 09:14 PM
Fuzzy's Avatar
Fuzzy Fuzzy is offline
SageTVaholic
 
Join Date: Sep 2005
Location: Jurupa Valley, CA
Posts: 9,957
No I was copying them in Midnight Commander so I didn't watch the transfer time.
__________________
Buy Fuzzy a beer! (Fuzzy likes beer)

unRAID Server: i7-6700, 32GB RAM, Dual 128GB SSD cache and 13TB pool, with SageTVv9, openDCT, Logitech Media Server and Plex Media Server each in Dockers.
Sources: HRHR Prime with Charter CableCard. HDHR-US for OTA.
Primary Client: HD-300 through XBoxOne in Living Room, Samsung HLT-6189S
Other Clients: Mi Box in Master Bedroom, HD-200 in kids room
Reply With Quote
  #17  
Old 01-24-2017, 06:37 AM
stuckless's Avatar
stuckless stuckless is offline
SageTVaholic
 
Join Date: Oct 2007
Location: London, Ontario, Canada
Posts: 9,711
Like Fuzzy... when I do network copies... I see about 70mb/s consistent... i don't really see a burst and then a slowdown... it's just fairly consistent.
Reply With Quote
  #18  
Old 01-24-2017, 06:43 AM
stuckless's Avatar
stuckless stuckless is offline
SageTVaholic
 
Join Date: Oct 2007
Location: London, Ontario, Canada
Posts: 9,711
I did some googling and found a few articles on performance tuning the unRAID system... I'm ok with my performance, but these might be of interst to those trying to get more IO bandwidth.

https://lime-technology.com/forum/in...?topic=31073.0

https://lime-technology.com/wiki/ind...ID_Performance

http://lime-technology.com/wiki/inde...ser_Benchmarks
Reply With Quote
  #19  
Old 01-24-2017, 08:26 AM
wayner wayner is offline
SageTVaholic
 
Join Date: Jan 2008
Location: Toronto, ON
Posts: 7,373
Now that my array is built i am getting much faster transfers just using Windows File Explorer to transfer media files from one of my existing SageTV servers. I seem to be able to sustain close to 100MB/s for at least thirty minutes. That's probably good enough for me as that is about 0.36TB/hour.

The only issue is that occasionally you get this stupid dialog box about certain properties not able to transfer over to the new drive. You have to then confirm that. You don't want that to happen a few minutes into an overnight file transfer.
__________________
New Server - Sage9 on unRAID 2xHD-PVR, HDHR for OTA
Old Server - Sage7 on Win7Pro-i660CPU with 4.6TB, HD-PVR, HDHR OTA, HVR-1850 OTA
Clients - 2xHD-300, 8xHD-200 Extenders, Client+2xPlaceshifter and a WHS which acts as a backup Sage server
Reply With Quote
  #20  
Old 01-24-2017, 08:56 AM
Taddeusz Taddeusz is offline
SageTVaholic
 
Join Date: Nov 2004
Location: Yukon, OK
Posts: 3,919
Dang, so I have a few drives to replace if I want better performance. I ran that performance script and my slowest drive is an old WD Green 1TB. It averages 75MB/s and of course is only 5400 RPM. I have two other drives that average below 100MB/s that I will also be replacing as early as I can.
__________________
Server: i5 8400, ASUS Prime H370M-Plus/CSM, 16GB RAM, 15TB drive array + 500GB cache, 2 HDHR's, SageTV 9, unRAID 6.6.3
Client 1: HD300 (latest FW), HDMI to an Insignia 65" 1080p LCD and optical SPDIF to a Sony Receiver
Client 2: HD200 (latest FW), HDMI to an Insignia NS-LCD42HD-09 1080p LCD
Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Moving Recordings From Windows to Linux Taddeusz SageTV Github Development 16 12-28-2015 05:56 PM
Moving from Windows to Linux jmode SageTV Linux 4 06-23-2011 12:31 PM
No playback of files on UnRAID server ckewinjones SageTV Software 10 02-12-2011 10:44 AM
Moving a Sage install to a Windows 7 box tonysathre SageTV Software 15 01-06-2010 02:28 PM
Moving Sage Windows To WHS mayhewda SageTV Software 5 10-27-2009 08:26 AM


All times are GMT -6. The time now is 06:03 PM.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, vBulletin Solutions Inc.
Copyright 2003-2005 SageTV, LLC. All rights reserved.