STV progress; virtualization + NAS

[drvnbysound]

TLDR upfront: I want to use this thread to document some thoughts and progress of my STV setup, including virtualization, NAS selection and setup, etc. I was recently involved in a lot of discussion about my system in another thread (which I kinda hi-jacked) so I wanted to start a dedicated thread. So here goes:

Introduction:
I didn't get my STV system setup until shortly after the Google purchase, but I've fought with myself regarding virtualizing STV since around that time. Right now I've got a dedicated desktop for daily use, a dedicated STV server, another machine that I was (at one point) going to dedicate for a CCTV server [but decided to go with a pre-built DVR appliance instead], as well as some other misc. spare parts.

I've always had some network security concerns, but never really cared enough (low priority) to do anything about it. I've got co-worker/friend who is an IA/IT specialist and he likes to send me results of blocked attempts to his home network; I don't know his exact setup, but I know he has a Cisco router (among other enterprise hardware) at his home for testing (lab for learning/certification). Knowing that the attempts aren't necessarily targeted attempts to his network, and that my network (and yours) receive the same attempts I finally decided to take some action, albeit a small step...

About 2 months ago I flashed my WRT54G router with DD-WRT, and got some firewall rules which were provided to me. This is currently working well. After it was setup, he asked me one evening if I could 'see' his attempts to access my network. I followed that with a question of, 'what do I need to do see you attempts?' To which he responded, network monitoring software.

I'm now looking at all sorts of software tools which are not very processor/memory intensive. Most of these tools are designed to run as stand-alone appliances, generally in VMs, to which their interfaces are accessed via web-based GUIs.

All of this has now led me to re-considering and moving toward virtualizing STV and running various other VMs via ESXi on my current STV hardware.

Where I am now:
So, based on the above, and the discussion that was had in the previous thread (here)... I decided it would be a good idea to document my current hardware, hopefully in an effort to justify that I'm making (or attempting to make) sound decisions. We utilize STV on a daily basis now, my wife enjoys and uses it, and we've got plenty of content available for our 4mo old (when needed) as well.

Here is a listing of my current STV hardware:

OS: Windows 7 Professional, x64
Case: Antec Three Hundred Mid-Tower
Power Supply: Antec BP550
Motherboard: Gigabyte GA-EP45T-UD3LR rev 1.1
Processor: Intel Core2Quad Q9650, 3.0GHz
RAM: 8GB (4 x 2GB), DDR3 (plan to upgrade to 16GB for ESXi)
HDDs:

• (1) WD5002AALX, 500GB, SATAIII - OS
• (2) WD20EARS, 2TB, SATAII - currently dedicated for Movies
• (2) WD10EACS, 1TB, SATAII; currently dedicated for DVR

Video: nVidia GeForce 9500GT, 1GB DDR2 (also likely to be moved as I only use extenders)
Drives: Lite-On BD-ROM (likely to remove and place in primary desktop for BD rips)

Here is the hardware listing for the other computer I built (currently collecting dust):

OS: Windows 7 Home, x86
Case: Antec Three Hundred Mid-Tower
Power Supply: Antec BP550
Motherboard: Gigabyte GA-G41MT-S2PT vx.x (likely to swap and use elsewhere)
Processor: Intel Core2Quad Q9300, 2.5GHz
RAM: 2GB, DDR3 (currently installed; will be replaced with min. 4GB when MOBO is swapped; preferably 8GB (4 x 2GB), DDR3 from STV/ESXi server)
HDD(s):

• (1) WD5000AADS, 500GB, SATAII - OS

Video: Integrated

Regarding my STV setup, I have a total of 6 tuners; (2) HDHRs and (2) HD-PVRs. Due to the HD-PVRs, I will utilize USB-Passthrough. As mentioned above, I am only accessing STV via extenders (currently (3) HD-200s).

Some have suggested that I just run a dedicated STV server (as I'm doing now) and just put the networking VMs on the secondary machine. Personally, this isn't something that I want to do. Why? Well, mainly because the specs of that machine; the Core2Quad processor for starters. It would likely display 1% CPU usage because that's the minimum that it could display. I also don't want to lose the physical space that I could dedicate to HDDs. So, what am I thinking? I'm proposing (to myself) that I virtualize (with ESXi) the machine that's currently dedicated to STV, and utilize the second machine for a NAS (likely unRAID or tRAID). Doing so, I will be moving some parts around, and likely upgrading some others... some of these I've notated above. I'll utilize the next post to detail what the proposed configuration(s) may be...

[drvnbysound]

Proposed hardware configurations: (changes are notated with an *)

ESXi Server (currently dedicated STV server):

Case: Antec Three Hundred Mid-Tower
Power Supply: Antec BP550
Motherboard: Gigabyte GA-EP45T-UD3LR rev 1.1
Processor: Intel Core2Quad Q9650, 3.0GHz
* RAM: 16GB (4 x 4GB; max. memory supported by MOBO); G.Skill or similar
HDDs:

• * (2) 500GB, SATAIII - ESXi; RAID 1 (also to be used as primary datastore)
• (2) WD20EARS, 2TB, SATAII - dedicated for DVR storage (previously dedicated 'Movie' drives)

*Video: Integrated (remove existing video card due to lack of utilization; will likely use video card in another to-be-built-PC)

NAS (unRAID/tRAID) machine; currently un-utilized machine:
Case: Antec Three Hundred Mid-Tower
Power Supply: Antec BP550
* Motherboard: TBD; requirements: LGA775... Nice-to-have: On-board RAID for OS; DDR3; Dual GB NICs
Processor: Intel Core2Quad Q9300, 2.5GHz
* RAM: TBD; MOBO dependent, will likely use max. supported; preferably 8GB (4 x 2GB), DDR3 from STV/ESXi server
HDD(s):

• * If tRAID: (2) xxxGB, SATAIII; RAID 1 host OS
• * Min. (5) (up to 9; physical space limitation) 2TB, SATAIII for storage volumes

* HDD Cage: Supermicro CSE-M35T-1B, (3) 5.25" to (5) 3.5"
* RAID controller: TBD; recommendation IBM ServeRAID M1015
Video: Integrated

The motherboard change for the unRAID server may not be necessary, and is only a consideration at this point. However, there are certain limitations to the existing MOBO in that machine that are having me lean towards a replacement; some of these include minimal RAM slots and supported capacity, and lack of on-board RAID controller if a host OS is required (e.g. tRAID).

I just priced the cost of HDDs and the price per GB is about the same for 2GB and 4GB drives. That said, I could get away with less drives at the same cost for a particular amount of storage. My consideration here is that I don't expect to max out the NAS in any near time. I currently have ~150 movies stored (no compression) and with the 4TB I have now, I still have ~80% free space. I don't intend to be a DVR hoarder, and presently only plan to utilize the NAS for the movie library.

[drvnbysound]

This thread is worthless without pics

Here is a shot of my current STV server as I was just finished with the hardware build (taken in Sept of 2011). This is the machine that will likely be dedicated to ESXi, including the virtualization of STV:

Sorry for the crappy cell phone quality pic, but it's all I've got right now.

I'm a fan of the Antec Three Hundred case. Here is a shot of (4) machines that I was working at a while back; the one in the desk is my dedicated daily use desktop, the far left one is the STV server, next is the machine that's currently un-used and will likely be used for the NAS, and lastly a build I was doing for someone else:

[drvnbysound]

So my tentative plan for now is to work on the NAS first. The STV server is obviously already in place and working fine right now, and the VMs aren't requirements but rather 'nice-to-have'. So, I'm planning to work on the NAS (unRAID/tRAID) system first... test and play with it... figure out its limitations, etc. and go from there.

[drvnbysound]

I've re-installed my STV system once, but at that time I left all movies and DVR'd shows in the same location. Can anyone comment on what needs to be done with regard to moving movies from one location to another? What's the best practice on doing this? I'd obviously prefer to do any necessary steps to maintain existing BMT metadata.

[drvnbysound]

Anyone?

Originally Posted by drvnbysound

I've re-installed my STV system once, but at that time I left all movies and DVR'd shows in the same location. Can anyone comment on what needs to be done with regard to moving movies from one location to another? What's the best practice on doing this? I'd obviously prefer to do any necessary steps to maintain existing BMT metadata.

[Skirge01]

Originally Posted by drvnbysound

Anyone?

I'm quite certain it's in an FAQ on the forums somewhere, but I'm too lazy to look it up at the moment.

[BobPhoenix]

Originally Posted by drvnbysound

Anyone?

This FAQ entry explains how to move recordings.

[drvnbysound]

Originally Posted by BobPhoenix

This FAQ entry explains how to move recordings.

Thanks for finding that... I was about to do some searching. What I'm not clear about is movies (e.g. DVD library) from DVD rips and their associated metadata. I just quickly browsed through that link and didn't see anything for DVDs w/ metadata, but I need to read through it more in-depth again; actually the mention of DVD was always an exclusion... that the method(s) wouldn't work for DVD media.

[BobPhoenix]

Originally Posted by drvnbysound

Thanks for finding that... I was about to do some searching. What I'm not clear about is movies (e.g. DVD library) from DVD rips and their associated metadata. I just quickly browsed through that link and didn't see anything for DVDs w/ metadata, but I need to read through it more in-depth again; actually the mention of DVD was always an exclusion... that the method(s) wouldn't work for DVD media.

If you are talking about graphics then just move them. If you are talking about descriptions then it is probably in a properties file along with the files of graphics. It should then just take moving it to the new location. You might want to check the properties files for paths and change those as well since they will probably be different. Or you can delete all the metadata and download it again when you are done moving the DVDs. The key to the move is still the same as recordings. Have the old location available so that SageTV will see that the recordings/DVDs no longer exist in their original location but DO exist in the new location.

[drvnbysound]

Originally Posted by BobPhoenix

If you are talking about graphics then just move them. If you are talking about descriptions then it is probably in a properties file along with the files of graphics. It should then just take moving it to the new location. You might want to check the properties files for paths and change those as well since they will probably be different. Or you can delete all the metadata and download it again when you are done moving the DVDs. The key to the move is still the same as recordings. Have the old location available so that SageTV will see that the recordings/DVDs no longer exist in their original location but DO exist in the new location.

Should be interesting once I do it... Yes, I was referring to both graphics (movie covers and backgrounds) as well as all of the text type data (e.g. movie description, actors, etc) that BMT collects. One of the things that you mentioned... having the old location available... definitely isn't planned, as that's my main purpose for moving them... is that I don't plan to have those locations available anymore. I was hoping that I could do a seamless move, but I doubt that's going to happen; worse case I have to update them again... not the end of the world, but definitely a considerable amount of time.

[reggie14]

Moving recordings and imported videos is easy. But, OP also asked about making sure BMT metadata (and presumably fanart) are transferred.

I think you just have to copy over your fanart folder. When I rebuilt my server, I copied/moved/configured recordings and imported videos, I installed Sage, copied fanart folder, copied wiz.bin, then installed BMT and other plugins.

It seemed like everything worked fine after that.

[BobPhoenix]

Originally Posted by drvnbysound

Should be interesting once I do it... Yes, I was referring to both graphics (movie covers and backgrounds) as well as all of the text type data (e.g. movie description, actors, etc) that BMT collects. One of the things that you mentioned... having the old location available... definitely isn't planned, as that's my main purpose for moving them... is that I don't plan to have those locations available anymore. I was hoping that I could do a seamless move, but I doubt that's going to happen; worse case I have to update them again... not the end of the world, but definitely a considerable amount of time.

You can try the advanced video recovery option then for your recordings. That should delete the old entries out of the database. My problem was if I didn't have both recording directories available (I use UNC paths) I would get two entries for each recording. One pointing to the old location and one to the new location. So I would then have to use the linked method to get rid of them. As long as both locations were available then I didn't have to follow the link directions.

Originally Posted by reggie14

Moving recordings and imported videos is easy. But, OP also asked about making sure BMT metadata (and presumably fanart) are transferred.

I think you just have to copy over your fanart folder. When I rebuilt my server, I copied/moved/configured recordings and imported videos, I installed Sage, copied fanart folder, copied wiz.bin, then installed BMT and other plugins.

It seemed like everything worked fine after that.

This sounds better than what I was suggesting for DVDs using BMT. Haven't used BMT in so long I forgot about the fanart folder. So ignore my directions completely for your DVDs and use reggie14s directions. I'm sure you will have better luck.

[drvnbysound]

So I've finally picked a route and begun to acquire equipment. I found that the MOBO that I was wanting to use for unRAID was PCI-E x8 (v1.1) and was not going to support the speeds of the HDD controller I was looking at. Long story short, I've decided to re-build my daily use PC, re-purpose my current PC as the STV server, and re-purpose my current STV server as the NAS.

I'm not going to run ESXi or bother to try and virtualize STV. Instead, I'm probably going to install VirtualBox, and run it along side STV on the Windows box. This will still allow me to run VMs as needed without requiring me to have yet another PC.

I mainly decided to go this route, because I wasn't going to save much money buying new hardware for the NAS, and anything modern I bought was going to far exceed system requirements for the NAS anyway. Not to mention it's been about 4 years since I built my daily use machine, so it was about time for an upgrade.

[BobPhoenix]

That will certainly be easier to setup then going down the virtualization route with ESXi.

I'm starting to setup one of my unRAID servers with unRAID 6B4 so that the virtualization is directly on the unRAID server. unRAID is not a VM it is the host. I will be setting up TWO VMs one that will be running all the time will be WHS2011. The other VM will be WHSv1 so that I can keep all of my current backups available since WHS2011 isn't an upgrade for WHSv1. That VM will be off most of the time and only started if I ever have to restore something.