New Server - SageTV+unRAID under ESXi 4.1

[brainbone]

Originally Posted by stanger89

I'm using the USB passthrough, my board doesn't support PCI passthrough.

I was looking into doing the same as you, combining my unraid and SageTV server. Before ESXi 4.1 came out, one option considered was was a USB over IP device, but the added cost and uncertainty of something like that was pushing me towards a new MB with VMDirectPath support. But both of these options, especially VMDirectPath, removes some of the hardware abstraction I like with VMs -- making it more difficult to migrate to new hardware in an emergency.

I was excited to see USB passthrough in ESXi 4.1, but based on what I've been reading, it seems to be limited to ~7MBs max, with most seeing around 3-4MBs. Hopefully this will be fixed in future versions.

[Fuzzy]

Originally Posted by brainbone

I was looking into doing the same as you, combining my unraid and SageTV server. Before ESXi 4.1 came out, one option considered was was a USB over IP device, but the added cost and uncertainty of something like that was pushing me towards a new MB with VMDirectPath support. But both of these options, especially VMDirectPath, removes some of the hardware abstraction I like with VMs -- making it more difficult to migrate to new hardware in an emergency.

I was excited to see USB passthrough in ESXi 4.1, but based on what I've been reading, it seems to be limited to ~7MBs max, with most seeing around 3-4MBs. Hopefully this will be fixed in future versions.

Yes, VMDirectPath does limit your VM portability.. somewhat.. but when you're talking about something like IOMMU'ing a USB controller, that's a pretty simple task to handle after a host change. It's not like there are problems with incompatable drivers and such, USB controllers are pretty standard.

[Fuzzy]

I am now looking more at going with running an UnRAID VM in my windows host, though. this will allow direct access to hardware for the things that need it (R-5000, SageTV Playback), but things that work fine over abstraction can be in the VM (UnRAID).

[brainbone]

Originally Posted by Fuzzy

Yes, VMDirectPath does limit your VM portability.. somewhat.. but when you're talking about something like IOMMU'ing a USB controller, that's a pretty simple task to handle after a host change.

Assuming the hardware you're migrating to supports IOMMU.

Originally Posted by Fuzzy

I am now looking more at going with running an UnRAID VM in my windows host.

Something to think about, but I still like the idea of having my SageTV server in a VM. I may hold out for better USB pass-through support -- if it ever comes.

[Fuzzy]

Originally Posted by brainbone

Assuming the hardware you're migrating to supports IOMMU.



Something to think about, but I still like the idea of having my SageTV server in a VM. I may hold out for better USB pass-through support -- if it ever comes.

Unfortunately, i don't see that really happening... I'm thinking they, at this point, are just banking on the fact that there really aren't that many performance intensive USB applications out there, so the current passthrough is completely adequate... and for the few that really need high performance, all you need is a IOMMU capable system...

I'm actually thinking that my next 'upgrade' of my system would be to upgrade to an 890FX system (currently have a Phenom II X2 already, so it would be a MB only upgrade)

[brainbone]

Originally Posted by Fuzzy

all you need is a IOMMU capable system...

And with that, my quest for a cheap IOMMU motherboard that is relatively compatible with ESXi 4.1 continues...

Problem is, most desktop boards only have Realtek Ethernet -- requiring you to hack in third party drivers to ESXi for support, with questionable results. That, or adding ESXi supported add-on nics, wasting PCIe slots that may come in handy for an unraid server.

[Fuzzy]

Originally Posted by brainbone

...wasting PCIe slots that may come in handy for an unraid server.

Yet another reason I wish there were more PCIe splitter products on the market.. ;-) There are quite a few boards out there with more than one x16 slot.. would be nice to be able to split those out to multiple x8's, x4's, or even a ton of x1's...

[ehfortin]

Hi,

I didn't saw this post before but I can comment a bit as I actually had this working since ESXi 4.1 is out. I have Unraid in a VM with 10 dedicated hard disk (RDM). I have a second VM running my Astaro 8 firewall. Finally, I had a third VM running XP with SageTV 7 beta on it with a USB HD-PVR.

Everything was working smoothly and I never lost a single recording doing this. However, I still had an issue: some of the recordings had problems with playback where it is starting with no audio and in some fast forward mode. Doing a pause/play or a skip forward would sometime fix the problem. Some recordings were just "loosing" the first 10-20 seconds, some other were going for a few minutes.

I have not been able to fully establish it is because of the HD-PVR over USB in ESXi but... I've put SageTV back on a small Atom 330 machine and it fixed the problem (even if Unraid is still under VMware ESXi). My guess would be that the USB virtualization may loose a few "block" of data when a recording is starting (waking up the virtualization process of the USB) which may cause that kind of problem. That's the only thing I can think about as... HD-PVR is creating the stream so SageTV/CPU is never involved in that except for saving the data to disks.

So, except for the beginning of the recording that seems "incomplete", the remaining of each recording have never showed any problem that I noticed.

I would expect it is because of this "incompleteness" at the beginning of the files that I was unable to convert those to MKV which seems to really indicate there is missing information in those files.

Outside of these 3 constant VM, I have a few other VM for various other temporary usage.

The whole box is running on a Q9400 processor with 8 GB of ram on an Gigabyte motherboard (will have to look back at the exact model number) with a Supermicro SAS PCIe board (there was only 6 SATA connectors on the mainboard so... was not enough) and a few Intel NIC (1x PCIe and 2x PCI). Realtek ports were not working well under ESXi (unsupported third-party driver but... loosing connectivity).

I was thinking at returning to Windows with VMware server running on it so that I could keep my firewall and Unraid virtualized but... I have 8 GB of RAM so my W2K3 or XP won't allow me to use it. I'm not really interested in buying W2K8 just for this. Maybe I could try with Win7x64 but I have not looked into this as I can see possible issue with HD-PVR driver. And I hate the fact that any upgrade to Windows will mean rebooting all the VMs...

Anyway, that's the story at high-level. If you have questions or comments, don't hesitate.

Have a great day.

ehfortin

[Fuzzy]

Originally Posted by ehfortin

I was thinking at returning to Windows with VMware server running on it so that I could keep my firewall and Unraid virtualized but... I have 8 GB of RAM so my W2K3 or XP won't allow me to use it. I'm not really interested in buying W2K8 just for this. Maybe I could try with Win7x64 but I have not looked into this as I can see possible issue with HD-PVR driver. And I hate the fact that any upgrade to Windows will mean rebooting all the VMs...

Shouldn't have any problem with drivers for the HD-PVR on win7 64. The only driver limitation would be the lack of firewire channel changing (if you use it). Otherwise, win7-64 is a perfectly fine platform for a sage server.

As for updates forcing a reboot, You really shouldn't need to even run windows updates all that often, and when you do, you should just be able to pause the VM's, reboot, and start hem back up.

[ehfortin]

Hi Fuzzy,

I may try that with Win7 64 then. It may allow me to remove a machine from the environment. While thinking at this, I may decide to go with Virtualbox instead of VMware Server as it is lighter and more current (VMware Server has not been updated in over a year now). I've did that before running my FW on VBox and I currently use it with physical raw disk mapped on my laptop (running Windows XP in a VM and Ubuntu as the host).

I have a non-related question after having a look at your setup. I saw that you are using your SageTV server for playback as well. I did that a few years ago but I was having issue back then with stuttering. Now that I successfully got the system to do full hardware decoding (so the CPU is only used at 0-2%), I guess I could look back at using the server as the main HTPC client (using extender in 2 other rooms).

Would you think it could work correctly or it is still a risky operation if I don't want to put numerous days doing fine-tuning? My CPU would be a simple Intel dual-core 2 Ghz (not a C2D however). As I've runned my SageTV server on an atom box for a long time, I know the server is not hard on ressource but... client/server together, I'm not sure.

Thank you.

ehfortin

[Fuzzy]

Honestly, most would be better off with an extender. I use it for more than just sage in my primary location, so no point to the extender for me. The system itself has been updated since that graphic/sig was made, but its no really any more powerful. (Phenom II X2 on an AMD 785 chipset). I've been using sage since long before extenders, so I've always used the server as the main client. No reason to change now.

[najames]

Geesh, I go on vacation and ignore this stuff for a while and just look at all the fun I've missed out on!!

I pulled the Win7 hard drive, used 4 others, installed ESXi 4.1 on my Gigabyte X58-UD4P board, but when I clicked on the tab for passthrough it said my hardware didn't meet specs, grrrr. I tried updating the BIOS and also tried a newer beta BIOS, still no joy. I installed Hyper-V for a look but also had problems (client?). At this point I shelved it and went on vacation for a week, been back since Sept 12 but haven't done anything with it.

I have either the same 780 Gigabyte board or one very similar with a PCI Intel 1000 NIC for our daily use PC. I am surprised you got as far with ESXi 4.1 as you did.

I have 3 new Samsung F4 2TB drives, so I need to at least test them out soon in case of DOA. I'm almost to the point of running another low power server with an OS, might try VirtualBox or something for unRAID or just punt and run Sage straight on WHS or Linux.

EDIT: I forgot that research shows that AMD C3 stepping CPUs seem to run at lower idle watts, not far from i3 530 & i5 650. Look for GM in the part numbers for C3 CPUs (ADX640WFGMBOX) as I recall.

[broconne]

Chiming in here..

I have been running ESXi 4.1 with a SageTV partition since June using HDHRs without issues. This month I added the HDPVR - I have it set to max and have not had any USB issues.

This is running on a core i7 860, 8GB of ram, with a RAID 1 setup for the VM drives. I don't record directly to the machine, instead I have it record on my NAS which runs OpenSolaris(RIP), and the drives are in a raidz2 config.

I did have one issue, where I was recording 4 things at once (3 on the 2 HDHRs, 1 on the HDPVR) - watching a 5th show and comskip was processing in another VM. I was getting some skipping during playback. I have since tuned down comskip, I think it was causing too much seeking, and have not run into that issue again.

[ehfortin]

Hi,

Well... a few subject to cover:

First, I've tried what I described in my previous post, ie: running Windows 7x64 with VirtualBox and Unraid in a VM. This works but not for me (Had it working with 1-2 disks without parity). The reason it is not good for me is that I have 12 HD in this machine and... rebooting Windows seems to move drives around. As Virtualbox is using raw device based on the notation \\.\PhysicalDrive0 and so on... there is not much intelligence to make sure the drive that sit at a specific location is always the same. Moving disks around may corrupt data (moving the parity disk is really not a good idea).

So, I'm back with Unraid under ESXi 4.1 and SageTV on physical.

The issue I had with SageTV/HDPVR under ESXi 4.1 may be related to the fact that Unraid is also running in a VM on the same hardware. I don't have this problem if SageTV is running on a physical machine and saving the recording to Unraid. However, I must admit I have not tested the other way around (SageTV/HDPVR with a VM with Unraid on physical). As Broconne is not having issue, maybe it would work fine for me as well. Just don't have the setup and cycle to try it this way. I could make sense that starting to use USB and starting a new recording over the net on a VM sitting on the same hardware is kind of asking a lot at the same split second (USB, NIC (2x), disks (I'm using parity under Unraid so... at least 2 disks are working then). Maybe I should try this with the same setup but with a cache drive in Unraid (SSD may help here...). Maybe I will try this after all. I can easily add a cache drive without changing anything else and my SageTV VM is always ready to go. Will keep you informed.

That's all for now. See you later.

ehfortin

[Fuzzy]

Have you tried running the UnRaid in VMWare Player instead of VirtualBox? Not sure on how the direct access is set up in VMWare, but it may be more 'predictable' than VirtualBox...

That said, I have no idea why windows would be reassigning physical addresses to your drives. They are usually based on SATA connections, so if you aren't changing anything in reboots, it shouldn't change. Also, you could always present 'virtual drives' to UnRAID, pointing to files on the windows accessed drive.. not sure if this would really impact the performance, as the extra level may be offset by some extra caching as the files are accessed through window's IO system.

[broconne]

While we are talking about virtualization here - I will mention that I had tried running SageTV under virtualbox running on the same corei7 with the host OS of ubuntu. That was not a stable configuration. Recording a single HD stream would cause my CPU to spike to around 80% - it looked like most of it was the network emulation.
I found a strong difference in performance between a level 0 hypervisor (esxi) and a level 1 hypervisor (virtualbox/vmware server).

[ehfortin]

You are right Broconne... I just figured how my drives are physically seen by Windows 7 in order to pass them to VBox (raw disk). It is working. However, performance is less then 30% of what I'm getting on ESXi 4.1 (don't forget that I have 12 SATA drives under that Unraid VM). And the best of it.... it crashed at some point while doing a parity check (intensive task on all disks at the same time).

So, I'm back, again to ESXi 4.1. Will try to do some testing with a cache drive under Unraid VM to see if it's helping to reduce the recording issue when everything is virtualized on the same hardware (SageTV/HDPVR, Unraid and the Astaro8 firewall).

It is nearly working fully virtualized.... just get the problem where some recordings seems to have some block corrupted at the beginning which do weird thing at playback.

If somebody has an idea... I'm listening (well... reading I should say).

Thank you.

ehfortin

[ehfortin]

New status. I've added a cache disk in my Unraid VM. This is not helping at all because it makes SageTV to hang. Very strangely, if I point on my user share from SageTV and try to restart the service, I'll see SageTVService in the list of process for a few seconds and then, it just disappear. I've enable the log but nothing is showing in there. I can see all the steps in the log until the process disappear with absolutely no warning or error message.

If I put back the original share (the one pointing directly to a disk under unraid), life is good and SageTVService is working as expected.

Which means... I can't test the setup with the cache disk. Right now, I'm all virtualized again but that's the state I've got corrupted blocks in recordings before so... that's what I except again.

Very weird. Next test is to keep SageTV/HDPVR under ESXi but with my shares on a physical machine, as Broconne's setup. I'll see if I can reproduce his success this way (inverse of this is working fine).

Keep you posted.

ehfortin

[brainbone]

Even with my current physical unraid and SageTV server, I have all recording go to a local disk on the SageTV server, and then archive/move to the unraid server later.

[ehfortin]

Hi,

Even with my current physical unraid and SageTV server, I have all recording go to a local disk on the SageTV server, and then archive/move to the unraid server later.

If I understand correctly, you have 2 physical servers (Unraid and SageTV), is that correct?

The advantage of such a setup is that you can do maintenance on Unraid without having to care about upcoming recordings (or actual recording) on SageTV. Is that why you are doing it?

How have you automated the recordings movement?

And how have you configured SageTV to not use your share that is on Unraid for recordings but only for playback? Is it by using the "Use only x GB" rule to make sure it won't ever be use as a recording path?

Right now, I've tried to have my virtual SageTV recording on a physical server (Windows 2003 Server) but... it break SageTV. I even tried that on my physical SageTV server and it doesn't work. So I may not be able to test a similar setup to Broconne.

With that said, I still have a SageTV client for my main HTPC as the machine is also used for physical DVD playback (rented stuff or when the network is done for SageTV or server maintenance). Now that I've successfully got it to fully decode HDPVR content in the GPU, I may put back the server on the same machine, ultimately removing one PC (the current dedicated SageTV server) from my 3 PC setup. Or I'll just let everything as it is (it is working well so...).

Thank you.

ehfortin