Question for vmware ESX users.

[drewg]

Originally Posted by jpappas

Despite the warning, I am going to post this anyway (as I posted about ZFS a while back). For Linux there is a ZFS fuse port, as well as BTRFS, plus Sage runs on OSX which has native ZFS. I have not tested either, but btrfs has been mainlined, so that may be "the future." Researching btrfs now, since I just noticed that I picked up the backported version in the repos...
jp

Due to the way it does caching, ZFS is a huge memory hog and I would not run it on a 32-bit kernel, since it is likely to run the kernel out of address space. All released (10.5) MacOSX kernels are 32-bit. MacOSX will not have a 64-bit kernel until 10.6, so I wouldn't trust ZFS on MacOSX until 10.6. Further, I don't think ZFS is really even supported on MacOSX in 10.5.

FWIW, I really wanted to run ZFS too. I'd considered running it under Fuse, but the idea of a user-level process handling my storage just seemed like a bad idea. I did some benchmarks between native RAID-Z ZFS on FreeBSD, RAID-Z ZFS under Fuse on Ubuntu 8.04, and XFS over a raid5 LVM. I'll try to dig them up. The Fuse performance was surprisingly not bad for bulk I/O, but terrible for things like file creates and deletes, which is unsurprising.

Some other creative options I'd considered:

- Run SageTV under Linux binary compatibility on FreeBSD/amd64. FreeBSD has had ZFS support for a few releases now and has quite good support for Linux binaries. This will only work if you have no tuner hardware which requires drivers. Eg, this will only work with a tuner like an HD Homerun.

- Run SageTV in a Linux zone under Solaris. Again, this will only work with a driverless tuner, like HD-HR.

Since I had 3 existing PCI tuners, I just decided to stick with Linux for this machine. My next machine will probably be either FreeBSD or Solaris..

Drew

[kbyrd]

The "warning" was me just having fun. I guess the reasons I want ZFS are: It's shiny and new. Oh, and the checksuming. With that and a parity chunk you can actually correct errors at read or write time. I don't want it enough to get off of Linux for everything else my NAS/Sage/etc box does though. As much as I say I want ZFS or FlexRaid or whatever, I'm really comfortable with md and I'll just trust that for a while. Maybe the ESX based solution will be for the next time!

I had the same thought about FUSE. It seems like too much to trust. Of course, I'm perfectly willing to trust VMs for that. That's not overkill at all ;-). Because of work, I guess I tend to see everything as a VM waiting to be created. Doing something like a Solaris VM that offered up a ZFS pool sover NFS or iSCSI (either to other VMs or even to the ESX host itself as storage for the other VMs virtual disks) isn't so crazy. At least a few companies offer solutions that do things like that. LeftHand networks has got this thing that lets you combine a bunch of local storage on ESX boxes a single pool of shared storage. The ESX host sees this as an NFS or iSCSI share you can then create your actual workload VMs on. I believe it lets you add new boxes with disks as needed and it provides some parity-based redundancy. The "agent" on each ESX box is a virtual machine.

[drewg]

Originally Posted by kbyrd

I want ZFS are: It's shiny and new. Oh, and the checksuming. With that and a parity chunk you can actually correct errors at read or write time.
<...>
Because of work, I guess I tend to see everything as a VM waiting to be created. Doing something like a Solaris VM that offered up a ZFS pool sover

What I like about ZFS is it combines the volume manager with the filesystem. The first FS I used which did that was AdvFS back on DEC OSF/1 (aka, Digital UNIX, aka Tru64) in the mid 90s. It gives you an incredible amount of power and flexibility. Adding a drive to an array and expanding the fs to cover it is a one- or two-step operation, not a complex operation like expanding a linux s/w raid array, lvm, and growing the fs.

Drew

[gtallan]

Originally Posted by drewg

What I like about ZFS is it combines the volume manager with the filesystem. The first FS I used which did that was AdvFS back on DEC OSF/1 (aka, Digital UNIX, aka Tru64) in the mid 90s. It gives you an incredible amount of power and flexibility. Adding a drive to an array and expanding the fs to cover it is a one- or two-step operation, not a complex operation like expanding a linux s/w raid array, lvm, and growing the fs.

Drew

Amazing the threads you find in the Sage forums. We are STILL using Tru64, mainly because of advfs, while waiting for another decent filesystem to become available (ideally ZFS becoming nice and stable on FreeBSD).

No way to run SageTV on an alphaserver though - erm, could try NT 4.0 for Alpha with FX!32 to run x86 apps? Ah, painful memories...

[killervette]

Originally Posted by chain

Hallo Rex,

how do you pass thru SATA disk as a SCSI device...

do you change the *.vmx file or what ever
I know this is not the Problem here but some info would be grate

As far as i know, ESXi will not allow a sata disk to be "passed through." I am experiencing this problem right now.

[ecsrun]

Sorry to dig this one up from the grave but...

I am using ESXi 4.0 with an Intel 5400 chipset.

Certain intel chipsets/motherboards feature directpath (VTD), mainly high-end or server grade motherboards. This technology allows you to "pass through" a PCI/PCI-Express or a PCI/PCI-Express usb card to your virtual machine. You must use ESX or ESXi 4.0 (3.5 did not have this feature).

I have linux based sagetv running in a vm on top of ESXi 4.0 and it is using several tuner cards with success (including a usb tuner via a usb pci card).

Please let me know if you need more information on this.

[kbyrd]

Originally Posted by ecsrun

Sorry to dig this one up from the grave but...

I am using ESXi 4.0 with an Intel 5400 chipset.

Certain intel chipsets/motherboards feature directpath (VTD), mainly high-end or server grade motherboards. This technology allows you to "pass through" a PCI/PCI-Express or a PCI/PCI-Express usb card to your virtual machine. You must use ESX or ESXi 4.0 (3.5 did not have this feature).

I have linux based sagetv running in a vm on top of ESXi 4.0 and it is using several tuner cards with success (including a usb tuner via a usb pci card).

Please let me know if you need more information on this.

VT-d wasn't widely deployed when this thread first started, but I can confirm this works with ESX 4.x and above. I've seen lots of problems with VT-d and BIOS on various platforms, but I believe it's all pretty shaken out by now. With VT-d, you can dedicate a thing like a PCI or PCIe to a virtual machine, and have the VM's operating system drive the card, rather than needing drivers for vmkernel.

[stanger89]

Wild curiosity, can you dedicate a graphics/PEG to a VM?

[kbyrd]

Originally Posted by stanger89

Wild curiosity, can you dedicate a graphics/PEG to a VM?

I don't think so, yet. I'll ask around.

[mikejaner]

Originally Posted by killervette

As far as i know, ESXi will not allow a sata disk to be "passed through." I am experiencing this problem right now.

I know your post is kinda old, but if you never found an answer to this, I am currently doing it with ESXi 4.
Here's how I got it working http://www.vm-help.com/forum/viewtopic.php?f=14&t=1025

[gtallan]

Originally Posted by kbyrd

VT-d wasn't widely deployed when this thread first started, but I can confirm this works with ESX 4.x and above. I've seen lots of problems with VT-d and BIOS on various platforms, but I believe it's all pretty shaken out by now. With VT-d, you can dedicate a thing like a PCI or PCIe to a virtual machine, and have the VM's operating system drive the card, rather than needing drivers for vmkernel.

That's interesting, so potentially even an old tuner card like PVR500 could be passed through to a VM under ESX 4.0? It would be interesting to try (though perhaps academic since the PVR500 will be obsolete soon enough, if/when Comcast kill analog around here...).

I'd seen it described that the device passthrough should work on any recent Intel/AMD system with support of virtualization - does that sound right or is it really Intel-only?

[mikejaner]

Originally Posted by gtallan

That's interesting, so potentially even an old tuner card like PVR500 could be passed through to a VM under ESX 4.0? It would be interesting to try (though perhaps academic since the PVR500 will be obsolete soon enough, if/when Comcast kill analog around here...).

I'd seen it described that the device passthrough should work on any recent Intel/AMD system with support of virtualization - does that sound right or is it really Intel-only?

Unfortunately no. AFIK Only Server level Intel boards, like ones based on the 5400 chipsets can do this. The reason why is the VT-d extension. I haven't seen any AMD systems which do this.

[kbyrd]

Originally Posted by mikejaner

Unfortunately no. AFIK Only Server level Intel boards, like ones based on the 5400 chipsets can do this. The reason why is the VT-d extension. I haven't seen any AMD systems which do this.

mikejaner has it right. AMD/Intel each have hardware assisted cpu virtualization, that's been out for a bit. Recently, they have each added device virtualization, it's known as an IOMMU, VT-d, AMD-Vi, and probably several other names. This stuff is usually only found on server motherboards and it's a separate feature from the CPU hardware virtualization (VT and AMD-V).