Virtualize with ESXi 4.0

[mikejaner]

Originally Posted by stanger89

I've got a question... I see the new i5's have VT-d, but I'm having a heck of a time finding a motherboard that supports it to go with it. Does anyone know if any of the intel H55 or H57 board support VT-d? Or any i5 boards with confirmed VT-d?

-er scratch that maybe. Looks like the DQ57TM officially includes VT-d.

I second that question.
Any others? and any brands like Asus etc..?

[stanger89]

I know this is the linux forum, but seems the best thread for an ESXi question. I've had a rather radical idea that has led me to try running ESXi on my desktop. It's a i7 920 machine in a DX58SO board that supports VT-d.

I'm having some trouble getting the passthrough to work for my graphics card. I believe I've read that ESXi should just give up the graphics card to the passthrough when the VM asks for it, but that doesn't seem to be happening and the video card (while it shows up and drivers are loaded) shows as Stopped (code 43) in Windows Device Manager.

Anything obvious I might have set wrong, or forgot to do to get it working?

[DigitalMan]

Interesting. I haven't tried a video card, perhaps there's a conflict with ESXi using/needing it. Maybe you could try 2 video cards in the system and passthrough the second one.

Originally Posted by stanger89

I know this is the linux forum, but seems the best thread for an ESXi question. I've had a rather radical idea that has led me to try running ESXi on my desktop. It's a i7 920 machine in a DX58SO board that supports VT-d.

I'm having some trouble getting the passthrough to work for my graphics card. I believe I've read that ESXi should just give up the graphics card to the passthrough when the VM asks for it, but that doesn't seem to be happening and the video card (while it shows up and drivers are loaded) shows as Stopped (code 43) in Windows Device Manager.

Anything obvious I might have set wrong, or forgot to do to get it working?

[drewg]

Originally Posted by stanger89

I'm having some trouble getting the passthrough to work for my graphics card. I believe I've read that ESXi should just give up the graphics card to the passthrough when the VM asks for it, but that doesn't seem to be happening and the video card (while it shows up and drivers are loaded) shows as Stopped (code 43) in Windows Device Manager.

What kind of graphics card is it? Nvidia?

ESX messes with the page tables as well. This prevents a guest from using PAT to change cache attributes on PCIe memory spaces (like marking the Graphic's card's memory write-combine). Setting up a PAT mapping is something that is highly unlikely to ever fail on real hardware, and might be your problem. I've seen this problem with Nvidia drivers on linux when our NIC driver loaded first and chose a different PAT scheme than Nvidia's drivers were expecting. The Nvidia PAT setup failed, and the driver failed to load.

Drew

[stanger89]

Yeah, it's a 9600GT.

[drewg]

Originally Posted by stanger89

Yeah, it's a 9600GT.

Interesting problem... I wonder what would happen if you tried an ATI card..?

Drew

[brainbone]

Originally Posted by stanger89

I believe I've read that ESXi should just give up the graphics card to the passthrough when the VM asks for it

I haven't found anything that specifically says ESXi supports GPU passthough, but I can find threads, albeit under the ESXi 3.5 section, saying that it is NOT supported.

For example:
http://communities.vmware.com/thread...35E4?tstart=30

[stanger89]

Yeah, it's weird, it all looks like it would work. ESXi allows me to select it for passthrough and add it to the VM. Windows picks it up and installs the drivers for it. It just can't use it. My first reaction was "well ESXi is using it for the console output so it's probably taken". But then I throught I read that ESXi would "give up" the card when a VM asked for it. Maybe that was in error, guess I need to see if I've got a spare card lying around....

[brainbone]

Looking at this list:
http://www.vm-help.com/esx40i/esx40_...itebox_HCL.php

Someone has reported VMDirectPath success with a "Sapphire Radeon 4550 512MB/PCI-Ex16", so appears possible.

Out of curiosity, do you have VMware Tools installed on the Windows guest?

Also, what version Windows? Following the link for the Sapphire Radeon 4550 512MB/PCI-Ex16 in the HCL, it appears it was working under a Windows 7 guest. The post reads:

The graphics card actually works very well. Under windows7 you can set the extra adapter as your primary display. Somewhere during the startup process the output then switches from the VMware "display" to the VMDirectPath video card attached display.

Only problem I have is that if the virtual PC goes into Suspend mode I cannot get it back online with the VMDirectath mapped mouse/keyboard.

The graphics card I'm using is a Sapphire Radeon 4550 512MB/PCI-Ex16.

Edit: If you still can't get it working, Xen may be an option, having official support for VGA passthrough, though it looks like some nvidia cards can be troublesome.

[stanger89]

Cool, at least I know I'm not crazy

I've found a reference to making sure you use a "supported" nVidia card, and the "new" drivers, but I've thus far been unable to find what either of those are...
http://professionalvmware.com/2009/0...tions-and-you/

[brainbone]

I'm not sure if it holds true with ESXi, but from Xen's list of tested graphics cards:
"Nvidia officially supports only Quadro FX 3800, 4800 and 5800 for graphics passthrough usage (they've tested and verified their binary drivers for these graphics cards in combination with graphics passthrough)."

[stanger89]

l found on the xen messageboards a reference that for some (older, perhaps like mine) the virtual memory addresess need to match exactly the physical ones. Haven't been able to find a way to do that in ESXi yet....

[PosterBoy]

On my system, the video chip is on the same PCI bus as my firewire card. Because of that, the video chip is reserved for VMDirectPath. Although ESXi gives up the video board when my guest os boots and it takes over the firewire card, it does not see the video chip. I tried allocating the video chip to the guest OS and the guest OS hung on startup, although ESXi was fine. The video chip is some oddball ASPEED AST2000. I am guessing ESXi doesnt do well giving guest OSes access to video chips. They only claim support for a limited set of network and storage cards, but clearly it works for my firewire board and USB interface.

Hope that helps

Kent

[brainbone]

So, now I'm thinking of moving my SageTV server to a VM...

Stanger, is your plan to eventually move SageTV to ESXi?

[stanger89]

It's just a possibility at this point.

As far as my latest line of inquiry, I was investigating if I could turn my i7 machine into an ESXi box, and actually run my desktop as a VM, along side unRAID and Sage, but hit a roadblock with the current itteration of ESXi's passthrough and my nVidia video card.

[brainbone]

I'd recommend a new graphics card, but without a list of supported GPUs (official or otherwise), you could end up with a pile of them before finding one that work well enough.

For my use, I'm thinking that the 2 VMDirectPath devices per VM limit is going to be a headache when trying to get SageTV under ESXi. To get USB devices to work, you still need to pass-through a USB port -- so USB tuners don't help here.

I wonder how stable the unstable branch of Xen is ... ...

[stanger89]

Originally Posted by brainbone

I'd recommend a new graphics card, but without a list of supported GPUs (official or otherwise), you could end up with a pile of them before finding one that work well enough.

Yeah, that's my concern. Supposedly the new Quadros support it, but thosre are $$$. There was the supposed ATI card that worked that I could try, but I really don't do much on the machine so I'm not big on the idea of buying a card just to run a VM.

For my use, I'm thinking that the 2 VMDirectPath devices per VM limit is going to be a headache when trying to get SageTV under ESXi. To get USB devices to work, you still need to pass-through a USB port -- so USB tuners don't help here.

Well I really only need two, I've got an HDHR and 2 tuners (an HD PVR and an R5000. I think the simple solution to the USB think is getting a USB controller, then you can just pass through the whole PCI USB controller with 4 ports or something and be relatively set. I guess I would need 3 USB ports passed through, R5000, HD PVR and USB UIRT.

I don't know the lack of real documentation for any of this (ie some semblance of a guarantee it will work) is making it look increasingly likely that I'll just not do it and my Athlon XP 1800 server will live on.

Maybe I'll swap out my 1.5TB drives in my NAS for 2TB drives which will solve my storage issues, which are what's really driving my investigation.

I wonder how stable the unstable branch of Xen is ... ...

The problem for me is the learning curve for Xen looks way beyond my level of interest

[harrijay]

All of this talk about passing through a graphics adapter to a VM has got me thinking how well does it work. Right now I am passing through 1 PCI tuner - A180, 1 NIC and also a drive passsed through with RDM to a linux VM. I do have a radeon 4000 series card that I can test out to see what happens along with a copy of windows 7. Maybe I will give it a try this weekend and let everyone know how it works.

Jayson

[whartsell]

What kind of drive did you pass through? I cant get any (SCSI or SATA) to work

[harrijay]

I pass through a SATA drive. I followed this and it worked perfectly

Jayson