ESXI help

[Hunter69]

I am having a weird issue and am in the middle of troubleshooting the problem. First the issue I am having involves this error:

unable to connect to the MKS:error connecting to /bin/vmx process

Sagetv is unresponsive. I literally have to reboot the entire esxi server. Esxi is responsive but I have to perform a hard shutdown. The only machine having issues is my sagetv.

The difference between sagetv and the others is: I am passing through 2 hauppage colossus. I am not positive this is the issue.

I am currently investigating a possible issue and am looking for input. I have fout VM's. I do not think this issue started until I recently created a WHS vm. My first thought was this:
My server has a Xeon. It has 1 socket with 4 processors. 8 logical processors. Between the 4 machines I had 9 processors assigned. Here is a list of the machines and what is assigned relating to processor:
Unraid 1 virtual socket with 1 core
Sagetv 1 virtual socket with 4 core
WHS 2011 1 virtual socket with 2 core
Homeseer server 1 virtual socket with 2 core

I am wondering if this could be my issue.
What are your thoughts?
I record HD. Run 1 instance of comskip. 4 total tuners (2 colossus and 1 hdhomerun w 2 tuners). What have you guys assigned to your sagetv vm's?

Thanks for the help

[BobPhoenix]

I have 2 ESXi 5.0 servers both are Xeon E3-1230 (4 cores with HT - 8 logical cores) both with 16GB memory. With following VMs:

Server1:
1 IBM M1015 connected to Intel RES2SV240 sas expander.
2 AOC-SASLP-MV8
2 AVer Media Duet tuners
1 NEC based USB 2.0 PCI card
3 VMs:
1) SageTV/Win7x64 6GB 4 cores, 1 SASLP-MV8 - 6 2TB recording drives, 120 GB SSD boot drive RDM'd to IDE controller, USB 2.0 - 2 HVR-950Qs, 2 Duets, 1 HDHR - will be changing.
2) WHSv1 2GB 1 core, 1 SASLP-MV8 - 4 1TB laptop & 1 320GB laptop, MB USB 2.0 controller - 2TB USB drive for backup of server.
3) unRAID 3GB 1 core, M1015 and SAS expander - 16 2TB drives

Server2:
1 IBM M1015 connected to Intel RES2SV240 sas expander.
1 AOC-SASLP-MV8
1 Highpoint 1742 PCI controller - will be changing
1 NEC USB 2.0 PCIe controller - will be changing
1 AVer Media Duet
1 HVR-2250
2 VMs:
1) SageTV/Win7x64 6GB 4 cores, Highpoint 1742 - 5 3TB drives in external port multipler enclosure, 120 GB SSD boot drive RDM'd to IDE controller, NEC USB 2.0 - 2 HD-PVRs, Duet, 2250, 1 HDHR
2) unRAID 3GB 1 core, M1015 and SAS expander - 15 2TB & 3 3TB drives.

Server 1 has no problems at all. Server 2 has a weird startup problem on the SageTV instance. Quite often I have to start the VM twice because the first time it is executing VEEERY slowly and a power off and then power up fixes the issue. Could be a bad SSD as I have a mini-PC client that freezes with the HDD light active also just haven't replaced the SSD's yet to see if that makes a difference. It could also be MB problem as I had to RMA the first one back to Retailer because of some missing epoxy coating that exposed the MB traces - could have been scratched but didn't really look like a scratch. Anyway one I boot the VM a second time it works without problems. Also unRAID never has a problem. I will have to check the ESXi logs on server 2 to see if there are any errors.

[Hunter69]

Thanks for the information. I think I have found (with emphasis on think) what my problem is. I went ahead and change around the setup. I changed from a RDM type drive setup to a vdmk on a new datastore. I reinstalled. Then I had the same issue. I lost my new datastore. The new datastore is the same disk that was rdm'd as the boot drive for the sage server. So I either have a bad drive or bad sata cable. The drive is a new wd raptor.

What stinks is that I was trouble shooting this, lost all my recordings . It was my stupid fault. Boy am I kicking myself right now............

[Hunter69]

So last night I changed drives. I have a WD red 1tb (it was my recording drive). I went ahead and rdm'd the WD red drive as the boot drive. I have another 1tb drive (form my WHS server, that changed since I virtualized it and moved to whs 2011). I rdm'd that drive as the recording. I am getting really good at reloading/recreating my sage servers since virtualizing it. I guess time will tell.

[Hunter69]

Well I experienced the same issue once again. Last time I changed out the hd and the sata cable. I experienced the same crash. Now I am switching the hd to a different sata port to see if it crashes again.

[doc]

What version of ESXi are you using?

There are some passthrough issues with 5.1, it had me scratching my head for a while. If you're using that it might be worth downgrading to 5.0 until they've resolved them.

[Protoman]

I use ESXi pretty extensively at my work. One thing we have found to cause issues is if the Datastore assigned to the VM needs to have enough free space on it for the amount of RAM assigned to that VM. It can be a bit confusing, and I will try to explain better.

Lets say you have a 300 GB drive you are going to use in your ESXi environment. You add it to ESXi and say you want to use the "Maximum Amount of Space." When you create your Sage VM with lets say 4 GB RAM, when you create the Datastore on the 300 GB drive you would only have 295 GB size for the OS Drive, and essentially 5 GB "Available Space." ESXi will create a Pagefile, similar to Windows, and put it in the first drive for that VM (unless modified to go somewhere else). That pagefile size equals the amount of RAM allocated. So if you go from 4 GB to 8 GB you might experience issues.

One other thing I have learned with ESXi, and virtualization in general is that vCPU (Virtual) are very different that pCPU (Physical). Having too many vCPUs might actually hurt performance of that VM. If you have 4 vCPUs assigned and only have a Quad Core pCPU, that VM has to wait until all pCPU are free to be able to process what it needs. It is best to start with 1 vCPU and scale upward only if it is needed.


By the way, I am thinking of moving my dedicated SageTV server into a virtual environment at home as well. I want run other VMs on it, since it only gets used 5 Hours a day, most of which is recording.

I hope this information helps,
Protoman

[Skirge01]

Originally Posted by Protoman

Lets say you have a 300 GB drive you are going to use in your ESXi environment. You add it to ESXi and say you want to use the "Maximum Amount of Space." When you create your Sage VM with lets say 4 GB RAM, when you create the Datastore on the 300 GB drive you would only have 295 GB size for the OS Drive, and essentially 5 GB "Available Space." ESXi will create a Pagefile, similar to Windows, and put it in the first drive for that VM (unless modified to go somewhere else). That pagefile size equals the amount of RAM allocated. So if you go from 4 GB to 8 GB you might experience issues.

One other thing I have learned with ESXi, and virtualization in general is that vCPU (Virtual) are very different that pCPU (Physical). Having too many vCPUs might actually hurt performance of that VM. If you have 4 vCPUs assigned and only have a Quad Core pCPU, that VM has to wait until all pCPU are free to be able to process what it needs. It is best to start with 1 vCPU and scale upward only if it is needed.

Hopefully I'm not entirely threadjacking this and am actually keeping it on topic for the OP. That was an interesting read and it brought two thoughts to mind:

First, would it make sense to use multiple drives for storing your VMs to spread out the disk load or does it not really make much of a difference? I currently have mine on a single 3TB drive, but your post made me think that maybe I'd be better off using 3 1TB drives or even a single 2.5" (non-SSD) drive for each VM. Assuming it's dependent upon each VM's workload, I currently have 4 VM's: 1 running pfSense, 1 running XP (SageTV/comskip), 1 running Windows 7 (FlexRAID, uTorrent, and other "server" duties), 1 running WHS 2011 (backups only).

Second, if you were going to assign more vCPUs to any VM, would you suggest a rule of thumb is to never assign more than 50% of your pCPUs? 75%? My only VM which "needs" more than one vCPU is the SageTV one, so that I can run multiple comskip sessions simultaneously.

[Protoman]

Originally Posted by Skirge01

First, would it make sense to use multiple drives for storing your VMs to spread out the disk load or does it not really make much of a difference? I currently have mine on a single 3TB drive, but your post made me think that maybe I'd be better off using 3 1TB drives or even a single 2.5" (non-SSD) drive for each VM. Assuming it's dependent upon each VM's workload, I currently have 4 VM's: 1 running pfSense, 1 running XP (SageTV/comskip), 1 running Windows 7 (FlexRAID, uTorrent, and other "server" duties), 1 running WHS 2011 (backups only).

In my experience with VMware at my organization, we have had issues with VMs. Part of that issues was that the VM having problems (general slowness) was a high resource utilization VM. It was a SQL Server and needed lots of Disk I/O and lots of CPU I/O. The ideal usage for virtualization is to have lower resource utilization servers running in the environment.

With that said, us home users don't have access to redundant 4 GB fiber SANs and dedicated Gigabit links for servers and RAID 10 storage and etc, so we must plan out our environments a bit more. I plan on having a dedicated SSD for my ESXi install, and using the left over space for my SageTV Windows 7 OS, and for my recordings having a dedicated 1.5 TB drive. The reason is that I know I record 4 shows simultaneously, run 1 ShowAnalyzer during recording, and even watch 1 show all at the same time. This level of activity really only happens for 4 - 5 hours a day at best and even less on the weekends.

With your current VM environment you could easily combine pfSense, and WHS 2011 onto the same physical disk, since I would assume you backup when you are sleeping, and pfSense does not need a lot of diskspace nor lots of resources. For the SageTV server I would have dedicated just because of the Disk I/O. The Windows 7 VM is kind of up in the air, since I am not sure how much utilization it gets and it is something you should look into to get an idea, worst case is try it and see.

Second, if you were going to assign more vCPUs to any VM, would you suggest a rule of thumb is to never assign more than 50% of your pCPUs? 75%? My only VM which "needs" more than one vCPU is the SageTV one, so that I can run multiple comskip sessions simultaneously.

This issue recently came up for my organization, and everything that I researched and understand is only assign 1 vCPU, unless you know your server/application will use more than 1 at a time. If comskip runs like ShowAnalyzer does then I could see assigning more vCPUs, but I would try 2 vCPUs and see how it goes then maybe 4 but would be very analytical on the performance it offers at 4 vCPUs and the performance impact of the other VMs as well. The other question that I would ask is how many comskips do I really need to run at once. I personally run only 1 at a time, and am almost never subjected to seeing ads.


Hope this is helpful,
Protoman

[Skirge01]

@Protoman: Thanks for taking the time to write all that out. It was extremely helpful and gives me quite a few things to consider going forward. My plan will be to consolidate pfSense and WHS2011 onto a single drive and separate the W7 and Sage to their own, smaller disks. As far as the W7 VM goes, due to the I/O from FlexRAID (software RAID) and uTorrent, I think it probably makes sense to put that on its own disk, as well. Speaking of which, does thin provisioning cause noticeable, unnecessary disk overhead I should be worrying about?

You also said you were going to install ESXi onto an SSD. Since ESXi doesn't have trim support, does that make sense? I picked up an SSD for that, but chose not to use it when I found out it still didn't support trim. In the end, I installed it on a 4GB USB flash drive. From what I read, there's minimal activity there once the hypervisor has booted. Is that incorrect and the flash drive could also be causing a bottleneck?

[Protoman]

I am not entirely sure if thin provisioning has an disk overhead or not. We always use Thick Provisioning since we do Live Migrations of VMs.

I seem to forget about TRIM, maybe bcause I don't have an SSD yet, so I might be going back to the drawing board on getting one if ESXi does not support when I start actually upgrading. You could use that SSD as a Physical RDM drive for a VM and still get the TRIM support from my little bit of research.

There might be bit of a bottleneck, I believe ESXi does use the drive it is installed on and does some Paging on it similar to Windows. Not 100% sure on it, but wouldn't be surprised either.

Starting to make me think.
Protoman

[BobPhoenix]

Get an SSD with built in Garbage Collection. Works like Windows Trim but with any OS. The drives that have it are more expensive than those without.

[Skirge01]

I did a quick look for this feature and most of what I'm finding says that all SSD's have this, which can't possibly be correct. If that's the case, what the heck is TRIM for? I did some searching and found this from 2009. Does that still hold true? If so, then it would appear that GC is not nearly as effective as TRIM.

If I'm mistaken, please point me to a link that's more accurate. Thanks!

[BobPhoenix]

Originally Posted by Skirge01

I did a quick look for this feature and most of what I'm finding says that all SSD's have this, which can't possibly be correct. If that's the case, what the heck is TRIM for? I did some searching and found this from 2009. Does that still hold true? If so, then it would appear that GC is not nearly as effective as TRIM.

If I'm mistaken, please point me to a link that's more accurate. Thanks!

From this post on lime tech forums from Johnm. He's never steered me wrong yet.

[Skirge01]

Originally Posted by BobPhoenix

From this post on lime tech forums from Johnm. He's never steered me wrong yet.

Thanks for the info. That's quite interesting. Since the two SSD's he listed aren't available any longer, what models are currently suggested or what feature do I need to be on the lookout for?

[BobPhoenix]

Originally Posted by Skirge01

Thanks for the info. That's quite interesting. Since the two SSD's he listed aren't available any longer, what models are currently suggested or what feature do I need to be on the lookout for?

Look for SSDs with the Marvell 88SS9174 controller then read reviews. The reason I say that is that I believe some manufacturers have written there own firmware for the 88S9174 and it may not be as effective as the ones listed. The newer model 88SS9187 appears to be an upgrade to the 88SS9174 and is probably what will be in future offerings and that might be better than the older SSDs.

When doing a search it looks like OCZ Vertex 4 use that controller but a custom firmware.

[Protoman]

Looks like Plextor's "True Speed Technology" is a fancy term for garbage collection. The current models, or at least available ones are the Plextor M5S and M5 Pro. The MS5 uses the same controller as the M3 models, and the M5 Pro is using a Marvell 88SS9187 controller. Xbitlabs testing seems to take into count garbage collection and make it obvious. Here is the relevant charts to backup this info.

http://www.xbitlabs.com/articles/sto...s_6.html#sect0

Newegg has the M5s for $109, and the M5P for $129.

We should almost start a new thread, since we have seemingly hijacked this one almost completely, about ESXi usage for SageTV.


I hope that Hunter69 (OP) comes back with if the problem is resolved yet or not, and what the resolution was.

Thanks,
Protoman

[Skirge01]

Nice find on the technology info. I was looking at those very models yesterday. You just sealed the deal for me and I picked up two of the 128GB models, one for the W7 VM and one for the SageTV VM. I'll keep pfSense and WHS2011 on their current HDD for now. The Plextor's are a little overpriced at the moment (low price point was $85 back in September), but I need to get this lag resolved. I think I'll be needing another M1015 soon, too.

I was thinking about Hunter69 yesterday. Hopefully he'll find the rest of the discussion relevant and at least somewhat useful.

[Skirge01]

I built the new server, installed the additional SSD's (1 for W7/FlexRAID & 1 for XP/SageTV), and moved all the VM's over without too many issues. However, the lag is still present in precisely the same way. I keep getting the circle of doom when stopping playback of anything. I'm wondering if FlexRAID is the issue here. I'll need to figure out how best to test playback from a non-FlexRAID controlled drive.

[Skirge01]

Looks like my issue stems from a "back to basics" oversight. When I installed the new drives, I forgot to format them with 64k clusters; every one was using the Windows 7 default of 4k. I inserted another empty drive, outside of FlexRAID and noticed the "default" cluster size option and that's when it hit me. To test, I added it first with the 4k cluster size and copied a single file to that drive; the client hung when I stopped playback. Deleting the file, I then formatted using 64k clusters and the client didn't hang at all on the same file. I'm now copying all the files off each drive, reformatting and copying them back. A long process, but it'll be well worth it to rid myself of that dreaded hang.