SageTV on Ubuntu in Vmware server 2.0 using a HDHR

[SuiDog]

**WARNING**
THIS HOW TO IS NOT COMPLETE!

I am still working on it. I have to still finish the sagetv config section and also fill in some gaps. If you have comments/corrections/suggestions please let me know.

HowTo

Ok. Here is my howto on getting SageTV setup on Ubuntu 8.10 server in VMware 2.0 server running on Ubuntu 8.10

** Note This HowTo will also work if you are using windows as the host. After installing VMware server on your Windows server, just skip to the creating the VM section and go from there.**

Installing VMware 2.0 on Ubuntu 8.10:

I used the following very good howto:

http://www.howtoforge.com/how-to-ins...on-ubuntu-8.10

I deviated from the script twice. First when it asked for a username for the web interface I used my account rather than leave it as root and second, if you are running the latest kernel (which you should be) the vsock module will not compile (which means nat only in your vm which is no good). When you get to the section of the install script where it asks to run the "vmware-config.pl" script say no. Then do the following

Download the following from the Ubuntu forums (you might need an account)

http://ubuntuforums.org/attachment.p...7&d=1227872015 to your Desktop

Then open a terminal and do the following:

cd $HOME/Desktop
sudo patch /usr/bin/vmware-config.pl $HOME/Desktop/vmware-config.pl.patch
sudo /usr/bin/vmware-config.pl

The just accept the defaults .. make sure you tell it you want to use bridged networking a specify you network interface that you want it to attach to. You can have it use NAT or Local only for other things so I'd still install them but you won't be using it for the Sagetv VM.

Finish the Howto and should should be able to create a new VM.

Creating the VM

First things first. If you don't have a copy of ubuntu server 8.10 iso, download it and place a copy in the directory where you told it to store your vm's (/var/lib/vmware/Virtual Machines) is the default.

Then via the console create a new VM:


Name it:


Guest OS (32bit)


Memory and Processor. (I used two processors because one couldn't cut it with the amount of extenders I have.) I seem to do just fine with 512mb but I don't run anything else on it and I don't use any custom themes. You can always bump the memory up if needed at a later point.


Hard Drive: Now this one is up to you. You can either create a really big HD and store all your video inside the VM or do like me and just create a NFS share back to the host and use the disk attached to the host instead. I use the NFS method (you could also use samba or something else)



Network adapter: You need to create one and tell it to use the bridged adapter other wise your clients won't be able to get to it easily



CD: use the ISO image at first so you can install Ubuntu



The rest of the setting should be default. It's up to you if you want to use USB but I don't know if you will be able to use a USB adapter for the tuner



Installing the OS in the VM

Once finished. Open a console and power on. You should see something like this:


The install is straight forward for installing Ubuntu. I recommend you at least install the openssh server so you can putty to it for the rest of the config. I find it easier to use putty then configure it through the console of vm (I like to cut and paste which you can't do with the console).

When finished disconnect the CD ISO image from the vm before rebooting
http://wheet.homeip.net/disconnect.jpg

Once rebooted you should have a prompt for login


From this point on I recommend using putty to connect to the box so you can just cut and paste the commands.

So once connected we need to do some things to make sure the vm runs good.

VMware Tools:
From the webpage select install tools


Then in the remote session do the following

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install linux-headers-`uname -r` build-essential
sudo mount /media/cdrom0
cd $HOME/Desktop
cp /media/cdrom0/VMwareTools<tab>
tar -zxvf VMwareTools<tab>
cd $HOME/Desktop/vmware-tools-distrib
sudo ./vmware-install.pl

Once again you will have problems with the vsock so when prompted don't run the "vmware-config.pl" say no.

Get the following and save it ON THE VM in your home directory.

http://ubuntuforums.org/attachment.p...4&d=1228735141

Once again since the ubuntu forums require a log in you will need to download it .. then upload it to your VM (another reason for ssh). You can use any sftp/scp client to do this or something like Winscp if you are using a windows client. You should be able to dump it in your home directory (/home/<username>). When done do the following in the remote connection

cd $HOME/
sudo patch /usr/bin/vmware-config-tools.pl $HOME/vmware-config-tools.pl.patch
sudo /usr/bin/vmware-config-tools.pl

You should be able to just select the defaults for everything. NOTE: HGFS is another way that could be used for connecting to storage on the host instead of NFS/Samba. I didn't use it so I won't cover configuring it. If someone else write the instructions I will update my HowTo with them.

Once the tools are installed I would recommend rebooting and making sure everthing is good to go.. you should see something like this on the console:


Not sure what the two error messages are but it didn't seem to affect the operation. If someone else figures out how to fix it I'll update my HowTo.

Java and Quicktime install

SageTV requires java. For this HowTo I will be using SageTV 6.5.12(beta) which requires 1.6.x. I wouldn't normally say get the beta but in my case the beta has specific fixes for the HDHR.

I also like to install the quicktime libs so I can convert my stuff to Qicktime and watch Quicktime formatted movies.

So on the remote console do the following.

sudo apt-get sun-java6-jre libquicktime-dev libquicktime1 quicktime-utils

You will have to accepted the agreement for java but it should all install correctly.

Static IP

Setting up a static IP is pretty straight forward.

Edit the interfaces

sudo nano /etc/network/interfaces

The syntax you will use is:

# The primary network interface
auto eth0
iface eth0 inet static
address 192.168.1.(IP you want to assign the VM)
network 192.168.1.0
netmask 255.255.255.0
broadcast 192.168.1.255
gateway 192.168.1.1

You may need to change the above to match your internal network.

Restart networking

/etc/init.d/networking restart

SageTV install

*********
NOTE: YOU MUST HAVE THE SERIAL READY BEFORE YOU INSTALL!
*********

Now I wouldn't normally recommend using a beta but since the 6.5.12 beta has specific updated for the HDHR (which is what I plan on using) I'm going to grab it.

So from the console:

sudo mkdir -p /var/media/music
sudo mkdir -p /var/media/pictures
sudo mkdir -p /var/media/videos
cd $HOME/
wget http://download.sage.tv/linuxbeta/sagetvserver_6.5.12_i386.deb
dpkg -i sagetvserver_6.5.12_i386.deb

I would select the defaults for everything here (you can specify different media folders later via the client/placeshifter/MVP...) and when prompted enter the serial number for copy of SageTV.

After the install I like to reboot the server and make sure everything is running. To verify that SageTV is running do the following on the remote console after you log in.

ps aux|grep sage

It should return something like this:

root      4791  1.4  9.7 436612 50288 ?        Sl   14:07   1:09 java 
-Dsage.paths.tools=/opt/sagetv/server -Djava.awt.headless=true 
-DUSERKEY=<YOUR KEY HERE> -Xmx256m 
-XX:+UseAdaptiveSizePolicy -XX:MaxGCPauseMillis=25 -XX:GCTimeRatio=24 
-cp Sage.jar:.:/:JARs/commons-jxpath-1.1.jar:JARs/commons-logging-
api.jar:JARs/commons-logging.jar:JARs/jcifs-1.1.6.jar:JARs/mx4j-impl.jar:JARs
/mx4j-jmx.jar:JARs/mx4j-remote.jar:JARs/mx4j-tools.jar:JARs/sbbi-
jmx-1.0.jar:JARs/sbbi-upnplib-1.0.3.jar SecuredLoader 0 0 x sagetv 
Sage.properties

If it does you are running. You should be able to connect to your SageTV with a client/placeshifter/MVP/.. and do the initial config.

SageTV Configuration

I will finish this out later.
Suidog

[SuiDog]

Anyone have any comments/corrections/suggestions yet?

TODO:
Fix grammar and typos
Add instructions on how to start VM on reboot automatically
Add instructions on configuring HDHR
Add instructions on configuring SageTV with HDHR
Possibly expand Debian OS install section.

[Zervun]

I have to say excellent readme - So far going through your steps I have not found anything wrong.

I know sometimes lack of replies in this forum can be a downer but I don't think there are alot of us, and when it just works, it keeps working.

I on the other had have reinstalled so many times it's not even funny

[SuiDog]

It's been a while since I looked at this thread. Is there anyone that would like me to update this or finish it out? Didn't seem like there was a lot of response to it. If people find it useful then I'll maintain it, otherwise I won't waste time. I only documented it because I was setting it up and thought people might like to try it.

I don't use this method anymore because I needed firewire support to control my comcast box. Other than that it seemed to work pretty good in a VM.

Sui

[drewg]

Originally Posted by SuiDog

It's been a while since I looked at this thread. Is there anyone that would like me to update this or finish it out?

Where did you put your media, and was there a measurable overhead to accessing it from the VM? Eg, did you give it access to a raw disk, use an NFS mount, a VMware shared folder, etc?

Did you run into issues giving the VM only 512MB? I set my java heap size that big alone..

Thanks,

Drew

[loonsailor]

Thanks for posting this.

What are the advantages of doing it this way? Is it just for ease of moving to a new system / load balancing? I guess it would also make it easy to do snapshots for backup. Is that it, or am I missing something?

[SuiDog]

Originally Posted by drewg

Where did you put your media, and was there a measurable overhead to accessing it from the VM? Eg, did you give it access to a raw disk, use an NFS mount, a VMware shared folder, etc?

Did you run into issues giving the VM only 512MB? I set my java heap size that big alone..

Thanks,

Drew

At the time I had all my files on the host. I created an NFS share to the host. I only had on dual tuner HDhomeRun at the time and it worked great. I was able to record two SD shows and watch a third and it worked pretty good. I don't think the memory effected me too much.

[SuiDog]

[QUOTE=loonsailor;390732]Thanks for posting this.

What are the advantages of doing it this way? Is it just for ease of moving to a new system / load balancing? I guess it would also make it easy to do snapshots for backup. Is that it, or am I missing something?[/QUOTE

At the time I was using my Linux box as my Wireless Access Point but was having problems with corruption of my DB. It was because the MAC on the bridged wireless changed every time on boot and Sage didn't like that. So the solution was to put it in a VM. When I wrote this, ESXi (free) was out and a lot of people were looking at it so the could better utilize the hardware they had. I was also thinking about moving to ESXi on my existing box. I think I might still go that way when I upgrade my hardware. So this was just a way for people to check it out and show that you could have a full system that could be built in a VM and didn't rely on any hardware cards for TV capture.

The advantages.. Well the same as any box running in a VM .. plus things like snapshots for upgrades.. cloning.. all that good stuff.. and migrating to new hardware would be a snap. At the time it was kinda like .. I wonder if I could do it .. type of thing.

[jimwhite]

Originally Posted by SuiDog

It's been a while since I looked at this thread. Is there anyone that would like me to update this or finish it out? Didn't seem like there was a lot of response to it. If people find it useful then I'll maintain it, otherwise I won't waste time. I only documented it because I was setting it up and thought people might like to try it.

I don't use this method anymore because I needed firewire support to control my comcast box. Other than that it seemed to work pretty good in a VM.

Sui

I, for one, would like to see it completed!!

[SuiDog]

ok. I'll work on it next week and update it for the lastest version of sagetv

Sui

[Zervun]

This last incantation of my server I did esxi on 680i with WHS on one VM, and ubuntu 64 on another VM.

I'm running a dell perc 6i 4x seagate 1.5TB 7200 rpm drives with XFS raid5

I found performance to be pretty poor at times and some wierd bugs.

I am running 4 feeds of 2x HDhomeruns with all pumping in at once some times.

Right now I'm reformating back to just ubuntu raid10.

I just feel that there are some performance issues with running sage in a VM right now.

[mikejaner]

Originally Posted by Zervun

This last incantation of my server I did esxi on 680i with WHS on one VM, and ubuntu 64 on another VM.

I'm running a dell perc 6i 4x seagate 1.5TB 7200 rpm drives with XFS raid5

I found performance to be pretty poor at times and some wierd bugs.

I am running 4 feeds of 2x HDhomeruns with all pumping in at once some times.

Right now I'm reformating back to just ubuntu raid10.

I just feel that there are some performance issues with running sage in a VM right now.

I am currently running some tests in this exact fashion. Not conclusive, yet but trying to eliminate possible bottlenecks. I have two phenom X4 esxi servers attatched to an ubuntu based iscsi target hosting a Raid Array. SageTV lives on a VM on the iscsi, and is recording to another Ubuntu based server hosting an XFS SMB share.

[DigitalMan]

Originally Posted by Zervun

I just feel that there are some performance issues with running sage in a VM right now.

Perhaps there are some configuration issues with your setup.

I have sage running under ESXi 4 w/3 virtual cpu's and 2gb of ram assigned (on a win 7 guest, I would expect ubuntu to perform even better). 2 HDHR's are connected and it supports a pc client and an HD100. When I'm recording 4 shows and watching 2 others guest cpu usage is less than 20%. If its converting video at the same time it stays less than 50%. On top of this I can get better than 170MB/s copying videos on the same volume while this is going on.

I have separate raid 6 arrays for VMs and video. My drive mappings are virtual not raw physical. It would be nice to see what the performance benefit would be for that. My system is running Xeon 5520's.

[Zervun]

Possibly, although it was a fairly basic install.

Your xeon's are probably quite a bit quicker than my e6600.

I don't think I had any cpu issues, it seemed more I/O based buggyness.

Are you recording all HD? or are some of your tuners doing non-hd content.

[DigitalMan]

Its possible there was at least one SD recording in that test I mentioned. Honestly though, I think there's enough bandwidth there to have more than 30 tuners going. IOPs can definitely pile up in VMs if you have too many things hitting a limited resource.

I'm using hw raid arrays, rather than sw. I have a separate array for the VMs. Pretty soon I'm going to move my capture directories to separate physical disks to leave more space on the array for my favorites. I used this type of setup on my previous server which wasn't virtual but used software raid and didn't have quite enough write performance to the array for what I wanted.

You mentioned you have a perc 6i, have you tried configuring a raid 5 volume with that rather than XFS? I have a bunch that are extended together in a VM since ESXi has a 2TB limit for volumes. Its not ideal but performance has been excellent. Rebuild takes less than 7 hours.

I don't know how you have everything arranged but in my experience it doesn't take much to overwhelm S/W raid when you're writing data. 4 HD streams isn't much at all however. Its possible SW raid (in a VM?) is introducing too much latency in this case.

[Zervun]

I'm doing all my raiding on hardware on the 6i. XFS isn't raid, XFS is a filesystem such as ext3, NTFS and so forth which is supposidly better for sage due to it being nice with bigger files.

I'm running actually running raid 10 right now (0+1 or whatever you want to call it) which is significantly faster than 5 (less space though). When I had raid 5 setup though I had it carved into 2x 1.99TB carves and another smaller one for the left over.

I might try and do vmware again this weekend. I'm having some glitchyness on recordings and I can't figure out if it's a hdhomerun issue, or maybe that I'm using 64bit.

I just need to get my headless WHS box up so I can save my recordings

I really want to get it to work, just not having much luck at the moment.

[kbyrd]

A few comments:

- Any install guide is useful, even if we don't comment, imagine how many new people search in google or bing and find this post and then decide doing Sage in Linux isn't that hard!

- Maybe it would be helpful to talk about using ESX or Workstation as well. I treat Workstation 7.0 as a "server", I run a PlayOn in a Win7 VM, and it's up all the time, with scripts to launch it at boot. Why Win7 over Server? Newer, better, faster VMware code. Even if you don't run through an ESX install, you could treat it like you treat: "this will work on Windows hosts as well, just skip to..."

- Maybe add some discussion about why you would want to do this, what advantages does the VM have for you (and what disadvantages, e.g. using PCIe tuners without a VT-d motherboard).

- At this point in time, I'm not happy with performance of shared folders/HGFS. I would be either NFS or writing directly to the virtual machine's vmdk. Streaming writes (even 4...6...8 streams at once) shouldn't produce that many IOPS I would expect ESX

- A few people here and other places have said things indicating they were using one Linux VM for Sage, and another Linux VM (or WHS) for storage. I get having a separate storage VM if that VM gets you something you can't get from the Linux Sage VM (like WHS), but it seems like arbitrarily adding latency to have a second Linux VM serving up something over NFS to Sage Linux VM.

[kbyrd]

Ok, it's been a while since I posted the above and I've just started trying out a new config. I'm using a Linux host as the fileserver. It's does RAID10 with the md driver serves it up via Samba/NFS/AFP server for the whole house. I'm more comfortable doing this in Linux. Using md ensures I can always recover my data on any random PC with a live CD.*

For Sage TV, PlayOn, BluRay processing, I'm now trying out a Win7 VM. I start it automatically at boot with this line in /etc/rc.local

vmrun -T ws start /path/to/vmx_file.vm nogui

Recording is done via a HDHR to a samba share provided by the Linux host.

It's been running for a day now and I noticed some recording glitches. Am I doing anything obviously wrong? Before this, I've been running Sage on Linux for a few years without a problem.

* If someone could convince me that using particular RAID card will let not lock me into that card's vendor in case I need to recover the data. I would be happy and use ESX.

[zoop]

why not just use the linux box as a network encoder? seems likely that recording to a network share may be causing your issue...

[kbyrd]

Originally Posted by zoop

why not just use the linux box as a network encoder? seems likely that recording to a network share may be causing your issue...

Was this a response to my post? If so, there is only one box, not two. I can't have just a network encoder.