Server Lockups with 3 or more HD300s connected

[sdsean]

Hello:

So recently I added a 3rd HD300 to my setup for our guest bedroom, which is now getting used a lot more.

However since then I keep experiencing circle's of death / server lock ups, so bad that the service itself won't shut down and I have to actually reboot the whole machine.

I'm running Sage v7 still, on windows 7 ultimate (64bit), Java VM heap size set to 1020mb.

when it happens I can see that the memory usage for the SageTVService has gone well above 1100mb.

I only run a few plugins, play on, and imdb search / metadata tools, no comskip or crazy ui mods or anything.

I suspect I'm running into the old problem of the server simply not being able to support more than 2 extenders at a time on Windows. . .any thing I can do? I'd rather not switch over to linux (although I may have to I guess).

Thoughts?

Sean

[Fuzzy]

Does sounds like the heap is too small. The biggest thing that takes up space is fanart. Turning it off might seriously smooth it out until you can sort out what you're wanting to do long term. Honestly, if I'm building a system from scratch right now, I'm going to use Linux for it - and I'm really not familiar with Linux at all - it's just that 64-bit jvm is such a huge benefit in my eyes, especially for a fanart heavy setup.

[stuckless]

Originally Posted by Fuzzy

Does sounds like the heap is too small. The biggest thing that takes up space is fanart. Turning it off might seriously smooth it out until you can sort out what you're wanting to do long term. Honestly, if I'm building a system from scratch right now, I'm going to use Linux for it - and I'm really not familiar with Linux at all - it's just that 64-bit jvm is such a huge benefit in my eyes, especially for a fanart heavy setup.

Sort of off topic, but... I wonder windows 10 Bash shell (Ubuntu) integration if you can run SageTV 64bit under windows, and get the 64 large heap benefits.

[EnterNoEscape]

Originally Posted by stuckless

Sort of off topic, but... I wonder windows 10 Bash shell (Ubuntu) integration if you can run SageTV 64bit under windows, and get the 64 large heap benefits.

I can speculate that you probably won't suddenly have Windows capture devices showing up as v4l2 devices, but it would be very interesting if at least network based devices would work.

[sdsean]

Yeah I mean I could switch to linux, but then it ends up being a dedicated piece of hardware, whereas right now my server i also use a workstation of sorts for media processing and what not. Plus playon, and other things. . .

Curious though I wonder if I could run a linux VM on windows and things would work well. . . (i seriously doubt the drivers for hardware would all work correctly. . but maybe).

It's too bad you can't have a master/slave server-setup. . . then you could just have multiple instances of the server running to solve that problem regardless of the JVM, and just have the master act as bridge/message engine. . . hmmm.

[sdsean]

I'll definitely try shutting off the fan art stuff.

Right now with just 1 extender turned on. . .(and someone watching tv), the memory usage of the server is ~524MB.

[stuckless]

Originally Posted by sdsean

I'll definitely try shutting off the fan art stuff.

Right now with just 1 extender turned on. . .(and someone watching tv), the memory usage of the server is ~524MB.

It's hard to just look at the mem usage. Java will just keep eating memory as it needs it, but eventually the garbage collector will kick in. So if you set a max jvm size of 1.2 gigs java will eventually reach that, but it doesn't mean it's out of memory... just means that now it will start having to run the garbage collector more often. Really until you get an actual OutOfMemory error in the JVM... you haven't actually run out memory, even if it's consuming the full amount.

[sdsean]

well yeah of course. . . and it may not be a direct memory usage thing either, but it's pretty clearly IO related.. . the behavior is super consistent.

After 24-36 hours: Mem usage grows above what should be the max
When it's close to this I can notice on my extenders that the frame rate slows down slightly even though I know for sure it's not a bandwidth problem and it's all hardwired.

From there someone goes and tries to change a channel or start recording something new.

At that point one of the HD-PVRs kicks on just fine, and it all looks like the recording is happening, and sure enough I can even see data getting written to the given TS file.

But then the Circle of Death appears. . .

At that point, like i said, I can't shut down the service, it just totally fails to respond.
And if I say, reset the given HD-PVR at that point, all the sudden my whole system slows to a crawl so much that even taskmanager or procexp cannot paint to show me what's happening.

Either way I'm basically forced to reboot the whole PC.

And this only seems to happen if I have more than 2 extenders running. Never had this problem with just 2.

I'll enable debug-logging and see what else I can see.

[BobPhoenix]

Originally Posted by sdsean

well yeah of course. . . and it may not be a direct memory usage thing either, but it's pretty clearly IO related.. . the behavior is super consistent.

After 24-36 hours: Mem usage grows above what should be the max
When it's close to this I can notice on my extenders that the frame rate slows down slightly even though I know for sure it's not a bandwidth problem and it's all hardwired.

From there someone goes and tries to change a channel or start recording something new.

At that point one of the HD-PVRs kicks on just fine, and it all looks like the recording is happening, and sure enough I can even see data getting written to the given TS file.

But then the Circle of Death appears. . .

At that point, like i said, I can't shut down the service, it just totally fails to respond.
And if I say, reset the given HD-PVR at that point, all the sudden my whole system slows to a crawl so much that even taskmanager or procexp can paint to show me what's happening.

Either way I'm basically forced to reboot the whole PC.

And this only seems to happen if I have more than 2 extenders running. Never had this problem with just 2.

I'll enable debug-logging and see what else I can see.

I was having this after upgrading to V9. The only solution that worked for me was to blow away the wiz.bin and start over. I didn't notice a problem before I upgraded but it could have been there and just noticeable after the upgrade.

I'm working on setting up a linux vm along side my Windows VM on unRAID for the linux version of SageTV. I have to have both because my AVerMedia Duets don't have linux drivers so I'm going to use the current Windows SageTV install with the windows drivers to provide them as network encoders. HDHomeRuns already work in linux for me on a test I ran several months ago. Just have to upgrade the linux kernel so the HDPVRs and HVR-2250 are recognized natively. Hope to get time next month.

[sdsean]

Well Bob if you get that working let me know. .. if I could setup a beefy enough linux vm on Windows and get the drivers all to bridge right (serial tuning and the hd-pvrs), then that would be ideal for me so that I can still stay on a windows machine. Plus a vm would be easier to deal with in terms of reboots and what not.

unraid looks more to me like a partitioned boot though. . . am I reading the docs wrong?
I was thinking more like running VMware with a linux ubuntu or redhat image.

[BobPhoenix]

Originally Posted by sdsean

Well Bob if you get that working let me know. .. if I could setup a beefy enough linux vm on Windows and get the drivers all to bridge right (serial tuning and the hd-pvrs), then that would be ideal for me so that I can still stay on a windows machine. Plus a vm would be easier to deal with in terms of reboots and what not.

unraid looks more to me like a partitioned boot though. . . am I reading the docs wrong?
I was thinking more like running VMware with a linux ubuntu or redhat image.

I've got ESXi on another server with an unRAID and Windows VM running SageTV on it. I had 4 SageTV servers before my job got down sized last November - I now have 3 servers. Now that I have a new job I'm re-configuring my network to prolong the life of my HDDs as long as I can.

unRAID is a NAS primarily but they have added VMs and dockers to v6. The problem with running SageTV on a docker is the drivers have to be installed on the boot OS (unRAID) to work natively with the SageTV docker. Now there is a plugin that will compile new linux kernels for unRAID to have some of the media tuner drivers but I decided that I want it to be just a VM platform since I have to have at least one Windows VM for my AVerMedia Duets. So I'm going the linux VM route on unRAID along side the Windows VM. I may even have a second unRAID installed on the same box as a VM and use the BOOT unRAID as a virtualization platform only. I can't use my ESXi 5.0 on my new MB because it is a dual CPU MB and the free version of ESXi 5.0 will only work with a SINGLE physical CPU (I believe even the more current free versions are also limited this way).

My new MB and dual CPUs has dual 8 core CPUs with hyperthreading (32 theads) and 128GB memory. My other unRAID servers are going to be shut down completely most of the time to save HDDs as long as possible rather then running the servers and just spinning down the drives but leaving power to the drives in standby mode. I got the CPUs and memory off eBay for greatly reduced prices.

[Fuzzy]

Originally Posted by sdsean

Yeah I mean I could switch to linux, but then it ends up being a dedicated piece of hardware, whereas right now my server i also use a workstation of sorts for media processing and what not. Plus playon, and other things. . .

Curious though I wonder if I could run a linux VM on windows and things would work well. . . (i seriously doubt the drivers for hardware would all work correctly. . but maybe).

It's too bad you can't have a master/slave server-setup. . . then you could just have multiple instances of the server running to solve that problem regardless of the JVM, and just have the master act as bridge/message engine. . . hmmm.

Going off your Linux VM on a windows host idea. It is possible to do a sort of master slave if you like. On windows, you run SageTV as a network encoder, so it has access to the tuners (this is if you have local tuners - if everything is a network encoder already, than it isn't needed). In the VM, you have Linux installed, running the 64-bit server, and THIS is your sagetv server that clients/extenders connect to.

[valnar]

Check this out too, what happened to me.
http://forums.sagetv.com/forums/showthread.php?t=62873