out of memory

[Nelbert]

It could be related to the plugins, but only occuring when something happens such as an update.

I don't know anything about the internals of BMT/Pheonix or the other plugins, but if they have alot of complex types, quickly firing requests they could be filling the young generation heap and causing the JVM to move things into the tenured space.

While increasing JVM max heap by setting -Xmx you haven't increased the young generation space to go with it. If a plugin fires and fills it up the young space will overflow into the tenured space. Try also setting -Xmn to 3 and see if that helps.

Items in the young heap are collected much more frequently with little to no interruption and are supposed to be short lived.

rtrski: if you're problem is memory heap related it sounds more like a Garbage Collection pause than an OOM. If it was OOM it would be in the sagetv_?.log files. Once the main heap starts to become full the JVM will run a full GC. This is an expensive operation for the JVM and the JVM will pause all threads until the GC completes. That's why things moving unneccesarily from young heap space to tenured/main heap is bad.

It is very easy to be a situation where memory is under such pressure that the GC actually accelerates the death of the app because it can't run quickly enough often enough to keep the heap size under control and spirals out of control.

For trouble shooting JVM memory issues snapshots aren't much help, you need a trace of what was happening upto the OOM and being able to interpret it. Using that you piece together what was running and taking more than it's expected share of resources.

We used to use jConsole from the JDK as a starting point for getting a thread/memory overview of the system and left it running on a separate machine to the Java server. Looks like visualVM is taking over in the post Oracle java world.

The other main culprit of memory problems is often PermGen as you can run out of permgen and still have plenty of heap space free (>50%). If this is a cause expect to see PermGen exceptions in the logs when things start to crash and burn.

[bastafidli]

I have the following plugins...
Phoenix Core Services
Season and Episode Number in Details
CVF - Custom Video Folder
SageTV H2 Database

in addition
Context Sensitive Main Menu
Enable/Disable Favorites
Hide Exit n Main Menu
IMDB Search
Movie Trailers
OSD Clock and Play Time
OSD Enhancements
Phoenix Fanart for Malore Menus
PlayOn for SageTV: STVI
Recordings List Show Original Air Date
Restart Sage from Setup Menu
Signal Strength on OSD
STVI Configurator
Upcoming Movies
TWiT.TV Video Podcast
BlueTwo (Addon for MBDiamond)
MBDiamond

[rtrski]

Originally Posted by bastafidli

I have the following plugins...
Phoenix Core Services
Season and Episode Number in Details
CVF - Custom Video Folder
SageTV H2 Database

in addition
Context Sensitive Main Menu
Enable/Disable Favorites
Hide Exit n Main Menu
IMDB Search
Movie Trailers
OSD Clock and Play Time
OSD Enhancements
Phoenix Fanart for Malore Menus
PlayOn for SageTV: STVI
Recordings List Show Original Air Date
Restart Sage from Setup Menu
Signal Strength on OSD
STVI Configurator
Upcoming Movies
TWiT.TV Video Podcast
BlueTwo (Addon for MBDiamond)
MBDiamond

I'm at work, so I'll get a more complete list up this weekend. But the only commonality I have with you appears to be Phoenix core services.

In addition I've got Jetty, BMT, Serenity UI, Commercial Detector UI and dependencies which Stuckless listed. I do have Comskip set to not run during typical recording times (6pm - 11pm every night), set to use the "-playnice" parameter, and limited to 2 threads.

I know the problems started after the last Phoenix / BMT updates. While that's still just correlation not causation...Phoenix is the only thing we have in common.

I'm on Win Vista (x64), with a quad core (C2D Q6600) CPU and 4 GB of RAM. I'll try to capture the actual failure message but usually it starts as a noted sluggishness of the UI through the HD300. Some time thereafter I'll see a windows message that the SageService failed. So I've been assuming it's a Java heap error but I've never really been able to see the heap size to verify. Usual heap runs about 151M or so (after a restart) and is set to 768M max. I've not been reposting but I'm up to about every other to every third day now.

[stuckless]

Originally Posted by rtrski

I'm at work, so I'll get a more complete list up this weekend. But the only commonality I have with you appears to be Phoenix core services.

In addition I've got Jetty, BMT, Serenity UI, Commercial Detector UI and dependencies which Stuckless listed. I do have Comskip set to not run during typical recording times (6pm - 11pm every night), set to use the "-playnice" parameter, and limited to 2 threads.

I know the problems started after the last Phoenix / BMT updates. While that's still just correlation not causation...Phoenix is the only thing we have in common.

I'm on Win Vista (x64), with a quad core (C2D Q6600) CPU and 4 GB of RAM. I'll try to capture the actual failure message but usually it starts as a noted sluggishness of the UI through the HD300. Some time thereafter I'll see a windows message that the SageService failed. So I've been assuming it's a Java heap error but I've never really been able to see the heap size to verify. Usual heap runs about 151M or so (after a restart) and is set to 768M max. I've not been reposting but I'm up to about every other to every third day now.

I'd look at pulling the phoenix core services (and everything that depends on it), and then run for a few days. See if things change.

[rtrski]

Originally Posted by stuckless

I'd look at pulling the phoenix core services (and everything that depends on it), and then run for a few days. See if things change.

OK, here's my complete list from the Installed Plugins - All page (showing anything Enabled or installed, ignoring only those that say "no need to download directly" or "dependency"):

BMT 4.9.7
Commercial Detector 0.8 Beta
Commercial Detector UI 1.31
Comskip Playback 2.06
jetty Web Server 2.2.0.9
MB Diamond 1.63 (Installed, but is not in use)
Metadata Tools for SageTV - Commandline 4.8 (says "Update Available" but I don't seem to have a system message for this, strange)
Nielm's Sage Utilities 1.4.1
Phoenix API, Phoenix Core Services, both 2.3.13
Phoenix Fanart for SageTV7 0.1.9
SageTV H2 Database 1.0.8
sagex-api - SageTV API Extensions 7.0.23.5 (also shows Update Available, no system message)
Serenity 1.0 (Currently my active UI theme is Serenity7)


Today I did a little DVD ripping and then used the Metadata web interface to adjust the data the way I like it (I rip TV episodes as separate folders e.g. "Eureka" ->"Season1"->"S01E01" folder with the DVD structure, using DVDFab, so the 'automatic' metadata gathering always fails which is expected. I also sometimes edit the rather cheesy summaries for movies you get from the MovieDB ("This guy did this and then the girl was like boo-yah! Then they all get eaten! This movie R0XX! --by Teh Inte4weB5_LOL"). I also hit the 'Scale' button for the FanArt images in the Sage UI.

I swapped back to the Sage interface occasionally while editing and noticed that the Java heap which had been at 150MB-ish when I started had crept to 180ish. Going back to the Home screen and then back to System messages, each time it would re-query I'd see the heap go up by another 6-10MB. Don't know if that was as it was doing image scaling or what.

BMT log didn't have anything showing (last file update was like 12-18-2010, and all that's in there is "browsing folder" messages from when I did a name search to pop up a DVD rip directly for editing a month ago). Phoenix log did have a bunch of new stuff, a lot of warnings and the like, but in a quick scan I didn't see anything that screamed problem, just a lot of file in skip list sort of things. After a few minutes and re-pinging the system info page (heap up to 286MB or so) I just used the main Sage UI to 'Exit' and then restarted it, and usage was back down at about 86MB when it came back. Just checked again after typing all this and it's crept on its own, with the UI doing nothing but sitting on the plugins screen, to 93MB.

Right now I'm tempted to just update the two items that say update available. I know I didn't delete any system messages without requesting the update for them first, so don't know how I got out of sync. Let me know if you think the noted items could be causing any problem, but frankly I don't think they would as I have no BMT log updates recently, so per the above commonality with bastafidli it still looks like Phoenix is the only potential culprit.

Let me know if you'd like the Phoenix log I captured this a.m. (copied it over when I shut Sage down just in case it got overwritten). Nothing new was written to it after restart, and I have a one-line sagex-api log after restarting as well. Will try to watch it during the day and see if the heap size (now 108MB) stabilizes somewhere or if it just keeps on creeping, even though the system is just sitting idle, we're not watching anything on it at all nor is it recording.

EDIT - just checked again after waiting about 15 min and the heap is back down to 80MB or so. Seems like it did a flush (or garbage collection?) of some sort on its own. So now I've got no clue what caused my earlier crashes. Also confirmed that Comskip ran on a manual recording made last night, as I've got the timeline color-coding showing on it, so I still don't think comskip misbehavior was the cause either. My problems may be resolving themselves - aside from my manual restart this morning out of paranoia I haven't seen a crash for 3 days now.

[stuckless]

@rtrski - think you need to understand how java allocates memory. ie, using the system, even doing trivial task will cause the heap to grow. But a growing heap is much different than a memory leak. Based on your actions, I'd say it was very normal for the heap to grow to 256+ mb, especially if you are configured with a heap of 512mb+.

My current used memory is 86mb (system has been up for about week) and it grows and shrinks as I use the system. When there clients connected, it will grow quite rapidly, but as the system remains idle, it will go down again (that's how the jvm memory allocation works).

Now, as you do memory intensive operations, like download fanart/metadata or rescale images, then your memory is going to spike, and that is not an indication of a memory leak, that an indication that you are using the system.

Scaling images (which is done when you download fanart) or if you use the Scale Images button will quickly eat through memory, as each image has to be loaded into memory before it can be scaled. Images in memory are uncompressed, so a 1mb image may take 10mb or more in memory. You can see how loading 100s of those images will quickly eat through memory as java will simply keep allocating memory (especially under load) until it is forced to do a garbarge collection (ie, it can't allocate a chunk). It will then pause and try to clean up memory and then re-allocate. If it can't allocate a large enough chunk to hold the uncompressed image, then it will fail with OOM.

The scaling operations try to minimize their references to the uncompressed images, so that the jvm can reclaim the memory at a later point.

For the record as well, just this morning I added 2 new files to my collection and had sagetv scan them, and my used memory is still sitting at 80+mb.

After starting a placeshifter client, it jumped to 115mb+, which would be normal, as far as I'm concerned. I would think that after a some time of sitting idle it will go back down again. At least that's been my experience, given that every night and usually thoughout the day, people are using the sage server, but by morning, the allocated memory drops back to 80-100mb.

[rtrski]

Stuckless - thanks. I too am starting to think my prior failures were something other than OOM after all. So I'll stop dogging the heap and just wait for the next Windows failure message (and not hold my breathe <grin>).

[stuckless]

Originally Posted by rtrski

Stuckless - thanks. I too am starting to think my prior failures were something other than OOM after all. So I'll stop dogging the heap and just wait for the next Windows failure message (and not hold my breathe <grin>).

I'm not disputing OOM, since I've gotten it myself, form time to time (including when doing a scale images in 1000s of images ), but I just wanted to clarify that heap growing is not the same as a memory leak. (but heap growing, and never shinking after idle periods is a memory leak)

[bastafidli]

Context sensitive menu plugin allows me to see the java heap very easily (without diving into submenus). What I have observed over last couple of days is that my heap grows every day by 100MB. When it reaches 600+ I will get OoM soon. I use the Restart SaveTV plugin to then restart manually which brings me back to 150MB heap. Now when I can easily see this behaviour I will start experimenting with removing plugins.

BTW. My usage of Sage is (1 x HD100)
- record usually 2 programs a day (morning show and nightly news)
- watch usually 1 kid show
- watch usually 1 adult show
- sometimes liste to music
- use the photo screensaver 24/7

[Nelbert]

I'd switch the photo screensaver off first. See if that makes any difference.

Photos are usually memory intensive... that 2 Mb jpeg on the h/disk can easily become 64Mb+ in memory. It's possible as it's cycling through the photos it's not freeing up memory because it's needing buffer/scratch areas for the next photo. It could be the photo screen saver has a memory leak, which as you run it 24/7 shows up on your system.

[gabrielcab]

I've started to see the memory heap error and sage turns unbrowsable. I have sagetv installed on a windows home server and i'm using a hd300. How can i view how much memory java is using?

[tmiranda]

Originally Posted by gabrielcab

I've started to see the memory heap error and sage turns unbrowsable. I have sagetv installed on a windows home server and i'm using a hd300. How can i view how much memory java is using?

The information is in the "System Information" menu. You can graphically monitor the usage by install the Java Heap Monitor Plugin.

[bastafidli]

I am in a process of disabling plugins on my installation to figure out which one is causing this issue. One thing I learned is that this out-of-memory is usage based. On Monday we had whole family sick day and therefore we were watching it whole day and we were able to get OoM in a single day. If we are watching it only 1-2 hours a day, it takes several days to get OoM.

[bastafidli]

In my last round I have disabled "Upcoming movies" and "OSD Clock and Play Time" and after several hours of using Sage the memory stays at the same level as when newly restarted (195MB). Time to start enabling them back to reproduce the problem. My money is on "OSD Clock and Play time", which would be too bad, since I love this plugin.

[toricred]

As the author of OSD Clock I would concur that it may be a problem. I had problems with the clock some time ago and fixed a memory leak in it. When I added playtime it had the same problem and I didn't notice until today. I've uploaded a new version that incorporates the same fix. If you use this plugin, please try the new version and let me know if it is fixed.

[bastafidli]

Thank you. I will defintely try it. I am still running tests to verify that this is the only source of leak.

[bastafidli]

After testing it for few days I can confirm that all is well again in Sage land after updating the OST Clock plugin :-). Thanks to toricred for fast fix. My heap starts at 178 of 954MB, sometimes go up to 230MB but in the morning it is always around 190ish.

[toricred]

I'm glad it fixed your problem. One other thing that might help with this type of problem would be to set allow_periodic_explicit_gc to true in your sage.properties file.