SageTV 9.1 Out of Memory (Fixed)

[JREkiwi]

Thought I'd post this so the information is available for anyone else running into issues with running out of memory following the changes made in 9.1. I suspect this will only affect the Windows users who don't have the benefit of the extra heap headroom enjoyed by the linux folks.

After upgrading to 9.1 my Windows 7 system went from being incredibly stable to falling over multiple times a day with "Out of Memory"

Following the notes from enternoescape in the pull request for the code change I made the following change in sage.properties

disable_carny_maps=true

My system has now being running 3 days since the change.

John

[EnterNoEscape]

Interesting. I tested this on a heap as small as 384MB with multiple miniclients and didn't run into these issues. What Java version are you running? What do you have your heap size set to?

[sacrament055]

I've noticed a major improvement in memory management in my case. I'm running java 1.8 on Windows 10 64bit. Maybe the issue is OS version of plugin related?

[Taddeusz]

I've noticed that over several days memory usage increases significantly. It definitely starts at more than it did prior to the Carny improvement. It starts at around 600MB usage and after several days it grew till SageTV was using about 1500MB. Concerned there may be a memory leak of some kind in there somewhere.

I've also noticed that the total java heap has increased significantly. I increased my heap to 3GB as I was afraid of SageTV running out of memory. But with the new Carny the total heap seems to be nearly as large as the max heap. Where before the total heap would remain relatively small.

I'm not necessarily concerned about my own system's memory usage. Running it on unRAID with 24GB of RAM so memory is not a concern for me personally. Just want to report what I'm seeing.

[Taddeusz]

Not sure how Java works with garbage collection but in both JavaScript and C# an object won't get garbage collected when something still has a reference to that object. These newer languages make memory management easier but you have to make sure you clean up object references before "disposing" of objects.

[stuckless]

Originally Posted by Taddeusz

Not sure how Java works with garbage collection but in both JavaScript and C# an object won't get garbage collected when something still has a reference to that object. These newer languages make memory management easier but you have to make sure you clean up object references before "disposing" of objects.

That's pretty standard for garbage collection, and Java would work the same. The thing with Java and garbage collection, is that it will run when it needs to, and it will reclaim easy to re-claim objects first. ie, Object A holds a reference to Object B and nothing is holding onto Object A... then B will get reclaimed in one pass and then detached from A, and then in another pass, later, Object A would be freed. (Garbage collection is a science onto itself, so trivializing as I've done, is not 100% correct, but that's the general idea).

So, it's not uncommon in Java for people to go, "Oh my God, I have a memory leak", because they give it 2G jvm and it just keeps filling up. That's how java works. It's not going to aggressively keep the memory low, if it has lots of memory play with.

Generally, how you know if you have a memory leak, is if you actually get an out of memory error. Since that would happen when Java has exhaustively tried to free up enough memory for your new object, but it can't, so it dies with OOM. OOM is not always an indication of memory leak, but, it's good yard stick. It could just be that you are trying to load too much stuff into memory, period, and with SageTV (Phoenix/Gemstone) loading HUGE backgrounds and posters that that take 20-50mb each... you can quickly get to a point where you run out of memory... (Although SageTV is pretty good at evictiving those things when it needs to).

I see SageTV memory (java heap) grow over time as well... On a reboot it's around 250mb then as more clients connect it jumps to 600+mb... I don't think I've ever seen it go to 1G, but I have gotten OOM errors with SageTV (Usually takes me a few months), so something (Phoenix, Gemstone, or core sagetv) likely does have a memory leak.

[KarylFStein]

Originally Posted by stuckless

I see SageTV memory (java heap) grow over time as well... On a reboot it's around 250mb then as more clients connect it jumps to 600+mb... I don't think I've ever seen it go to 1G, but I have gotten OOM errors with SageTV (Usually takes me a few months), so something (Phoenix, Gemstone, or core sagetv) likely does have a memory leak.

I'm pretty sure it's Gemstone. I got the OOM errors on a regular basis only when I had Gemstone installed.

[Taddeusz]

I do have the Phoenix framework installed but we aren't using the Phoenix UI. Still using the default STV. Never used Gemstone.

[wayner]

I always use Gemstone and could only use about 3 clients with a 1GB JVM heap size. I now have a 3GB heap size and have had over a dozen clients connected simultaneously without even hitting 2GB used. As someone else has mentioned, SageTV for Linux seems to be more parsimonious in its use of memory.

[EnterNoEscape]

The memory usage in some cases has very little to do with the Carny changes. Yes, Carny makes it use more RAM at startup, but if you do something very trivial like scroll through the home screen menu over and over rapidly, you'll see the used heap get to something like 1600MB (with a 2GB max heap) before it jumps back down to ~500MB. We also changed to G1 GC around the same time as the Carny change, and that affects anyone on Java 8. I will not pretend to know how the GC makes all of its decisions, but I know that one of them is time based. My observation is that if your server is fast enough to clear all of those objects in under 25ms, the heap will grow to fairly impressive sizes before it cleans up. I do know that the old GC was more aggressive because it has to stop the world longer to get it's job done and that's more likely what some people are seeing.

[stuckless]

Originally Posted by KarylFStein

I'm pretty sure it's Gemstone. I got the OOM errors on a regular basis only when I had Gemstone installed.

Gemstone uses a lot of phoenix, so, if there is a memory leak, it's likely a combination of both. I use gemstone with 3 extenders (rarely all 3 at once), and I run for months without hitting a memory issue.

Originally Posted by EnterNoEscape

The memory usage in some cases has very little to do with the Carny changes. Yes, Carny makes it use more RAM at startup, but if you do something very trivial like scroll through the home screen menu over and over rapidly, you'll see the used heap get to something like 1600MB (with a 2GB max heap) before it jumps back down to ~500MB. We also changed to G1 GC around the same time as the Carny change, and that affects anyone on Java 8. I will not pretend to know how the GC makes all of its decisions, but I know that one of them is time based. My observation is that if your server is fast enough to clear all of those objects in under 25ms, the heap will grow to fairly impressive sizes before it cleans up. I do know that the old GC was more aggressive because it has to stop the world longer to get it's job done and that's more likely what some people are seeing.

Yeah, likely when people are saying the new release is using more memory, I agree, it's likely the GC changes are just allowing it grow the heap more quickly, but in reality, it's not a memory leak (unless we can't reclaim the memory).

[EnterNoEscape]

Originally Posted by stuckless

Gemstone uses a lot of phoenix, so, if there is a memory leak, it's likely a combination of both. I use gemstone with 3 extenders (rarely all 3 at once), and I run for months without hitting a memory issue.

I've heard that the deallocators for direct ByteBuffer objects can be a little latent and might not actually free up before an OOM exception is thrown. I don't think I saw any custom native code being used by Phoenix, so my next thought would be how does it perform image caching (if at all)? And how?

I know SageTV stores all of that off heap and does it's own management. I noticed that EMBEDDED was using a LinkedHashMap as an LRU cache and it looks like the default uses a much slower approach that iterates through everything to find what's not being used. I was able to feel a significant performance deficit navigating recordings when I cranked up the cache size just to see what would happen. MetaImage.java is a very interesting class that probably deserves a little attention.

Originally Posted by stuckless

Yeah, likely when people are saying the new release is using more memory, I agree, it's likely the GC changes are just allowing it grow the heap more quickly, but in reality, it's not a memory leak (unless we can't reclaim the memory).

Yep, no leak in Linux at least. I noticed the behavior change immediately when I was using the new GC. I had a feeling it wasn't just me and we would eventually hear someone asking about it.

[Taddeusz]

Originally Posted by EnterNoEscape

Yep, no leak in Linux at least. I noticed the behavior change immediately when I was using the new GC. I had a feeling it wasn't just me and we would eventually hear someone asking about it.

I figured that might be part of it. I wasn't sure so I just thought I should mention it. I was keeping tabs on the pull request comments and noticed you say the new Carny uses more memory. So I was expecting a little more usage. Just not expecting the huge change that I've observed.

[wubdich]

I'm not sure if it of help or interests, but switching to 9.1.1.651 seems to cut memory consumption in half - doesn't exceed 350 MB and running rock solid. (Java 1.8.0_131, Windows 10 64bit, standard UI).

[EnterNoEscape]

Originally Posted by wubdich

I'm not sure if it of help or interests, but switching to 9.1.1.651 seems to cut memory consumption in half - doesn't exceed 350 MB and running rock solid. (Java 1.8.0_131, Windows 10 64bit, standard UI).

The most likely reason is because we switched to G1 GC so that we could enable String deduplication. For many people that will save a significant amount of memory. The catch is that you need to be using Java 8 (like yourself).

[stuckless]

Originally Posted by EnterNoEscape

I've heard that the deallocators for direct ByteBuffer objects can be a little latent and might not actually free up before an OOM exception is thrown. I don't think I saw any custom native code being used by Phoenix, so my next thought would be how does it perform image caching (if at all)? And how?

Phoenix doesn't actually cache image bytes... it does cache image filenames. For example in Phoenix you can reference an image, apply custom functions against the image, to create a new file cached image, but that image is never cached in memory, but the filenames of images, are. And to be honest, the caching in phoenix is pretty simple could lead to an OOM if you were loading millions of image filenames.

Originally Posted by EnterNoEscape

I know SageTV stores all of that off heap and does it's own management. I noticed that EMBEDDED was using a LinkedHashMap as an LRU cache and it looks like the default uses a much slower approach that iterates through everything to find what's not being used. I was able to feel a significant performance deficit navigating recordings when I cranked up the cache size just to see what would happen. MetaImage.java is a very interesting class that probably deserves a little attention.

The MetaImage class (which is where sagetv does cache images) does need some attention. It's doing a lot work, and I think it should be split out into serveral classes.

[EnterNoEscape]

Originally Posted by stuckless

Phoenix doesn't actually cache image bytes... it does cache image filenames. For example in Phoenix you can reference an image, apply custom functions against the image, to create a new file cached image, but that image is never cached in memory, but the filenames of images, are. And to be honest, the caching in phoenix is pretty simple could lead to an OOM if you were loading millions of image filenames.

The Java community seems to be all over the place about if this is an appropriate application for memory sensitive caches, but have you considered using soft references? Or would that not really make sense here?

[JREkiwi]

Originally Posted by EnterNoEscape

What Java version are you running? What do you have your heap size set to?

Java 1.8.131
Windows 7 64bit
Java Heap 1280MB (Had OOM in the past with 1024MB and 1280MB has worked fine for 2+ years)
Wiz.Bin 100MB (13 years)
Video Content 10TB
Library Content 60TB (Video, Pictures, Music)

John

[stuckless]

Originally Posted by EnterNoEscape

The Java community seems to be all over the place about if this is an appropriate application for memory sensitive caches, but have you considered using soft references? Or would that not really make sense here?

Phoenix really only caching a java.io.File object using a string key, and then, only for actual fanart lookups. So if we have 100,000 files, but we are only showing fanart for 100, then only 100 items are in the cache. The "leak" is that I never clear it out, so, it could possible grow, but you'd need millions before this really became a problem for someone. My sagetv server instance generally remains up for 100+ days without a restart, and I've rarely encountered a memory issues. Where I see the most memory consumed is when I connect extenders, and then disconnect. When I've gotten OOM issue in my sagetv setup, it's usually because I'm doing lots of testing with the MiniClient connecting 100s of times in the course of a weekend, and eventually I get an OOM. But I never really get OOM under normal usage.

I could use a Weak cache, and phoenix does have a SimpleWeakCache implementation for it's ICache... but I'd need to review what I did there... Java's WeakHashMap is sort of counter intuitive in that the "keys" are weak, and not the values (but as I type this... I can see the logic in doing that).

When I get back to Phoenix, I'll review the caching, but I don't think it's the cause of severe memory leaks, mainly because there's so little in the caches.

[EnterNoEscape]

In John's case, at least for the log file provided to me, the OOM was because the JVM couldn't allocate a new thread. There are no OOM errors for any other object creation in the JVM. This is a native memory problem and the logging also showed that the JVM heap likely had little to do with the problem (more than half of it was free at the time). Native thread allocation issues are OS specific and can also happen due to outside forces. The implication is that the OS ran out of memory that could be used to allocate threads. The obvious question is why? I really wish there was a thread dump in there.