Sage Process on Server Crashing

[rickgillyon]

This is happening several times a day. I'm not using the service. Latest v9 on Win10 64 Pro. Any ideas what's causing it? hs_err_pid and log attached.

[rickgillyon]

Also getting separately, around once per day, an error that the SageTV process has stopped, which stops all SageTV activity until I click it. Details below. It's better if the process just stops itself rather than waiting for input as it restarts...

Faulting application name: SageTV.exe, version: 9.1.5.683, time stamp: 0x594a4a96
Faulting module name: ntdll.dll, version: 10.0.15063.447, time stamp: 0xd51d5c5e
Exception code: 0xc0000005
Fault offset: 0x0002ef5d
Faulting process id: 0x27bc
Faulting application start time: 0x01d30e39f60e6701
Faulting application path: C:\Program Files (x86)\SageTV\SageTV\SageTV.exe
Faulting module path: C:\WINDOWS\SYSTEM32\ntdll.dll
Report Id: d6b5dcc2-69bf-4969-9da6-d416ca855e4b
Faulting package full name: 
Faulting package-relative application ID:

Any hints on these would be helpful.

[wnjj]

Your system is running out of native memory. There are a few others posting in other threads having the same issue. Try turning off the photo screensaver if you are running it. Something is wrong in V9 that is affecting several users in the same way. The log you posted already was out of memory at the top. If it crashes again, post the last log file that isn't showing 'out of memory' issues. Sagetv_1.txt is older than sagetv_0.txt and sagetv_2 is older than that, etc.

[rickgillyon]

Thanks, will do. Every time I look it's running at <30% RAM usage, but I guess it just suddenly maxes out and crashes...

[wnjj]

Originally Posted by rickgillyon

Thanks, will do. Every time I look it's running at <30% RAM usage, but I guess it just suddenly maxes out and crashes...

A 32-bit process only has access to 2GB of user memory. The Java heap (default 768MB) comes out of that, as does the JVM itself (overhead plus per-thread) and every other DLL loaded to run sage. Then, sage itself has native code in DLL's that request some memory plus Java can request native memory byte buffers directly. The native memory requests are often related to image processing o file buffering so that's why the photo screensaver could be a suspect.

One of the above or the combination of all is using too much. It does seem to be a new issue in V9 for some. I've been looking through the source code for a memory leak and so far haven't found anything. Keep those log files coming and I'll keep looking.

[Narflex]

If you go to this photo in the photos menu and view it, does it crash?

V:\Photos\2006\200612\IMG_3829.jpg

There may be something specifically odd about that photo that's causing it to allocate a huge chunk of native memory incorrectly which would create a problem like this. If it does reproduce it...please DO NOT delete that photo so we can fix the underlying problem.

[rickgillyon]

There is something odd about that photo. I'm on it.

But I currently have it displayed on a PC client and an Android extender with no apparent ill effect. I've attached it anyway, but it looks like all the others from this end. Nope, too big to attach, it's here:
http://www.gillyon.com/downloads/IMG_3829.jpg

[wnjj]

Originally Posted by Narflex

If you go to this photo in the photos menu and view it, does it crash?

V:\Photos\2006\200612\IMG_3829.jpg

There may be something specifically odd about that photo that's causing it to allocate a huge chunk of native memory incorrectly which would create a problem like this. If it does reproduce it...please DO NOT delete that photo so we can fix the underlying problem.

Jeff,
Several users are crashing in imageloader but some are even crashing the JVM itself. The imageloader crash is because there are mallocs that don't check for success (null pointer). Those will crash even if a small amount is requested but not available. The real issue is why is so much native memory being used up or fragmented? It seems that some who upgraded V7 systems started having this issue. Perhaps they were using close to the limit before and are now just over or maybe their media types or photo formats bring out a memory leak or bug?

[JREkiwi]

My system was more stable after turning off "Optimized Image Loader" but was still having crashes using any of the 9.1 versions. I've had to go back to 9.0.13 to get a stable system.

John

[wnjj]

Originally Posted by JREkiwi

My system was more stable after turning off "Optimized Image Loader" but was still having crashes using any of the 9.1 versions. I've had to go back to 9.0.13 to get a stable system.

John

That's quite interesting. The most significant change memory-wise in 9.1.x was turning on G1GC string deduplication. I wonder if that feature enabling saving of Java heap has a cost to native memory?

[JREkiwi]

Originally Posted by wnjj

That's quite interesting. The most significant change memory-wise in 9.1.x was turning on G1GC string deduplication. I wonder if that feature enabling saving of Java heap has a cost to native memory?

I wonder if we're running into an issue with Metaspace with G1GC or whether the -XX:+ParallelRefProcEnabled flag needs to be set ?

Also Narflex gives and explanation here why turning off the "Optimized Image Loader" would reduce the impact on native memory

John

[JREkiwi]

I also wonder whether we should be using G1GC when it still has bugs like this

John

[wnjj]

Originally Posted by JREkiwi

I also wonder whether we should be using G1GC when it still has bugs like this

John

If you want to try some more experimenting, go back to a 9.1.x version but use the .exe from 9.0.x. The .exe is where the G1GC is enabled when launching the JVM.

[JREkiwi]

Originally Posted by wnjj

If you want to try some more experimenting, go back to a 9.1.x version but use the .exe from 9.0.x. The .exe is where the G1GC is enabled when launching the JVM.

I've installed 9.1.5 with the 9.0.13 exes. We'll see how that goes. Thanks for the suggestion.

John

[rickgillyon]

I watched today as the memory usage on my server went from the usual 21% to 76%, for an extended time. The server has 8GB. I've attached the logs, but no crash today. I think the high usage coincides with an EPG update.

Logs here:
http://www.gillyon.com/downloads/sagetv_0.zip

[wnjj]

Originally Posted by JREkiwi

I've installed 9.1.5 with the 9.0.13 exes. We'll see how that goes. Thanks for the suggestion.

John

How's the change in GC working? I'm curious to know if that was the issue.

[JREkiwi]

Originally Posted by wnjj

How's the change in GC working? I'm curious to know if that was the issue.

Didn't have any problems and have been running 9.1.6 which uses the default GC without problems as well. I do still have "Optimized Image Loader" disabled though.

Not sure if G1GC was the entire problem, but from my reading it certainly seems more suited to large JAVA heaps.

John

[EnterNoEscape]

Originally Posted by JREkiwi

Didn't have any problems and have been running 9.1.6 which uses the default GC without problems as well. I do still have "Optimized Image Loader" disabled though.

Not sure if G1GC was the entire problem, but from my reading it certainly seems more suited to large JAVA heaps.

John

Ironically, it was supposed to help out Windows users with memory issues because of the really good string duplication.

[Narflex]

The optimized image loader should actually use less system memory overall and definitely performs faster than that non-optimized one (as one would expect). It does use more native memory and less Java memory...but in the overall total it will be less. I definitely think it would be worth trying to enable that again now that you've seen improvement...I wouldn't want people disabling that as a potential fix for things that aren't broken now that the GC setting has been changed back.

[JREkiwi]

Originally Posted by Narflex

The optimized image loader should actually use less system memory overall and definitely performs faster than that non-optimized one (as one would expect). It does use more native memory and less Java memory...but in the overall total it will be less. I definitely think it would be worth trying to enable that again now that you've seen improvement...I wouldn't want people disabling that as a potential fix for things that aren't broken now that the GC setting has been changed back.

I believe part of the problem is/was that Java 8 uses native memory for MetaSpace and G1GC in the current JAVA versions has a bug with cleaning up that used native memory. That bug is supposed to be fixed in JAVA 8.151

John