OutOfMemory Exception

[mgpaulus]

My system has suddenly started throwing OutOfMemory Exceptions about every other day or so. I'm not recalling any new plugins I've installed, and I'm not sure what's causing the issue.

I upped my JVMMaxHeapSizeMB to 0x600, and it appears that Sage is respecting that: SysInfo shows Jave Heap: Size: 264MB/586MB/1557MB and JVM Heap Size (Used/Total/Max): 324MB/586MB/1557MB.

When I look in the logs, I see that the exception is thrown by several processes, but all of them wind up throwing this message: ERROR processing message of:java.lang.OutOfMemoryError: unable to create new native thread

Not sure how to debug this. Every time I try to run jmap,
I get this:

$ /cygdrive/c/Program\ Files\ \(x86\)/Java/jdk1.7.0_80/bin/jmap -histo -F 4068
Attaching to process ID 4068, please wait...
Debugger attached successfully.
Client compiler detected.
JVM version is 24.80-b11
Iterating over heap. This may take a while...
Exception in thread "main" java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at sun.tools.jmap.JMap.runTool(JMap.java:197)
        at sun.tools.jmap.JMap.main(JMap.java:128)
Caused by: sun.jvm.hotspot.debugger.DebuggerException: Windbg Error: ReadVirtual failed!
        at sun.jvm.hotspot.debugger.windbg.WindbgDebuggerLocal.readBytesFromProcess0(Native Method)
        at sun.jvm.hotspot.debugger.windbg.WindbgDebuggerLocal.readBytesFromProcess(WindbgDebuggerLocal.java:485)
        at sun.jvm.hotspot.debugger.DebuggerBase$Fetcher.fetchPage(DebuggerBase.java:76)
        at sun.jvm.hotspot.debugger.PageCache.getPage(PageCache.java:178)
        at sun.jvm.hotspot.debugger.PageCache.getInt(PageCache.java:96)
        at sun.jvm.hotspot.debugger.DebuggerBase.readCInteger(DebuggerBase.java:355)
        at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
        at sun.jvm.hotspot.debugger.windbg.WindbgDebuggerLocal.readOopHandle(WindbgDebuggerLocal.java:327)
        at sun.jvm.hotspot.debugger.windbg.WindbgAddress.getOopHandleAt(WindbgAddress.java:117)
        at sun.jvm.hotspot.oops.Oop.getKlassForOopHandle(Oop.java:233)
        at sun.jvm.hotspot.oops.ObjectHeap.newOop(ObjectHeap.java:380)
        at sun.jvm.hotspot.oops.ObjectHeap.iterateLiveRegions(ObjectHeap.java:466)
        at sun.jvm.hotspot.oops.ObjectHeap.iterate(ObjectHeap.java:244)
        at sun.jvm.hotspot.tools.ObjectHistogram.run(ObjectHistogram.java:47)
        at sun.jvm.hotspot.tools.ObjectHistogram.run(ObjectHistogram.java:37)
        at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
        at sun.jvm.hotspot.tools.ObjectHistogram.main(ObjectHistogram.java:56)
        ... 6 more

(I get it also if I try to run jmap in a normal DOS window with Administrator rights).

If anyone has any ideas on how to get to the bottom of this, I would appreciate it. I wind up having to reboot every other day or so, which is not what I would consider stable.

Win 7 64 bit
Jdk 1.7 - 32-bit - b/c I'm using OpenDCT
16GB RAM

[jusjoken]

Originally Posted by mgpaulus

My system has suddenly started throwing OutOfMemory Exceptions about every other day or so. I'm not recalling any new plugins I've installed, and I'm not sure what's causing the issue.

I upped my JVMMaxHeapSizeMB to 0x600, and it appears that Sage is respecting that: SysInfo shows Jave Heap: Size: 264MB/586MB/1557MB and JVM Heap Size (Used/Total/Max): 324MB/586MB/1557MB.

When I look in the logs, I see that the exception is thrown by several processes, but all of them wind up throwing this message: ERROR processing message of:java.lang.OutOfMemoryError: unable to create new native thread

Not sure how to debug this. Every time I try to run jmap,
I get this:

$ /cygdrive/c/Program\ Files\ \(x86\)/Java/jdk1.7.0_80/bin/jmap -histo -F 4068
Attaching to process ID 4068, please wait...
Debugger attached successfully.
Client compiler detected.
JVM version is 24.80-b11
Iterating over heap. This may take a while...
Exception in thread "main" java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at sun.tools.jmap.JMap.runTool(JMap.java:197)
        at sun.tools.jmap.JMap.main(JMap.java:128)
Caused by: sun.jvm.hotspot.debugger.DebuggerException: Windbg Error: ReadVirtual failed!
        at sun.jvm.hotspot.debugger.windbg.WindbgDebuggerLocal.readBytesFromProcess0(Native Method)
        at sun.jvm.hotspot.debugger.windbg.WindbgDebuggerLocal.readBytesFromProcess(WindbgDebuggerLocal.java:485)
        at sun.jvm.hotspot.debugger.DebuggerBase$Fetcher.fetchPage(DebuggerBase.java:76)
        at sun.jvm.hotspot.debugger.PageCache.getPage(PageCache.java:178)
        at sun.jvm.hotspot.debugger.PageCache.getInt(PageCache.java:96)
        at sun.jvm.hotspot.debugger.DebuggerBase.readCInteger(DebuggerBase.java:355)
        at sun.jvm.hotspot.debugger.DebuggerBase.readAddressValue(DebuggerBase.java:454)
        at sun.jvm.hotspot.debugger.windbg.WindbgDebuggerLocal.readOopHandle(WindbgDebuggerLocal.java:327)
        at sun.jvm.hotspot.debugger.windbg.WindbgAddress.getOopHandleAt(WindbgAddress.java:117)
        at sun.jvm.hotspot.oops.Oop.getKlassForOopHandle(Oop.java:233)
        at sun.jvm.hotspot.oops.ObjectHeap.newOop(ObjectHeap.java:380)
        at sun.jvm.hotspot.oops.ObjectHeap.iterateLiveRegions(ObjectHeap.java:466)
        at sun.jvm.hotspot.oops.ObjectHeap.iterate(ObjectHeap.java:244)
        at sun.jvm.hotspot.tools.ObjectHistogram.run(ObjectHistogram.java:47)
        at sun.jvm.hotspot.tools.ObjectHistogram.run(ObjectHistogram.java:37)
        at sun.jvm.hotspot.tools.Tool.start(Tool.java:221)
        at sun.jvm.hotspot.tools.ObjectHistogram.main(ObjectHistogram.java:56)
        ... 6 more

(I get it also if I try to run jmap in a normal DOS window with Administrator rights).

If anyone has any ideas on how to get to the bottom of this, I would appreciate it. I wind up having to reboot every other day or so, which is not what I would consider stable.

Win 7 64 bit
Jdk 1.7 - 32-bit - b/c I'm using OpenDCT
16GB RAM

The max heap on windows is a setting of 400hex or approx 1038 so lower your setting to that and the issues may go away.

k

[EnterNoEscape]

Originally Posted by jusjoken

The max heap on windows is a setting of 400hex or approx 1038 so lower your setting to that and the issues may go away.

k

Agreed. The reason no one goes above 1024 is because the JVM gets unstable. I presume it has to do with how the JVM allocates the heap being limited to 32-bit addresses (and they might in reality be 31-bit addresses effectively if they are being used signed like everything else in Java) plus it's own overhead. I have tried even small amounts over 1024 and it doesn't work out well. On Java 6, I couldn't even go above 768 without strange problems.

[Narflex]

I'd got with their suggestions...the problem isn't that you've run out of heap space as the message indicates:

ERROR processing message of:java.lang.OutOfMemoryError: unable to create new native thread

The problem is that it can't create a new native thread which means either the OS is out of memory, the process/OS hit its thread limit or there's something else odd happening (which could be the JVM max heap size was set too big).

[mgpaulus]

It was originally at 0x400 when I first started experiencing the problem, which is why I bumped it up. I've bumped it back down, and I'll monitor.

Here's a stupid question: Since I am running sagetv as a service, how do I get stats on the Service JVM? It appears that the JVM heap size info is related to the Client, not the service (I say this, because I changed the setting in the registry, and then restarted the Client and noticed my values had changed, even though I have not restarted the service (recording going on right now, so I'll wait to bounce the service).

(Any chance of getting a jmx port enabled so that we could connect via jconsole and instrument the JVM?)

[Narflex]

If you use the Placeshifter then it will report the server's memory state.

[Narflex]

And feel free to propose changes in GitHub that add command line parameters to enable extra debugging modes.