SageTV docker Java crash

[wayner]

I am getting the occasional crash of the SageTV Java process. Last night it crashed while watching a show but there was nothing done that seemed to precipitate the crash as it happened midway through a recorded TV show. It also crashed again at about 7am this morning when I don't think anyone was watching a show.

When it crashes I see a message like:

./startsagecore: line 48: 106 Aborted java -Djava.awt.headless=$HEADLESS $JAVAMEM -XX:+UseAdaptiveSizePolicy -XX:MaxGCPauseMillis=25 -XX:GCTimeRatio=24 $(if [[ $EUID -eq 0 ]]; then echo '-XX:ThreadPriorityPolicy=1'; fi) $JAVAOPTS -cp Sage.jar:JARs/lucene-core-

in the log for the SageTV docker file.

The crash last night did create a hs_err_pidXXX.log file although the one this morning did not. Here is the beginning of that file.

Anyone have any idea on what could be causing this?

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00002af415e0221b, pid=105, tid=0x00002af418dc0700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_111-b14) (build 1.8.0_111-b14)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.111-b14 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libImageLoader.so+0x5c21b]  put1bitbwtile+0x8b
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

---------------  T H R E A D  ---------------

Current thread (0x00002af3a8011000):  JavaThread "Fork-BASE-53879" daemon [_thread_in_native, id=365, stack(0x00002af418cc0000,0x00002af418dc1000)]

siginfo: si_signo: 11 (SIGSEGV), si_code: 1 (SEGV_MAPERR), si_addr: 0x0000000000e01c80

Registers:
RAX=0x0000000000e01ca0, RBX=0xffffffffffffcb00, RCX=0x00002af378c60231, RDX=0x00002af378358eb0
RSP=0x00002af418dbe718, RBP=0x00000000000000d4, RSI=0x0000000000e01c80, RDI=0x00002af378c60304
R8 =0x00000000000006a0, R9 =0x0000000000000000, R10=0x00002af3783586b0, R11=0x0000000000000875
R12=0x00002af415e291a0, R13=0x0000000000000000, R14=0x0000000000001a80, R15=0x00000000ffffffff
RIP=0x00002af415e0221b, EFLAGS=0x0000000000010206, CSGSFS=0x002b000000000033, ERR=0x0000000000000006
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00002af418dbe718)
0x00002af418dbe718:   0000000000000875 00002af418dbe800
0x00002af418dbe728:   0000000000000876 0000000000000000
0x00002af418dbe738:   0000000000000876 0000000000000000
0x00002af418dbe748:   00002af415e0703f ffffffff00000000
0x00002af418dbe758:   00000000fffff2c0 00002af378c60230
0x00002af418dbe768:   00002af378770520 00002af378c60230
0x00002af418dbe778:   000006a0000000d4 0000000000000000
0x00002af418dbe788:   00002af415e02190 00000000fffff2c0
0x00002af418dbe798:   0000000100000001 0000000000000590
0x00002af418dbe7a8:   ffffffff7890ce20 00002af37890d2f8
0x00002af418dbe7b8:   00002af378770520 0000000000000000
0x00002af418dbe7c8:   0000000000000000 0000000000000000
0x00002af418dbe7d8:   00002af418dbe880 00002af418dbe800
0x00002af418dbe7e8:   00002af415e095e0 00002af3ffffffff
0x00002af418dbe7f8:   ffffffff00000008 00002af378770520
0x00002af418dbe808:   0000000100000000 000006a000000000
0x00002af418dbe818:   0001000100000876 0000000000040001
0x00002af418dbe828:   0000000000000000 0000000000000000
0x00002af418dbe838:   0000000000000000 00002af415e06f10
0x00002af418dbe848:   00002af415e02190 0000000000000000
0x00002af418dbe858:   00002af3783586b0 0000000000000000
0x00002af418dbe868:   0000000000000000 0000000000000000
0x00002af418dbe878:   0000000000000000 0000000000000000
0x00002af418dbe888:   0000000000000000 0000000000000000
0x00002af418dbe898:   0000000000000000 0000000000000000
0x00002af418dbe8a8:   0000000000000000 0000000000000000
0x00002af418dbe8b8:   0000000000000000 0000000000000000
0x00002af418dbe8c8:   0000000000000000 0000000000000000
0x00002af418dbe8d8:   0000000000000000 0000000000000000
0x00002af418dbe8e8:   0000000000000000 0000000000000000
0x00002af418dbe8f8:   0000000000000000 0000000000000000
0x00002af418dbe908:   0000000000000000 0000000000000000 

Instructions: (pc=0x00002af415e0221b)
0x00002af415e021fb:   8d 3c 29 48 89 f0 0f 1f 80 00 00 00 00 48 83 c1
0x00002af415e0220b:   01 0f b6 51 ff 48 83 c0 20 49 8b 14 d2 44 8b 3a
0x00002af415e0221b:   44 89 78 e0 44 8b 7a 04 44 89 78 e4 44 8b 7a 08
0x00002af415e0222b:   44 89 78 e8 44 8b 7a 0c 44 89 78 ec 44 8b 7a 10 

Register to memory mapping:

RAX=0x0000000000e01ca0 is an unknown value
RBX=0xffffffffffffcb00 is an unknown value
RCX=0x00002af378c60231 is an unknown value
RDX=0x00002af378358eb0 is an unknown value
RSP=0x00002af418dbe718 is pointing into the stack for thread: 0x00002af3a8011000
RBP=0x00000000000000d4 is an unknown value
RSI=0x0000000000e01c80 is an unknown value
RDI=0x00002af378c60304 is an unknown value
R8 =0x00000000000006a0 is an unknown value
R9 =0x0000000000000000 is an unknown value
R10=0x00002af3783586b0 is an unknown value
R11=0x0000000000000875 is an unknown value
R12=0x00002af415e291a0: <offset 0x831a0> in /opt/sagetv/server/libImageLoader.so at 0x00002af415da6000
R13=0x0000000000000000 is an unknown value
R14=0x0000000000001a80 is an unknown value
R15=0x00000000ffffffff is an unknown value


Stack: [0x00002af418cc0000,0x00002af418dc1000],  sp=0x00002af418dbe718,  free space=1017k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libImageLoader.so+0x5c21b]  put1bitbwtile+0x8b

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
J 9614  sage.media.image.ImageLoader.loadScaledImageFromFile(Ljava/lang/String;IIII)Lsage/media/image/RawImage; (0 bytes) @ 0x00002af37caebb1f [0x00002af37caebac0+0x5f]
J 11264 C2 sage.media.image.ImageLoader.loadResizedRotatedImageFromFile(Ljava/lang/String;IIII)Lsage/media/image/RawImage; (44 bytes) @ 0x00002af37ce67f08 [0x00002af37ce67ee0+0x28]
J 21938 C2 sage.MetaImage.getRawImage(I)Lsage/media/image/RawImage; (4210 bytes) @ 0x00002af37f9a6da8 [0x00002af37f9a5400+0x19a8]
J 15613 C2 sage.MiniClientSageRenderer.createNativeImage(Lsage/MetaImage;I)Z (5868 bytes) @ 0x00002af37f02e72c [0x00002af37f02b760+0x2fcc]
J 8314 C2 sage.MetaImage.getNativeImage(Lsage/NativeImageAllocator;I)J (335 bytes) @ 0x00002af37d63320c [0x00002af37d632c40+0x5cc]
J 12762 C2 sage.MiniClientSageRenderer.preloadImage(Lsage/MetaImage;)V (448 bytes) @ 0x00002af37cd11564 [0x00002af37cd11180+0x3e4]
J 13350 C1 sage.api.Utility$52.runSafely(Lsage/Catbert$FastStack;)Ljava/lang/Object; (259 bytes) @ 0x00002af37e7a9864 [0x00002af37e7a7ae0+0x1d84]
J 5773 C2 sage.PredefinedJEPFunction.run(Lsage/Catbert$FastStack;)V (155 bytes) @ 0x00002af37cf068ac [0x00002af37cf06840+0x6c]
J 12939 C2 sage.jep.JEP$CommandEvaluator.evaluate([Lsage/jep/JEP$CommandElement;Lsage/jep/SymbolTable;)Ljava/lang/Object; (190 bytes) @ 0x00002af37e85e6ec [0x00002af37e85e640+0xac]
J 12906 C2 sage.jep.JEP.getValueAsObject(Lsage/Catbert$Context;Lsage/ZPseudoComp;)Ljava/lang/Object; (337 bytes) @ 0x00002af37e8749d8 [0x00002af37e874640+0x398]
J 12925 C2 sage.Catbert.evaluateExpression(Ljava/lang/String;Lsage/Catbert$Context;Lsage/ZPseudoComp;Lsage/Widget;)Ljava/lang/Object; (967 bytes) @ 0x00002af37e957964 [0x00002af37e9577e0+0x184]
J 13178 C2 sage.ZPseudoComp.processChain(Lsage/Widget;Lsage/Catbert$Context;Ljava/util/Set;Lsage/ZPseudoComp;ZZLjava/util/Set;)Lsage/Catbert$ExecutionPosition; (1534 bytes) @ 0x00002af37e8c35b8 [0x00002af37e8c2e80+0x738]
J 13178 C2 sage.ZPseudoComp.processChain(Lsage/Widget;Lsage/Catbert$Context;Ljava/util/Set;Lsage/ZPseudoComp;ZZLjava/util/Set;)Lsage/Catbert$ExecutionPosition; (1534 bytes) @ 0x00002af37e8c5bbc [0x00002af37e8c2e80+0x2d3c]
J 20761 C1 sage.ZPseudoComp$2.run()V (79 bytes) @ 0x00002af37f5b78cc [0x00002af37f5b74e0+0x3ec]
j  sage.Pooler$PooledThread.run()V+41
v  ~StubRoutines::call_stub

[EnterNoEscape]

It looks like it crashed loading an image, but I couldn't tell you what image. Do you have the log files from when it crashed?

[wayner]

Thanks - that helped me to nail it down.

It turns out that the image that it crashed on happens to be a tif file (which appears to be a scan) which is two pages long. When I tried to browse to this file via the Pictures menu of SageTV it crashed again as soon as I got to the folder when it was presumably trying to generate a thumbnail.

I am able to view this file properly in other applications so it appears that the file is not corrupt but that loading this (type of?) file causes SageTV (or perhaps more correctly Java) to crash. I will try on my other servers to see if this happens on V7 and Windows. I will also try to see if this happens to other tif files, either single or multi page.

In the meantime I can delete this file as I don't even want to keep the file around. But to prevent this from inadvertently happening again should I remove the extensions .tif and .tiff from the following line in Sage.properties.

seeker/picture_library_import_filename_extensions=.jpg,.gif,.jpeg,.jpe,.png,.tif,.tiff

If I do this will that prevent Sage from trying to import or view tif files in the future, either in the screen saver or in the Browse Pictures menu?

Should I also add this as a bug in github?

Or is it possible a Java bug rather than a SageTV bug?

I am guessing that this bug is not present in SageTV/Java on Windows either V7 or V9 as I have been had the same file show up in the screen saver without causing crashes for years. The other difference is the Java version - as my Windows servers are both (obviously) 32 bit and I believe that one or both of them are Java 7 -particularly my production server which has been running for years on Win 7 32bit with Java 7 with none of these crashes and it must have displayed that image many, many times.

[EnterNoEscape]

The funny thing is that I'm fairly certain no one has actually packaged a newly compiled version of the Windows DLL that caused this crash, so the one that crashed was the same code as v7. I would create an issue on GitHub and attach the picture that crashed it (is possible), but the catch is that since no one that I'm aware of has actually compiled the image loader on Windows, that part would also need to be worked out before you'll see this fixed.

If you're using Schedules Direct, you might have noticed channel logos that are green and half width. That's a gray-scale PNG image bug involving the same DLL that was fixed, but only Linux has enjoyed the fix because we still don't have it compiling on Windows.

[wayner]

By the Windows DLL that caused this crash do you mean /opt/sagetv/server/libImageLoader.so?

To be honest I don't care as much about this on Windows as my primary server is now on unRAID but I will likely keep Sage running on my VR PC, which runs Win10 and SageTV V9, as my backup SageTV server. But we will want to fix this bug as others may discover it in the future.

[EnterNoEscape]

Originally Posted by wayner

By the Windows DLL that caused this crash do you mean /opt/sagetv/server/libImageLoader.so?

To be honest I don't care as much about this on Windows as my primary server is now on unRAID but I will likely keep Sage running on my VR PC, which runs Win10 and SageTV V9, as my backup SageTV server. But we will want to fix this bug as others may discover it in the future.

I just looked at your signature. In Linux, it's a library and that's the right path. So it's possible that recent changes created a new problem that needs to be addressed.

[wayner]

I just tried viewing the file through the Sage UI on my former production server running V7 and Win7 and it crashed as well - although it looks as if sometimes may only crash the UI context rather than . So it looks like it is an old bug.

One of the crashes on V7 was in the ThumbnailGen daemon.

# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x024a00e7, pid=12608, tid=9644
#
# JRE version: Java(TM) SE Runtime Environment (7.0_45-b18) (build 1.7.0_45-b18)
# Java VM: Java HotSpot(TM) Client VM (24.45-b08 mixed mode windows-x86 )
# Problematic frame:
# C  [ImageLoader.dll+0x400e7]
#
# Failed to write core dump. Minidumps are not enabled by default on client versions of Windows
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

---------------  T H R E A D  ---------------

Current thread (0x4dafac00):  JavaThread "ThumbnailGen" daemon [_thread_in_native, id=9644, stack(0x57c90000,0x57d90000)]

siginfo: ExceptionCode=0xc0000005, writing address 0x00e01c80