Linux: Core Dump Scanning HDHR QAM

[Taddeusz]

Testing on an ESXi vm. Hooking to one of my HDHR's. It starts the scan. As soon as it hits the first channel with any QAM it crashes with a core dump. The channel it hits, 13, has only encrypted programming.

[Taddeusz]

Same thing happens when I hook to my other OTA HDHR.

[Narflex]

I'm assuming this is from the open source build. If you could post any additional details on the crash that'd be great...ideally, open up the core dump file with gdb and then post the stack trace of where it's failing.

[Taddeusz]

Ok, I hope this helps. Could not figure out gdb. I ran strace on the sagetv's java pid and this is what I got. This is from a clean folder. Running the "default-jre" on Ubuntu Server 15.04.

Too large to attach. Link to my Dropbox:

https://dl.dropboxusercontent.com/u/...etv_strace.zip

[Narflex]

Originally Posted by Taddeusz

Ok, I hope this helps. Could not figure out gdb. I ran strace on the sagetv's java pid and this is what I got. This is from a clean folder. Running the "default-jre" on Ubuntu Server 15.04.

Too large to attach. Link to my Dropbox:

https://dl.dropboxusercontent.com/u/...etv_strace.zip

No, I don't want strace output...that won't help. But this did remind me that there's likely 'hs_err_pidxxx.log' files being created in the SageTV folder every time this happens. That should have the valuable data to track down the source of the problem.

[Taddeusz]

Ok, here is that file. I copied it off before I nuked the vm just in case it could be useful. I tried going to Ubuntu 32-bit but the buildso fails because it's looking for amd64 rather than i386 under the jre/lib.

[Narflex]

Looks like it's crashing in libNativeCore.so. Can you do a debug build of that library and then crash it again?

This is what it's pointing to right now...but I don't see any 64 bit problems in there.

# C [libNativeCore.so+0x300f5] CalTSCRC32+0x35

It's very possible we will need somebody to trap this with GDB to figure out exactly what's going on.

[Taddeusz]

Ok, to update my own thread I figured out how to get SageTV to build for 32-bit. Channel scanning works fine under 32-bit, although it doesn't seem to be able to detect ATSC. I had to manually edit the Sage.Properties file to change the tuner to ATSC & Air from QAM & Cable. After that it sees the channels just fine.

[Taddeusz]

Originally Posted by Narflex

Looks like it's crashing in libNativeCore.so. Can you do a debug build of that library and then crash it again?

This is what it's pointing to right now...but I don't see any 64 bit problems in there.

# C [libNativeCore.so+0x300f5] CalTSCRC32+0x35

It's very possible we will need somebody to trap this with GDB to figure out exactly what's going on.

New to Linux development. How do I go about doing that?

[Narflex]

Originally Posted by Taddeusz

New to Linux development. How do I go about doing that?

Do some googling...it'd take me quite a while to explain it.

[Taddeusz]

It may be my lack of experience but when I try to attach gdb to sagetv's pid it completely freezes the process and my Placeshifter session goes unresponsive. Is this by design or something I'm doing wrong?

[Taddeusz]

This looks pretty useless to me but maybe it will help. Still trying to figure out how to add symbols to that so. I find it odd that the "file" command says that it is not stripped. GDB says that it can't find symbols for that library though.

[Taddeusz]

Hold on. Learning as I go here. Looks like I finally got the gdb symbols built into the libNativeCore.so. Doing a full rebuild "real quick".

[Narflex]

After you hit the segfault with GDB...then type 'bt' so it'll give you a backtrace for the thread that crashed.

And I contacted Qian (who wrote this code...donkey on the forums) and he's on vacation right now...but said he'll take a look when he gets back.

[Taddeusz]

Originally Posted by Narflex

After you hit the segfault with GDB...then type 'bt' so it'll give you a backtrace for the thread that crashed.

And I contacted Qian (who wrote this code...donkey on the forums) and he's on vacation right now...but said he'll take a look when he gets back.

Ok, I put "ulimit -c unlimited" in the startsagecore script so now it's generating core dumps for me. I did a bt on the core dump and this is what I got:

#0  0x00007f40e45f3267 in __GI_raise (sig=sig@entry=6)
    at ../sysdeps/unix/sysv/linux/raise.c:55
#1  0x00007f40e45f4eca in __GI_abort () at abort.c:89
#2  0x00007f40e3936439 in os::abort(bool) ()
   from /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server/libjvm.so
#3  0x00007f40e3ac55ff in VMError::report_and_die() ()
   from /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server/libjvm.so
#4  0x00007f40e393f266 in JVM_handle_linux_signal ()
   from /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/amd64/server/libjvm.so
#5  <signal handler called>
#6  0x00007f40c8c420f5 in CalTSCRC32 (pData=<optimized out>, 
    len=<optimized out>) at TSCRC32.c:94
#7  0x00007f40c8c3f0a5 in SectionCrcCheck (pSection=0x7f40bc00d240, 
    pSection=0x7f40bc00d240) at SectionData.c:219
#8  UnpackSection (bStartFlag=<optimized out>, pSection=0x7f40bc00d240, 
    pbData=pbData@entry=0x7f407460c464 "", nSize=nSize@entry=184)
    at SectionData.c:170
#9  0x00007f40c8c4495d in UnpackPAT (pTSPacket=0x7f40751c0c80, 
    pTSFilter=0x7f40bc00b9b0) at TSFilter.c:380
#10 ParseTSTable (pTSPacket=0x7f40751c0c80, pTSFilter=0x7f40bc00b9b0)
    at TSFilter.c:860
#11 TSProcess (pTSFilter=0x7f40bc00b9b0, pData=pData@entry=0x7f407460c460 "G@")
    at TSFilter.c:1067
#12 0x00007f40c8c4d7b6 in PushDataTSParser (pTSParser=0x7f40bc00b670, 
    pData=<optimized out>, nSize=nSize@entry=155288) at TSParser.c:982
#13 0x00007f40c8c2b77d in PushDemuxStreamData (Handle=0x7f40bc00b590, 
    pData=pData@entry=0x7f40745f4ae8 "G", nBytes=nBytes@entry=155288, 
    nExpectedBytes=nExpectedBytes@entry=0x7f40bc00afc8) at Demuxer.c:1479
#14 0x00007f40c8c27041 in PushScanStreamData (pScan=0x7f40bc00b0f0, 
    pData=pData@entry=0x7f40745f4ae8 "G", nBytes=nBytes@entry=155288, 
    nExpectedBytes=nExpectedBytes@entry=0x7f40bc00afc8) at ChannelScan.c:472
#15 0x00007f40c8c4e0a4 in PushChannelScanTunerData (lDataLen=155288, 
    pData=<optimized out>, pScanFilter=0x7f40bc00aeb0) at ScanFilter.c:107
#16 ProcessScan (pScanFilter=0x7f40bc00aeb0, pData=<optimized out>, 
    lDataLen=155288) at ScanFilter.c:130
#17 0x00007f40c8e7eaa4 in DTVChannel::splitStream (this=0x7f405c0025b0, 
    buffer=0x7f40745f4ae8, size=155288) at DTVChannel.cp:700
#18 0x00007f40c8e7e9e7 in DTVChannel::pushData (this=0x7f405c0025b0, 
    buffer=0x7f40745f4ae8, size=155288) at DTVChannel.cp:678
#19 0x00007f40c8e7c4ad in HDHRDevice::captureThread (this=0x7f40dc409690)
    at HDHRDevice.cp:676
#20 0x00007f40c8e7c3af in HDHRDevice::captureThreadEntry (arg=0x7f40dc409690)
    at HDHRDevice.cp:643
#21 0x00007f40e41a36aa in start_thread (arg=0x7f40751c1700)
    at pthread_create.c:333
#22 0x00007f40e46c4eed in clone ()
    at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109

[MrD]

I am hitting this issue too, but from a different direction.

When I start a network recording (SageDCT cable card). I get a bunch of ReprocessHook exceptions and the JVM aborts in the same library same location.

The interesting thing is SageDCT starts the recording, but that recording is a poison pill for Sage (i.e. subsequent restarts of sage die trying to process this recording). Once I remove the recording Sage starts up fine.

I am trying to find the code but here is a snippet from the hs_err pid file.

Stack: [0x00007f6b443a6000,0x00007f6b444a7000],  sp=0x00007f6b444a39e8,  free space=1014k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libNativeCore.so+0x300f5]  CalTSCRC32+0x35

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  sage.media.format.MPEGParser.getMediaAVInf0(Ljava/lang/String;JZJ)Ljava/lang/String;+0
j  sage.media.format.FormatParser.getFileDuration(Ljava/io/File;)J+78
j  sage.MediaFile.initializeRecovery(Ljava/io/File;)V+12
j  sage.Wizard.addMediaFileRecovered(Lsage/Airing;Ljava/io/File;)Lsage/MediaFile;+62
j  sage.Seeker.checkDirsForFiles(Z)V+1797
j  sage.Seeker.init()V+191
j  sage.Seeker.prime()Lsage/Seeker;+11
j  sage.SageTV.<init>()V+2069
j  sage.Sage.startup([Ljava/lang/String;)V+2445
j  sage.Sage.main([Ljava/lang/String;)V+10
v  ~StubRoutines::call_stub

[Narflex]

I have an idea what the problem is here if somebody wants to try it out.

In native/ax/Native2.0/NativeCore/TSCRC32.c

I think the 'unsigned long' data types should be changed to be uint32_t (proper includes will need to be added for that). What could be happening is that the unsigned long data type now has a 64 bit value on a 64 bit platform (Whereas it was 32 bits in size on a 32 bit platform). So we start off with this:

unsigned long crc = 0xffffffff;

then when it does this:

crc = (crc << 8) ^ ts_crc32_table[ (crc >> 24) ^ (*p_byte) ];

That first left shift of 8 bits won't cause any to drop off (if unsigned longs are 64 bits now)...and it'll have stuff in the higher 32 bits. That also means then when it does the 24 bit right shift the next time around in the loop (or maybe 2 more times)...that there'll be data beyond the lowest 8 bits, making the value larger than 255 which will then walk off the array that stores the CRC values (and by a very large amount, enough to make Linux unhappy).

[Taddeusz]

Ok, I made that change to the file and it seems to have worked. Did a full scan successfully on 64-bit. When you say proper includes what do you mean? I put "#include <stdint.h>" at the top of the file. Is this ok or is there somewhere else, by convention, that include should go?

[Taddeusz]

Scanning channels works but it doesn't appear that any data is being gotten from the HDHR. I keep getting "halt detected in recording" errors when trying to record.

Keep in mind, this is on 64-bit. The 32-bit one I set up is working fine. There must be deeper differences.

I've tried upgrading libhdhomerun to the latest version without any change.

[Fuzzy]

Not to derail it, but this is part of the reason I think all sources should be moved to the network encoder mechanism. There's no functional reason to have to compile libhdhomerun in 64-bit other than the fact that it's jammed inside an otherwise 64-bit server process. By segregating the sources out to the network encoders, it could easily run in 32-bit, and it wouldn't affect the sagetv core at all.