SageTV V9 core dumping when I stop recording with HD PVR

[turak]

I'm setting up a HD PVR to record from my directv genie. I am able to tune live TV just fine with it, but the sagetv server process crashes when I hit stop.

*** stack smashing detected ***: java terminated
./startsagecore: line 48: 12359 Aborted (core dumped) java -Djava.awt.headless=$HEADLESS $JAVAMEM -XX:+UseAdaptiveSizePolicy -XX:MaxGCPauseMillis=25 -XX:GCTimeRatio=24 -XX:ThreadPriorityPolicy=1 $JAVAOPTS -cp Sage.jar:.:/:$(echo JARs/*.jar | sed 's/ **/:/g') sage.Sage 0 0 x "sagetv Sage.properties" 0>&-



I'm running Ubuntu 14.04 64bit.

[turak]

Here it is with a bunch of debugging turned on. As always, it core dumps when I hit stop on a HD PVR recording:


Sat 11/7 20:34:09.615 [Seeker@128e9671] MARK 1 currRecord=null enc=HDHomeRun 1039a37a Tuner 0 clients=[] ir=true
Sat 11/7 20:34:09.615 [Seeker@128e9671] defaultRecord=A[20963967,20963611,"The Green Hornet",92064@1107.20:30,30,T]
Sat 11/7 20:34:09.615 [EventRouter-0023a5000596@1378810a] "REM Now, check the menu history to see if we can return to where playback started."
Sat 11/7 20:34:09.615 [Seeker@128e9671] Seeker in AUTOMATIC mode nextRecord=A[20923060,20922657,"The Walking Dead",32725@1108.15:00,60,T] nextTTA=66350385
Sat 11/7 20:34:09.615 [Seeker@128e9671] newRecord=null
Sat 11/7 20:34:09.615 [Seeker@128e9671] NOTHING TO RECORD FOR NOW...
Sat 11/7 20:34:09.615 [Seeker@128e9671] MARK 1 currRecord=A[21378000,21312574,"Ghostbusters",14873@1107.20:00,135,T] enc=Hauppauge HD PVR 00A82A46 clients=[] ir=true
Sat 11/7 20:34:09.616 [Seeker@128e9671] Seeker in AUTOMATIC mode nextRecord=A[21418349,21417840,"Bubble Guppies",19211@1107.21:30,30,T] nextTTA=3350385
Sat 11/7 20:34:09.616 [Seeker@128e9671] newRecord=null
Sat 11/7 20:34:09.616 [Seeker@128e9671] Change in record, logging recorded data.
Sat 11/7 20:34:09.616 [Seeker@128e9671] Seeker.endRecord(Sat 11/7 20:34:09.615) currRecord=A[21378000,21312574,"Ghostbusters",14873@1107.20:00,135,T] readySwitch=true
Sat 11/7 20:34:09.616 [Seeker@128e9671] NOTHING TO RECORD FOR NOW...
Sat 11/7 20:34:09.616 [Seeker@128e9671] stopEncoding for V4L capture device
Sat 11/7 20:34:09.616 [Seeker@128e9671] Waiting for V4L capture thread to terminate
Sat 11/7 20:34:09.616 [EventRouter-0023a5000596@1378810a] "REM Show the various OSD portions."
Sat 11/7 20:34:09.616 [EventRouter-0023a5000596@1378810a] "REM Show the various OSD portions."
Sat 11/7 20:34:09.616 [EventRouter-0023a5000596@1378810a] "REM Show the various OSD portions."
Sat 11/7 20:34:09.616 [EventRouter-0023a5000596@1378810a] "REM Show the various OSD portions."
Sat 11/7 20:34:09.616 [EventRouter-0023a5000596@1378810a] "REM Show the various OSD portions."
Sat 11/7 20:34:09.616 [EventRouter-0023a5000596@1378810a] "REM Show the various OSD portions."
Sat 11/7 20:34:09.616 [EventRouter-0023a5000596@1378810a] "REM Show the various OSD portions."
Sat 11/7 20:34:09.616 [EventRouter-0023a5000596@1378810a] "REM Show the various OSD portions."
Sat 11/7 20:34:09.616 [EventRouter-0023a5000596@1378810a] "REM Show the various OSD portions."
Sat 11/7 20:34:09.617 [EventRouter-0023a5000596@1378810a] "REM Show the various OSD portions."
Sat 11/7 20:34:09.617 [EventRouter-0023a5000596@1378810a] "REM Show the various OSD portions."
Sat 11/7 20:34:09.617 [EventRouter-0023a5000596@1378810a] "REM Show the various OSD portions."
Sat 11/7 20:34:09.617 [EventRouter-0023a5000596@1378810a] "REM Show the various OSD portions."
Sat 11/7 20:34:09.617 [EventRouter-0023a5000596@1378810a] "REM Show the various OSD portions."
Sat 11/7 20:34:09.617 [EventRouter-0023a5000596@1378810a] "REM Show the various OSD portions."
Sat 11/7 20:34:09.617 [EventRouter-0023a5000596@1378810a] "REM Show the various OSD portions."
Sat 11/7 20:34:09.617 [EventRouter-0023a5000596@1378810a] "REM Show the various OSD portions."
Sat 11/7 20:34:09.617 [EventRouter-0023a5000596@1378810a] "REM Show the various OSD portions."
Sat 11/7 20:34:09.617 [ActiveRender-0023a5000596@3f56cf6a] "REM FX for the lower OSD portion."
Sat 11/7 20:34:09.617 [ActiveRender-0023a5000596@3f56cf6a] "REM FX to show/hide OSD messages"
Sat 11/7 20:34:09.617 [ActiveRender-0023a5000596@3f56cf6a] "REM FX to show/hide OSD messages"
Sat 11/7 20:34:09.618 [Scheduler@cf76937] # Airs=617
Sat 11/7 20:34:09.625 [V4L-Encoder-video0@6eee4260] V4LCap null 394936320
Sat 11/7 20:34:09.625 [V4L-Encoder-video0@6eee4260] V4L: closeEncoding
Sat 11/7 20:34:09.625 [V4L-Encoder-video0@6eee4260] V4L: join capture thread
Sat 11/7 20:34:09.627 [EventRouter-0023a5000596@1378810a] "REM Global Back listener - set up reverse menu animation & check for return to custom STV."
Sat 11/7 20:34:09.627 [EventRouter-0023a5000596@1378810a] "REM Set up reverse menu animation & let the default Back action proceed."
Sat 11/7 20:34:09.628 [EventRouter-0023a5000596@1378810a] setUI(sage.PseudoMenu@2a88f251[SageRecordings w/ Optional Preview & AutoCategories]) histIdx=3 uiHistory=[sage.PseudoMenu@3cce57cf[Main Menu], sage.PseudoMenu@2537e5de[SageRecordings w/ Optional Preview & AutoCategories], sage.PseudoMenu@1bb164d0[Main Menu], sage.PseudoMenu@2a88f251[SageRecordings w/ Optional Preview & AutoCategories], sage.PseudoMenu@72f489fd[MediaPlayer OSD]] redo=true performingActivation=false
Sat 11/7 20:34:09.628 [EventRouter-0023a5000596@1378810a] VideoFrame got registration of a subtitle UI component: null
*** stack smashing detected ***: java terminated
./startsagecore: line 48: 15141 Aborted (core dumped) java -Djava.awt.headless=$HEADLESS $JAVAMEM -XX:+UseAdaptiveSizePolicy -XX:MaxGCPauseMillis=25 -XX:GCTimeRatio=24 -XX:ThreadPriorityPolicy=1 $JAVAOPTS -cp Sage.jar:.:/:$(echo JARs/*.jar | sed 's/ **/:/g') sage.Sage 0 0 x "sagetv Sage.properties" 0>&-

[turak]

I switched to a 32bit build of SageTV (V7 install with V9 Sage.jar dropped in) and it's not crashing anymore. I'll keep my 64bit version in case anyone wants to troubleshoot this and needs me to test for them.

[oldsage]

I get the same error for HD PVR.
(Also the video preview when adding tuner does not work)

Last relevant entry in log seems to be the "V4L: join capture thread".
(at line 466 in sage_IVTVCaptureDevice.c)

sagetv_0.txt

Thu 2/4 14:27:40.634 [V4L-Encoder-video0@3c9283f8] V4L: closeEncoding
Thu 2/4 14:27:40.634 [V4L-Encoder-video0@3c9283f8] V4L: join capture thread
Thu 2/4 14:27:40.634 [Scheduler@7abab0e1] Hauppauge HD PVR 00A2531F
Thu 2/4 14:27:40.634 [Scheduler@7abab0e1] []
Thu 2/4 14:27:40.644 [Scheduler@7abab0e1] HDHomeRun 10192689 Tuner 1
Thu 2/4 14:27:40.644 [Scheduler@7abab0e1] []
Thu 2/4 14:27:40.644 [Scheduler@7abab0e1] HDHomeRun 10192689 Tuner 0
Thu 2/4 14:27:40.644 [Scheduler@7abab0e1] []
Thu 2/4 14:27:40.644 [Scheduler@7abab0e1] Total Schedule eval time=15 msec

I tried to add a few "-g" and compile the code, so I could enable debugging.
As far as I got it seems to die on the pthread_join().
If you comment what calls that out, then it will not core dump.

stacktrace

Program received signal SIGABRT, Aborted.
0x00007fdb0f8e9cc9 in __GI_raise (sig=sig@entry=6)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
56	../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  0x00007fdb0f8e9cc9 in __GI_raise (sig=sig@entry=6)
    at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007fdb0f8ed0d8 in __GI_abort () at abort.c:89
#2  0x00007fdb0f1ef505 in os::abort(bool) ()
   from /usr/lib/jvm/java-8-oracle/jre/lib/amd64/server/libjvm.so
#3  0x00007fdb0f38dbf3 in VMError::report_and_die() ()
   from /usr/lib/jvm/java-8-oracle/jre/lib/amd64/server/libjvm.so
#4  0x00007fdb0f1f4d2f in JVM_handle_linux_signal ()
   from /usr/lib/jvm/java-8-oracle/jre/lib/amd64/server/libjvm.so
#5  0x00007fdb0f1eb4c3 in signalHandler(int, siginfo*, void*) ()
   from /usr/lib/jvm/java-8-oracle/jre/lib/amd64/server/libjvm.so
#6  <signal handler called>
#7  0x00007fdabfde4c04 in Java_sage_IVTVCaptureDevice_closeEncoding0 ()
   from /home/john/src/sagetv/build/ubuntuserver/opt/sagetv/server/libIVTVCapture.so
Cannot access memory at address 0x7fda00000008

Comment out line 466 as:

/* ACL_ThreadJoin(x->capThread); */

will let it continue, but not a real fix.

Maybe a bad pointer somewhere in there or overwritten memory.
Hope this info helps someone to fix it.

(Maybe a quick lazy fix to detach that thread )

[troll5501]

Good timing on your post...I was just looking into this today. See my other post here: http://forums.sagetv.com/forums/show...2&postcount=30.

I believe I have a fix for this, although it may need a code review by someone who is more versed in C than I am. I will try to get more details posted tomorrow, but as I type this I have 4 HD-PVRs recording at the same time on CentOS 7 64-bit Sage V9. I can stop and start recordings without any crashes.

Maybe you guys could help to test the changes...

[wnjj]

I think I found the bug here. In the ACL_ThreadJoin function in thread_util.c, the pthread_join call is expecting a void** but the function is passing an &int. Int and a pointer and not interchangeable in 64-bit code. 'retval' needs to be declared as 'void *' and ACL_ThreadJoin needs to return void*.

Edit: Was writing this while troll5501 posted. Did you come to the same conclusion?

[stuckless]

Originally Posted by troll5501

Good timing on your post...I was just looking into this today. See my other post here: http://forums.sagetv.com/forums/show...2&postcount=30.

I believe I have a fix for this, although it may need a code review by someone who is more versed in C than I am. I will try to get more details posted tomorrow, but as I type this I have 4 HD-PVRs recording at the same time on CentOS 7 64-bit Sage V9. I can stop and start recordings without any crashes.

Maybe you guys could help to test the changes...

Sounds like you are testing the changes and it's working (which was better than before ). Maybe submit a pull request and then it can be reviewed.

[oldsage]

Good to hear that this can be fixed. And maybe filtering out compiler warnings can catch some other similar problems that may be in there.

Maybe I will be able to use Linux after all.
I had thought HD PVR was in more popular use, but maybe only in windows.

Now if I can only find out how to configure USB-UIRT and Lirc.
I think from another thread mentions the one plugin also has a problem.
But I can't seem to find a thread with any of the basic setup of it.

I will dig around ubuntu and MythTv forums and see what I find.
If anyone knows of any postings on the setup on this forum I would appreciate.
But will it be able to work on 64 bit?

[troll5501]

Originally Posted by wnjj

Edit: Was writing this while troll5501 posted. Did you come to the same conclusion?

Yep, basically the same conclusion here. I also made a few other small changes to clean up some compiler warnings and potential issues. In addition, there was another crash occurring when using tuner plugins on Linux 64-bit and I think I've got that fixed too.

Per stuckless's recommendation, I will be creating a pull request shortly. Then we'll see if anyone has additional comments or recommendations. C coding is not my area of expertise but I've done some occasionally over the years. I just hope the changes don't introduce any new problems on 32-bit platforms.

[troll5501]

Well that was fast! The changes have been merged into the project:

https://github.com/google/sagetv/pull/75

[oldsage]

Great! That fixes the crash when adding an ir plugin too!

So I got channel changes working but sometimes get a black screen.
Maybe my script needs some more fine tuning sending the ir codes.

By the way, there is another thread_util.c in the native DVBCapture directory also.

[troll5501]

Originally Posted by oldsage

Great! That fixes the crash when adding an ir plugin too!

Glad to hear it's working for you! It's been working well here too...I've had two extenders playing live TV for 24 hours and no crashes. Although I'm seeing some occasional recording halts that I didn't have with the same HD-PVRs in Windows SageTV 7. Trying to track that down now...

Originally Posted by oldsage

By the way, there is another thread_util.c in the native DVBCapture directory also.

Thanks for pointing this out. It appears to be identical code, so maybe I will submit a fix for it too. I found another forum thread that discusses a JVM crash in closeEncoding0 when using DVBCapture. Maybe someone over there can test the changes since I don't have the necessary hardware.

[oldsage]

Originally Posted by troll5501

Glad to hear it's working for you! It's been working well here too...I've had two extenders playing live TV for 24 hours and no crashes. Although I'm seeing some occasional recording halts that I didn't have with the same HD-PVRs in Windows SageTV 7. Trying to track that down now...

Unfortunately I get a black screen and then my extenders turn off after watching a recording for a while or watching live tv.

I'll have to recheck the logs, but it seems like some other thread is shutting down the stream and then the extender is getting an AsynchronousCloseException or something like that.

Then the extender powers off and I have to power it back on.

But I think I had the error with hd-pvr as well as hdhomerun recordings.

I have the hd100 extenders.

Originally Posted by troll5501

Thanks for pointing this out. It appears to be identical code, so maybe I will submit a fix for it too. I found another forum thread that discusses a JVM crash in closeEncoding0 when using DVBCapture. Maybe someone over there can test the changes since I don't have the necessary hardware.

Sounds like similar problem, so will probably fix it.

[troll5501]

I'm using the HD200 extenders and they seem to be fine, but I do have several HD100 that I haven't used for years. Maybe I'll hook one up to see if that issue occurs on my system.

[oldsage]

Well this info should probably end up in a different thread, but to record for now in case anyone else sees the same problem.

This seems to be what causes the problem:

Sat 2/6 13:55:01.771 [Pusher@62b43c18] Error with MiniPlayer, closing UI: java.io.EOFException
Sat 2/6 13:55:01.772 [Pusher@62b43c18] java.io.EOFException
Sat 2/6 13:55:01.772 [Pusher@62b43c18]  at sage.BufferedFileChannel.read(BufferedFileChannel.java:183)
Sat 2/6 13:55:01.772 [Pusher@62b43c18]  at sage.FastMpeg2Reader.read(FastMpeg2Reader.java:987)
Sat 2/6 13:55:01.772 [Pusher@62b43c18]  at sage.FastMpeg2Reader.transfer(FastMpeg2Reader.java:880)
Sat 2/6 13:55:01.772 [Pusher@62b43c18]  at sage.MiniPlayer.pushBuffer0(MiniPlayer.java:3339)
Sat 2/6 13:55:01.772 [Pusher@62b43c18]  at sage.MiniPlayer$1.run(MiniPlayer.java:2076)
Sat 2/6 13:55:01.772 [Pusher@62b43c18]  at java.lang.Thread.run(Thread.java:745)
Sat 2/6 13:55:01.773 [Pusher@62b43c18] pushBuffer call failed; terminating push loop

which causes screen to go black, then a minute or so later the extender goes down, and this error in log:

Sat 2/6 13:55:33.162 [MiniUIClientReceiver@7618012d] Exception in the MiniUIClientReceiver of: java.nio.channels.AsynchronousCloseException
Sat 2/6 13:55:33.162 [MiniUIClientReceiver@7618012d] java.nio.channels.AsynchronousCloseException
Sat 2/6 13:55:33.163 [MiniUIClientReceiver@7618012d]    at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:205)
Sat 2/6 13:55:33.163 [MiniUIClientReceiver@7618012d]    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:407)
Sat 2/6 13:55:33.163 [MiniUIClientReceiver@7618012d]    at sage.MiniClientSageRenderer$MiniUIClientReceiver.run(MiniClientSageRenderer.java:8210)
Sat 2/6 13:55:34.717 [KillUIMgr@428a849f] Killed RootPanel
Sat 2/6 13:55:34.718 [KillUIMgr@428a849f] Saving properties file to clients/001e40b5a153.properties
Sat 2/6 13:55:34.728 [KillUIMgr@428a849f] Done writing out the data to the properties file
Sat 2/6 13:55:34.728 [KillUIMgr@428a849f] Disposed Window

[turak]

Originally Posted by troll5501

Although I'm seeing some occasional recording halts that I didn't have with the same HD-PVRs in Windows SageTV 7. Trying to track that down now...

I've been getting several "halt in recording" messages every day since switching to Linux. I rarely notice anything missing when I play back the recordings. I've been running 32bit though due to the server crashing when I'd stop recording and the irtunerplugin crashes.

[stuckless]

Originally Posted by turak

I've been getting several "halt in recording" messages every day since switching to Linux. I rarely notice anything missing when I play back the recordings. I've been running 32bit though due to the server crashing when I'd stop recording and the irtunerplugin crashes.

If you are seeing halt detected with HDPVR recordings, on linux, I'll put money on the powersupply. I've replaced my HDPVR powersupplies about 3-4 times (each) in 7 years (The last time was about a year ago, so I suspect I only have a couple months). Next time it happens, I'll be moving to solution whereby the HDPVR will use the PSU from the computer (someone posted that solution a while ago).

[troll5501]

Originally Posted by stuckless

If you are seeing halt detected with HDPVR recordings, on linux, I'll put money on the powersupply. I've replaced my HDPVR powersupplies about 3-4 times (each) in 7 years (The last time was about a year ago, so I suspect I only have a couple months). Next time it happens, I'll be moving to solution whereby the HDPVR will use the PSU from the computer (someone posted that solution a while ago).

Hmmm, I'm not so sure, and I've had plenty of the HD-PVR power bricks go bad over the years. I had this halt issue years ago when I was testing V7 on Linux and the HD-PVR drivers and firmware were not very mature yet. Moving to Windows XP on the same hardware resolved it.

I later virtualized everything and setup PCI passthrough for the USB bus and that VM has been running fine for years with V7 on XP and with 4 HD-PVRs. Last week I re-mapped the USB bus to my Linux VM without touching any hardware or cables, and now I have the occasional halts. I was hoping there were some improvements in the drivers since 2011 but I don't know.

I'm thinking there are still certain scenarios that the Linux driver doesn't handle well, but the Windows driver being from Hauppauge is more robust. Last night I enabled hdpvr_debug in the Linux module and it appears that suddenly the incoming data stream just stops completely...often when I'm doing something in the UI (FF/REW or exiting from the Weather screen) that creates some load on the system, but occasionally it happens when I'm not even near the TV.

I found some threads in other forums discussing this and people were seeing the same type of hdpvr module debug output and sudden loss of incoming data. So I'm not sure if there is an easy fix for this if it's a driver issue.

[stuckless]

Originally Posted by troll5501

Hmmm, I'm not so sure, and I've had plenty of the HD-PVR power bricks go bad over the years. I had this halt issue years ago when I was testing V7 on Linux and the HD-PVR drivers and firmware were not very mature yet. Moving to Windows XP on the same hardware resolved it.

I later virtualized everything and setup PCI passthrough for the USB bus and that VM has been running fine for years with V7 on XP and with 4 HD-PVRs. Last week I re-mapped the USB bus to my Linux VM without touching any hardware or cables, and now I have the occasional halts. I was hoping there were some improvements in the drivers since 2011 but I don't know.

I'm thinking there are still certain scenarios that the Linux driver doesn't handle well, but the Windows driver being from Hauppauge is more robust. Last night I enabled hdpvr_debug in the Linux module and it appears that suddenly the incoming data stream just stops completely...often when I'm doing something in the UI (FF/REW or exiting from the Weather screen) that creates some load on the system, but occasionally it happens when I'm not even near the TV.

I found some threads in other forums discussing this and people were seeing the same type of hdpvr module debug output and sudden loss of incoming data. So I'm not sure if there is an easy fix for this if it's a driver issue.

It could be a driver issue... I can only go on my own experience of running 2 HDPVRs on linux for 7 years now. If I see more than 2 halts in a week, I replace the power brick and then I'll never see another halt for about 1-2 years. I record everything .ts streams using h264 BEST quality. Other than the halts caused by the power brick, the HDPVR on linux has been rock solid for me... recording stuff daily. (Still running v7 native 32bit libraries on my server, so I haven't moved my server to the full open source version yet, other than the Sage.jar)

[troll5501]

Originally Posted by stuckless

It could be a driver issue... I can only go on my own experience of running 2 HDPVRs on linux for 7 years now. If I see more than 2 halts in a week, I replace the power brick and then I'll never see another halt for about 1-2 years. I record everything .ts streams using h264 BEST quality. Other than the halts caused by the power brick, the HDPVR on linux has been rock solid for me... recording stuff daily. (Still running v7 native 32bit libraries on my server, so I haven't moved my server to the full open source version yet, other than the Sage.jar)

Are you using SPDIF audio or RCA? I've read that some people have more success with the analog audio input.

I've been keeping track of the halts I've seen this week, and they've occurred on all 4 of my HD-PVRs across 2 USB busses (integrated and PCIe card). With Windows, I could easily go months without seeing a halt, and when I did get one it was usually because the recording directory was full.