Comskip is crashing SageTV inside docker

[BobPhoenix]

Here is a link for the full set of SageTV logs if you want to look at more than my samples below: https://www.dropbox.com/s/96p9hhp31b...-logs.zip?dl=0. The full docker log is attached.

I've had this on two different unRAID servers with SageTV Dockers. Both unRAID servers are on version 6.3.5 stable. One is a Dual Processor 32 thread server with lots of VMs and one Docker - SageTV. The other server has NO VMs and only SageTV docker with E31240 Xeon 4 core 8 thread processor with 32GB memory. That server has never had a problem like this before upgrading to the latest version of SageTV and/or docker if that container was updated recently. Current SageTV version: 9.1.5.166. Both of the recordings that were being comskipped at the time appear to have correct markings. The tuners that recorded them were Hauppauge HVR-2255 because they show LG as the tuner. My HVR-2250s show Samsung. Other recordings were going on at the time of the crash and you can see they missed about the last minute of the recording consistent with the last SageTV log time of 22:29 below.

Docker log of SageTV docker (not sure it is relevant but near same time as SageTV crash):

Jun 24 22:17:01 Media2 CRON[31176]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
./startsagecore:  line 55:   117 Killed                  java  -Djava.awt.headless=$HEADLESS $JAVAMEM -XX:+UseAdaptiveSizePolicy  -XX:MaxGCPauseMillis=25 -XX:GCTimeRatio=24 $(if [[ $EUID -eq 0 ]]; then  echo '-XX:ThreadPriorityPolicy=1'; fi) $JAVAOPTS -cp  Sage.jar:JARs/lucene-core-3.6.0.jar:/:$(echo JARs/*.jar | sed 's/   */:/g') sage.Sage 0 0 x "sagetv Sage.properties" 0>&-
Jun 24 23:17:01 Media2 CRON[31192]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)

Last entries in the SageTV Log (no entries after this) 12 minutes after the "Killed" in Docker log so may not be related (but?):

Sat 6/24 22:29:05.217 [Seeker@ce960a9] Enforcing keep at most limits for the files...
Sat 6/24 22:29:05.775 [Seeker@ce960a9] DONE enforcing keep at most limits for the files.
Sat 6/24 22:29:05.790 [Seeker@ce960a9] Seeker clearing unwanted and partial files...
Sat 6/24 22:29:05.824 [Thread-536@1e61a751] CD: ComskipJob.executeComskipLinux: comskip failed with return code = 137
Sat 6/24 22:29:05.829 [Thread-536@1e61a751] java.lang.Exception
Sat 6/24 22:29:05.829 [Thread-536@1e61a751]     at tmiranda.cd.Log.write(Log.java:64)
Sat 6/24 22:29:05.829 [Thread-536@1e61a751]     at tmiranda.cd.ComskipJob.executeComskipLinux(ComskipJob.java:386)
Sat 6/24 22:29:05.829 [Thread-536@1e61a751]     at tmiranda.cd.ComskipJob.executeComskip(ComskipJob.java:173)
Sat 6/24 22:29:05.829 [Thread-536@1e61a751]     at tmiranda.cd.ComskipJob.executeUsingDefault(ComskipJob.java:187)
Sat 6/24 22:29:05.829 [Thread-536@1e61a751]     at tmiranda.cd.ComskipJob.run(ComskipJob.java:130)
Sat 6/24 22:29:05.829 [Thread-536@1e61a751]     at java.lang.Thread.run(Thread.java:745)
Sat 6/24 22:29:05.831 [Thread-536@1e61a751] CD: ComskipJob.updateRatio: Old, Current and new Ratios 0.09118217:0.0042235656:0.08248631
Sat 6/24 22:29:06.490 [Seeker@ce960a9] DONE clearing unwanted and partial files.
Sat 6/24 22:29:07.100 [Thread-528@50fc8f6e] CD: ComskipJob.executeComskipLinux: comskip failed with return code = 137
Sat 6/24 22:29:07.125 [Thread-528@50fc8f6e] java.lang.Exception

Feel free to move this to a more appropriate thread but since it is running on Docker instance of SageTV and Comskip is built into the docker I figured this was as good a place as any.

Added unRAID log because it shows OOM errors from java.
Added the comskip log of recording that caused the crash. Way I read that log it is an audio problem. Hopefully this might help make comskip work better or at least change to not crash the whole docker.

[Cooper]

This seems to be the same issue I am having on Windows 10 but in my case I get the Blue screen and reboots. Happens if something is recording or watching a recorded show or live TV. I will try to get logs as well next time but now I only allow Comskip to run from Midnight to 3 AM and no crashes.

[stuckless]

Do you know if this was recent? I updated comskip a few weeks back in the docker container. I checked the comskip github site and I'm not seeing an reports like ones being reported here.

[BobPhoenix]

Originally Posted by stuckless

Do you know if this was recent? I updated comskip a few weeks back in the docker container. I checked the comskip github site and I'm not seeing an reports like ones being reported here.

The one I posted above happened last night. Didn't notice it until earlier today. The other server wasn't too long ago but I am unable to find the logs to tell me exactly when it happened. But I will keep looking for them because I thought I had saved them. I did post about it in the Commercial Detector thread after it happened and that was on the 14th of June. Here is the post: https://forums.sagetv.com/forums/sho...postcount=1034

Also Comskip crashing on some videos isn't the end of the world but it it taking out the rest of SageTV with it. If something could be done to prevent that I would be satisfied. I may try something myself and it wouldn't be pretty. My brute force approach would be to communicate with a job running in a VM that would launch comskip from there rather than within the SageTV docker. But as I said that is the brute force approach and I'm sure there are better ways.

[stuckless]

I just checked my logs on my server, I don't have any comskip errors in mine (which is good for me ), so I wonder if it's a particular file type or something else.

[BobPhoenix]

Originally Posted by stuckless

I just checked my logs on my server, I don't have any comskip errors in mine (which is good for me ), so I wonder if it's a particular file type or something else.

I imagine it is the adds that get inserted in the video that are causing the problems. I'll see if I can find a comskip.ini setting that will make a difference. And it is going to take a while as it has only happened every other week so far.

[BobPhoenix]

Haven't figured this out yet. Not 100% sure it is comskip causing the crash or not.

For instance on my VM server it just crashed again today I just restarted the Docker and these are my JVM specs in SageTV Info:
JVM Heap Size (Used/Total/Max): 5850MB/13149MB/15032MB

I never thought when I increased the heap size on the docker I would ever see SageTV using all of it.

For the docker container I limit by this parameter:
--memory="16G"

And SageTV java memory is 14336

If it isn't comskip (although it is always running when the crash occurs - but it could be just coincidence I suppose) - any thoughts on what it could be?

Plugins:

• Anti-Favorites for Dummies
• Commercial Detector UI
• Complementary Metadata Tools
• Comskip Playback
• CPU Meter
• Encoder Names
• Java Heap Monitor
• Move Imported TV shows to SageTV recordings
• Restart SageTV from Setup menu
• Season Premiere borders for Guide
• Signal Strength on OSD
• VideoOut Tools for Extenders
• Batch Meta Tools
• Commercial Detector
• Jetty Web Server
• Phoenix Core services (with update available showing so will update just in case
• sageTV addons License Server
• SageTV Groovy Tools
• SageTV Web Interface (with update available so will update this too)

There are several that I could get rid of but they really are not likely to be causing this as they have been installed for a long time and the only thing new has been SageTV itself and the Docker container.

[stuckless]

have you tried running with a much smaller java memory size? In looking at the unraid system log, it appears that the system is killing java because it's getting an OOM. Are you seeing OOM errrors in the SageTV log?

Java will use everything you throw at it. If you give it 16g of memory it will use 16g of memory...eventually.

I would set your JVM to 4G (which I think is extremely large) and leave your other settings.

I'm wondering if because your java is set 14G andyou docker is set 16G then that's not much room to run other things, like comskip, ffmpeg, etc.

[BobPhoenix]

Originally Posted by stuckless

have you tried running with a much smaller java memory size? In looking at the unraid system log, it appears that the system is killing java because it's getting an OOM. Are you seeing OOM errrors in the SageTV log?

Java will use everything you throw at it. If you give it 16g of memory it will use 16g of memory...eventually.

I would set your JVM to 4G (which I think is extremely large) and leave your other settings.

I'm wondering if because your java is set 14G andyou docker is set 16G then that's not much room to run other things, like comskip, ffmpeg, etc.

OK. That what I had it set at originally. Didn't think about leaving room for comskip & ffmpeg. Will reduce both servers back to that. I did get one out of memory error in SageTV on 6/24 when it crashed previously. Specifically the error was "General System Message" - Java Heap has exceeded warning level. The message comes from the Java Heap Monitor plugin which has the following settings: "Warning Threshhold%" - 80, "Error Threshhold%" - 95. I had the docker set to 6G and SageTV set to 4G. I will leave both servers dockers at 16G like currently but I will reduce SageTV limit to 6G since when I restarted one server the memory usage was at 5G. 10G should be enough for ffmpeg and comskip.

[stuckless]

Originally Posted by BobPhoenix

OK. That what I had it set at originally. Didn't think about leaving room for comskip & ffmpeg. Will reduce both servers back to that. I did get one out of memory error in SageTV on 6/24 when it crashed previously. Specifically the error was "General System Message" - Java Heap has exceeded warning level. The message comes from the Java Heap Monitor plugin which has the following settings: "Warning Threshhold%" - 80, "Error Threshhold%" - 95. I had the docker set to 6G and SageTV set to 4G. I will leave both servers dockers at 16G like currently but I will reduce SageTV limit to 6G since when I restarted one server the memory usage was at 5G. 10G should be enough for ffmpeg and comskip.

yeah, the heap warning plugin... not sure what it does, but that can likely be nothing to worry about. Java will just keep using memory and avoid garbarge collecting until it has to. So, it can grow the heap very quickly and result in those kinds of "warnings", but an OOM error is really a case where the JVM can't allocate enough room for a new object AFTER it tries all possible means to garbage collect. But your errors seems to be coming from unRAID (linux os) and not Java itself.