SageTV Community  

Go Back   SageTV Community > SageTV Products > SageTV Linux > SageTV for unRAID/Docker

Notices

SageTV for unRAID/Docker Discussion related to SageTV for unRAID/Docker. Questions, issues, problems, suggestions, etc. relating to SageTV for unRAID/Docker should be posted here.

Reply
 
Thread Tools Search this Thread Display Modes
  #1  
Old 06-25-2017, 10:12 AM
BobPhoenix BobPhoenix is offline
SageTVaholic
 
Join Date: Oct 2004
Posts: 3,152
Comskip is crashing SageTV inside docker

Here is a link for the full set of SageTV logs if you want to look at more than my samples below: https://www.dropbox.com/s/96p9hhp31b...-logs.zip?dl=0. The full docker log is attached.

I've had this on two different unRAID servers with SageTV Dockers. Both unRAID servers are on version 6.3.5 stable. One is a Dual Processor 32 thread server with lots of VMs and one Docker - SageTV. The other server has NO VMs and only SageTV docker with E31240 Xeon 4 core 8 thread processor with 32GB memory. That server has never had a problem like this before upgrading to the latest version of SageTV and/or docker if that container was updated recently. Current SageTV version: 9.1.5.166. Both of the recordings that were being comskipped at the time appear to have correct markings. The tuners that recorded them were Hauppauge HVR-2255 because they show LG as the tuner. My HVR-2250s show Samsung. Other recordings were going on at the time of the crash and you can see they missed about the last minute of the recording consistent with the last SageTV log time of 22:29 below.

Docker log of SageTV docker (not sure it is relevant but near same time as SageTV crash):
Code:
Jun 24 22:17:01 Media2 CRON[31176]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
./startsagecore:  line 55:   117 Killed                  java  -Djava.awt.headless=$HEADLESS $JAVAMEM -XX:+UseAdaptiveSizePolicy  -XX:MaxGCPauseMillis=25 -XX:GCTimeRatio=24 $(if [[ $EUID -eq 0 ]]; then  echo '-XX:ThreadPriorityPolicy=1'; fi) $JAVAOPTS -cp  Sage.jar:JARs/lucene-core-3.6.0.jar:/:$(echo JARs/*.jar | sed 's/   */:/g') sage.Sage 0 0 x "sagetv Sage.properties" 0>&-
Jun 24 23:17:01 Media2 CRON[31192]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Last entries in the SageTV Log (no entries after this) 12 minutes after the "Killed" in Docker log so may not be related (but?):
Code:
Sat 6/24 22:29:05.217 [Seeker@ce960a9] Enforcing keep at most limits for the files...
Sat 6/24 22:29:05.775 [Seeker@ce960a9] DONE enforcing keep at most limits for the files.
Sat 6/24 22:29:05.790 [Seeker@ce960a9] Seeker clearing unwanted and partial files...
Sat 6/24 22:29:05.824 [Thread-536@1e61a751] CD: ComskipJob.executeComskipLinux: comskip failed with return code = 137
Sat 6/24 22:29:05.829 [Thread-536@1e61a751] java.lang.Exception
Sat 6/24 22:29:05.829 [Thread-536@1e61a751]     at tmiranda.cd.Log.write(Log.java:64)
Sat 6/24 22:29:05.829 [Thread-536@1e61a751]     at tmiranda.cd.ComskipJob.executeComskipLinux(ComskipJob.java:386)
Sat 6/24 22:29:05.829 [Thread-536@1e61a751]     at tmiranda.cd.ComskipJob.executeComskip(ComskipJob.java:173)
Sat 6/24 22:29:05.829 [Thread-536@1e61a751]     at tmiranda.cd.ComskipJob.executeUsingDefault(ComskipJob.java:187)
Sat 6/24 22:29:05.829 [Thread-536@1e61a751]     at tmiranda.cd.ComskipJob.run(ComskipJob.java:130)
Sat 6/24 22:29:05.829 [Thread-536@1e61a751]     at java.lang.Thread.run(Thread.java:745)
Sat 6/24 22:29:05.831 [Thread-536@1e61a751] CD: ComskipJob.updateRatio: Old, Current and new Ratios 0.09118217:0.0042235656:0.08248631
Sat 6/24 22:29:06.490 [Seeker@ce960a9] DONE clearing unwanted and partial files.
Sat 6/24 22:29:07.100 [Thread-528@50fc8f6e] CD: ComskipJob.executeComskipLinux: comskip failed with return code = 137
Sat 6/24 22:29:07.125 [Thread-528@50fc8f6e] java.lang.Exception
Feel free to move this to a more appropriate thread but since it is running on Docker instance of SageTV and Comskip is built into the docker I figured this was as good a place as any.

Added unRAID log because it shows OOM errors from java.
Added the comskip log of recording that caused the crash. Way I read that log it is an audio problem. Hopefully this might help make comskip work better or at least change to not crash the whole docker.
Attached Files
File Type: txt Log for stuckless-sagetv-server-java8.txt (35.7 KB, 86 views)
File Type: txt syslog.txt (514.6 KB, 231 views)
File Type: txt 13Newsat9onFox17-1612679-0.log.txt (513.0 KB, 97 views)
__________________
"Keep your goals away from the trolls"

Last edited by BobPhoenix; 06-25-2017 at 12:23 PM. Reason: Added ComSkip log on video that caused the problem
Reply With Quote
  #2  
Old 06-25-2017, 01:40 PM
Cooper Cooper is offline
Sage User
 
Join Date: Oct 2008
Location: Washington
Posts: 35
This seems to be the same issue I am having on Windows 10 but in my case I get the Blue screen and reboots. Happens if something is recording or watching a recorded show or live TV. I will try to get logs as well next time but now I only allow Comskip to run from Midnight to 3 AM and no crashes.
Reply With Quote
  #3  
Old 06-25-2017, 04:17 PM
stuckless's Avatar
stuckless stuckless is offline
SageTVaholic
 
Join Date: Oct 2007
Location: London, Ontario, Canada
Posts: 9,711
Do you know if this was recent? I updated comskip a few weeks back in the docker container. I checked the comskip github site and I'm not seeing an reports like ones being reported here.
Reply With Quote
  #4  
Old 06-25-2017, 04:45 PM
BobPhoenix BobPhoenix is offline
SageTVaholic
 
Join Date: Oct 2004
Posts: 3,152
Quote:
Originally Posted by stuckless View Post
Do you know if this was recent? I updated comskip a few weeks back in the docker container. I checked the comskip github site and I'm not seeing an reports like ones being reported here.
The one I posted above happened last night. Didn't notice it until earlier today. The other server wasn't too long ago but I am unable to find the logs to tell me exactly when it happened. But I will keep looking for them because I thought I had saved them. I did post about it in the Commercial Detector thread after it happened and that was on the 14th of June. Here is the post: https://forums.sagetv.com/forums/sho...postcount=1034

Also Comskip crashing on some videos isn't the end of the world but it it taking out the rest of SageTV with it. If something could be done to prevent that I would be satisfied. I may try something myself and it wouldn't be pretty. My brute force approach would be to communicate with a job running in a VM that would launch comskip from there rather than within the SageTV docker. But as I said that is the brute force approach and I'm sure there are better ways.
__________________
"Keep your goals away from the trolls"

Last edited by BobPhoenix; 06-25-2017 at 05:04 PM.
Reply With Quote
  #5  
Old 06-26-2017, 04:58 AM
stuckless's Avatar
stuckless stuckless is offline
SageTVaholic
 
Join Date: Oct 2007
Location: London, Ontario, Canada
Posts: 9,711
I just checked my logs on my server, I don't have any comskip errors in mine (which is good for me ), so I wonder if it's a particular file type or something else.
Reply With Quote
  #6  
Old 06-26-2017, 04:31 PM
BobPhoenix BobPhoenix is offline
SageTVaholic
 
Join Date: Oct 2004
Posts: 3,152
Quote:
Originally Posted by stuckless View Post
I just checked my logs on my server, I don't have any comskip errors in mine (which is good for me ), so I wonder if it's a particular file type or something else.
I imagine it is the adds that get inserted in the video that are causing the problems. I'll see if I can find a comskip.ini setting that will make a difference. And it is going to take a while as it has only happened every other week so far.
__________________
"Keep your goals away from the trolls"
Reply With Quote
  #7  
Old 07-01-2017, 11:08 PM
BobPhoenix BobPhoenix is offline
SageTVaholic
 
Join Date: Oct 2004
Posts: 3,152
Haven't figured this out yet. Not 100% sure it is comskip causing the crash or not.

For instance on my VM server it just crashed again today I just restarted the Docker and these are my JVM specs in SageTV Info:
JVM Heap Size (Used/Total/Max): 5850MB/13149MB/15032MB

I never thought when I increased the heap size on the docker I would ever see SageTV using all of it.

For the docker container I limit by this parameter:
--memory="16G"

And SageTV java memory is 14336

If it isn't comskip (although it is always running when the crash occurs - but it could be just coincidence I suppose) - any thoughts on what it could be?

Plugins:
  • Anti-Favorites for Dummies
  • Commercial Detector UI
  • Complementary Metadata Tools
  • Comskip Playback
  • CPU Meter
  • Encoder Names
  • Java Heap Monitor
  • Move Imported TV shows to SageTV recordings
  • Restart SageTV from Setup menu
  • Season Premiere borders for Guide
  • Signal Strength on OSD
  • VideoOut Tools for Extenders
  • Batch Meta Tools
  • Commercial Detector
  • Jetty Web Server
  • Phoenix Core services (with update available showing so will update just in case
  • sageTV addons License Server
  • SageTV Groovy Tools
  • SageTV Web Interface (with update available so will update this too)
There are several that I could get rid of but they really are not likely to be causing this as they have been installed for a long time and the only thing new has been SageTV itself and the Docker container.
__________________
"Keep your goals away from the trolls"
Reply With Quote
  #8  
Old 07-02-2017, 04:24 AM
stuckless's Avatar
stuckless stuckless is offline
SageTVaholic
 
Join Date: Oct 2007
Location: London, Ontario, Canada
Posts: 9,711
have you tried running with a much smaller java memory size? In looking at the unraid system log, it appears that the system is killing java because it's getting an OOM. Are you seeing OOM errrors in the SageTV log?

Java will use everything you throw at it. If you give it 16g of memory it will use 16g of memory...eventually.

I would set your JVM to 4G (which I think is extremely large) and leave your other settings.

I'm wondering if because your java is set 14G andyou docker is set 16G then that's not much room to run other things, like comskip, ffmpeg, etc.
Reply With Quote
  #9  
Old 07-02-2017, 02:51 PM
BobPhoenix BobPhoenix is offline
SageTVaholic
 
Join Date: Oct 2004
Posts: 3,152
Quote:
Originally Posted by stuckless View Post
have you tried running with a much smaller java memory size? In looking at the unraid system log, it appears that the system is killing java because it's getting an OOM. Are you seeing OOM errrors in the SageTV log?

Java will use everything you throw at it. If you give it 16g of memory it will use 16g of memory...eventually.

I would set your JVM to 4G (which I think is extremely large) and leave your other settings.

I'm wondering if because your java is set 14G andyou docker is set 16G then that's not much room to run other things, like comskip, ffmpeg, etc.
OK. That what I had it set at originally. Didn't think about leaving room for comskip & ffmpeg. Will reduce both servers back to that. I did get one out of memory error in SageTV on 6/24 when it crashed previously. Specifically the error was "General System Message" - Java Heap has exceeded warning level. The message comes from the Java Heap Monitor plugin which has the following settings: "Warning Threshhold%" - 80, "Error Threshhold%" - 95. I had the docker set to 6G and SageTV set to 4G. I will leave both servers dockers at 16G like currently but I will reduce SageTV limit to 6G since when I restarted one server the memory usage was at 5G. 10G should be enough for ffmpeg and comskip.
__________________
"Keep your goals away from the trolls"

Last edited by BobPhoenix; 07-02-2017 at 02:58 PM.
Reply With Quote
  #10  
Old 07-02-2017, 05:37 PM
stuckless's Avatar
stuckless stuckless is offline
SageTVaholic
 
Join Date: Oct 2007
Location: London, Ontario, Canada
Posts: 9,711
Quote:
Originally Posted by BobPhoenix View Post
OK. That what I had it set at originally. Didn't think about leaving room for comskip & ffmpeg. Will reduce both servers back to that. I did get one out of memory error in SageTV on 6/24 when it crashed previously. Specifically the error was "General System Message" - Java Heap has exceeded warning level. The message comes from the Java Heap Monitor plugin which has the following settings: "Warning Threshhold%" - 80, "Error Threshhold%" - 95. I had the docker set to 6G and SageTV set to 4G. I will leave both servers dockers at 16G like currently but I will reduce SageTV limit to 6G since when I restarted one server the memory usage was at 5G. 10G should be enough for ffmpeg and comskip.
yeah, the heap warning plugin... not sure what it does, but that can likely be nothing to worry about. Java will just keep using memory and avoid garbarge collecting until it has to. So, it can grow the heap very quickly and result in those kinds of "warnings", but an OOM error is really a case where the JVM can't allocate enough room for a new object AFTER it tries all possible means to garbage collect. But your errors seems to be coming from unRAID (linux os) and not Java itself.
Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Manual Comskip in Docker dstanley SageTV for unRAID/Docker 5 06-01-2017 04:33 AM
How are you running ComSkip in unRaid/Docker setups with v9? mgpaulus SageTV v9 Customizations 5 01-22-2017 05:53 PM
SageTV Server Linux with Comskip for Docker (and unRAID) Support Thread stuckless SageTV for unRAID/Docker 409 10-10-2016 07:22 AM
Server crashing...comskip? nettech_gt SageTV v7 Customizations 8 04-04-2013 09:02 AM
Media Streaming inside SageTV 6? plawlor SageTV Software 4 03-25-2009 05:46 PM


All times are GMT -6. The time now is 07:33 AM.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, vBulletin Solutions Inc.
Copyright 2003-2005 SageTV, LLC. All rights reserved.