SageTV Community  

Go Back   SageTV Community > SageTV Development and Customizations > SageTV Github Development

Notices

SageTV Github Development Discussion related to SageTV Open Source Development. Use this forum for development topics about the Open Source versions of SageTV, hosted on Github.

Reply
 
Thread Tools Search this Thread Display Modes
  #1  
Old 09-21-2017, 11:27 AM
DynamoBen DynamoBen is offline
Sage Aficionado
 
Join Date: Nov 2003
Posts: 283
Angry EPG Outofmemory killing service

A few days ago my server starting dropping offline. After some investigation I discovered that EPG updates where throwing JAVA outofmemory exceptions which was completely killing sage. I was having to restart the service manually every few hours.

Since then I've updated to the latest version of sage, ensured I had the latest JAVA version, removed all plugins, and reduced the JAVA heap size. Reducing heap has helped keep it online longer but the problem is still occurring every few days.

I read in another thread that this might be a JAVA issue, although it could be a SageTV issue to0. Any ideas?

Example from Log:
Wed 9/20 10:30:45.443 [SDEPG-Import@13f39eb] SDEPG Unable to get head-shots for: SDPerson{personId='536610', nameId='558195', name='Larry Detwiler', role='Director', characterName='null', billingOrder='07'}
Wed 9/20 10:30:45.447 [SDEPG-Import@13f39eb] java.lang.OutOfMemoryError: unable to create new native thread

Last edited by DynamoBen; 09-22-2017 at 01:16 PM.
Reply With Quote
  #2  
Old 09-21-2017, 12:01 PM
wnjj wnjj is online now
Sage Icon
 
Join Date: Jan 2009
Posts: 1,088
What did you reduce your heap size to?

To confirm, you're on version 9.1.6 now?
Reply With Quote
  #3  
Old 09-21-2017, 12:08 PM
DynamoBen DynamoBen is offline
Sage Aficionado
 
Join Date: Nov 2003
Posts: 283
Quote:
Originally Posted by wnjj View Post
What did you reduce your heap size to?
From 1024 to 512

Quote:
To confirm, you're on version 9.1.6 now?
9.1.6.747
Reply With Quote
  #4  
Old 09-21-2017, 12:39 PM
wnjj wnjj is online now
Sage Icon
 
Join Date: Jan 2009
Posts: 1,088
Quote:
Originally Posted by DynamoBen View Post
From 1024 to 512


9.1.6.747
Those should be fine. The Java garbage collection issue was when using G1GC in V9.1.0 - V9.1.5.

I read that Java 9 is supposed to be out today and it unfortunately turns on G1GC by default. Are you by chance on Java 9?
Reply With Quote
  #5  
Old 09-21-2017, 03:23 PM
DynamoBen DynamoBen is offline
Sage Aficionado
 
Join Date: Nov 2003
Posts: 283
Quote:
Originally Posted by wnjj View Post
Are you by chance on Java 9?
Nope. Java 8 Update 144.
Reply With Quote
  #6  
Old 09-22-2017, 01:17 PM
DynamoBen DynamoBen is offline
Sage Aficionado
 
Join Date: Nov 2003
Posts: 283
Got another Failure overnight will the OTA guide was updated. Any ideas for a workaround or fix?
Reply With Quote
  #7  
Old 09-22-2017, 03:13 PM
wnjj wnjj is online now
Sage Icon
 
Join Date: Jan 2009
Posts: 1,088
It sure seems like you've done what you can. Others haven't been complaining since the G1GC revert. Can you post the hs_pid_err* file from the last crash? There may be something of note in there.
Reply With Quote
  #8  
Old 09-22-2017, 03:18 PM
Narflex's Avatar
Narflex Narflex is offline
Sage
 
Join Date: Feb 2003
Location: Redondo Beach, CA
Posts: 6,308
Also, grep your logs for the "MemStats" line and see if your JVM heap is getting used a lot or not. I would actually expect that the JVM heap isn't getting too full because this error is about not enough memory to create a native thread. You should check to see if maybe the OS itself isn't running out of memory (but it should be using swap space if that's the case)...and look at the memory usage of the SageTV process (for the process itself, not just the JVM heap like I mentioned regarding MemStats).
__________________
Jeffrey Kardatzke
Google
Founder of SageTV
Reply With Quote
  #9  
Old 09-22-2017, 04:07 PM
DynamoBen DynamoBen is offline
Sage Aficionado
 
Join Date: Nov 2003
Posts: 283
Quote:
Originally Posted by wnjj View Post
It sure seems like you've done what you can. Others haven't been complaining since the G1GC revert. Can you post the hs_pid_err* file from the last crash? There may be something of note in there.
I can find no such file, where does this normally live?
Reply With Quote
  #10  
Old 09-22-2017, 04:52 PM
wnjj wnjj is online now
Sage Icon
 
Join Date: Jan 2009
Posts: 1,088
Quote:
Originally Posted by DynamoBen View Post
I can find no such file, where does this normally live?
Sorry, I meant hs_err_pid*.log.

I'm not entirely sure where but with a hard crash they get created. I'm guessing in the directory where SageTV.exe resides. They include a memory map of the native memory used by loaded DLL's and the Java engine.
Reply With Quote
  #11  
Old 09-22-2017, 04:54 PM
DynamoBen DynamoBen is offline
Sage Aficionado
 
Join Date: Nov 2003
Posts: 283
Quote:
Originally Posted by wnjj View Post
Sorry, I meant hs_err_pid*.log.

I'm not entirely sure where but with a hard crash they get created. I'm guessing in the directory where SageTV.exe resides. They include a memory map of the native memory used by loaded DLL's and the Java engine.
Nope, nothing.
Reply With Quote
  #12  
Old 09-22-2017, 06:48 PM
wnjj wnjj is online now
Sage Icon
 
Join Date: Jan 2009
Posts: 1,088
Quote:
Originally Posted by DynamoBen View Post
Nope, nothing.
That means the JVM isn't crashing but Sage is. I guess this makes sense because the out of memory error is trapped with the EPG code. Other users were running out of memory when loading screensaver images. Hard crashes of the JVM create those logs.

Unfortunately I don't have any further suggestions, other than what Narflex suggested.
Reply With Quote
  #13  
Old 09-22-2017, 06:58 PM
DynamoBen DynamoBen is offline
Sage Aficionado
 
Join Date: Nov 2003
Posts: 283
Quote:
Originally Posted by Narflex View Post
Also, grep your logs for the "MemStats" line and see if your JVM heap is getting used a lot or not.
Scanned through and it hovers between 300-450M. At one point total usage was 512 but it recovered. I was expected that there would be a consistent climb leading to the exceptions but that wasn't the case. Weird.

Quote:
You should check to see if maybe the OS itself isn't running out of memory
Nothing unusual here, plenty of room there doesn't appear to be any sort of memory leak.

Quote:
look at the memory usage of the SageTV process
Nothing odd there either, although I'd really need to monitor during an EPG update.

I'm gone ahead and stopped the nightly backups and uninstalled the AV software which was updated right around the time this started. Hopefully one of those is what pushed it over the edge. Beyond that I'm not sure what else I can do.
Reply With Quote
  #14  
Old 09-23-2017, 08:33 AM
waynedunham waynedunham is offline
Sage Icon
 
Join Date: Sep 2004
Posts: 1,374
I'm not sure if my issue is related to this, but especially within the last couple weeks I'm seeing both of my SageTV V9 machines throwing Java Heap exceeded warnings into the system messages during EPG processing.

I have seen it in the past, but am seeing almost daily now and almost always either at the EPG download/processing time or if I go in to my scheduled recordings and make any changes to fix conflicts etc.

I am thinking it is most likely due to, or made worse by the fall season starting up and it having a lot more processing/conflict resolution to do with all the new shows and starting up the new season of old favorites.

I have a ton of favorites, and quite a large # of tuners on each machine with 3 lineups on one machine(HDPVR/STB, Cable HDHR Prime, OTA) and 2(Cable HDHR Prime, OTA) on the other.

I'm sure it factors in that I also use Intelligent Recording on both machines as well.
__________________
Wayne Dunham
Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
OutOfMemory Exception mgpaulus SageTV Github Development 6 02-09-2017 04:44 PM
Placeshifter killing SageTV Service valnar SageTV Software 6 02-07-2015 05:38 PM
HD-PVR is killing me.... CanadianEh Hardware Support 30 02-25-2010 11:45 AM
Killing Fork() toricred SageTV Studio 9 07-26-2009 07:50 PM


All times are GMT -6. The time now is 02:58 AM.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.
Copyright 2003-2005 SageTV, LLC. All rights reserved.