SageTV Community  

Go Back   SageTV Community > SageTV Development and Customizations > SageTV Github Development

Notices

SageTV Github Development Discussion related to SageTV Open Source Development. Use this forum for development topics about the Open Source versions of SageTV, hosted on Github.

Reply
 
Thread Tools Search this Thread Display Modes
  #1  
Old 08-05-2017, 06:15 AM
rickgillyon's Avatar
rickgillyon rickgillyon is offline
Sage Icon
 
Join Date: Sep 2005
Location: Whitley Bay, England
Posts: 1,950
Sage Process on Server Crashing

This is happening several times a day. I'm not using the service. Latest v9 on Win10 64 Pro. Any ideas what's causing it? hs_err_pid and log attached.
Attached Files
File Type: zip crash.zip (104.3 KB, 147 views)
__________________
unRAID Server: Intel Core i5 7600K, 48GB DDR4, 2x512GB PCIe M.2 Cache Pool, 2x10TB SATA3 Parity Drive, 3x8TB SATA Array, 1x hdHomeRun DVB-T2 Quattro, IPTV via xTeVe, unRAID 6.8.3, tvHeadEnd for recording back end, Emby
Clients: 3 Nvidia Shields, 3 FireTV, 3 Win10 Pro PC Clients
Reply With Quote
  #2  
Old 08-06-2017, 05:19 PM
rickgillyon's Avatar
rickgillyon rickgillyon is offline
Sage Icon
 
Join Date: Sep 2005
Location: Whitley Bay, England
Posts: 1,950
Also getting separately, around once per day, an error that the SageTV process has stopped, which stops all SageTV activity until I click it. Details below. It's better if the process just stops itself rather than waiting for input as it restarts...
Code:
Faulting application name: SageTV.exe, version: 9.1.5.683, time stamp: 0x594a4a96
Faulting module name: ntdll.dll, version: 10.0.15063.447, time stamp: 0xd51d5c5e
Exception code: 0xc0000005
Fault offset: 0x0002ef5d
Faulting process id: 0x27bc
Faulting application start time: 0x01d30e39f60e6701
Faulting application path: C:\Program Files (x86)\SageTV\SageTV\SageTV.exe
Faulting module path: C:\WINDOWS\SYSTEM32\ntdll.dll
Report Id: d6b5dcc2-69bf-4969-9da6-d416ca855e4b
Faulting package full name: 
Faulting package-relative application ID:
Any hints on these would be helpful.
__________________
unRAID Server: Intel Core i5 7600K, 48GB DDR4, 2x512GB PCIe M.2 Cache Pool, 2x10TB SATA3 Parity Drive, 3x8TB SATA Array, 1x hdHomeRun DVB-T2 Quattro, IPTV via xTeVe, unRAID 6.8.3, tvHeadEnd for recording back end, Emby
Clients: 3 Nvidia Shields, 3 FireTV, 3 Win10 Pro PC Clients
Reply With Quote
  #3  
Old 08-06-2017, 06:00 PM
wnjj wnjj is online now
Sage Icon
 
Join Date: Jan 2009
Posts: 1,493
Your system is running out of native memory. There are a few others posting in other threads having the same issue. Try turning off the photo screensaver if you are running it. Something is wrong in V9 that is affecting several users in the same way. The log you posted already was out of memory at the top. If it crashes again, post the last log file that isn't showing 'out of memory' issues. Sagetv_1.txt is older than sagetv_0.txt and sagetv_2 is older than that, etc.
Reply With Quote
  #4  
Old 08-06-2017, 06:06 PM
rickgillyon's Avatar
rickgillyon rickgillyon is offline
Sage Icon
 
Join Date: Sep 2005
Location: Whitley Bay, England
Posts: 1,950
Thanks, will do. Every time I look it's running at <30% RAM usage, but I guess it just suddenly maxes out and crashes...
__________________
unRAID Server: Intel Core i5 7600K, 48GB DDR4, 2x512GB PCIe M.2 Cache Pool, 2x10TB SATA3 Parity Drive, 3x8TB SATA Array, 1x hdHomeRun DVB-T2 Quattro, IPTV via xTeVe, unRAID 6.8.3, tvHeadEnd for recording back end, Emby
Clients: 3 Nvidia Shields, 3 FireTV, 3 Win10 Pro PC Clients
Reply With Quote
  #5  
Old 08-06-2017, 07:15 PM
wnjj wnjj is online now
Sage Icon
 
Join Date: Jan 2009
Posts: 1,493
Quote:
Originally Posted by rickgillyon View Post
Thanks, will do. Every time I look it's running at <30% RAM usage, but I guess it just suddenly maxes out and crashes...
A 32-bit process only has access to 2GB of user memory. The Java heap (default 768MB) comes out of that, as does the JVM itself (overhead plus per-thread) and every other DLL loaded to run sage. Then, sage itself has native code in DLL's that request some memory plus Java can request native memory byte buffers directly. The native memory requests are often related to image processing o file buffering so that's why the photo screensaver could be a suspect.

One of the above or the combination of all is using too much. It does seem to be a new issue in V9 for some. I've been looking through the source code for a memory leak and so far haven't found anything. Keep those log files coming and I'll keep looking.

Last edited by wnjj; 08-06-2017 at 07:19 PM.
Reply With Quote
  #6  
Old 08-07-2017, 10:53 AM
Narflex's Avatar
Narflex Narflex is offline
Sage
 
Join Date: Feb 2003
Location: Redondo Beach, CA
Posts: 6,320
If you go to this photo in the photos menu and view it, does it crash?

V:\Photos\2006\200612\IMG_3829.jpg

There may be something specifically odd about that photo that's causing it to allocate a huge chunk of native memory incorrectly which would create a problem like this. If it does reproduce it...please DO NOT delete that photo so we can fix the underlying problem.
__________________
Jeffrey Kardatzke
Google
Founder of SageTV
Reply With Quote
  #7  
Old 08-07-2017, 11:07 AM
rickgillyon's Avatar
rickgillyon rickgillyon is offline
Sage Icon
 
Join Date: Sep 2005
Location: Whitley Bay, England
Posts: 1,950
There is something odd about that photo. I'm on it.

But I currently have it displayed on a PC client and an Android extender with no apparent ill effect. I've attached it anyway, but it looks like all the others from this end. Nope, too big to attach, it's here:
http://www.gillyon.com/downloads/IMG_3829.jpg
__________________
unRAID Server: Intel Core i5 7600K, 48GB DDR4, 2x512GB PCIe M.2 Cache Pool, 2x10TB SATA3 Parity Drive, 3x8TB SATA Array, 1x hdHomeRun DVB-T2 Quattro, IPTV via xTeVe, unRAID 6.8.3, tvHeadEnd for recording back end, Emby
Clients: 3 Nvidia Shields, 3 FireTV, 3 Win10 Pro PC Clients
Reply With Quote
  #8  
Old 08-07-2017, 11:54 AM
wnjj wnjj is online now
Sage Icon
 
Join Date: Jan 2009
Posts: 1,493
Quote:
Originally Posted by Narflex View Post
If you go to this photo in the photos menu and view it, does it crash?

V:\Photos\2006\200612\IMG_3829.jpg

There may be something specifically odd about that photo that's causing it to allocate a huge chunk of native memory incorrectly which would create a problem like this. If it does reproduce it...please DO NOT delete that photo so we can fix the underlying problem.
Jeff,
Several users are crashing in imageloader but some are even crashing the JVM itself. The imageloader crash is because there are mallocs that don't check for success (null pointer). Those will crash even if a small amount is requested but not available. The real issue is why is so much native memory being used up or fragmented? It seems that some who upgraded V7 systems started having this issue. Perhaps they were using close to the limit before and are now just over or maybe their media types or photo formats bring out a memory leak or bug?
Reply With Quote
  #9  
Old 08-07-2017, 07:00 PM
JREkiwi's Avatar
JREkiwi JREkiwi is offline
Sage Icon
 
Join Date: Jan 2005
Location: Auckland, New Zealand
Posts: 2,129
My system was more stable after turning off "Optimized Image Loader" but was still having crashes using any of the 9.1 versions. I've had to go back to 9.0.13 to get a stable system.

John
Reply With Quote
  #10  
Old 08-07-2017, 07:43 PM
wnjj wnjj is online now
Sage Icon
 
Join Date: Jan 2009
Posts: 1,493
Quote:
Originally Posted by JREkiwi View Post
My system was more stable after turning off "Optimized Image Loader" but was still having crashes using any of the 9.1 versions. I've had to go back to 9.0.13 to get a stable system.

John
That's quite interesting. The most significant change memory-wise in 9.1.x was turning on G1GC string deduplication. I wonder if that feature enabling saving of Java heap has a cost to native memory?
Reply With Quote
  #11  
Old 08-07-2017, 09:08 PM
JREkiwi's Avatar
JREkiwi JREkiwi is offline
Sage Icon
 
Join Date: Jan 2005
Location: Auckland, New Zealand
Posts: 2,129
Quote:
Originally Posted by wnjj View Post
That's quite interesting. The most significant change memory-wise in 9.1.x was turning on G1GC string deduplication. I wonder if that feature enabling saving of Java heap has a cost to native memory?
I wonder if we're running into an issue with Metaspace with G1GC or whether the -XX:+ParallelRefProcEnabled flag needs to be set ?

Also Narflex gives and explanation here why turning off the "Optimized Image Loader" would reduce the impact on native memory

John

Last edited by JREkiwi; 08-07-2017 at 09:10 PM.
Reply With Quote
  #12  
Old 08-07-2017, 10:50 PM
JREkiwi's Avatar
JREkiwi JREkiwi is offline
Sage Icon
 
Join Date: Jan 2005
Location: Auckland, New Zealand
Posts: 2,129
I also wonder whether we should be using G1GC when it still has bugs like this

John
Reply With Quote
  #13  
Old 08-07-2017, 11:04 PM
wnjj wnjj is online now
Sage Icon
 
Join Date: Jan 2009
Posts: 1,493
Quote:
Originally Posted by JREkiwi View Post
I also wonder whether we should be using G1GC when it still has bugs like this

John
If you want to try some more experimenting, go back to a 9.1.x version but use the .exe from 9.0.x. The .exe is where the G1GC is enabled when launching the JVM.
Reply With Quote
  #14  
Old 08-08-2017, 12:05 AM
JREkiwi's Avatar
JREkiwi JREkiwi is offline
Sage Icon
 
Join Date: Jan 2005
Location: Auckland, New Zealand
Posts: 2,129
Quote:
Originally Posted by wnjj View Post
If you want to try some more experimenting, go back to a 9.1.x version but use the .exe from 9.0.x. The .exe is where the G1GC is enabled when launching the JVM.
I've installed 9.1.5 with the 9.0.13 exes. We'll see how that goes. Thanks for the suggestion.

John
Reply With Quote
  #15  
Old 08-08-2017, 05:39 AM
rickgillyon's Avatar
rickgillyon rickgillyon is offline
Sage Icon
 
Join Date: Sep 2005
Location: Whitley Bay, England
Posts: 1,950
I watched today as the memory usage on my server went from the usual 21% to 76%, for an extended time. The server has 8GB. I've attached the logs, but no crash today. I think the high usage coincides with an EPG update.

Logs here:
http://www.gillyon.com/downloads/sagetv_0.zip
__________________
unRAID Server: Intel Core i5 7600K, 48GB DDR4, 2x512GB PCIe M.2 Cache Pool, 2x10TB SATA3 Parity Drive, 3x8TB SATA Array, 1x hdHomeRun DVB-T2 Quattro, IPTV via xTeVe, unRAID 6.8.3, tvHeadEnd for recording back end, Emby
Clients: 3 Nvidia Shields, 3 FireTV, 3 Win10 Pro PC Clients
Reply With Quote
  #16  
Old 08-18-2017, 10:09 AM
wnjj wnjj is online now
Sage Icon
 
Join Date: Jan 2009
Posts: 1,493
Quote:
Originally Posted by JREkiwi View Post
I've installed 9.1.5 with the 9.0.13 exes. We'll see how that goes. Thanks for the suggestion.

John
How's the change in GC working? I'm curious to know if that was the issue.
Reply With Quote
  #17  
Old 08-18-2017, 02:11 PM
JREkiwi's Avatar
JREkiwi JREkiwi is offline
Sage Icon
 
Join Date: Jan 2005
Location: Auckland, New Zealand
Posts: 2,129
Quote:
Originally Posted by wnjj View Post
How's the change in GC working? I'm curious to know if that was the issue.
Didn't have any problems and have been running 9.1.6 which uses the default GC without problems as well. I do still have "Optimized Image Loader" disabled though.

Not sure if G1GC was the entire problem, but from my reading it certainly seems more suited to large JAVA heaps.

John

Last edited by JREkiwi; 08-18-2017 at 02:16 PM.
Reply With Quote
  #18  
Old 08-21-2017, 10:52 AM
EnterNoEscape's Avatar
EnterNoEscape EnterNoEscape is offline
SageTVaholic
 
Join Date: Jun 2010
Location: Harrisburg, PA
Posts: 2,657
Quote:
Originally Posted by JREkiwi View Post
Didn't have any problems and have been running 9.1.6 which uses the default GC without problems as well. I do still have "Optimized Image Loader" disabled though.

Not sure if G1GC was the entire problem, but from my reading it certainly seems more suited to large JAVA heaps.

John
Ironically, it was supposed to help out Windows users with memory issues because of the really good string duplication.
__________________
SageTV v9 Server: ASRock Z97 Extreme4, Intel i7-4790K @ 4.4Ghz, 32GB RAM, 6x 3TB 7200rpm HD, 2x 5TB 7200rpm HD, 2x 6TB 7200rpm HD, 4x 256GB SSD, 4x 500GB SSD, unRAID Pro 6.7.2 (Dual Parity + SSD Cache).
Capture: 1x Ceton InfiniTV 4 (ClearQAM), 2x Ceton InfiniTV 6, 1x BM1000-HDMI, 1x BM3500-HDMI.

Clients: 1x HD300 (Living Room), 1x HD200 (Master Bedroom).
Software: OpenDCT :: WMC Live TV Tuner :: Schedules Direct EPG
Reply With Quote
  #19  
Old 08-22-2017, 10:46 AM
Narflex's Avatar
Narflex Narflex is offline
Sage
 
Join Date: Feb 2003
Location: Redondo Beach, CA
Posts: 6,320
The optimized image loader should actually use less system memory overall and definitely performs faster than that non-optimized one (as one would expect). It does use more native memory and less Java memory...but in the overall total it will be less. I definitely think it would be worth trying to enable that again now that you've seen improvement...I wouldn't want people disabling that as a potential fix for things that aren't broken now that the GC setting has been changed back.
__________________
Jeffrey Kardatzke
Google
Founder of SageTV
Reply With Quote
  #20  
Old 08-22-2017, 02:10 PM
JREkiwi's Avatar
JREkiwi JREkiwi is offline
Sage Icon
 
Join Date: Jan 2005
Location: Auckland, New Zealand
Posts: 2,129
Quote:
Originally Posted by Narflex View Post
The optimized image loader should actually use less system memory overall and definitely performs faster than that non-optimized one (as one would expect). It does use more native memory and less Java memory...but in the overall total it will be less. I definitely think it would be worth trying to enable that again now that you've seen improvement...I wouldn't want people disabling that as a potential fix for things that aren't broken now that the GC setting has been changed back.
I believe part of the problem is/was that Java 8 uses native memory for MetaSpace and G1GC in the current JAVA versions has a bug with cleaning up that used native memory. That bug is supposed to be fixed in JAVA 8.151

John
Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Server crashing corrupt wiz.bin? rxnelson SageTV Software 12 11-08-2015 09:12 PM
SageTV Server Restore - Validate my process... greggerm SageTV Software 2 02-26-2013 12:21 PM
Streamlined process for BD to server superteancum SageTV Media Extender 11 02-13-2009 08:32 AM
SageTV Server keeps crashing grauchy SageTV Software 2 02-21-2008 09:35 AM
MVP Server crashing Ralnee SageTV Software 2 11-11-2003 08:39 AM


All times are GMT -6. The time now is 10:03 AM.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2023, vBulletin Solutions Inc.
Copyright 2003-2005 SageTV, LLC. All rights reserved.