out of memory

[DIBU]

Greetings,

after i have add 2 DVD's to SagetTV-import, the scan for imported media will bring me a out of memory in sage (service). i have set the java heap to 1024 and more. but no effect.

when i set the java heap to much, sage (not the service) can't start.

[stuckless]

Originally Posted by DIBU

Greetings,

after i have add 2 DVD's to SagetTV-import, the scan for imported media will bring me a out of memory in sage (service). i have set the java heap to 1024 and more. but no effect.

when i set the java heap to much, sage (not the service) can't start.

I would try it without any plugins installed If it still fails the I'd file a bug report. That's not normal.

[DIBU]

ok i will do that...

here the info before the import scan start:

SageTV Version 7.0.23.246
Java Version 1.6.0_22
Server true
Recordings Used Diskspace 1.575,60 G
Library Import Scan in Progress false
JVM Max Memory 1.520.448 M
JVM Allocated Memory 1.472.524 M

i think it is a problem with to much import and "batch metadata tools"

i think this is wrong..i have more
* Music: 638 albums / 15812 tracks
* Video: 260 files
* DVDs: 221 files
* Pictures: 5582 files
* Total Media Library Duration: 83 Tage 7 Stunden 12 Minuten

[stuckless]

Your allocated memory seems to almost match your max memory before the import scan starts. That seems odd. I run a jvm of 784mb and currently after about of week of running (and many scans later), I'm sitting with an allocated memory of 300mb.

I'd remove or disable the bmt plugin (and any other plugins) and try again. BMT will only scan new files files, so even if you have 1000s of files existing, bmt will not scan those automatically, only the new files that are added.

[heffe2001]

I've noticed lately that my java heap usage is almost maxed out most of the time, and although I've not done much testing, it does appear to be BMT using most of it. It looks like it maxes it out especially when I use the browse feature in the web menus to fix fanart or miss-identification of new movies and such. I don't recall it using quite this much before the version before last.

[stuckless]

Originally Posted by heffe2001

I've noticed lately that my java heap usage is almost maxed out most of the time, and although I've not done much testing, it does appear to be BMT using most of it. It looks like it maxes it out especially when I use the browse feature in the web menus to fix fanart or miss-identification of new movies and such. I don't recall it using quite this much before the version before last.

In the last 3 versions of bmt, not much has changed on the server code, but lots have changed in the web ui code, so I doubt bmt is consuming all that memory. Also, no one runs bmt more than I do, and my memory is very stable. Don't get wrong, BMT will consume huge amounts of memory. When it downloads images, it has to scale them, which means loading them into memory and then performing the scaling operations. If you are updating 100s of movies them memory will spike, and then go down. 2 days ago, I scanned 400 movies. My memory this morning is sitting around 300mb (with 400+mb free).

I would try running for a few weeks without BMT installed and see what happens. I suspect not much, since, it bmt sits idle most of the time.

If you have fanart plugins installed, then those will also consume large amounts of memory as well, since backgrounds, posters, banners, etc, will need to be loaded into memory.

[DIBU]

ok i think i have the answer

i have put 2 new dvd (as rip) on my drive U:

this is not only a dvd image drive, also a recording device for the dbox. also i make a folder DVD_Archiv and a folder DBOX_Recording and put the files in this.

ok...when i say to sage use U:\ as video import..out of memory

when i say to sage use U:\DVD_Archiv AND U:\DBOX_Recording as video import..NO PROBLEM

this is now...sage is now refresh the import media directorys without problems

i will wait and see what is coming




ok that is is not

when i start the import.. after the out of memory some video files are more importet..508 next scan error 512 next scan error 518...

is there any error on the folder.jpg?

i will see..

[heffe2001]

I wondered if it might be caching of images, the only reason I even noticed my usage is because I installed the Java heap monitor plugin. I do a scan on new content, and it send a system message every time about memory usage. It's sitting right now with 914 max, 701 used, and 507 free (I'm guessing some of that used is marked as scratch or something along those lines, and is available to be reallocated when needed?). I'll be happy if they ever release a 64b version of Sage, so we can use more memory for the java heap... Although I assume that if I were running x64 windows with say 8g ram, even running the 32b version of Java I could allocate 3g+ memory if I really wanted to (not that it would be really needed though).

Originally Posted by stuckless

In the last 3 versions of bmt, not much has changed on the server code, but lots have changed in the web ui code, so I doubt bmt is consuming all that memory. Also, no one runs bmt more than I do, and my memory is very stable. Don't get wrong, BMT will consume huge amounts of memory. When it downloads images, it has to scale them, which means loading them into memory and then performing the scaling operations. If you are updating 100s of movies them memory will spike, and then go down. 2 days ago, I scanned 400 movies. My memory this morning is sitting around 300mb (with 400+mb free).

I would try running for a few weeks without BMT installed and see what happens. I suspect not much, since, it bmt sits idle most of the time.

If you have fanart plugins installed, then those will also consume large amounts of memory as well, since backgrounds, posters, banners, etc, will need to be loaded into memory.

[DIBU]

ok i'm now one step further..

i have 2 new video files in asx format. i think this make the out of memory



i will report later

yep, when i move this asx -files out of import directory, NO error. Move it back..out of memory..

[QuasiInsomniac]

Thought I would chimein here. I have a new install of V7 and its been running more or less stable for about 2 weeks.

At 7:44am this morning I had that heap size error show up in the system messages. I wasnt aware of the problem until late morning when my wife said she turned on the TV and Sage was frozen on what looked like two different pictures from our photo album. It looked like it froze in transition. We did have the screen saver set to display images from the photo gallery.

As a help to troubleshoot, all hard drives are new. Upon restart, the heap size under system information was 133/197/400. I dont know what it was during the lockup. And lastly, we have an a-load of pictures; like 12,000 photos or something. I'm thinking this has something to do with it since it looked like the slideshow, the freezing, and the error all happend abouts the same time.

Upon restart of Sage (force quit TaskMan), it has run fine all day best we can tell. But I will update here should that change.

[bastafidli]

I have been running all 7 betas and since I have upgraded to full release I had already 2 out of memory situations. Saved the logs but didnt have time to investigate yet.

[QueOnda]

I had a same issue. I saved the logs but haven't done anything with them. I was going to send them to sage but was waiting to happen again. I have had issues before but not sure related as I didn't save the logs or look into it.

[bastafidli]

I just had my 3rd out of memory in 3 months, I do restart sage now and then. This last time it was running for 10 days (1/2 -> 1/12) when I experience OutOfMemory. If it happens again before the end of the month I will start investigating.

[stuckless]

I had an OOM error the other day as well, which is the first OOM I've ever had.

From the BMT web ui, I saw this...

Sun Jan 09 00:53:21 GMT-500 2011 - Out of Memory Detected
SageTV has detected that it has run out of memory. You may need to increase your JVM heap size or there could be a plugin/customization that is causing an issue.

Sat Jan 08 21:39:45 GMT-500 2011 - Failure Loading Video Capture Device
Failed to load the video capture device Hauppauge HD PVR 00A1FEA2. Please check to make sure the device is properly connected and the proper drivers are installed. You may need to restart the device or reboot your machine to correct the problem.

Sat Jan 08 21:39:45 GMT-500 2011 - Halt Detected in Recording
The device input "" stopped producing data while recording "The Suite Life on Deck" on TFCP on physical channel . SageTV will try to reset the device. This may be caused by a channel that is no longer available.

When I looked in the logs, I could see that after sagetv failed to load my video capture card, it went into some loop trying to load the capture device and then I finally got the OOM several hours later. That's what I can tell from the SageTV logs, since I have 4 log files with nothing except...

Sun 1/9 7:00:15.102 [Seeker@15cd6a3] java.lang.NullPointerException
Sun 1/9 7:00:15.127 [Seeker@15cd6a3] SEEKER EXCEPTION THROWN:java.lang.NullPointerException
Sun 1/9 7:00:15.127 [Seeker@15cd6a3] java.lang.NullPointerException
Sun 1/9 7:00:15.144 [Seeker@15cd6a3] SEEKER EXCEPTION THROWN:java.lang.NullPointerException
Sun 1/9 7:00:15.144 [Seeker@15cd6a3] java.lang.NullPointerException
Sun 1/9 7:00:15.169 [Seeker@15cd6a3] SEEKER EXCEPTION THROWN:java.lang.NullPointerException
Sun 1/9 7:00:15.169 [Seeker@15cd6a3] java.lang.NullPointerException
Sun 1/9 7:00:15.194 [Seeker@15cd6a3] SEEKER EXCEPTION THROWN:java.lang.NullPointerException
Sun 1/9 7:00:15.194 [Seeker@15cd6a3] java.lang.NullPointerException
Sun 1/9 7:00:15.211 [Seeker@15cd6a3] SEEKER EXCEPTION THROWN:java.lang.NullPointerException

Based on the timestamps, it tried doing this in an endless loop until I got the OutOfMemory

Sun 1/9 7:39:51.365 [ErrorStreamConsumer@198886e] Exception in thread "ErrorStreamConsumer" Sun 1/9 7:41:04.750 [ErrorStreamConsumer@198886e] java.lang.OutOfMemoryError: Java heap space
Sun 1/9 7:42:14.880 [InputStreamConsumer@9b5df8] Exception in thread "InputStreamConsumer" Sun 1/9 7:44:31.605 [Seeker@15cd6a3] SEEKER EXCEPTION THROWN:java.lang.OutOfMemoryError: Java heap space
Sun 1/9 7:44:31.605 [LocatorRegistrationClient@669ec3] MAC address was not able to be detected, using zero for the MAC!
Sun 1/9 7:44:42.280 [MsgManager@19ed523] ERROR processing message of:java.lang.OutOfMemoryError: Java heap space
Sun 1/9 7:44:42.282 [MsgManager@19ed523] java.lang.OutOfMemoryError: Java heap space
Sun 1/9 7:44:42.282 [Seeker@15cd6a3] java.lang.OutOfMemoryError: Java heap space
Sun 1/9 7:44:42.282 [LocatorRegistrationClient@669ec3] WARNING: Aborting properties file save!

I don't know what it all means, but I suspect that SageTV tried forever to load the capture device until it finally died.

As it turns out, the power adapter on my HDPVR was nuked, so that's why I lost the tuner and why SageTV could not re-load it.

Anyways... that's my only experience with an OOM in SageTV.

Right now my memory looks good, and it's been running since I restarted on the 9th.

JVM Max Memory
713,664 M

JVM Allocated Memory
121,408 M

JVM Total Free Memory
656,755 M

[rtrski]

I think something like this just happened to me for the first time, too.

Noticed the HD300 was really sluggish trying to watch some TV (converted, not original as-recorded) rather than watching our Dear Leader's Tucson Shooting Pep-Rally. (ok, ok...no more politics)

Anyway, the recording was 'surging' and pausing almost perfectly every 5 seconds like the disk was having a hard time keeping up, not like the recording had been spotty in the first place. Tried a hard power off and reboot of the HD300 and got the same thing, but by then the speech was over so forgot about it.

Then yesterday came upstairs to see a Windows notice that "SageService has stopped working." Don't have logging on so don't know what really went wrong, but the sluggish UI (thumbnails and backgrounds really slow to load) and playback would seem consistent with an out of memory error. Forgot to actualy check the Java heap size (mine is set at 768) before forcing a restart - after the restart it's down at about 150 or so used.

I did also recently update the BMT, Jetty, and Phoenix core services. There was also an update on like 1/8 I think to the Comskip playback, and that one did not seem to want to force a restart.

Will keep an eye on it - if anything like it appear to happen again will enable logging and report back in on this thread as well.

[nyplayer]

I got this too after updating BMT and phoenix... never had this before. One thing I dislike about the plugins you cannot go back to a previous version to see if it is the BMT upgrade.

[rtrski]

Yep, happened again tonight. Really don't see anything in logs - but probably don't have them turned on. Will fiddle with it this weekend; too buried at work (13 hr day today and yesterday) right not to do other than restart it and hope it makes it thru to the wife's shows tomorrow.

[bastafidli]

It happened again for me as well but this time I was able to connect to the VM with VisualVM and create snapshot. My heap was over 600M with 14million objects present.

22% java.lang.String, total size 78MB (12%)
22% char[] total size 200MB (30%)
14% tv.sage.b.e[] total size 27MB (4%)
10% java.util.HasMap$Entry 35MB (5%)
...
1% java.lang.String[] 50MB (8%)

Definitely feels like memory leak. I will try to figure out if I can send the snapshot to Sage.

[bastafidli]

I have exchanged few emails with Sage support and they thing that the problem is caused by one of the plugins rather than bySage itself. Due to this there is not much they can/want to do. Analyzing the log itself may also not help since plugins often directly modify code/stv shipped by sage. The best option seems to be to try to identify what plugins do we have in common, then try to remove those and see if it solves the problem. When you get a chance, can you please post here list of your plugins you have installed and I will do the same. Thanks.

[stuckless]

I have the following plugins...
Batch metadata tools - Web Interface
Jetty Web Server
Phoenix Core Services
SageTV Web Interface
Phoenix Fanart for SageTV
Season and Episode Number in Details
CVF - Custom Video Folder
Commercial Detector
Commercial Detector UI
Comskip Playback
Media Streaming Services
SageTV H2 Database
SageTV Mobile Interface

I run those plugins, and my kids usually connect every day to sage to watch saget for 30min - hr. I have 1 tuner with HDPVR and I record on average 1-2 shows a day.

I'm throwing this out as a baseline, because with that activity, my current Allocated JVM memory is 72Mb. My JVM size is 784MB, and my free memory is 656MB (as reported by the bmt web ui)

While OOM erros are rare for me (I've only had one, and it was due to bad tuner), I feel pretty confident that the above listed plugins are not the culprit, given how low my memory usage is.

One final note, I'm running on Ubuntu 10.10.