Problem with Movies and DVD/BD Section on 64 bit Server and HD-300 Extenders

[SteveW]

I have a fresh install of the 64 bit SageTV server on my media server. It's installed in its own directory. This media server also has our production SageTV 32 bit (9.1.8) in its own directory, which has been running perfectly - minus shutting it down every now and then to clear the java heap.

Once I had the new 64 bit version fully setup (including plugins), I copied over the wiz.bin file, pointed the import directories to where our production shares are, and away I went.

TV portion runs fine, and music seems to be fine. The problem I'm running into is the movies portion. I have BMT installed, as well as CMT and Phoenix Fan Art installed just as I do on my old system.

I first noticed this on an my HD-300 extender. Some of the fanart posters for the movies weren't showing up, and as you scrolled down through the movies, on every page, there is a spinning circle for about 10 seconds per page. As you move down the list, the spinning circle takes longer and longer.

On the off chance that there was some sort of plugin issue, I disabled CMT, Phoenix Fanart, and BMT, and went back into the DVD/BD menu.

With no fan art plugins, Sage natively will try to read a folder.jpg file in the bluray disk directories as a poster for the movie. Otherwise it just loads the first image in the movie file as a poster. Again scrolling through the list of movies, though I wasn't getting as much spinning circle, it is taking Sage quite a while to replace the Bluray and DVD logos with an image or poster from the folder.jpg files. Sometime a few minutes.

Once you've scrolled to the bottom of the movie list, and then scroll back up (either with native SageTV or with plugins enabled), posters it had already found, now its looking for some of them again and there are delays again scrolling back up.

I went to the server's built in client (with the fanart plugins re-enabled), and though nearly all the Phoenix fanart shows correctly, again there are spinning circles between pages of movies loading, and once you have let them load as you scroll to the bottom of the list for movies, scrolling back up, it is again trying to load posters and backgrounds again after having already displayed them once.

I have about 350 BD/DVD's in Sage, so not what I would think is a huge amount.

I'm going to have to switch back to 32 bit 9.1.8 at the server for now as the movies section is unusable for the family.

It almost seems like the system is starved for resources and unable to cache the thumbnails for the movie posters.

I had already before I started increased the java heap. Wasn't sure what I could increase it to. I set it at 2048 decimal, though on the server and extenders it says the max is 1909. It says currently I'm at 888MB/1229MB/1909MB. I find those numbers kind of surprising considering I can run 3 HD-300's withing the confines of 32 bit java with all of the fanart plugins with no issues.

I vaguely remember something similar to this happening many years ago wrt to slow loading thumbnails when version 7 first came out. I think at the time SageTV support had me increase a thumbnail generation cache or something? That was like upteen years ago, so no idea if that's why my original 32 bit server is running without issue.

Would like very much to use the 64 bit version, if someone can suggest what might be going on.

Was going to try the 64 bit client on my Win 10 PC to see how it behaved, but the install failed, but that's another thread...

Interestingly, running my 32 bit client at 9.1.8 against my 64 bit server at 9.2.1 and the Movies BD/DVD area runs perfectly - but then again I think with the windows client, all of that processing for fanart (other than pulling the fanart from the server share) runs local...

Would appreciate any suggestions. Was going to try to keep the 64 bit install as a fresh go of it, but maybe I'm going to have to install it on top of my old 32 bit install...

-Steve

[wnjj]

That would be a pain. I’ve always had a bit of that with my 32-bit system but not unusably bad. I’ll do some poking around but in the meantime you can try comparing your properties files between the 32 and 64 bit server to see if you can find whatever setting you may have adjusted years ago.

[wnjj]

How about one of these? Are they set differently in your 32-bit properties?

ui/system_memory_2dimage_cache_scale
ui/system_memory_2dimage_cache_size
ui/system_memory_2dimage_cache_limit

[jusjoken]

May help if you post the sagetv server log when this occurs.

If you are using gemstone and/or phoenix post those logs as well.

Phoenix creates a cache for these images and that cache is empty with a new install. If using gemstone then use the prebuild cache option in utilities...takes time but run it when not using the system.

But it could be something else so the logs may assist.

k

[SteveW]

Sorry, should have enclosed logs. I've zipped them up. Since I shut the server down a few times while working on the problem, there should be something in some if not all of these logs... SageTV_1.log is 9 MB has the most info in it. A lot of hang times mentioned...

Even the 9 MB log compressed far exceeds what I can post here. You can pull them from my Google Drive.

Just using the stock UI - no Gemstone or Phoenix at play as far as UIs...

https://drive.google.com/open?id=1Bq...DIPxiw7S2_1lU4

[SteveW]

"That would be a pain. I’ve always had a bit of that with my 32-bit system but not unusably bad. I’ll do some poking around but in the meantime you can try comparing your properties files between the 32 and 64 bit server to see if you can find whatever setting you may have adjusted years ago."




I'll try a compare on the two property files. Tnx

[SteveW]

Originally Posted by wnjj

How about one of these? Are they set differently in your 32-bit properties?

ui/system_memory_2dimage_cache_scale
ui/system_memory_2dimage_cache_size
ui/system_memory_2dimage_cache_limit

I checked these 3. Same in both files.

[SteveW]

Originally Posted by wnjj

How about one of these? Are they set differently in your 32-bit properties?

ui/system_memory_2dimage_cache_scale
ui/system_memory_2dimage_cache_size
ui/system_memory_2dimage_cache_limit

So have been digging further into this...

There are a boatload of differences between my 64-bit properties file and my production 32-bit, so not sure where to even start there. A couple under UI_Disable stood out, so I tried those, no diff.

I temporarily renamed my production 9.1.8 32-bit directory to SageTV.temp, and installed a brand new 9.2.1 32-bit install the same way I did the 64-bit to see what would happen. Same deal - fresh install, configure, add the same plugins, and then copied over the production wiz.bin to the new install

I'm seeing the same behaviour on the extenders as I did with the 64-bit server. I can scroll up and down through the movies, then things start to slow down, random movie posters are blank, spinning circles start showing up more often and longer as you keep exploring the list of DVDs/BDs.

The server client side is a little different. On the 64-bit server, scrolling around movies would give similar behaviour as the extender. Movie posters randomly blank, or folder.jpg images never loading, spinning circles, etc. The 32-bit server's client is not behaving that way at all. It runs as fluidly as my 9.1.8 32-bit server client. No issues.

I have to put everything back where it belongs at the moment to get ready for the family to come home, but next step is I will again make a backup copy of my production 9.1.8 server, and then install 9.2.1 32-bit server directly onto my 9.1.8 32-bit server, and check the behaviour of the server client and the extenders in the movies section.

After I take a run at that, I'll reset again, and install the 64-bit server directly over my production 32-bit server to see what happens.

I'll fill you in as I test out the various scenarios.

Tnx
-Steve

Update #1:

The extender running against the fresh 32-bit 9.2.1 server seems to have settled down. The two blank movie posters, when I went into the metadata for each, even though there were acceptable posters selected, I changed the default poster to be a different poster, and now those movies are showing their posters. No more blanks, and the spinning circles have stopped. Will carry on with the "drop the 32-bit 9.2.1 server right on top of the production 32-bit 9.1.8 server later this evening.

[Tiki]

Out of curiosity, did you notice if it is always the same images that give you problems? Did you happen to check them to see if there is something different about the images that work vs those that don’t? For example, maybe the images that work are jpeg and those that don’t are png or something like that.

[SteveW]

Originally Posted by Tiki

Out of curiosity, did you notice if it is always the same images that give you problems? Did you happen to check them to see if there is something different about the images that work vs those that don’t? For example, maybe the images that work are jpeg and those that don’t are png or something like that.

Each time after a restart, some were the same and some were different...

[SteveW]

Originally Posted by SteveW

So have been digging further into this...

There are a boatload of differences between my 64-bit properties file and my production 32-bit, so not sure where to even start there. A couple under UI_Disable stood out, so I tried those, no diff. ...

Update #1:

The extender running against the fresh 32-bit 9.2.1 server seems to have settled down. The two blank movie posters, when I went into the metadata for each, even though there were acceptable posters selected, I changed the default poster to be a different poster, and now those movies are showing their posters. No more blanks, and the spinning circles have stopped. Will carry on with the "drop the 32-bit 9.2.1 server right on top of the production 32-bit 9.1.8 server later this evening.

So here's what I've found (Production = Ver 9.1.8 32-bit Installed at C:\SageTV)


Trial 1) 64-bit fresh 9.2.1 install in new directory on production server, prod wiz.bin copied in, rest reconfigured to match productions:

Result: Aforementioned spinning circle problem while browsing through DVD/BD list. Disabling CMT, Phoenix FanArt, and Phoenix Core, as well as other plugins did not resolve. Default folder.jpgs that core Sage tried to load sometimes loaded, sometimes didn't, spinning circles. Happened both on server client and HD-300s. Got to the point where the SageTV server software would have to be restarted due to constant spinning circle.


Trial 2) 32-bit fresh 9.2.1 install in existing C:\SageTV directory (old 32-bit version temporarily copied out and C:\SageTV directory emptied before install):

Result: A couple blank fanart posters on HD-300. Server client had no issue. Was able to choose in edit metadata fanart menu a different poster, which then fixed the blanks (this did not work in the 64 bit version). Some spinning circle, but then it settled out after a few minutes and seemed fine afterwards.


Trial 3) 64-bit install over top of production 9.1.8 32 bit server

Result: That turned into a huge dumpster fire. Errors on install, SageTV server wouldn't load due to error. Somehow my USBUIRT driver in SYSWOW64 got mangled. Took awhile to figure that out. Don't do this...


Trial 4) 32-bit 9.2.1 installed on top of production 9.1.8 32-bit server

Result: No noted issues. Moving up and down movie list is speedy on the server client. On the HD-300 seems a little slower than it was in 9.1.8... seems to be more pauses after each page.


So currently, for me, there are issues trying to run the 64 bit server and browsing movies. TV seems OK and speedy (fanart shows). There are quite a few differences between my production sage properties file and the one in the 64 bit directory, even after I reconfigure everything to look the same as much as possible. Looks like my only option now is to try copying my production Sage properties file over from C:\SageTV\SageTV to C:\SageTV64\SageTV, and try a search and replace from C:\SageTV to C:\SageTV64... and see if something in there makes a difference. The only other obvious difference is the 32 bit version of Java versus the 64 bit version...


-Steve

[jusjoken]

The is so much going on in those logs that its too hard to trace the issue.

Could you stop the 64 bit server, start it, do the steps to see the hang and then exit and post those logs.

I also see you are using some type of plugin for fanart that uses phoenix calls as i see the phoenix cache being built... so the phoenix log under logs folder if there is one may help too.

k

[jerrym]

Some thoughts.

If you are using Phoenix, you may want to check the Plugin configuration for the Phoenix Core Services.

Make sure the directory for Fanart exists and has enough space available to it.

It also has settings for scaling, resizing, etc... that could contribute to performance.

If you are running Gemstone, there is an option to pre-build the Cache so that it doesn't try to build it on the fly as you are using it.

[SteveW]

Ok so after some more testing, I know what's going on here. I just don't know why, or how to fix it. I'll try to keep this simple as this may get confusing.

So for reference, I'm currently using 32-Bit SageTV server 9.2.1 with no issues I've noticed.

I assume most of us know, that natively in SageTV, under the DVD/BD menu, Sage will show you the movie titles, and as graphic or poster for the title, it will look for a folder.jpg from the movie's directory to display. If it doesn't find it, it seems to generate a thumbnail from an early part of the movie, or it will show a Bluray Disk logo or DVD logo until it loads one of those. If you scroll up and down the list of all the movies, it seems to load and unload these images and regenerates them on the fly - they don't seem to get stored in a cache - or certainly not all of them when you have 350 movies. Sometimes the images will reappear when you scroll over, sometimes they'll just sit there with the Bluray logo or DVD logo. On an HD-300, the behaviour is the same if not slower. See my attachment "Capture1" as an example of what I you see in the DVD/BD menu on the server client and the extender. To show that, I simply turned off JREkiwi's Fanart For SageTV Plugin 1.0.8, rather then disabling all of Phoenix core.

So when JREkiwi's plugin is enabled in 32-bit SageTV 9.2.1, it allows the nice Phoenix fanart to be shown again - you see the nice movie posters for each movie title, and matching fanart background above the description on the right above the details - see my attachment "Capture2" as an example. You see the same on the server client and the HD-300, and everything runs as it should. Again, HD-300 is a little slower rendering, but not very noticeable.

Now when I run the same plugins, all connected to the same fanart directory, in the 64-bit SageTV server, something different is happening - see my attachments "Capture3" and "Capture4". For some reason, the Phoenix fanart is showing incorrectly. Most of the movie posters show fine, but some of them show as just black. On top of that, instead of the phoenix fanart backgrounds being displayed on the right correctly above the movie details, what's being displayed are the folder.jpg images or movie thumbnails like you'd see for the posters if fanart is off (again reference my "Capture1"). As you scroll up and down the posters on the server client (and it's worse on the extender), you start getting more delays, more spinning circles as it tries to display the fanart, and eventually Sage grinds to a halt with a spinning circle. You have to kill the 64 bit SageTV process to close it.

I can't find any setting Phoenix setting that would be causing this. I even took my 32-bit sage.properties file, replaced the one in the 64-bit directory with my one from the 32-bit directory, and did a search and replace of C:\SageTV with C:\SageTV64 so that the properties file would work, and same DVD/BD menu behaviour is still happening in the 64-bit server, so I don't think it's a different plugin setting, unless some of them are hiding settings somewhere else in another file.

The 64-bit SageTV log file is again too big to attach, so I'm making it available at:

https://drive.google.com/open?id=1Bq...DIPxiw7S2_1lU4

This 64-bit log file is me browsing up and down the DVD/BD menu of movie titles, causing it to load and unload the graphics with each movie title. Only big difference in the 64-bit log that I notice off hand is a lot of mentions of "Hang Time" that seem to line up with spinning circles.

Hopefully this gives you fine folks something to work with in chasing this. Many thanks!
-Steve

[wnjj]

I've just started to look at this a little a found this interesting bit of code:

        while (!imageStats[0] && Sage.eventTime() - startWait < 30000)
        {
          try
          {
            imageLock.wait(5000);
          }
          catch (InterruptedException e)
          {}
        }

I noticed one of your really bad hangs stopped just above 30 seconds which matches this timeout loop. The code above is inside ImageUtils.java and is called right before you see the "ImageUtils creating BI copy" messages. This is part of the "ensureImageIsLoaded" function which incidentally is optional with this property setting: ui/load_images_with_awt_toolkit. Setting that to "false" would disable this code.

That said, I have no clue what this code all does and why it would be messed up on 64-bit other than the JVM is different and could have a bug in 64-bit.

Also, for what it's worth, I have tons of issues with all all black pictures in my photo albums on 32-bit HD300. Sometimes they load eventually, sometimes never. There is no spinning circle though and it doesn't affect the UI responsiveness. Maybe related...maybe not.

[SteveW]

Originally Posted by wnjj

Also, for what it's worth, I have tons of issues with all all black pictures in my photo albums on 32-bit HD300. Sometimes they load eventually, sometimes never. There is no spinning circle though and it doesn't affect the UI responsiveness. Maybe related...maybe not.

Can’t say that i’ve checked the Photo’s menu since upgrading. I’ll have to have a look at that.

The Phoenix fan art for TV seems fine as well - it’s just movies for some reason. Phoenix fan art is rock solid in movies in 32 bit. Is this maybe an issue with JREkiwi’s plugin handling the fan art correctly in the 64 bit code? I could also try a different version of 64 bit java if you think it’s worth a try.

-S

[wnjj]

Can you try setting that register setting to false?

[SteveW]

Originally Posted by wnjj

Can you try setting that register setting to false?

Sorry, yes. Meant to say I'll be trying that as well. Will give it a try when I get home from work today.

[wnjj]

Originally Posted by SteveW

Sorry, yes. Meant to say I'll be trying that as well. Will give it a try when I get home from work today.

Also, was replying on my phone before so brief. To answer your question, plugins are Java and Java doesn’t have the concept of 32 or 64 bit since it’s a VM.

Differences can only be within native code or the Java engine itself.

[JREkiwi]

Steve, can you tell me what the value in sage.properties is for phoenix/mediametadata/fanartCentralFolder

It doesn't look right in the logs

John