Gemstone laggy with new SageTV v9 install

[crarbo1]

I have a fresh install of SageTV v9 and installed the Gemstone plugin as I liked it on v7.

I'm seeing the main menu screens being laggy. What I mean is after I load the Gemstone UI, it has "TV" highlighted, when I hit the right button to see the submenu for TV, it takes about 3-4 seconds to show the submenu and then another second to highlight the default view. If I select that view, it does go into it and all is smooth. If I go back, it takes the 3-4 second period. This also happens if I navigate through the main menu selections, after the first 3-4 second pause to go from say "TV" to "Videos", the scrolling is smooth.

I go back to the revert back to the default UI and everything is smooth.

I didn't have this issue on my v7 install, but I wasn't sure if it was a setting I had changed in it to make it better that got reset when I did a fresh install of v9.

Any help is appreciated.

Thanks,
Chuck

[stuckless]

Originally Posted by crarbo1

I have a fresh install of SageTV v9 and installed the Gemstone plugin as I liked it on v7.

I'm seeing the main menu screens being laggy. What I mean is after I load the Gemstone UI, it has "TV" highlighted, when I hit the right button to see the submenu for TV, it takes about 3-4 seconds to show the submenu and then another second to highlight the default view. If I select that view, it does go into it and all is smooth. If I go back, it takes the 3-4 second period. This also happens if I navigate through the main menu selections, after the first 3-4 second pause to go from say "TV" to "Videos", the scrolling is smooth.

I go back to the revert back to the default UI and everything is smooth.

I didn't have this issue on my v7 install, but I wasn't sure if it was a setting I had changed in it to make it better that got reset when I did a fresh install of v9.

Any help is appreciated.

Thanks,
Chuck

In my experience, lagginess is caused by a couple of things...
1. cpu starvation
2. very large fanart
3. disk io

I rarely see #1, but if the sever is old enough, then it might show up if the server is doing other stuff.

If you are seeing lagginess when transitioning menus, I'm thinking it might be disk IO. I had a case like this recently, where my drives would spin down... so on a multi-drive machine, all of sudden sagetv would access something on another drive, and it would take 2-10 seconds before something would happen. I fixed it in my case by preventing my drives from spinning down.

In gemstone I also experience lagginess when navigating views... ie, I'd see 1-2 second pauses as I go from item to item, and this was caused by very large fanart... most posters were were 2000x3000 pixels in size and it would take 1-2 seconds to load the image.

Disk IO would also play a role if your server is accessing fanart over the network. ie, you fanart is not stored on the same machine as the server.

[crarbo1]

Originally Posted by stuckless

In my experience, lagginess is caused by a couple of things...
1. cpu starvation
2. very large fanart
3. disk io

I rarely see #1, but if the sever is old enough, then it might show up if the server is doing other stuff.

If you are seeing lagginess when transitioning menus, I'm thinking it might be disk IO. I had a case like this recently, where my drives would spin down... so on a multi-drive machine, all of sudden sagetv would access something on another drive, and it would take 2-10 seconds before something would happen. I fixed it in my case by preventing my drives from spinning down.

In gemstone I also experience lagginess when navigating views... ie, I'd see 1-2 second pauses as I go from item to item, and this was caused by very large fanart... most posters were were 2000x3000 pixels in size and it would take 1-2 seconds to load the image.

Disk IO would also play a role if your server is accessing fanart over the network. ie, you fanart is not stored on the same machine as the server.

Thanks stuckless for the feedback.
I doubt that it was due to the cpu, even though this build is older it has been running great with my SageTV v7 install, which also used the same Gemstone plugin.
I may be related to fanart, as I didn't have it enabled in my v7 install but wanted to see how it worked with my new v9 install. I did disable it via BMT to see if that helped but it didn't.
Not sure about the disk IO part or how to stop them from spinning down.

However, I did see something interesting. I knew that the JVM Heap size could be at play here so I went into my HD300 to see what it was last night while I was having this issue and this is what it had:
JVM Heap Size (Used/Total/Max): 700MB/1038MB/1038MB
I couldn't remember hot to increase it and I didn't have time to search for it in the forums last night, so I left it as it was.

I then had to reboot my server to debug an OpenDCT issue I am having and when I turned the HD300 back on everything was great when going through the menu items. The JVM Heap now said JVM Heap Size (Used/Total/Max): 232MB/441MB/1038MB, so it may have been that.

I'm not sure what those numbers mean or how to change the heap size but I was wondering if you could answer my questions about this.
Is the Max of 1038MB large enough and if not, how do I change it and to what value?
What could have caused it to be so large before the reboot? Was it because I did have fanart enabled at one point and didn't reboot after I disabled it?

Thanks for any and all help,
Chuck

[wayner]

I see this issue from time to time and I have a superfast CPU (i7-6700K) with 16GB of memory and a superfast hard drive - a PCIex4 NVME SSD mounted on the mobo.

[jusjoken]

Originally Posted by crarbo1

I have a fresh install of SageTV v9 and installed the Gemstone plugin as I liked it on v7.

I'm seeing the main menu screens being laggy. What I mean is after I load the Gemstone UI, it has "TV" highlighted, when I hit the right button to see the submenu for TV, it takes about 3-4 seconds to show the submenu and then another second to highlight the default view. If I select that view, it does go into it and all is smooth. If I go back, it takes the 3-4 second period. This also happens if I navigate through the main menu selections, after the first 3-4 second pause to go from say "TV" to "Videos", the scrolling is smooth.

I go back to the revert back to the default UI and everything is smooth.

I didn't have this issue on my v7 install, but I wasn't sure if it was a setting I had changed in it to make it better that got reset when I did a fresh install of v9.

Any help is appreciated.

Thanks,
Chuck

To help identify any issues you need to have sage and gemstone logging on debug and when the issue occurs upload the sagetv and the gemstone log and i can review.

k

[crarbo1]

Originally Posted by jusjoken

To help identify any issues you need to have sage and gemstone logging on debug and when the issue occurs upload the sagetv and the gemstone log and i can review.

k

jusjoken,
Thanks for the feedback. I haven't seen that issue again after I disabled the fanart. I never used it in v7, just thought I would try it out in v9 with Gemstone and had issues. So, my issues went away with I disabled the fanart and then rebooted the server.

Thanks,
Chuck

[phelme]

Originally Posted by crarbo1

I may be related to fanart, as I didn't have it enabled in my v7 install but wanted to see how it worked with my new v9 install. I did disable it via BMT to see if that helped but it didn't.

However, I did see something interesting. I knew that the JVM Heap size could be at play here so I went into my HD300 to see what it was last night while I was having this issue and this is what it had:
JVM Heap Size (Used/Total/Max): 700MB/1038MB/1038MB
I couldn't remember hot to increase it and I didn't have time to search for it in the forums last night, so I left it as it was.
Chuck

It's the fanart. I have a similar problem and I find that a maximum heap size of 1038 just isn't enough for Gemstone when using fanart with my collection of shows and movies. Within 24 hours of use when fanart is enabled of any kind on my HD300's (posters, banners or background) I hit the ceiling and it affects UI performance and server recording gets sketchy too as the memory space is shared. I've messed with various caching settings hoping to quell it but haven't found a solution. Even with garbage collection, just too much needs to be memory resident I guess.

at the moment the only fix is to switch from Windows to Linux so the Java heap size can be raised. until then, as you've found, turning off fanart is the best solution.

[jusjoken]

The latest phoenix release from a few weeks back should help in this as stuckless added some code to adjust the size of fanart to the max resolution of the system (as many are WAY larger which causes much overhead in memory).

I believe you need to force all the fanart to refresh in BMT as well as check the BMT settings as there are a few related new ones.

I have not had time to test this out but stuckless said it made a big difference on the android mini client so I expect similar on the HDxxx clients.

k

[stuckless]

Originally Posted by jusjoken

The latest phoenix release from a few weeks back should help in this as stuckless added some code to adjust the size of fanart to the max resolution of the system (as many are WAY larger which causes much overhead in memory).

I believe you need to force all the fanart to refresh in BMT as well as check the BMT settings as there are a few related new ones.

I have not had time to test this out but stuckless said it made a big difference on the android mini client so I expect similar on the HDxxx clients.

k

Actually I haven't actually done a public release of that version of phoenix, yet

[jusjoken]

Originally Posted by stuckless

Actually I haven't actually done a public release of that version of phoenix, yet

oops.... cat out of bag

[Fuzzy]

Originally Posted by jusjoken

oops.... cat out of bag

Additionally, are you doing the caching differently in gemstone2 so that all cached posters are the same size, regardless of original size? the current method in gemstone of simply scaling it to a percentage of the original id inconsistent, and doesn't make a lot of sense. If percentages are desired, I think going by a percentage of UI resolution would be best, or even just each artifact type to a specific resolution.

[jusjoken]

Originally Posted by Fuzzy

Additionally, are you doing the caching differently in gemstone2 so that all cached posters are the same size, regardless of original size? the current method in gemstone of simply scaling it to a percentage of the original id inconsistent, and doesn't make a lot of sense. If percentages are desired, I think going by a percentage of UI resolution would be best, or even just each artifact type to a specific resolution.

That is how gemstone 1 does it or am i missing something here?

It gets the UI width, which if not found and not overridden then uses a default. then uses a percent of the screen width that the fanart element would consume on the ui, unless overridden, and applies that to the fanart. At least thats how its supposed to be doing it.

k

[Fuzzy]

Originally Posted by jusjoken

That is how gemstone 1 does it or am i missing something here?

It gets the UI width, which if not found and not overridden then uses a default. then uses a percent of the screen width that the fanart element would consume on the ui, unless overridden, and applies that to the fanart. At least thats how its supposed to be doing it.

k

That's how I thought it was being done as well. Stuckless at some point in one of the miniclient threads found that it was instead running a percent of the original image instead (I haven't verified, just going off what he said).

[stuckless]

Originally Posted by jusjoken

That is how gemstone 1 does it or am i missing something here?

It gets the UI width, which if not found and not overridden then uses a default. then uses a percent of the screen width that the fanart element would consume on the ui, unless overridden, and applies that to the fanart. At least thats how its supposed to be doing it.

k

When I was doing testing I was seeing poster sizes that were coming across with heights that were much greater than my UI height, and yet smaller than the original poster size. It almost seemed as if the scaling factor of 80% was being applied to the original image size.

[Fuzzy]

Originally Posted by stuckless

When I was doing testing I was seeing poster sizes that were coming across with heights that were much greater than my UI height, and yet smaller than the original poster size. It almost seemed as if the scaling factor of 80% was being applied to the original image size.

I went back and read what you had written in your miniclient performance thread, and it looks like it was taking the UI Width (1920) and using that as the basis, so with the scaling set at 100%, that meant it was scaling the 2000x3000 posters to 1920x2880 (or something like that), scaling the image so that the final width is x% or the UI width.

[stuckless]

Originally Posted by Fuzzy

I went back and read what you had written in your miniclient performance thread, and it looks like it was taking the UI Width (1920) and using that as the basis, so with the scaling set at 100%, that meant it was scaling the 2000x3000 posters to 1920x2880 (or something like that), scaling the image so that the final width is x% or the UI width.

Actually you are right... it could be that... i would think that the logic for scaling posters ban banners and backgrounds should be separate. ie, it doesn't make sense to create a poster that is larger than the screen's height, so the 80% should be applied against the height and then calculate the width.

In the next release of Phoenix, this is happening automatically when the posters are downloaded (by default, but can be turned off). ie, a 2000x3000 poster will be scaled when it is saved to ensure that the height is not greater than the "preferred" screen height. (gemstone/phoenix will then further scale it in the UI).

For optimal performance for highly graphical UIs (ie fanart UIs), then images should be scaled, per client, to be the exact size you want. This is both efficent in terms of memory storage, but also, smaller images take up less cache, and are less likely to be evicted.

This was the issue in the miniclient. it has 32mb image cache. So even though it was showing a poster that that was only 400 pixels wide, it was sending a 1920x2880 size poster. On the server, sagetv had to load it and cache, and on the miniclient it has to load it and cache (~20mb). So what was happening, was that I'd press down, and it would load the next poster to show, and then sagetv would get the request to cache another ~20mb, and it wouldn't have room, so then it would go through and start evicting the "oldest" objects first (which were likely very small fonts, etc) until it had room to cache the new poster. During this process, there is constant back and forth since the images are removed from the cache, and then unloaded, and this "pause" was taking ~9-10 seconds. After I made changes in the MiniClient to use a larger cache, it was better, but just meant that I hit it further down the list of images. Finally, adding the auto-resize for the posters and backgrounds to ensrue they were an "optimal" source size, the performance started to come under control.

It would be ideal, if in gemstone 2, the images were scaled "exactly" to the size they need to ensure the most optimal use of memory and caching. Using more diskspace vs faster client experience... i think the faster experience wins out

[Fuzzy]

Originally Posted by stuckless

It would be ideal, if in gemstone 2, the images were scaled "exactly" to the size they need to ensure the most optimal use of memory and caching. Using more diskspace vs faster client experience... i think the faster experience wins out

Part of the issue is that often times, UI elements in sage are not a fixed size. They are often percentages of whatever container they are in, so sizing ends up being 'close', but not exact. And scaling an image from just barely larger than displayed will end up looking far worse than scaling an image from much larger than it will be displayed, because there is more information to work with.

[stuckless]

Originally Posted by Fuzzy

Part of the issue is that often times, UI elements in sage are not a fixed size. They are often percentages of whatever container they are in, so sizing ends up being 'close', but not exact. And scaling an image from just barely larger than displayed will end up looking far worse than scaling an image from much larger than it will be displayed, because there is more information to work with.

I get that... I guess what I'm saying is that if you know the image size should be an approx size, then specifying that size (or a little larger) would be preferred. In sagetv, if load 2000x3000 image into a 200x300 area, sagetv will not scale it... the MiniClient gets the 2000x3000 image (~22mb) and it is told the source and dest rectangles and the miniclient scales it down to 200x300. So, any time we can make the images close to the original size, we are going to have much better performance in the UI.

[Fuzzy]

I know that would maximize performance, by minimizing memory footprint, but like I said, shrinking a 2000x3000 image in the cache to 533x800 on the display is going to look a LOT better than shrinking a 2000x3000 image to 720x1080 in the cache and then shrinking that image to 533x800. In fact, the closer the cache and final render sizes are to each other, the worse the scaling will look.

I'm wondering if this is something that could be fixed architecturally, to change it so that the image is scaled to actual rendering size on the server instead of the miniclient.

[stuckless]

Originally Posted by Fuzzy

I know that would maximize performance, by minimizing memory footprint, but like I said, shrinking a 2000x3000 image in the cache to 533x800 on the display is going to look a LOT better than shrinking a 2000x3000 image to 720x1080 in the cache and then shrinking that image to 533x800. In fact, the closer the cache and final render sizes are to each other, the worse the scaling will look.

I'm wondering if this is something that could be fixed architecturally, to change it so that the image is scaled to actual rendering size on the server instead of the miniclient.

It probably could be scaled on the server, but sagetv will likely still have the original 22mb file in memory as welll, and it has a finite cache limit as well, and will start it's own eviction process when it reaches the limit. You are right in that there likely isn't a good answer to this... but I think personally, I'd rather pixelated fanart, than long pauses on the UI (and as of the next phoenix release, no fanart that is downloaded will be larger than the 1920x1080, by default, and for posters, no poster will will exceed a height of 1080. People can turn it off if they want, but out of the box, the images sizes are going to start getting smaller. When I first started doing fanart, posters were like 500px wide... now they are 2000 pixels wide... i get it why... We have 4K TVs, and as such, we want the highest possible display.