STV Import: TVExplorer for the Default STV

[stuckless]

Originally Posted by razrsharpe

I just updated the phoenix api to the latest (was running an ancient 1.7.something version on my client... yes its been that long since i've done any sage tinkering).... now i am getting the error failed to find a cached image for: blah blah errors...

I deleted (renamed) my TVExplorerBanners directory so that images would be recreated.... Banners are getting created, saved to that folder, and displayed in the stv, but I am getting the same error in the phoenix log (fixed the bracket issue on my end...)

2011-01-05 19:56:23,320 [AWT-EventQueue-0] ERROR phoenix.impl.ImageAPI - Failed to find a cached image for: \\svr1\Sage\Fanart\TV\Southland\Banners\84645-g.jpg; tag: TVExplorerBanners

I'll need to dig into this and see if i can figure anything out...

That ERROR was the "red herring" that I was referring to earlier. Based on your logic, you'll see that error at least once per image, since when the GetImage api returns null, then you scale and cache the image. If you are seeing this ALL the time for the same image, then that's not normal. While I log it as an error, it's really a warning at best. GetImage, when it can't find a cached file, will show that "error". It really should be a warn or info, since it's not an error.

[razrsharpe]

ahhhh... i see now... yes i only get that error once per image.... now im stumped though... before i upgraded the phoenix api (from 1.7.something to 2.3.something (today)) tve on my client here was reasonably fast with fanart and all... after upgrading its slow... i cant browse up the list fast like i could before... so i must be doing something not quite right because if i disable displaying the banner (short circuiting all that code) things go super fast again... needless to say I'm gonna have to dig into it more to figure out exactly whats happening...

Skirge - to make tve usable again until i can figure what's going on... disable the banner display... TVE Options -> Browser -> Group Style -> choose one of the "No Banner" options... after i did this (leaving all the "fanart" options enabled) TVE sped up for me again.

[Skirge01]

Thanks for the correction. I again fixed the brackets, after deleting the folders you mentioned. Then, I scrolled through for a good 5 minutes, up and down. It started out around 6 seconds per page up/down, then dropped down to 4 seconds, which is where it was before. I did the math and it could potentially need to create up to 610 banners (61 shows times 10 max banners for BMT to download). It's currently at 545, but it's slow even when no banners are being created (I'm watching the folder).

One other thing I noticed is that the background posters only appear when scrolling UP. When scrolling down, they never appear. If I arrow down, they do, though. Also, if I do just a single arrow down, followed by all page downs, they show up. Probably unrelated, but thought I'd mention it.

EDIT: Looks like I was sitting on this post for far too long. I see the new info now. Glad to know it's not my setup, at least.
EDIT #2: Thanks again guys!

[razrsharpe]

Originally Posted by Skirge01

Thanks for the correction. I again fixed the brackets, after deleting the folders you mentioned. Then, I scrolled through for a good 5 minutes, up and down. It started out around 6 seconds per page up/down, then dropped down to 4 seconds, which is where it was before. I did the math and it could potentially need to create up to 610 banners (61 shows times 10 max banners for BMT to download). It's currently at 545, but it's slow even when no banners are being created (I'm watching the folder)

im no where near 4 or 6 seconds per arrow down... for me its mostly notable when the focused item is the last or first item on the group panel and needing to scroll the table. Yes its slow even though no new banners are created... like i said it has to be something about displaying the banner... i messed around a bit today didn't come up with anything so I'm going to have to do more digging.... One thing i did find out though.... Disabling the banner display speeds things up again... let me know if it works with you too...

One other thing I noticed is that the background posters only appear when scrolling UP. When scrolling down, they never appear. If I arrow down, they do, though. Also, if I do just a single arrow down, followed by all page downs, they show up. Probably unrelated, but thought I'd mention it.

not related... at least i dont think... but good to know.. thanks for doing the debug

[Skirge01]

Disabling the banners, as you suggested, absolutely puts it at ludicrous speed. I can deal without banners... for now.

[Fuzzy]

Was just playing around with my test/dev setup with my malore tweaks(which doens't use any caching yet), and noticed it was VERY slow when the fanart folder was remote (just using the fanart from my production server). As soon as I pointed my fanart folder to a local C: drive copy, it was speedy again. I'm not sure why it would somehow be so much slower over the network than before, but something is slowing down the image grabbing over the network...

Of course, that shouldn't matter with your cached files.

[razrsharpe]

all my fanart is on a unc path (and always has been)... perhaps that is the problem though... even with caching... maybe something weird is going on with just the banner code in phoenix... or because i am displaying several banners at the same time (granted all are cached though) that is making something slow... I'll move my fanart to a local resource and see what happens... My gut though is that it is something in the phoenix core... because before i upgraded the phoenix api last evening things were running fine on my dev setup. I use my dev setup for occasional TV watching in the bedroom so would have definitely noticed if it was taking forever to scroll around tve. I was running phoenix 1.7.something before upgrading to 2.3.something yesterday.

[Skirge01]

The last time I recall it being fast for me was in SageTV V6, which I was still running up until V7's beta 18, I believe. According to Opus' post about beta 18, that was on September 30th. So, if the issue is in Phoenix, then it was probably somewhere around that time that something changed. I figured I'd point that out, just in case it helps to track changes down.

[razrsharpe]

just did some digging to determine which phoenix build i was running... it was phoenix 1.73.1 or 1.73.0 from April 2010 (yes i know suuuuuper ancient )

[stuckless]

Originally Posted by Fuzzy

Was just playing around with my test/dev setup with my malore tweaks(which doens't use any caching yet), and noticed it was VERY slow when the fanart folder was remote (just using the fanart from my production server). As soon as I pointed my fanart folder to a local C: drive copy, it was speedy again. I'm not sure why it would somehow be so much slower over the network than before, but something is slowing down the image grabbing over the network...

Of course, that shouldn't matter with your cached files.

Fanart is extremely IO intensive. The reason is that because phoenix uses a folder model for each fanart type, then you need to first find the folder, then get all files, and then return the first match (if all you want is a single poster). Setting a 'default' poster will speed things up because it will not need to list the files, since it will first try an exact file match based on the default.

This is further compounded by the fact that if you do this for tv shows, then it will first do all work for the current season, and then if nothing is found, it will do it again, for the series.

Originally Posted by razrsharpe

all my fanart is on a unc path (and always has been)... perhaps that is the problem though... even with caching... maybe something weird is going on with just the banner code in phoenix... or because i am displaying several banners at the same time (granted all are cached though) that is making something slow... I'll move my fanart to a local resource and see what happens... My gut though is that it is something in the phoenix core... because before i upgraded the phoenix api last evening things were running fine on my dev setup. I use my dev setup for occasional TV watching in the bedroom so would have definitely noticed if it was taking forever to scroll around tve. I was running phoenix 1.7.something before upgrading to 2.3.something yesterday.

The fanart process for posters, backgrounds, banners, etc, all use the exact some code. The only difference is the paths.

Some versions of phoenix toyed with the idea of using memory caching for the images, but I was finding that it was causing way too much memory overhead, and there was no way (easily) to update the cache without restart sagetv. ie, if you added a new poster it would not be seen until the next sagetv restart. Sure, it could have been fixed in code and still cached, but given that I currently manage over 50,000+ lines of code... its simply too much for one developer, so caching was removed instead of being fixed

[Fuzzy]

So, part of the problem with these being slow might be the HasFanartBanner call. The problem being, that even if a image is cached locally, it still has to reference the UNC path to get the directory listing. I'm not sure how much overhead this is adding, but it's gotta be something. I've found that SyncToy to put the fanart folder on each client speeds things up nicely.

[stuckless]

Originally Posted by Fuzzy

So, part of the problem with these being slow might be the HasFanartBanner call. The problem being, that even if a image is cached locally, it still has to reference the UNC path to get the directory listing. I'm not sure how much overhead this is adding, but it's gotta be something. I've found that SyncToy to put the fanart folder on each client speeds things up nicely.

Yeah... HasFanart basically boils down to GetFanart > 0. Since there is no caching, HasFanart basically does a GetFanart, and if there are items, it returns true. Later when you call GetFanart, it will do all that work again.

Maybe I'll spend some time on the weekend trying to put in a better caching model for the fanart calls.

[Fuzzy]

Well, I'm now thinking there may be a better way to cache these types of displays (my malore, and TVE). If the local cached file is ID'd for the show title, instead of the original file path, it could save on having to do the network lookup (after the first grab, of course). So, instead of a HasFanart to determine if an image is to be shown, it runs a GetImage(ShowTitle, "Banners"). This would be a local call to the cache folder. Then, if it fails (returns a null), that's when the createimage from the getfanart is called. If that also fails, you're then at using a text object. It would end up being the same (slower) speed from the failed createimage(getfanart) on missing banners as the current hasfanart call, but populated cached images would be MUCH faster. without needing to worry about maintaining some sort of localized list of fanart availability.

[razrsharpe]

ahhhh... a little light bulb just went off in my head ... when stuckless started posting about caching and there not being a local cache i was confused because i was thinking about the image cache that is created... then fuzzy cleared me all up with "maintaining some sort of localized list of fanart availability" so then it dawned on me that the caching stuckless was referring to was that list of what (if any) fanart is available....

Does a "default" poster or banner get set automatically? or is that something the user needs to do interactively?

Would using the series poster or banner be faster from an io perspective or not?

I still find it curious that it used to work with UNC paths.... that's the interesting part to me...

Sean, I just want to say thank you. I know you do way too much the way it is and your contributions to this community are huge, so just want to say thank you for helping to understand this problem and to try to come up with a solution

[stuckless]

Originally Posted by razrsharpe

Does a "default" poster or banner get set automatically? or is that something the user needs to do interactively?

No, a default poster does not get set automatically. By default, it's kind of random.

Would using the series poster or banner be faster from an io perspective or not?

All fanart lookups use exactly the same process. There is really only a couple of fanart apis (that accept a lot of parameters), and then for convenience, I created a GetFanartPoster, etc.

I still find it curious that it used to work with UNC paths.... that's the interesting part to me...

I think it should still work with unc paths.. it;ll just be slow. It probably got slower in the last couple of months, because i recall, totally removing the in memory caching when I added the Episode fanart for evilpenguin.

Sean, I just want to say thank you. I know you do way too much the way it is and your contributions to this community are huge, so just want to say thank you for helping to understand this problem and to try to come up with a solution

I have fun with this stuff

[Fuzzy]

So i did some tweaking this morning with implementing some form of banner caching in my malore tweaks. It seems to be working well (well, after first time a banner is displayed). Attached is what I've got going right now.

I fought for a while with this, and I think i found a bug Sean (or at least, I need to verify an intended use). CreateImage was failing if there was no transform (as I don't really want one in my implementation). I tried every iteration of "", null, or even "{}".. all three of which caused the reflection to fail. Is there a transform that will pass through the source untouched? All I'm really looking to do is copy the original file over, unmodified, but was unable to get that going (hence the tiny rounding done in this example code).

[razrsharpe]

Originally Posted by stuckless

No, a default poster does not get set automatically. By default, it's kind of random.

so if a user were to go through and set the default do i need to do something special to make sure i load the default first and then load the others if the default isnt set or doesnt exist? Also is there an api to set the default so one could theoretically automate it from the stv?

I think it should still work with unc paths.. it;ll just be slow. It probably got slower in the last couple of months, because i recall, totally removing the in memory caching when I added the Episode fanart for evilpenguin.

oh it works its just slow.... and yup if there used to be in memory caching and its not there anymore that would explain why it got slow when i upgraded from my ancient version of phoenix

Originally Posted by Fuzzy

So i did some tweaking this morning with implementing some form of banner caching in my malore tweaks.

what about if you have a lot (some) shows that don't have banners (or other fanart)... your code will always try to show retrieve a cached banner that doesnt exist.... does this bring navigating those shows to a standstill? when you have shows without a banner that will cause the phoenix.log to fill up with the "failed to retrieve cached image for... " errors we were seeing yesterday... (and that logging is likely expensive).

Is there a transform that will pass through the source untouched?

probably one of these will work (have not tried any of these but might work from my reading of the phoenix java docs for the image api):

{name:scale, width: -1, height: -1}
{name:opacity, opacity: 1}
{name:rounded, arcSize:0}

Agreed though, a {name:Copy} or {name:scalePercent, percent:1} would be nice (and cleaner)

[stuckless]

There is nothing that you need to do use the default poster. When you call get poster it will first check if there is a default and if not it will go ahead and fetch a random one from the directory.

To set a default poster from the stv calll SetDefaultPoster with a filename path.

Calling get image without a transform is kind of abusing the api for caching I'll see if I support a null transform.

[Fuzzy]

Originally Posted by razrsharpe

probably one of these will work (have not tried any of these but might work from my reading of the phoenix java docs for the image api):

{name:scale, width: -1, height: -1}
{name:opacity, opacity: 1}
{name:rounded, arcSize:0}

Agreed though, a {name:Copy} or {name:scalePercent, percent:1} would be nice (and cleaner)

I looked through the actual google code, and it looks like -1,-1 won't work well (as the code for the -1 uses the value of the other to fill in the scale values). what I've since changes is scaling them to 9999x9999. If the scaled size is larger than the input size, it aborts the scale and returns the inputed image. Still, the problem is this is still needlessly going through the costly parsing of the json string, and shuffling the bufferedimage object around from code to code just to get it back where it was t the beginning. this would be the same if there was a {name: copy} transform. that's why i requested either to detect null transform strings and simply skip the parsing step, or create another overloaded CreateImage call that would do the same. (perhaps a CacheImage(string id,string tag,object image,boolean overwrite) would fit the bill nicely and allow some improved speed.

As for the non-bannered items, yes, it does have to go through a failed call each time a banner is NOT available. I'm still working out the best approach to get around that limitation. either one would have to store a map of non-bannered items, or create transparent dummy images for those shows. Again, haven't gotten to that one yet.

Honestly, i haven't noticed a significant slowdown, though I've only got about 15-20% of my shows missing fanart.

[razrsharpe]

Originally Posted by stuckless

To set a default poster from the stv calll SetDefaultPoster with a filename path.

phoenix_fanart_SetDefaultPoster? relative to the central fanart folder correct?

Calling get image without a transform is kind of abusing the api for caching I'll see if I support a null transform.

certainly is abusing it for caching... but that's not such a terrible thing, right

Originally Posted by Fuzzy

(perhaps a CacheImage(string id,string tag,object image,boolean overwrite) would fit the bill nicely and allow some improved speed.

ahhh i see what you're getting at now... and ya a CacheImage would be nice to bypass all the needless shuffling... you could also scale them to be the size they need to be on a 720p or 1080p display.

As for the non-bannered items, yes, it does have to go through a failed call each time a banner is NOT available. I'm still working out the best approach to get around that limitation. either one would have to store a map of non-bannered items, or create transparent dummy images for those shows. Again, haven't gotten to that one yet.

transparent dummy images sounds like it would work... the problem being that you would need to check for new fanart periodically... otherwise it will continually load the blank image... even when new fanart exists for an item.

Honestly, i haven't noticed a significant slowdown, though I've only got about 15-20% of my shows missing fanart.

mine is over 50% without banners... I'll see how it behaves on my system... it could be letting the GetImage fail all the time is less expensive then the GetFanart methods... so it might be the more efficient way to go about it...


RE: Caching (or getting around file system access slowdowns): thinking out loud here... what about storing a list of fanart in the metadata for an item (similar to the defaultfanart but just a complete list)? Then calls to GetFanart would read that list or create it if it doesn't exist. I guess the problem then is how/when do you update that cache? That could be done with a scan in bmt.