Easy way to speed up Fanart browsing

[Dargason]

Thanks, I just needed a ballpark. My collection is quite a bit smaller than yours, so I shouldn't need to worry about it yet.

[routerunner]

Originally Posted by Dargason

Thanks, I just needed a ballpark. My collection is quite a bit smaller than yours, so I shouldn't need to worry about it yet.

I mean the worst thing that can happen is that you run out of space on the RAM disk and therefore no more thumbs are generated, so you just need to increase the size accordingly and re-run the script (which will clear the cache as well).

[Fuzzy]

Originally Posted by routerunner

I mean the worst thing that can happen is that you run out of space on the RAM disk and therefore no more thumbs are generated, so you just need to increase the size accordingly and re-run the script (which will clear the cache as well).

Another reason caching is just a better option...

[Dargason]

Ok, I got some time to play with this a bit, and I've run into a few issues.

First, I noticed that my SageTV installation is in "C:\Program Files (x86)\SageTV" so I modified the 5th line to reflect this.

Then, I noticed that I don't have a "Fanart" directory in "SageTV\SageTV\Phoenix". But since the part of the script that refers to it is REMed, I guess it doesn't matter.

The OSFMount lines work fine, but I have to "Run As Administrator" or I get access denied errors.

After that is where I get problems.. The purpose of the next few lines is a bit murky to me, but it looks like you're deleting the existing userdata directory, and replacing it with a soft link to one that is on the ram disk. Is that correct? If so, I'm a bit concerned because I have other things in that directory that aren't strictly cache -- I have ADM configuration settings too. Maybe this should be userdata\Phoenix?

It fails anyway... maybe it doesn't like the spaces I have in the path name?

The next sequence of lines appear to do the same thing for the Thumbnails, but they fail too, probably for the same reason. In this case, I tried wrapping the path name in "double quotes" and it failed because "The process cannot access the file because it is in use by another process." Probably because Sage is running on an extender in the other room.

Lastly, I noticed that my userdata\Phoenix\cache directory is over 2GB in size... is that normal?

Thanks again... there are a few things that I need to get fixed but this is looking promising so far!

Here's a copy of the script and its output:

C:\Program Files (x86)\SageTV>REM

C:\Program Files (x86)\SageTV>REM This batch will create a RamDisk via OSFMount
and copy the Fanart into for speed access

C:\Program Files (x86)\SageTV>REM

C:\Program Files (x86)\SageTV>set OSFMountDirectory=C:\Program Files\OSFMount\

C:\Program Files (x86)\SageTV>set JunctionDirectory=C:\Program Files\Junction\

C:\Program Files (x86)\SageTV>set SageTVDirectory=C:\Program Files (x86)\SageTV\
SageTV\

C:\Program Files (x86)\SageTV>set FanartSource=C:\Program Files (x86)\SageTV\Sag
eTV\STVs\Phoenix\Fanart\

C:\Program Files (x86)\SageTV>set FanartDestination=F:

C:\Program Files (x86)\SageTV>REM RAM Disk Size in Mb

C:\Program Files (x86)\SageTV>set RAMDiskSize=256

C:\Program Files (x86)\SageTV>REM Accepted values (as written): 512, 1024, 2048,
 4096, 8192, 16K, 32K, 64K

C:\Program Files (x86)\SageTV>set RAMDiskAllocationUnitSize=8192

C:\Program Files (x86)\SageTV>REM

C:\Program Files (x86)\SageTV>REM Creates the RAM Disk

C:\Program Files (x86)\SageTV>REM

C:\Program Files (x86)\SageTV>cd C:\Program Files\OSFMount\

C:\Program Files\OSFMount>OSFMount -d -m F:
Notifying applications...
Flushing file buffers...
Locking volume...
Dismounting filesystem...
Removing device...
Removing mountpoint...
Done.

C:\Program Files\OSFMount>OSFMount -a -t vm -s 256M -m F:
Creating device...
Created device 7: F: -> VM image
Notifying applications...
Done.

C:\Program Files\OSFMount>format F: /FS:NTFS /A:8192 /V:Fanart /Q /Y
The type of the file system is RAW.
The new file system is NTFS.
QuickFormatting 256M
Creating file system structures.
Format complete.
     256.0 MB total disk space.
     253.2 MB are available.

C:\Program Files\OSFMount>pause
Press any key to continue . . .

C:\Program Files\OSFMount>REM

C:\Program Files\OSFMount>REM Copy the Fanart into RAM Disk

C:\Program Files\OSFMount>REM

C:\Program Files\OSFMount>REM cd C:\Program Files (x86)\SageTV\SageTV\STVs\Phoen
ix\Fanart\

C:\Program Files\OSFMount>REM xcopy *.* F:\ /E

C:\Program Files\OSFMount>pause
Press any key to continue . . .

C:\Program Files\OSFMount>REM

C:\Program Files\OSFMount>REM Create a Windows virtual folder for the Fanart cac
he

C:\Program Files\OSFMount>REM

C:\Program Files\OSFMount>cd C:\Program Files\Junction\

C:\Program Files\Junction>rmdir /S /Q C:\Program Files (x86)\SageTV\SageTV\userd
ata
The system cannot find the file specified.
The system cannot find the file specified.
The system cannot find the path specified.

C:\Program Files\Junction>mkdir F:\userdata\Phoenix\cache

C:\Program Files\Junction>junction /accepteula -q "C:\Program Files (x86)\SageTV
\SageTV\userdata" "F:\userdata"

Junction v1.06 - Windows junction creator and reparse point viewer
Copyright (C) 2000-2010 Mark Russinovich
Sysinternals - www.sysinternals.com

Error creating C:\Program Files (x86)\SageTV\SageTV\userdata:
Cannot create a file when that file already exists.


C:\Program Files\Junction>pause
Press any key to continue . . .

C:\Program Files\Junction>REM

C:\Program Files\Junction>REM Create a Windows virtual folder for the Thumbnails
 cache

C:\Program Files\Junction>REM

C:\Program Files\Junction>cd C:\Program Files\Junction\

C:\Program Files\Junction>rmdir /S /Q C:\Program Files (x86)\SageTV\SageTV\Gener
atedThumbnails
The system cannot find the file specified.
The system cannot find the file specified.
The system cannot find the path specified.

C:\Program Files\Junction>mkdir F:\GeneratedThumbnails

C:\Program Files\Junction>junction /accepteula -q "C:\Program Files (x86)\SageTV
\SageTV\GeneratedThumbnails" "F:\GeneratedThumbnails"

Junction v1.06 - Windows junction creator and reparse point viewer
Copyright (C) 2000-2010 Mark Russinovich
Sysinternals - www.sysinternals.com

Error creating C:\Program Files (x86)\SageTV\SageTV\GeneratedThumbnails:
Cannot create a file when that file already exists.


C:\Program Files\Junction>pause
Press any key to continue . . .

[routerunner]

Goodness me what a post... OK, I'll try to answer to all your questions, please see my comments below.

Originally Posted by Dargason

After that is where I get problems.. The purpose of the next few lines is a bit murky to me, but it looks like you're deleting the existing userdata directory, and replacing it with a soft link to one that is on the ram disk. Is that correct? If so, I'm a bit concerned because I have other things in that directory that aren't strictly cache -- I have ADM configuration settings too. Maybe this should be userdata\Phoenix?

This is correct, however because you have other information which I don't have and you want obviously keep it, you need to focus on the scope a little bit. Basically you need to find the inner directory you want to replace with the soft link, make that directory on the fanart RAM disk and point that directory to the original place in C:, so for example:

suppose your inner directory in C: is:

C:\Program Files\SageTV\SageTV\userdata\Phoenix\cache\imagetrans

then you need to create the same directory in your RAM disk called:

F:\imagetrans

and finally link that directory back to your original in C:, so that any attempt to R/W into

C:\Program Files\SageTV\SageTV\userdata\Phoenix\cache\imagetrans

will effectively R/W into:

F:\imagetrans

Same thing applies to the "CachedThumbnails", but that would be easy as what is in the script already should be fine. Deleting your inner directory as per the script will not delete others directory/files that branches from the same tree.

Lastly, I noticed that my userdata\Phoenix\cache directory is over 2GB in size... is that normal?

Well, it looks like your have both scaling options enabled (Posters & Backgrounds scaling), which I personally found overkilling. I run both options disabled (I asked the plugin developer the favour to implement that way) because my fanart posters and backgrounds are optimised already at the source (not BMT involved).
You'll find out that a small poster (jpg) in the fanart directory gets scaled to a PNG with a huge size for no benefit, I didn't like that at all.

C:\Program Files\OSFMount>cd C:\Program Files\Junction\

C:\Program Files\Junction>rmdir /S /Q C:\Program Files (x86)\SageTV\SageTV\userd
ata
The system cannot find the file specified.
The system cannot find the file specified.
The system cannot find the path specified.

Sometimes Windows can be very cryptic on messages, it looks like either the path needs double quotes or that the "RMDIR" command is not supported on your Windows (7??), please try the exact equivalent "RD". Unfortunately those commands are internal so it is not the case that you need to add the full path to invoke it. Also, are you using the "SageTV Fanart plugin" by JRKiwi? As that directory should be there...

I can see whether I can post a small video on YouTube that shows this mod in action on my system. It is working really well considering that my server is running from a standard 80Gb 2.5" WD 5400 HDD with 128Mb RAM Disk drive, but again I took all the precautions to reduce the caching to the bare minimum and the Fanart file size never exceed the 100Kb on backgrounds (1024x576, no chroma sub-sampling, 50% quality JPG) and 40Kb on posters (256x360, no chroma sub-sampling, 30% quality JPG).

Eddy

[routerunner]

I've performed some speed tests on this RAM disk by using Parkdale (in attachment) and the results are pretty impressive, around 2/3Gb per second ballpark...

[Dargason]

I played around with it a bit more last night, and I got it to the point where I THINK it should be working. Here's the script that I'm using now.

REM
REM This batch will create a RamDisk via OSFMount and copy the Fanart into for speed access
REM
set OSFMountDirectory=C:\Program Files\OSFMount\
set JunctionDirectory=C:\Program Files\Junction\
set SageTVDirectory=C:\Program Files (x86)\SageTV\SageTV\
set FanartDestination=F:

REM RAM Disk Size in Mb
set RAMDiskSize=1024

REM Accepted values (as written): 512, 1024, 2048, 4096, 8192, 16K, 32K, 64K
set RAMDiskAllocationUnitSize=8192

REM
REM Creates the RAM Disk
REM
cd %OSFMountDirectory%
OSFMount -d -m %FanartDestination%
OSFMount -a -t vm -s %RAMDiskSize%M -m %FanartDestination%
format %FanartDestination% /FS:NTFS /A:%RAMDiskAllocationUnitSize% /V:Fanart /Q /Y

REM
REM Create a Windows virtual folder for the Phoenix Fanart cache
REM
cd %JunctionDirectory%
rmdir /S /Q "%SageTVDirectory%userdata\Phoenix\Fanart"
mkdir %FanartDestination%\userdata\Phoenix\Fanart
mkdir %FanartDestination%\userdata\Phoenix\Fanart\Movies
mkdir %FanartDestination%\userdata\Phoenix\Fanart\TV
junction /accepteula -q "%SageTVDirectory%userdata\Phoenix\Fanart" "%FanartDestination%\userdata\Phoenix\Fanart"

REM
REM Create a Windows virtual folder for the Phoenix\cache\imagetrans cache
REM
cd %JunctionDirectory%
rmdir /S /Q "%SageTVDirectory%userdata\Phoenix\cache\imagetrans"
mkdir %FanartDestination%\userdata\Phoenix\cache\imagetrans\bmtfanart
mkdir %FanartDestination%\userdata\Phoenix\cache\imagetrans\phoenix-20poster
mkdir %FanartDestination%\userdata\Phoenix\cache\imagetrans\phoenix-35banner
mkdir %FanartDestination%\userdata\Phoenix\cache\imagetrans\phoenix-85banner
mkdir %FanartDestination%\userdata\Phoenix\cache\imagetrans\phoenix-bgthumb
mkdir %FanartDestination%\userdata\Phoenix\cache\imagetrans\phoenix-fullbg
mkdir %FanartDestination%\userdata\Phoenix\cache\imagetrans\webfanart

junction /accepteula -q "%SageTVDirectory%userdata\Phoenix\cache\imagetrans" "%FanartDestination%\userdata\Phoenix\cache\imagetrans"
pause

This deletes the original directories, recreates them on the ramdisk, and then puts links in the appropriate places. Everything seems to look right, except that when I run Sage, nothing ever seems to appear on the ram disk... Fanart seems to work (though no faster than before) but it doesn't seem to be using the cache at all. Is there another place that these files are being saved? I checked the configuration settings and it says "userdata/Phoenix/Fanart"...

BTW, no, I don't use a separate plugin for Fanart. Isn't all that built into BMT/Phoenix?

Edit: Another thought. Any reason I can't just point the central fanart repository to the F: drive and avoid all the business with the links? Or would that mean I'd have to re-fetch the entire collection every time I reboot?

[Fuzzy]

Pointing teh fanart directory to the F: drive works for the initial pull of fanart, but most all the fanart systems in place utilize caching in the userdata or cache folder, so subsequent calls would pull the images from there instead of the ram drive.

[Fuzzy]

Originally Posted by Dargason

Fanart seems to work (though no faster than before) but it doesn't seem to be using the cache at all. Is there another place that these files are being saved? I checked the configuration settings and it says "userdata/Phoenix/Fanart"...

I wonder if by chance your fanart is being generated in a virtualstore folder (sage initially had problems dealing with vista/win7 UAC redirects).

Just saw your note on the plugins in use. BMT/Phoenix provides the download of the fanart, and provides the caching mechanism, but it is up to the subsequent UI Mod plugin that displays the fanart to actually utilize the cache features. What UI Mod plugin are you using to display the fanart in the UI?

[Dargason]

I wonder if by chance your fanart is being generated in a virtualstore folder (sage initially had problems dealing with vista/win7 UAC redirects).

Yeah, I thought of that too, and checked it before I left work this morning. I didn't see SageTV stuff in the virtualstore directory, but I didn't have time to look extensively.

Just saw your note on the plugins in use. BMT/Phoenix provides the download of the fanart, and provides the caching mechanism, but it is up to the subsequent UI Mod plugin that displays the fanart to actually utilize the cache features. What UI Mod plugin are you using to display the fanart in the UI?

I just use the Phoenix STV or Diamond STV. I didn't realize anything else was required. Maybe my performance is slow because nothing is being cached?

So, basically BMT fills up userdata/Phoenix/Fanart, and then the UI caches the stuff at userdata/Phoenix/cache? If this is true, what I want is to create a ramdisk version of the cache folder, and leave the Fanart folder alone, right?

[routerunner]

Originally Posted by Dargason

So, basically BMT fills up userdata/Phoenix/Fanart, and then the UI caches the stuff at userdata/Phoenix/cache? If this is true, what I want is to create a ramdisk version of the cache folder, and leave the Fanart folder alone, right?

Precisely, I use the "Fanart SageTV plugin" made by JREkiwi which populates the "Phoenix\cache\imagetrans" directory and that directory is now in the RAM Disk. Also, as I said before, I partially use the BMT as I customise the size of every image for even better performances.

The result can be observed in this YouTube file I've just uploaded HERE

[Dargason]

Thanks. Obviously my understanding of how and where these files are located is pretty limited.

My imagetrans directory has at least a gigabyte of stuff in it... since I don't use JREKiwi's plugin, can I assume that the Phoenix UI caches stuff here as well?

On a related topic, I didn't see anything in the BMT configuration that allows me to determine file sizes, image qualities, etc. Is this something that JREWiki's plugin does? I would gladly sacrifice some image quality to speed things up.

[routerunner]

Originally Posted by Dargason

My imagetrans directory has at least a gigabyte of stuff in it... since I don't use JREKiwi's plugin, can I assume that the Phoenix UI caches stuff here as well?

This is what I think is more logical, but I'm not using Phoenix Fanart at all so my knowledge is pretty limited. Also, I think this thread should be moved from here as is not strictly related to Phoenix or its incarnations, but just Fanart...

On a related topic, I didn't see anything in the BMT configuration that allows me to determine file sizes, image qualities, etc. Is this something that JREWiki's plugin does? I would gladly sacrifice some image quality to speed things up.

The BMT or the JREwiki plugin's doesn't allow you to specify those parameters unfortunately. My workflow is different, but if you're interested then send me a PM with your email address and I'll be more than happy to discuss in details my workflow with you which is a balanced compromise between quality and speed. Did you see the video I posted? You cannot tell from the video which I captured with my wife's cheap camera, but the Fanart are pretty high quality and on a 110" screen they look stunning.

Eddy

[Dargason]

I just watched the video. Very impressive!

I'll play around with the ramdisk caching a bit more and see if I can figure out how to get it working...

[routerunner]

Originally Posted by Dargason

I just watched the video. Very impressive!

I'll play around with the ramdisk caching a bit more and see if I can figure out how to get it working...

On a second look at your script you shouldn't do this really as you should leave your fanart repository on the SSD otherwise everytime you power cycle your server you have to refetch everything.

REM
REM Create a Windows virtual folder for the Phoenix Fanart cache
REM
cd %JunctionDirectory%
rmdir /S /Q "%SageTVDirectory%userdata\Phoenix\Fanart"
mkdir %FanartDestination%\userdata\Phoenix\Fanart
mkdir %FanartDestination%\userdata\Phoenix\Fanart\Movies
mkdir %FanartDestination%\userdata\Phoenix\Fanart\TV
junction /accepteula -q "%SageTVDirectory%userdata\Phoenix\Fanart" "%FanartDestination%\userdata\Phoenix\Fanart"

Also, why did you remove the "GeneratedThumbnails"? Is very useful for photo browsing also is a good way to check whether your script works, just browse some photo and if that directory gets populated on the RAM disk as well as on your SSD then the virtual link works.

Also, did you try the script from the command line? Does the "RMDIR" command works or need to be replaced by "RD" as per my suggestion?

EDIT: Just found this on another thread:

C:\Program Files\SageTV\SageTV\userdata\Phoenix\cache\videothumb

Is maybe where your video/movie thumbnails get cached?

[Dargason]

I just took out the generatedthumbnails thing because I was simplifying things... I'll put it back in once I get things working.

The RMDIR command works fine, but you have to enclose the path in double quotes if it has spaces in it.

Yes, now that I understand what the directories are for, I am now only caching the phoenix/cache/imagetrans directory, and not the Fanart directory itself.

So now I have the imagetrans directory on the ramdisk, and it gets populated with stuff when I run Sage on the server client, but running Sage on an extender (at least my HD200) didn't seem to do it. Do extenders cache in a different place?

I didn't have time tonight to do much testing, so I'll let it run the way it is for a while.

[routerunner]

Originally Posted by Dargason

So now I have the imagetrans directory on the ramdisk, and it gets populated with stuff when I run Sage on the server client, but running Sage on an extender (at least my HD200) didn't seem to do it. Do extenders cache in a different place?

It might be possible that the plugin uses the already generated thumbnails for the server on the HD200 as well. One of the beauty of the JREkiwi plugin is that it generates different thumbnails for different type of client/extender in separate directories, as every client/extender runs at different resolution. For instance thumbnails generated for HD300 which runs at 1280x768 will get scaled down for the HD200 which runs at 1024x576, but they wont be upscaled in the opposite case, so depending upon which client/extender you are using to generate the thumbnail in the first place it will influence the display on others client/extenders, and the result can be really bad visually, also this is more noticeable between software clients which can run at more different resolutions and not always at full screen.

I recommend you to use your HD200 first to see whether the directories gets populated so you know whether is the case or not.

EDIT: To clarify, the above thumbnails generation mechanism was happening with Phoenix fanart prior the JREkiwi plugin whom did merge two previous plugins implementation (SageTV v7 Fanart and Fanart for MC), I've no idea whether the existing Phoenix Fanart incarnation does this or not.

[Dargason]

I see that I wrote that in a misleading way... the order of events was

1) create empty ramdisk cache
2) Play around with movie wall on HD200 to see if it populates the cache
3) discover that there is nothing being added to the cache
4) start up the client on the server machine and play around with movie wall
5) observe that there are now files in the cache.

I'll try this again tonight and see what happens, but I'll try playing with the HD300 as well.

btw, phoenix runs super fast on the client machine once all the images are cached.

[routerunner]

Originally Posted by Dargason

I see that I wrote that in a misleading way... the order of events was

1) create empty ramdisk cache
2) Play around with movie wall on HD200 to see if it populates the cache
3) discover that there is nothing being added to the cache
4) start up the client on the server machine and play around with movie wall
5) observe that there are now files in the cache.

I'll try this again tonight and see what happens, but I'll try playing with the HD300 as well.

btw, phoenix runs super fast on the client machine once all the images are cached.

I'm very happy you start to see some benefits

Have you noticed this comment below as well?

EDIT: Just found this on another thread:

C:\Program Files\SageTV\SageTV\userdata\Phoenix\cache\videothumb

Is maybe where your video/movie thumbnails get cached?

[Dargason]

Yes, I saw the comment about the videothumbs directory, but I didn't have a chance to check on that until this evening. I don't have a videothumbs directory.

I repeated the test I did before... I deleted files from the cache, then played around with the HD200 and then the HD300. I didn't get any new files added to the cache in either case. Then I played with the client that is installed on the server machine and the cache repopulated itself.

It seems pretty clear that the files are being cached somewhere else for extenders (or not being cached at all... but it acts like there's a cache somewhere).