Easy way to speed up Fanart browsing

[routerunner]

All,

if you're looking for speeding up the Fanart browsing on extenders by caching the whole fanart directory on a RAM Disk drive without spending a penny, then please read through:

• Download OSFMount from here
• Install the OSFMount with defaults, but please do not run the OSFMount as requested during the installation process.
• Unzip the "FanartRamDisk.zip" file in attachment into your SageTV installation folder.
• Edit the "FanartRamDisk.bat" batch file and configure your RAM Disk drive size, Fanart source directory and RAM Disk drive letter as appropriate (Default is F:\).
• Run the "FanartRamDisk.bat" batch file (WARNING! The RAM Disk drive is formatted, so please make sure the assigned drive letter is not used already).
• You should now have a copy of your current fanart directory into the RAM disk drive.
• Configure your extender's fanart directory in system setup to point to the root of your newly created RAM Disk drive.
• Browse your video collection and see the improvement.

To update the fanart RAM Disk cache, just run the batch script again or automate the process as you wish.

There is a parameter in the batch script that you need to experiment a little bit and is the "RAMDiskAllocationUnitSize". For my usage I noticed that 8192 bytes is a good compromise, but it depends on the average size of your Fanart files really. Valid values are only the one listed and must be entered as is, just change the value in the batch file, run it and browse to see the difference.

Please let me know whether it did help...

Enjoy!
Eddy

[Fuzzy]

Based on the batch file, it looks like you are only redirecting the source fanart folder to the ramdrive. This is only partially useful, as all the fanart heavy UI's use scaled caching of the images, which in most cases are stored in the SageTV/userdata or SageTV/cache folders. So, while it might be faster on the very first time an image is accessed, it is then back to harddrive speeds on subsequent loads. You'd be better off leaving full fanart folder alone, and ramdisking the cache folder, or what i do, and just run a massive disk cache that is larger than pretty much everything that is accessed by sage.

[routerunner]

Originally Posted by Fuzzy

Based on the batch file, it looks like you are only redirecting the source fanart folder to the ramdrive. This is only partially useful, as all the fanart heavy UI's use scaled caching of the images, which in most cases are stored in the SageTV/userdata or SageTV/cache folders. So, while it might be faster on the very first time an image is accessed, it is then back to harddrive speeds on subsequent loads. You'd be better off leaving full fanart folder alone, and ramdisking the cache folder, or what i do, and just run a massive disk cache that is larger than pretty much everything that is accessed by sage.

Partially true, in my case the Fanart background scale and poster scale options are disabled and all files in the Fanart directory are already of a good size so that they never get scaled (a part form the small background which get only scaled after you stop browsing for more than 1 second and stored into the Fanart cache), this way I can take full advantage of the RamDisk. I've tried to redirect the SageTV/userdata or SageTV/cache folders into the RamDisk as well, by changing the path in the Sage.properties without success as it looks like that those directories are always relative to the Sage installation dir, but very happy to be proven wrong

Bear in mind that this solution is meant to be a free option, not willing to shed $80 dollars on what your're using, which since you said you have your write cache disabled I don't understand how it could be on any help on ramdisking the Fanart cache folder, unless I completely misunderstood

However, if anyone is brave enough (and got an UPS), and want to do a similar thing as you do, you can still use this free solution like this:

- Server PC starts up in the morning
- Create the RamDisk
- Copy the entire SageTV directory into RamDisk for the day.
- Run SageTV from RamDisk.
- On the evening stop SageTV service copy the Win.biz/Wiz.bak and properties files back into HDD.
- Switch server PC off for the night.

Eddy

[Fuzzy]

You should be able to mount your ramdrive to the sage cache folder location, that way it is transparent to sage. Not as smooth or simple as a real ram cache, but it should work.

Still, nothing compares to a write-through mirror mode cache of an entire volume.. :-)

[routerunner]

Originally Posted by Fuzzy

You should be able to mount your ramdrive to the sage cache folder location, that way it is transparent to sage. Not as smooth or simple as a real ram cache, but it should work.

Are you talking about this which allows to make a Windows folder virtual? With Linux is dead easy...

[routerunner]

OK, I'm going to use Windows junctions to cache the "userdata\Phoenix\cache" into the RamDisk tomorrow (time to go to bed for me now) and if it works properly will post a new batch file...

Eddy

[Fuzzy]

I was referring to actually mounting the ramdisk itself to C:\Program Files (x86)\SageTV\SageTV\cache. Volumes don't HAVE to be mounted to drive letters, that's just the convention. They can also be mounted to any existing (empty) NTFS folder, just like in Linux.

That said, this is very inflexible, and using directory junctions would be more versatile (since you'd be able to put more than one folder on a ramdisk).

Speaking of flexible, FancyCache is 'free' while in beta - might be worth a look.

[routerunner]

Originally Posted by Fuzzy

I was referring to actually mounting the ramdisk itself to C:\Program Files (x86)\SageTV\SageTV\cache. Volumes don't HAVE to be mounted to drive letters, that's just the convention. They can also be mounted to any existing (empty) NTFS folder, just like in Linux.

Apparently cannot be done with OSFMount, the volume produced is not shown in the "Disk Management"

That said, this is very inflexible, and using directory junctions would be more versatile (since you'd be able to put more than one folder on a ramdisk).

Agree and this is what I'm going to try...

[routerunner]

I managed to get the Windows directory junction working, therefore now even the userdata\Phoenix\cache directory is transparently cached in the RamDisk together with fanart.

I'll do some testing tonight and report back results...

Eddy

[routerunner]

I forgot to mention that this solution is currently designed for extenders only; if you want to enable the SageTV clients to use the cached Fanart as well you must enable the Windows share on the newly created RamDisk and change the Fanart UNC path obviously.

Please look here for instructions on how to do it from the command line...

Eddy

[wayner]

Will this still speed things up if your Sage OS is on a SSD?

[routerunner]

Originally Posted by wayner

Will this still speed things up if your Sage OS is on a SSD?

Generally speaking nothing is faster than memory, however it all depends on:

- How good is the OFSMount implementation (no idea).
- How big are you fanart files.
- Whether you have the scaling of your fanart enabled or not (mine are all disabled as I produce the right size at the source ignoring BMT, so posters are 256x360 and backgrounds 1024x576 looking good on a 110" screen)

I had however noticeable improvement without having the Windows junction enabled yet, but doing more tests tonight with the latests script so whether the userdata being cached as well improves even further.

Eddy

[Fuzzy]

Originally Posted by routerunner

I forgot to mention that this solution is currently designed for extenders only; if you want to enable the SageTV clients to use the cached Fanart as well you must enable the Windows share on the newly created RamDisk and change the Fanart UNC path obviously.

Please look here for instructions on how to do it from the command line...

Eddy

Client caching is actually best done locally on the client. Not to say you couldn't just as easily utilize a ramdisk on the client as well (using this same script). Just have to run a sync job between the client and the server's fanart folders.

I'm actually running fancycache on my client, which does a wonderful job of caching the local copy of my server's fanart folder, which is kept in sync using windows offline storage.

[routerunner]

Folks, as promised I've done some testing with the junctions enabled and for the time being I can consider myself happy. I believe the next room of improvement will be is to cache the "WIZ" as well, however not brave enough to mess about it, will you?

Please find in attachment my version of the batch which of course you need to adapt to you circumstances, also the batch contains junctions for the standard thumbnails as well (pictures, recordings).

You might consider to split the file in different parts to allow independent update of the different cache for instance, but is up to you.

The only extra instruction is that you need to download the junction software in attachment as well. Default installation directory of the junction is "C:\Program Files\Junction\" as the batch assumes that directory, just unzip the file content in that directory.

Please have a go and post your result good or bad, here...

Have fun
Eddy

[Fuzzy]

Originally Posted by routerunner

Folks, as promised I've done some testing with the junctions enabled and for the time being I can consider myself happy. I believe the next room of improvement will be is to cache the "WIZ" as well, however not brave enough to mess about it, will you?

I run a read-only cache my entire sagetv installation (along with a few other items). It's no more dangerous than running completely cache-less. Writes (the risky part) are still done direct to disk. I have no concerns about THAT causing corruption. I have, of course, had a couple wiz.bin corruptions over the 7 years I've used the same wiz.bin, but I've always had a backup for it. These corruptions occurred long before I even thought about doing any sort of supplemental caching.

Backup strategy is important whether you cache in ram or not.

[routerunner]

I did more experiment and so far it looks like that the best speed on extenders is achieved by leaving the Fanart repository on HDD/SSD and just put the "cache" and "thumbnails" folder in the RAM Disk as per the second version of the "FanartRamDisk" script. This way I didn't see any "spinning circles" at all

EDIT: I reckon that a possible explanation is because when reading fanart from HDD/SSD they are cached naturally by the RAM on board of the HDD/SSD controller, however if you perform write operations (by generating the fanart cache as well) on the same drive at the same time the RAM on board of the HDD/SSD controller gets invalidated and the interface can clog up easily on SSD and on HDD there also the mechanical movement that makes this delay more evident; by moving everything on RAM disk you have a similar effect but mitigated by the fact that no mechanical part are involved, however by splitting as suggested you get the best of both worlds which has the benefit of not interfering with each other and not performing any write operation on your HDD/SSD.

[Dargason]

Thanks for your work on this. I'm thinking of trying this out this weekend. I have an SSD, but I still get the occasional spinning circles and it's the most annoying thing about Sage right now.

Have you updated the original post with the latest scripts? I haven't had a chance to dig into the files yet and some of the discussion here has me a little lost.

[routerunner]

As requested, please find in attachment version 3 of the FanartRamDisk script. Basically I've just commented out the part that copy the Fanart files into your RamDisk drive, the rest of the script is the same as v2. Also, despite the file size, the script is not evil at all

To summarise you need to do this:

• Download OSFMount from here
• Install the OSFMount with defaults, but please do not run the OSFMount as requested during the installation process.
• Download the junction software in attachment to post #14 of this thread.
• Install the junction software in "C:\Program Files\Junction\" as the batch assumes that directory.
• Unzip the "FanartRamDisk v3.zip" file in attachment into your SageTV installation folder.
• Edit the "FanartRamDisk.bat" batch file and configure your RAM Disk drive size and RAM Disk drive letter as appropriate (Default is F:\ WARNING! The RAM Disk drive will be formatted, so please make sure the assigned drive letter is not used already).
• Run the "FanartRamDisk.bat" batch file.
• You should have two directories in your new RAM Disk drive which will be soft-linked to their original place into your SageTV installation directory.

Please let me know how you're getting on...

Eddy

[Dargason]

Excellent, thanks!

I'll give it a try this weekend (or earlier, if I get a chance.) About how much RAM does this require? I have an extra 4GB I can install if needed... just haven't had a reason to do it.

[routerunner]

Originally Posted by Dargason

Excellent, thanks!

I'll give it a try this weekend (or earlier, if I get a chance.) About how much RAM does this require? I have an extra 4GB I can install if needed... just haven't had a reason to do it.

There isn't a rule, it really depends on your usage and how long you keep your server on...for instance take the size of your current "C:\Program Files\SageTV\SageTV\GeneratedThumbnails" directory and "C:\Program Files\SageTV\SageTV\userdata\Phoenix\cache\imagetrans" sum them together and multiply by two, that will give you some idea as a start.

In my case my movie collection is about a couple of thousand movies, however my server is shutting down every night and my current RAM Disk size is just 128Mb and is a huge size compared on what I'm really using just browsing my movie collection or recordings.