SageTV Community  

Go Back   SageTV Community > SageTV Development and Customizations > SageTV v7 Customizations > Gemstone for v7

Notices

Gemstone for v7 This forum is for discussing the user-created Gemstone custom interface for SageTV.

Reply
 
Thread Tools Search this Thread Display Modes
  #1  
Old 12-03-2012, 04:24 PM
Hunter69 Hunter69 is offline
Sage Aficionado
 
Join Date: Oct 2008
Posts: 301
Speeding up Fanart- Has anyone done this from the Phoenix forum

Here is the thread

http://forums.sagetv.com/forums/show...545#post543545

If you have or if you could help me accomplish this task I would be grateful
__________________
Server: Antec 300, AMD Phenom 9750, 4 gig ram, 4 tb-with pooled recording , Lite on Blu-ray drive
Tuners: 2 HD-PVR, 1 HDHR
Clients: 2 HD200
Reply With Quote
  #2  
Old 12-03-2012, 08:56 PM
smak smak is offline
Sage User
 
Join Date: Jul 2008
Posts: 40
I have been working on it. I loaded up with RAM and installed FancyCache (fuzzy's recommendation) . Once everything is cached works well.

I played around with a ram drive but getting the files pre-cached was challenging.
Reply With Quote
  #3  
Old 12-03-2012, 10:45 PM
Hunter69 Hunter69 is offline
Sage Aficionado
 
Join Date: Oct 2008
Posts: 301
So did you follow a tutorial in order to set this up? I saw fancy cache being talked about but did not search for a tutorial? Any help is greatly appreciated. I can load up as memory as needed. Which fancy cache are you using? Volume or Disk?

Thank you reply and sharing your experience
__________________
Server: Antec 300, AMD Phenom 9750, 4 gig ram, 4 tb-with pooled recording , Lite on Blu-ray drive
Tuners: 2 HD-PVR, 1 HDHR
Clients: 2 HD200

Last edited by Hunter69; 12-03-2012 at 10:55 PM.
Reply With Quote
  #4  
Old 12-04-2012, 05:07 AM
routerunner's Avatar
routerunner routerunner is offline
Sage Icon
 
Join Date: May 2008
Location: Wiltshire, UK
Posts: 1,383
Quote:
Originally Posted by smak View Post
I have been working on it. I loaded up with RAM and installed FancyCache (fuzzy's recommendation) . Once everything is cached works well.

I played around with a ram drive but getting the files pre-cached was challenging.
Maybe a combination of both? Unless you have an UPS having the write cache enabled is not recommended, so you truly end up to have benefit on reading the fanart only, but what about the fanart thumbnails generation that inevitably gets written on disk? I think you can have best of both by using the RamDisk for the thumbnails generation and the disk cache for reading the fanart.

Even better, having two RamDisks, one for the Fanart copy that get's updated from the repository when is necessary and one for the thumbnails, unfortunately the Fanart repository relocation is currently limited because you should be able to specify two directories in the BMT, one for the general fanart repository and one for the fanart to use to display which can then be relocated into a RamDisk which I reckon is the best option by far.

Ideal is to ask Gemstone developers to implement a RamDisk cached solution by default

Eddy
__________________

Automatic Power Off | Squeezeslave | DVB-S Importer | DVB Decrypter & Card Client | Tuner Preroll


Every man is a damn fool for at least five minutes every day; wisdom consists in not exceeding the limit. ~ Elbert Hubbard

Last edited by routerunner; 12-04-2012 at 05:33 AM.
Reply With Quote
  #5  
Old 12-04-2012, 07:02 AM
stuckless's Avatar
stuckless stuckless is offline
SageTVaholic
 
Join Date: Oct 2007
Location: London, Ontario, Canada
Posts: 9,675
Quote:
Originally Posted by routerunner View Post
Maybe a combination of both? Unless you have an UPS having the write cache enabled is not recommended, so you truly end up to have benefit on reading the fanart only, but what about the fanart thumbnails generation that inevitably gets written on disk? I think you can have best of both by using the RamDisk for the thumbnails generation and the disk cache for reading the fanart.

Even better, having two RamDisks, one for the Fanart copy that get's updated from the repository when is necessary and one for the thumbnails, unfortunately the Fanart repository relocation is currently limited because you should be able to specify two directories in the BMT, one for the general fanart repository and one for the fanart to use to display which can then be relocated into a RamDisk which I reckon is the best option by far.

Ideal is to ask Gemstone developers to implement a RamDisk cached solution by default

Eddy
Does Windows support symbolic links? All Cached data in phoenix is written to SAGE_HOME/userdata/Phoenix/cache/, so if you moved the cache to a ramdisk and then symlinked the directory, then would that work?
Reply With Quote
  #6  
Old 12-04-2012, 07:11 AM
routerunner's Avatar
routerunner routerunner is offline
Sage Icon
 
Join Date: May 2008
Location: Wiltshire, UK
Posts: 1,383
Quote:
Originally Posted by stuckless View Post
Does Windows support symbolic links? All Cached data in phoenix is written to SAGE_HOME/userdata/Phoenix/cache/, so if you moved the cache to a ramdisk and then symlinked the directory, then would that work?
Unfortunately not the same as Linux

Windows has a concept of Junctions, which unfortunately requires the original directory to be empty before applying the junction, so if you you want to move the Fanart directory into a RamDisk you have to:

- create the RamDisk
- copy the whole cache into the RamDisk
- delete the original fanart directory
- junction the RamDisk to the original directory

Now as you can see all your fanart are now in volatile memory, which is not good.

I've implemented already the idea by splitting the concept in two, whereby the thumbnails are in RamDisk (doesn't matter if you loose them), but the fanart repository is still on the OS drive.

Ideally we should have BMT allowing to specify two directories, one for the safe central fanart repository and one for the fanart to be displayed so that the latter can be relocated on a RamDisk and let BMT keep them both on sync

That would be the ideal solution, however I asked John already to implement this feature in his Fanart plugin and let me keeping the two directory in sync as I understand this might not be anyone cup of tea.

By doing this I will achieve:

- Best berformance than an SSD (nothing can beat RAM speed).
- Not spending any money on a DriverCache solution which unless you have an UPS have limited benefits.
- Zero disk fragmentation issues, very useful for SSD users indeed.

Eddy
__________________

Automatic Power Off | Squeezeslave | DVB-S Importer | DVB Decrypter & Card Client | Tuner Preroll


Every man is a damn fool for at least five minutes every day; wisdom consists in not exceeding the limit. ~ Elbert Hubbard

Last edited by routerunner; 12-04-2012 at 07:22 AM.
Reply With Quote
  #7  
Old 12-04-2012, 07:29 AM
stuckless's Avatar
stuckless stuckless is offline
SageTVaholic
 
Join Date: Oct 2007
Location: London, Ontario, Canada
Posts: 9,675
Phoenix already allows that the central fanart be relocated anywhere you want, by changing the centralFanart property (by default it's in the userdata area). The cache is stored in the userdata area as well, and phoenix does allow you to set a new userdata location, although it is relative to the sagetv home at the moment.

If John is using the Phoenix apis, then then he's probably using the phoenix APIs for caching, etc, and as such, he wont have much control over the cache locations without changing phoenix.



Quote:
Originally Posted by routerunner View Post
Unfortunately not the same as Linux

Windows has a concept of Junctions, which unfortunately requires the original directory to be empty before applying the junction, so if you you want to move the Fanart directory into a RamDisk you have to:

- create the RamDisk
- copy the whole cache into the RamDisk
- delete the original fanart directory
- junction the RamDisk to the original directory

Now as you can see all your fanart are now in volatile memory, which is not good.

I've implemented already the idea by splitting the concept in two, whereby the thumbnails are in RamDisk (doesn't matter if you loose them), but the fanart repository is still on the OS drive.

Ideally we should have BMT allowing to specify two directories, one for the safe central fanart repository and one for the fanart to be displayed so that the latter can be relocated on a RamDisk and let BMT keep them both on sync

That would be the ideal solution, however I asked John already to implement this feature in his Fanart plugin and let me keeping the two directory in sync as I understand this might not be anyone cup of tea.

By doing this I will achieve:

- Best berformance than an SSD (nothing can beat RAM speed).
- Not spending any money on a DriverCache solution which unless you have an UPS have limited benefits.
- Zero disk fragmentation issues, very useful for SSD users indeed.

Eddy
Reply With Quote
  #8  
Old 12-04-2012, 07:44 AM
routerunner's Avatar
routerunner routerunner is offline
Sage Icon
 
Join Date: May 2008
Location: Wiltshire, UK
Posts: 1,383
Quote:
Originally Posted by stuckless View Post
The cache is stored in the userdata area as well, and phoenix does allow you to set a new userdata location, although it is relative to the sagetv home at the moment.
Correct, but what I'm asking is different, I would like to have a mirror of the original central fanart repository in a RamDisk. But what are you saying is that John has not control where to get the original fanart, but it just gets the ready scaled fanart in the cache? If is the case then is not true as John allows you to get the unscaled original files from the Fanart directly if you wish, I personally asked him for this option while ago because my fanart are optimally scaled up front, the only images that John stores in the cache are the small background thumbnails when browsing the fanart view which are scaled at runtime.

The main reason of asking John to get direct access to the fanart repository and not the cached one was because I found out that the cached images where 5 times bigger (file size) and definitevely worse quality than the original. So for John it should be pretty easy to add the option to allow another directory for fanart display as the infrastructure is there already.

Eddy
__________________

Automatic Power Off | Squeezeslave | DVB-S Importer | DVB Decrypter & Card Client | Tuner Preroll


Every man is a damn fool for at least five minutes every day; wisdom consists in not exceeding the limit. ~ Elbert Hubbard

Last edited by routerunner; 12-04-2012 at 07:52 AM.
Reply With Quote
  #9  
Old 12-04-2012, 12:02 PM
JREkiwi's Avatar
JREkiwi JREkiwi is offline
Sage Icon
 
Join Date: Jan 2005
Location: Auckland, New Zealand
Posts: 2,115
Quote:
Originally Posted by routerunner View Post
Correct, but what I'm asking is different, I would like to have a mirror of the original central fanart repository in a RamDisk. But what are you saying is that John has not control where to get the original fanart, but it just gets the ready scaled fanart in the cache? If is the case then is not true as John allows you to get the unscaled original files from the Fanart directly if you wish, I personally asked him for this option while ago because my fanart are optimally scaled up front, the only images that John stores in the cache are the small background thumbnails when browsing the fanart view which are scaled at runtime.
I'm only using the phoenix APIs to handle the fanart.

GetFanartBackgrounds() for unscaled art which pulls from the original fanart repository
phoenix_image_CreateImage() with a tag for scaled art which pulls from the userdata cache directory.
Quote:
Originally Posted by routerunner View Post
So for John it should be pretty easy to add the option to allow another directory for fanart display as the infrastructure is there already.
The infrastructure is not already there.

John
Reply With Quote
  #10  
Old 12-04-2012, 01:26 PM
routerunner's Avatar
routerunner routerunner is offline
Sage Icon
 
Join Date: May 2008
Location: Wiltshire, UK
Posts: 1,383
Quote:
Originally Posted by JREkiwi View Post
I'm only using the phoenix APIs to handle the fanart.

GetFanartBackgrounds() for unscaled art which pulls from the original fanart repository
phoenix_image_CreateImage() with a tag for scaled art which pulls from the userdata cache directory.
The infrastructure is not already there.

John
Ouch...so "GetFanartBackgrounds()" implicitly uses the Fanart Folder Directory specified in the property files?
__________________

Automatic Power Off | Squeezeslave | DVB-S Importer | DVB Decrypter & Card Client | Tuner Preroll


Every man is a damn fool for at least five minutes every day; wisdom consists in not exceeding the limit. ~ Elbert Hubbard
Reply With Quote
  #11  
Old 12-06-2012, 03:34 AM
routerunner's Avatar
routerunner routerunner is offline
Sage Icon
 
Join Date: May 2008
Location: Wiltshire, UK
Posts: 1,383
Quote:
Originally Posted by JREkiwi View Post
I'm only using the phoenix APIs to handle the fanart.

GetFanartBackgrounds() for unscaled art which pulls from the original fanart repository
phoenix_image_CreateImage() with a tag for scaled art which pulls from the userdata cache directory.
The infrastructure is not already there.

John
Hi John,

a thought about your reply and I think I've got a better idea. When the scale option is disabled, what if you copy the unscaled fanart "as is" from the repository into the cache? This way if someone want to relocate the cache into a RamDisk it would be completely transparent for you and the user and your access to the cache consistent.

Think about it and please let me know.

Eddy
__________________

Automatic Power Off | Squeezeslave | DVB-S Importer | DVB Decrypter & Card Client | Tuner Preroll


Every man is a damn fool for at least five minutes every day; wisdom consists in not exceeding the limit. ~ Elbert Hubbard
Reply With Quote
  #12  
Old 12-06-2012, 07:30 AM
stuckless's Avatar
stuckless stuckless is offline
SageTVaholic
 
Join Date: Oct 2007
Location: London, Ontario, Canada
Posts: 9,675
Quote:
Originally Posted by routerunner View Post
Ouch...so "GetFanartBackgrounds()" implicitly uses the Fanart Folder Directory specified in the property files?
And you don't think GetFanartBackground should use the settings that know where the fanart is stored???
Reply With Quote
  #13  
Old 12-06-2012, 08:04 AM
routerunner's Avatar
routerunner routerunner is offline
Sage Icon
 
Join Date: May 2008
Location: Wiltshire, UK
Posts: 1,383
Quote:
Originally Posted by stuckless View Post
And you don't think GetFanartBackground should use the settings that know where the fanart is stored???
Easy mate, I never said that...of course GetFanartBackground should use that pathname is even implicit in the method name, mine was a simple consideration after realising that it wasn't that easy for John as I originally thought...

Anyway, I gave him another idea that hopefully should be simple, well I guess...

Eddy
__________________

Automatic Power Off | Squeezeslave | DVB-S Importer | DVB Decrypter & Card Client | Tuner Preroll


Every man is a damn fool for at least five minutes every day; wisdom consists in not exceeding the limit. ~ Elbert Hubbard
Reply With Quote
  #14  
Old 12-06-2012, 12:18 PM
Dargason Dargason is offline
Sage Expert
 
Join Date: Oct 2003
Posts: 516
Has anyone verified that this actually makes performance on extenders any faster? I tried it quite a while ago when it was in the Phoenix forums and it didn't really help me at all.

I certainly might not have had it set up correctly, but in the end I figured that it already caches stuff in ram (the heap) and therefore a ramdisk would only help when loading stuff that's not in the heap, which is only occasionally (at least in my case.)

Does that make any sense?
Reply With Quote
  #15  
Old 12-06-2012, 02:22 PM
routerunner's Avatar
routerunner routerunner is offline
Sage Icon
 
Join Date: May 2008
Location: Wiltshire, UK
Posts: 1,383
Quote:
Originally Posted by Dargason View Post
Has anyone verified that this actually makes performance on extenders any faster? I tried it quite a while ago when it was in the Phoenix forums and it didn't really help me at all.

I certainly might not have had it set up correctly, but in the end I figured that it already caches stuff in ram (the heap) and therefore a ramdisk would only help when loading stuff that's not in the heap, which is only occasionally (at least in my case.)

Does that make any sense?
I always said that this mechanism works providing you're using the JREkiwi plugin and in particular by setting both scaling options (Poster and Background) to "Don't scale" which means that the Fanart are taken "as-is" from the Fanart Central repository.

However by doing this you must make sure that your files in the fanart repository are optimized in terms of file size, because the memory on board of the extender is limited, so the last thing you want is to have big files that will fill up the extender's heap quickly.

But why this? The reason is pretty simple and you can try by yourself if you wish. Using BMT your posters and backgrounds are typically JPG with a given size/quality which can be good for a client which typically has got CPU power and fast Ethernet controller speed, but definitively not for an extender. Now browse a little bit and wait until the fanart cache has some files inside, you'll instantly notice that the corresponded files in the cache are PNG with a much bigger file size and a questionable quality, now imagine that the very same file has to be loaded into the limited extender memory and network bandwidth, you can immediately think that this is not going to work.

For that reasons I asked JREkiwi to add the options to disable the scaling altogether because I did want to take care myself of providing files with optimal size and quality and you can see the results by clicking on my signature "Video" link.

Eddy
__________________

Automatic Power Off | Squeezeslave | DVB-S Importer | DVB Decrypter & Card Client | Tuner Preroll


Every man is a damn fool for at least five minutes every day; wisdom consists in not exceeding the limit. ~ Elbert Hubbard
Reply With Quote
  #16  
Old 12-06-2012, 02:37 PM
stuckless's Avatar
stuckless stuckless is offline
SageTVaholic
 
Join Date: Oct 2007
Location: London, Ontario, Canada
Posts: 9,675
Quote:
Originally Posted by routerunner View Post
But why this? The reason is pretty simple and you can try by yourself if you wish. Using BMT your posters and backgrounds are typically JPG with a given size/quality which can be good for a client which typically has got CPU power and fast Ethernet controller speed, but definitively not for an extender. Now browse a little bit and wait until the fanart cache has some files inside, you'll instantly notice that the corresponded files in the cache are PNG with a much bigger file size and a questionable quality, now imagine that the very same file has to be loaded into the limited extender memory and network bandwidth, you can immediately think that this is not going to work.
This is a very good explanation... and my take away from this is that we don't need a more complicated dual location fanart structure... but rather we need BMT/Phoenix to fix the scaling so that it's more efficient. I'll be honest, I've never looked at the resulting file sizes from the cached files, so I've never noticed that the cached files are in fact larger in size than the originals.

File size will certainly affect the performance in terms of loading images (especially from the network), but the goal of scaling is to reduce the pixel size of the image, since smaller images will consume less memory on an extender, even if the file size is much larger.

If I recall the code for Phoenix, it probably wouldn't be too much effort to make the "scaler" pluggable, so that you could use your own program to scale images. In the code today, it does force a png output, but again, that could easily be changed to .jpg, I just prefererred png files, so that's what I used.
Reply With Quote
  #17  
Old 12-06-2012, 02:50 PM
routerunner's Avatar
routerunner routerunner is offline
Sage Icon
 
Join Date: May 2008
Location: Wiltshire, UK
Posts: 1,383
Quote:
Originally Posted by stuckless View Post
This is a very good explanation... and my take away from this is that we don't need a more complicated dual location fanart structure...
@stuckless
What do you think instead of my last proposal to copy the original fanart file when required in the cache so that can be easily relocatable in ramdisk for instance? In the end is not much different that caching the other files the only difference is that the file is not scaled but taken "as it" and therefore it should fit better with the current implementation.

I've spent quite a long time in the past to find a process that would produce good quality but small file size which would still looking good on a 120" projected image and I've published already HERE what I'm doing if you're interested.

Eddy
__________________

Automatic Power Off | Squeezeslave | DVB-S Importer | DVB Decrypter & Card Client | Tuner Preroll


Every man is a damn fool for at least five minutes every day; wisdom consists in not exceeding the limit. ~ Elbert Hubbard

Last edited by routerunner; 12-06-2012 at 03:16 PM. Reason: better explanation
Reply With Quote
  #18  
Old 12-10-2012, 08:40 AM
Fuzzy's Avatar
Fuzzy Fuzzy is offline
SageTVaholic
 
Join Date: Sep 2005
Location: Jurupa Valley, CA
Posts: 9,957
Quote:
Originally Posted by routerunner View Post
Unfortunately not the same as Linux

Windows has a concept of Junctions, which unfortunately requires the original directory to be empty before applying the junction, so if you you want to move the Fanart directory into a RamDisk you have to:

- create the RamDisk
- copy the whole cache into the RamDisk
- delete the original fanart directory
- junction the RamDisk to the original directory

Now as you can see all your fanart are now in volatile memory, which is not good.

I've implemented already the idea by splitting the concept in two, whereby the thumbnails are in RamDisk (doesn't matter if you loose them), but the fanart repository is still on the OS drive.

Ideally we should have BMT allowing to specify two directories, one for the safe central fanart repository and one for the fanart to be displayed so that the latter can be relocated on a RamDisk and let BMT keep them both on sync

That would be the ideal solution, however I asked John already to implement this feature in his Fanart plugin and let me keeping the two directory in sync as I understand this might not be anyone cup of tea.

By doing this I will achieve:

- Best berformance than an SSD (nothing can beat RAM speed).
- Not spending any money on a DriverCache solution which unless you have an UPS have limited benefits.
- Zero disk fragmentation issues, very useful for SSD users indeed.

Eddy
Actually, windows vista and 7 DO support symbolic links, pretty much the same as linux. You have to have administrator permissions to create them, and you use the mklink.exe command to do so.
Code:
MKLINK [[/D] | [/H] | [/J]] Link Target

        /D      Creates a directory symbolic link.  Default is a file
                symbolic link.
        /H      Creates a hard link instead of a symbolic link.
        /J      Creates a Directory Junction.
        Link    specifies the new symbolic link name.
        Target  specifies the path (relative or absolute) that the new link
                refers to.
That said, what you COULD do for fanart in RAM, is to actually point your fanart folder to the RAMDrive, and then sync it with a hdd folder. no need to work with junctions, links, etc. New fanart is downloaded directly to the ramdrive. A sync process keeps a constant backup of the folder onto the HDD. On startup, the same sync process would bring the data back into the ram disk.. again... for my use, it's too much complication when this is essentially what fancycache does for me...
__________________
Buy Fuzzy a beer! (Fuzzy likes beer)

unRAID Server: i7-6700, 32GB RAM, Dual 128GB SSD cache and 13TB pool, with SageTVv9, openDCT, Logitech Media Server and Plex Media Server each in Dockers.
Sources: HRHR Prime with Charter CableCard. HDHR-US for OTA.
Primary Client: HD-300 through XBoxOne in Living Room, Samsung HLT-6189S
Other Clients: Mi Box in Master Bedroom, HD-200 in kids room

Last edited by Fuzzy; 12-10-2012 at 08:48 AM.
Reply With Quote
  #19  
Old 12-10-2012, 08:57 AM
routerunner's Avatar
routerunner routerunner is offline
Sage Icon
 
Join Date: May 2008
Location: Wiltshire, UK
Posts: 1,383
Quote:
Originally Posted by Fuzzy View Post
Actually, windows vista and 7 DO support symbolic links, pretty much the same as linux. You have to have administrator permissions to create them, and you use the mklink.exe command to do so.
Code:
MKLINK [[/D] | [/H] | [/J]] Link Target
 
        /D      Creates a directory symbolic link.  Default is a file
                symbolic link.
        /H      Creates a hard link instead of a symbolic link.
        /J      Creates a Directory Junction.
        Link    specifies the new symbolic link name.
        Target  specifies the path (relative or absolute) that the new link
                refers to.
That said, what you COULD do for fanart in RAM, is to actually point your fanart folder to the RAMDrive, and then sync it with a hdd folder. no need to work with junctions, links, etc. New fanart is downloaded directly to the ramdrive. A sync process keeps a constant backup of the folder onto the HDD. On startup, the same sync process would bring the data back into the ram disk.. again... for my use, it's too much complication when this is essentially what fancycache does for me...
Yeah...sorry, I keep assuming people still using XP these days
__________________

Automatic Power Off | Squeezeslave | DVB-S Importer | DVB Decrypter & Card Client | Tuner Preroll


Every man is a damn fool for at least five minutes every day; wisdom consists in not exceeding the limit. ~ Elbert Hubbard
Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Phoenix Fanart with SMM + Diamond 3.1 nateziemann Sage My Movies 5 04-06-2011 08:30 PM
getting banners and other fanart using the phoenix api razrsharpe Batch Metadata Tools 11 01-17-2011 01:23 PM


All times are GMT -6. The time now is 03:38 PM.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2021, vBulletin Solutions Inc.
Copyright 2003-2005 SageTV, LLC. All rights reserved.