Utility: mediaScraper - metadata made easy!

[jaminben]

Originally Posted by evilpenguin

Long story short, its almost done, but I have to a lot of testing to make sure I didn't break anything.

Cool, looking forward to seeing how this will work out.

[evilpenguin]

Originally Posted by JimmyBalboa

I am a novice at meta data but, Rating=TVPG, etc... should work right?

You're absolutely right, I missed that before. I'll make sure it gets added in.

[Zorglub]

How did you handle the creation of the xml file ?

While you are working on the integration (which will be wonderful !),I was wondering if you mind sharing whatever code you may have used to generate or convert to xml format, so I can do a little conversion script to make it work in the mean time.

At first I was planning to write one in Perl, but before I do that, maybe your code will save me some time...

[lobosrul]

Great utility EvilPenguin! Finally something that works well with TV files.

The only problem I've encountered so far is (other than thetvdb.com being very slow) the title in the .my or .properties file is changed from the original filename. This causes a few problems for me. First I have all my TV shows and movies with the word "The" at the front moved to the back (ie "Fugitive, The"). Also, some foreign movies have their title changed to the original language (ie "The Good, the Bad, and the Ugly" becomes "Buono, il brutto, il cattivo., Il"). I don't want the title in either case to be changed. I have all of my movies stored on an external NAS device and all the background jpg's really slow things down unless I move them to the central directory. But now that the title doesn't match the filename, SageMC doesn't load them.

Is there any way to keep the titles exactly the same as the filename?

[JimmyBalboa]

Originally Posted by evilpenguin

You're absolutely right, I missed that before. I'll make sure it gets added in.

Do you have an ETA on the next release? Is it days or weeks away?

[evilpenguin]

mediaScraper 1.0 beta9

• XML output and automatic web server importing (See this post for details and see the bottom of the 2nd post for the CLI switches)
• Switched over to using theTVDB api which is much faster than just regular HTML calls.
• Downloading video thumbnails for TV shows
• Downloading season specific banners/posters for TV shows
• Fix for individual TV shows stored in VIDEO_TS folders
• Added rating to properties file
• Added /useOriginalName so that the original file name will be used in the properties file.
• Fix for shows with reserved perl REGEX chars in the name ({}[]()^$.|*+?\) (ex: M*A*S*H)

The two biggest changes in this one are behind the scenes. First off I switched over to using theTVDB API which makes everything much faster (5kb vs. 150kb downloads). Secondly, I ported over most of the code from my video encoding tool into this code base which allows me to detect the duration for the XML files as well as set me up for re-releasing my other tool under this code base (I've already got AutoEncode working like a champ ).

Both of those things should be 100% transparent and I've tested everything pretty thoroughly so I don't expect anyone will run into any problems, but if stuff that worked before stops working be sure and let me know and i'll get it fixed right away.

Aside from that the big new feature is the ability to automatically scrape show info and add it into the SageTV DB which will allow your imported TV to show up in your Sage recordings screen right next to your recorded TV. I've been playing with it for a few days and its pretty neat addition, many thanks to who ever added that to the web server.

[matt91]

EP -

My interest in your great tool, at this point, is to try and grab fanart for my existing Sage recordings (ie, not ripped/imported).

When I run the tool on:
R:\tv\Crusoe-Hour12TheTraveler-9484598-0.mpg

I get nothing. However, if I rename the file to appear as if I imported it:
R:\tv\Crusoe 1x11.mpg

then the tool grabs all the art.

I understand that your intent it to grab metadata for imported files, but is there any way to accomplish what I'm looking for?

thanks
matt

[popechild]

I'm still unable with the newest version to figure out how to run it on my tv shows ripped as VIDEO_TS folders. It keeps coming back with crazy results that aren't accurate. I'm sure I'm just doing something wrong.

My "base folder" for my tv rips is on a network share (\\NASLITE\Disk-5\Owned DVDs\TV Shows\)

My defaults.txt currently just has the following:
/genMyFile
/downloadFanArt
/baseFolder

For /baseFolder I've tried "\\NASLITE\Disk-5\Owned DVDs\TV Shows" and I've also tried making a mapped drive to the Owned DVDs folder and using baseFolder "Z:\TV Shows". I also tried without /baseFolder. Regardless of whether I drop a single episode folder or a full season or full show folder (I have them organized "Series\Season\Episode\VIDEO_TS") nothing ever comes back with the right info.

Any suggestions?

Also, is there a way to disable downloading of certain FanArt elements? I'd rather it not download (or generate, whichever it's doing) the episode thumbnails because Sage doesn't use the poster cover art when it does this. But I still want it to download all the other art.

[popechild]

Just had a thought for a nice addition to future downloads. What do you think about putting a text file in the folder that has all of the usage instructions from your 2nd post in it? I can "scrape" it myself for now, but knowing that it'll update with each version... I just realized that I keep having to come back to this thread whenever I use the scraper to remember what the different switch options are. Just a thought.

[JimmyBalboa]

Originally Posted by evilpenguin

mediaScraper 1.0 beta9

Kudos to you Bro...

beta9 Rocks!

Thanks for the hard work you put in to this awesome utility!

[popechild]

Sorry, one more from me. I'm playing around with the XML import to sage DB feature and it's very cool. I have a couple of questions about how it works though.

1) I get a little lock icon next to the imported shows in my Recorded TV list. It doesn't seem to prevent me from deleting the episodes - does anyone know what this means? I've never seen it on anything before.

2) Does anyone know how Sage handles these in terms of freeing up space? I have these imported episodes stored on a different drive, so I'm hoping Sage doesn't delete them when it's trying to clear out room for new shows. Anybody know whether Sage takes these into consideration when calculating available space or whether it might delete them thinking that would free up space? That's the only thing that kind of scares me.

3) Finally, I've got 5 episodes of one show (Damages) all in the same folder. All run through the scraper great and all the properties file and FanArt stuff is generated. However, when I use the options to do the XML import into Sage's DB, it's only importing 2 of the 5. (The one other show I have tried it on has 4 episodes, and all of them imported properly.) Can you think of a reason why only 2 of the 5 would get imported into Sage's DB when all 5 are recognized and properly scraped?

[evilpenguin]

Originally Posted by matt91

When I run the tool on:
R:\tv\Crusoe-Hour12TheTraveler-9484598-0.mpg

Oh shoot, I half-way fixed that and forgot to finish it. I'll try and have something out for that in the next few days.

Originally Posted by popechild

I'm still unable with the newest version to figure out how to run it on my tv shows ripped as VIDEO_TS folders. It keeps coming back with crazy results that aren't accurate. I'm sure I'm just doing something wrong.

I'll need to see scraper.log files for all the problems your having.

[evilpenguin]

Originally Posted by popechild

1) I get a little lock icon next to the imported shows in my Recorded TV list. It doesn't seem to prevent me from deleting the episodes - does anyone know what this means? I've never seen it on anything before.

I think that means the show is archived. Its the same thing it would do if you would manually send a show to the archive.

Originally Posted by popechild

2) Does anyone know how Sage handles these in terms of freeing up space?

I'm not sure, i'd have to read the manual, but i'm pretty sure Sage won't delete archived shows.

Originally Posted by popechild

3) Finally, I've got 5 episodes of one show (Damages) all in the same folder. All run through the scraper great and all the properties file and FanArt stuff is generated. However, when I use the options to do the XML import into Sage's DB, it's only importing 2 of the 5.

I'll need to see scraper.log's for this.

[jaminben]

Great job E.P

I've gotten the xml files to import into Sage directly by manually doing it but I can't seem to get it to work automatically.

So far I've tried various configurations of:

/userName "******" /password "*****" /sageTVServer "192.168.***.***:****" /addToSageDB /genXMLFile /downloadFanArt /organizeFiles /tvSE

I'm sure I've misread your guide but I can't see where, any ideas?

edit:

ooops, I used the wrong username. Its working very well now



I was also thinking...dangerous hobby I know... is their any need to have all my imported shows placed in a seperate folder to my actual TV recordings? Once this new method of importing an xml file works would it be better to place my imported media directly into my TV recordings folder?

Anyway, nice job once again

[popechild]

Originally Posted by jaminben

I was also thinking...dangerous hobby I know... is their any need to have all my imported shows placed in a seperate folder to my actual TV recordings? Once this new method of importing an xml file works would it be better to place my imported media directly into my TV recordings folder?

Anyway, nice job once again

Not EP, but in an effort to help take some of the burden off him if I can I'll try to answer. I'm pretty sure there's no reason you couldn't keep the shows in the same folder. I can't think of anything it would help either though, unless you're just wanting to do it that way for your own organization reasons.

[JimmyBalboa]

I do have one small issue to report with beta9.

The episode graphics are blank on the HD200 extenders. On the PC interface everything is fine, but my HD Theater Extenders have no picture at all. This may be a Sage issue...

Any ideas?

[jaminben]

Originally Posted by popechild

Not EP, but in an effort to help take some of the burden off him if I can I'll try to answer. I'm pretty sure there's no reason you couldn't keep the shows in the same folder. I can't think of anything it would help either though, unless you're just wanting to do it that way for your own organization reasons.

Thanks for the reply popechild

I decided to keep them where they are for now as I did have a thought that the folder.override wouldn't work if all shows where in the same folder.

Is their a way to prevent the download of images and backgrounds? I don't remember seeing a post about this and I've just looked at the 1st/2nd post but found no answers.

I'm also noticing that I'm getting a .properties file and an .xml file placed in the shows folder. Is it possible to have these deleted automatically once the show .xml has been imported into Sage's DB?

Cheers

Ben

[JimmyBalboa]

Originally Posted by jaminben

I'm also noticing that I'm getting a .properties file and an .xml file placed in the shows folder. Is it possible to have these deleted automatically once the show .xml has been imported into Sage's DB?

Ben

No expert here, but logically thinking, the next time you refreshed your media library the meta data would be removed from those episodes if you delete those files.

[evilpenguin]

Originally Posted by JimmyBalboa

but my HD Theater Extenders have no picture at all. This may be a Sage issue...

Any ideas?

Have you tried restarting Sage on the Server, it may be an issue with image cache. It could also be an issue with absolute paths being used for recording drives rather than UNC.

Originally Posted by jaminben

Thanks for the reply popechild

I decided to keep them where they are for now as I did have a thought that the folder.override wouldn't work if all shows where in the same folder.

Yeah, there's really no need to move them if you don't want to. I'm planning on putting all of mine on a NAS.

Originally Posted by jaminben

Is their a way to prevent the download of images and backgrounds? I don't remember seeing a post about this and I've just looked at the 1st/2nd post but found no answers.

Specifically which images/backgrounds, I'm downloading so many. Right now /downloadFanArt grabs *everything* it can, but I could easily have a switch for disabling specific ones you don't want.

Originally Posted by jaminben

I'm also noticing that I'm getting a .properties file and an .xml file placed in the shows folder. Is it possible to have these deleted automatically once the show .xml has been imported into Sage's DB?

That makes sense, I'll get it into the next version.


@Popechild - I still need to see logs and/or filenames for the problems your having so I can duplicate them.

[jaminben]

Originally Posted by JimmyBalboa

No expert here, but logically thinking, the next time you refreshed your media library the meta data would be removed from those episodes if you delete those files.

I thought that at first but then I thought it has imported the data into Sage's wiz.bin file so their is no need to read an external file anymore. I've deleted alot of my .xml, .my & .properties files from my TV show folders and all the data is still available in Sage. So I'm assuming that its ok to do this as it still works.