Utility: mediaScraper - metadata made easy!

[evilpenguin]

Release History:

• Beta13
• Beta9
• Beta8
• Beta7
• Beta6
• Beta5

What it is:
mediaScrapper is my answer XBMC's excellent, but very confusing, scraping engine and you can use it to track down all the metadata and fanart for any TV/Movies you may have ripped/bought/downloaded/whatever.

Usage Instructions: See 2nd post

How to use metadata files:

* To make SageMC use the .my files, go into SageMC Properties -> Enhancments -> Use .my Files -> Enabled.
* To make the default STV use .properties files, read this thread.

Download @ Sourceforge (Source Code)

Submitting Issues:

If you have any issue I'll need to see the EXACT PATH/FILE name that is giving you trouble and/or (preferably) the scraper.log file that gets generated next to the .exe after every run.

Example Outputs:

.properties file for default STV:

MediaType=TV
Title=Wonderfalls 1x05 - Crime Dog
MediaTitle=Wonderfalls
Actor=Caroline Dhavernas;Tracie Thoms;Tyron Leitso;Diana Scarwid;Lee Pace;Katie Finneran;William Sadler;
ActorRoles=Jaye Tyler;Mahandra;Eric;Karen Tyler;Aaron Tyler;Sharon Tyler;Darrin Tyler;
Guest=Audrey Wasilewski;Bill Lake;Kimberly Scott;
Director=Allan Kroeker;
Writer=Krista Vernoff;
Genres=Drama;
Description=When Jaye's brother sees her talking to inanimate objects, he begins to question her state of mind; the animals help Jaye reunite her family with the housekeeper.
Rated=
UserRating=6.0
ReleaseDate=2004-07-23
MediaProviderDataID=theTVDB:78845
SeasonNumber=1
EpisodeNumber=5
EpisodeTitle=Crime Dog

.my file for SageMC:

Title=Burn Notice
overview=A spy, Michael Weston, receives a burn notice for an unstated reason, effectively firing him. He has spent the previous decade working for the government in Eastern Europe and returns to his hometown of Miami to get his life in order. Michael will stop at nothing to find out why. Shut out from his normal contacts, but still driven to right wrongs, Michael needs to stay under the radar in order to stay in the game.
TVOriginalAiringDate=2007-07-26
actors=Jeffrey Donovan, Gabrielle Anwar, Bruce Campbell, Paul Gutrecht, Guri Weinberg, Audrey Landers, Scott Michael Campbell, Joel Swetow, Hannia Guillen, Jessica Baldwin, Rangel Hernandez Martinez,
TVEpisode=Family Business
TVOverview=Michael infiltrates a family of gunrunners who are pressuring an airport supervisor and his pregnant wife. The FBI begin to put pressure on Sam to gather more information on Michael.
TVDuration=60 minutes
TVGenre=Action and Adventure
TVChannelName=USA

List of Non-Working names that I know about and intend to Fix:

[evilpenguin]

Supported Naming Conventions:
MediaScaper works by looking at your file names and attempting to pull out all of the information it needs to track down the metadata. Chances are if mediaScraper can't find the metadata for your video then the file name is too confusion for it to figure out. Below are the supported naming conventions that will work

Television: Requires Show Title and Season/Episode Number

Show Title (S##E##|#x##|###) randomJunk.avi
BaseFolder\Show Title\(S##E##|#x##|###) randomJunk.avi
BaseFolder\Show Title\Season #\(S##E##|#x##|###) randomJunk.avi

Movies: Requires Movie Title

Movie Title.avi
Movie Title (YEAR).avi
Movie Title (YEAR).randomJunk.avi

NOTE: This will not work on shows in Sage Recording format (ShowTitle-EpisodeName-#####-#.mpg)!!!


Basic Usage:

For general usage all you need to do is drag and drop videos and/or folders containing videos onto mediaScraper.exe and, by default, it will download the metadata and fanart and place them right next to the original videos.

TV:

TV\Scrubs\Scrubs 2x01 - My Overkill.avi
TV\Scrubs\Scrubs 2x01 - My Overkill.avi.properties (metadata)
TV\Scrubs\background.jpg (Fanart)
TV\Scrubs\folder.jpg (Thumbnail/Poster)
TV\Scrubs\banner.jpg (Banner)

Movies:

Movies\300.avi
Movies\300.avi.properties (metadata)
Movies\300_background.jpg (Fanart)
Movies\300.jpg (Thumbnail/Poster)

Movies (VIDEO_TS Folder):

Movies\300\VIDEO_TS
Movies\300.properties (metadata)
Movies\300\background.jpg (Fanart)
Movies\300\folder.jpg  (Thumbnail/Poster)

folder.override:

There will be times when mediaScraper just won't be able to find a match for your file. Common problems include

• Actual show name contains illegal window characters. ex: Terminator: The Sarah Conner Chronicles.
• Name conflicts with other TV shows. ex: The Office (US) vs. The Office (UK) or Battlestar Gallactica (2003) vs. Battlestar Gallactica.

In these cases, rather than going through and renaming every file to get it to match you can create a folder.override file next to the original video(s) that is just a text file that contains the exact show title that theTVDB or IMDB is expecting for every show in that folder.

Example:

Lets say that you try and run this file through mediaScraper:

TV\Terminator The Sarah Connor Chronicles\Terminator The Sarah Connor Chronicles 1x01.avi

It would decide that its a TV show with...

Show Title = Terminator The Sarah Connor Chronicles
Season = 1
Episode = 1

Which looks all good, but when you search for it on theTVDB it returns 0 matches.

With a little manual investigating you'll find that theTVDB will only recognize the title if it contains the ':'.

Series Title = Terminator: The Sarah Conner Chronicles

However, you can't use a ':' in a Windows file name. This is where folder.override come in.

To fix this you'd create a folder.override in the folder with all the episodes.

TV\Terminator The Sarah Connor Chronicles\Terminator The Sarah Connor Chronicles 1x01.avi
TV\Terminator The Sarah Connor Chronicles\folder.override

and use notepad to make its contents be the exact show title that theTVDB or IMDB is expecting....

Terminator: The Sarah Connor Chronicles

Then next time mediaScraper gets a video from that folder, it'll see the folder.override file and instead of trying to figure out the Show Title, it will just read it out of that file and use that to search.

You can also place a folder.override it in the parent directory if you want it to apply to all folders directly above it.

TV\Terminator The Sarah Connor Chronicles\Season 1\Terminator The Sarah Connor Chronicles 1x01.avi
TV\Terminator The Sarah Connor Chronicles\folder.override

In addition, mostly for movies, you can create a whatEverYourFileNameIs.override so that it applies to only one file.

The Dark Knight.avi
The Dark Knight.override

mediaScraper.skip:

If you have a folder full of videos you know have no metadata (ex. home movies, clips, etc.) then you can place a file named mediaScraper.skip in the folder and then mediaScaper will ignore it and all its sub folders.

TV\Home Movies\mediaScraper.skip

Advanced Usage:

MediaScraper reads all of its options out of defaults.txt which is right next to mediaScraper.exe and it is just a list of command line switches that will always be used.

These are the default options:

/genPropertyFile /downloadFanArt /baseFolder "TV"

In addition you can also add any of the below, available, switches to further customize the output.

Available Switches:

• /genPropertyFile - Generate a .properties file.
• /genMyFile - Generate a .my file, contains extra data for SageMC
• /genInfoFile - Generate a .info file. These don't do anything useful, they are just a dump of all of the available metadata, some of which, doesn't fit into the .my or .properties files.
• /downloadFanArt - Download fanart/thumbnails/banners if available.
• /baseFolder "C:\Example\Folder Name" - If you keep all your TV organized in a common base folder you can specify it here to help with scraping accuracy
• /organizeFiles - If the mediaScraper is able to find metadata for your file it can also rename your file to match. (Will not work with VIDEO_TS folders)
  

  Scrubs.201.lol.hdtv.avi -> Scrubs 2x01 - My Overkill.avi

  The.Dark.Knight.2008.DVDrip.xor.aiv -> The Dark Knight (2008).avi

• /tvSE - This will have mediaScraper use the TV naming format S02E01 rather than 2x01. This will apply to both organizing and metadata files.
• /organizeBaseFolder "D:\Videos" - If you specify a base folder when you have /organizeFiles set this will move the videos into a folder structure as well.
  

  Scrubs.201.lol.hdtv.avi -> D:\Videos\TV\Scrubs\Season 2\Scrubs 2x01 - My Overkill.avi

  The.Dark.Knight.2008.DVDrip.xor.aiv -> D:\Videos\Movies\The Dark Knight (2008).avi

• /4digitTV - Allow TV shows that don't use a separator between the season/episode (S07E22 -> 722) to be 4 digits long (S11E22 -> 1122). By default this is disabled to prevent a conflict with movies that have the year in the file name (2008 season/episode vs 2008 year).
• /updateInfo - Normally mediaScraper will ignore videos that already have existing properties files. You can set this to have it run anyway.
• (NEW) /useOriginalName - Use this if you want to use your original file name in the .properties files rather than the official name scraped from the show info.
• (NEW) /genXMLFile - Generate an XML file that can be imported in to SageTV web server
• (NEW) /userName "User" - User name for logging onto a SageTV web server
• (NEW) /password "Password" - Password for logging onto a SageTV web server
• (NEW) /sageTVServer "localhost:8080" - Host name and port number of web server
• (NEW) /addToSageDB - When used with /genXMLFile and web server settings will automatically add a show to Sage's DB using the web servers XML import function. You can use this to have your imported TV show up with your recorded TV.

/addToSageDB:
If you pair this switch with /userName, /password, /sageTVServer, and /genXMLFile then mediaScraper will pack the metadata into a SageTV Webserver XML file and use an experimental feature that will add trick Sage into treating the show as if it were a SageTV Recording. I've been playing around with it and it seems to be working rock solid, but I make no guarantees that this won't blow out your entire wiz.bin: Use at your own risk!

[mickp]

Ooooh. If Mike or Dirk support this for fan art I might just give the feature a go.

Downloading now. Thanks E.P.

Mick.

[mickp]

Hey E.P.

Just gave it a burl on a few files

Filenames that didn't work;

"c:\shares\movies\Atlantis\Season 3\Stargate.Atlantis.S03E01.WS.DSR.XviD-DIMENSION.avi"

"c:\shares\movies\Atlantis\Season 3\Stargate.Atlantis.S03E01.avi"

Filename that did work;

"c:\shares\movies\Atlantis\Season 3\Stargate Atlantis S03E01.avi"

Unfortunately I'm utter rubbish at regex so can't be of much help with detailed suggestions .

Can I suggest replacing the (.) period with a space and also if a postitive result/hit hasn't yet been found try removing the last word of the file (assuming . replaced with space) and giving the search another go. Then removing another, and another?

I realise that I could rename the files so that they work but it would be nice to have it just work with typical download file names.

I'll go have more of a play now

Mick.

Edit: Also, would it be possible to not create the metadata files if no result was found?

[evilpenguin]

Yeah, good catch, looks like the '.' in the series name is causing some trouble. I'll that (and a whole bunch of other bugs I found ) all sorted out tomorrow.

[mickp]

Cool. Thanks.

Mick.

[popechild]

Just had a thought for a nice addition to future downloads. What do you think about putting a text file in the folder that has all of the usage instructions from your 2nd post in it? I can "scrape" it myself for now, but knowing that it'll update with each version... I just realized that I keep having to come back to this thread whenever I use the scraper to remember what the different switch options are. Just a thought.

[popechild]

Sorry, one more from me. I'm playing around with the XML import to sage DB feature and it's very cool. I have a couple of questions about how it works though.

1) I get a little lock icon next to the imported shows in my Recorded TV list. It doesn't seem to prevent me from deleting the episodes - does anyone know what this means? I've never seen it on anything before.

2) Does anyone know how Sage handles these in terms of freeing up space? I have these imported episodes stored on a different drive, so I'm hoping Sage doesn't delete them when it's trying to clear out room for new shows. Anybody know whether Sage takes these into consideration when calculating available space or whether it might delete them thinking that would free up space? That's the only thing that kind of scares me.

3) Finally, I've got 5 episodes of one show (Damages) all in the same folder. All run through the scraper great and all the properties file and FanArt stuff is generated. However, when I use the options to do the XML import into Sage's DB, it's only importing 2 of the 5. (The one other show I have tried it on has 4 episodes, and all of them imported properly.) Can you think of a reason why only 2 of the 5 would get imported into Sage's DB when all 5 are recognized and properly scraped?

[evilpenguin]

Originally Posted by matt91

When I run the tool on:
R:\tv\Crusoe-Hour12TheTraveler-9484598-0.mpg

Oh shoot, I half-way fixed that and forgot to finish it. I'll try and have something out for that in the next few days.

Originally Posted by popechild

I'm still unable with the newest version to figure out how to run it on my tv shows ripped as VIDEO_TS folders. It keeps coming back with crazy results that aren't accurate. I'm sure I'm just doing something wrong.

I'll need to see scraper.log files for all the problems your having.

[Homebuilder]

Hi,
just downloaded and it seems to work great..except... Itworks when a file has the extension avi, but not .divx... I rename the file to avi and it works.

Anyway to make this recognize a divx file?

[freedml]

mediascraper seems to be getting stuck here. can someone tell me why:

- Remaining Commands (3): (/get ******AiringInfoXML /user "%%userName%%" /pa
ssword "%%password%%" /realm "SageTV Web Interface")
+ Command : /get ******AiringInfoXML /user "Sage" /password "******" /real
m "SageTV Web Interface"
- Adding Credientials: Sage@localhost:8080(SageTV Web Interface)
- Adding Credientials: Sage@localhost:8080(SageTV Web Interface)
+ Requesting WebAddress: http://localhost:8080/******/DetailedInfo?AiringI
d=1309749&xml=yes
- Attempt #1 - Attempt #2 - Attempt #3 - Attempt #4
- Attempt #5 Failure: 500 Internal server error
+ Sleeping for 10 seconds as to not throttle website

[freedml]

Maybe a password problem? Webserver installs with a default 4 character password, but mediascraper seems to assume a 6 character password.

[rmac321]

EP,

Should MediaScraper (1.13) be producing a MediaTitle field for TV shows and putting it in the .properties file? It turns out that the problem I'm having with SageMC is due to SageMC expecting to see the MediaTitle to load FanArt for TV shows in the media import directory.

see: http://forums.freytechnologies.com/f...7&postcount=18
and: http://forums.freytechnologies.com/f...2&postcount=19

Thanks!

[GollyJer]

Is /AddToSageDB working for anyone? It was working great but doesn't anymore with the 6.5.17 beta.

[rmac321]

EP,

Sorry for the nag. Any chance you will be adding the MediaTitle field into the .properties output? I love MediaScraper and have used it to organize all my media files and FanArt, so I'd prefer to stay with it.

I know you mentioned you use BMT now, so I would certainly understand if you don't intend to do any more updates. OTOH, if you do intend to add this feature, I'll wait patiently for the next beta.

Thanks!

[rmac321]

EP, did I offend?