Utility: mediaScraper - metadata made easy!

[megamojo]

I found a very minor but very annoying problem tonight. It seems that leading zeros in filenames aren't preserved in the title field of the .properties file. Instead of trying to explain it perfectly, I'll just show the example of what's happening to me. I name all my TV show files as like "Heroes 1x04 Collision.avi" and after running this tool the title in the .properties file would be "Heroes 1x4 Collision." This means that in the regular Sage view episode 4 would come after episode 10, not cool. Also, is there a way that we can control what files are output? I don't really need all the different metadata filetypes and they make for lots of clutter.

[Halstead]

Is /TV/ going to stay hard coded? I currently have all of my shows in a /TV Shows/ top-level directory, and I'd hate to loose all of the 'watched' information.

Needless to say, I understand this probably isn't your top priority right now.

[evilpenguin]

Originally Posted by megamojo

I found a very minor but very annoying problem tonight. It seems that leading zeros in filenames aren't preserved in the title field of the .properties file.

Did you download beta4 that I just put out tonight. It should fix this problem.

Title=Scrubs 4x03 - My New Game

Originally Posted by megamojo

I don't really need all the different metadata filetypes and they make for lots of clutter.

I'll make it easer once i've sorted out all the technical stuff, but until they you can just make a batch file like this...

@ECHO OFF
set DIRECTORY=G:\mediaScraper

cd /D "%DIRECTORY%"
mediaScraper.exe /profile inputFile /genPropertyFile /outputFile "%~dp1%~n1" /inputFile "%~f1"
pause

Where you change G:\mediaScraper to the path to your folder. Then you can just drag and drop your files onto the batch file.

Originally Posted by Halstead

Is /TV/ going to stay hard coded? I currently have all of my shows in a /TV Shows/ top-level directory, and I'd hate to loose all of the 'watched' information.

Bah, my bad, I meant to do this this time, I'll get it in the next beta, but in the meantime just open up

...\scrapingProfiles\inputFile.scrape

and change where it says TV[\\\/] to TV Shows[\\\/]

# Profile: Series\Season\Episode format
#
#   INPUT : %%inputFile%%    - Input file with unknown format
#

    Profile           =TV_Series_Season_Episode
    Encode CLI #1     =?>!showTitle||!seasonNum||!episodeNum<:>TV Shows[\\\/]+($$showTitle$$[^\\\/]+)[\\\/]+[^0-9\\\/]+($$seasonNum$$[0-9]+)[\\\/]+\s*($$episodeNum$$[0-9]{2})<?
    Encoder #1        =/use "%%inputFile%%" /noOverWrite
    Encode CLI #1     =?>!showTitle||!seasonNum||!episodeNum<:>TV Shows[\\\/]+($$showTitle$$[^\\\/]+)[\\\/]+[^0-9\\\/]+($$seasonNum$$[0-9]+)[\\\/]+\s*S?[0-9][xE]($$episodeNum$$[0-9]{2})<?
    Encoder #1        =/use "%%inputFile%%" /noOverWrite
    Encode CLI #1     =?>!showTitle||!seasonNum||!episodeNum<:>TV Shows[\\\/]+($$showTitle$$[^\\\/]+)[\\\/]+[^0-9\\\/]+($$seasonNum$$[0-9]+)[\\\/]+\s*[0-9]{1,2}($$episodeNum$$[0-9]{2})<?
    Encoder #1        =/use "%%inputFile%%" /noOverWrite


# Profile: Series\SeasonXEpisode format
#
#   INPUT : %%inputFile%%    - Input file with unknown format
#

    Profile           =TV_Series_SeasonXEpisode
    Encode CLI #1     =?>!showTitle||!seasonNum||!episodeNum<:>TV Shows[\\\/]+($$showTitle$$[^\\\/]+)[\\\/]+.*S?($$seasonNum$$[0-9]+)[xE]($$episodeNum$$[0-9]{2})<?
    Encoder #1        =/use "%%inputFile%%" /noOverWrite
    Encode CLI #1     =?>!showTitle||!seasonNum||!episodeNum<:>TV Shows[\\\/]+($$showTitle$$[^\\\/]+)[\\\/]+.*($$seasonNum$$[0-9]{1,2})($$episodeNum$$[0-9]{2})<?
    Encoder #1        =/use "%%inputFile%%" /noOverWrite

And that should get it working.

[joey3002]

Have a problem.
When I select multiple files, it only does 1 of the multiples.

I did update input.properties with correct path (TV Archive)

Attaching the log file.. hmm just read log.. its only doing 1 file of the 2 i selected... so not attaching log because nothing in it.. how can i get it to do multiple files or directories (recursive)?

Thanks and nice app..

[joey3002]

Originally Posted by mickp

Awesome!

I've adapted my old comskip batch file to process all files in a directory. The log files from this should be quite interesting.

I'll give the new version a run over one of my download directories and pm the logs.

In the meantime i've attached the two batch files required for bulk scraping. Hopefully someone will find them useful.

Usage is scrape [unc path]

Mick.

Edit: Fixed a bug which would cause all shows to be processed every time. Now only files without a .properties file are scraped on subsequent runs.

Just a quick FYI on this.. if you are using long filenames, you need to do the following:

option1:
scrape.cmd "G:\my tvshows\seinfeld\season 1\"
option2:
use short filenames..
scrape.cmd g:\mytvsh~1\seinfeld\season~1\

hope this helps anyone..

[joey3002]

wow.. sorry for the many many posts in this forum today..
does this also download the artwork?

[evilpenguin]

Now that I'm mostly happy with the scrapping, I'm going to start adding new features and usability improvements. JPG downloading and dragging and dropping multiple files are at the top of my list.

[jaminben]

Originally Posted by evilpenguin

Now that I'm mostly happy with the scrapping, I'm going to start adding new features and usability improvements. JPG downloading and dragging and dropping multiple files are at the top of my list.

Cool, looking forward to it

[bluenote]

Is there a "main" config file? I want to turn off the generation of the file formats that I don't need. I'm browsing the included files but so far I don't see a way to do that.

thanks for this tool,

Cory

[evilpenguin]

Originally Posted by cnovak

Is there a "main" config file? I want to turn off the generation of the file formats that I don't need. I'm browsing the included files but so far I don't see a way to do that.

thanks for this tool,

Cory

That's also high on my todo list

[noobgoldberg]

Originally Posted by mickp

Sure. Appologies in advance if this is a bit basic.

Download the zip file and extract the two batch files (scrape.cmd and scrapAction.cmd) to the same directory that you have E.P's mediaScraper.exe file in.

launch a command prompt. There are various means to do this but if you're stuck press {windows key}+R and type "cmd" into the Run dialog that comes up and press enter.

At the command prompt type (assuming that you extracted it somewhere to teh C: drive)
"c:\whichever\directory\it\is\in\scrape \\ComputerName\ShareName" e.g. "c:\scraper\scrape \\myserver\mymovies"

NB: Swap the paths in the above example for your own. You could also use a local directory path rather than a share name e.g. c:\shares\mymovies rather than \\mypc\movies

The batch file will run mediaScraper.exe for each of the files in your movie share (and sub directories) which don't already have a .properties file.

During the process you will see some minimised command windows flashing on and off in the taskbar as the files are processed. A log file will be created in the root of the share that you're processing called ScrapeLog.txt

The log file will give you details of which files were processed, not processed and what metadata was saved.

The more obvious way to use the batch file would be to schedule it on a regular basis but that's another discussion. You could also drag a shortcut to the scrape.cmd batch file, edit the shortcut and add the path that you wish to process to the end of the target.

Please feel free to pm me if you have any questions or run into any problems. I'll be glad to work through the specifics with you.

Mick.

I'm not having luck with the batch processing. Individually dragging and dropping seems to work fine, but as soon as I type:

"J:\Downloads\SageTVAddons\scraper\scrape" "J:\Television\Battlestar Galactica\Season 3\"

into the command line I get:

Folder path is "J:\Television\Battlestar Galactica\Season 3\"

File Not Found

The file it should be looking for in the "Season 3" directory is "S03E01 - Occupation.avi" but it doesn't seem to find it. I've used an administrator command line as well to no avail. Any ideas? I'm hoping I'm just having a stupid moment and it's an easy fix. I haven't updated sageMC to the latest "b" version (still "a") but I can't imagine that would influence this in the least...other than that just an x86 Vista computer here...

[evilpenguin]

Try not including that trailing "\".

[evilpenguin]

Just updated to Beta5 and, since it includes actual features, I have release notes. The big ones are in bold.

mediaScraper 1.0 beta5

• Added Drag and Drop support for multiple files and folders
• Added support for VIDEO_TS folders
• Downloading fanart from theTVDB and theMovieDB
• Caching webpages so when doing multiple episodes of the same show we don't hammer theTVDB.
• Skip files that already have metadata. Can be overrided with /updateInfo
• Added customizable TV basefolders. (ex: /basefolder "TV Shows")
• Added defaults.txt which you can use to configure your default options.
• Added year to movie titles (ex: 300 (2006))
• Lots of minor RegEx tweaks.

[jaminben]

Originally Posted by evilpenguin

Just updated to Beta5 and, since it includes actual features, I have release notes. The big ones are in bold.

mediaScraper 1.0 beta5

• Added Drag and Drop support for multiple files and folders
• Added support for VIDEO_TS folders
• Downloading fanart from theTVDB and theMovieDB
• Caching webpages so when doing multiple episodes of the same show we don't hammer theTVDB.
• Skip files that already have metadata. Can be overrided with /updateInfo
• Added customizable TV basefolders. (ex: /basefolder "TV Shows")
• Added defaults.txt which you can use to configure your default options.
• Added year to movie titles (ex: 300 (2006))
• Lots of minor RegEx tweaks.

You soon banged that one out

One small request that I think would benifit other people and not just me is an option to select where to place fanart images rather than placing an image in the same folder as the TV show. Or rather than naming the fanart background the same as the TV show name it "background" as mine doesnt use the fanart unless I move it or rename it to "background". Other than that it worked first time, only tried Bones so far though, will try more later.

Very Cool Tool

Thankyou

edit:

I've been playing with the tool for awhile now and have noticed some issues with certain files not working and was hoping you maybe able to get these to work as well.

Example files that do work:

burn.notice.s01e06.720p.hdtv.x264.bia
Bones.s03e11.Xvid-Orpheus
stargate.atlantis.s05e20.hdtv.xvid-preair
smallville.s08e01.720p.hdtv.x264-ctu
lost.s04e13-e14.720p.hdtv.x264-ctu
sanctuary.us.s01e13.720p.hdtv.x264-ctu

Example files that dont work:

stargate.atlantis.s05e15.720p.hdtv
Heroes.S02E11.720p.HDTV.X264-DIMENSION
reaper.s01e18.720p.hdtv.x264-ctu
Star Wars The Clone Wars S1E05 HDTV XviD-SC-SDH

As you can see there isnt that much difference between the ones that do work and others that dont. It does appear to timeout on the files that dont work giving a text line of attempt#1, attempt#2 etc. But I could be reading it wrong and I dont really know what I'm looking at anyway. I've attached my scraper log just incase you need it.

Another request I have would be to remove unwanted text from the file name (see attched screenshot). So the original title of a file is "Bones.s03e11.Xvid-Orpheus" and a cleaned up file name would look like "Bones Season 3 Episode 11 The Pain in the Heart".

I also hope that one day the extra data avaiable via a .info file could be imported into Sage as it does seem a shame to waste all that extra information seeing as its available.

Dragging of folders worked superbly for Bones and Smallville I havent tried movies yet but I'll get around to that later.

Other than that, great job

Regards

Ben

[Halstead]

EP- thought you's like to know that MediaScraper seems to be playing along just fine with DirMon.

Any ETA on a fix/workaround for shows like Termiantor: The Sarah Connor Chronicles?

Oh, and where do i put the /basefolders string? in the defaults file?

Thanks again for all the hard work!

[evilpenguin]

Originally Posted by jaminben

One small request that I think would benifit other people and not just me is an option to select where to place fanart images rather than placing an image in the same folder as the TV show. Or rather than naming the fanart background the same as the TV show name it "background" as mine doesn't use the fanart unless I move it or rename it to "background". Other than that it worked first time, only tried Bones so far though, will try more later.

I wasn't really sure how to name the jpg I downloaded or where to put it so I just guessed. If you can give me some examples of what you're talking about I can get it in there pretty quickly.

Originally Posted by jaminben

Example files that dont work:

stargate.atlantis.s05e15.720p.hdtv
Heroes.S02E11.720p.HDTV.X264-DIMENSION
reaper.s01e18.720p.hdtv.x264-ctu
Star Wars The Clone Wars S1E05 HDTV XviD-SC-SDH

All of those, minus "The Clone Wars", is a bug where I thought that 720p was the season/episode. It was an easy fix and you can download just the updated profile and put it in your scrapingProfiles folder. This is just a quick and dirty patch, I'll add something a little more robust this weekend where it tries to strip out a lot of that crap that the encoders groups put in the file name.

The Clone Wars issue is kinda a tough one. theTVDB.com won't recognize it unless its has the ":" (Star Wars: The Clone Wars). Normally I'd fix this by running it through imdb, but imdb is kicking back "Star Wars: Episode II - Attack of the Clones" instead of the TV show. I've got the same issue with "X-Files" and I'm brainstorming for a good solution.

[evilpenguin]

Originally Posted by Halstead

Any ETA on a fix/workaround for shows like Termiantor: The Sarah Connor Chronicles?

Getting the metadata should be working now, but I ran into a new issue where it can't save the fanart because the name contains an illegal windows file name character (. I'll get that sorted out this weekend.

Originally Posted by Halstead

Oh, and where do i put the /basefolders string? in the defaults file?

Unfortunately, that's not working like I want it to yet. I'll put out a fix tonight and let you know how to add it then.

[jaminben]

Originally Posted by evilpenguin

I wasn't really sure how to name the jpg I downloaded or where to put it so I just guessed. If you can give me some examples of what you're talking about I can get it in there pretty quickly.

I use a central folder for films and tv shows I record via sage and these are located at \\SAGEWHS\Fanart\Movies & \\SAGEWHS\Fanart\TV. I place all my tv shows in a separate share all placed in their individual folders. All I then have to do is place a background.jpg in each series folder for it to work and I don't then need to rename each file everytime I get a new one. This is a bit long winded but it works.

What would be cool is if your scraper could rename a file to make it more friendly to fanart, something like my example above. I see you've also mentioned this in your first post.

* Add option to rename files after successfully getting metadata

This is the way I currently have my fanart setup and is proberly not the best way or what other people would use\like. Another thought would be to allow the user to specify where they place fanart using your default file. I don't know how that would work or even if it is workable.

Originally Posted by evilpenguin

I'll add something a little more robust this weekend where it tries to strip out a lot of that crap that the encoders groups put in the file name.

Using your temp file I got all to work except:

stargate.atlantis.s05e15.720p.hdtv

I didn't try starwars.

Regards

Ben

[justme]

Much Thanx. I'm glad to have a CLI batch scraper again.

2 requests ,please, while you're still fairly early in development...

1) Can we have the metadata files created in a user specified location(either via a settings file, but preferably a command line switch)? This is needed for files that are already archived to DVDs and network shares that lack write acccess permission/ability. I know that 2 of the .my metadata import options for SageTV allow metadata to be put in a subdir, or a common user specified dir. This allows .my files to function alot like VSfilter looks for subtitles. ie
.\metadata would put the files into a subdir of the show folder called "metadata". This is usefull for keeping clutter down in the dir your media files are stored in. With the small downside of adding a metadata subdir to each of your show directories that contain media files. Still, as an example, one subdir. added is "cleaner" than 25 .my files added to a show folder for a series that has 25 episodes.
c:\metadata is obvious where they'd go. This is usefull for the write access issues I mentioned with DVDs and certain network shares. This is a big one for me as I have many shows already archived/burnt to DVD before we started using .my files. I obviously don't wish to reburn those disc, but would love to have your tool be able to make metadata for them.

2) Can you add the ability to handle files that don't use season numbers? Instead they are numbered by absolute Episode count. An Example of something like this would be the american version of "Dragonball Z". It has 276 reqular episodes, not counting movies. So a file like "Dragonball Z-EP201.avi" would be proccessed as the 201st episode(aka "A Dark and Secret Power") instead as episode 1 of season 2(aka "A New Goal... Namek!").
I realize that this would need a switch/setting(so the app knew which way to process the file), as other use may want that same files proccessed as season2 episode 1. Still this was a nice feature in the older TV.com scraper before it was broken by site changes. I would really appreciate if you could add this functionality to you scraper, as I rarely ever use season numbers in either the directory tree structure or filename. IMHO season info is one of the many pieces of info that metadata files are for.

Apologies if your app is already capable of any of these actions. I just found this thread today and my file server is torn down for a rebuild(upgrading CPU and harddrives) this weekend. So I haven't yet gotten a chance to truely test your app of my media collection. I just did some simple tests using fake(empty text files renamed to avi) files.

Thank you for all your time and effort,
/justme

[evilpenguin]

Originally Posted by jaminben

This is the way I currently have my fanart setup and is proberly not the best way or what other people would use\like. Another thought would be to allow the user to specify where they place fanart using your default file. I don't know how that would work or even if it is workable.

I'm all about supporting what the SageMC dev's and plug-in dev's such as yourself decide is the preferred method. So if that's what you like I'll make sure it gets in there.

Originally Posted by jaminben

What would be cool is if your scraper could rename a file to make it more friendly to fanart, something like my example above. I see you've also mentioned this in your first post.

My current TV folder is incredibly ugly, which actually makes it good for my testing . I've already laid all the ground work for auto-organizing files, now I juts have to double and triple check my work to make sure I don't blow out peoples files.

Originally Posted by jaminben

stargate.atlantis.s05e15.720p.hdtv

Hmm, thought I tested that one, oh well, I'll get that sorted out over the weekend.