Utility: mediaScraper - metadata made easy!

[heffe2001]

I sent a zip4test pack to you just a minute ago, hopefully I have some good, odd filenames for you to test against, lol.

[justme]

I noticed you updated zip4test to add OGM. Thank you. It worked great as far as I could see.

So... I just sent you a new email with the new zip attached. It's of the exact same directory as the first email, but now has those ogm files. To make it clear which zip is which, I renamed the new zip file to "VIDEOS(with OGM)".

/justme

[evilpenguin]

Thanks for all the .zip files, they've helped me track down a bunch of bugs this weekend

BTW this little tool is something I've been looking for, for a long time. The closest thing I found cost 20 dollars. In other words your test tool is a great resource that I'd use myself, even if you didn't want my dir structure. As long as that is OK with you, that I use it for other personal uses.

I'm glad/surprised you liked it. I put out all my work under the GPL (Generel Public License) so you're free to use it as you like. I'll also expand zip4test a little bit this week to make it so you can specify all the files types you want it to catch at the command line, I imagine that'll make it more useful for you.

Also, I don't take any donations, but I'd encourage you to send a few dollars over to theTVDB who provides all the great metadata I use.

Originally Posted by megamojo

In this case, I want overview out of the .my file, but TVOverview should stay.

I'll take that out of the next version, I don't really see a reason for it to be there either.

[justme]

Originally Posted by evilpenguin

Thanks for all the .zip files, they've helped me track down a bunch of bugs this weekend .

Glad we could help you. But for me it was no trouble at all. Especially since we are helping you help us, by making your metadata scaper work with our files. So thank you for looking over the .zip files.

Originally Posted by evilpenguin

I'm glad/surprised you liked it. I put out all my work under the GPL (Generel Public License) so you're free to use it as you like...

Yeah I definitely find it usefull. I originally wanted a tool like this for making a mirror of video files on DVDs, so I could generate metadata using these local copies on my hard drive. The fact that you're adding user specified output location, makes this extra step unneccessay. Still zip4test will be very handy for helping make a mirror(on my harddrive) of all my media that's on DVD. This will help me in tracking my library, and that's only one use. So another, thank you, for releasing it under GPL.

Originally Posted by evilpenguin

I'll also expand zip4test a little bit this week to make it so you can specify all the files types you want it to catch at the command line, I imagine that'll make it more useful for you.

Very cool and yes this functionality will make it even more usefull. I was actually considering trying to modify the .pl sourecode version on your site. I think I found the right line but... I'm not sure it's the only line that needs modifing and I certianly don't know how to compile a exe from the .pl. Let alone how to add a zip compression library/tool into the exe.

Originally Posted by evilpenguin

Also, I don't take any donations, but I'd encourage you to send a few dollars over to theTVDB who provides all the great metadata I use..

Cool with me. It's done, as of this post. I used the link you gave me to give them $5. It's not much but I'll make a note to try and make another donation in the future. I'd do it sooner but I see many people who accept donations say to wait and save up to make a larger donation. Appearantly(according those sites) smaller donations like $1, can be largely ineffective due to paypal proccessing fees.

Anyway, looking forward to future versions. Just don't work too hard.
/justme

[megamojo]

Originally Posted by evilpenguin

I'll take that out of the next version, I don't really see a reason for it to be there either.

Sounds fine. I just deleted the line from the .my maker in the profiles directory and all is well. I'm really not sure why I didn't just look before coming on here begging for help. Oh and one more thing, I can't seem to get the scraper to work for the new Knight Rider series. If I leave the file/directory names as Knight Rider, it grabs data from the old show with Hasselhof, and if I name it as "Knight Rider (2008)," which matches with tvdb exactly, it won't find any info. I'll attach a log file when I get home. Thanks for all your work!

[evilpenguin]

Originally Posted by megamojo

If I leave the file/directory names as Knight Rider, it grabs data from the old show with Hasselhof, and if I name it as "Knight Rider (2008)," which matches with tvdb exactly, it won't find any info. I'll attach a log file when I get home. Thanks for all your work!

I've got a fix/workaround for that kind of issue that will hopefully be out tonight-ish.

Originally Posted by megamojo

I'm really not sure why I didn't just look before coming on here begging for help.

You should feel free to let me know if somethings not working like you'd expect it to. I'll shoot you down if I disagree but more often than not its either something I haven't fully thought out or I haven't looked at the results in Sage.

[megamojo]

It doesn't seem to like files with an ! at the end. In one case the entry on thetvdb.org also didn't have that at the end, but in another it was there. In both cases it didn't find anything. I'm talking about episode 12x05 of South Park and 2x16 of Gossip Girl if you want to know.

[evilpenguin]

It doesn't look like much, but I changed a ton of code so be sure and let me know if I broke anything.

mediaScraper 1.0 beta6

• Fixed UTF-8 issues so now special characters like quotes and accent marks should be working. This also includes show titles with accent marks, although you'll probably need to use a folder.override for those.
• Added folder.override files for shows with tough names that otherwise would fail (more on this below)
• If a folder has a mediascraper.skip file in it then it and its sub directories won't be scanned. Useful if you have folders full of random clips and home movies mixed in with your TV/Movies.
• Reworked RegEx's so that movie files with the year not in () get recognized (Juno 2007.avi).
• TV files that don't have a separator between the season/episode (7x22 = 722) will only work when the season is under 10 (10x01 != 1001). That should make 99% of people happy.
• Added ability to auto rename files. (More on this below)
• Downloading folder.jpg and banner.jpg in addition to fan art.
• Added ogm and ts to list of file extensions to look for.
• Added year, run time, and rating to movie .properties file
• Moved all downloading and output to the end of the profiles
• Created new profile with all downloading functions
• Lots of minor tweaks and improvements to the profiles
• Reworked defaults.txt so it all goes on one line and won't break if you put a path in quotes with a space in it.
• Removed overview from .my files
• Fixed bug where an ! in the filename would make the conditionals think they were negated

folder.override:
If you have a show/movie that my scraper just can't handle for one reason or another, all you need to do is create a text file next to it named folder.override or whateverTheFileNameIs.override that just contains the exact show title to search for in theTVDB or IMDB and i'll use that instead of trying to guess it from the file name. One folder.override will work for everything in that folder and you can also place a folder.override it in the parent directory if you want it to apply to all folders above it.


/centralFanartPath "G:\Videos\Fanart":
Mike and Dirk put a lot of hard work into getting fanart working, the least we can do is help them out and naming and organizing our fanart in the cleanest way possible. If you put this into your default.txt then all fanart will be downloaded into a central folder.

/organizeFiles:
If you put this switch into defaults.txt then if mediaScraper finds the metadata for your video it will in place rename it to the the format...

Scrubs 2x01 - My Overkill.mkv

The Matrix Revolutions (2003).avi

/organizeBaseFolder "G:\Videos" :
If you want to go the extra mile, you can also set a basefolder to organize into...

G:\Videos\TV\Scrubs\Season 2\Scrubs 2x01 - My Overkill.mkv

G:\Videos\Movies\The Matrix Revolutions (2003).avi

I'm currently working on ripping/encoding all of my movies and TV shows to my HTPC and using this makes everything much easier.

zip4test.exe:
Before you try to batch organize your files you'll probably want to test it out first to make sure it doesn't screw anything up. You can do this by dragging your video base folder onto zip4test.exe which will create a mirror of your video folder in a zip file using empty video files. You can then extract that zip to somewhere safe and try running it through mediaScraper to see if everything works all without touching your real videos.

[jaminben]

Very nice work EP
I'll have a go at this later today.

[mickp]

Thanks E.P.
I notice that it doesn't appear to be dowloading banners etc. Is there a switch I'm neglecting? Also don't notice the fanart appearing. Does it only come down at all now if you're using a central directory?

If so any chance of implementing the following if you've not specified a central directory?

Originally Posted by brent

If instead of the Central FanArt Directory Method described in step 3 above you want to place your backgrounds locally with the individual media files follow these steps:

Movies - Rename your Movie FanArt background image to be "background.jpg" and place it in the same location where the media is. For example if you have a ripped DVD located in DarkKnight/Video_TS/ you'll put your Background in DarkKnight/Video_TS/background.jpg.


For non-DVD format video files rename your video FanArt background image to be VideoFileName_background.jpg

Mick.

[jaminben]

Originally Posted by mickp

Also don't notice the fanart appearing.

You've proberly already done this but I added the defaults file from the previous beta to grab the fanart and used the line

/downloadFanArt

Just tried it with Lost and all is working for me.

[mickp]

Originally Posted by jaminben

You've proberly already done this but I added the defaults file from the previous beta to grab the fanart and used the line

/downloadFanArt

Just tried it with Lost and all is working for me.

Hey Jaminben,

I didn't have it in the first instance but went looking, found it missing and put it back. I'm not seeing any images comming down tho .

Mick.

[jaminben]

Originally Posted by mickp

Hey Jaminben,

I didn't have it in the first instance but went looking, found it missing and put it back. I'm not seeing any images comming down tho .

Mick.

I've just tried again and its not working now, very strange. It defo worked the first time as I just created a new folder for the new season of lost and didn't have any fanart for that show. Perhaps the website is not working correctly?

I also didnt have the default file in beta 6 but just copied it from beta 5, not sure if this the correct way to do it but I tried anyway.

edit:

I think I must have mixed up my folders and used the wrong beta as I can't get it to work now.

My bad, Sorry.

[megamojo]

Originally Posted by evilpenguin

[*]Removed TVOverview from .my files

I hope you mean you left TVOverview and removed overview. TVOverview has the episode summary, which is nice, overview has a summary which applies to the entire show and is always the same, which is almost useless.

Edit: Yes, you did it the right way now that I look at the code. Might want to edit your post so as not to confuse people. Thanks for all your work, adding renaming is completely amazing! The other renamers I've tried are not console based and so can't be scripted, and they're extrememly unstable. In fact, so far the only way to get them usable for me has been to wrap them in a service with proper recovery options. I'm excited to try yours out!

[evilpenguin]

Opps, I forgot to include the updated defaults.txt

mediaScraper 1.0 beta6a

• Added defaults.txt to download

I updated the download, but you can also just grab the attachment here and overwrite your current one.

[megamojo]

Is there any way we can customize the way the files get renamed? My own personal format of choice is almost exactly like yours, minus the dash between the episode number and episode title. I definitely don't mind using yours, but I'm sure some people are pretty used to other formats such as SxxExx for the numbers, and maybe they don't want the show name in the filename, just the episode number and the episode title for instance. I see 2 easy ways to change it. Either make it something we can set in defaults.txt where we enter the variable names and whatever punctuation we want for formatting, or just describe how to change the source code without breaking it. I've looked at doing that and I'm sure I could figure it out, but it's not something I immediately knew how to do just by looking.

[evilpenguin]

Yeah, I figured people would want to name stuff their own way. The defaults will always be how I use it, but I also added two extra switches so people could customize it fit their own preferences.

Default:

/organizeTVSeasonXepisode: Scrubs 2x01 - My Overkill.avi

Additional Options:

/organizeTVSeasonEpisode: Scrubs S2E01 - My Overkill.avi

/organizeTVEpisode: Scrubs E01 - My Overkill.avi

And any other customizations you want to make can be done by manually editing organize.scrape. For instance for what you want you'd change this function to remove the dash...

organize.scrape:

   51 #
   52 # Snip: TVseasonXepisode
   53 #
   54 
   55     Profile           =TVseasonXepisode
   56     Encode CLI #1     =?>!organizeTVNoTitle<:>%%showTitle_WIN32%% <?%%seasonNum%%x%%episodeNumFill%% %%episodeTitle_WIN32%%.%%inputFile_EXT%%
   57     Encoder #1        =

[megamojo]

Ah, makes perfect sense now. I couldn't figure out why it seemed that I'd need to remove the dash in 3 places, which is why I was confused. I didn't notice that it was 3 different built-in ways to name it. Since you're so good with requests, can you add in a few more custom switches for us? Here's some that I would like.
/doubleBankAccountBalance
/doTheDishes
/doMyTaxes
/cleanTheHouse
/fixTheEconomy
/change201kBackTo401k

I'm sure others can come up with more, but that should be a good start.

[jaminben]

Originally Posted by megamojo

/doubleBankAccountBalance

I'd also like to see this in a future version. Could you also make it autorepeat maybe 10 or 20 times.

Very nice job

[Halstead]

Awesome additions, EP. Found one issue on the auto renaming- it seems like the show title is getting partially appended on both sides of the season/episode section for Terminator: The Sarah Connor Chronicles (is this show becoming your bete noir?).

So:
Terminator.The.Sarah.Connor.Chronicles.s02.e10.720p.XVID.[VTV].avi

Becomes:
Terminator The Sarah Connor Chronicles 2x10 - The Sarah Connor Chronicles Strange Things Happen at the One Two Point.mkv

I checked the episode names on TVDB, and they don't have this issue (e.g. the episode is just titled 'Strange Things Happen at the One Two Point'). FYI, I'm using the folder.override option, which worked like a charm. Could it be the hyphen?