Utility: mediaScraper - metadata made easy!

[evilpenguin]

Originally Posted by justme

1) Can we have the metadata files created in a user specified location(either via a settings file, but preferably a command line switch)?

Consider it done.

Originally Posted by justme

2) Can you add the ability to handle files that don't use season numbers? Instead they are numbered by absolute Episode count. An Example of something like this would be the american version of "Dragonball Z".

Well, first of all, i'll prolly never support TV.com... its really ugly from a development standpoint.

Secondly, theTVDB has its anime sorted by arc/episode which is going to be the easiest for me to support. But, if you have a lot of stuff in that format I can look at trying to translate between total # and arc/episode # with a special option.

[jaminben]

Originally Posted by evilpenguin

Hmm, thought I tested that one, oh well, I'll get that sorted out over the weekend.

Ooops my mistake, I've just tried again and it works


edit:

I've found a couple more examples that dont work:

burn.notice.203.hdtv-0tv
burn_notice.2x04.comrades.hdtv_xvid-fov

I maybe getting a bit extreme but it would be good to get a 100% hit rate

[evilpenguin]

Wow, you have way better test content than I do because both of those hit some simple bugs right on the head

Both are easy fixes, just need to allow show title words to be separated by "_" and ".".

[jaminben]

Originally Posted by evilpenguin

Both are easy fixes, just need to allow show title words to be separated by "_" and ".".

Cool

TVDB also supports series banners which could be used as coverart, any chance you could grab this as well? This would be a neat way to make different seasons stand out.


Regards

Ben

[mickp]

Originally Posted by evilpenguin

I'm all about supporting what the SageMC dev's and plug-in dev's such as yourself decide is the preferred method. So if that's what you like I'll make sure it gets in there.

My current TV folder is incredibly ugly, which actually makes it good for my testing . I've already laid all the ground work for auto-organizing files, now I juts have to double and triple check my work to make sure I don't blow out peoples files.

Hi E.P.

My directories tend toward chaotic. For me the best way would be to have the fanart placed in the same directory as the media file itself as per Brent's instructions below;

Originally Posted by brent

If instead of the Central FanArt Directory Method described in step 3 above you want to place your backgrounds locally with the individual media files follow these steps:

Movies - Rename your Movie FanArt background image to be "background.jpg" and place it in the same location where the media is. For example if you have a ripped DVD located in DarkKnight/Video_TS/ you'll put your Background in DarkKnight/Video_TS/background.jpg.


For non-DVD format video files rename your video FanArt background image to be VideoFileName_background.jpg

Relying on having tv shows grouped into their own sub directories wouldn't work well for me most of the time. It makes sense if you're manually placing the files but for an automated system probably not so useful.

Having a command line switch to determine where/how they go is obviously a good option.

Mick.

[mickp]

Hi E.P.

Could you also download the poster (first available I guess) as filename.jpg (where filename is the name of the file)?

Mick.

[mickp]

Help!!!

I'm very pleased at the moment with the accuracy of the metadata using the existing filenames for most of my files. I've been going through doing a bit of tidying, renaming files where there's obviously a problem to see if I can get a match. Mostly this seems to be working with the exception of "Doctor Who (2005)".

I've tried various permutations but can't get it across that the show name is not "Doctor Who" (I have the original 26 seasons in a different directory ).

The desired series is:

File name is http://www.thetvdb.com/?tab=series&id=78804&lid=15

An example filename is:

\\samael\movies\Doctor Who (2005)\Doctor Who (2005) season two\Doctor Who (2005) s02e11.avi

Anyone have any suggestions?

Mick.

[justme]

Originally Posted by evilpenguin

Consider it done..

Wow, thanks very much. This(user specified output location) will come in very handy for me. Hopefully to others as well.

Originally Posted by evilpenguin

Well, first of all, i'll prolly never support TV.com... its really ugly from a development standpoint.

Sorry about that. I didn't mean to confuse the issue with a link to TV.com. I somehow got it in my head that you were using some kind of hybrid method that involved both TVDB and TV.com. Nevermind, I'm happy with TVDB support. It's a nice user supported site. And I agree about TV.com being ugly, but not just from a development side... Ever since the recent change(I think ABC bought them?), the HTML/CSS formatting has been bad. For a while some show pages were totally unusable in IE7, due to 4 or more hyperlinks appearing visually to be stacked on each other. This made selecting or even reading the text in the links nearly impossible. It's a shame, before the recent changes they were my favorite TV site. Now my wife won't even use it, at all, anymore.

Originally Posted by evilpenguin

Secondly, theTVDB has its anime sorted by arc/episode which is going to be the easiest for me to support. But, if you have a lot of stuff in that format I can look at trying to translate between total # and arc/episode # with a special option.

I see what you mean about the difference in sorting. TV.com says that EP201 is season 12 episode 7, while TVDB says it's Season11 episode 7. That's not really a big deal to me and DBZ has many disputes about season orderng, since the american version was chopped by ~20 episodes. The important thing IMO is that they both show it as the same episode title. In hindsight DBZ was a bad example to use.

As far as my gerenal request for the ability to parse for metadata using absolute episode numbers... I'd appreciate it if you could come up with that option to translate the numbering, as you mentioned. It's not just anime, that I have like that. I also tend to use that system for other/most TV shows, too. Like I said, I feel metadata is where things like season number go.

So I tend to use a format of Showtitlte-EPXXX-*.avi, where X is the absolute episode number and * is often the episode title, but not always included.The only real variation is that if a show has under 100 episodes I use only 2 digits. So for a series that had 20 episodes a season and less that 5 seasons... ShowTitle-EP67-*.avi Would be episode 7 of season 3. Showtitlte-EP01-*.avi Would be episode1 of season 1. It's only when a series has an episode count of >= 100 that I use 3 digits, as in my DBZ example. I do this so the episodes are sorted properly when viewed/managed from a Windows file explorer. Without the addition of leading 0s the lower episodes( ie 1-9) are interlaced through the file list.

Thanks, for the great work and especially for listening to requests.
/justme

[evilpenguin]

Originally Posted by mickp

Anyone have any suggestions?

Someone had an idea earlier that I liked about putting an override.txt file in the folder with videos with difficult titles. Once I do that all you need to do is make an override.txt file with the exact theTVDB.com name and it was automatically use it rather than trying to figure it out from the file.

[evilpenguin]

Originally Posted by justme

As far as my gerenal request for the ability to parse for metadata using absolute episode numbers... I'd appreciate it if you could come up with that option to translate the numbering, as you mentioned.

The best I'd be able to do is grab a list of all of the episodes in the series and blindly take the N'th one. Implementing it will be a bit complicated so I wouldn't start working on it until everything else is working solid.

[evilpenguin]

I'm starting to see that I just don't have enough test content so I created a tool that people can use to scan their videos folders and create a zip file that mirrors their directory structure with dummy, empty, video files that I can use for testing.

The usage is pretty simple, just drag and drop the base folder of your video directory onto zip4test.exe and it'll do the rest.

So, if your comfortable sharing your entire directory structure with me, I'd appreciate it if you would use the tool and send me the resulting zip file @ mediaScraper(at)gmail(dot)com. It'll really make my testing a lot easier.

[Addict]

Originally Posted by evilpenguin

I'm starting to see that I just don't have enough test content so I created a tool that people can use to scan their videos folders and create a zip file that mirrors their directory structure with dummy, empty, video files that I can use for testing.

The usage is pretty simple, just drag and drop the base folder of your video directory onto zip4test.exe and it'll do the rest.

So, if your comfortable sharing your entire directory structure with me, I'd appreciate it if you would use the tool and send me the resluting zip file @ mediaScraper(at)thegmail(dot)com. It'll really make my testing a lot easier.

I'll send my directory, but this tool doesn't create blank .ts files...which is the majority of my files now thanks to the HD-PVR.

[evilpenguin]

Originally Posted by Addict

I'll send my directory, but this tool doesn't create blank .ts files

It does now

[megamojo]

Any way we could be able to turn off/specify which fields get created in the files? Reason I ask is that the .my files I'm getting look like this one, for example:

Title=Private Practice
overview=From Shonda Rhimes, the Golden Globe-winning creator of "Grey's Anatomy" comes a story about finding a way to begin the rest of your life. Addison Forbes Montgomery is a renowned surgeon. Having left behind Seattle Grace Hospital, a broken marriage with Derek "McDreamy" Shepherd, and her torn relationship with Mark "McSteamy" Sloan, Addison moves to Los Angeles for sunnier weather and happier possibilities. Addison reunites with her friends from medical school, Naomi and Sam Bennett, joining them at their chic, co-op, Oceanside Wellness Center in Santa Monica. Naomi, a fertility and hormone specialist and majority owner of the clinic, and Sam, an internist whose self-help books have made him the "common man's medical guru," are newly divorced. They co-exist for the sake of their teenaged daughter, Maya, and their medical practice, but the relationship is hardly amicable. The cent
TVOriginalAiringDate=2009-01-15
actors=Kate Walsh, Audra McDonald, Taye Diggs, Kadee Strickland, Timothy Daly, Paul Adelstein, Amy Brenneman, Chris Lowell,
TVEpisode=2x12 - Homeward Bound
TVOverview=With her relationship with Kevin on the rocks, Addison finds herself attracted to Wyatt. Violet looks for a way to have a relationship with both Pete and Sheldon. Meanwhile, Cooper and Charlotte become closer after a family emergency.
TVDuration=60 minutes
TVGenre=Drama
TVChannelName=ABC

As you can see, the overview field is just general info that applies to the whole series, which is pretty much useless on a per episode basis. The TVOverview field contains exactly what I want. I use the R/W metadata plugin to add the info from the .my file into the SageTV DB, which I find a lot more pleasant than somehow forcing the files to seem "new" to sage so that the .properties file gets parsed. With the old tv.com scraper, only the TVOverview field existed, so the show description was perfect. Now that a field called just "overview" exists, that is what gets added as the description of the show, making the description that sage displays on the screen pretty much useless.

[evilpenguin]

Originally Posted by megamojo

Any way we could be able to turn off/specify which fields get created in the files? Reason I ask is that the .my files I'm getting look like this one, for example:

So... what field do you want out of what file?

[justme]

Originally Posted by Addict

I'll send my directory, but this tool doesn't create blank .ts files

Originally Posted by evilpenguin

It does now

It didn't catch .ogm files in my quick test in my fake dir. Can you add that too?
BTW this little tool is something I've been looking for, for a long time. The closest thing I found cost 20 dollars. In other words your test tool is a great resource that I'd use myself, even if you didn't want my dir structure. As long as that is OK with you, that I use it for other personal uses.

I'll gladly run it the one video directory I currently have access to, the others will have to wait till I rebuild my main media server(hopefully Sunday). Just keep in mind the structure is a little disorganized on this directory, as it is my temp holding directory for anime.

The best I'd be able to do is grab a list of all of the episodes in the series and blindly take the N'th one. Implementing it will be a bit complicated so I wouldn't start working on it until everything else is working solid.

That's cool. I'd appreciate anything you could do. Besides it's your tool and your time, so I'm just glad you will consider it.

Thanx again, especially if you say I can use zip4test for my personal use.

PS: Do you accept donations? I'd gladly give you a few dollars for .ogm support in zip4test, not to mention your work on the metadata tool. It wouldn't be much but I like to support tools I like and use.

/justme

[Addict]

Originally Posted by evilpenguin

It does now

Nice. Sending over my directory shortly. Tons of mis-hits or non-hits on my directory though. I would recommend the following in your parsing (apologies if this has already been discussed, haven't read the entire thread):

1) Look for the show ID and go from that...if nothing found, or no show ID, then...
2) Look for capital letters in the file name, and add spaces in front of them...then search by show name, title, etc...

Most of my files look like this, and most are failing. Typical SageTV file naming structure:
TheColbertReport-5960988-0.ts
Top20Countdown-5975341-0.ts
Survivorman-AustralianOutback-5671255-0.ts
DestroyedinSeconds-5869892-0.ts
FamilyGuy-BlueHarvestParts1and2-1303810-0.mpg

You get the point...thanks for working on this!

[justme]

Just sent mine. Just keep in mind likely 1/3 of the files were missed since they were .OGM. Also like I said above the structure and naming may be erractic. I also haven't run your metadata tool on this entire dir yet, due to being a holding dir that all files and Dirs will be moved out of. Also these shows are almost entirely anime and some rare in the US(hence not in many english databases). So who knows what will happen when you run a test on it.

I like the idea of a txt file in the dir as a override to find the "true" showtitle. It worked well as a last resort fallback method in the old TV.com scraper.

/justme

[evilpenguin]

Originally Posted by Addict

1) Look for the show ID and go from that...if nothing found, or no show ID, then...

I don't handle SageTV recordings yet, but i'll add that eventually add it by using the episodeID and running it through the web server.

[megamojo]

Originally Posted by evilpenguin

So... what field do you want out of what file?

In this case, I want overview out of the .my file, but TVOverview should stay.

And btw, you're awesome for making this and especially for listening to us the users and responding so kindly and quickly to our requests and comments, kudos to you!