Utility: mediaScraper - metadata made easy!

[Phydeaux]

Fabulous utility! Thanks so much for sharing this and the zip4test tool that is essential for a guy like me who has no Internet connection at home. The tool allows me to take a light copy of my media structure to work where I DO have an Internet connection and can collect the data to take home. Only on my lunch hour, of course...

One question, one issue. The issue I have is the MASH TV series, where the entry at TheTVDB is listed as M*A*S*H. Clearly I can't make name series directory the same way...

The question is - where can I find some documentation? Could a basic text file be created which lists options, switches etc and general usage suggestions? I realize some of that is scattered throughout this thread, but I'm thinking a helpme file distributed with the utility might be handy.

RLW

[evilpenguin]

Originally Posted by Phydeaux

The question is - where can I find some documentation? Could a basic text file be created which lists options, switches etc and general usage suggestions? I realize some of that is scattered throughout this thread, but I'm thinking a helpme file distributed with the utility might be handy.

RLW

Hi, you're absolutely, right, I need to get some documentation done. I was just waiting until the scraping accuracy was good enough, which I think it is now. Keep an eye on the 2nd post, i'll get that updated with all the necessary info.

[megamojo]

Originally Posted by jaminben

Now all we need is to have it automatically detect when the top folder changes in size and have it scan on its own so I dont have to interact with it at all

I just script it using dirmon2. It's working extrememly well for me and I don't have to touch a thing.

[evilpenguin]

Originally Posted by megamojo

I just script it using dirmon2. It's working extrememly well for me and I don't have to touch a thing.

It automatically ignores any file where all the specified output files (my/property/info) exist, so it should be pretty efficient at running periodically.

[megamojo]

Originally Posted by evilpenguin

It automatically ignores any file where all the specified output files (my/property/info) exist, so it should be pretty efficient at running periodically.

That's pretty much what I have dirmon set up to do. The nice thing about dirmon is that it's easy to just have it run on a few directories instead of your entire TV folder, and it also checks on whatever schedule you want, instead of the rather limited options you get with windows scheduled tasks. Using it this way enables me to have random media that I don't want metadata for in the same root directory without worrying about mediascraper trying to find metadata for it. So, I have a dozen or more jobs in dirmon that monitor individual show directories and look for .avi files that don't have corresponding .my files. When it finds these files, it runs a little file I made called meta.bat that lives in each show directory. The batch file is nothing more than this.

"D:\Program Files\mediaScraperBeta8a\mediaScraper.exe" "%~dp0"

The %~dp0 just tells it to run on the directory that it is in. In order to make it easy to change without updating the batch file in every single directory (such as every time a new version of mediascraper comes out), the batch files are all symlinks on my linux box of one master file. Any time I update any of them they are all updated. I have dirmon set to check every 3 minutes for new .avi files that don't have .my files to go with them. Every 3 minutes is a little overkill during most of the day, but in the evenings when new TV shows come out I like the shows to be automatically renamed and organized right away.

I'm interested to hear anyone else's automation solution.

[popechild]

Very awesome utilty! I just started trying to use it, so I may end up with further questions or issues, but so far of the two series I've tried running it on, one of them worked great (Damages) and one it didn't recognize (Flight of the Conchords). I'm attaching a .zip from zip4test of the parent folder for both the shows, plus the log when I run it on an episode of FOTC.

Thanks for any guidance you can give if I'm doing something wrong.

[popechild]

Hmm... Okay. Just tried to run this on all of my ripped tv series, as that's 99% of what I'd be using this for, and it's getting all kinds of crazy results. Not sure if it's the way I have them organized or it's something else I'm doing wrong. Any chance that you're defaulting to looking for movies when it finds a VIDEO_TS folder? Because I have all of the individual episodes ripped from their original dvds to individual VIDEO_TS folder.

I've attached another zip4test for this directory structure so you can see what I'm talking about.

[evilpenguin]

Originally Posted by popechild

and one it didn't recognize (Flight of the Conchords).

Downloaded TV\Flight-of-the-Conchords\Flight.of.the.Conchords.S02E01.HDTV.XviD-aAF.avi

If you use a central TV folder like that, then I assume that you've already done a little organization yourself and am using the folder name for the series title rather than trying to pull it out of the file. In this case I'm using Flight-of-the-Conchords and those hypens are messing it up. Remove them and you should be fine.

Hmm... Okay. Just tried to run this on all of my ripped tv series, as that's 99% of what I'd be using this for, and it's getting all kinds of crazy results.

Wow, i've never seen anyone organize their TV like that before. Let me play around with that zip4test file and see if I can't work something out for you, you've got it organized so well I don't think it will be a problem.

[popechild]

Originally Posted by evilpenguin

Downloaded TV\Flight-of-the-Conchords\Flight.of.the.Conchords.S02E01.HDTV.XviD-aAF.avi

If you use a central TV folder like that, then I assume that you've already done a little organization yourself and am using the folder name for the series title rather than trying to pull it out of the file. In this case I'm using Flight-of-the-Conchords and those hypens are messing it up. Remove them and you should be fine.

Okay, I think I can do that. That folder (with the hyphens) was created by the automated torrent download program that I use. I would think I could specify the name somewhere (otherwise the next episode that downloads will just recreate a new FOTC folder with the hyphens again). I'll look into it.

Wow, i've never seen anyone organize their TV like that before. Let me play around with that zip4test file and see if I can't work something out for you, you've got it organized so well I don't think it will be a problem.

How do people typically have it organized? I was thinking series/season/episode would be pretty standard. Or does it have more to do with the fact that I did VIDEO_TS rips of each individual episode? I decided to go with that instead of the full disk rips because that way I could get individual metadata for each episode, which wouldn't have worked very well if I had 3-4 episodes per VIDEO_TS rip. Either way, thanks for checking it out for me.

[evilpenguin]

Originally Posted by popechild

Okay, I think I can do that. That folder (with the hyphens) was created by the automated torrent download program that I use. I would think I could specify the name somewhere (otherwise the next episode that downloads will just recreate a new FOTC folder with the hyphens again). I'll look into it.

Go ahead and check the 2nd post for how to use folder.override files. That should do what you need.

Originally Posted by popechild

How do people typically have it organized?

I meant most people encode 'em

I made a few tweaks to the profiles so that these will start working, i'll do an update tonight. Just make sure that you specify your /baseFolder "???" for those DVD's so that my tool knows to pull the series/season info out of the folder names.

[popechild]

Originally Posted by evilpenguin

Go ahead and check the 2nd post for how to use folder.override files. That should do what you need.

That's actually one thing I had trouble understanding how to use. Do i put anything IN the folder.override file? Where exactly does it go, and what does it do? The part about it using exactly what thetvdb expects is confusing to me. Do I need do tell it what thetvdb expects? What's the difference between using folder.override and "shownname".override? Sorry, that whole section just isn't making sense for me.

I meant most people encode 'em

Ahh, got ya. I just had so many episodes to do I figured this way would be alot quicker than encoding everything.

I made a few tweaks to the profiles so that these will start working, i'll do an update tonight. Just make sure that you specify your /baseFolder "???" for those DVD's so that my tool knows to pull the series/season info out of the folder names.

Will do. I actually didn't even notice baseFolder when reading the list, I noticed /organizeBaseFolder, which I knew I didn't want, and then forgot about /baseFolder by itself.

Edit: Also using VIDEO_TS folders isn't optimal from a fanart standpoint as it requires the same images to be downloaded and placed in each VIDEO_TS folder where as if they were all encoded you'd only need one image per season.

Hadn't thought about that. I haven't had any real issues with speed loading the fanart, and since I'll only really have to do this once per show, I'm not too concerned about extra speed downloading the fanart. I just can't imagine how long it would take me to encode everything for so many episodes.

[evilpenguin]

Originally Posted by popechild

That's actually one thing I had trouble understanding how to use. Do i put anything IN the folder.override file? Where exactly does it go, and what does it do? The part about it using exactly what thetvdb expects is confusing to me. Do I need do tell it what thetvdb expects? What's the difference between using folder.override and "shownname".override? Sorry, that whole section just isn't making sense for me.

I rewrote that section give it a read again and see if it makes sense.

[mickp]

Originally Posted by evilpenguin

The problem is that I can't detect if your shows are in a dedicated directory or not. I have to make the assumption that they're at least organized in separate folders by series name otherwise I'd be wasting theTVDB's bandwidth by downloading the same images over and over.

The best way around this would be to take advantage of the auto-organizing I added in the last beta which would sort the videos before trying to download the images. All you'd need to do is add these lines to your defaults.txt...

/organizeFiles /organizeBaseFolder "\\hal\hmovies"

I'm sure that the auto organizing feature is fine though it wouldn't be very practical on an active torrent directory as many of the files are still being shared (need to remain in the same spot) for some time. Moving tv series which are only partially downloaded would also not be pretty.

Re: bandwidth, I too was concerned about the ammount of bandwidth that would be used so checked the .info file and found that the paths appear to be the same for the same shows. I'm pretty sure that my transparent proxy will save thetvdb from abuse.

From what I can see the file naming is controlled by downloading.scrape. I'll try to muddle through hacking this to get the behavior that I need.

Mick.

[evilpenguin]

Originally Posted by mickp

From what I can see the file naming is controlled by downloading.scrape. I'll try to muddle through hacking this to get the behavior that I need.

Yep, your changes should be pretty straight forward, be sure and let me know if you have any trouble.

[popechild]

Originally Posted by evilpenguin

I rewrote that section give it a read again and see if it makes sense.

Awesome. It's very clear now.

[evilpenguin]

Originally Posted by popechild

Awesome. It's very clear now.

Excellent, i'm not very good at documenting so feel free to let me know when something is confusing. It all makes sense to me

[jaminben]

Originally Posted by megamojo

I just script it using dirmon2. It's working extrememly well for me and I don't have to touch a thing.

I've been playing with Dirmon2 and although I dont understand it I do have it working with mediascraper

Many Thanks for the tip

[Zorglub]

I love this tool ! Thank you !

I was wondering if it would be possible to add the possibility to generate the information in XML format (SageXmlInfo) ?

I believe that it is possible to fill up enough information in the xml file, so that we can then load it in the wiz.bin database with the webserver tools and make it appear on the Recorded TV list in the SageTV UI.

Here an exemple I construct with the information in your .info and was able to import easily with the webserver:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE sageShowInfo PUBLIC "-//NIELM//DTD SAGESHOWINFO XML 1.2//EN" "http://sageplugins.sourceforge.net/nielm_modules/sagexmlinfo/sageshowinfo_1_2.dtd">
<sageShowInfo version="1.2">
     <showList>  
        <show>
           <title>Battlestar Galactica</title>
           <episode>A Disquiet Follows My Soul</episode> 
           <duration>3600</duration>
           <category>Science-Fiction</category> 
	   <description>Tigh and Number Six take their first glimpse at their unborn child, Tyrol offers Adama Cylon technology in return for his fellow Cylons properly reintegrated into the fleet, and an attempted coup leads to a costly struggle to maintain civility.</description>
           <peopleList>
                <person role="Actor">Edward James Olmos</person>
                <person role="Actor">Mary McDonnell</person>
                <person role="Actor">Katee Sackhoff</person>
                <person role="Actor">Jamie Bamber</person>
                <person role="Actor">James Callis</person>
 		<person role="Actor">Tricia Helfer</person>
                <person role="Actor">Michael Hogan</person>
                <person role="Actor">Kandyse McClure</person>
                <person role="Actor">Michael Trucco</person>
                <person role="Actor">Grace Park</person>
                <person role="Director">Ronald D. Moore</person>
                <person role="Writer">Ronald D. Moore</person>
           </peopleList>
	   <year>2003</year>
	   <language>English</language>
           <originalAirDate>2009-01-23T23:00:00.00Z</originalAirDate>
           <airing duration="3600" startTime="2009-01-23T23:00:00.00Z">     
		<isHDTV></isHDTV>
                <stereo></stereo>
                <mediafile type="TV">
                    <segmentList>
                        <segment filePath="\\naslite\disk-0\MEDIA\Videos\TV\Battlestar Galactica\Season 4\Battlestar Galactica (2003) 4x12 - A Disquiet Follows My Soul.mkv"/>
                    </segmentList>
                </mediafile>
           </airing>
        </show>
     </showList>
</sageShowInfo>

That would permit me (and a whole bunch of other Sage users ) to keep all of our TV show, recorded or acquired via other means, in the same place.

This would be a BIG improvement on the WAF !!!

[evilpenguin]

Originally Posted by Zorglub

I was wondering if it would be possible to add the possibility to generate the information in XML format (SageXmlInfo) ?

I like the sound of that, can you go into more detail on you would use these XML files to import the shows into SageTV? I'm not familiar with it.

[QueOnda]

Originally Posted by evilpenguin

I like the sound of that, can you go into more detail on you would use these XML files to import the shows into SageTV? I'm not familiar with it.

I LOVE the sound of that!!!

This would allow you if you have dvd tv shows in the video section, now it would be in the recorded tv section???