Utility: mediaScraper - metadata made easy!

[GollyJer]

Is /AddToSageDB working for anyone? It was working great but doesn't anymore with the 6.5.17 beta.

[freedml]

I can get beta9 to make .properties files, but no XML or import into SageTV.

I can't get beta13 to create any files. Not even a log file.

Any hints?

beta13 doesn't create a log file, so I copy and paste it here:

Welcome to mediaEngine v1.0beta
Staring Proccesing at 17:11:39 6-14-2009

  + Executable   : C:\Documents and Settings\Administrator\Desktop\mediascraper
13\mediaEngine.exe
  + EXE path     : C:\Documents and Settings\Administrator\Desktop\mediascraper
13
  + Parsing switches
    - optionsString: "/genPropertyFile" "/downloadAllFanArt" "/baseFolder" "TV"
"/defaultProfile" "inputFile" "/findFileRegEx" "avi|mpg|mkv|mp4|mpeg|VIDEO_TS|ts
|ogm|divx|mpts|dvr-ms" "/throttle" "10" "/maxImagesToDownload" "3" "/genxmlfile"
 "/username" "Sage" "/password" "Sage" "/Addtosagedb" "/mediaScraper" "fixBug" "
G:\TwoandaHalfMen-1309749-0.mpg" "G:\TwoandaHalfMen-1311564-0.mpg"
        - Replacing quote0: /genPropertyFile
        - Replacing quote1: /downloadAllFanArt
        - Replacing quote2: /baseFolder
        - Replacing quote3: TV
        - Replacing quote4: /defaultProfile
        - Replacing quote5: inputFile
        - Replacing quote6: /findFileRegEx
        - Replacing quote7: avi|mpg|mkv|mp4|mpeg|VIDEO_TS|ts|ogm|divx|mpts|dvr-m
s
        - Replacing quote8: /throttle
        - Replacing quote9: 10
        - Replacing quote10: /maxImagesToDownload
        - Replacing quote11: 3
        - Replacing quote12: /genxmlfile
        - Replacing quote13: /username
        - Replacing quote14: Sage
        - Replacing quote15: /password
        - Replacing quote16: Sage
        - Replacing quote17: /Addtosagedb
        - Replacing quote18: /mediaScraper
        - Replacing quote19: fixBug
        - Replacing quote20: G:\TwoandaHalfMen-1309749-0.mpg
        - Replacing quote21: G:\TwoandaHalfMen-1311564-0.mpg
    - Adding to to options Hash
      + Key: genPropertyFile
    - Adding to to options Hash
      + Key: downloadAllFanArt
    - Adding to to options Hash
      + Key: baseFolder
      + Value: TV
    - Adding to to options Hash
      + Key: defaultProfile
      + Value: inputFile
    - Adding to to options Hash
      + Key: findFileRegEx
      + Value: avi|mpg|mkv|mp4|mpeg|VIDEO_TS|ts|ogm|divx|mpts|dvr-ms
    - Adding to to options Hash
      + Key: throttle
      + Value: 10
    - Adding to to options Hash
      + Key: maxImagesToDownload
      + Value: 3
    - Adding to to options Hash
      + Key: genxmlfile
    - Adding to to options Hash
      + Key: username
      + Value: Sage
    - Adding to to options Hash
      + Key: password
      + Value: Sage
    - Adding to to options Hash
      + Key: Addtosagedb
    - Adding to to options Hash
      + Key: mediaScraper
      + Value: fixBug
    - Adding Inputfile: G:\TwoandaHalfMen-1309749-0.mpg
    - Adding Inputfile: G:\TwoandaHalfMen-1311564-0.mpg
    - Scanning Directory: C:\Documents and Settings\Administrator\Desktop\medias
craper 13\mediaEngineProfiles (profile|func|snip|output|scrape)
    - Scanning Directory: C:\Documents and Settings\Administrator\Desktop\medias
craper 13\mediaEngineProfiles\encodingProfiles (profile|func|snip|outp...
    - Scanning Directory: C:\Documents and Settings\Administrator\Desktop\medias
craper 13\mediaEngineProfiles\scrapingProfiles (profile|func|snip|outp...
  + Seraching for avaialble binaries
    - Scanning Directory: C:\Documents and Settings\Administrator\Desktop\medias
craper 13\mediaEngineProfiles (exe)
    - Scanning Directory: C:\Documents and Settings\Administrator\Desktop\medias
craper 13\mediaEngineProfiles\encodingProfiles (exe)
    - Scanning Directory: C:\Documents and Settings\Administrator\Desktop\medias
craper 13\mediaEngineProfiles\scrapingProfiles (exe)
    - Scanning Directory: C:\Documents and Settings\Administrator\Desktop\medias
craper 13\mediaEngineBins (exe)
    - Scanning Directory: C:\Documents and Settings\Administrator\Desktop\medias
craper 13\mediaEngineBins\Curl (exe)
    - Scanning Directory: C:\Documents and Settings\Administrator\Desktop\medias
craper 13\mediaEngineBins\ffmpeg (exe)
      + mtee.exe        : C:\Documents and Settings\Administrator\Desktop\medias
craper 13\mediaEngineBins\mtee.exe
      + touch.exe       : C:\Documents and Settings\Administrator\Desktop\medias
craper 13\mediaEngineBins\touch.exe
      + wget.exe        : C:\Documents and Settings\Administrator\Desktop\medias
craper 13\mediaEngineBins\wget.exe
      + curl.exe        : C:\Documents and Settings\Administrator\Desktop\medias
craper 13\mediaEngineBins\Curl\curl.exe
      + ffmpeg.exe      : C:\Documents and Settings\Administrator\Desktop\medias
craper 13\mediaEngineBins\ffmpeg\ffmpeg.exe
  + Seraching for input files (avi|mpg|mkv|mp4|mpeg|VIDEO_TS|ts|ogm|divx|mpts|dv
r-ms)
    - Found File: (G:\TwoandaHalfMen-1309749-0.mpg)
    - Found File: (G:\TwoandaHalfMen-1311564-0.mpg)
------------ Processing ----------------
  + Adding per run options: (/inputFile "G:\TwoandaHalfMen-1309749-0.mpg" /profi
le inputFile)(/inputFile "G:\TwoandaHalfMen-1309749-0.mpg" /profile in...
    + Parsing switches
      - optionsString: /inputFile "G:\TwoandaHalfMen-1309749-0.mpg" /profile inp
utFile
      - Adding to to options Hash
        + Key: inputFile
        + Value: G:\TwoandaHalfMen-1309749-0.mpg
      - Adding to to options Hash
        + Key: profile
        + Value: inputFile
  + Processing: inputFile = TwoandaHalfMen-1309749-0
  + Looking for profile: inputFile
    - Found "inputfile"
  + Getting Video Info: (G:\TwoandaHalfMen-1309749-0.mpg)(6318)
    - Input Line:          Input #0, mpeg, from 'G:\TwoandaHalfMen-1309749-0.mpg
':
    - Duration Line:       Duration: 00:00:04.04, start: 0.766311, bitrate: 1281
0 kb/s
    - Video Info Line:     Stream #0.0[0x1e0], 1/90000: Video: mpeg2video, yuv42
0p, 1920x1080, 1001/30000, AR: 16:9, interlaced, 19392 kb/s, 29.97 fps...
    - Audio Info Line:     Stream #0.1[0x82], 1/90000: Audio: ac3, 48000 Hz, ste
reo, s16, 192 kb/s
    - Audio Info Line:     Stream #0.2[0x81], 1/90000: Audio: ac3, 48000 Hz, 5.1
, s16, 384 kb/s

######## Starting Processing

    - Remaining Commands (16): (/use %%inputFile%%)
      + Command : /use G:\TwoandaHalfMen-1309749-0.mpg
      + Target  : ([0-9]*)\-[0-9]\.[a-zA-Z0-9]+$
      + Success
        - [1309749-0.mpg]
      + Success (1)
        - $$airingID$$ = (1309749)

  + Found Password: ******
    - Scrubbing Password:

    - Rewriting Logfiles: TwoandaHalfMen-1309749-0.inputFile.(2).log,

    - Remaining Commands (15): (/branch)
      + Command : /branch

    - Remaining Commands (5): (/use %%inputFile%%)
      + Command : /use G:\TwoandaHalfMen-1309749-0.mpg
      + Target  : ([0-9]*)\-[0-9]\.[a-zA-Z0-9]+$
      + Success
        - [1309749-0.mpg]
      + Success (1)
        - $$airingID$$ = (1309749)

    - Remaining Commands (4): (/insertFunction)
      + Command : /insertFunction

    - Remaining Commands (5): (/use "C:\Program Files\******TV\******TV\webserve
r\realms.properties" /readFile)
      + Command : /use "C:\Program Files\******TV\******TV\webserver\realms.prop
erties" /readFile
      + Target  : ******TV Web Interface=/sage,([^:]+):([^ ]+)\n
      + Success
        - [******TV Web Interface=/sage,******:******
]
      + Success (1)
        - $$userName$$ = (******)
        - $$password$$ = (******)

    - Remaining Commands (4): (/use "C:\Program Files\******TV\******TV\******.p
roperties" /readFile)
      + Command : /use "C:\Program Files\******TV\******TV\******.properties" /r
eadFile
      + Target  : nielm/webserver/port=([0-9]+)\n
      + Success
        - [nielm/webserver/port=8080
]
      + Success (1)
        - $$port$$ = (8080)

    - Remaining Commands (3): (/get sageAiringInfoXML /user "%%userName%%" /pass
word "%%password%%" /realm "******TV Web Interface")
      + Command : /get sageAiringInfoXML /user "******" /password "******" /real
m "******TV Web Interface"
        - Adding Credientials: ******@localhost:8080(******TV Web Interface)
        - Adding Credientials: ******@localhost:8080(******TV Web Interface)
      + Requesting WebAddress: http://localhost:8080/sage/DetailedInfo?AiringId=
1309749&xml=yes
        - Attempt #1        - Attempt #2        - Attempt #3        - Attempt #4
        - Attempt #5 Failure: 500 Internal server error
          + Sleeping for 10 seconds as to not throttle website

    - Remaining Commands (2): (/use sageAiringInfoXML /flatten)
      + Command : /use sageAiringInfoXML /flatten
      + Target  : <title>([^<]+)</title>.*<originalAirDate>([0-9\-]+)T[^<]+</ori
ginalAirDate>
      ! Failure

    - Remaining Commands (1): (/use sageAiringInfoXML /flatten)
      + Command : /use sageAiringInfoXML /flatten
      + Target  : <title>([^<]+)</title>.*<originalAirDate>([0-9\-]+)T[^<]+</ori
ginalAirDate>
      ! Failure

    - Remaining Commands (0): (/branch)
      + Command : /branch
      + Command Empty, Skipping: ""

  + Deleting temp files (G:\TwoandaHalfMen-1309749-0.workFolder)
    - Deleting File: G:\TwoandaHalfMen-1309749-0.workFolder
      + Deleting Folder: rmdir /S /Q "G:\TwoandaHalfMen-1309749-0.workFolder"
------------ Processing ----------------
  + Adding per run options: (/inputFile "G:\TwoandaHalfMen-1311564-0.mpg" /profi
le inputFile)(/inputFile "G:\TwoandaHalfMen-1309749-0.mpg" /profile in...
    + Parsing switches
      - optionsString: /inputFile "G:\TwoandaHalfMen-1311564-0.mpg" /profile inp
utFile
      - Adding to to options Hash
        + Key: inputFile
        + Value: G:\TwoandaHalfMen-1311564-0.mpg
      - Adding to to options Hash
        + Key: profile
        + Value: inputFile
  + Processing: inputFile = TwoandaHalfMen-1311564-0
  + Looking for profile: inputFile
    - Found "inputfile"
  + Getting Video Info: (G:\TwoandaHalfMen-1311564-0.mpg)(3332070)
    - Input Line:          Input #0, mpeg, from 'G:\TwoandaHalfMen-1311564-0.mpg
':
    - Duration Line:       Duration: 00:30:51.29, start: 0.713844, bitrate: 1474
4 kb/s
    - Video Info Line:     Stream #0.0[0x1e0], 1/90000: Video: mpeg2video, yuv42
0p, 1920x1080, 1001/30000, AR: 16:9, interlaced, 19392 kb/s, 29.97 fps...
    - Audio Info Line:     Stream #0.1[0x81], 1/90000: Audio: ac3, 48000 Hz, 5.1
, s16, 384 kb/s
    - Audio Info Line:     Stream #0.2[0x82], 1/90000: Audio: ac3, 48000 Hz, ste
reo, s16, 192 kb/s

######## Starting Processing

    - Remaining Commands (16): (/use %%inputFile%%)
      + Command : /use G:\TwoandaHalfMen-1311564-0.mpg
      + Target  : ([0-9]*)\-[0-9]\.[a-zA-Z0-9]+$
      + Success
        - [1311564-0.mpg]
      + Success (1)
        - $$airingID$$ = (1311564)

    - Remaining Commands (15): (/branch)
      + Command : /branch

    - Remaining Commands (5): (/use %%inputFile%%)
      + Command : /use G:\TwoandaHalfMen-1311564-0.mpg
      + Target  : ([0-9]*)\-[0-9]\.[a-zA-Z0-9]+$
      + Success
        - [1311564-0.mpg]
      + Success (1)
        - $$airingID$$ = (1311564)

    - Remaining Commands (4): (/insertFunction)
      + Command : /insertFunction

    - Remaining Commands (5): (/use "C:\Program Files\******TV\******TV\webserve
r\realms.properties" /readFile)
      + Command : /use "C:\Program Files\******TV\******TV\webserver\realms.prop
erties" /readFile
      + Target  : ******TV Web Interface=/sage,([^:]+):([^ ]+)\n
      + Success
        - [******TV Web Interface=/sage,******:******
]
      + Success (1)
        - $$userName$$ = (******)
        - $$password$$ = (******)

    - Remaining Commands (4): (/use "C:\Program Files\******TV\******TV\******.p
roperties" /readFile)
      + Command : /use "C:\Program Files\******TV\******TV\******.properties" /r
eadFile
      + Target  : nielm/webserver/port=([0-9]+)\n
      + Success
        - [nielm/webserver/port=8080
]
      + Success (1)
        - $$port$$ = (8080)

    - Remaining Commands (3): (/get sageAiringInfoXML /user "%%userName%%" /pass
word "%%password%%" /realm "******TV Web Interface")
      + Command : /get sageAiringInfoXML /user "******" /password "******" /real
m "******TV Web Interface"
        - Adding Credientials: ******@localhost:8080(******TV Web Interface)
        - Adding Credientials: ******@localhost:8080(******TV Web Interface)
      + Requesting WebAddress: http://localhost:8080/sage/DetailedInfo?AiringId=
1311564&xml=yes
        - Attempt #1

[rmac321]

EP,

Sorry for the nag. Any chance you will be adding the MediaTitle field into the .properties output? I love MediaScraper and have used it to organize all my media files and FanArt, so I'd prefer to stay with it.

I know you mentioned you use BMT now, so I would certainly understand if you don't intend to do any more updates. OTOH, if you do intend to add this feature, I'll wait patiently for the next beta.

Thanks!

[rmac321]

EP, did I offend?

[evilpenguin]

Not at all, I just haven't had a chance to verify if the mediaTitle field is in mediaScrape or not, but i'm fairly certain it should be.

[PiX64]

Hey EP,

AMAZING tool, it has done wonders for me. Quick question for you. I searched the thread already and was unable to find a clear answer to my issue so here it goes.

i have DVD box sets and i can't seem to get the properties files to generate:

example
--------
Simpsons -> Simpsons Season 1 -> Simpsons Season 1 - 1, Simpsons Season 1 - 2, etc.

what naming convetion should i use to get your tool to find the fanart, and generate teh correct properties files?

[PiX64]

*Bump*

Anyone?

[evilpenguin]

Mediascraper doesn't' really handle TV show DVD's. You'd need to rip them into their individual episodes and name them accordingly to make it work right.

[PiX64]

Thanks EP. Any suggestions on a good tool to use for ripping TVSeries DVD into their individual episodes...

[rmac321]

Originally Posted by PiX64

Thanks EP. Any suggestions on a good tool to use for ripping TVSeries DVD into their individual episodes...

Dude, that would be EP's other Super Cool Tool: MediaShrink. It doesn't do the actual ripping, but it sounds like you already have them ripped. Mediashrink will take your ripped Video_TS folders and create individual videos for each episode.

Step 1. Drag and Drop a Video_TS folder on to MediaShrink. come back in a few hours.
Step 2. create a folder.override text file in the new _compress directory or drag the files into a folder for that series that already has a folder.override file.
Step3. rename the episode files MediaShrink created by placing a 3 digit number at the beginning of the file name. First digit is season number, 2nd and 3rd digits are the episode number. ie 105. Don't bother to rename with episode title.
Step 4. Drag the _compress folder onto MediaScraper and depending on your settings the files will get renamed and moved to their proper subfolders.

[PiX64]

friggin sweeeeeet!

thanks ill give it a whirl tomorrow.

[ohpleaseno]

is anybody else having issues with apostrophes in movie titles? I can't get the file to move to my directory if there is an apostrophe.

[ohpleaseno]

Let me rephrase this so that it actually says what my question is (why oh why must I ask things in such a non eloquent way?)

I have movies with ' in the title (Dude Where's My Car) that I run through Media Scraper. I've set up media scraper to move the file to my movies folder and name it movie name (year) .extension. Everything works flawlessly except for the movies with an ' in the title. Is there a workaround to get them to rename and move?

[evilpenguin]

Add /guessOK to the mediaScaper.properties file, then it'll work. I should know because I just used mediaScraper on "Dude Wheres My Car.mkv" last night

[ohpleaseno]

didn't work here's my defaults

/genPropertyFile /downloadAllFanArt /baseFolder "Videos" /defaultProfile inputFile  /findFileRegEx "avi|mpg|mkv|mp4|mpeg|VIDEO_TS|ts|ogm|divx|mpts|dvr-ms" /updateInfo  /organizeBaseFolder "D:\shares\Videos" /downloadFanArt /basefolder "D:\shares\Videos" /throttle 3 /maxImagesToDownload 3  /centralFanartpPath "D:\shares\Videos\Fan Art" /organizeFiles /tvSE /guessOK

[ohpleaseno]

Am I better off taking the apostrophe out of the filename and only using it in the folder.override?

[evilpenguin]

Bah, I guess that's only in my dev version, no problem though. Just open up imdbMovies.scrape and modify this line in the first profile...

# Scraper: imdbMovie
#
#     INPUT: %%showTitle%% - Title of movie to search
#

    Profile           =imdbMovie
    Encode CLI #1     =?>!centralFanartPath<:>getCentralFanArtPath<?
    Encoder #1        =/insertFunction
    Encode CLI #1     =($$showTitle$$.*)
    Encoder #1        =/use "%%showTitle_STRIPSPACES_DOTTOSPACE%%"
    Encode CLI #1     =/isMovie
    Encoder #1        =/setOptions
    Encode CLI #1     =?>showTitle<:>imdbTitleSearch<?
    Encoder #1        =/insertFunction
    Encode CLI #2     =?>!imdbMovieID<:>imdbGetFirstPopular<?
    Encoder #2        =/insertFunction
    Encode CLI #1     =?>imdbMovieID<:>imdbGetMovieInfo<?
    Encoder #1        =/insertFunction
    Encode CLI #1     =?>imdbMovieID<:>imdbTheMovieDBSearch<?
    Encoder #1        =/insertFunction
    Encode CLI #1     =?>imdbMovieID&&organizeFiles&&(>guessMatch||guessOK<)&&!videoTS&&inputFile<:>organizeFiles<?
    Encoder #1        =/insertFunction
    Encode CLI #1     =?>downloadFanArt<:>imdbDownloadImages<?
    Encoder #1        =/insertFunction
    Encode CLI #1     =?>imdbMovieID&&genInfoFile&&inputFile<:>writeMovieInfoFile<?
    Encoder #2        =/insertFunction
    Encode CLI #1     =?>imdbMovieID&&genMyFile&&inputFile<:>writeMovieMyFile<?
    Encoder #2        =/insertFunction
    Encode CLI #1     =?>imdbMovieID&&(>genPropertyFile||(>!genInfoFile&&!genMyFile&&!genPropertyFile<)<)<:>writeMoviePropertiesFile<?
    Encoder #2        =/insertFunction

[ohpleaseno]

A Bug's Live.avi

became

A Bug#x27s Life (1998).avi

[evilpenguin]

Yeah, I noticed that recently too. IMDB must have changed their page formating... again

[stuckless]

actually i noticed that imdb is referrring to several non alpha numeric characters by using html entities. I had a problem in bmt, with & being someing &x39; (or something like). I changed the code to html decode the titles after I parse them, which translates the html entities into regular characters.