SageTV Community

SageTV Community (http://forums.sagetv.com/forums/index.php)
-   Batch Metadata Tools (http://forums.sagetv.com/forums/forumdisplay.php?f=46)
-   -   Utility: Batch Metadata Importer 3.x (http://forums.sagetv.com/forums/showthread.php?t=34301)

stuckless 07-19-2008 08:55 PM

Utility: Batch Metadata Importer 3.x
 
4 Attachment(s)
Documentation | Download | Fanart FAQ | Releases Thread | sflamm's Install Guide | Configure Logging

http://forums.sagetv.com/forums/atta...4&d=1250646956

Overview
Batch Metadata Tools (BMT) is a tool (command line) and a Sage plugin that scans a media collection and attempts to automatically search/download metadata for each item.
  • Multi-Platform (Windows/Linux/Mac)
  • Multple Metadata Sources (imdb, tvdb, themoviedb, dvd profiler, mymovies)
  • Full TV Support for Sage Recordings or Archived TV shows (tvdb)
  • Downloads Posters and Backdrops (support for central fanart using phoenix apis)
  • Scrubs filenames to better search hits (works on Sage Recordings, TV Files (sXX eXX), and Movies)
  • Optionally Prompts for entries that cannot be located (and stores selected result for future reference) (command line)
  • Many configuration options (use --showProperties)
  • Includes STVi for Default STV and SageMC for doing on demand fanart and metadata for Movies, TV, and EPG items.
  • WebUI for configuration (since 3.0)

NOTE: Plugin mode runs metadata searches automatically when new files are added to the Sage Library. If metadata cannot be found the the file is skipped.

BMT Supports the Following Metadata Locations
  • IMDB (imdb, niel's imdb, xbmc's imdb using imdb.xml scraper file)
  • TheMovieDB.org
  • DVD Profiler (using DVD Profiler's Xml and images)
  • Xbmc Sources Using Xbmc's Scraper Language (imdb.xml)
  • thetvdb
  • MyMovies (Uinsg MyMovies collection xml)

How should my files be named, to ensure best results?
Movie Files
Finding Nemo.avi
Finding Nemo (2005).avi
Finding Nemo cd1.avi
Finding Nemo cd2.avi

DVDs
Finding Nemo/VIDEO_TS/
Finding Nemo/*.vob
Finding Nemo (2005)/*.vob

Sage Recordings (Movie)
FindingNemo-000000-0.ts

Sage Recordings (TV)
Futurama-MothersDay-0000000-0.ts

Archived TV Shows
Futurama - S01E02 - Mother's Day.avi
Futurama - S01E02.avi
Lie To Me - S01E01.avi

Ripped TV DVDs
Lie To Me S01D01/*.vob


Avoid using dots in spaces, ie, Lie.To.Me-S01E01.avi.
Avoid using too many tokens in your title, like, DVDRip, etc.

BMT does attempt to clean up the filenames, but, the cleaner they are to to start with, the better your chances of getting a good hit.

Known Issues
In Windows, after a server restart, BMT web will fail to show the UI. Instead it will show only the directory listing or generate an error. This post explains the problem and the solution.



Hint's Tips from Users
How do I force Sage to reindex my entire collection?
Add this to your Sage.properties to force it to completely rebuild your metadata. Useful to clean up Genres.
Quote:

Originally Posted by bcjenkins
force_full_content_reindex=true

Note: As of SageTV 6.5.18, you can now tell sage to do a Full Reindex from withint the STV
Note: As fo bmt 3.0 you can force sage to do a full reindex from the web ui

Windows Batch File
razrsharpe provided this windows .bat file that some windows users may find useful. You still need to configure the bat file with your default options, but once you do, you can then drag and drop a file/folder onto the bat file and it will invoke the scan/update.

How to update the timestamp of a Media File so sagetv will re-read the .properties (Windows)
babgvant posted this tip
sflamm posted this tip
And since 3.0-beta8, you can "touch" a file using the --touch commandline option.

Code:

# java -jar MetadataTool.jar --touch YOUR_FILE|YOUR_DIR

Do not make your files read-only
Quote:

Originally Posted by ChuckSchick
Before running the script, make sure none of your movies are set to "Read Only".
Otherwise, the file dates won't update and Sage won't import the .properties info. Somewhere in the history of my library I was setting the files to Read Only in an effort to protect them. About half of my collection still had this property.

Submitting Bugs/Requesting Help
Because of the complexities of how BMT can be used, I need to know some information about your environment before I can help your. Use the following a starting point for information.
Code:

What bmt version 2.x (stable) or 3.x (beta)?
What SageTV version?
SageMC or default STV?
SageMC version?
Java version?
Phoenix version?

SageTV setup:
Is central FanArt folder enabled?
Is central FanArt folder local or a network share?
Do you have SageTV set up to do automatic scans?
If yes, how often?
Do the scans import new videos?

BMT setup:
Command Line Tool:
Have you successfully used the command line tool?

On-demand Tool:
Have you imported the on-demand plugin (stvi of xml)?
Do you have the on-demand plug-in enabled?

Automatic tool:
Do you have these 2 properties set in sage.properties?

       
Code:

       
mediafile_metadata_parser_plugins=org.jdna.sage.MetadataUpdaterPlugin
custom_metadata_properties=UserRating;EpisodeNumber;EpisodeTitle;MediaTitle;MediaType;OriginalAirDate;SeasonNumber;MediaProviderDataID;DiscNumber


Please give a description of the steps to reproduce your issue:


PGPfan 07-20-2008 02:06 AM

Thanks Sean,

I'm definately going to give this a spin!

-PGPfan

jusjoken 07-20-2008 08:09 AM

Excellent tool.

So far my results look good and it processed 36 movies very quickly.

1 file I needed to manually provide a title for "Serenity (Aka.Firefly).avi", I provided Firefly and that worked.

1 movie it got wrong "Terminator 1.avi", it found Terminator II. I then ran the command for --force with the filename and then manually selected "The Terminator" and that worked, except that it left the jpg for Terminator II that it had previously downloaded. I then had to create a metadataupdater.properties file and adjust the overwriteThumbnail value to true. I would have assumed if I was using --force then the thumbnail would have been overwritten by default, but others might think otherwise as potentially I might have had my own thumbnail collection that I didn't want overwritten. Either way it worked well with just a little effort.

I have not uploaded into sage yet, so I will let you know the results then.

bcjenkins 07-20-2008 08:19 AM

From Java 1.5

Code:

sagetv@sagetv-server:~/scripts/imdb_importer$ PATH=/opt/jre1.5.0_15/bin:$PATH
sagetv@sagetv-server:~/scripts/imdb_importer$ export PATH
sagetv@sagetv-server:~/scripts/imdb_importer$ java -jar MetadataTool.jar
Exception in thread "main" java.lang.UnsupportedClassVersionError: Bad version number in .class file
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(Unknown Source)
        at java.security.SecureClassLoader.defineClass(Unknown Source)
        at java.net.URLClassLoader.defineClass(Unknown Source)
        at java.net.URLClassLoader.access$100(Unknown Source)
        at java.net.URLClassLoader$1.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at java.lang.ClassLoader.loadClassInternal(Unknown Source)

Will use 1.6, but FYI

B

bcjenkins 07-20-2008 08:36 AM

Is there support for VIDEO_TS?

/var/media/videos/$DVD_NAME/VIDEO_TS/(vob files)

TIA

B

stuckless 07-20-2008 08:41 AM

Quote:

From Java 1.5
I mentioned the release info that I suspected that java 5 would not work, simple because I use java 6... But I will attempt to recompile using Java 5 next time... I don't think i'm using any Java 6 extensions yet :)

stuckless 07-20-2008 08:45 AM

Quote:

1 movie it got wrong "Terminator 1.avi", it found Terminator II. I then ran the command for --force with the filename and then manually selected "The Terminator" and that worked, except that it left the jpg for Terminator II that it had previously downloaded. I then had to create a metadataupdater.properties file and adjust the overwriteThumbnail value to true. I would have assumed if I was using --force then the thumbnail would have been overwritten by default, but others might think otherwise as potentially I might have had my own thumbnail collection that I didn't want overwritten. Either way it worked well with just a little effort.
I wrestled with the idea of overwriting the thumbnails or not. In the end, I opted to NOT overwrite ever, but that functionality can be overridded using a custom metadataupdater.properties and setting the
Code:

org.jdna.media.metadata.impl.sage.SageVideoMetaData.overwriteThumbnail=true
I could potentialy use another command line arg to set that value as well.

stuckless 07-20-2008 08:48 AM

Quote:

Is there support for VIDEO_TS?
Not yet... it is on my todo list... and it's not hard to implement (i think). I actually check for VIDEO_TS directories and currently skip them :(

FYI.... there tool does create a log file in your user's home directory called metadataupdater.log... you'll see messages in there about "Skipping DVD Folders for now" :)

bcjenkins 07-20-2008 08:56 AM

It isn't giving me that error, I am getting:

Code:

73 ERROR [main] org.jdna.metadataupdater.MetadataUpdater - Unknown Media Resourse for: file:/var/media/videos/MovieName/;org.jdna.media.impl.MediaFolder
It doesn't seem to be finding VIDEO_TS

B

stuckless 07-20-2008 09:02 AM

Quote:

It doesn't seem to be finding VIDEO_TS
I'll look into that for you. I'll just go ahead and implement dvd folder support :)

Update: I can see the problem now... I'm only checking for DVD Folders when --recurse is used. I'll try to add in dvd folder support sometime today (later :) )

jusjoken 07-20-2008 09:08 AM

My import into Sage worked well. The only issue I see is that the Categories (Genre) seem to import as a complete string including the ; seperators. I would have thought that the first one would be the Category and the second would be the SubCategory and the rest would be ignored. This is likely an issue with the import in Sage and not your tool unless the formating in the .properties is not what Sage is expecting.

The other thing I noticed is that I thought there was a propertiy for a Rating and for an Extended Rating. A Rating might be PG where and extended rating might be "Rated PG for ....". Not sure IMDB offers both nor if the properties file support both, I just remember getting a simpler Rating (only the PG) when using the imports in SageMC (now using the Default STV).

Otherwise, great so far.

bcjenkins 07-20-2008 10:26 AM

Quote:

Originally Posted by stuckless (Post 293938)
I'll look into that for you. I'll just go ahead and implement dvd folder support :)

Update: I can see the problem now... I'm only checking for DVD Folders when --recurse is used. I'll try to add in dvd folder support sometime today (later :) )

Thanks! All of my videos are in DVD format. Oddly the properties file for DVD folders needs to go at the same level as the $DVD_NAME directory. Instead of in the VIDEO_TS folder like the folder.jpg.

B

JREkiwi 07-20-2008 12:59 PM

Quote:

Originally Posted by bcjenkins (Post 293944)
Oddly the properties file for DVD folders needs to go at the same level as the $DVD_NAME directory. Instead of in the VIDEO_TS folder like the folder.jpg.

You can have a .jpg in the same level as the $DVD_NAME directory with the .properties file. It just needs to be called $DVD_NAME.jpg

John

bcjenkins 07-20-2008 01:16 PM

I would prefer a folder.properties at the same level as the folder.jpg. It makes for a cleaner videos directory.

:)

B

stuckless 07-20-2008 03:44 PM

So if I understand this, for dvd folders, the property name is the same name as the dvd folder, and the thumbnail can either be the same as video folder name or VIDEO_TS/folder.jpg?

eg, if the DVD Folder is called Terminator,
Terminator/Terminator.properties
Terminator/Terminator.jpg
or
Terminator/Terminator.properties
Terminator/VIDEO_TS/folder.jpg

bcjenkins 07-20-2008 03:49 PM

./Terminator/VIDEO_TS/folder.jpg
./Terminator.properties

B

JREkiwi 07-20-2008 04:11 PM

With 6.4 the requirement for a VIDEO_TS was removed for DVD's and support for .properties file for DVD's was introduced and defined as being located in the parent folder. So for DVD's

In the parent folder DVDs
DVDs\<DVDName>\dvd files
DVDs\<DVDName>.properties
DVDs\<DVDName>.jpg
DVDs\<DVDName>\folder.jpg does not work
and
DVDs\<DVDName>\VIDEO_TS\dvd files
DVDs\<DVDName>.properties
DVDs\<DVDName>.jpg
or
DVDs\<DVDName>\VIDEO_TS\folder.jpg

John

Brent 07-20-2008 04:21 PM

Quote:

Originally Posted by JREkiwi (Post 293995)
With 6.4 the requirement for a VIDEO_TS was removed for DVD's and support for .properties file for DVD's was introduced and defined as being located in the parent folder. So for DVD's

In the parent folder DVDs
DVDs\<DVDName>\dvd files
DVDs\<DVDName>.properties
DVDs\<DVDName>.jpg
DVDs\<DVDName>\folder.jpg does not work
and
DVDs\<DVDName>\VIDEO_TS\dvd files
DVDs\<DVDName>.properties
DVDs\<DVDName>.jpg
or
DVDs\<DVDName>\VIDEO_TS\folder.jpg

John

True, but since SageMC (with DVDProfiler) hasn't caught up with that requirement many still have the "video_ts" folder.

JREkiwi 07-20-2008 04:28 PM

Quote:

Originally Posted by Brent (Post 294000)
True, but since SageMC (with DVDProfiler) hasn't caught up with that requirement many still have the "video_ts" folder.

I suppose the point I was really trying to make is that folder.jpg only works if you are using VIDEO_TS folders. <DVDNAME>.jpg works in both configurations, and there is only 1 valid location for the .properties file

John

Brent 07-20-2008 04:31 PM

Quote:

Originally Posted by JREkiwi (Post 294004)
I suppose the point I was really trying to make is that folder.jpg only works if you are using VIDEO_TS folders. <DVDNAME>.jpg works in both configurations, and there is only 1 valid location for the .properties file

John

Okay, understood.


All times are GMT -6. The time now is 11:08 AM.

Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2019, vBulletin Solutions, Inc.
Copyright 2003-2005 SageTV, LLC. All rights reserved.