SageTV Community  

Go Back   SageTV Community > SageTV Development and Customizations > SageTV v7 Customizations > Batch Metadata Tools

Notices

Batch Metadata Tools This forums is for discussing the user-created Batch Metadata Tools for SageTV.

Reply
 
Thread Tools Search this Thread Display Modes
  #1  
Old 07-19-2008, 08:55 PM
stuckless's Avatar
stuckless stuckless is offline
SageTVaholic
 
Join Date: Oct 2007
Location: London, Ontario, Canada
Posts: 9,554
Arrow Utility: Batch Metadata Importer 3.x

Documentation | Download | Fanart FAQ | Releases Thread | sflamm's Install Guide | Configure Logging



Overview
Batch Metadata Tools (BMT) is a tool (command line) and a Sage plugin that scans a media collection and attempts to automatically search/download metadata for each item.
  • Multi-Platform (Windows/Linux/Mac)
  • Multple Metadata Sources (imdb, tvdb, themoviedb, dvd profiler, mymovies)
  • Full TV Support for Sage Recordings or Archived TV shows (tvdb)
  • Downloads Posters and Backdrops (support for central fanart using phoenix apis)
  • Scrubs filenames to better search hits (works on Sage Recordings, TV Files (sXX eXX), and Movies)
  • Optionally Prompts for entries that cannot be located (and stores selected result for future reference) (command line)
  • Many configuration options (use --showProperties)
  • Includes STVi for Default STV and SageMC for doing on demand fanart and metadata for Movies, TV, and EPG items.
  • WebUI for configuration (since 3.0)

NOTE: Plugin mode runs metadata searches automatically when new files are added to the Sage Library. If metadata cannot be found the the file is skipped.

BMT Supports the Following Metadata Locations
  • IMDB (imdb, niel's imdb, xbmc's imdb using imdb.xml scraper file)
  • TheMovieDB.org
  • DVD Profiler (using DVD Profiler's Xml and images)
  • Xbmc Sources Using Xbmc's Scraper Language (imdb.xml)
  • thetvdb
  • MyMovies (Uinsg MyMovies collection xml)

How should my files be named, to ensure best results?
Movie Files
Finding Nemo.avi
Finding Nemo (2005).avi
Finding Nemo cd1.avi
Finding Nemo cd2.avi

DVDs
Finding Nemo/VIDEO_TS/
Finding Nemo/*.vob
Finding Nemo (2005)/*.vob

Sage Recordings (Movie)
FindingNemo-000000-0.ts

Sage Recordings (TV)
Futurama-MothersDay-0000000-0.ts

Archived TV Shows
Futurama - S01E02 - Mother's Day.avi
Futurama - S01E02.avi
Lie To Me - S01E01.avi

Ripped TV DVDs
Lie To Me S01D01/*.vob


Avoid using dots in spaces, ie, Lie.To.Me-S01E01.avi.
Avoid using too many tokens in your title, like, DVDRip, etc.

BMT does attempt to clean up the filenames, but, the cleaner they are to to start with, the better your chances of getting a good hit.

Known Issues
In Windows, after a server restart, BMT web will fail to show the UI. Instead it will show only the directory listing or generate an error. This post explains the problem and the solution.



Hint's Tips from Users
How do I force Sage to reindex my entire collection?
Add this to your Sage.properties to force it to completely rebuild your metadata. Useful to clean up Genres.
Quote:
Originally Posted by bcjenkins
force_full_content_reindex=true
Note: As of SageTV 6.5.18, you can now tell sage to do a Full Reindex from withint the STV
Note: As fo bmt 3.0 you can force sage to do a full reindex from the web ui

Windows Batch File
razrsharpe provided this windows .bat file that some windows users may find useful. You still need to configure the bat file with your default options, but once you do, you can then drag and drop a file/folder onto the bat file and it will invoke the scan/update.

How to update the timestamp of a Media File so sagetv will re-read the .properties (Windows)
babgvant posted this tip
sflamm posted this tip
And since 3.0-beta8, you can "touch" a file using the --touch commandline option.

Code:
# java -jar MetadataTool.jar --touch YOUR_FILE|YOUR_DIR

Do not make your files read-only
Quote:
Originally Posted by ChuckSchick
Before running the script, make sure none of your movies are set to "Read Only".
Otherwise, the file dates won't update and Sage won't import the .properties info. Somewhere in the history of my library I was setting the files to Read Only in an effort to protect them. About half of my collection still had this property.
Submitting Bugs/Requesting Help
Because of the complexities of how BMT can be used, I need to know some information about your environment before I can help your. Use the following a starting point for information.
Code:
What bmt version 2.x (stable) or 3.x (beta)?
What SageTV version?
SageMC or default STV?
SageMC version?
Java version?
Phoenix version?

SageTV setup:
Is central FanArt folder enabled?
Is central FanArt folder local or a network share?
Do you have SageTV set up to do automatic scans?
If yes, how often?
Do the scans import new videos?

BMT setup:
Command Line Tool:
Have you successfully used the command line tool?

On-demand Tool:
Have you imported the on-demand plugin (stvi of xml)?
Do you have the on-demand plug-in enabled?

Automatic tool:
Do you have these 2 properties set in sage.properties?
Code:
mediafile_metadata_parser_plugins=org.jdna.sage.MetadataUpdaterPlugin
custom_metadata_properties=UserRating;EpisodeNumber;EpisodeTitle;MediaTitle;MediaType;OriginalAirDate;SeasonNumber;MediaProviderDataID;DiscNumber
Please give a description of the steps to reproduce your issue:
Attached Images
File Type: png bmt-scanresults1.png (170.0 KB, 1491 views)
File Type: png bmt-metadatasearch.png (208.1 KB, 1188 views)
File Type: png bmt-viewbackground.png (326.5 KB, 1138 views)
File Type: jpg bmt-viewbackground-scaled.jpg (36.9 KB, 18020 views)

Last edited by stuckless; 05-02-2010 at 09:41 PM. Reason: Added version to the title
Reply With Quote
  #2  
Old 07-20-2008, 02:06 AM
PGPfan's Avatar
PGPfan PGPfan is offline
Sage Fanatic
 
Join Date: Apr 2003
Location: Oldtown, Idaho USA
Posts: 862
Thanks Sean,

I'm definately going to give this a spin!

-PGPfan
__________________
Sage Server: Gigabyte 690AMD m-ATX, Athlon II X4 620 Propus, 3.0 GB ram, (1) VistaView dual analog PCI-e tuner, (2) Avermedia Purity 3D MCE 250's, (1) HD-Homerun, 1.5 TB of hard drives in a Windows Home Server drive pool, Western Digital 300GB 'scratch' disk outside the pool, Gigabit LAN
Sage Clients: MSI DIVA m-ATX, 5.1 channel 100w/channel amplifier card, 2 GB ram, , (1) Hauppauge MVP, (1) SageTV HD-100 Media Storage: unRAID 3.6TB server
Reply With Quote
  #3  
Old 07-20-2008, 08:09 AM
jusjoken jusjoken is offline
Sage Icon
 
Join Date: Dec 2005
Location: Minneapolis MN
Posts: 2,177
Excellent tool.

So far my results look good and it processed 36 movies very quickly.

1 file I needed to manually provide a title for "Serenity (Aka.Firefly).avi", I provided Firefly and that worked.

1 movie it got wrong "Terminator 1.avi", it found Terminator II. I then ran the command for --force with the filename and then manually selected "The Terminator" and that worked, except that it left the jpg for Terminator II that it had previously downloaded. I then had to create a metadataupdater.properties file and adjust the overwriteThumbnail value to true. I would have assumed if I was using --force then the thumbnail would have been overwritten by default, but others might think otherwise as potentially I might have had my own thumbnail collection that I didn't want overwritten. Either way it worked well with just a little effort.

I have not uploaded into sage yet, so I will let you know the results then.
Reply With Quote
  #4  
Old 07-20-2008, 08:19 AM
bcjenkins bcjenkins is offline
SageTVaholic
 
Join Date: Jan 2006
Posts: 3,760
From Java 1.5

Code:
sagetv@sagetv-server:~/scripts/imdb_importer$ PATH=/opt/jre1.5.0_15/bin:$PATH
sagetv@sagetv-server:~/scripts/imdb_importer$ export PATH
sagetv@sagetv-server:~/scripts/imdb_importer$ java -jar MetadataTool.jar
Exception in thread "main" java.lang.UnsupportedClassVersionError: Bad version number in .class file
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(Unknown Source)
	at java.security.SecureClassLoader.defineClass(Unknown Source)
	at java.net.URLClassLoader.defineClass(Unknown Source)
	at java.net.URLClassLoader.access$100(Unknown Source)
	at java.net.URLClassLoader$1.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(Unknown Source)
	at java.lang.ClassLoader.loadClass(Unknown Source)
	at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
	at java.lang.ClassLoader.loadClass(Unknown Source)
	at java.lang.ClassLoader.loadClassInternal(Unknown Source)
Will use 1.6, but FYI

B
__________________
Running SageTV on unRAID via Docker
Tuning handled by HDHR3-6CC-3X2 using OpenDCT
Reply With Quote
  #5  
Old 07-20-2008, 08:36 AM
bcjenkins bcjenkins is offline
SageTVaholic
 
Join Date: Jan 2006
Posts: 3,760
Is there support for VIDEO_TS?

/var/media/videos/$DVD_NAME/VIDEO_TS/(vob files)

TIA

B
__________________
Running SageTV on unRAID via Docker
Tuning handled by HDHR3-6CC-3X2 using OpenDCT
Reply With Quote
  #6  
Old 07-20-2008, 08:41 AM
stuckless's Avatar
stuckless stuckless is offline
SageTVaholic
 
Join Date: Oct 2007
Location: London, Ontario, Canada
Posts: 9,554
Quote:
From Java 1.5
I mentioned the release info that I suspected that java 5 would not work, simple because I use java 6... But I will attempt to recompile using Java 5 next time... I don't think i'm using any Java 6 extensions yet
Reply With Quote
  #7  
Old 07-20-2008, 08:45 AM
stuckless's Avatar
stuckless stuckless is offline
SageTVaholic
 
Join Date: Oct 2007
Location: London, Ontario, Canada
Posts: 9,554
Quote:
1 movie it got wrong "Terminator 1.avi", it found Terminator II. I then ran the command for --force with the filename and then manually selected "The Terminator" and that worked, except that it left the jpg for Terminator II that it had previously downloaded. I then had to create a metadataupdater.properties file and adjust the overwriteThumbnail value to true. I would have assumed if I was using --force then the thumbnail would have been overwritten by default, but others might think otherwise as potentially I might have had my own thumbnail collection that I didn't want overwritten. Either way it worked well with just a little effort.
I wrestled with the idea of overwriting the thumbnails or not. In the end, I opted to NOT overwrite ever, but that functionality can be overridded using a custom metadataupdater.properties and setting the
Code:
org.jdna.media.metadata.impl.sage.SageVideoMetaData.overwriteThumbnail=true
I could potentialy use another command line arg to set that value as well.
Reply With Quote
  #8  
Old 07-20-2008, 08:48 AM
stuckless's Avatar
stuckless stuckless is offline
SageTVaholic
 
Join Date: Oct 2007
Location: London, Ontario, Canada
Posts: 9,554
Quote:
Is there support for VIDEO_TS?
Not yet... it is on my todo list... and it's not hard to implement (i think). I actually check for VIDEO_TS directories and currently skip them

FYI.... there tool does create a log file in your user's home directory called metadataupdater.log... you'll see messages in there about "Skipping DVD Folders for now"
Reply With Quote
  #9  
Old 07-20-2008, 08:56 AM
bcjenkins bcjenkins is offline
SageTVaholic
 
Join Date: Jan 2006
Posts: 3,760
It isn't giving me that error, I am getting:

Code:
73 ERROR [main] org.jdna.metadataupdater.MetadataUpdater - Unknown Media Resourse for: file:/var/media/videos/MovieName/;org.jdna.media.impl.MediaFolder
It doesn't seem to be finding VIDEO_TS

B
__________________
Running SageTV on unRAID via Docker
Tuning handled by HDHR3-6CC-3X2 using OpenDCT
Reply With Quote
  #10  
Old 07-20-2008, 09:02 AM
stuckless's Avatar
stuckless stuckless is offline
SageTVaholic
 
Join Date: Oct 2007
Location: London, Ontario, Canada
Posts: 9,554
Quote:
It doesn't seem to be finding VIDEO_TS
I'll look into that for you. I'll just go ahead and implement dvd folder support

Update: I can see the problem now... I'm only checking for DVD Folders when --recurse is used. I'll try to add in dvd folder support sometime today (later )

Last edited by stuckless; 07-20-2008 at 09:04 AM.
Reply With Quote
  #11  
Old 07-20-2008, 09:08 AM
jusjoken jusjoken is offline
Sage Icon
 
Join Date: Dec 2005
Location: Minneapolis MN
Posts: 2,177
My import into Sage worked well. The only issue I see is that the Categories (Genre) seem to import as a complete string including the ; seperators. I would have thought that the first one would be the Category and the second would be the SubCategory and the rest would be ignored. This is likely an issue with the import in Sage and not your tool unless the formating in the .properties is not what Sage is expecting.

The other thing I noticed is that I thought there was a propertiy for a Rating and for an Extended Rating. A Rating might be PG where and extended rating might be "Rated PG for ....". Not sure IMDB offers both nor if the properties file support both, I just remember getting a simpler Rating (only the PG) when using the imports in SageMC (now using the Default STV).

Otherwise, great so far.
Reply With Quote
  #12  
Old 07-20-2008, 10:26 AM
bcjenkins bcjenkins is offline
SageTVaholic
 
Join Date: Jan 2006
Posts: 3,760
Quote:
Originally Posted by stuckless View Post
I'll look into that for you. I'll just go ahead and implement dvd folder support

Update: I can see the problem now... I'm only checking for DVD Folders when --recurse is used. I'll try to add in dvd folder support sometime today (later )
Thanks! All of my videos are in DVD format. Oddly the properties file for DVD folders needs to go at the same level as the $DVD_NAME directory. Instead of in the VIDEO_TS folder like the folder.jpg.

B
__________________
Running SageTV on unRAID via Docker
Tuning handled by HDHR3-6CC-3X2 using OpenDCT
Reply With Quote
  #13  
Old 07-20-2008, 12:59 PM
JREkiwi's Avatar
JREkiwi JREkiwi is offline
Sage Icon
 
Join Date: Jan 2005
Location: Auckland, New Zealand
Posts: 2,073
Quote:
Originally Posted by bcjenkins View Post
Oddly the properties file for DVD folders needs to go at the same level as the $DVD_NAME directory. Instead of in the VIDEO_TS folder like the folder.jpg.
You can have a .jpg in the same level as the $DVD_NAME directory with the .properties file. It just needs to be called $DVD_NAME.jpg

John
Reply With Quote
  #14  
Old 07-20-2008, 01:16 PM
bcjenkins bcjenkins is offline
SageTVaholic
 
Join Date: Jan 2006
Posts: 3,760
I would prefer a folder.properties at the same level as the folder.jpg. It makes for a cleaner videos directory.



B
__________________
Running SageTV on unRAID via Docker
Tuning handled by HDHR3-6CC-3X2 using OpenDCT
Reply With Quote
  #15  
Old 07-20-2008, 03:44 PM
stuckless's Avatar
stuckless stuckless is offline
SageTVaholic
 
Join Date: Oct 2007
Location: London, Ontario, Canada
Posts: 9,554
So if I understand this, for dvd folders, the property name is the same name as the dvd folder, and the thumbnail can either be the same as video folder name or VIDEO_TS/folder.jpg?

eg, if the DVD Folder is called Terminator,
Terminator/Terminator.properties
Terminator/Terminator.jpg
or
Terminator/Terminator.properties
Terminator/VIDEO_TS/folder.jpg
Reply With Quote
  #16  
Old 07-20-2008, 03:49 PM
bcjenkins bcjenkins is offline
SageTVaholic
 
Join Date: Jan 2006
Posts: 3,760
./Terminator/VIDEO_TS/folder.jpg
./Terminator.properties

B
__________________
Running SageTV on unRAID via Docker
Tuning handled by HDHR3-6CC-3X2 using OpenDCT
Reply With Quote
  #17  
Old 07-20-2008, 04:11 PM
JREkiwi's Avatar
JREkiwi JREkiwi is offline
Sage Icon
 
Join Date: Jan 2005
Location: Auckland, New Zealand
Posts: 2,073
With 6.4 the requirement for a VIDEO_TS was removed for DVD's and support for .properties file for DVD's was introduced and defined as being located in the parent folder. So for DVD's

In the parent folder DVDs
DVDs\<DVDName>\dvd files
DVDs\<DVDName>.properties
DVDs\<DVDName>.jpg
DVDs\<DVDName>\folder.jpg does not work
and
DVDs\<DVDName>\VIDEO_TS\dvd files
DVDs\<DVDName>.properties
DVDs\<DVDName>.jpg
or
DVDs\<DVDName>\VIDEO_TS\folder.jpg

John

Last edited by JREkiwi; 07-20-2008 at 04:15 PM.
Reply With Quote
  #18  
Old 07-20-2008, 04:21 PM
Brent Brent is offline
SageTVaholic
 
Join Date: May 2006
Location: KC, Missouri
Posts: 3,695
Quote:
Originally Posted by JREkiwi View Post
With 6.4 the requirement for a VIDEO_TS was removed for DVD's and support for .properties file for DVD's was introduced and defined as being located in the parent folder. So for DVD's

In the parent folder DVDs
DVDs\<DVDName>\dvd files
DVDs\<DVDName>.properties
DVDs\<DVDName>.jpg
DVDs\<DVDName>\folder.jpg does not work
and
DVDs\<DVDName>\VIDEO_TS\dvd files
DVDs\<DVDName>.properties
DVDs\<DVDName>.jpg
or
DVDs\<DVDName>\VIDEO_TS\folder.jpg

John
True, but since SageMC (with DVDProfiler) hasn't caught up with that requirement many still have the "video_ts" folder.
Reply With Quote
  #19  
Old 07-20-2008, 04:28 PM
JREkiwi's Avatar
JREkiwi JREkiwi is offline
Sage Icon
 
Join Date: Jan 2005
Location: Auckland, New Zealand
Posts: 2,073
Quote:
Originally Posted by Brent View Post
True, but since SageMC (with DVDProfiler) hasn't caught up with that requirement many still have the "video_ts" folder.
I suppose the point I was really trying to make is that folder.jpg only works if you are using VIDEO_TS folders. <DVDNAME>.jpg works in both configurations, and there is only 1 valid location for the .properties file

John
Reply With Quote
  #20  
Old 07-20-2008, 04:31 PM
Brent Brent is offline
SageTVaholic
 
Join Date: May 2006
Location: KC, Missouri
Posts: 3,695
Quote:
Originally Posted by JREkiwi View Post
I suppose the point I was really trying to make is that folder.jpg only works if you are using VIDEO_TS folders. <DVDNAME>.jpg works in both configurations, and there is only 1 valid location for the .properties file

John
Okay, understood.
Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
STV Import: Read & Write Metadata files for Default STV 5/6 (v2.2.9 18/Feb./2010) JREkiwi SageTV Customizations 251 10-09-2019 11:11 AM
Automated VideoRedo (DOS batch files) grauchy SageTV Customizations 3 09-08-2011 10:01 PM
Simple utility to control multiple USB or Serial HD DirecTV STBs jchiso Hardware Support 15 02-19-2009 06:27 PM
DVB-S setup with Digiguide EPG Grabber & XMLTV importer MCE-Refugee SageTV United Kingdom 27 09-08-2008 09:10 AM
Keeping custom metadata for imports via AddShow() Opus4 SageTV Studio 9 02-20-2008 06:35 PM


All times are GMT -6. The time now is 04:04 AM.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.
Copyright 2003-2005 SageTV, LLC. All rights reserved.