Utility: Batch Metadata Importer 3.x

[stuckless]

Originally Posted by OferL

And I would ask one thing to add to it: an option to select several files (series) and batch search for all of them. Updating them for series, season episode etc. manually.

The "scan" of the webui is pretty much done... but I'm working on a separate "browse" that would allow for this sort of this.

[QUOTE=technazz;382350I've checked the "Enable Default STV compatibility" and then re-indexed my video folders. I do not see movname.jpg being generated. Do I have to restart the server before this setting takes effact? **[/QUOTE]
You'd have to tell it to overwrite fanart and metadata or else, it won't recreate the files (or add new ones).

Originally Posted by rosemary

I also found out that the "Ignore These Folders" apparently doesn't like single backslash space characters. I tried forward slashes and started the scan, but only now do I see you must want double backslashes (should have realized it from the "Illegal/unsupported escape sequence" error.

This is a regular expression. The folder will be passed to the regular expression as a uri, so if you do include a directory, it should be forward slashes (/). The better results, i'd either not include directory separators and just use the directory name, or use the forward slash. I'd avoid things like, "c:\mydir\" since trying to enter that as a regular expression is uri format is almost impossible

sagex.phoenix.configuration.XmlMetadataProvider (XmlMetadataProvider.java:20) - Missing Configuration File: C:\Program Files\SageTV\SageTV\.\STVs\Phoenix\Configuration\Phoenix.xml

not an issue... bmt uses the phoenix config apis... and it's missing the actual phoenix config, which is normal.

Originally Posted by joejoeyjojo

Hi I am also having problems getting the BMT Web UI to work. I have installed following the intructions. I get a 404 not found error.

I have the Sage TV WEb Interface running on jetty as well and can get ot that fine. However when i go to http://localhost:8081/bmt it gives me the 404 error

a 404 is basically telling you that it's not installed. either it's not installed , or you are hitting a different server than the jetty server. is the 404 error message a Jetty error?

Originally Posted by rosemary

Any other ideas on what to check?

I just installed the Sage web interface, and it works fine on every restart of Sage. So it's not affecting all jetty apps. The one main difference I spot between the web UI and BMT is that BMT creates a temp dir. It seems it only fails if the bmt dir exists.

What temp dir? what's it called?

Furthermore: I decided to watch what is going on when SageTV gets closed. Well, a bunch of the files from the bmt dir get deleted. But not all. Here's the ones left:

Then when you restart Sage, these are the ONLY files in the bmt dir. The other files never get re-created. This would seem like what might be screwing things up.

Sorry, this seems a bit stream-of-consciousness, but I just found this:

http://jira.codehaus.org/browse/JETTY-848

Now I'm just trying to figure out where to go from here. They point to a sweeper daemon, but I'm trying to understand where this will fit with a jetty install in Sage.

I'll check this out... maybe there is something that I can do in the bmt context.xml file.

[stuckless]

1. Shutdown SageTV/Jetty
2. replace your jetty/contexts/bmt.xml with the attached one
3. remove jetty/webapps/bmt/ (directory not .war, if it exists)
4. Startup SageTV/Jetty

The issue is that because BMT specified a temp area (mainly because I have very little experience with the jetty context xml and I was just copy and pasting stuff), and when jetty starts up, it extracts the war if the tmp area does not exist, and when it shuts down, it deletes it. The problem is that under windows, files become locked quite easily, and as a result, the removal of the temp area doesn't fully happen. So when jetty starts up, it will not re-extract it's war, because the tmp area already exists. I've never seen this issue, since I don't use windows.

I've checked a few other jetty app context xml files, and none of them specify a temp area, so I removed it from my context xml as well.

hopefully this will will fix the issues that people are having with seeing a directory listing instead of the actual web page.

[MeInMaui]

FYI,

With pre-release 12, I think I'm finally done playing whack-a-mole with the SageMC video library and it should now be compatible with BMT 3.1. Thanks for your patience.

Aloha,
Mike

[rosemary]

Excellent. Looks like getting rid of the temp directory made it survive sage restarts. Thanks!

[stuckless]

Originally Posted by MeInMaui

With pre-release 12, I think I'm finally done playing whack-a-mole with the SageMC video library and it should now be compatible with BMT 3.1. Thanks for your patience.

Thanks Mike... No one appreciates this more than I do besides who doesn't like whack-a-mole

[ke6guj]

minor pet peeve.

Would it be possible to leave the time stamps of the files tht you put in the zip file be the original time stamp from when that file was made? For instance, every JAR files you have in the BMT3.1 zip file has a 9/8/2009 date stamp, no matter when it was actually last updated. It makes it hard to know what has changed, and what is the most current version of a file when all hte time stamps are altered.

[mkanet]

Sean, I'm not sure if this will help any to improve BMT any, but tonight the show Smallville Recorded, but the fanart didn't automatically download. So, I manually added Smallville (2001) via on demand method of BMT. It's very unusal for it to fail for me; as, all TV series shows have worked for me except this. The file name is: Smallville-Savior-5103378-0.mpg. Who knows, I might have messed something up just recently trying to get the jetty version of neils webserver working; which I havent gotten to work yet.

[stuckless]

Originally Posted by ke6guj

Would it be possible to leave the time stamps of the files tht you put in the zip file be the original time stamp from when that file was made? For instance, every JAR files you have in the BMT3.1 zip file has a 9/8/2009 date stamp, no matter when it was actually last updated. It makes it hard to know what has changed, and what is the most current version of a file when all hte time stamps are altered.

That's never to going to happen Each time bmt is built, it basically fetches all dependencies, and then rebuilds everything. While some of the dependency jars may not have changed, they are downloaded, copied, etc, so preserving a timestamp will be next to impossible, and I'll never invest time into it.

[Qusix]

When I do a scan on the web interface for videos should it return all my videos?

It only returns the videos that have missing metadata, however I did not specify that option. I only selected scan videos.

[stuckless]

Originally Posted by Qusix

When I do a scan on the web interface for videos should it return all my videos?

Assuming you are using 3.1... the Scan dialog has a comment at the top, explaining that only media that cannot be automatically updated will be returned.

It only returns the videos that have missing metadata, however I did not specify that option. I only selected scan videos.

Many of the dialogs in 3.1 have a ? icon. When you click this icon, you will get help for a given section. The help for that section explains that the default behaviour is to scan everything, unless you check one of the other options.

If you are looking for something that will simply list the videos, without scanning, then use the Browse link.

[joejoeyjojo]

Ok SO i posted before i get a 404 error which i still do. However if i stop the sage interface which i have specified needs a login and then go to the bmt page it comes up fine. Howver the second i run the sage web interface and then try to go to the bmt web page it comes up with 404 error. I can only run one or the other.

I have a login required for the Sage web interface any idea?


By the way to get BMT web interface ot work form my last post all I did was stop the service. Copy everything back over to the jetty folder etc. And then hit the BMT web page first. IF i hit the Sage web interface first then bmt never comes up.


Regards

Joe

[stuckless]

Originally Posted by joejoeyjojo

Ok SO i posted before i get a 404 error which i still do.

Did you apply the new bmt.xml from this post?

[Skirge01]

I'm having issues getting my videos (non-SageTV recordings) to have their metadata populated by BMT. I ran the detailed reports and here are the results of my system info, followed by a single example of a video which didn't get any metadata:

         Phoenix Diagnostic Report - Date: Sep 26, 2009 7:42:54 PM         


                          === Version Checks ===                           
               Phoenix Version: 1.43
             Sagex.api Version: 6.6.1
                  Sage Version: 6.6.2.218

                   === Metadata/Fanart Configuration ===                   
        Custom Metadata Fields: UserRating;EpisodeNumber;EpisodeTitle;MediaTitle;MediaType;OriginalAirDate;SeasonNumber;MediaProviderDataID;DiscNumber
    MediaFile Metadata Plugins: org.jdna.sage.MetadataUpdaterPlugin
Manual Metadata Lookups Enabled: true

                 === Phoenix Metadata/Fanart Settings ===                  
phoenix/mediametadata/fanartEnabled: true
phoenix/mediametadata/fanartCentralFolder: \\Mediaserver\SageTV_DVR\Fanart
phoenix/mediametadata/parseMediaFileForTVSeries: false
phoenix/mediametadata/tvSeriesRegex: null
phoenix/mediametadata/tvSeasonRegex: null
phoenix/mediametadata/fanartSupportClass: sagex.phoenix.fanart.PhoenixFanartSupport
phoenix/mediametadata/mediaMetadataSupportClass: bmt.BMTMetadataSupport



                     === Java Environment Settings ===                     
                    http.agent: SageTV SJQ/2.1.5.511
             java.runtime.name: Java(TM) SE Runtime Environment
    java.protocol.handler.pkgs: com.sun.net.ssl.internal.www.protocol
         sun.boot.library.path: C:\Program Files\SageTV\SageTV\jre\bin
               java.vm.version: 1.6.0-b105
              sun.awt.nopixfmt: false
                sun.java2d.d3d: false
                jetty.ssl.port: 8446
                java.vm.vendor: Sun Microsystems Inc.
               java.vendor.url: http://java.sun.com/
                path.separator: ;
                  java.vm.name: Java HotSpot(TM) Client VM
             file.encoding.pkg: sun.io
                  user.country: US
            sun.os.patch.level: Service Pack 2
    java.vm.specification.name: Java Virtual Machine Specification
                      user.dir: C:\Program Files\SageTV\SageTV
          java.runtime.version: 1.6.0-b105
          java.awt.graphicsenv: sun.awt.Win32GraphicsEnvironment
            jetty.ssl.keystore: C:\Program Files\SageTV\SageTV\jetty\etc\keystore
            java.endorsed.dirs: C:\Program Files\SageTV\SageTV\jre\lib\endorsed
                       os.arch: x86
                java.io.tmpdir: C:\WINDOWS\TEMP\
                line.separator: 

  java.vm.specification.vendor: Sun Microsystems Inc.
        sun.java2d.ddoffscreen: false
                  user.variant: 
                       os.name: Windows 2003
            sun.java2d.noddraw: true
              sun.jnu.encoding: Cp1252
             java.library.path: C:\Program Files\SageTV\SageTV\;
       java.specification.name: Java Platform API Specification
            java.class.version: 50.0
                    jetty.home: C:\Program Files\SageTV\SageTV\jetty
       sun.management.compiler: HotSpot Client Compiler
                    os.version: 5.2
                     user.home: C:\Documents and Settings\Default User
                 user.timezone: America/New_York
           java.awt.printerjob: sun.awt.windows.WPrinterJob
                         DEBUG: true
    java.specification.version: 1.6
                 file.encoding: Cp1252
                     user.name: SYSTEM
               java.class.path: C:\Program Files\SageTV\SageTV\Sage.jar;C:\Program Files\SageTV\SageTV\xerces.jar;C:\Program Files\SageTV\SageTV\plugin.jar;C:\Program Files\SageTV\SageTV\;;JARs\ant-1.6.5.jar;JARs\commons-codec-1.3.jar;JARs\commons-io-1.4.jar;JARs\commons-jxpath-1.1.jar;JARs\commons-lang-2.4.jar;JARs\commons-logging-api.jar;JARs\commons-logging.jar;JARs\commons-vfs-20070823.jar;JARs\core-3.1.1.jar;JARs\htmlparser.jar;JARs\jcifs-1.1.6.jar;JARs\jdic.jar;JARs\jetty-6.1.19.jar;JARs\jetty-ajp-6.1.19.jar;JARs\jetty-rewrite-handler-6.1.19.jar;JARs\jetty-sslengine-6.1.11.jar;JARs\jetty-sslengine-6.1.19.jar;JARs\jetty-starter.jar;JARs\jetty-util-6.1.19.jar;JARs\jsp-2.1-jetty-6.1.19.jar;JARs\jsp-2.1.jar;JARs\jsp-api-2.1.jar;JARs\log4j-1.2.15.jar;JARs\lucene-core-2.3.2.jar;JARs\metadata-updater.jar;JARs\miglayout-3.6.3-swing.jar;JARs\mx4j-impl.jar;JARs\mx4j-jmx.jar;JARs\mx4j-remote.jar;JARs\mx4j-tools.jar;JARs\nielm_sageimdb.jar;JARs\nielm_sageutls.jar;JARs\nielm_sagewebserver.jar;JARs\nielm_sagexmlinfo.jar;JARs\phoenix.jar;JARs\sagex.api.jar;JARs\sbbi-jmx-1.0.jar;JARs\sbbi-upnplib-1.0.3.jar;JARs\servlet-api-2.5-6.1.19.jar;JARs\servlet.jar
 java.vm.specification.version: 1.0
           sun.arch.data.model: 32
                     java.home: C:\Program Files\SageTV\SageTV\jre
     java.specification.vendor: Sun Microsystems Inc.
                 user.language: en
                   awt.toolkit: sun.awt.windows.WToolkit
                  java.vm.info: mixed mode
                  java.version: 1.6.0
                 java.ext.dirs: C:\Program Files\SageTV\SageTV\jre\lib\ext;C:\WINDOWS\Sun\Java\lib\ext
           sun.boot.class.path: C:\Program Files\SageTV\SageTV\jre\lib\resources.jar;C:\Program Files\SageTV\SageTV\jre\lib\rt.jar;C:\Program Files\SageTV\SageTV\jre\lib\sunrsasign.jar;C:\Program Files\SageTV\SageTV\jre\lib\jsse.jar;C:\Program Files\SageTV\SageTV\jre\lib\jce.jar;C:\Program Files\SageTV\SageTV\jre\lib\charsets.jar;C:\Program Files\SageTV\SageTV\jre\classes
                   java.vendor: Sun Microsystems Inc.
             jetty.configfiles: "C:\Program Files\SageTV\SageTV\jetty/etc/jetty.xml" "C:\Program Files\SageTV\SageTV\jetty/etc/jetty-ssl.xml" "C:\Program Files\SageTV\SageTV\jetty/etc/jetty-logging.xml"
                file.separator: \
           java.vendor.url.bug: http://java.sun.com/cgi-bin/bugreport.cgi
                sun.cpu.endian: little
       sun.io.unicode.encoding: UnicodeLittle
          jetty.ssl.truststore: C:\Program Files\SageTV\SageTV\jetty\etc\keystore
      java.rmi.server.hostname: MediaServer
                    jetty.logs: C:\Program Files\SageTV\SageTV\jetty\logs
                   sun.desktop: windows
                    jetty.port: 8125
               sun.cpu.isalist: pentium_pro+mmx pentium_pro pentium+mmx pentium i486 i386 i86

<snip>

BEGIN MediaFile \\Mediaserver\Videos\TV Series\Stargate SG-1\Season 01\Stargate SG-1 - S01E17 - Enigma.avi
                 Display Title: Stargate SG-1 - S01E17 - Enigma

                        --- RAW Metadata Fields ---                        
                    MediaTitle: 
                     MediaType: 
                  SeasonNumber: 

                    --- Calculated Metadata Fields ---                     
                    MediaTitle: Stargate SG-1 - S01E17 - Enigma
                     MediaType: MOVIE
                  SeasonNumber: 0

                           --- Fanart Paths ---                            
               Background Path: \\Mediaserver\SageTV_DVR\Fanart\Movies\Stargate SG-1 - S01E17 - Enigma\Backgrounds
                   Banner Path: \\Mediaserver\SageTV_DVR\Fanart\Movies\Stargate SG-1 - S01E17 - Enigma\Banners
                   Poster Path: \\Mediaserver\SageTV_DVR\Fanart\Movies\Stargate SG-1 - S01E17 - Enigma\Posters

                   --- Fanart Single Files and Times ---                   
                    Background: null
          Background Time (ms): 0
                        Banner: null
              Banner Time (ms): 0
                        Poster: null
              Poster Time (ms): 0
END MediaFile \\Mediaserver\Videos\TV Series\Stargate SG-1\Season 01\Stargate SG-1 - S01E17 - Enigma.avi

I tried to read up on this as much as possible before asking for help and this is what I could figure out to get you on my own. If there's more data you need, just let me know how to get it for you.

[stuckless]

Originally Posted by Skirge01

I'm having issues getting my videos (non-SageTV recordings) to have their metadata populated by BMT. I ran the detailed reports and here are the results of my system info, followed by a single example of a video which didn't get any metadata:

I think your particular issue with this video is that the filename has "Stargate SG-1". If you rename the file to "Stargate SG1 - SXXEXX - Episode", you'll probably get a hit (ie, just remove the dash from the SG-1 in the filename only, directory doesn't matter). This is a bug in how bmt "cleans" titles. I'm working on a way to fix this, but it's not ready to be released.

[Skirge01]

At least I know I'm not going nuts. I was just about pulling my hair out trying to figure out what I had done wrong. Of course, these were the first ones I tried after installing BMT, so it figures!

That fixed it all up. Thanks for the quick reply and the amazing work you've done on this tool!

[ldavis]

Stuckless,

I am new to BMT/fanart (I was out of the country for a majority of this year) and am just now getting around to installing this utility. I have 2 questions that I am sure have been asked earlier so I will apologize for the repetition in advance.

1.) When I use BMT to get fanart on media import it does get the proper information but it puts the information in the folder with the media file and not in the central file (I know that I have the central file set up to the default). How can I change the behavior?

2.) When I manually get the fanart it is put in the correct folder but for movies (at least in folder view), they do not show up in the correct folder. For example, all of my movies are in a movies folder. When I do the manual scan, the movie shows up not in a folder at all. Note that the file has not been physically moved - I just shows up in Sage this way. How can I change this behavior?

[scm]

ldavis,

Make sure you don't find the fan art in the following location:
C:\Program Files\SageTV\SageTV\STVs\Phoenix\Fanart

By default, bmi put all my images here. Otherwise, make sure you have the latest release of BMI and SageMC. Changes to both could be giving this problem.

SCM

[ldavis]

SCM,

I already checked all of that and I know the folder is set correctly, else how would BMI put the manually imported images in the right place?

[ldavis]

Problem 1 solved (which reduces problem 2 to more of a curosity) . Turns out that even though the BMI STVi has a default Fanart location that shows up in the STV and in the sage.properties file. Even though the shown folder works for manual imports, (at least on my system) I had to go into this setting within the STVi and essentially reaffirm this default location to get it to work.

I am still interested in question 2 – Why a manual import of a movie results in an (apparent) change of folder location for movies but this is much less important now.

[stuckless]

Originally Posted by ldavis

I am still interested in question 2 – Why a manual import of a movie results in an (apparent) change of folder location for movies but this is much less important now.

I think that this issue can be resolved by updating to the very latest SageMC. The way in which bmt writes the metadata caused an issue in the SageMC folder display. Mike has provided a fix for this (you can search back a few posts for more information). You didn't specifically mention SageMC, so I am just assuming that you are using it, because of the issue that is described.