Utility: Batch Metadata Importer 3.x

[stuckless]

Originally Posted by electronics4lif

Stuckless, are you still working on the duration for DVDs problem? Just wanted to make sure I didn't miss the fix.

I've added a bug for it... but i'm not sure if it's a bmt thing, or a DVD thing, since RunningTime does appear to work for regular avis. BMT doesn't distinguish between videos and dvds. ie, they both get their properties filled the same way.

[ehfortin]

Hi,

I have installed BMI a few weeks ago and had success with it for most of my recording. Lately, I"ve reinstalled DVD Profiler and added a few DVD to my collection. As usual, I have exported the file Collection.xml and try to update the metadata and posters from it with BMI. I can see that BMI is using the file correctly as metadata is updated. I also see that it recognize the poster and I can look at it if I select the link that appears under "FanArt". However, when I save the config, everything is changed but the poster is not copied to the appropriate folder. Actually, folder is not created either.

My fanart profile is using DVDProfiler + themoviedb.org. If the movie has any posters/backgrounds coming from themoviedb.org as well, then the folder will be created when I do save and all the images will be there, except the poster coming from dvdprofiler (again, it is showing as one of the recognized image so the issue seems with the copy to the appropriate place).

Is that a known problem? Is there a known fix? I would really like to update all the movies that don"t have any poster yet because they are not into themoviedb.org or any other online db.

Thank you for your time and assistance.

ehfortin

[stuckless]

Originally Posted by ehfortin

Hi,
Is that a known problem? Is there a known fix? I would really like to update all the movies that don"t have any poster yet because they are not into themoviedb.org or any other online db.

For bmt to use dvd profiler, you need to configure both the collection.xml and the images dir in the bmt configuration. From the webui, you can access the bmt configuration using Configure -> Metadata Tools -> DVD Profiler. It sounds as if your images dir is not set.

[ehfortin]

Hi Stuckless,

Yes, it is configured. This is what I"m seeing after doing a search in BMI:

Fanart 
 
POSTER \\fileserver\public\DVD Profiler\Databases\Default\Images\065935820994.19f.jpg

If I click on this link, I can see the image. If I do a save on the metadata found (including this link to the image), all the metadata is recorded but the image is not copied into the FanArt directory.

As I was saying, if the title I"m working on is also getting posters or backgrounds from themoviedb, these are copied correctly but the poster coming from DVD Profiler is never copied at all. I even tried to create the directory structure manually to see if it was a problem with the fact that they don"t exist for a new movie (even if they are created when images are coming from other sources like themoviedb) and it didn"t work either. It really sound as if there is no copy action at all when the URI is from a local file system (//fileserver/public/... in my case) instead of an external URL (http://themoviedb.org/... when using external data).

Can it be because of this difference?


ehfortin

[stuckless]

@ehfortin - that's an interesting case, and i think that you are right about the "save" not working because it's a local image file. I remember adding some code a couple of builds back, where I would skip fanart if the resource was a local file... I did this so that when you loaded an item with existing fanart, then it would not try and overwrite the file with the same local file. I didn't figure in that mymovies and dvdprofiler would always have "local" images (mainly because I don't use either of those sources )

I'll add a bug for this, but there is no short term fix

[ehfortin]

Hi,

Would be this method that induce the wrong behaviour?

    private boolean shouldSkipFile(File imageFile) {
        boolean skip = false;
        // only do this if we are using the central fanart
        if ((updaterConfig.isFanartEnabled() && !StringUtils.isEmpty(updaterConfig.getFanartCentralFolder()))) {
            try {
                File storedFile = new File(imageFile.getParentFile(), "images");
                StoredStringSet set = new StoredStringSet();
                String name = imageFile.getName();
                if (storedFile.exists()) {
                    StoredStringSet.load(set, storedFile);
                    if (set.contains(name)) {
                        log.debug("Skipping Image file: " + imageFile.getPath() + " because it's in the image skip file.");
                        skip = true;
                    }
                }
                if (!skip) {
                    log.debug("Adding Image file: " + imageFile.getPath() + " to the image skip file.");
                    set.add(name);
                    // create the file's parent dir if it's not exist
                    if (!storedFile.getParentFile().exists()) {
                        storedFile.getParentFile().mkdirs();
                    }
                    StoredStringSet.save(set, storedFile, "Ignoring these image files");
                }
            } catch (Exception e) {
                log.error("Failed to load/save the skip file for image: " + imageFile.getAbsolutePath(), e);
            }
        }
        return skip;
    }

DVD Profiler is effectively using the structure that end with "images" so, if I understand correctly the code that is here, it would generate a true each time this method is called even if it is not what is intented with DVD Profiler.

Is there a way to compare the path with what is saved in SageTV config regarding DVD Profiler? If so, the easy fix would be to add a condition where if the path is the one that is use for DVD Profiler database, not to generate a true to the skip function.

I wanted to do a quick test with this but... I don't seems to set properly a project environment within NetBeans so... I have errors everywhere and can't compile or run anything. Does the svn checkout include everything for a working environment? If so, with which development IDE? Or maybe it is directly at the command line?

Anyway, let me know if this make sense and if that's something that could be fixed quickly (if it's the proper place to do it).

Thank you.

ehfortin

[razrsharpe]

Originally Posted by ehfortin

Does the svn checkout include everything for a working environment?

i havent messed around with the checkout for phoenix/bmt but my guess is you need to include the sagex, log4j, and maybe some other jars in your java classpath...

[ehfortin]

Well, you are right. Most of the jars were present but... Netbeans didn't considered them until I kind of register them to the project. I actually had to do the same with the source code as well. It was visible in the project but not used/linked. I've added the sources as "Source" and the jars as "Library" and everything is now clean. I've build the batchmetadatatools.jar correctly.

I guess I'll try a quick change to the method and see if fix the problem (without any intelligence, just returning false for test purpose).

Thank you.

ehfortin

[stuckless]

@ehfortin - I use eclipse for development... but as razr pointed out, there are some dependency jars as well.

But, as for the code, I don't think that this is what you are looking for.... the "images" file is a file that is stored in the images dir that records the filenames of all files that you've downloaded. Once a name is written to that file, then it will not be downloaded again. This is done so that if you download fanart but then remove SOME of the images, then bmt will not attempt to download those images again.

I think your issue is actually comming from the webui, and only from the webUI... unless i'm misunderstanding your issue. In the webui, when you press save, I remove all "local" files from the fanart list and ONLY save fanart from remote locations.

I think your issues is comming from this file in the bmtweb project...

            // process the fanart images
            for (Iterator<IMediaArt> i = file.getMetadata().getFanart().iterator(); i.hasNext();) {
                IMediaArt ma = i.next();
                if (ma instanceof GWTMediaArt) {
                    GWTMediaArt gma = (GWTMediaArt) ma;
                    if (gma.isLocal()) {
                        log.debug("Skipping Download of Local Fanart: " + gma.getLabel());
                        i.remove();
                    }
                    if (gma.isDeleted() && gma.isLocal()) {
                        try {
                            File f = new File(gma.getLabel());
                            f.delete();
                        } catch (Throwable t) {
                            log.error("Unablet to delete: " + gma.getDownloadUrl());
                        }
                    }
                }
            }

Now, here's the real kicker.... the current bmt codebase is an unstable 4.x codebase... So, the onlyway to reliably fix this in bmt, is to do a checkout of the source code by date... ie, you'dhave to find out when I released 3.1 and then do a svn checkout for that date (or even a couple weeks after).

Normally I tag/branch a release, but since i'm pretty much the only one working in bmt, I didn't do that

[skiingwiz]

Originally Posted by stuckless

Now, here's the real kicker.... the current bmt codebase is an unstable 4.x codebase... So, the onlyway to reliably fix this in bmt, is to do a checkout of the source code by date... ie, you'dhave to find out when I released 3.1 and then do a svn checkout for that date (or even a couple weeks after).

Normally I tag/branch a release, but since i'm pretty much the only one working in bmt, I didn't do that

If it helps, I have revision 268 and it is version 3.1.

[ehfortin]

That would be great to get it, yes. The only package I was able to get were 266 for BMT (it is flagged as 3.1 so it should be fine) but for BMTWeb, I got 263 (which is related to 3.0) and then, it jump up to 281 (which is nearly current). And... 263 was an unclean commit (multiple errors with a few very hard to fix without a lot of coding around).

I just tried to capture directly 268 but I'm not sure that's what I got. So, if you can post a zip of either the whole thing or at least the source for BTM and BTMWeb, it would be appreciated.

Thank you.

ehfortin

PS: I'm sure I've not 268 because I have the same errors as I was getting in 263. Error in BrowserServicesImpl.java where MetadataAPI.normalizeMetadata is called with only two parameters while it is defined with three of those and the second error is with SageShowPeristence that is called with a parameter while there is none in the definition. The first one impose severe modifications as I can see in the newer revision so... there was something done that is not committed in the repository I'm using or I don't know what. I can't seems to have access to 268...

[Clift]

I've got an interesting problem:
Recently (and I'm not sure exactly when) my videos do not have any fanart or box art, and the description is not showing either. This is only for new(er) videos added after a certain date. All previous videos show up just fine with descriptions, box art, fan art, etc. I checked various .properties files and they are generated and look just like the other files. So I'm not sure what is going on. I'm going to have to check into further. But it's weird.

Any ideas?

Update: I loaded up a previous release of SageMC on one of my clients and all the new media fanart, desc, and box art are showing up finenow. So the problem seems to be with SageMC.

[skiingwiz]

Originally Posted by ehfortin

PS: I'm sure I've not 268 because I have the same errors as I was getting in 263. Error in BrowserServicesImpl.java where MetadataAPI.normalizeMetadata is called with only two parameters while it is defined with three of those and the second error is with SageShowPeristence that is called with a parameter while there is none in the definition. The first one impose severe modifications as I can see in the newer revision so... there was something done that is not committed in the repository I'm using or I don't know what. I can't seems to have access to 268...

I just looked at my copy and it also shows two arguments in that call in BrowserServicesImpl. I have never loaded BMTWeb into my ide (I have only been concerned with BMT proper). So, I guess 268 won't work for you either. Sorry about that.

[ehfortin]

Hi again,

Have you checked if the actual method is asking for 2 or 3 parameters? It may be OK that BrowserServicesImpl is using two parameters if the method is made this way as well. The code I have is awaiting for 3 parameters (that's 266) so maybe it was changed for 268 to work with 2 parameters?

If you send me the src, I'll be able to check or, you can let me know if you prefer.

Thank you.

ehfortin

[skiingwiz]

Originally Posted by ehfortin

Hi again,

Have you checked if the actual method is asking for 2 or 3 parameters? It may be OK that BrowserServicesImpl is using two parameters if the method is made this way as well. The code I have is awaiting for 3 parameters (that's 266) so maybe it was changed for 268 to work with 2 parameters?

If you send me the src, I'll be able to check or, you can let me know if you prefer.

Thank you.

ehfortin

Yeah, I checked. It is not corrent (2 params in the call and 3 in the definition). I loaded BMTWeb revision 268 into my IDE and found several such errors.

[ehfortin]

Ok. So it seems that the thing to do is to wait for the next release as it seems a too big job to recreate 3.1.

@stuckless: what is the current status of the new code? You told that it was unstable but how unstable is that? If I remember well, the code was compiling correctly (will have to redo a checkout of current code later on). That would be a good start. What is the plan for 4.x? If you have a documented plan of what is left to do, I can see if I can help here and there.

Let us know. Thank you for your good work, it is appreciated.

ehfortin

[stuckless]

Originally Posted by ehfortin

Ok. So it seems that the thing to do is to wait for the next release as it seems a too big job to recreate 3.1.

Sorry about that.... From now on, I will branch all new development after a major release to enable others to work on the released code. It's a practice that I should have started some time ago, but I didn't because I didn't think that anyone was actually checking out the code

@stuckless: what is the current status of the new code? You told that it was unstable but how unstable is that? If I remember well, the code was compiling correctly (will have to redo a checkout of current code later on). That would be a good start. What is the plan for 4.x? If you have a documented plan of what is left to do, I can see if I can help here and there.

The 4.0 code is a rewrite of the processing engine to better support all 4 models of scanning... WebUI, On Demand, Command-Line and Automatic Plugin.

Thew 4.0 code will add some concepts, such as scan queues where multiple threads can deplete the queue... this will give some benefits to scanning large collections.

In addition to that, I'm also adding better support for manually configuring titles. (ie a replacement for the metadata-titles). I'm also adding in ability to select titles base don year, (currently the year is ignored). And, the filename matching is being handled by the filename scrapers, (which is how the tv matching works corrently).

From the UI perpsective, a new browser is being built, and the scanner is being updated to allow for better progress reporting and even cancelling a scan.

There are bits and pieces of each piece that is currently done, but it's still a couple of months away. Nothing is really changing in terms of the metadata providers and persistence engines, so this is really a refactoring of how the engine works, and to pull in more and more of the core phoenix code. In cases where phoenix has functionality that duplicates bmt, then I remove that functionality form bmt and use the phoenix apis.

If you are interesting in helping with the coding, then give me a couple of weeks to add in a couple of changes, and then I'll contact you.

thanks,
Sean.

[ehfortin]

Hi,

I could be interested. I've done lot of web development using Java/Servlet implementing my own brew of MVC but that's now about 9 years old so I'm probably rusty but I think I should be able to get back on track with hands-on. I've always been more of an architect then a coder but I was not too bad

Let me know once you have something in the state you want. Until then, I'll do a checkout of the current code and try to understand by myself how you've implemented this up to now.


ehfortin

[SteveW]

Quick Question... I have 3 Extenders and a Client on Windows 7.... If I want to give BMT a try, can I just install it on my client to try it there first before my Sage server, or do I have to have it on my server as well if I want it on my client.

My client is more for playing in right now, so I'd rather blow that up than my server trying this out. Tnx!

[toricred]

I need some help here. I've had BMT working very well for quite a while now. I've noticed recently that CSI: NY is not getting picked up. I'm assuming it is based on the : in the title. Any suggestions how to fix this?