Metadata Tools 4.0 Beta Thread (aka BMT 4.0)

[stuckless]

Originally Posted by petereader

I think I may have a problem crop back up that I had on a previous build. The show title for the last two shows that recorded seem to have adopted the show title of the recorded movie that I manually grabbed metadata for just a few hours ago using the webGUI (the first one I did after installing the first beta4). The media type also appears to have been adopted from that first scan. It was a movie, while these new recordings are TV shows. Would like to look at some log files but not sure which ones to look at.

That's interesting. Things like the title and media type are never stored as configuration, so the only way this can be happening is if, somehow, when I'm updating the metadata, that it's re-using an older show/airing.

The log files are either... bmt.log*, phoenix.log* or mailcheck.*

i know it's confusing... but you can force set the logfile name in the log4j.properties and from the log4j configuration in the webui. The name of the log file really depends on what was installed first, and if there was an existing configuration. ie, most apps will not overwrite the logj4 setting, but if it's not there, then the app will create one. So depending on the app that gets run first, when there isn't a configuration, decides the file. Typically if sagemc is isntalled the file is mailcheck.txt (i think).

[toricred]

So far I've had the automatic plugin succeed once and fail once. This was right about the time I posted that thetvdb.com had come back up. The first recording was the one that failed so I'm hoping it was just that thetvdb.com hadn't quite come up all the way yet. So far this is working much better than the previous versions and I'm in love with how well it's doing on accuracy. I know it's still beta, but this is as good as the typical Sage beta. I'm really impressed.

[stuckless]

Originally Posted by petereader

I think I may have a problem crop back up that I had on a previous build. The show title for the last two shows that recorded seem to have adopted the show title of the recorded movie that I manually grabbed metadata for just a few hours ago using the webGUI (the first one I did after installing the first beta4). The media type also appears to have been adopted from that first scan. It was a movie, while these new recordings are TV shows. Would like to look at some log files but not sure which ones to look at.

The interesting thing about this, is that I no longer allow bmt to update the metadata for recorded shows. It can still update the fanart for recorded shows (which includes setting the season/episode), but it no longer updates the wiz.bin airing/show details. If you have logs then send them along to metadatatools at gmail dot com.

[bikesquid]

The latest build hasn't changed anything for me that I can tell, first letter still missing, nothing seems to be 'found'....
the VIDEO_TS has gone away as promised.
New log file seems to have lots more errors and such if interested I can email.

Once I do set a file up manually it seems happy, it's the automagic that's not.

[stuckless]

Originally Posted by petereader

toricred,
I just did a manual scan using thetvdb and got results back (beta6)

Sean,
I think I may have a problem crop back up that I had on a previous build. The show title for the last two shows that recorded seem to have adopted the show title of the recorded movie that I manually grabbed metadata for just a few hours ago using the webGUI (the first one I did after installing the first beta4). The media type also appears to have been adopted from that first scan. It was a movie, while these new recordings are TV shows. Would like to look at some log files but not sure which ones to look at.

Originally Posted by bikesquid

The latest build hasn't changed anything for me that I can tell, first letter still missing, nothing seems to be 'found'....
the VIDEO_TS has gone away as promised.
New log file seems to have lots more errors and such if interested I can email.

Once I do set a file up manually it seems happy, it's the automagic that's not.

sure email the logs... that's best place to figure out what's happening.

[Tiki]

Originally Posted by stuckless

The interesting thing about this, is that I no longer allow bmt to update the metadata for recorded shows. It can still update the fanart for recorded shows (which includes setting the season/episode), but it no longer updates the wiz.bin airing/show details. If you have logs then send them along to metadatatools at gmail dot com.

Sean,

Could you explain this a little more please? Are you saying that for Sage Recordings, you no longer get any metadata (if so, then the new options in the API for overwrite-metadata: 'false' would not really have any affect for Sage Recordings, right)? Where are you setting the season/episode info if not in Wiz.bin?

I'm just trying to make sure I understand the intended operation.

[toricred]

OK, this time I know it should have been able to get the metadata through the automatic plugin and it didn't. I was able to get it through the web GUI so I know it was available. I think the automatic plugin is not quite totally stable yet.

[stuckless]

Originally Posted by Tiki

Could you explain this a little more please? Are you saying that for Sage Recordings, you no longer get any metadata (if so, then the new options in the API for overwrite-metadata: 'false' would not really have any affect for Sage Recordings, right)? Where are you setting the season/episode info if not in Wiz.bin?

Metadata/Fanart can be grouped into 3 sections; Core Metadata, Custom Metadata, and Fanart.

Core Metadata is the data that is a part of the Airing/Show objects, ie, wiz.bin. The only way to update that metadata is to create a new Show object using AddShow() and then attach that Show to the MediaFile. When you add a new Show and then attach it to the MediaFile, sagetv automatically creates a new Airing as well. So, the process of updating Core Metadata basically involves removing the old, and then replacing it with new metadata. The whole process is quite cumbersome.

The current rules for Core Metadata is basically you can Import TV files in as Recordings, and you can update that Metadata later, if you want. You cannot import Movies as recordings. And, for native sage recordings (ie, recordings that sage actually created), you cannot update the Core Metadata. The assumption here is that sage probably has valid metadata for it's recordings, so rather than risk corrupting the metadata, I simply refuse to update it. (that's new in 4.0). I was hoping that this stance would actually get rid of the issue that some people experienced, where Airing/Show information appeared to become cross linked. Although there are reports of that happening in this beta as well, which is very puzzling.

In addition to Core Metadata, there is Custom Metadata. Custom Metadata is the metadata defined in the custom_metadata_properties property. Sage provides a Read/Write interface for these properties, so updating them is quite trivial. While they are stored in the wiz.bin, I tend to not consider it an act of updating the wiz.bin, since, there is no risk of corrupting the wiz.bin core metadata by setting these fields. SeasonNumber, EpisodeNumber, MediaTitle, and MediaType are all stored in the Custom Metadata fields. As a side note, these fields MUST be defined in the custom_metadata_properties or else sage will simply discard them.

Lastly, there is Fanart. Fanart is directly tied to the Custom Metadata, and does not depend on the Core Metadata at all. Fanart uses the MediaType and MediaTitle fields to determine where to find fanart. The Season is also used if it's a TV MediaType and the Season has a value.

The properties for overwrite-fanart and overwrite-metadata are taken into consideration when udpating all 3, but in the case of native sage recordings, if overwrite-metadata is true, then the Core Metadata is not touched, but the Custom Metadata will be. Also, if overwrite-fanart is true, but overwrite-metadata is false, then the Custom Metadata Fields are still updated, since it is those fields that ensure that fanart works correctly.

Lastly we have the .properties files. It's basically a backup of what's in Core, Custom and Fanart. When Custom Metadata is updated using the SageTV apis, it will also write those properties to the .properties file. But, when Core Metadata is updated using the SageTV apis, the .properties are not updated. BMT provides it's own .properties updater, that can be disabled in 4.0.

It's all rather complicated... It's taken me a long time to figure it all out... and I'm still figuring

[stuckless]

Originally Posted by toricred

OK, this time I know it should have been able to get the metadata through the automatic plugin and it didn't. I was able to get it through the web GUI so I know it was available. I think the automatic plugin is not quite totally stable yet.

The logs will help here... the process of updating is so similar in all cases, that obviously the plugin is doing something subtly different.

[petereader]

Some more info,

Sean I just emailed you some log files and a screenshot. I also added a movie to my imported videos a few hours ago and the automatic plugin doesn't seem to have picked up metadata/fanart for it.

One other thing I noticed, on my client that uses a 1280 x 720 resolution, I don't seem to have a way to scroll down and view what below the viewable area of my screen. This also happens on the "pop-up" windows for search results. I included a few screenshots in case I didn't describe this well.

I also wanted to take the opportunity to thank you Sean, for all the time and effort you have put into this. This is a fantastic plugin, one I can't imagine doing without. This beta version,as mentioned previously, yields better results than previous versions (that weren't too shabby either), the webui is much more responsive, especially with things like browsing. And I really like the redesign.

Cheers

[stuckless]

@Peter, i just set my screen size.... i can see where that can get a little frustrating

[bikesquid]

Originally Posted by petereader

One other thing I noticed, on my client that uses a 1280 x 720 resolution, I don't seem to have a way to scroll down and view what below the viewable area of my screen. This also happens on the "pop-up" windows for search results. I included a few screenshots in case I didn't describe this well.
Cheers

Does Ctrl+ and Ctrl- not make things bigger/smaller...? it does on my setup.

[toricred]

One more clue I've found. The plugin seems to only have a problem with shows I've had to define in MetadataTitles.

[petereader]

Originally Posted by bikesquid

Does Ctrl+ and Ctrl- not make things bigger/smaller...? it does on my setup.

Yeah, forgot about the zoom feature; I've been using F11. I just thought I would mention it since I noticed it.

[stuckless]

Originally Posted by bikesquid

The latest build hasn't changed anything for me that I can tell, first letter still missing, nothing seems to be 'found'....
the VIDEO_TS has gone away as promised.
New log file seems to have lots more errors and such if interested I can email.

Once I do set a file up manually it seems happy, it's the automagic that's not.

The missing first letter is quite puzzling. When you are browsing the collection, you'll notice a sprocket icon on each thumbnail. This sprocket is mainly for me for debugging. When you click on it, it will pop open a dialog where it given you dropdown selection. It should show the actual Wiz.bin values for that item. Check the title and see if it is missing the first letter. Next select the "Custom Metadata Fields" from the drop down list. See if the Media Title is missing the first letter. If those are showing the correct title, then the problem lies somewhere within the UI side of things, which helps me narrow things down.

[stuckless]

Originally Posted by toricred

One more clue I've found. The plugin seems to only have a problem with shows I've had to define in MetadataTitles.

could you send me you media titles xml?

[bikesquid]

Originally Posted by stuckless

The missing first letter is quite puzzling. When you are browsing the collection, you'll notice a sprocket icon on each thumbnail. This sprocket is mainly for me for debugging. When you click on it, it will pop open a dialog where it given you dropdown selection. It should show the actual Wiz.bin values for that item. Check the title and see if it is missing the first letter. Next select the "Custom Metadata Fields" from the drop down list. See if the Media Title is missing the first letter. If those are showing the correct title, then the problem lies somewhere within the UI side of things, which helps me narrow things down.

I only get the sprocket for non 'sage imports' anything in sage imports (includes most everything that's not right) I don't get it and clicking on the item shows me nothin. I suspect something got screwed up whereever the import stuff goes, is there a way to flush that out and start over? When doing a 'refresh' it's not finding anything new, so it thinks it's all there and ok.... I'm guessing...

[Tiki]

Originally Posted by stuckless

In addition to Core Metadata, there is Custom Metadata. Custom Metadata is the metadata defined in the custom_metadata_properties property. Sage provides a Read/Write interface for these properties, so updating them is quite trivial. While they are stored in the wiz.bin, I tend to not consider it an act of updating the wiz.bin, since, there is no risk of corrupting the wiz.bin core metadata by setting these fields. SeasonNumber, EpisodeNumber, MediaTitle, and MediaType are all stored in the Custom Metadata fields. As a side note, these fields MUST be defined in the custom_metadata_properties or else sage will simply discard them.

Sean,

I have not seen the cross link problem yet with BMT4.0, but haven't done much testing that would be likely to uncover it. I did, however, see the problem several times with the previous release (3.1).

In my experience, it appeared to be the custom metadata that was getting corrupted, not the core data. Here is what I found:

1. If I did an on-demand scan from the STVi plugin on a show that wasn't an actual recording (eg: on something from the program guide or an upcoming airing that had not been recorded yet), I never had a problem.

2. If I did an on-demand scan on a Sage Recording (something that would have an actual mediafile object and not just an airing object), and then I later deleted that mediafile, the crosslink problem would happen every time (I only tried a few times, and then stopped out of fear of corrupting my wiz.bin).

3. I wonder if this is actually a bug with the way Wiz.bin handles the custom metadata. Perhaps when a show is deleted the custom metadata is left behind in Wiz.bin and randomly linked to some other mediafile object?

[stuckless]

Here's the next beta

Fixed
1. Fixed bug in tvdb where search results contained year in brackets, which could cause potential hits to fail
2. Added ScrollBars to Preferences Panel, and Status Panel
3. Shortened the Search Result dialog to fit within a smaller window (short term fix)
4. Added Language support to tvdb (configure -> Metadata Tools -> TheTVDB Configuration -> Language
5. Fixed bug in setLastModified code
6. Added more debug data to the debug panel
7. Verified that cached url does work correctly. Adjusted the default settings to 1800 (30mins). (Configure -> Metadata Tools -> Url Settings -> Cache Expiry)
8. Adjusted the 'touch' value to 10seconds because SageTV doesn't appear to detect changes unless the change is more than a couple of seconds.
9. Added more debug logging for various issues

Still no progress on the missing first letter issue and the duplicate metadata issue.

[stuckless]

@tiki - i'll give that a shot and see if I have make this happen.

@bikesquid - it almost seems as if your install is not quite right. I know that you have a very large collection... and I don't know if that is causing some issues or not. When you click on "Sage Imports" does it just show nothing at all? Does the spinning circle start and then stop spinning? If you have DVDs, does the DVD View show anything? You should be able to start over by removing the imports and then adding them back in. You can leave the Recordings, but maybe remove the bulk of your other imports and then start by adding in a small import.