Questions about metadata and the pecking order...

[PLUCKYHD]

Okay I have some questions about metadata and the SageDB objects since I have never written to them much before.

I am more concerned with the pecking order which things get written.

Lets say a new file is found during the scan that contains a .properties file. I am going to assume that properties file info gets written first to the database. Then the even listener for mediafile added gets fired. That is where my worry comes in and maybe I shouldn't be worried, but if a user has BMT and say my program installed it seems to be it is going to be last one called that gets the info written. As if BMT is called first and then my listener files then my info is going to overwrite BMT's info. Is that correct? What stops both from trying to write at the same time and causing errors?

Maybe I am over worrying thinking this but I just now users are going to install both even if I tell them not to and then someone is going to complain I or BMT is breaking their metadata and wanting to make sure I understand the process correctly before I decide to change my ways and go this route.

cheers

[stuckless]

Originally Posted by PLUCKYHD

Okay I have some questions about metadata and the SageDB objects since I have never written to them much before.

I am more concerned with the pecking order which things get written.

Lets say a new file is found during the scan that contains a .properties file. I am going to assume that properties file info gets written first to the database. Then the even listener for mediafile added gets fired. That is where my worry comes in and maybe I shouldn't be worried, but if a user has BMT and say my program installed it seems to be it is going to be last one called that gets the info written. As if BMT is called first and then my listener files then my info is going to overwrite BMT's info. Is that correct? What stops both from trying to write at the same time and causing errors?

Maybe I am over worrying thinking this but I just now users are going to install both even if I tell them not to and then someone is going to complain I or BMT is breaking their metadata and wanting to make sure I understand the process correctly before I decide to change my ways and go this route.

cheers

I've been meaning to start this discussion for some time now, so the timing is good. With BMT/Phoenix, Ortus, and MyMovies all contributing metadata and fanart, I think we need to cordially come up with some nice ways of working together.

Currently BMT is written to NEVER update metadata, automatically, if the the mediafile already has an existing MediaTitle field, or in the case of TV if it already has a SeaonNumber or EpisodeNumber. I made this change to BMT some time ago, in preparation for the fact the Ortus and MM was on the horizon. I didn't want BMT overwriting metadata if people were getting it from another location. You can also turn off the automatic lookups as well, so that you can still have the bmt web ui, but it will never automatically update metadata.

There are some drawbacks to this approach...

First, it assumes that all metadata providers use the same custom fields. Initially for phoenix fanart, we had an open discussion around which fields we think we should all agree upon, but perhaps we need to be a little more transparent and create "known" set of custom fields.

Second, bmt only checks a couple of fields to see if it should fetch metadata automatically. So if Ortus or MyMovies populates the metadata, but doesn't actually download the fanart, for some reason, then BMT will never actually download any fanart either.

Third, I can't be sure that MyMovies or Ortus doesn't overwrite BMT metadata automatically. And there is no way to debug if another plugin updates the metadata. While BMT writes a log entry to the phoenix-metadata.log stating that it updated the metadata, but there's no indication that onother plugin may have updated it as well.

Moving forward, since there are going to be several players, I think it would be wise to come up with a "standard" way of allowing the users to choose which metadata/fanart providers to use... or to NEVER enable automatic lookups by default, and only enable them if the user goes into a plugin and enables it, etc, etc, etc.

[PLUCKYHD]

Glad I am not the only one worried about this and it was initially and currently SMM doesn't write to the Database (outside of if it is creating property files). I didn't want to overwrite or mess with BMT.

I like your method but see issues with it as well as if BMT gets called first it won't find metadata so it will do it's lookup and then if MyMovies is checking the same thing it won't do anything.

and of course there is the issue of fields that are written to and respected by all developers. I know some/most of the BMT fields it writes to and of course i would need several additional ones that I create.

I have one suggestions in thinking out loud. What if I or someone else creates a simple STVi (if even necessary). The allows editing of a 2 properties such as

DefaultMetadataFetcherForMovies=
DefaultMetadataFetcherForTV=

and me and you and hopefully everyone else agree to check that property for our name. If our name exist we do our thing if it doesn't our automatic writers to the db are disabled. Of course any new players in the game and Ortus would have to respect the same two server properties when implementing their plugins and listeners.

It would be fairly easy to keep the stvi (if necessary) updated with the choices currently available as long as every dev gave me the name/ value they needed in the property to have it as an option.

My thought is this would give all users regardless of their metadata installations a choice to switch back and fourth between the fetchers they have at will/demand. Without interaction of each individual plugin except for us checking the current state of the above mentioned properties. It would also allow for control seperately of TV and movie scrapers (I still wish Sagetv would add a TV import type so us scrapers could easily determine what is what on imported movies and tv without having to guess).

thoughts?

[Tiki]

Why not devise a custom metadata field that indicates what plugin updated the data last? Then any metadata plugins could check this field and know not to automatically overwrite data for an object whose metadata had been set by some other plugin.

[PLUCKYHD]

Originally Posted by Tiki

Why not devise a custom metadata field that indicates what plugin updated the data last? Then any metadata plugins could check this field and know not to automatically overwrite data for an object whose metadata had been set by some other plugin.

That wouldn't work when people want to change providers very well.

[Fuzzy]

Originally Posted by PLUCKYHD

That wouldn't work when people want to change providers very well.

What complications do you see in this method (not that I'm saying it WOULD work - I'm just not seeing the problem). If a mediafile is added, and say, a the time, BMT is the preferred source. The 'MetadataSource' property - originally null - is set with something like "BMT:TVDB:71398". No other 'friendly' plugins would modify metadata once that is populated - unless specifically asked to do so, via a manual update. Simply changing a preferred provider shouldn't trigger a re-hash of all existing data. (I'd also like to point out that having this somewhat easily readable 'MetadataSource' property might be useful for the user in general to figure out where the data is coming from).

[PLUCKYHD]

Yeah I guess that does make since but then still requires a property somewhere that providers check to make sure they are default. As if they aren't doing that and a user thinks they are default and it isn't writing because that property on the mediafile already exist....in otherwords if I was set as default provider but another provider wasn't checking the default property and wrote before mine and my check against the file reveals metadata already written by another provider so I don't write anything the user will wonder why.

[Fuzzy]

Perhaps, instead of making a simple STVi that would be used for setting a property that all plugins would have to check - what if instead a 'metadata clearing house' general plugin was developed that would handle the arbitration. Of course, the plugins would still then have to utilize that - but it would probably provide a bit more control over how the plugins are called (could not only offer a 'preferred' option - but perhaps a preferred order).

So the ClearingHouse is what actually responds to the MediaFileImported event - then fires off whatever specific metadata gathering doodads are in it's list. The individual behavior of those doodads should be defined - but I'd say by default, it shouldn't overwrite any already populated metadata field. this would allow a higher priority doodad to fill in what it can - but more specific information could be added by lower priority plugins. (MyMovies might be used first - and would get a majority of information - but then BMT could also be used, which could populate the things MyMovies might miss, fanart, etc.).

[PLUCKYHD]

Originally Posted by Fuzzy

Perhaps, instead of making a simple STVi that would be used for setting a property that all plugins would have to check - what if instead a 'metadata clearing house' general plugin was developed that would handle the arbitration. Of course, the plugins would still then have to utilize that - but it would probably provide a bit more control over how the plugins are called (could not only offer a 'preferred' option - but perhaps a preferred order).

So the ClearingHouse is what actually responds to the MediaFileImported event - then fires off whatever specific metadata gathering doodads are in it's list. The individual behavior of those doodads should be defined - but I'd say by default, it shouldn't overwrite any already populated metadata field. this would allow a higher priority doodad to fill in what it can - but more specific information could be added by lower priority plugins. (MyMovies might be used first - and would get a majority of information - but then BMT could also be used, which could populate the things MyMovies might miss, fanart, etc.).

hmm I think that might work although multiple providers (ie one filling in where another misses out) is a bit much for what I was thinking. I can see having different scrapers for tv vs movies but in all honesty I can't see having multiple scrapers installed and expecting one to fill in where the other leaves off. That is a nice idea but would be very complicated on the "clearing house" plugin to decipher all of that and rely on the providers to cohere to allot more code. Personally I think that is overkill being that we are all doing this for hobby or whatever reason you want to give it. I think I simply property check/stvi is more than sufficient and even then you might have a provider that doesn't adhere to even that as there is nothing forcing them to.

[stuckless]

Personally... I think that Tiki's and Fuzzy's idea (which are the same), of creating a custom metadata field on the mediaitem, is the simplest approach. Even if we were to create a "clearing house" type of app, then there is still no guarantee that another plugin will adhere to the rules.

In terms of what is stored, I'd prefer to go with Tiki's approach, and only store the plugin id of what updated the metadata, instead of Fuzzy's approach of also adding in the provider id and provider's data id, mainly because I already store those against the media item as well in 2 different fields (MetadataProviderID, MetadataProviderDataId).

In terms of the process flow, bmt's automatic scraper would simply check the MetadataSource field, and if it is null, or has bmt, then it will automatically update the mediafile.

BUT, from the web ui, if you fired off a scan or manual update, then I would NOT check that field, since there is real user driving the application.

I could also easily show the MetadataSource in the details for a media item.

Of course, during an automatic update, there is no way that bmt will handle the media item first, or last. But if a user ONLY wants to have SMM to handle the metadata automatically, then they can already turn off the automatic scraping in bmt, but still use the web ui, if they want to do manual updates.

We don't have that many plugins updating metadata automatically, so I don't really see this as being that huge of an issue. And, if other plugins come along and start duplicating the features of bmt in that they also pull metadata form tvdb, imdb, tmdb, dvdprofiler, etc, etc, then maybe I'll change bmt to use those... it'll be less work for me.

[PLUCKYHD]

My only fear on that route is the users themselves (yes not allot of faith sometimes) not disabling one plugin scraping when enabling another plugin. I mean wouldn't you want to fallback to writing if your plugin was enabled to even if another plugin had already written it?

Either way I am open and a bit of from actually writing to the db and still not 100% convinced I am going to do it (hit some caveats with it in testing), but still to good to have a plan of action.

[stuckless]

Originally Posted by PLUCKYHD

I mean wouldn't you want to fallback to writing if your plugin was enabled to even if another plugin had already written it?

No. Actually I go out of my way to find excuses to NOT update the metadata on an automatic scan. ie, if detect that there is an existing .properties, i do nothing. If I detect that the the mediafile already has a MediaTitle, then I skip it. Bottom line, I don't want BMT fetching metadata, automatically, for stuff that could already be updated. The "MediaTitle" field, today, acts the the proposed "MediaSource" fields, in that if there's a value in that field, then I skip the file.

Just because I work this way, I don't expect that your plugin should do the same. if people complain that BMT is not getting the right metadata, and they have other scrapers installed, then I'll probably have people uninstall the other plugins before submitting bugs.

I think a "good faith" solution is the best we can hope for in terms of fetching metadata. I'm not against Fuzzy's broker solution, but BMT is already a broker. ie, you can have different plugins, you can order the plugins, you can have different plugins for tv, movies, and music, etc, etc. I don't see the point in implementing another broker (from my point of view).

[PLUCKYHD]

Originally Posted by stuckless

No. Actually I go out of my way to find excuses to NOT update the metadata on an automatic scan. ie, if detect that there is an existing .properties, i do nothing. If I detect that the the mediafile already has a MediaTitle, then I skip it. Bottom line, I don't want BMT fetching metadata, automatically, for stuff that could already be updated. The "MediaTitle" field, today, acts the the proposed "MediaSource" fields, in that if there's a value in that field, then I skip the file.

I gotcha and understand your reasoning for doing it that way.

[Fuzzy]

Originally Posted by stuckless

I think a "good faith" solution is the best we can hope for in terms of fetching metadata. I'm not against Fuzzy's broker solution, but BMT is already a broker. ie, you can have different plugins, you can order the plugins, you can have different plugins for tv, movies, and music, etc, etc. I don't see the point in implementing another broker (from my point of view).

I agree, and I think BMT's methodology for it works quite well. The limitation is that broker doens't extend to the other potential metadata sources out there. Perhaps Plucky should work on implementing his metadata source as a BMT source plugin? I don't know if it would give the capabilities required. I'm certainly not knowledgable in BMT's scraper api, or plucky's needs - but that's never stopped me from chiming in before.. ;-)

Maybe BMT 5.0 could become the SageTV Advanced Metadata Retreival System of the future, and would be sources would be built as addons to that system...

[PLUCKYHD]

Originally Posted by Fuzzy

I agree, and I think BMT's methodology for it works quite well. The limitation is that broker doens't extend to the other potential metadata sources out there. Perhaps Plucky should work on implementing his metadata source as a BMT source plugin? I don't know if it would give the capabilities required. I'm certainly not knowledgable in BMT's scraper api, or plucky's needs - but that's never stopped me from chiming in before.. ;-)

Maybe BMT 5.0 could become the SageTV Advanced Metadata Retreival System of the future, and would be sources would be built as addons to that system...

Yeah great create more work for Stuckless just want he wants....


As for me I probably will still not use the default DB as I have been testing heavily and it really just doesn't do enough to fit the two providers am using with support for offline videos. I could make dummy files and then attach metadata to them but that creates the issue of constantly having to check the external database to make sure no new offline videos exist. As well as it creates the issue of keeping 2 database's in synch the sage db and the external ones. I have been giving the default db a fair shot in the last couple of weeks but I am afraid it will just create more issues/problems for me then it will fix.