Cross Linked Metadata Discussions (Any BMT Version)

[cmaloney]

Oh, the movies in question are .ISO files from my dvd collection. After a dvd gets linked I rename it to something else (ie the movie name and year) and Sagetv picks it up on the next manual scan. If I manually use the imdb search via the sagetv application it works all the time, however the imdb search option doesn't allow me to save the information if I manually specify the title for the movie so I need something a little more robust. BMT is perfect for this, other than the cross-link file problem. Once I use BMT, if I dont restart the server in between edits, two successive edits create a link. It is link the information isn't being properly cleared out between edits and it is reusing an id from the last movie or something.

[stuckless]

Originally Posted by cmaloney

Oh, the movies in question are .ISO files from my dvd collection. After a dvd gets linked I rename it to something else (ie the movie name and year) and Sagetv picks it up on the next manual scan. If I manually use the imdb search via the sagetv application it works all the time, however the imdb search option doesn't allow me to save the information if I manually specify the title for the movie so I need something a little more robust. BMT is perfect for this, other than the cross-link file problem. Once I use BMT, if I dont restart the server in between edits, two successive edits create a link. It is link the information isn't being properly cleared out between edits and it is reusing an id from the last movie or something.

Maybe this is still related to the bug that was fixed in 7.0.13... I can see this being a bmt issue, if the files are not linked, but if you are experiencing cases where when you change data in one file, and those changes show up in another file's metadata, then I'd also look at submitting a core issue, since bmt cannot do that.

But, if this is a case where you are saving the metadata and it gets stored incorrectly, but does not affect other media items, then it's probably a bmt issue.

BMT does use a new set of objects for each search, and they are never re-used.

[skyeclad]

For me, I'm seeing newly scanned items turning all into the same Caprica TV episode across multiple new items. Example, I'm transcoding a movie now and in the video conversion area it says that I'm transcoding Caprica instead. Box art seems fine but the title, description and actors are starting to become a mix of a different movie and Caprica.

So, in my case, it's just getting worse as new imports are coming in. I'm considering removing BMT to stop corrupting the database. Is there anything I can do to sort this out?

[stuckless]

Originally Posted by skyeclad

For me, I'm seeing newly scanned items turning all into the same Caprica TV episode across multiple new items. Example, I'm transcoding a movie now and in the video conversion area it says that I'm transcoding Caprica instead. Box art seems fine but the title, description and actors are starting to become a mix of a different movie and Caprica.

So, in my case, it's just getting worse as new imports are coming in. I'm considering removing BMT to stop corrupting the database. Is there anything I can do to sort this out?

Personally, I'd remove bmt, or at least disable the automatic metadata fetching... but I'm not sure this is a bmt issue.

[skyeclad]

Originally Posted by stuckless

Personally, I'd remove bmt, or at least disable the automatic metadata fetching... but I'm not sure this is a bmt issue.

Based on the issue, I think I have no other choice. I will remove it but is there anything that you would recommend I try to fix the issue. Remove and reinstall? Delete any files? Check any logs? Submit a bug report?

I did notice that the properties files looked ok so I wasn't sure if the bad metadata was being stored in the Wiz.bin or maybe the files themselves? The other weird thing was that the metadata imported was all mixed up, i.e. The images from the correct file, the description and plot from Caprica, the actors from There Will Be Blood! All files that I had touched recently though through BMT Web UI.

[stuckless]

Originally Posted by skyeclad

Based on the issue, I think I have no other choice. I will remove it but is there anything that you would recommend I try to fix the issue. Remove and reinstall? Delete any files? Check any logs? Submit a bug report?

I did notice that the properties files looked ok so I wasn't sure if the bad metadata was being stored in the Wiz.bin or maybe the files themselves? The other weird thing was that the metadata imported was all mixed up, i.e. The images from the correct file, the description and plot from Caprica, the actors from There Will Be Blood! All files that I had touched recently though through BMT Web UI.

In a few days, I'm going to try an put out another bmt release with some extra logging, mainly so that I can verify that BMT is actually handling the files.

Something to keep in mind is that IF you have a .properties with a media item, then bmt should not try to update the metadata for that item, since by default bmt will not overwrite metadata for existing items.

If you are updating from the web ui, then bmt does not touch the .properties, but rather it will write directly to the wiz.bin using the sage apis. Actually bmt doesn't directly write to .properties unless you are running in a command line mode.

[skyeclad]

Originally Posted by stuckless

In a few days, I'm going to try an put out another bmt release with some extra logging, mainly so that I can verify that BMT is actually handling the files.

Something to keep in mind is that IF you have a .properties with a media item, then bmt should not try to update the metadata for that item, since by default bmt will not overwrite metadata for existing items.

If you are updating from the web ui, then bmt does not touch the .properties, but rather it will write directly to the wiz.bin using the sage apis. Actually bmt doesn't directly write to .properties unless you are running in a command line mode.

I recall that I cleaned up all my properties files(I had lots of them and wanted to delete the ones that I didn't need) and through the BMT UI I refreshed my metadata which created new files. Then I went through and selected media that didn't have any metadata, of which Caprica was one of them, and added some episode info to help BMT the relevant metadata. I also had a DVD- that needed help and that's when BMT added Caprica's metadata to the DVD. I thought it was odd, but no big deal since it was only one issue. Then it happened again, so I knew something was wrong.

Then last night I started to transcode a video file already imported and when I went into the File Conversion screen to check on the status, it said that it was transcoding Caprica even though the file was a BD rip of another movie! I guess the temporary file that was being created ended up picking up the metadata that it stuck within Sage or BMT. Not sure which one would do this.

Here's the thing, I'm pretty sure that the Properties files created look OK so I'm thinking that whatever is being posted to Wiz.bin is incorrect.

[skyeclad]

Well, there definitely cross-linked. I can update any one of the incorrect media files which then changes about 3 other items to the corrected entry of the first item. I also noticed that these items don't have properties files so I guess they are all cross linked in wiz.bin.

I reinstalled BMT but I don't know if any new items will get cross-linked. Sage issue?

[cmaloney]

Originally Posted by skyeclad

Well, there definitely cross-linked. I can update any one of the incorrect media files which then changes about 3 other items to the corrected entry of the first item. I also noticed that these items don't have properties files so I guess they are all cross linked in wiz.bin.

I reinstalled BMT but I don't know if any new items will get cross-linked. Sage issue?

Yah, I am beginning to think it is a Sagetv problem as well. v7 is still beta, so hopefully it will be fixed before the official release :P Should we submit a bug report for this?

[stuckless]

Originally Posted by skyeclad

Well, there definitely cross-linked. I can update any one of the incorrect media files which then changes about 3 other items to the corrected entry of the first item. I also noticed that these items don't have properties files so I guess they are all cross linked in wiz.bin.

I reinstalled BMT but I don't know if any new items will get cross-linked. Sage issue?

Originally Posted by cmaloney

Yah, I am beginning to think it is a Sagetv problem as well. v7 is still beta, so hopefully it will be fixed before the official release :P Should we submit a bug report for this?

I would submit a bug report... there maybe something about the types of files on which this is happening, etc... or maybe the files have embedded metadata, etc.

[Fuzzy]

Originally Posted by skyeclad

Well, there definitely cross-linked. I can update any one of the incorrect media files which then changes about 3 other items to the corrected entry of the first item. I also noticed that these items don't have properties files so I guess they are all cross linked in wiz.bin.

I reinstalled BMT but I don't know if any new items will get cross-linked. Sage issue?

I'm thinking you should post the info on the files. Most importantly, filename and ShowID.

[jorton]

On a whim I tried to do a fresh install on my server and reimport all my movies using the My Movies Properties Files. New wiz.bin importing from the My Movies Properties files. As soon as I tried to update the files in BMT I started getting the cross linking again.

Next I removed the import directories and waited for awhile for things to remove themselves from the wiz.bin.

I used the My Movies clean up tool to remove all the properties files (originally generated from My Movies Colleciton Manager) from all the import folders.

Added back the import folders and waited for the refresh.

I am now in BMT updating metadata without the cross linking. I don't know why but it seems that Sagetv importing the My Movies generated properties files might be part of the problem, at least for me.

I have done about 20 files now and haven't had a problem.

I hope this isn't just a fluke...

Jeremy

[Fuzzy]

As mentioned, it was probably that the .properties files were sharing some ID numbers or something, screwing up the database. They were most likely generated in previous version, before the fix. ALL that bad data has to be purged to get rid of the problem.

[cmaloney]

The issue affecting some people might be related to .properties files, but it definitely is not the case for me. Let me give you a concrete example of exactly what I am doing and what is happening.

1. I am running the latest beta release (7.0.14.236), and I remove all imported videos from sagetv, completely remove ALL .properties files, and restart sagetv.

2. The Wiz.bin file is now 565 KB, which presumably is the configuration data plus the metadata from the few songs/pictures that I have.

3. I import my movies that are stored on my NAS in the directory structure: Media/HD Movies/NAME_HERE/NAME_HERE.mkv and Media/SD Movies/NAME_HERE/NAME_HERE.ISO

4. I wait until sagetv is done importing the files and I have 174 HD movies and 74 SD movies. The Wiz.bin file is now 35,979 KB.

5. Using Metadata Tools WebUI, I click browse, DVD and it displays all my SD movies.

6. Blade II is being seen as Blade since the file name is Blade2.ISO, so I fix it and everything seems to update properly, I can click on Blade and Blade II and see different descriptions.

7. I click on my Futurama movie and attempt to fix it since BMT thought it was "Four Christmasas" somehow. After I click Save, Blade II's title is renamed to "Futurama: Into the Wild Green Yonder" and it appears right before the real "Futurama: Into the Wild Green Yonder" movie. When I click on the first "Futurama: Into the Wild Green Yonder" movie (Blade II), the information for Blade II is displayed correctly (including the Blade II title).

8. The issue is that the title is being recording in sagetv as "Futurama: Into the Wild Green Yonder" instead of Blade II, so the description is completely different when I try to view the movie data in the sagetv app. I have included a picture that should clearly show the results of this problem. This problem continues to happen every single time I edit more than 1 movie.

Once again, I do not have a single .properties file in either Blade 2 or futurama's directory and the files were completely removed and readded.

Any thoughts?

[Narflex]

I just looked at a Wiz.bin from someone with this issue and I'm fairly sure this is a bug in BMT. The problem was that the 'ExternalID' for the Show object linked to all the items was blank. So all of the cross-linked items were sharing the same Show object. It's probably a bug in BMT where the external ID is not getting generated properly.

I'm going to make a change for the next version so that it won't allow the external ID for a Show object to be set to the empty string....not that it's a real fix; but it may help prevent this from occurring.

[stuckless]

Originally Posted by Narflex

I just looked at a Wiz.bin from someone with this issue and I'm fairly sure this is a bug in BMT. The problem was that the 'ExternalID' for the Show object linked to all the items was blank. So all of the cross-linked items were sharing the same Show object. It's probably a bug in BMT where the external ID is not getting generated properly.

I'm going to make a change for the next version so that it won't allow the external ID for a Show object to be set to the empty string....not that it's a real fix; but it may help prevent this from occurring.

Thanks Jeff. At least we know "why" it's happening. From there I can certainly look in bmt to see if I'm doing something wrong. I prefill the external id field with the original value, and it's never set back, unless it changes. I don't actually change the externalid field myself, unless the user requests to import as tv, and in that case, i validate that the externalid is not in use (which is not what is happening in this case). I don't validate the externalid from the webui, so I should add that as well. I'm guessing that if a user were to manually change the externalid to that of another file, then they would become linked?

I'll add another check on writes to ensure that externalid is never null/empty, now that I know that having a null value causes an issue for sage. It makes sense that you add the null/empty check on your side as well, since bmt is not the only metadata tool that is using the new metadata apis, so it's safe to assume that other providers may incorrectly set that field as well.

Thanks for looking into this.

[Fuzzy]

thats why i was asking what peoples showid/externalid's were for the problem files... with no responses...

[stuckless]

Originally Posted by Fuzzy

thats why i was asking what peoples showid/externalid's were for the problem files... with no responses...

Since no one responded, i just assumed that they were ok as well. But since Jeff confirmed the issue, I did find the issue in the bmt web ui. Basically it was blanking out the externalid field when you do a manual update. This only happens in the web ui for a manual update. It does not happen for the automatic updates.

I have the issue fixed now, and I'm just running some tests. So hopefully I get a build out tonight or tomorrow.

My apologies to everyone that screwed up their wiz.bin... thankfully, you made backups

[cmaloney]

Originally Posted by Fuzzy

thats why i was asking what peoples showid/externalid's were for the problem files... with no responses...

I just read through all your posts and I don't see any request for externalid, just showid and you never made the request to me otherwise I would have responded to it.

Regardless, the good news is that ExternalId is definitely the problem (at least on my end). Whenever I would do a manual update, it would clear out the field and cause problems when it was saved. If I manually put the old ExternalId back into the field it will save properly and everything will work as expected. Thanks for finding the problem and hopefully the fix whenever you can!

I didn't make any backups of my Wiz.bin file, but it is beta software so bugs are to be expected :P

[Fuzzy]

Originally Posted by cmaloney

I just read through all your posts and I don't see any request for externalid, just showid and you never made the request to me otherwise I would have responded to it.

Regardless, the good news is that ExternalId is definitely the problem (at least on my end). Whenever I would do a manual update, it would clear out the field and cause problems when it was saved. If I manually put the old ExternalId back into the field it will save properly and everything will work as expected. Thanks for finding the problem and hopefully the fix whenever you can!

I didn't make any backups of my Wiz.bin file, but it is beta software so bugs are to be expected :P

ShowID and ExternalID are the same thing. ExternalID is what it is actually called in the back-end, but in the UI, it's called ShowID.