SageTV Community  

Go Back   SageTV Community > SageTV Development and Customizations > SageTV v7 Customizations > Batch Metadata Tools

Notices

Batch Metadata Tools This forums is for discussing the user-created Batch Metadata Tools for SageTV.

Reply
 
Thread Tools Search this Thread Display Modes
  #21  
Old 08-09-2010, 07:19 PM
cmaloney cmaloney is offline
Sage User
 
Join Date: Feb 2009
Posts: 22
Oh, the movies in question are .ISO files from my dvd collection. After a dvd gets linked I rename it to something else (ie the movie name and year) and Sagetv picks it up on the next manual scan. If I manually use the imdb search via the sagetv application it works all the time, however the imdb search option doesn't allow me to save the information if I manually specify the title for the movie so I need something a little more robust. BMT is perfect for this, other than the cross-link file problem. Once I use BMT, if I dont restart the server in between edits, two successive edits create a link. It is link the information isn't being properly cleared out between edits and it is reusing an id from the last movie or something.
Reply With Quote
  #22  
Old 08-09-2010, 09:07 PM
stuckless's Avatar
stuckless stuckless is offline
SageTVaholic
 
Join Date: Oct 2007
Location: London, Ontario, Canada
Posts: 9,580
Quote:
Originally Posted by cmaloney View Post
Oh, the movies in question are .ISO files from my dvd collection. After a dvd gets linked I rename it to something else (ie the movie name and year) and Sagetv picks it up on the next manual scan. If I manually use the imdb search via the sagetv application it works all the time, however the imdb search option doesn't allow me to save the information if I manually specify the title for the movie so I need something a little more robust. BMT is perfect for this, other than the cross-link file problem. Once I use BMT, if I dont restart the server in between edits, two successive edits create a link. It is link the information isn't being properly cleared out between edits and it is reusing an id from the last movie or something.
Maybe this is still related to the bug that was fixed in 7.0.13... I can see this being a bmt issue, if the files are not linked, but if you are experiencing cases where when you change data in one file, and those changes show up in another file's metadata, then I'd also look at submitting a core issue, since bmt cannot do that.

But, if this is a case where you are saving the metadata and it gets stored incorrectly, but does not affect other media items, then it's probably a bmt issue.

BMT does use a new set of objects for each search, and they are never re-used.
Reply With Quote
  #23  
Old 08-10-2010, 05:13 AM
skyeclad skyeclad is offline
Sage Aficionado
 
Join Date: Nov 2007
Location: NY DMA
Posts: 441
Exclamation

For me, I'm seeing newly scanned items turning all into the same Caprica TV episode across multiple new items. Example, I'm transcoding a movie now and in the video conversion area it says that I'm transcoding Caprica instead. Box art seems fine but the title, description and actors are starting to become a mix of a different movie and Caprica.

So, in my case, it's just getting worse as new imports are coming in. I'm considering removing BMT to stop corrupting the database. Is there anything I can do to sort this out?
Reply With Quote
  #24  
Old 08-10-2010, 05:29 AM
stuckless's Avatar
stuckless stuckless is offline
SageTVaholic
 
Join Date: Oct 2007
Location: London, Ontario, Canada
Posts: 9,580
Quote:
Originally Posted by skyeclad View Post
For me, I'm seeing newly scanned items turning all into the same Caprica TV episode across multiple new items. Example, I'm transcoding a movie now and in the video conversion area it says that I'm transcoding Caprica instead. Box art seems fine but the title, description and actors are starting to become a mix of a different movie and Caprica.

So, in my case, it's just getting worse as new imports are coming in. I'm considering removing BMT to stop corrupting the database. Is there anything I can do to sort this out?
Personally, I'd remove bmt, or at least disable the automatic metadata fetching... but I'm not sure this is a bmt issue.
Reply With Quote
  #25  
Old 08-10-2010, 08:28 AM
skyeclad skyeclad is offline
Sage Aficionado
 
Join Date: Nov 2007
Location: NY DMA
Posts: 441
Quote:
Originally Posted by stuckless View Post
Personally, I'd remove bmt, or at least disable the automatic metadata fetching... but I'm not sure this is a bmt issue.
Based on the issue, I think I have no other choice. I will remove it but is there anything that you would recommend I try to fix the issue. Remove and reinstall? Delete any files? Check any logs? Submit a bug report?

I did notice that the properties files looked ok so I wasn't sure if the bad metadata was being stored in the Wiz.bin or maybe the files themselves? The other weird thing was that the metadata imported was all mixed up, i.e. The images from the correct file, the description and plot from Caprica, the actors from There Will Be Blood! All files that I had touched recently though through BMT Web UI.
Reply With Quote
  #26  
Old 08-10-2010, 11:37 AM
stuckless's Avatar
stuckless stuckless is offline
SageTVaholic
 
Join Date: Oct 2007
Location: London, Ontario, Canada
Posts: 9,580
Quote:
Originally Posted by skyeclad View Post
Based on the issue, I think I have no other choice. I will remove it but is there anything that you would recommend I try to fix the issue. Remove and reinstall? Delete any files? Check any logs? Submit a bug report?

I did notice that the properties files looked ok so I wasn't sure if the bad metadata was being stored in the Wiz.bin or maybe the files themselves? The other weird thing was that the metadata imported was all mixed up, i.e. The images from the correct file, the description and plot from Caprica, the actors from There Will Be Blood! All files that I had touched recently though through BMT Web UI.
In a few days, I'm going to try an put out another bmt release with some extra logging, mainly so that I can verify that BMT is actually handling the files.

Something to keep in mind is that IF you have a .properties with a media item, then bmt should not try to update the metadata for that item, since by default bmt will not overwrite metadata for existing items.

If you are updating from the web ui, then bmt does not touch the .properties, but rather it will write directly to the wiz.bin using the sage apis. Actually bmt doesn't directly write to .properties unless you are running in a command line mode.
Reply With Quote
  #27  
Old 08-10-2010, 12:01 PM
skyeclad skyeclad is offline
Sage Aficionado
 
Join Date: Nov 2007
Location: NY DMA
Posts: 441
Quote:
Originally Posted by stuckless View Post
In a few days, I'm going to try an put out another bmt release with some extra logging, mainly so that I can verify that BMT is actually handling the files.

Something to keep in mind is that IF you have a .properties with a media item, then bmt should not try to update the metadata for that item, since by default bmt will not overwrite metadata for existing items.

If you are updating from the web ui, then bmt does not touch the .properties, but rather it will write directly to the wiz.bin using the sage apis. Actually bmt doesn't directly write to .properties unless you are running in a command line mode.
I recall that I cleaned up all my properties files(I had lots of them and wanted to delete the ones that I didn't need) and through the BMT UI I refreshed my metadata which created new files. Then I went through and selected media that didn't have any metadata, of which Caprica was one of them, and added some episode info to help BMT the relevant metadata. I also had a DVD- that needed help and that's when BMT added Caprica's metadata to the DVD. I thought it was odd, but no big deal since it was only one issue. Then it happened again, so I knew something was wrong.

Then last night I started to transcode a video file already imported and when I went into the File Conversion screen to check on the status, it said that it was transcoding Caprica even though the file was a BD rip of another movie! I guess the temporary file that was being created ended up picking up the metadata that it stuck within Sage or BMT. Not sure which one would do this.

Here's the thing, I'm pretty sure that the Properties files created look OK so I'm thinking that whatever is being posted to Wiz.bin is incorrect.
Reply With Quote
  #28  
Old 08-10-2010, 06:45 PM
skyeclad skyeclad is offline
Sage Aficionado
 
Join Date: Nov 2007
Location: NY DMA
Posts: 441
Well, there definitely cross-linked. I can update any one of the incorrect media files which then changes about 3 other items to the corrected entry of the first item. I also noticed that these items don't have properties files so I guess they are all cross linked in wiz.bin.

I reinstalled BMT but I don't know if any new items will get cross-linked. Sage issue?
Reply With Quote
  #29  
Old 08-10-2010, 07:44 PM
cmaloney cmaloney is offline
Sage User
 
Join Date: Feb 2009
Posts: 22
Quote:
Originally Posted by skyeclad View Post
Well, there definitely cross-linked. I can update any one of the incorrect media files which then changes about 3 other items to the corrected entry of the first item. I also noticed that these items don't have properties files so I guess they are all cross linked in wiz.bin.

I reinstalled BMT but I don't know if any new items will get cross-linked. Sage issue?
Yah, I am beginning to think it is a Sagetv problem as well. v7 is still beta, so hopefully it will be fixed before the official release :P Should we submit a bug report for this?
Reply With Quote
  #30  
Old 08-10-2010, 08:19 PM
stuckless's Avatar
stuckless stuckless is offline
SageTVaholic
 
Join Date: Oct 2007
Location: London, Ontario, Canada
Posts: 9,580
Quote:
Originally Posted by skyeclad View Post
Well, there definitely cross-linked. I can update any one of the incorrect media files which then changes about 3 other items to the corrected entry of the first item. I also noticed that these items don't have properties files so I guess they are all cross linked in wiz.bin.

I reinstalled BMT but I don't know if any new items will get cross-linked. Sage issue?
Quote:
Originally Posted by cmaloney View Post
Yah, I am beginning to think it is a Sagetv problem as well. v7 is still beta, so hopefully it will be fixed before the official release :P Should we submit a bug report for this?
I would submit a bug report... there maybe something about the types of files on which this is happening, etc... or maybe the files have embedded metadata, etc.
Reply With Quote
  #31  
Old 08-10-2010, 11:02 PM
Fuzzy's Avatar
Fuzzy Fuzzy is offline
SageTVaholic
 
Join Date: Sep 2005
Location: Jurupa Valley, CA
Posts: 9,957
Quote:
Originally Posted by skyeclad View Post
Well, there definitely cross-linked. I can update any one of the incorrect media files which then changes about 3 other items to the corrected entry of the first item. I also noticed that these items don't have properties files so I guess they are all cross linked in wiz.bin.

I reinstalled BMT but I don't know if any new items will get cross-linked. Sage issue?
I'm thinking you should post the info on the files. Most importantly, filename and ShowID.
__________________
Buy Fuzzy a beer! (Fuzzy likes beer)

unRAID Server: i7-6700, 32GB RAM, Dual 128GB SSD cache and 13TB pool, with SageTVv9, openDCT, Logitech Media Server and Plex Media Server each in Dockers.
Sources: HRHR Prime with Charter CableCard. HDHR-US for OTA.
Primary Client: HD-300 through XBoxOne in Living Room, Samsung HLT-6189S
Other Clients: Mi Box in Master Bedroom, HD-200 in kids room
Reply With Quote
  #32  
Old 08-12-2010, 02:38 AM
jorton jorton is offline
Sage Icon
 
Join Date: Jul 2005
Location: Canada
Posts: 1,273
On a whim I tried to do a fresh install on my server and reimport all my movies using the My Movies Properties Files. New wiz.bin importing from the My Movies Properties files. As soon as I tried to update the files in BMT I started getting the cross linking again.

Next I removed the import directories and waited for awhile for things to remove themselves from the wiz.bin.

I used the My Movies clean up tool to remove all the properties files (originally generated from My Movies Colleciton Manager) from all the import folders.

Added back the import folders and waited for the refresh.

I am now in BMT updating metadata without the cross linking. I don't know why but it seems that Sagetv importing the My Movies generated properties files might be part of the problem, at least for me.

I have done about 20 files now and haven't had a problem.

I hope this isn't just a fluke...

Jeremy
Reply With Quote
  #33  
Old 08-12-2010, 11:35 AM
Fuzzy's Avatar
Fuzzy Fuzzy is offline
SageTVaholic
 
Join Date: Sep 2005
Location: Jurupa Valley, CA
Posts: 9,957
As mentioned, it was probably that the .properties files were sharing some ID numbers or something, screwing up the database. They were most likely generated in previous version, before the fix. ALL that bad data has to be purged to get rid of the problem.
__________________
Buy Fuzzy a beer! (Fuzzy likes beer)

unRAID Server: i7-6700, 32GB RAM, Dual 128GB SSD cache and 13TB pool, with SageTVv9, openDCT, Logitech Media Server and Plex Media Server each in Dockers.
Sources: HRHR Prime with Charter CableCard. HDHR-US for OTA.
Primary Client: HD-300 through XBoxOne in Living Room, Samsung HLT-6189S
Other Clients: Mi Box in Master Bedroom, HD-200 in kids room
Reply With Quote
  #34  
Old 08-12-2010, 04:10 PM
cmaloney cmaloney is offline
Sage User
 
Join Date: Feb 2009
Posts: 22
The issue affecting some people might be related to .properties files, but it definitely is not the case for me. Let me give you a concrete example of exactly what I am doing and what is happening.

1. I am running the latest beta release (7.0.14.236), and I remove all imported videos from sagetv, completely remove ALL .properties files, and restart sagetv.

2. The Wiz.bin file is now 565 KB, which presumably is the configuration data plus the metadata from the few songs/pictures that I have.

3. I import my movies that are stored on my NAS in the directory structure: Media/HD Movies/NAME_HERE/NAME_HERE.mkv and Media/SD Movies/NAME_HERE/NAME_HERE.ISO

4. I wait until sagetv is done importing the files and I have 174 HD movies and 74 SD movies. The Wiz.bin file is now 35,979 KB.

5. Using Metadata Tools WebUI, I click browse, DVD and it displays all my SD movies.

6. Blade II is being seen as Blade since the file name is Blade2.ISO, so I fix it and everything seems to update properly, I can click on Blade and Blade II and see different descriptions.

7. I click on my Futurama movie and attempt to fix it since BMT thought it was "Four Christmasas" somehow. After I click Save, Blade II's title is renamed to "Futurama: Into the Wild Green Yonder" and it appears right before the real "Futurama: Into the Wild Green Yonder" movie. When I click on the first "Futurama: Into the Wild Green Yonder" movie (Blade II), the information for Blade II is displayed correctly (including the Blade II title).

8. The issue is that the title is being recording in sagetv as "Futurama: Into the Wild Green Yonder" instead of Blade II, so the description is completely different when I try to view the movie data in the sagetv app. I have included a picture that should clearly show the results of this problem. This problem continues to happen every single time I edit more than 1 movie.

Once again, I do not have a single .properties file in either Blade 2 or futurama's directory and the files were completely removed and readded.

Any thoughts?
Attached Images
File Type: jpg problem.JPG (57.5 KB, 204 views)
Reply With Quote
  #35  
Old 08-12-2010, 05:18 PM
Narflex's Avatar
Narflex Narflex is offline
Sage
 
Join Date: Feb 2003
Location: Redondo Beach, CA
Posts: 6,308
I just looked at a Wiz.bin from someone with this issue and I'm fairly sure this is a bug in BMT. The problem was that the 'ExternalID' for the Show object linked to all the items was blank. So all of the cross-linked items were sharing the same Show object. It's probably a bug in BMT where the external ID is not getting generated properly.

I'm going to make a change for the next version so that it won't allow the external ID for a Show object to be set to the empty string....not that it's a real fix; but it may help prevent this from occurring.
__________________
Jeffrey Kardatzke
Google
Founder of SageTV
Reply With Quote
  #36  
Old 08-12-2010, 05:52 PM
stuckless's Avatar
stuckless stuckless is offline
SageTVaholic
 
Join Date: Oct 2007
Location: London, Ontario, Canada
Posts: 9,580
Quote:
Originally Posted by Narflex View Post
I just looked at a Wiz.bin from someone with this issue and I'm fairly sure this is a bug in BMT. The problem was that the 'ExternalID' for the Show object linked to all the items was blank. So all of the cross-linked items were sharing the same Show object. It's probably a bug in BMT where the external ID is not getting generated properly.

I'm going to make a change for the next version so that it won't allow the external ID for a Show object to be set to the empty string....not that it's a real fix; but it may help prevent this from occurring.
Thanks Jeff. At least we know "why" it's happening. From there I can certainly look in bmt to see if I'm doing something wrong. I prefill the external id field with the original value, and it's never set back, unless it changes. I don't actually change the externalid field myself, unless the user requests to import as tv, and in that case, i validate that the externalid is not in use (which is not what is happening in this case). I don't validate the externalid from the webui, so I should add that as well. I'm guessing that if a user were to manually change the externalid to that of another file, then they would become linked?

I'll add another check on writes to ensure that externalid is never null/empty, now that I know that having a null value causes an issue for sage. It makes sense that you add the null/empty check on your side as well, since bmt is not the only metadata tool that is using the new metadata apis, so it's safe to assume that other providers may incorrectly set that field as well.

Thanks for looking into this.
Reply With Quote
  #37  
Old 08-12-2010, 06:36 PM
Fuzzy's Avatar
Fuzzy Fuzzy is offline
SageTVaholic
 
Join Date: Sep 2005
Location: Jurupa Valley, CA
Posts: 9,957
thats why i was asking what peoples showid/externalid's were for the problem files... with no responses...
__________________
Buy Fuzzy a beer! (Fuzzy likes beer)

unRAID Server: i7-6700, 32GB RAM, Dual 128GB SSD cache and 13TB pool, with SageTVv9, openDCT, Logitech Media Server and Plex Media Server each in Dockers.
Sources: HRHR Prime with Charter CableCard. HDHR-US for OTA.
Primary Client: HD-300 through XBoxOne in Living Room, Samsung HLT-6189S
Other Clients: Mi Box in Master Bedroom, HD-200 in kids room
Reply With Quote
  #38  
Old 08-12-2010, 06:52 PM
stuckless's Avatar
stuckless stuckless is offline
SageTVaholic
 
Join Date: Oct 2007
Location: London, Ontario, Canada
Posts: 9,580
Quote:
Originally Posted by Fuzzy View Post
thats why i was asking what peoples showid/externalid's were for the problem files... with no responses...
Since no one responded, i just assumed that they were ok as well. But since Jeff confirmed the issue, I did find the issue in the bmt web ui. Basically it was blanking out the externalid field when you do a manual update. This only happens in the web ui for a manual update. It does not happen for the automatic updates.

I have the issue fixed now, and I'm just running some tests. So hopefully I get a build out tonight or tomorrow.

My apologies to everyone that screwed up their wiz.bin... thankfully, you made backups
Reply With Quote
  #39  
Old 08-12-2010, 09:21 PM
cmaloney cmaloney is offline
Sage User
 
Join Date: Feb 2009
Posts: 22
Quote:
Originally Posted by Fuzzy View Post
thats why i was asking what peoples showid/externalid's were for the problem files... with no responses...
I just read through all your posts and I don't see any request for externalid, just showid and you never made the request to me otherwise I would have responded to it.

Regardless, the good news is that ExternalId is definitely the problem (at least on my end). Whenever I would do a manual update, it would clear out the field and cause problems when it was saved. If I manually put the old ExternalId back into the field it will save properly and everything will work as expected. Thanks for finding the problem and hopefully the fix whenever you can!

I didn't make any backups of my Wiz.bin file, but it is beta software so bugs are to be expected :P
Reply With Quote
  #40  
Old 08-12-2010, 10:43 PM
Fuzzy's Avatar
Fuzzy Fuzzy is offline
SageTVaholic
 
Join Date: Sep 2005
Location: Jurupa Valley, CA
Posts: 9,957
Quote:
Originally Posted by cmaloney View Post
I just read through all your posts and I don't see any request for externalid, just showid and you never made the request to me otherwise I would have responded to it.

Regardless, the good news is that ExternalId is definitely the problem (at least on my end). Whenever I would do a manual update, it would clear out the field and cause problems when it was saved. If I manually put the old ExternalId back into the field it will save properly and everything will work as expected. Thanks for finding the problem and hopefully the fix whenever you can!

I didn't make any backups of my Wiz.bin file, but it is beta software so bugs are to be expected :P
ShowID and ExternalID are the same thing. ExternalID is what it is actually called in the back-end, but in the UI, it's called ShowID.
__________________
Buy Fuzzy a beer! (Fuzzy likes beer)

unRAID Server: i7-6700, 32GB RAM, Dual 128GB SSD cache and 13TB pool, with SageTVv9, openDCT, Logitech Media Server and Plex Media Server each in Dockers.
Sources: HRHR Prime with Charter CableCard. HDHR-US for OTA.
Primary Client: HD-300 through XBoxOne in Living Room, Samsung HLT-6189S
Other Clients: Mi Box in Master Bedroom, HD-200 in kids room
Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Metadata Tools 4.0 Beta Thread (aka BMT 4.0) stuckless Batch Metadata Tools 885 09-02-2013 08:03 AM
BMT: Read this if you don't see any BMT topics Opus4 SageTV Customizations 0 04-25-2010 09:17 PM
Is there a way of doing Metadata WITHOUT BMT plnewton SageTV Customizations 12 08-17-2009 06:53 PM
BMT Metadata Provider List? cowboyway SageMC Custom Interface 1 08-12-2009 05:54 AM
Multiple Sage Servers and cross-connecting Orthogonite General Discussion 3 03-12-2004 11:49 AM


All times are GMT -6. The time now is 08:15 AM.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2020, vBulletin Solutions Inc.
Copyright 2003-2005 SageTV, LLC. All rights reserved.