BMT data corruption for russian shows

[scanner95]

Hello,

Quite frequently I face a problem when populating metadata information for Russian TV shows (it may also occur in some other cases; not sure, ).
What happens is that I go to BMT Web interface, select a show, fill in all necessary details (name of the show, season, episode) and click "search". The show is located correctly; all information from TVDB is displayed in Russian; all looks fine. I save the metadata.
However, at later stage (sometimes immediately after I save; sometimes in a few hours), if I browse for the same show, it suddenly has corrupted information. In some cases, the name of the show is still correct, and only actors names are scrambled, in other cases, the name if the show is also corrupted.
I have attached two screenshots; one - of a correctly discovered Russian sitcom; the other one - how the same show looks after couple of hours.

Thanks,
Yuly

[stuckless]

Keep in mind that once bmt updates a show, it never updates it again, unless you manually do it yourself from the web ui.

So, if you are seeing where bmt saves the data... all is well (even for a few minutes) and then things get corrupted... then BMT is not doing this, since it never touches your metadata once you save it (unless you manually request it again).

I see a lot of complaints lately of people complaining the "bmt is wrecking my metadata several hours/days/months after I've updated it"... and this is not the case, since BMT does not re-update metadata after it has been saved.

That being said... There are 3 things that might be going on here...
1. BMT doesn't properly send down the unicode data for the russian character set, and as such, it gets corrupted during save, and you'll notice this immediately after saving the data, since once your save, bmt then reloads the saved data, and sends it back to the UI, so that you can see in real time, what the current state of your metadata looks like.

2. The sagetv apis for reading/writing metadata is not handling the unicode characters correctly.

3. Some other process is mucking with your metadata.


Do you have .properties files lying around for these shows? (they'll be in the same directory as the media files and named the same with a .properties extension). If so, then open them, and see if you see proper russian characters in those files? Or,if you have one lying aruond, then post it here, so that I can take a look at the file.

[scanner95]

Hello,

I have attached one such .properties file, for a show that yesterday I updated it and it looked fine in the BMT WebUI, and now it all garbage.
Since the show file name is in russian, the . properties is in russian too. I have added the .txt extension to it.
For me, the .properties file looks just fine. I also attached the screenshot of how it looks in the BMT.
yuly

[JREkiwi]

Originally Posted by scanner95

For me, the .properties file looks just fine. I also attached the screenshot of how it looks in the BMT.

Yuly, the properties file has the bad data in it. It is SageTV is that is writing that .properties file, not BMT, so you should do a bug submission on this problem to get them to have a look at it.

It seems that there was a recent change in SageTV so it now periodically does a refresh of metadata from the media file and the .properties file. When it does this, it is loading the bad data from the .properties file and overwriting the good data.

John

[stuckless]

@jrekiwi - thank John, that's my findings as well.

On a related note... It would appear that SageTV is not using the standard java apis for saving properties, but they are using the standard java apis for reading them. If you can read the russian text in your .properties then that is an issue, because .properties as basically Latin1 character encoded files (which doesn't suppport russian), and as a result all unicode character sets need to be specially encoded so that the java .properties reader can correctly build the data coming back in. Typically in a .properties file, you'll see a bunch of "\u#####" entries to represent the unicode character, which is what I would have expected to see in your code.

Since SageTV is doing its own .properties writer, they'll need to convert unicode characters into \u#### characters instead. If you are submitting a bug, you can pass along this URL, since it shows how one might go about "escaping" those characters.

[scanner95]

Hello all,

I do not have insides on the java encoding things and artifacts obviously. However, I did another test: I downloaded a russian movie, and made a fresh metadata update through BMT WebUI. I have attached the BMT-generated .properties file. It is readable and I can see proper russian fonts via Windows notepad, so, probably, there is more to it here...

[stuckless]

Originally Posted by scanner95

Hello all,

I do not have insides on the java encoding things and artifacts obviously. However, I did another test: I downloaded a russian movie, and made a fresh metadata update through BMT WebUI. I have attached the BMT-generated .properties file. It is readable and I can see proper russian fonts via Windows notepad, so, probably, there is more to it here...

The fact that it is human readable (in russian... i can't read it ) is enough to tell me the the file is incorrect.... as a russian encoded .properties file should not be human readable to anyone (even russians )

Keep in mind... BMT is NOT creating those .properties files, and as such, there is nothing I can do to resolve this.

[scanner95]

Originally Posted by stuckless

The fact that it is human readable (in russian... i can't read it ) is enough to tell me the the file is incorrect.... as a russian encoded .properties file should not be human readable to anyone (even russians )

Keep in mind... BMT is NOT creating those .properties files, and as such, there is nothing I can do to resolve this.

I'm afraid I'm lost here. I thought it's BMT who is initially creating those .properties files. Could you please explain how it really works then?
When I select a title from BMT UI - where does the information comes from (after the title was recognized)? Does it not come from the .properties file?

[stuckless]

Originally Posted by scanner95

I'm afraid I'm lost here. I thought it's BMT who is initially creating those .properties files. Could you please explain how it really works then?
When I select a title from BMT UI - where does the information comes from (after the title was recognized)? Does it not come from the .properties file?

.properties is a side effect of calling the sagetv tv apis for updating metadata. BMT writes directly to your database (called Wiz.bin) using the SageTV metadata apis. When this happens, sagetv writes an entry to the .properties file and record your change. This file is not used by bmt at all. When you view the metadata for an item in the bmt web ui, then bmt is reading the data directly from the Wiz.bin and NOT the .properties.

[stuckless]

I posted a bug with sagetv... it's fixed, and it should be in the 7.1.18 release.

[scanner95]

Thanks!

[stuckless]

Originally Posted by stuckless

I posted a bug with sagetv... it's fixed, and it should be in the 7.1.18 release.

Just so you know that you don't have to wait for 10 more releases... my comment should have stated version 7.1.8 not 7.1.18

[scanner95]

I figured; thanks again