Wiz.bin problems and getting recordings treated as recordings

[fresnoboy]

Hi everyone. I'll spare you the gory details, but I was trying upgrade my Linux V9 Sage setup to the latest version of Sage, and ran into issues that corrupted the Wiz.bin. I hadn't copied the prior one over, so have been trying to recover from older versions.

The good news is that I had the version from when I did the V9 update (and went from windows to Linux at the same time), and discovered a ton of old metadata and file references that were somehow not carried over when I did the original install.

Note the system is quite large - the old Wiz.bin was pushing 900MB and has 20K recordings and about 39TB of storage consumed, and that's just for Sage recordings since about 2009, and doesn't include rips, photos, etc...

Things came back pretty well (though needed to up the Java heap storage), but there are some weird issues with metadata. BMT shows 15K shows in the category of "No Metadata" when I browse that section, but a ton of those show up properly in Sage.

For example, an old DIY Network show (10 things you must know) that has no TVDB entry, has about 7 recordings that actually show up when I select the show in the Sage UI, but Sage knows about 10 more (SageMC says it has 17 recordings, but only 8 show up as playable). The others are there, and show up in BMT when I look for the show, but Sage doesn't show them as playable.

Also, Adam-12 for example, shows 174 recordings in 7 seasons when I look at the show in BMT (under the TV seasons view), but only 92 show up when I select the entry in the Sage UI as playable.

A ton of shows (I suspect the missing ones) show up in the videos category on Sage and not as recordings, so I suspect that is where Sage is displaying them, so how do I get them reintegrated with the rest of the recordings?

I have set advanced_video_file_recovery=true and started it with that, but that didn't seem to find these issues. The "missing shows" show up in BMT as Sage recordings, but still are marked as missing metadata, so something is wrong, but I am not sure of what it is.

Anyone have any ideas as to where to start?

Thanks,
mike

[Narflex]

The first thing I'd do is look at the log file when it's re-importing all of the recordings. I'm assuming you changed the file paths going from Windows to Linux...and advanced_video_file_recovery is the way to do it right. If the file name exists in the database...then it should just re-use all the old metadata. The logs will rollover and be huge since you have so many recordings...but if you increase the logfile_rollover_size and num_logfiles_to_keep in Sage.properties then it'll retain more information.

You do need to make sure the recordings are in configured recording paths or import directories when you run SageTV for the first time with the old database, otherwise those entries will get cleared to some degree and it makes recovery more difficult (it still has 2 more ways to recover data even if that happens though, using the Airing IDs or the embedded metadata in the recordings).

Feel free to post relevant sections of the log here from examples you find where it didn't go as expected and I can tell you what I think is going on.

[fresnoboy]

Jeff, thanks for the reply.

I think I have found the problem. When I go into BMT and look at a recording that shows up in Sage as a recording, the file name in the metadata is something like:

/media/media2/SageTV/Adam12-S01E01-TheImpossibleMission-50092439-0.avi

For the recordings that DON'T show up, the file name looks something like this:

/opt/sagetv/server/\\tbstore2\Media2\SageTV\Adam12-S01E11-TheBoaConstrictor-50241685-0.avi

When I converted from Sage V7 to Sage V9, I moved from windows to Linux. All the files are stored in a NAS, and mounted on the windows machine using SMB, but under Linux using NFS, and just have the directories show up as normal filesystem access.

What I don't understand is why advanced file recovery didn't pick the latter up and map the metadata properly. BMT shows the recording, but Sage doesn't pick the missing video up as something that is playable.

Any idea as to how to fix this?

Thx
Mike

[fresnoboy]

Ok, pretty weird. The files that BMT shows as having that weird SMB path in them do show up as videos detected by sage. They don't have that SMB stuff in the real names they have on the disk, but it's even weirder. They have names like:

Adam12S02E01Exactly100Yards504672280-Adam12S02E01Exactly100Yards504672280-56735842-0.avi

instead of

Adam12-S02E01-Exactly100Yards-50467228-0.avi

like they are supposed to have.

If I manually rename them so they are supposed to be in the proper format, then then restart sage with advanced file recovery set, sage picks them up just fine and adds them to the list of shows for that series.

Also, at least for some of them, the shows are set to do automatic transcoding to MPEG4. So they went through some autocompression process, but the names got garbled in this odd way with -'s removed.

After this is done, BMT shows them with proper names too.

Some questions: Where did BMT get this odd name for these recordings with the SMB name embedded in them?

What changed the names from their original name to this odd doubled name with all the -'s removed?

Was that related to the autocompression processs?


Lastly, and maybe more importantly, does someone know of a utility I can use to go rename all these files, or should I just bite the bullet and try and write a script?

Thx
mike

[Narflex]

I can't think of any reason SageTV would have changed these...and especially if they worked before the move to Linux, then the paths should have been correct before. What's also odd is they have a different airing ID appended to them as well...and that really makes no sense to me at all. I'd mainly suspect some kind of plugin did this, but I have no idea.

And I don't think there'd be any normal utility available to do this kind of renaming...you'd need to write some kind of script to do it...which is going to be somewhat tricky because detecting the boundaries where the '-' characters go isn't going to be all that simple (especially if a show/episode name starts/ends with a number).