WIZ.BIN is insane; solved

[src666]

I've been wondering why my WIZ.BIN file is over 168MB big. Granted, I've been using SageTV for 5 years, but that seems excessive. Playing around with Neilm's web interface showed the problem.

It appears that once you watch a show/movie/whatever, every single occurrence of that show that appears in the guide, ever, is stored as a watched episode.

For instance, when browsing the watched list, I saw several _hundred_ watched instances of the movie "27 Dresses". While I might grant you that my wife watched it more than once, she certainly did NOT watch it a couple hundred times. It certainly looks like once she watched it, then from then on whenever it came up in the guide Sage squirreled the information away. That's just ridiculous.

This same pattern is evident throughout the list. Can anyone explain a logical reason for this behavior, because it is evading me? I just know that a 168MB WIZ.BIN can't be healthy for the system.

[Opus4]

That doesn't sound like anything I've seen SageTV do -- are you using some sort of customization that is affecting the Watched status like that?

- Andy

[src666]

The only things I have installed are SageMC, Jetty, Neilm's Jetty web interface and Metadata Tools.

But Neilm's web inteface has only been installed a couple hours, Jetty/Metadata tools have only been installed for a month. I started out with SageTV 3 way back when, and some of this stuff goes back pretty far. SageMC has been installed off and on for about 3 years, but spent quite a bit of time uninstalled due to WAF issues around the release of SageTV 6.

As an example of what I am seeing, the earliest recording of "A Man Apart" was August 15, 2005, but it is showing several years of "watched" data for that movie. It does appear that the behavior started in early 2008, since that seems to be the time that most of the duplicated information begins.

That was during the time that SageMC wasn't installed, so I don't think I can blame it, but I'm open to ideas.

[Opus4]

I'm only guessing & don't know whether this could be a cause, but do you have wizard/disable_maintenance=true in sage.properties?

- Andy

[Fuzzy]

Things like movies, if the guide data is fully correct, will share watched status. They So once you've watched 27 dresses, it will always show as watched whenver it airs... this, however, is not like it is storing that you actually watched every one of those airings. the same is true for episodes of shows. If you've seen an episode of a show, and it was marked watched, reruns of that show will also show as watched. this IS normal behavior.

[src666]

Originally Posted by Opus4

I'm only guessing & don't know whether this could be a cause, but do you have wizard/disable_maintenance=true in sage.properties?

- Andy

Yes, I do, and I suspect that may be the problem. I probably set that when I moved between boxes once (it has happened several times by now), and never changed it back. I'll turn it off and see what happens.

When does the maintenance run? Just wondering how long I should give it before I check for results.

[src666]

Originally Posted by Fuzzy

Things like movies, if the guide data is fully correct, will share watched status. They So once you've watched 27 dresses, it will always show as watched whenver it airs... this, however, is not like it is storing that you actually watched every one of those airings. the same is true for episodes of shows. If you've seen an episode of a show, and it was marked watched, reruns of that show will also show as watched. this IS normal behavior.

The problem isn't that it was showing as watched for future occurrences in the guide. The problem is that it was showing as a "watched" item hundreds of times, over several years of past airings (see the example screenshot). This was _not_ accurate, since we did not watch it every time it came on for the past 2 years.

I'm hoping that changing the disable_maintenance flag will fix the issue.

[src666]

Just FYI, the change appears to be working. I'm already down from over 50,000 watched airings to about 49,000. It does appear to be cleaning house now.

Thanks, Opus!

[matt91]

Interesting. I'm curious what size it drops to after a few hours of cleaning up.

[src666]

Originally Posted by matt91

Interesting. I'm curious what size it drops to after a few hours of cleaning up.

Actually, it seems to be growing. Down to 45,500 showings, but the WIZ.BIN is up to 174MB. But I'm not reading anything into that until it's done.

[src666]

Final result, 23MB Wiz.bin. It got up to 320MB during the process, and took a couple of hours, but all's well in the end.

The object lesson is make sure you clean up after yourself when you make major changes to the system. And don't leave the disable_maintenance set to true for 2 years.

Thanks again for pointing me in the right direction, Opus.

[Fuzzy]

just out of curiosity, what prompted you to ever SET that property? I can't thin of a time when I've ever needed to do so.

[src666]

Originally Posted by Fuzzy

just out of curiosity, what prompted you to ever SET that property? I can't thin of a time when I've ever needed to do so.

I was moving Sage to a new server, and also to a new drive structure. The last time I did that I forgot to set that flag, and when I restarted the new server things weren't exactly right, so I lost the information on a large fraction of my shows. I didn't want a repeat of that nightmare.

This time, I forgot to turn it off. Imagine my surprise when the watched list included more than 50K items. There aren't enough hours in the day to watch that many shows over 5 years. It's down to a more reasonable 6600 shows. Well, more reasonable if you don't mind spending almost as much time in front of the TV as you do at work.

[Fuzzy]

Newer versions of sage (I think since like 6.1 or something) shouldn't need that flag anymore. It was a fix for an old problem with how it did database cleanup (it used to purge ALL old data for shows that are no longer there). It now keeps the data for any show that was recorded, regardless of whether the file is deleted or not. It shouldn't be a problem, as it's really not THAT much data to keep (even over MANY years).

[Opus4]

Oh yeah, that property was mentioned in an old FAQ for older versions of SageTV because airing info could be removed if there was no recording for it & no other reason was found to keep the airing details. It is not a part of the updated FAQs for moving recording files.

- Andy

[src666]

Originally Posted by Opus4

Oh yeah, that property was mentioned in an old FAQ for older versions of SageTV because airing info could be removed if there was no recording for it & no other reason was found to keep the airing details. It is not a part of the updated FAQs for moving recording files.

- Andy

I can't recall what version I was on when I did the move (I held off on v6 for a while), but my original move was under v3 or v5, so I'm sure that's when I learned about the setting.