QUESTION: Is it time for a new container?

[Fuzzy]

I've been brainstorming a bit lately about sage, and how it is working, and some common problems people have with it. One, it seems, is the dependance on the wiz.bin file. I understand the need for having a central database for this information, for performance reasons, but it's downside is obvious. I'm curious, with the current work on extending the features and capabilities of the recording and playback filters, if thought has been given to moving sage's recordings into a different, more capable container format - one that could easily support all the formats currently in use by sage (like the current MPEG2-TS container does) but also the ability to store robust metadata in the file.

I'm not saying sage should use this metadata as the primary source, as that would definately hurt performance. But, I do think it makes sense to have this information in the recording files, so that when the wiz.bin inevetably goes south, the data can be rescanned back in to repopulate a NEW wiz.bin, instead of restoring an older backup that may contain further un-found errors. This, combined with redownloading the guide data would allow the wiz.bin's loss to be relatively painless.

So, what do the experts think?

Fuzzy.

[Lucas]

I was actually thinking something similar last night.

I am not sure if the benefits of using mkv for recording are that great but I SageTV should at least provide an mkv format in the "Convert" feature.

That would allow post recording conversion to mkv for those who need it.

[Fuzzy]

Thre are plenty of tools to convert to mkv. However, I don't think that really gains you all that much (people are either converting to H.264, to save space, or converting to another format for a portable device, which already IS supported, and wouldn't use mkv anyways). To me, the only advantage of going to a more robust container is to actually use the more advanced features. If all your going to have in it is video and audio, there's nothing wrong with MP2TS files. They do that perfectly fine, and very efficiently. I really think there needs to be a standardized way for sage to store metadata IN THE FILE.. which m2ts cannot do... but mkv can. This really needs to be written when the file is created, which is why i believe it needs to be done as the main recording format from sage.

[SWKerr]

I have found that the current recording formats are well supported across a number of different platforms and software so I am good with them. Not sure what any change would do for me.

I was using mkv for my movie collection for awhile but have recently started putting everything in a m2ts container. I am really just putting everything in h.264 at this point and the m2ts seems to have great support across in the market. (I am guessing because of BluRays.) Once the video file is in h.264 I have found it relatively easy to switch to mkv or mp4.

[stanger89]

Yeah, I don't think there's any reason to record to MKV, I don't think it gains you anything. As far as metadata goes, mpeg-ts supports a ton of supporting data. There's the whole PSIP strucuture that OTA broadcasts use to deliver both information about the current program, and program guide data. Also, IIRC BTV put metadata in standard mpeg-ps files.

Personally I think it would be a mistake to record to an essentially computer only, "underground" format when they can just as easilly record to a 100% industry standard container.

[Fuzzy]

well, if m2ts/ps would support the metadata, then they REALLY need to start using that capability. Using an undescribed file with just a practically randomly generated id number in the filename that points to a spot in a historically unreliable database is hardly a practical method.

[gplasky]

I'm not sure what you're doing with your wiz.bin but I've been running Sage since 2003 and I had 1 bad wiz.bin that was corrupted from a hard down with a power outage. Previous day backup fixed that. (and that is with running all the betas.)
If I wanted the metadata stored with the file I probably would have moved to Windows Media Center. But with tools like BMT, imdb plugin, DVDProfiler plugin, nielm's webserver I already have the alternatives in place for retrieving and keeping that metadata.
To me the database format doesn't matter because it is all back-end work. As long as Sage has its APIs and those keep improving I'm pretty sure Sage can keep up with the rest. Ansd I also think the plugin developers here help push that envelope with Sage. Look how quick they responded to support EP's UpnpPodcast with Hulu. Netflix and sub-categories within the core product.
For me I don't see another container buying me anything and support for the standards out there just continues to work fine. I like change as much as the next person but it has to get me something more than I already have. I do look forward to see what Ortus and other projects bring to the table.
I voted no. Not because change is bad but this change wouldn't really get me anything, in my opinion.

Gerry

[GKusnick]

Like Gerry, I take exception to the idea that Wiz.bin failure is inevitable. I've never lost a Wiz.bin other than to hard drive failure (and even then was able to restore a working copy from backup).

That said, if you want to keep a backup copy of the metadata outside of Wiz.bin, Nielm's XML generator does that nicely, and his web server allows re-importation of those files into Wiz.bin. So the tools already exist to do what you want without changing file formats.

I did not actually vote since "change is bad" doesn't really describe my position. I just don't think this particular change is a productive use of the devs' time.

[stanger89]

Originally Posted by GKusnick

I did not actually vote since "change is bad" doesn't really describe my position. I just don't think this particular change is a productive use of the devs' time.

I did, but only because I took as a not-optimally worded "No I don't think a new recording container is necessary".

And I to have had a single wiz.bin for ages.

[evilpenguin]

I 100% agree. While I know its easy enough to backup wiz.bin (and I do it weekly), but heres how I see it: wiz.bin has no chioce but to useless without the recordings but there's no reason the recordings have to be useless without wiz.bin.

As for recording straight into a new format, I personally could go either way. It'd really all depend on how easy it would actually be to record straight into a more advanced format for Sage. And I wouldn't worry about the tools for handing those recordings in the new format, writting something to convert the recordings back to mpeg2 so you could process them with your normal workflow would be cake.

[SWKerr]

I would much rather have a file format where the Metadata was embedded in the file. That way the format and changes can be easily shared between systems.

Wouldn't it be nice if when you recorded you shows in sage the description info appeared in iTunes or the Zune player. Now with my music collection I maintain the tag into in iTunes and it magically shows up in Sage when I import it.

I would much rather have the metadata in the file or an associated file than having it exclusively stored in the wiz.bin. Transportable metadata would be great.

[nick_l]

I voted yes, not so much for the specific container, but to the concept of metadata being kept with the file. Imo, metadata should always, always, always be kept with the file! One of the reasons I chose Sage was no drm, and thats great, but let's make it a usefull non-drm'd file. If I take a file outside of sage I'm left with a filename that may or may not tell me what the episode is and a random number that I can't even go and reference somewhere else.

I do understand the need for Sage to keep some info in a propritairy format. After all, they are a for profit company and the difficulty of getting certain info out of Sage and into another program easily (ie. favorites lists) is what helps to keep people from jumping around, at least in some small way. But the media files that Sage generates should be complete and portable files, with complete information included. Heck, even Apple is including metadata within all of it's files now (or so I'm told, I wouldnt go near it with a ten foot pole, personally).

That's my opinion anyway,
Nick

[Fuzzy]

Originally Posted by gplasky

I'm not sure what you're doing with your wiz.bin but I've been running Sage since 2003 and I had 1 bad wiz.bin that was corrupted from a hard down with a power outage. Previous day backup fixed that. (and that is with running all the betas.)

Originally Posted by GKusnick

Like Gerry, I take exception to the idea that Wiz.bin failure is inevitable. I've never lost a Wiz.bin other than to hard drive failure (and even then was able to restore a working copy from backup).

Those are perfect examples of what I'm talking about. Those ARE the situations that end up damaging the wiz.bin files. And they DO happen (as you have just confirmed). The problem is, Sage's built-in backup mechanism is inadequate, as it usually results in the backup being overwritten with the bad file, when you inevitably restart to figure out why something isn't right. Sure, you can keep 10-30 days of sequential backups of the wiz.bin file, but:
1. That's not built into sage, and if it is that important, SHOULD be.
2. Wouldn't it be better if the data was backed up IN THE FILE? Aside from the other advantages of this, such as metadata portability. In your circumstances that caused your damaged wiz.bin's, how many of your video files got corrupted? And if they DO get corrupted/lost, there's no need for the metadata anyways.

The argument that it isn't needed because you prefer to keep your own backups and manually restore doesn't seem to make sense to me.

[GKusnick]

Originally Posted by Fuzzy

The argument that it isn't needed because you prefer to keep your own backups and manually restore doesn't seem to make sense to me.

Of course I prefer to make backups of important files. This has nothing to do with Sage or Wiz.bin; it's just common sense. Disk failures happen, but that's not an argument for requiring a special backup mechanism just for Wiz.bin; it's an argument for including Wiz.bin in your general system backup procedures.

Now there may be other valid reasons for wanting metadata stored in the recording files. I just don't find the alleged fragility of Wiz.bin to be a particularly compelling one.

[gplasky]

In my example it wouldn't matter what database file was being used, wiz.bin, sql, oracle, mysql, postgreSQL, firebird, etc. Database and other types of file can get corrupt especial if the OS does not shutdown gracefully. Backups are a necessity based on what you think you can afford to lose. The wiz.bin contains other info, historical info that I don't think you are going to want to keep in files individually, i.e watched, station recorded from, like, don't like, etc that makes the most sense to be contained in a central database so it can be sorted quickly. We don't keep our recordings forever. We usually watch and within the month delete them. Without a central database that info would be lost without keeping all the files.
Now it would be nice if there was a standard like the ID3 music tags, let's call it ID4 for video files. But add it to the standard file formats just like it has been added to the different music file formats. I don't think it takes a another type of container to do it.

Gerry

[Fuzzy]

I think your missing where i'm going with this Gerry. I'm not talking about a replacement to the wiz.bin. I am talking about metadata being stored in the files, just like you seem to want with the ID4 proposal. And no, I'm not talking about watched status, history, etc. I'm just meaning the non-dynamic show inforamtion. title, episode, description, cast/crew, etc. THAT is they stuff that hurts to lose when teh wiz.bin goes south. Yes, someone COULD develop a standardized form of metadata to be placed in the M2TS, TS, MPG, whatever Mpeg container you choose, and attempt to get the MPEG to approve it... But wouldn't it be easier to just switch to a container format that already supports robust metadata, like MKV?

[stanger89]

Originally Posted by Fuzzy

I think your missing where i'm going with this Gerry. I'm not talking about a replacement to the wiz.bin. I am talking about metadata being stored in the files, just like you seem to want with the ID4 proposal. And no, I'm not talking about watched status, history, etc. I'm just meaning the non-dynamic show inforamtion. title, episode, description, cast/crew, etc. THAT is they stuff that hurts to lose when teh wiz.bin goes south.

How often does your wiz.bin go south, really? But I think Gerry's point was this, IMO the above info is not what "really hurts" to lose, that's all "easy" to recover with online searches. The stuff that would be a real loss is your watched history, likes dislikes, all the stuff that as you say, has no place tied up inside a recording.

Sounds like Gerry and I use Sage similarly, recordings never really stick around that long. I usually watch any recording within a couple days (heck usually within a day) and it gets deleted.

So I don't see how this would be of any real benefit, at least not for those who use Sage like Gerry and I.

Yes, someone COULD develop a standardized form of metadata to be placed in the M2TS, TS, MPG, whatever Mpeg container you choose, and attempt to get the MPEG to approve it... But wouldn't it be easier to just switch to a container format that already supports robust metadata, like MKV?

But that container is harder to work with. Less well supported.

[wayner]

Originally Posted by stanger89

The stuff that would be a real loss is your watched history, likes dislikes, all the stuff that as you say, has no place tied up inside a recording.

I guess everyone works a little differently but I don't give a damn about watched history, likes or dislikes. A lot of my shows are kids' shows and they will watch the same show a thousand times. When it comes to my shows a lot of them are sports and if I don't watch them right away I will delete as there is no point in keeping them around. Most of the stuff I record I will know if I have seen it before and often I don't care because I am interested in watching it again. I have seen every Seinfeld many times so I if "The Contest" has been watched - that is irrelevant to me.

But I do see lots of value in storing metadata with the media file. But I am not sure that I want everything in a less supported media type like MKV/H.264 until we get decent support for those files in other programs like VideoReDo, etc.

[nyplayer]

The watched history, like and dislike is the most important for me. I watch and delete. If I want a series I buy the DVD so meta data to me is not that important... but one thing I don't want is to record a series or show I have already watched and lose one that I have not watched because there is no watched history.

[jlmdxtv]

I'm desperately trying to keep all of my recordings to mpg format. I haven't bothered with HD-PVR because it doesn't record in mpg. I'm also not open to the time involved in converting files because I watch them once, often skipping through them and immediately deleting them.

I'm stuck on mpg because for me WINDVD7 is the best video player out there. It plays with sound at faster playback speeds (better than VLC) so I can really zip through all the crap and sports I'm recoding. (I know WINDVD9+ does other formats too but I've had little luck with anything but mpg).

BTW: my Wiz.bin has never crashed.

john