Fix for bad timelines in MPEG recordings

[Narflex]

This is probably one of the only things left in SageTV that can totally piss me off and completely ruin my day. It's when you try to playback a recording and you realize the timeline is either stuck at zero, the end of the file or it increments at some bizarre rate that does not match reality. Then of course if you try to skip...it'll just end, go back to the beginning or completely ruin the game by showing you the 4th quarter score even though just skipped 15 seconds into it.

These files work fine if you just start them and ignore the timeline and don't try to skip...but what's the fun in that? It really annoys me when I forget I'm watching one of these problems and then out of habit at a commercial, I grab the remote and hit skip only to be reset back to the beginning (I can't count how many times I've pissed off the wife trying to navigate around one of these recordings).

Well, I've just posted a fix (not ideal, but it gets the job done) to GitHub for this!

This problem happens generally due to bad commercial insertions by cable providers where they do not readjust the PTS values in the MPEG packets so that the timeline isn't increasing throughout the entire recording. The solution is to just completely ignore the PTS values for purpose of showing the timeline in the UI and for executing seek operations (it just assumes the file is constant bitrate, and uses byte positions instead...not very accurate...but way better than the crap you deal with otherwise).

Anyhoo...here's the commit I just posted to GitHub: https://github.com/google/sagetv/com...b7f2671d4da6f0

And here's the details of the commit which give you all the other details you are likely interested in.

Byte based seeking support for MPEG files

This helps to resolve a long standing issue where files that have bad PTS timelines cause the playback time to jump wildly and infuriate you when you try
to skip around them.

What it does is detect when the parser's calculated duration for a file that is
being played back (extender only, no transcoding) is different by a decent amount (25%) from that of what SageTV thinks the recorded duration should be. Then it uses the read file position to estimate the current playback time and then for seeking, it guesses the proper file position as if the file were constant bitrate. This works WAY better than being stuck without the ability to seek or even know what time you are at in the file.

This feature is off by default, and it can be enabled by setting this in your Sage.properties file:
disable_byte_based_seek_check=false

This does NOT work on files that are currently recording.
This does NOT allow you to fast forward or rewind properly in files like this, only seek.
You CAN force this to work for currently recording files by setting this property in the properties file for the extender you want it to work on:
force_byte_based_seeking=true
(this is so if you're recording something you really want to watch right now that has this issue, you can turn off the extender, edit the properties file and then it'll make it happen...don't forget to reset that property later).
This is somewhat inaccurate, but that's what you get with byte-based time estimation.

[Narflex]

And after more people have tried this out and think it's OK then I'll rename the property setting that controls this feature to something else that's actually ON by default rather than OFF. Currently this feature is off by default.

[peternm22]

Hey Narflex,

Is this the same issue that people had with R5000 recordings years ago when playing back on the HD300?

Occasionally the file timestamps would be wonky, and once a certain point of the file was reached things would start to jump around (during playback as well, not just seeking).

The fix for this at the time was to run the recordings through Videoredo's Quickstream Fix to fix all the timestamps.

I remember actually burning some of these recordings to a DVD and mailing them down to SageTV in California to test (this was right before the Google buyout).

I don't have an R5000 any longer so I won't be able to test this though Just funny if this is actually fixed after all these years.

Edit: Here is one of the old threads about this problem: https://forums.sagetv.com/forums/showthread.php?t=51645

[wayner]

Awesome, thanks for fixing this Jeff.

On a related note, how does the original show recording date/time matter for the timeline?

I have a process that uses Handbarke to re-encode files. I found that with some files that timeline got screwed up. This was fixed when I changed the date/time created stamp of the reencoded file to be the same as the original file.

Here is that the thread that discusses that issue. https://forums.sagetv.com/forums/showthread.php?t=59800

Presumably this fix will also work with that issue, will it not?

[tmiranda]

Some of my recordings have timelines that start at -60 minutes. Will it fix that issue too?

[Taddeusz]

Interesting. I've not figured out what causes it but I've had a problem where all of a sudden at the same point in the recording doing a smooth FF or REW on my media extenders will cause a large seek jump.

I've also had a problem where after performing a smooth FF the playback all of a sudden starts freezing and jumping during playback and then the audio starts losing sync. I've "fixed" this by using the 10 second jump rather than smooth FF.

[Narflex]

Originally Posted by wayner

Awesome, thanks for fixing this Jeff.

On a related note, how does the original show recording date/time matter for the timeline?

I have a process that uses Handbarke to re-encode files. I found that with some files that timeline got screwed up. This was fixed when I changed the date/time created stamp of the reencoded file to be the same as the original file.

Here is that the thread that discusses that issue. https://forums.sagetv.com/forums/showthread.php?t=59800

Presumably this fix will also work with that issue, will it not?

This fix has nothing to do with that. What you're talking about relates to the metadata for the recording, not the actual video content.

Originally Posted by tmiranda

Some of my recordings have timelines that start at -60 minutes. Will it fix that issue too?

It's very possible it will fix it. I don't recall ever seeing that happen myself, but it's worth a try.

[tvmaster2]

Originally Posted by Narflex

This fix has nothing to do with that. What you're talking about relates to the metadata for the recording, not the actual video content.




It's very possible it will fix it. I don't recall ever seeing that happen myself, but it's worth a try.

I also have what appear to be good recordings, but when I try and export / edit them in VideoReDo, the program crashes out. Their tech support thinks it may have something do with the black frames at the transition having a really low bitrate - is it possible this is caused by the same thing you’re concerned with?

[trk2]

Very nice!

Will this negatively affect comskip in anyway since it relies on the time-stamps created during its scan? Or would comskip already be compromised be the bad PTS timeline so it's a moot point?

[Narflex]

Originally Posted by tvmaster2

I also have what appear to be good recordings, but when I try and export / edit them in VideoReDo, the program crashes out. Their tech support thinks it may have something do with the black frames at the transition having a really low bitrate - is it possible this is caused by the same thing you’re concerned with?

Probably not.

Originally Posted by trk2

Very nice!

Will this negatively affect comskip in anyway since it relies on the time-stamps created during its scan? Or would comskip already be compromised be the bad PTS timeline so it's a moot point?

Assuming comskip was able to process the file properly, it should work fine with it...with the exception that seeking isn't as accurate with this technique so you'll likely not land right on the boundaries like you'd expect. This was especially noticeable for cartoon content where the bitrate during the show was much lower than that of the commercials (so doing constant bitrate estimation for timeline purposes yielded results that were off by a decent amount).

[zoop]

Originally Posted by Narflex

Probably not.

Assuming comskip was able to process the file properly, it should work fine with it...with the exception that seeking isn't as accurate with this technique so you'll likely not land right on the boundaries like you'd expect. This was especially noticeable for cartoon content where the bitrate during the show was much lower than that of the commercials (so doing constant bitrate estimation for timeline purposes yielded results that were off by a decent amount).

Thanks for the work here Jeff... I pulled down the commit and have it running now. I do have some files that exhibit the "zero length" problem, so I'll try to dig those suckers up. I was hoping that this would also address the condition in the miniclient where a seek forward is requested but it doesn't quite make it to the requested seek point, and it jumps backwards after the attempted seek. This behavior is prominent on .ts containers while using the OSX placeshifter, although I would swear I have seen this behavior on the android-tv miniclient as well, although not nearly as repeatable. This never happens on the HD300. Anyway my initial tests indicate that it hasn't helped, and I don't see the log messages suggesting that the new routine is kicking in for that case. Just posting to make sure I am seeing the right behavior.

Thanks again

/jer

[EnterNoEscape]

About the constant bitrate assumption. Is there no way to ensure a file with any variable bit-rate streams doesn't get this fix applied or do you suspect that will not be a problem for the few times it could happen? I know most broadcast is not variable, but I'm just throwing it out there.

[Narflex]

Originally Posted by EnterNoEscape

About the constant bitrate assumption. Is there no way to ensure a file with any variable bit-rate streams doesn't get this fix applied or do you suspect that will not be a problem for the few times it could happen? I know most broadcast is not variable, but I'm just throwing it out there.

It will only apply the fix if it appears the timeline is corrupted...in which case the fix is better than the alternative of basically having non-functional seeking. So for that reason I didn't see much point to only applying it to constant bitrate streams.

[cat6man]

Originally Posted by Narflex

This is probably one of the only things left in SageTV that can totally piss me off and completely ruin my day. It's when you try to playback a recording and you realize the timeline is either stuck at zero, the end of the file or it increments at some bizarre rate that does not match reality. Then of course if you try to skip...it'll just end, go back to the beginning or completely ruin the game by showing you the 4th quarter score even though just skipped 15 seconds into it.

These files work fine if you just start them and ignore the timeline and don't try to skip...but what's the fun in that? It really annoys me when I forget I'm watching one of these problems and then out of habit at a commercial, I grab the remote and hit skip only to be reset back to the beginning (I can't count how many times I've pissed off the wife trying to navigate around one of these recordings).

Well, I've just posted a fix (not ideal, but it gets the job done) to GitHub for this!

This problem happens generally due to bad commercial insertions by cable providers where they do not readjust the PTS values in the MPEG packets so that the timeline isn't increasing throughout the entire recording. The solution is to just completely ignore the PTS values for purpose of showing the timeline in the UI and for executing seek operations (it just assumes the file is constant bitrate, and uses byte positions instead...not very accurate...but way better than the crap you deal with otherwise).

Anyhoo...here's the commit I just posted to GitHub: https://github.com/google/sagetv/com...b7f2671d4da6f0

And here's the details of the commit which give you all the other details you are likely interested in.

Byte based seeking support for MPEG files

This helps to resolve a long standing issue where files that have bad PTS timelines cause the playback time to jump wildly and infuriate you when you try
to skip around them.

What it does is detect when the parser's calculated duration for a file that is
being played back (extender only, no transcoding) is different by a decent amount (25%) from that of what SageTV thinks the recorded duration should be. Then it uses the read file position to estimate the current playback time and then for seeking, it guesses the proper file position as if the file were constant bitrate. This works WAY better than being stuck without the ability to seek or even know what time you are at in the file.

This feature is off by default, and it can be enabled by setting this in your Sage.properties file:
disable_byte_based_seek_check=false

This does NOT work on files that are currently recording.
This does NOT allow you to fast forward or rewind properly in files like this, only seek.
You CAN force this to work for currently recording files by setting this property in the properties file for the extender you want it to work on:
force_byte_based_seeking=true
(this is so if you're recording something you really want to watch right now that has this issue, you can turn off the extender, edit the properties file and then it'll make it happen...don't forget to reset that property later).
This is somewhat inaccurate, but that's what you get with byte-based time estimation.

thanks and looking forward to it.
i always wondered why that happened but it was never common enough to moan about

[bigbill]

This problem is similar to what I am experiencing for the last couple weeks. I do not have the erratic skip or ff rew issue. After the recording I try to watch it and it is at the end already and dumps me back to the recorded shows menu. I figured out if I did not select Watch Now, but highlighted the show and pressed play while simultaneously hold down the Skip Back button I could then go all the way to the beginning of the show and it works fine from there, skipping works without issue. Still has 1/2 the bar red and the first half green (see screen shots in other thread)
I had a couple more shows record just like that last night too. I included some screen shots of the show detail and they look odd, the Aired on and Recording from times Don't match.. they are 1 hour 1 minute off. And its not one of those shows that starts early or ends late.

Please look at this thread and my screen shots.

https://forums.sagetv.com/forums/showthread.php?t=65445

[Narflex]

That looks like a different problem...I replied in that thread.