SageTV Community  

Go Back   SageTV Community > SageTV Development and Customizations > SageTV Github Development

Notices

SageTV Github Development Discussion related to SageTV Open Source Development. Use this forum for development topics about the Open Source versions of SageTV, hosted on Github.

Reply
 
Thread Tools Search this Thread Display Modes
  #1  
Old 01-09-2018, 06:46 PM
Narflex's Avatar
Narflex Narflex is offline
Sage
 
Join Date: Feb 2003
Location: Redondo Beach, CA
Posts: 6,299
Cool Fix for bad timelines in MPEG recordings

This is probably one of the only things left in SageTV that can totally piss me off and completely ruin my day. It's when you try to playback a recording and you realize the timeline is either stuck at zero, the end of the file or it increments at some bizarre rate that does not match reality. Then of course if you try to skip...it'll just end, go back to the beginning or completely ruin the game by showing you the 4th quarter score even though just skipped 15 seconds into it.

These files work fine if you just start them and ignore the timeline and don't try to skip...but what's the fun in that? It really annoys me when I forget I'm watching one of these problems and then out of habit at a commercial, I grab the remote and hit skip only to be reset back to the beginning (I can't count how many times I've pissed off the wife trying to navigate around one of these recordings).

Well, I've just posted a fix (not ideal, but it gets the job done) to GitHub for this!

This problem happens generally due to bad commercial insertions by cable providers where they do not readjust the PTS values in the MPEG packets so that the timeline isn't increasing throughout the entire recording. The solution is to just completely ignore the PTS values for purpose of showing the timeline in the UI and for executing seek operations (it just assumes the file is constant bitrate, and uses byte positions instead...not very accurate...but way better than the crap you deal with otherwise).

Anyhoo...here's the commit I just posted to GitHub: https://github.com/google/sagetv/com...b7f2671d4da6f0

And here's the details of the commit which give you all the other details you are likely interested in.

Byte based seeking support for MPEG files

This helps to resolve a long standing issue where files that have bad PTS timelines cause the playback time to jump wildly and infuriate you when you try
to skip around them.

What it does is detect when the parser's calculated duration for a file that is
being played back (extender only, no transcoding) is different by a decent amount (25%) from that of what SageTV thinks the recorded duration should be. Then it uses the read file position to estimate the current playback time and then for seeking, it guesses the proper file position as if the file were constant bitrate. This works WAY better than being stuck without the ability to seek or even know what time you are at in the file.

This feature is off by default, and it can be enabled by setting this in your Sage.properties file:
disable_byte_based_seek_check=false

This does NOT work on files that are currently recording.
This does NOT allow you to fast forward or rewind properly in files like this, only seek.
You CAN force this to work for currently recording files by setting this property in the properties file for the extender you want it to work on:
force_byte_based_seeking=true
(this is so if you're recording something you really want to watch right now that has this issue, you can turn off the extender, edit the properties file and then it'll make it happen...don't forget to reset that property later).
This is somewhat inaccurate, but that's what you get with byte-based time estimation.
__________________
Jeffrey Kardatzke
Google
Founder of SageTV
Reply With Quote
  #2  
Old 01-09-2018, 06:48 PM
Narflex's Avatar
Narflex Narflex is offline
Sage
 
Join Date: Feb 2003
Location: Redondo Beach, CA
Posts: 6,299
And after more people have tried this out and think it's OK then I'll rename the property setting that controls this feature to something else that's actually ON by default rather than OFF. Currently this feature is off by default.
__________________
Jeffrey Kardatzke
Google
Founder of SageTV
Reply With Quote
  #3  
Old 01-09-2018, 07:04 PM
peternm22 peternm22 is offline
Sage Expert
 
Join Date: Jan 2005
Posts: 586
Hey Narflex,

Is this the same issue that people had with R5000 recordings years ago when playing back on the HD300?

Occasionally the file timestamps would be wonky, and once a certain point of the file was reached things would start to jump around (during playback as well, not just seeking).

The fix for this at the time was to run the recordings through Videoredo's Quickstream Fix to fix all the timestamps.

I remember actually burning some of these recordings to a DVD and mailing them down to SageTV in California to test (this was right before the Google buyout).

I don't have an R5000 any longer so I won't be able to test this though Just funny if this is actually fixed after all these years.

Edit: Here is one of the old threads about this problem: https://forums.sagetv.com/forums/showthread.php?t=51645

Last edited by peternm22; 01-09-2018 at 07:08 PM.
Reply With Quote
  #4  
Old 01-09-2018, 10:11 PM
wayner wayner is offline
SageTVaholic
 
Join Date: Jan 2008
Location: Toronto, ON
Posts: 6,491
Awesome, thanks for fixing this Jeff.

On a related note, how does the original show recording date/time matter for the timeline?

I have a process that uses Handbarke to re-encode files. I found that with some files that timeline got screwed up. This was fixed when I changed the date/time created stamp of the reencoded file to be the same as the original file.

Here is that the thread that discusses that issue. https://forums.sagetv.com/forums/showthread.php?t=59800

Presumably this fix will also work with that issue, will it not?
__________________
New Server - Sage9 on unRAID 2xHD-PVR, HDHR for OTA
Old Server - Sage7 on Win7Pro-i660CPU with 4.6TB, HD-PVR, HDHR OTA, HVR-1850 OTA
Clients - 2xHD-300, 8xHD-200 Extenders, Client+2xPlaceshifter and a WHS which acts as a backup Sage server
Reply With Quote
  #5  
Old 01-10-2018, 06:05 AM
tmiranda's Avatar
tmiranda tmiranda is offline
SageTVaholic
 
Join Date: Jul 2005
Location: Central Florida, USA
Posts: 5,621
Some of my recordings have timelines that start at -60 minutes. Will it fix that issue too?
__________________

Sage Server: ASUS/Intel C2D based system w/4GB RAM. Win7, Hauppauge PVR 150, Hauppauge HVR-2250 and an HDHR. Runs headless. Accessed via VNC when necessary. Four HD-300 Extenders.
Reply With Quote
  #6  
Old 01-10-2018, 10:30 AM
Taddeusz Taddeusz is offline
SageTVaholic
 
Join Date: Nov 2004
Location: Yukon, OK
Posts: 3,894
Interesting. I've not figured out what causes it but I've had a problem where all of a sudden at the same point in the recording doing a smooth FF or REW on my media extenders will cause a large seek jump.

I've also had a problem where after performing a smooth FF the playback all of a sudden starts freezing and jumping during playback and then the audio starts losing sync. I've "fixed" this by using the 10 second jump rather than smooth FF.
__________________
Server: i5 8400, ASUS Prime H370M-Plus/CSM, 16GB RAM, 15TB drive array + 500GB cache, 2 HDHR's, SageTV 9, unRAID 6.6.3
Client 1: HD300 (latest FW), HDMI to an Insignia 65" 1080p LCD and optical SPDIF to a Sony Receiver
Client 2: HD200 (latest FW), HDMI to an Insignia NS-LCD42HD-09 1080p LCD
Reply With Quote
  #7  
Old 01-10-2018, 12:18 PM
Narflex's Avatar
Narflex Narflex is offline
Sage
 
Join Date: Feb 2003
Location: Redondo Beach, CA
Posts: 6,299
Quote:
Originally Posted by wayner View Post
Awesome, thanks for fixing this Jeff.

On a related note, how does the original show recording date/time matter for the timeline?

I have a process that uses Handbarke to re-encode files. I found that with some files that timeline got screwed up. This was fixed when I changed the date/time created stamp of the reencoded file to be the same as the original file.

Here is that the thread that discusses that issue. https://forums.sagetv.com/forums/showthread.php?t=59800

Presumably this fix will also work with that issue, will it not?
This fix has nothing to do with that. What you're talking about relates to the metadata for the recording, not the actual video content.


Quote:
Originally Posted by tmiranda View Post
Some of my recordings have timelines that start at -60 minutes. Will it fix that issue too?
It's very possible it will fix it. I don't recall ever seeing that happen myself, but it's worth a try.
__________________
Jeffrey Kardatzke
Google
Founder of SageTV
Reply With Quote
  #8  
Old 01-10-2018, 12:58 PM
tvmaster2's Avatar
tvmaster2 tvmaster2 is offline
SageTVaholic
 
Join Date: Jun 2005
Location: tarana
Posts: 3,270
Quote:
Originally Posted by Narflex View Post
This fix has nothing to do with that. What you're talking about relates to the metadata for the recording, not the actual video content.




It's very possible it will fix it. I don't recall ever seeing that happen myself, but it's worth a try.
I also have what appear to be good recordings, but when I try and export / edit them in VideoReDo, the program crashes out. Their tech support thinks it may have something do with the black frames at the transition having a really low bitrate - is it possible this is caused by the same thing you’re concerned with?
__________________
Sage 7 server = Gigabyte AMD quad-core - 2 gigs - integrated ATI HD4200 chipset - Seagate/Hitachi Deskstar - HD-PVR - Colossus - Win7 32 bit. HD200 networked. "If you've given up on Weird Al, you've given up on life" - Homer Simpson

Last edited by tvmaster2; 01-10-2018 at 02:48 PM.
Reply With Quote
  #9  
Old 01-10-2018, 01:21 PM
trk2 trk2 is offline
Sage Aficionado
 
Join Date: Jan 2006
Location: Maine
Posts: 317
Very nice!

Will this negatively affect comskip in anyway since it relies on the time-stamps created during its scan? Or would comskip already be compromised be the bad PTS timeline so it's a moot point?
Reply With Quote
  #10  
Old 01-10-2018, 06:17 PM
Narflex's Avatar
Narflex Narflex is offline
Sage
 
Join Date: Feb 2003
Location: Redondo Beach, CA
Posts: 6,299
Quote:
Originally Posted by tvmaster2 View Post
I also have what appear to be good recordings, but when I try and export / edit them in VideoReDo, the program crashes out. Their tech support thinks it may have something do with the black frames at the transition having a really low bitrate - is it possible this is caused by the same thing you’re concerned with?
Probably not.

Quote:
Originally Posted by trk2 View Post
Very nice!

Will this negatively affect comskip in anyway since it relies on the time-stamps created during its scan? Or would comskip already be compromised be the bad PTS timeline so it's a moot point?
Assuming comskip was able to process the file properly, it should work fine with it...with the exception that seeking isn't as accurate with this technique so you'll likely not land right on the boundaries like you'd expect. This was especially noticeable for cartoon content where the bitrate during the show was much lower than that of the commercials (so doing constant bitrate estimation for timeline purposes yielded results that were off by a decent amount).
__________________
Jeffrey Kardatzke
Google
Founder of SageTV
Reply With Quote
  #11  
Old 01-12-2018, 07:31 AM
zoop's Avatar
zoop zoop is offline
Sage Advanced User
 
Join Date: Jan 2006
Location: Elkridge, MD
Posts: 184
Quote:
Originally Posted by Narflex View Post
Probably not.

Assuming comskip was able to process the file properly, it should work fine with it...with the exception that seeking isn't as accurate with this technique so you'll likely not land right on the boundaries like you'd expect. This was especially noticeable for cartoon content where the bitrate during the show was much lower than that of the commercials (so doing constant bitrate estimation for timeline purposes yielded results that were off by a decent amount).
Thanks for the work here Jeff... I pulled down the commit and have it running now. I do have some files that exhibit the "zero length" problem, so I'll try to dig those suckers up. I was hoping that this would also address the condition in the miniclient where a seek forward is requested but it doesn't quite make it to the requested seek point, and it jumps backwards after the attempted seek. This behavior is prominent on .ts containers while using the OSX placeshifter, although I would swear I have seen this behavior on the android-tv miniclient as well, although not nearly as repeatable. This never happens on the HD300. Anyway my initial tests indicate that it hasn't helped, and I don't see the log messages suggesting that the new routine is kicking in for that case. Just posting to make sure I am seeing the right behavior.

Thanks again

/jer
__________________

AMD Ryzen 7 1700 w/ SageTV Server Linux V9.1.8.774 - Ubuntu 16.04LTS AMD64
-- 1x HDHomeRun Prime
-- 1x HDHR-DT
-- 1x HD-PVR
2x SageTV HD-300 Media Extenders
1x NVIDIA ShieldTV
Reply With Quote
  #12  
Old 01-12-2018, 06:51 PM
EnterNoEscape's Avatar
EnterNoEscape EnterNoEscape is offline
Sage Icon
 
Join Date: Jun 2010
Location: Harrisburg, PA
Posts: 2,452
About the constant bitrate assumption. Is there no way to ensure a file with any variable bit-rate streams doesn't get this fix applied or do you suspect that will not be a problem for the few times it could happen? I know most broadcast is not variable, but I'm just throwing it out there.
__________________
SageTV v9 Server: ASRock Z97 Extreme4, Intel i7-4790K @ 4.4Ghz, 32GB RAM, 8x 3TB 7200rpm HD, 2x 5TB 7200rpm HD, 4x 256GB SSD, 4x 500GB SSD, unRAID Pro 6.3.0-rc6 (Dual Parity + SSD Cache).
Capture: 1x Ceton InfiniTV 4 (ClearQAM), 2x Ceton InfiniTV 6, 1x BM1000-HDMI, 1x BM3500-HDMI.

Clients: 1x HD300 (Living Room), 1x HD200 (Master Bedroom).
Software: OpenDCT :: WMC Live TV Tuner :: Schedules Direct EPG
Reply With Quote
  #13  
Old 01-16-2018, 11:44 AM
Narflex's Avatar
Narflex Narflex is offline
Sage
 
Join Date: Feb 2003
Location: Redondo Beach, CA
Posts: 6,299
Quote:
Originally Posted by EnterNoEscape View Post
About the constant bitrate assumption. Is there no way to ensure a file with any variable bit-rate streams doesn't get this fix applied or do you suspect that will not be a problem for the few times it could happen? I know most broadcast is not variable, but I'm just throwing it out there.
It will only apply the fix if it appears the timeline is corrupted...in which case the fix is better than the alternative of basically having non-functional seeking. So for that reason I didn't see much point to only applying it to constant bitrate streams.
__________________
Jeffrey Kardatzke
Google
Founder of SageTV
Reply With Quote
  #14  
Old 01-20-2018, 05:50 PM
cat6man's Avatar
cat6man cat6man is offline
Sage Fanatic
 
Join Date: Jan 2006
Location: West of NYC, East of SF
Posts: 848
Quote:
Originally Posted by Narflex View Post
This is probably one of the only things left in SageTV that can totally piss me off and completely ruin my day. It's when you try to playback a recording and you realize the timeline is either stuck at zero, the end of the file or it increments at some bizarre rate that does not match reality. Then of course if you try to skip...it'll just end, go back to the beginning or completely ruin the game by showing you the 4th quarter score even though just skipped 15 seconds into it.

These files work fine if you just start them and ignore the timeline and don't try to skip...but what's the fun in that? It really annoys me when I forget I'm watching one of these problems and then out of habit at a commercial, I grab the remote and hit skip only to be reset back to the beginning (I can't count how many times I've pissed off the wife trying to navigate around one of these recordings).

Well, I've just posted a fix (not ideal, but it gets the job done) to GitHub for this!

This problem happens generally due to bad commercial insertions by cable providers where they do not readjust the PTS values in the MPEG packets so that the timeline isn't increasing throughout the entire recording. The solution is to just completely ignore the PTS values for purpose of showing the timeline in the UI and for executing seek operations (it just assumes the file is constant bitrate, and uses byte positions instead...not very accurate...but way better than the crap you deal with otherwise).

Anyhoo...here's the commit I just posted to GitHub: https://github.com/google/sagetv/com...b7f2671d4da6f0

And here's the details of the commit which give you all the other details you are likely interested in.

Byte based seeking support for MPEG files

This helps to resolve a long standing issue where files that have bad PTS timelines cause the playback time to jump wildly and infuriate you when you try
to skip around them.

What it does is detect when the parser's calculated duration for a file that is
being played back (extender only, no transcoding) is different by a decent amount (25%) from that of what SageTV thinks the recorded duration should be. Then it uses the read file position to estimate the current playback time and then for seeking, it guesses the proper file position as if the file were constant bitrate. This works WAY better than being stuck without the ability to seek or even know what time you are at in the file.

This feature is off by default, and it can be enabled by setting this in your Sage.properties file:
disable_byte_based_seek_check=false

This does NOT work on files that are currently recording.
This does NOT allow you to fast forward or rewind properly in files like this, only seek.
You CAN force this to work for currently recording files by setting this property in the properties file for the extender you want it to work on:
force_byte_based_seeking=true
(this is so if you're recording something you really want to watch right now that has this issue, you can turn off the extender, edit the properties file and then it'll make it happen...don't forget to reset that property later).
This is somewhat inaccurate, but that's what you get with byte-based time estimation.
thanks and looking forward to it.
i always wondered why that happened but it was never common enough to moan about
__________________
Q: dad, when will you stop changing all the electronics?
A: never, so you might as well get used to it.
Reply With Quote
  #15  
Old 03-13-2018, 03:22 PM
bigbill's Avatar
bigbill bigbill is offline
Sage Aficionado
 
Join Date: Dec 2006
Location: San Diego, California
Posts: 362
This problem is similar to what I am experiencing for the last couple weeks. I do not have the erratic skip or ff rew issue. After the recording I try to watch it and it is at the end already and dumps me back to the recorded shows menu. I figured out if I did not select Watch Now, but highlighted the show and pressed play while simultaneously hold down the Skip Back button I could then go all the way to the beginning of the show and it works fine from there, skipping works without issue. Still has 1/2 the bar red and the first half green (see screen shots in other thread)
I had a couple more shows record just like that last night too. I included some screen shots of the show detail and they look odd, the Aired on and Recording from times Don't match.. they are 1 hour 1 minute off. And its not one of those shows that starts early or ends late.

Please look at this thread and my screen shots.

https://forums.sagetv.com/forums/showthread.php?t=65445
__________________
Sage Server: SageTV v9.1.8
Hardware Dell T20 Mini Tower, 4th Gen Pentium, 8GB RAM, Windows 10 Pro 64 bit, 3 x 4TB WD Green, 128GB SSD boot drive. Capture: 2 Dual HDHR, (OTA Winegard HD8200U, CM4221HD), 3 @ STP-HD200's fw=v20100909 TV's: 70", 55" & 40" Sony's - Playon/PlayLater, Blue Iris. automated FFMPEG script converting MPEG2 to MP4.

Motorhome Sage Server: SageTV v9.1.8 Intel NUC6CAYH, HD300, MiBox, Nvidia Shield, 1 Dual HDHR 4TB Passport storage, Winegard Bat-wing antenna.
Reply With Quote
  #16  
Old 03-14-2018, 11:28 AM
Narflex's Avatar
Narflex Narflex is offline
Sage
 
Join Date: Feb 2003
Location: Redondo Beach, CA
Posts: 6,299
That looks like a different problem...I replied in that thread.
__________________
Jeffrey Kardatzke
Google
Founder of SageTV
Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
VideoRedo & Sage Timelines graywolf General Discussion 3 02-19-2012 12:34 PM
Tinelines, timelines, timelines... Polypro SageTV Beta Test Software 3 09-13-2007 10:17 PM
recordings timelines' get messed up JonTom SageTV Software 26 01-30-2007 04:46 PM
timelines (?) in some files are wrong... JonTom SageTV Software 1 01-22-2006 12:03 PM
Currupt MPEG Recordings Muchacho SageTV Software 1 08-30-2004 06:52 PM


All times are GMT -6. The time now is 11:33 PM.


Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2018, vBulletin Solutions, Inc.
Copyright 2003-2005 SageTV, LLC. All rights reserved.