How To: In-place recompression of Sage Recordings

[nielm]

I have had sucess with in-place recompression of Sage recordings: compressing the MPG to AVI, but keeping the file in the Sage Recordings list together with all the show information etc, so I though I would share my method with you.

Warning: You do this at your own risk! This is hacking sage to do something which it may not be intended to do. Any loss of data is not my fault!

This uses the 'media file recovery' feature of Sage. This is a feature when the Sage server/service starts up, if it finds any unknown files in the Sage video directory, it tries to match the numeric suffix (-xxxxxxx-0.) to an internal Airing ID in the EPG. If it finds one, then it links the mediafile to the airing and uses the file duration and file timestamp to set the start/end time of the recording.

With this feature, you can rename the first part of the filename, or the extension of any Sage recording file while the sage server/service is shut down, and it will still be linked to the airing info in the EPG when Sage restarts.

You can also use this feature to replace recordings with ones that have been edited to remove padding and commercials.

This is my full sequence of actions. If you don't understand what is going on here, don't try it, unless you are willing to lose the information for the recordings you are working on!

1. If the file is segmented (multiple sections ending in xxxxxx-0.mpg, xxxxxxx-1.mpg etc) then use an MPEG2 merge tool to create a single MPG.
   .
2. Recompress the MPG to an AVI, making sure that the AVI is written to the same partition but not in Sage Recordings Directory (see note 4 below)
   For compression, I use AutoGK (see note 5 below) and Xvid.
   AutoGK auto-crops black bars in letterbox recordings, auto-deinterlaces and auto sets quality which is useful!
   I use a custom size that I calculate as approx 700Mb/hr, or 1400Mb for a movie. (Using fixed sizes is the only way to get AutoGk to autocrop, and autoresize.)
   .
3. Check that the compressed file is playable.
   .
4. Verify in the web-interface that the AringID in the MPG file name is the real Airing ID (In webserver 1.2+ the AiringID is at the bottom of the Detailed Info page. In webserver 1.1, it is in the address of the detailed info page found from a search for airings 'in the past': AiringID=xxxx)
   .
5. Set the last modified timestamp on the generated AVI to the to the 'end time' of the program (normally same timestamp as the original MPG). (see note 2 below)
   There are several tools that can do this, I use the commandline tool touch from Cygwin or GnuWin32 Fileutils with a command line of either touch -d 'MMM dd hh:mm' file.avi, or to get the timestamp from the original MPG: touch -r file.mpg file.avi but there are also freeware Windows tools such as XTST
   .
6. make sure that an EPG update is not about to happen, or in progress (Setup/System information), and that nothing is currently being recorded (so you don't miss anything)
   .
7. Shut down the Sage server, or stop the service (net stop sagetv)
   .
8. backup Wiz.bin (see note 3 below)
   .
9. move the MPG to a different directory outside of Sage Recordings/video library directory
   .
10. move the AVI into the Sage Recordings directory, making sure it has the same airing ID in the filename suffix (xxxxxxx-0.avi) as the original MPG.
    .
11. start Sage or Sage service (net start sagetv)
    .
12. Check Sage Recordings, and start to playback the file to confirm that it is there and the timebar is correct.
    .
13. If all Ok, remove the orignal MPG from the disk (outside of Sage)
    If not, stop Sage or Sage service; restore backed up Wiz; return the MPG file to it's original location; move the AVI file back out of sage recordings and try to figure out what went wrong!
    .
14. Any manual recording flags are cleared so Sage sees it as an 'unwanted' recording after replacement. This puts it high on the list of things to delete, so it might be an idea to re-set the recording flag (CTRL-Y or 'record' remote button in the detailed information page, or when viewing the file) in Sage to prevent it from being from being automatically deleted.

Issues I have found:

1. The Airing IDs may actually be different to the filename suffix. This is because when the IDs reach 999999999, Sage automatically 'compresses' the number range and re-assigns low range numbers to everything. The only way of checking the airing ID is seaching for it in my webserver and checking the URL.
   .
2. If the timestamp of the AVI file is not changed, then the timebar will be messed up as Sage will think that this media file segment starts a long time after the airing starts
   .
3. If the airing ID is not matched, and the show was not marked as watched or don't like, the information for this airing will be deleted from the DB, and the file will show up in the library so a Wiz backup is important to restore the airing information.
   .
4. When recompressing, the generated file must be generated outside of the Sage video/library dirs so that it is still an 'unknown' file when it is moved into the Video dir.
   Keeping the compressed video file in the same partition as the original MPG allows simple move commands which operate quickly, rather than cross-partition copy/delete (which is very slow with large files). This minimises the time that ther Sage server/service is shut down
   .
5. Warning: latest 1.9x version of AutoGK contains Adware/Spyware... It can be cleanly installed from Add/remove programs apparently... I am still using 1.6.0
   .
6. I have Intelligent Recording on, so sage will never delete non-favorite, non-manual record or IR files. I cannot guarantee the behavior if IR is off, as Sage may decide that it is a partial or unwanted file and delete it.

Its a hack (in the traditional sense of the word) I know.

I have a plan (!) that at some point I will create a STV+java plugin that will use a 'compression queue' playlist and a command line encoder (ffmpeg or mencoder, os seamonkey's batch file!) to re-compress files, then automatically re-associate them with the airing info once compression is complete, without a need to shut sage down...
The only problem is extra features would need to be added to the Sage API to allow me to do this.

Good luck!
Niel

[bgorrell]

Originally Posted by nielm

I have a plan (!) that at some point I will create a STV+java plugin that will use a 'compression queue' playlist and a command line encoder (ffmpeg or mencoder, os seamonkey's batch file!) to re-compress files, then automatically re-associate them with the airing info once compression is complete, without a need to shut sage down...
The only problem is extra features would need to be added to the Sage API to allow me to do this.

Good luck!
Niel

Wow, that took a lot of work, but this is exactly what I've been waiting for. So far, it's been a trade-off between wasted space, and wasted info. I find the show info invaluable, so I've been wasting space instead. Can't wait untill it gets implemented/automated into an STV.

[Dav0id]

I agree completely! I've been waiting for this feature for some time now. I'm actually surprised that it isn't built into programs like sage - background, automatic, in-place compression during system idle time. It would be an effective hard drive doubler. (this is coming from someone w/ > 1/3 of a TB but needs MORE)

I also agree about the API part - I'd love to help implement this.

[abasu2003]

Two questions:

-How long does AutoGK take on your system. I found it takes a reallyt long time for me but I think it's supposed to get better the more tiems it does it right?

-What kind of quality loss do you get in the picture? any? Are you starting with Mpeg2 Max Quality?

[nielm]

Originally Posted by abasu2003

Two questions:

-How long does AutoGK take on your system. I found it takes a reallyt long time for me but I think it's supposed to get better the more tiems it does it right?

-What kind of quality loss do you get in the picture? any? Are you starting with Mpeg2 Max Quality?

AutoGK takes a while (10-15fps on an AMD64 3000+), but that is because it is doing a 2-pass encode by default to get the exact file size (this is not strictly necessary, but it is the only way AutoGK does resize and cropping tests). I just queue a couple of encodes up and leave it overnight... My only problem with AutoGk is that it runs all it's tools in Normal priority, which means that they fight with Sage for resources (unless you use Task Manager to re-assign them Low priority)

You don't have to use autoGK, you could use ffmpeg that is included with Cayars' STV, or mencoder but that does not crop the black bars from letterboxed (unless you play with the command line arguments)... (mencoder can do 31fps on my other box - a P4HT 2800Ghz)

I don't see any quality loss (starting from DVD standard play PAL), but then I am using TV-out... I am not sure, but I think I see an improvement in quality (with MPEG2 I am using BOB deinterlace at the moment, which results in half the resolution, AutoGK generates progressive output, which would use the full resolution). Anyway it is not noticable enough to comment... Note: I don't have a HUGE screen TV, and I am using my vidcard's S-Video out at 768x576 resolution, so differences in quality tend to be masked I am also not super obsessive about perfect quality

[dvd_maniac]

AutoGK takes a while (10-15fps on an AMD64 3000+), but that is because it is doing a 2-pass encode by default to get the exact file size (this is not strictly necessary, but it is the only way AutoGK does resize and cropping tests

This is at 700MB/hr? I use AutoGK for my recordings but for regular tv shows I use 300MB/hr and 500MB/hr for dvd rips on my 3.0P4 I am getting about 60-80fps on 2 pass. It takes me a little over 1 hour to encode 1 hour of video.
I record in Sage using Best, which is 6mbit/s VBR. Would I get better encoded results using CBR 8mbit/s?

[insomniac]

wow..this is the sole reason that i chose not to compress recordings! I hated the fact that you lost all the important data when you compressed and moved stuff to the library. I cannot wait to try this out! thanks!

Originally Posted by nielm

I have had sucess with in-place recompression of Sage recordings: compressing the MPG to AVI, but keeping the file in the Sage Recordings list together with all the show information etc, so I though I would share my method with you.

This is my full sequence of actions. If you don't understand what is going on here, don't try it, unless you are willing to lose the information for the recordings you are working on!

<snip>

[Wheemer]

All we need now it someone to make this process automated... Perhaps this little program posted on Snapstream forum could assist in some way: http://forums.snapstream.com/vb/showthread.php?t=20628

It monitors a folder and auto compresses using autogk.

[nielm]

Originally Posted by dvd_maniac

This is at 700MB/hr? I use AutoGK for my recordings but for regular tv shows I use 300MB/hr and 500MB/hr for dvd rips on my 3.0P4 I am getting about 60-80fps on 2 pass. It takes me a little over 1 hour to encode 1 hour of video.
I record in Sage using Best, which is 6mbit/s VBR. Would I get better encoded results using CBR 8mbit/s?

When recompressing to smaller file sizes, AutoGk will resize the video before compression, which results in less work for the MPEG4 compressor, which results in faster compression. This may be why my 700Mb/hr takes longer than your 3-500Mb/hr

By the way, my 15fps average is the overall time for the whole compression: ie test, first pass and second pass. (total time ~ 1.75 hrs compression time for each hour of video). The individual compression jobs go at about 15fps for the test, 45fps for the first pass and 25 fps for the second.

In general, when you intend to transcode, you should always capture at a best bitrate pratical to get the highest quality source, and avoid getting MPEG2 artifacts in the compressed MPEG4 output (think a photocopy of a photocopy: each stage adds noise). But I never know what I am going to recompress and I am too lazy to change the bitrate for each recording

[seamonkey420]

nice..!

hmm.....

i wonder if my divx encoding batchfile works like this?? when it re-encodes the mpg to a divx avi, it keeps all of the filename fully in tack, (ie SimpsonsMaggiePukes92993-1.avi ) and even puts the files into a new directory with the current date it was encoded (ie encoded\03152005\XXXXXX-0.avi )

actually, yes i think it would.. i have not imported my folder that i save my encoded shows to into sagetv.. i will try it once i get my new cpu and vid card tonight!!

its somewhat semi automated since its a batch file (.bat) and it can be setup to run at certain time using good ole windows scheduler.. (ie 4am..)
a quick link to the files and info (uses virtualdub for encoding)
http://forums.freytechnologies.com/f...ad.php?t=10612

peace
seamonkey420

[nielm]

For semi-automated setting of the timestamp, GNU FileUtils' touch has a flag where a timestamp can be set on a file using the timstamp of another file. So you can use the timestamp of the original MPG to set the timestamp of the new AVI:
touch -r file.mpg file.avi

[bgorrell]

I also found that 4NT's touch command works well for this. Of course, what we really need is one big batch file that runs AutoGK, touches the dates, shuts down Sage when it is idle, moves files around, and restarts Sage once daily. If version 3.0 doesn't have some sort of built-in solution that is.

[bgorrell]

Just a thought Nielm, since you have knowledge of how this works, and the ability to edit the database (i.e. webserver's features), could you somehow create a plug-in to automate this? Perhaps as simple as taking an existing database entry, letting us pick a new filename to associate with it, and then deleting the old file that was associated with it?

[nielm]

It is possible and is one of the things I am thinking about...

[thborchert]

One question I have: I tried this but I didn't recompress, I just cut padding and advertising away. This gives me a much shorter film length. The film shows up nicely in Sage, but the time bar doesn't work right at all. Any way to fix this? Thanks!

Thomas

[nielm]

Quick and easy way -- use my webserver to edit show info, check that filestart time is 0 seconds earlier (or later!), then just hit OK -- it re-scans the file and re-determines the length...

(this probably happened because the timestamp on the file did not corespond to airing start time + file duration...)

[thborchert]

Originally Posted by nielm

Quick and easy way -- use my webserver to edit show info, check that filestart time is 0 seconds earlier (or later!), then just hit OK -- it re-scans the file and re-determines the length...

Thanks, I'll do that.

Originally Posted by nielm

(this probably happened because the timestamp on the file did not corespond to airing start time + file duration...)

Let me make sure I got it right: I set the "last modified" time to the time the show ended according to the EPG. Is that the wrong way to do it?

Thanks!

Thomas

[nielm]

It should be OK, but if there were lots of adverts before you did your cut then you would get a red beginning on the timebar...
Airing: 2hrs from 12pm-2pm
Recorded file: 2hrs, timestamp 2pm (so 12pm-2pm)
Edited file: 1.5hrs, timestamp 2pm (so 12:30pm-2pm)
This would show up as a timebar with the first quarter showing red/not recorded.

(you did not mention how the timebar did not work, I was just making a guess!)

[thborchert]

Actually, it's more like: Playback starts at the beginning of the green time bar (which is showing 2 hours length), but the actual playback is starting about 10 or 15 minutes into the show. Then, I hit the 2.5-minutes-back button repeatedly, find the show start somewhere at minus 10 minutes on the time bar and start from there. The advance of time is then faster than real time (when a minute passes there, a minute hasn't passed in real life), so that at the end of the displayed 2 hours, the 1.5-hour-show will be over. This is all on an MVP, and I think the MVP plug-in influences time-bar display

[nielm]

This sounds like Sage has not re-scanned the file to determine it's length -- so it thinks the file is 2hrs long, then on playback sees that it is 1.5hrs, and re-scales the timebar accordingly...

This probably happened because the cut file has exactly the same name+ext as the original, so sage does not bother rescanning it to re-dertermine its length...