Retaining Recording Detail Info After Video Compression

[mayamaniac]

Skipped the paragraph below if you don't want to know about my conversion test stories between AutoGK and Divx Converter.
...............
I have over a hundred archived shows that needed to be compressed to free up space. I tried SageEncoder, but never got it working. So I waited for SageTV 6 beta for the Video Conversion tool. So after testing v6 with the various settings, the video quality is unacceptable as many other have also found out. So I looked at other conversion methods, and it was a choice between Divx Converter and AutoGK. After some test, I chose the Divx Converter as the quality was better, faster encode times, easier to use, and doesn't crash. In AutoGK, whenever I tried to change the resolution instead of the "Auto" setting, it would crash everytime. Divx Converter never crashed in my experience. The usual shows are 2.5G/hour Mpeg2 and compressed to Divx to about 632M. This is not great compression but I like to keep the quality high, plus it kept the original resolution. The deinterlacing is decent using the preset "Home Theater profile" setting.
...............

Anyway, after the conversions to Divx, I'm just wondering how is it that some show retained the Recording Detail Info and some don't? The ones that retained info went into the Media Center > Archived menu. The ones that don't went into the Imported Video folder. These are the steps I took to do the divx conversion to all the videos:

1. Create \\Server\D\Archived folder.
2. Add Added \\Server\D\Archived folder path to SageTV as an Imported Video folder.
2. Start Divx Converter and load up a batch of Mpeg2 files to convert. I kept all source MPeg2 files in the SageTV video folders, and stored the converted Divx files in the \\Server\D\Archived folder.
3. After conversion of each recording, I deleted the original Mpeg2 file. New divx files have .divx extension and SageTV6 plays them just fine.

Now on the last step, this varies in terms of "when" I delete the original Mpeg2 files. Sometime a few hours later, some time a day or two. I'm not sure if this matters or not. But its the only step that is different for each conversion. Also, I did all these steps while SageTV Service is running, I never shut it down. So again, how is it that some show retained the Recording Detail Info and some don't?

[GKusnick]

It's probably just a timing thing. If SageTV did an automatic refresh of imported media while both files existed, then it probably classified the converted file as an import. If it it did its rescan after the original was deleted, then it might have classified it as a TV recording that was moved to another folder. That's my guess anyway.

Which outcome were you hoping for? Did you want the recording info retained, or did you want them to end up as imports? If you want the info retained, the FAQ has some advice on how to accomplish that (i.e. move them to another recording dir, not an import dir, and do it while the service is stopped).

[nielm]

I wrote a somewhat complicated how-to replace a file with a recompressed version a while back for Sage 2.2 -- at that time I could reliably replace an MPG with an AVI..

Have not tried it recently (I have more disk space now) but the theory should still be sound... It may be that with V5 and V6, the server restart is no longer required, and you just need to rescan the DB...

[mayamaniac]

I've read about the other "tricks" you guys mentioned a while back to move recordings, but both requires shutting the service off, which I don't want to do. So since SageTV6 is doing it automatically, thats what I want to know, what SageTV is doing exactly. I don't really care about retaining the info, but it was a pleasant surprise to find out that about half of the recordings did. This includes "Watch" status. So do we know for sure that it is the "Refresh Media" is the solution? And does both recordings (Mpeg2 and Divx) recordings have to be present before doing a refresh? Or delete the mpeg2 first, then refresh?

[davephan]

Originally Posted by mayamaniac

I've read about the other "tricks" you guys mentioned a while back to move recordings, but both requires shutting the service off, which I don't want to do. So since SageTV6 is doing it automatically, thats what I want to know, what SageTV is doing exactly. I don't really care about retaining the info, but it was a pleasant surprise to find out that about half of the recordings did. This includes "Watch" status. So do we know for sure that it is the "Refresh Media" is the solution? And does both recordings (Mpeg2 and Divx) recordings have to be present before doing a refresh? Or delete the mpeg2 first, then refresh?

I would like to save disk space so I can put more files on my 1 TB diskspace. I downloaded SmartSoft Video Converter. What compression setting do you find that does not degrade the video recordings? I have my default settings SageTV setting to use the 'standard DVD' recording quality. I would like to keep the quality level the same (or not notice the quality decrease). I have a 60 inch TV.


Dave

[mayamaniac]

Originally Posted by davephan

I would like to save disk space so I can put more files on my 1 TB diskspace. I downloaded DivX Converter. What compression setting do you find that does not degrade the video recordings? I have my default settings SageTV setting to use the 'standard DVD' recording quality. I would like to keep the quality level the same (or not notice the quality decrease). I have a 60 inch TV.


Dave

When it comes to picture quality, it varies for each person's preference. This goes for audio compression (MP3) also. Some people can tell the difference between 128kbps and 320kbps, some can't.

What I did was used a one minute clip of video to do tests with divx converter. You can either edit out a one minute clip from existing show with some sort of video editor like VideoRedo or do a manual recording of one minute in length in SageTV. Use this clip to tests the various settings in Divx Converter to produce the quality you desire. I used the default "Home Theater" profile and pleased with the results, but they have other higher quality profiles also. With the one minute clip, its easy to calculate what the filesize of an actual full length show, just multiply the filesize x number of minutes.

[davephan]

Originally Posted by mayamaniac

When it comes to picture quality, it varies for each person's preference. This goes for audio compression (MP3) also. Some people can tell the difference between 128kbps and 320kbps, some can't.

What I did was used a one minute clip of video to do tests with divx converter. You can either edit out a one minute clip from existing show with some sort of video editor like VideoRedo or do a manual recording of one minute in length in SageTV. Use this clip to tests the various settings in Divx Converter to produce the quality you desire. I used the default "Home Theater" profile and pleased with the results, but they have other higher quality profiles also. With the one minute clip, its easy to calculate what the filesize of an actual full length show, just multiply the filesize x number of minutes.

Thanks for your quick response Mayamaniac. I disovered that I downloaded and installed the wrong DivX Converting program, called Smart Soft Divx Converter - there wasn't a 'Home Theatre' setting. I then found DivX Converter 6.2 at www.divx.com, priced just under $10. The GUI is a bit strange with the graphic that looks like a watch, and a list of files that are selected for conversion.

I set the quality to 'Home Theatre' and I am testing it now.

I use VideoReDo to edit SageTV recordings. I cannot edit the AVI files with VideoReDo, so I would need to edit the mpeg2 files first with VideoReDo, then convert them to DivX.

I am also looking for a good DVD ripper. I tried several of them so far. # 1 DVD Ripper is able to rip the DVD content to mpeg2 format. AnyDVD could not rip to mpeg format.

I used to have to archive most of my recordings with burned DVDs before I expanded my diskspace to over 1 TB. Now, I would like to keep a lot more content on my computer.

Dave

[nielm]

Originally Posted by mayamaniac

So do we know for sure that it is the "Refresh Media" is the solution?

it is not: all that will do is to make sage find the file immediatly, in stead of when it next does a library scan in xxx mins time

And does both recordings (Mpeg2 and Divx) recordings have to be present before doing a refresh?

this would probably be a reason for it not to work -- I believe you cannot have 2 files referencing the same airing, this may be why the simultaneous replace did not work: it finds the divx/avi before it sees that the mpeg2 has been deleted.

Or delete the mpeg2 first, then refresh?

probably... but thinking about this, a more reliable solution may be:

1. Remember to check the airing ID in the webserver to ensure that the divx/avi file is named correctly (see my HOWTO above for what the filename format should be)
2. put divx/avi on same disk as a Sage recordings Directory, but NOT in the recordings directory.
3. set the file/airing to watched (so that sage will keep the airing info in the DB forever)
4. delete the file from within Sage (see past Airings search to confirm that the airing is still in the DB)
5. MOVE AVI/divx into the recordings directory (note: not a library directory)
6. refresh + wait for refresh to complete

Theoretically, this should work, and it just worked for me...

[mayamaniac]

Originally Posted by nielm

[list=1]
Remember to check the airing ID in the webserver to ensure that the divx/avi file is named correctly (see my HOWTO above for what the filename format should be)

The Divx Converter app use the same exact filename as the original, with .divx extension of course.

put divx/avi on same disk as a Sage recordings Directory, but NOT in the recordings directory.

now this is interesting, why does it have to be on the same disk and what happens when it is not?

set the file/airing to watched (so that sage will keep the airing info in the DB forever)

interesting that the watch status determines whether the show is stored in the DB.

MOVE AVI/divx into the recordings directory (note: not a library directory)
refresh + wait for refresh to complete

Theoretically, this should work, and it just worked for me...

Thanks for trying it out. But I guess I should be more clear of what I'm asking. I actually don't want to do any work to get it to retain recording detail info. Before I did the conversion process, I was expecting all those detail info to be lost after the conversion. But I was surprised to see that majority of the shows retained the info. I just checked and apparently SageTV just refreshed again and I see more shows are showing up with detail info.

So what I wanted to know how SageTV was doing it on its own without me having to do anything except for converting to Divx and deleting the Mpeg2. For the files that lost detail info, what went wrong there and how do I avoid those shows from losing detail info? I think the answer to these questions require knowledge of what SageTV is doing and how it is doing it.

[nielm]

Originally Posted by mayamaniac

now this is interesting, why does it have to be on the same disk and what happens when it is not?

because its easier: moving the file between folders on the same disk is simply a fast rename operation, and not a slow copy from one disk to another + delete operation. It means also that sage will never see a file it cannot access (files in the process of being copied show up as unaccessible/size=0)

interesting that the watch status determines whether the show is stored in the DB.

Sage keeps watched, partially watched, shows marked with don't like, and shows with files in the DB -- and shows in the future of course: otherwise how would it know what you have watched/don't like!

What is happening is sort-of described in my How-to...

When Sage sees a *new* file in a recordings directory with the filename ending with -######-0, it looks for an airing with ID ######. If it finds one, and that airing does not have an existing file, then it links that airing with that file. This is called 'Media File recovery' (you may see that in the logs)

For files that lost the info there could be a variety of reasons:

Airing id in filename does not match real airing ID (can happen if something changed after the file got recorded), so mapping can never occur (why I say to check the airing ID in my howto)

Sage sees new file before it sees that old file had been deleted... So it cannot not map new file to the airing. Later it sees that the old file had been deleted; and then removes the shows' db entry. Because the new file has already been imported, it never sees it as a new file, and never maps the airing info to the new file. (why I say delete the file in Sage first, above)

[davephan]

Originally Posted by mayamaniac

When it comes to picture quality, it varies for each person's preference. This goes for audio compression (MP3) also. Some people can tell the difference between 128kbps and 320kbps, some can't.

What I did was used a one minute clip of video to do tests with divx converter. You can either edit out a one minute clip from existing show with some sort of video editor like VideoRedo or do a manual recording of one minute in length in SageTV. Use this clip to tests the various settings in Divx Converter to produce the quality you desire. I used the default "Home Theater" profile and pleased with the results, but they have other higher quality profiles also. With the one minute clip, its easy to calculate what the filesize of an actual full length show, just multiply the filesize x number of minutes.

Mayamaniac,

I tried using DivX Converter with the "Home Theatre" setting on several programs. The programs compressed to about 20% of their orginal size. I am able to play the files on my computer screen, but not on my TV. I am using SageTV version 5.0.4.92, the default STV, and a Hauppage PVR-350.

How can I get the files I converted to DivX to play back on the TV?


Dave

[blade]

Originally Posted by davephan

I am able to play the files on my computer screen, but not on my TV. I am using SageTV version 5.0.4.92, the default STV, and a Hauppage PVR-350.

How can I get the files I converted to DivX to play back on the TV?

Are you using the PVR-350 to output to the TV? It is a hardware decoder so you can only playback file types it supports and divx isn't one of them. I believe it can only playback mpeg2 files.

[mayamaniac]

yes, blade is correct. The 350 can only playback mpeg2 files. Buy an MVP Extender and license to playback many other transcoded file formats on your TV.

[mayamaniac]

Originally Posted by nielm

For files that lost the info there could be a variety of reasons:

Airing id in filename does not match real airing ID (can happen if something changed after the file got recorded), so mapping can never occur (why I say to check the airing ID in my howto)

Sage sees new file before it sees that old file had been deleted... So it cannot not map new file to the airing. Later it sees that the old file had been deleted; and then removes the shows' db entry. Because the new file has already been imported, it never sees it as a new file, and never maps the airing info to the new file. (why I say delete the file in Sage first, above)

Thanks for the explanation, that helped. I'll have to do some test to see what happens.