Commercial Advance Beta Release

[Deadbolt]

Malore,

Several questions --
1. When SageTV autodeletes a file (either IR or for a favorite) does it automatically delete the .txt file also? That would be nice as it would keep from them building up.
2. What happens when the auto CA tries to process a show with multiple parts...a show that is broken up by interuptions (noted by title_0,title _1,title _2,etc.)?
3. When using the SageClient...if the server is recording something, the server is not set to process CA, but the client is...will the client process the file? Does the recording/awake param in the properties file on the client switch refer to the server's state or the client's?

Thanks!

[malore]

Deadbolt,
1. Since the files are deleted automatically, my code isn't executed so the .txt file isn't deleted. It would be possible for someone to make a utility to delete any .txt files that don't have matching .mpg files.
2. Only processes the 1st file, I don't currently have any files like this, but I would guess that it would work for the duration of the 1st part. It would take a lot of extra work to support split shows, which are probably messed up anyway, so commercial detection wouldn't work correctly anyway.
3. It checks the recording schedule which is for the server and clients, so if one computer is recording, they should all wait to do commercial processing.

[Deadbolt]

Thanks Malore,

The code works great! One feature that would be great is the ability to not process selected channels (ex. HBO). Maybe it just puts a blank .txt file when it sees it or ignores it. This way channels that don't have commercials don't waste processing time and also don't have false positives.

Edit: Also, is there a way to cutomize the number of "saftey" seconds before and after an autoskip? Before you mentioned the 4 sec error at the beginning of the commerical due to time vs. frame differences. Is there a way to customize this margin? Same with the end of the commercial?

[BerrantRyke]

Originally posted by Deadbolt One feature that would be great is the ability to not process selected channels (ex. HBO).

YES! I'd like to leave the commercials in the kids programs. Even the commercials keep them busy for a while....

[beautye350]

First I would like to thank you guys for your hard work on this. I really love this feature. Now that I've buttered ya up I've noticed that the detection is off on some shows/channels. In comparing the result I get with Sage and this STV against my VCR that has this feature I am seeing my VCR have much better detection. In further research, I realize that my VCR is marking the tape going backwards. Would it be possible to alter comskip to mark the .mpg files backwards in an attempt to try and get better detection...assuming, of course, that is not how it does the detection now.

Thanks again.



Regards,

Beau

P.S. Now if we could just combine this with the STV for automatic compression I will have a complete product.

[malore]

Originally posted by beautye350
First I would like to thank you guys for your hard work on this. I really love this feature. Now that I've buttered ya up I've noticed that the detection is off on some shows/channels. In comparing the result I get with Sage and this STV against my VCR that has this feature I am seeing my VCR have much better detection. In further research, I realize that my VCR is marking the tape going backwards. Would it be possible to alter comskip to mark the .mpg files backwards in an attempt to try and get better detection...assuming, of course, that is not how it does the detection now.

Guys? I guess that would be me, myself, and I. (Of course, credit also goes to the developers of libmpeg2 and MythTV)

At the moment I'm just using black scene detection, your VCR may also be using sound levels or other additional techniques. Does the VCR actually do the detection backwards or does it just go back and mark all the position it found while recording forwards?

Anyway, comskip first processes the entire file noting the locations of all the black frames. Then it processes this list using the criteria found in the comskip.ini file. Since you've just volunteered, let me know when you've found better settings that work on your problem files and still work on regular files. I'm kind of disappointed that after taking the time to make it so anyone can make improvement, as far as I know, nobody has.

[Droid]

This is an awsome feature... Thanks malore!

I am however hoving trouble. I installed it, I get the menus and it is processing. The problem is that the markers are not at comercial breaks. They seem to be at random places in the shows. Could this be due to noise on the cable? ( I am running digital settop to my EPIA and Svideo and the picture looks perfect). What would cause this to be simply wrong on detection?

[malore]

Droid, great I have another volunteer.

Since my time and resources are limited, I’ve moved the criteria, that comskip uses to mark segments between black frames as commercials, into the comskip.ini settings file. Edit comskip.ini file with a text editor (notepad will work) and see if you can find settings that work better for you. The .txt files that comskip outputs are just a list of start and end frames for commercial breaks. I've been using VirtualDub-MPEG2 to double check the results. I like it because it's freeware and lets you jump to an exact frame number.

Once you get comskip giving you good results, attach the comskip.ini file to a post and then others can benefit from your work. I look forward to seeing what you come up with.

Basically, if I do all the testing then settings are going to be skewed to work best with the shows I watch, on the channels I watch, on the recording quality I use, etc. etc. It will be interesting to see if with everyones help (hint, hint) , there is a universal best settings or if everyone is best tweaking the settings for themselves.

[Droid]

Glad to be a volunteer Will let you know the results.

[Droid]

I am now getting comskip dieing. Everytime comskip starts to run (I am guessing) I get and error dialog saying comskip died and do I want to error report.

[malore]

Did this happen after you edited the .ini file? The .ini reader might not be very robust. Attach the problem .ini file to a post, so I can look at it. Then replace comskip.ini with the file from the malorebeta.zip file.

If that is not the case you have to give me more information about what you are doing.

[Jere_Jones]

malore,

Can you post the settings that the original comskip used? I seem to get better results with it than any settings that I can invent.

However, using the "scientific method," (and the last 8 hours of analyizing the same shows over and over and over...) these settings mark the most commercials without ever marking any of the actual show.

verbose=1 ;show extra info
fps=29.97 ;frames per second (NTSC=29.970, PAL=25)
border=10 ;border around edge of video to ignore
max_brightness=120 ;frame not black if any pixels checked are greater than this (scale 0 to 255)
test_brightness=80 ;frame not pure black if any pixels checked are greater than this, will check average brightness (scale 0 to 255)
max_avg_brightness=15 ;maximum average brightness for a dim frame to be considered black (scale 0 to 255)
max_commercialbreak=240 ;maximum length in seconds to consider a segment a commercial break
min_commercialbreak=15 ;minimum length in seconds to consider a segment a commercial break
max_commercial_size=62 ;maximum time in seconds for a single commercial or multiple commercials if no breaks in between
min_commercial_size=5 ;mimimum time in seconds for a single commercial
require_div5=0 ;0=off, set=1 to only mark breaks divisible by 5 as a commercial. Standard sizes (15,20,30,45,60,90,120 second)

I haven't messed with any of the frame identification settings because they seem to work flawlessly unless the recording is REALLY noisy. I only messed with the timing logic.

For what I record, I've found that setting div5=1 results in a lot of commercials not being marked. Has this setting resulted in better detection for anyone?

I'm also interested in what y'all've come up with.

Jere

[ToxMox]

So far I've found that the original comskip worked better than the newer one. I've tweaked the original now and will test that one for a day or so and post my results if it works out.

[Droid]

Unfortunatly this is prior to editing the file at all. I rebooted and undid the sage.properties setting of allowing processing while recording. That was the only thing I had changed. Will post tomorrow if this fixed it.

Update: comskip has stopped crashing. It must be related to the sage.properties background_commercial_processing_recording=true
I set this to false and it has been stable.

[beautye350]

Originally posted by malore
Guys? I guess that would be me, myself, and I. (Of course, credit also goes to the developers of libmpeg2 and MythTV)

At the moment I'm just using black scene detection, your VCR may also be using sound levels or other additional techniques. Does the VCR actually do the detection backwards or does it just go back and mark all the position it found while recording forwards?

Anyway, comskip first processes the entire file noting the locations of all the black frames. Then it processes this list using the criteria found in the comskip.ini file. Since you've just volunteered, let me know when you've found better settings that work on your problem files and still work on regular files. I'm kind of disappointed that after taking the time to make it so anyone can make improvement, as far as I know, nobody has.

I assume that the process by which my VCR does the marking is proprietary so I cannot answer for sure what the process is, but if I had to guess I would say that it is doing it backwards as I don't believe that the VCR has the ability to "store" any data for later retrieval to assist in marking. I also believe that marking in reverse assures that you get the black scene that happens right before the start of your target show and not one in the middle of the commercial block.

I think part of the reason my detection is not as optimal as it could be is that my current cable provider is low rent (read: a small player in the cable provider ranks). I'll explain:

For instance I watch Stargate SG1 on Sci Fi. At the commercial breaks, I get what first appear to be national run commercials and then sometimes in this block, I will see a local business commercial and this is usually where the problem comes from as my cable provider does not seem to be placing the commercial in the stream properly. Meaning sometimes you will see a few seconds of another commercial and then the local one plays like they are imposing the local over a national feed or something. This, I believe, is part of the problem as the black scenes sometimes show up in the middle of the "block" sometimes. Therefore maybe going in reverse would catch the last black scene before the desired programming material.

FYI, Sci Fi appears to be the worst channel.

On the other hand, I like to watch Celebrity Poker Showdown on Bravo and it detects the commercials PERFECTLY.

In my suggestion I was just throwing out another opinion or a brain nugget. I am happy with the results, just thought that maybe it could be better.

I will see what I can do on my own and I will of course share anything I find.


Regards,

Beau

[krutaw]

Originally posted by malore
I'm kind of disappointed that after taking the time to make it so anyone can make improvement, as far as I know, nobody has.

As a fellow developer, I completely understand your angst, and wish I had been able to test sooner. Fear not, you have another tester now.

[krutaw]

Has anyone else noticed a "freeze-up" on the tv-out of the pvr-350 when using Malore's stv for commercial advance? I thought it was just something I was doing at first until I rolled back to the stv I was using previous to finding the commercial advance beta release. After rolling back, the "freeze-up" issue stopped.

When the tv-out on the pvr-350 freezes up, I have to reload the video player to get the video back, and then have to re-enable the UI out on the pvr-350 to be able to see the menus.

Here's my system specs in case anyone's curious:

AMD Athlon 1000mhz
MSI motherboard (Nforce2 chipset)
512mb ram
100gb IDE Drive
250gb IDE Drive
(4) 40gb IDE Drives (mapped via network)

The videos lock up whether I'm watching a video via a local drive or a networked drive.

[Deadbolt]

I was just about to post this Krutaw! Exact same experience here! My system was fairly solid before...about 2 weeks between loosing pvr-350 tv-out/gui out. Now I get it 2-3 times per hour! I like the comercial skip, works great but can't use it at the expense of stability.

I am using a client to network comskip in the background, not comercial processing done on server, and autoskip on server is enabled. I watch from the server on the pvr-350.

[krutaw]

Originally posted by Deadbolt
I was just about to post this Krutaw! Exact same experience here! My system was fairly solid before...about 2 weeks between loosing pvr-350 tv-out/gui out. Now I get it 2-3 times per hour! I like the comercial skip, works great but can't use it at the expense of stability.

I am using a client to network comskip in the background, not comercial processing done on server, and autoskip on server is enabled. I watch from the server on the pvr-350.

No offense, but thank god it's not just me. My system is the server since I haven't upgraded the client to have a pvr-350 (YET!!!). I'm gonna try updating my drivers to the latest version, I believe right now I'm using 1.8.22020.

[mls]

Correct me if I'm wrong here, but I've always thought devices that auto-skipped commercials did so by monitoring data in the vertical blanking interval (such as rating, closed captioning, show title, station ID, etc.).

After the law passed for the V-chip and ratings requirement, the rating would seem to be the easiest to monitor (since it should be there during a show, but not during a commercial).

When the rating data is missing the device would just fast forward to wherever it came back again (seems simple enough).

Checking for blank or black spots doesn't sound like it would be very usefull since shows often fade in/out or have blank spots between scene changes.

However, it really doesn't matter how you try to do it because they (the broadcasters) know people are always trying to skip commercials.

I've noticed on my TV (which can be set to display all that extra data) that sometimes even in the middle of a show the rating will disappear for a while. Likewise, the show title or station ID sometimes goes out and comes back later.

In other words, the broadcasters know what we're trying to do so they modifiy things on their end to make sure any commercial skip feature will not always work correctly.

I know the CC data can be captured with Hauppauge encoders, but I don't know if enough of the other vertical blanking interval data is included to make use of it for commercial skip.

Anyway, that's just how I've assumed it was being done in other devices. Don't know if those methods could be used with SageTV?