Commercial Advance Beta Release

[ellweber]

Jere,

DespHous.zip attached. Inclides comskip.ini.

Block 27 is the offending Ford spot. 12, 13, 24, 25 and 47 are also similar cases


Thanks,

Lynn

[Jere_Jones]

Looking at your log I see the following:

890) S: 58271 E: 58289 L: 19 LIFE IS A JOURNEY.
891) S: 58357 E: 58390 L: 42 A ROAD THAT CAN TAKE YOU ANYWHERE.	 
892) S: 58532 E: 58550 L: 19 DOWN MAIN STREETS,
893) S: 58641 E: 58662 L: 29 AND OVER BACK-COUNTRY LANES.
894) S: 58779 E: 58800 L: 28 WHERE IT GOES IS UP TO YOU.
895) S: 58892 E: 58937 L: 48 AND AS LONG AS YOU KEEP YOUR PROMISES,		 
896) S: 59051 E: 59084 L: 42 AND NEVER LOSE SIGHT OF YOUR GOALS,	 
897) S: 59143 E: 59157 L: 15 YOUR DREAMS AR
898) S: 59158 E: 59178 L: 27 E JUST AROUND THE CORNER. 
899) S: 59275 E: 59308 L: 38 WELCOME TO THE NEXTGENERATION OF FORD.
900) S: 59401 E: 59434 L: 40 AN UNPRECEDENTED SIXALL-NEW VEHICLES. 
901) S: 59549 E: 59565 L: 22 RIGHT HERE. RIGHT NOW.

This is the transcript from the Ford commercial. Since it is a captioned commercial, it should be easy to deal with. Open up your comskip.dictionary and add some of the phrases you see above. Capitalization doesn't matter. Personally, I recommend adding the following:

life is a journey
generation of ford
all-new vehicles
right here. right now.

Those 4 hits would increase the block's score by 21%.

Additionaly, for this show at least, I would change the score_percentile to .63 in the ini file.

Jere

[ellweber]

Thank for looking at this. Seems like maintaining the dictionary could be a major project in itself! Any chance of getting the table column headings for the log block list defined?

Lynn

[sleonard]

Originally Posted by ellweber

Thank for looking at this. Seems like maintaining the dictionary could be a major project in itself! Any chance of getting the table column headings for the log block list defined?

Lynn

Just a guess on my part but -

S: Start
E: End
L: Length

[ellweber]

I understand those, however, at the end of the log there is a table that shows the scoring of the blocks in detail. The column headings are: # sbf bs be fs fe s len scc scr cmbnd AR cc

I understand what most of these stand for but am unclear on a few, such as cmbnd, which probably means combined but I would like some more detail so that I can better understand the impact of changes on the .ini file.

Lynn

[Jere_Jones]

I posted some explainations in the ComSkip FAQ.

If you still have questions, I will try to answer anything specific.

Regarding the combined column, ComSkip will try to combine adjacent blocks to see if they meet a standard commercial length. This was done so that commercials with black frames (like movie promos) are still identified.

Jere

[Jere_Jones]

Is NOBODY having crashes with comskip?

Jere

[ellweber]

Thanks Jere

[krutaw]

Originally Posted by Jere_Jones

Is NOBODY having crashes with comskip?

Jere

Actually, after having taken a stance of not upgrading (because it was working) I went ahead and upgraded today, and I gotta say, this version rocks. I've only tested it with a couple of my "problem child" shows, but thus far it was performed magnificantly.

Good work!

[snipeman]

Originally Posted by Jere_Jones

Is NOBODY having crashes with comskip?

Jere

I have had one since I upgraded to the latest verision .62. Before that I was running a really early version dated from August. I had no crashes with the early version ever, and one in about a week with the newest version.

I'm thrilled with the software, and didn't want to complain. I only posted this because you were looking for feedback.


Andy

[Jere_Jones]

I am honestly suprised by the lack of crashes considering the severity of the bug I found.

If you are not experiencing any crashing and don't have any filenames that unicode characters in the filename, there is no reason to upgrade. But I found bugs that have the potential to be severe, so I'm posting this version.

Enjoy...
Jere

[justme]

Originally Posted by Jere_Jones

I posting this for those who are curious about the future of comskip.

The simple answer is that I'm only going to fix bugs. There will be a version 0.63 because I seemed to have introduced some bugs that are a "bugger" to track down.

Is that the end of commercial detection development? No. The current framework that comskip is built on is simpy too limiting, so I'm developing another program that will be far more flexible. It will not be open source since I'm developing it from the ground up. The next generation will be much more "resource friendly" and adaptable... So if you were saddened by the "demise" of comskip, take heart, better things are around the corner.

Jere

Very glad to here about you thinking about Comskip:TNG. I love your app and am looking forward to the future. Glad to here you'll be keeping the next gen opensource, as well.EDIT:Actually Jere said it won't be opensource. Thanks to sleonard, for catching my misread. Anyway, I have two questions:

1) Since you're considering a new version and your current version can read CC, I was wondering if you could dump the CC data as a SMI file sort of like mpg2srt does? It's not that mpg2srt doesn't work, it's that running two apps on the same file is a little redundant. I didn't ask for this before, but I looked at the data from a verbose dump(optimizing comskip) and realized that you seem to be scanning all the needed info, already. If comskip could generate the SMI file, this would reduce my CPU's load/usage and only need one app. Also since you're working on realtime commercial detection, realtime CC(like 2.1s unofficial CC) using SMI should be doable as well. I'm still use the SMI system for CC, since it works with everything(including playback outside of SageTV). I'm hopefull SMI CC would work with a MVP when I can afford one. I also use the smi files to mux subtitles into the compressed(ogm) files, I use to save shows I want to archive. I basically just like having a hard copy of the CC data, since you never know how it may come in usefull in the future.

Apologies if I'm totally missunderstanding the difficulty level of this, as I'm no coder. I also realize this would be a feature that would not be as widely used as other comskip features. Still I figured it couldn't hurt to ask, anyway.

2)Anyway, question 2, and no it's not dependant on you adding #1. Is there a way I can donate to your project? It won't be much, maybe $5-$10, but I still like to support opensource apps I use.

Thanks for you hard work.
/justme

[sleonard]

I think he wrote that the next gen will not be open source. It would seem he intends to keep ownership. He made no mention of how he would distribute it though. So it could be anywhere from free executable (no source code) to whatever the market could bear.

Personally I think its a wise move on his part. The fictional Billionare in Carl Sagans "Contact" who solves the code for physicist Ellie Arroway (Jodie Foster in the movie) made his billions by inventing an algorithm that accurately zapped commercials

[justme]

I think he wrote that the next gen will not be open source.

Yep you're right. Thanks for catching that. I guess my mind was just reading what it wanted to see. I'm a big opensource fan. Still I trust he'll do right by us, with whatever he comes out with next.

/justme

[sleonard]

Jere, since you are writing a brand new version of comskip I'd like to start the feature requests. Some of these may not be possible due to Sage and STV limitations but here goes.

1) Ability to run on a Sage Server as it's own service or under the Sage service like the web server does.

Pros
a) This would allow all comskipping to be done automatically on the server without having the UI nor any clients running. I run Sage Server as a service and don't even log in unless I'm doing maintenance and the Sage clients are not always on.

b) This would reduce or eliminate dependance on 3rd party STV's.

Cons
a) May not be possible to communicate with Sage to get necessary info if it is a seperate service OR
b) may have to write the app in java to run under the Sage service like the web server (I have no clue myself).
c)This would also require added processing power on the server.

2) The ability to create profiles for either specific channels or shows and some way to give feedback to comskip on mistakes. For example, if I notice that it marked some frames as commercial that it shouldn't have there would be a way to let it know that info and rerun detection. Each time it is rerun with error feedback it would make the profile more accurate. Eventually we would have perfect profiles that could be shared among the user community.

[Jere_Jones]

Originally Posted by sleonard

Jere, since you are writing a brand new version of comskip I'd like to start the feature requests. Some of these may not be possible due to Sage and STV limitations but here goes.

Cool. I like requests. I can't think of everything.

Originally Posted by sleonard

1) Ability to run on a Sage Server as it's own service or under the Sage service like the web server does.

I see where you're coming from. This will be kindof a comskip manager service. It would monitor certain folders and if a new mpg showed up, it would start processing it.

Originally Posted by sleonard

Pros
a) This would allow all comskipping to be done automatically on the server without having the UI nor any clients running. I run Sage Server as a service and don't even log in unless I'm doing maintenance and the Sage clients are not always on.

b) This would reduce or eliminate dependance on 3rd party STV's.

All good points.

Originally Posted by sleonard

Cons
a) May not be possible to communicate with Sage to get necessary info if it is a seperate service OR

I've already been thinking about this. I'm imagining a conversation like this between Sage and ComSkip:

SageTV: Hey, I'm starting to watch CSI:NYSpeedKills-034495-0.
ComSkip: Cool, I'm processing that file right now, so I'll up the priority of that file.
{later}
SageTV: Hey, I'm done watching that show.
ComSkip: Thanks. Now I can concentrate on other shows,too.

Originally Posted by sleonard

b) may have to write the app in java to run under the Sage service like the web server (I have no clue myself).

Not a chance!

Originally Posted by sleonard

c)This would also require added processing power on the server.

More than you know. The current ComSkip uses a very stripped down MPEG decoder that is very fast. The ComSkipTNG is going to be a DirectShow application (necessary to process files of different codecs), and the best speed I can get is only half what I get now. Ugh...

Originally Posted by sleonard

2) The ability to create profiles for either specific channels or shows and some way to give feedback to comskip on mistakes. For example, if I notice that it marked some frames as commercial that it shouldn't have there would be a way to let it know that info and rerun detection. Each time it is rerun with error feedback it would make the profile more accurate. Eventually we would have perfect profiles that could be shared among the user community.

The profiles are easy. I'll have to figure someway for ComSkip to get information out of SageTV on demand. Something like:

ComSkip: Hey, what channel is CSI:NYSpeedKills-034495-0 being recorded from?
SageTV: Channel 57:CBS
ComSkip: Cool, thanks. How long is it?
SageTV: 1 hour, but it has 3 minutes of padding on each side.
ComSkip: Padding? Ugh. That always throws off my calculations. Thanks.

The feed back is a little harder, but is needed. It's on the backburner for thought.

Jere

[Cayars]

Jere, the new setting in the ini file "livetv". I noticed that this slows down processing a lot. To the point that it's taking 4 times longer to process on my machine. It took 4 hours to process a show that was already recorded. At present I can't use it for "real-time" shows.

Is there any down side to setting this to 0 and still having it run on files being recorded?

For shows being recorded, is it best to have the 2 pass stuff turned off?

If the shows have CC data in them is there any setting you can recommend to get it to run as fast as possible for real-time processing?

[Jere_Jones]

Originally Posted by justme

1) Since you're considering a new version and your current version can read CC, I was wondering if you could dump the CC data as a SMI file sort of like mpg2srt does?

Extracting the text is easy. Separating the garbage from the good stuff is a lot harder. Mpg2Srt ignores characters that comskip thinks are actual dialog and I haven't been able to figure out why. I originally thought it would be relatively easy. Now, I'm not too sure, but it is on the todo list. I can tell you it won't happen in the GPL'd comskip. I'll be maintaining it for bugfixes and to keep it compatible (and useful) with current and upcoming STV's, but I don't think I'm going to put any more major features in it.

Originally Posted by justme

I basically just like having a hard copy of the CC data, since you never know how it may come in usefull in the future.

What about translation to other languages? I played around with running the srt's through a translator so my wife can enjoy CSI with me. It worked amazingly well.

Originally Posted by justme

Apologies if I'm totally missunderstanding the difficulty level of this, as I'm no coder. I also realize this would be a feature that would not be as widely used as other comskip features. Still I figured it couldn't hurt to ask, anyway.

I'm sure that it will be in there eventually.

Originally Posted by justme

2)Anyway, question 2, and no it's not dependant on you adding #1. Is there a way I can donate to your project? It won't be much, maybe $5-$10, but I still like to support opensource apps I use.

I hadn't thought about it. ComSkip as it is uses so much code from other people that I don't know if you should contribute to me, malore, the mpeg2dec guys, or who. Bear in mind that even though I have taken comskip to some pretty sophisticated levels, I wouldn't have done any of it if malore hadn't laid the groundwork. I thought about it, but it was just "WAY" to overwhelming before I actually tried it.

Having said that, I'll probably open up a paypal account and pass that to y'all.

Originally Posted by justme

Thanks for you hard work.
/justme

Thanks for the kind words.

Jere

[Jere_Jones]

Originally Posted by Cayars

Jere, the new setting in the ini file "livetv". I noticed that this slows down processing a lot. To the point that it's taking 4 times longer to process on my machine. It took 4 hours to process a show that was already recorded. At present I can't use it for "real-time" shows.

The livetv setting turns on/off writting the commercials to the drive as they are found. It does greatly increase the I/O operations because it recalculates the commercials everytime a black frame is found. Everytime it recalculates the commercials, it rewrites it to the drive. I'll take a look at this and see what I can do.

Originally Posted by Cayars

Is there any down side to setting this to 0 and still having it run on files being recorded?

Just that commercials won't be written to the file until it finishes. Other than that, no.

Originally Posted by Cayars

For shows being recorded, is it best to have the 2 pass stuff turned off?

I always recommend 2 pass on. Since it takes comskip a little while to lock on to a logo, 2 pass runs the file again with the logo mask from the start. 2 pass doesn't mean that it processes the file fully twice. It only repeats the part of the show that it needed to lock on to the logo.

Originally Posted by Cayars

If the shows have CC data in them is there any setting you can recommend to get it to run as fast as possible for real-time processing?

I don't understand what CC data has to do with real time processing.

Jere

[Jere_Jones]

This is a slight adjustment to the livetv code. The I/O resource requirements are DRASTICALLY reduced with this one.

Jere