Utility: Comskip

[erik]

Lately I have not been tracking this forum.

If there is interrest I would not mind posting a message here when new functionality becomes available.

In case you want to know more about how Comskip develops (such as support for HD and visual tuning tools) I suggest you monitor the Comskip forum at http://mk.kaashoek.com/comskip/

Latest release notes

Changes in 0.79.24
- Made a big change to improve finding weak logo's (like on NZ's channel ONE)
- Some heuristics improvements for very short segments
- Comskip will check you logo_threshold setting and suggest a value in the log file. Can work only on recordings of at least 45 minutes.
- Fixed a buf that could cause the wrong ini file to be used
Changes in 0.79.23
- Added the -t commandline switch to enable Transport Stream input. TS files can have .ts or .mpg as an extension
- Files with a .ts extension are assumed to be a Transport Stream
- The debugwindow can show graphs of all measured parameters, use 'G' key to toggle
- It is now possible to zoom into the timeline fo accurate positioning of cutpoints, 'Z' to zoom, 'U' to unzoom
Changes in 0.79.22
- Fixed a bug in the .vcf output format
Changes in 0.79.21
- Corrected a bug causing comskip to crash when no EDL output was created.
Changes in 0.79.20
- Cutlist changes will be immidiately shown on the timeline
- New outputfile format .vcf for Virtualdub. Enable with output_vcf=1 in the ini file.
Changes in 0.79.19
- Improved identification of commercials
- Added a play_nice_sleep parameter to the ini file to allow configuration of the processor load
- Fixed a bug causing play_nice_start and play_nice_end in the ini file not to work .
Changes in 0.79.18
- Solved a crash when AR was disabled in detect_method
- Improved the heuristics for before_logo and after_logo
Changes in 0.79.17
- Any two logo segments with a gap less then min_commercial_size will be joined to one logo segment
- Any logo segment less then min_commercial_size will be deleted
- Added a delete_show_before_first_commercial ini parameter with default value of 0, usefull if you have a lot of prepadding.
- Improved the after_logo behaviour to set cutpoint at silence and maximum scene change in a 30 seconds windows after the after_logo, this solves the problem of a commercial break without starting black frame.
Changes in 0.79.16
- Extensive recheck of all the scoring and heuristics rules resulting in many small corrections
Changes in 0.79.15
- Fix a bug causing wrong aspect ratio processing for recordings with a low amount of horizontal pixels
- Added the posibility to always force cutting on aspect ratio by setting cut_on_ar_change=2
Changes in 0.79.13
- Fixed the overlapping of the last two last commercial segments when the last commercial found ended at the end of the recording.
- Fixed a wrong configuration causing the generation of a very large timing.csv file
Changes in 0.79.12
- Change the debugwindow title to Comskip so it can be found by other programs
- Increased the size of the commercial bar when using a .txt file as input to avoid overscan problems
- Introduced a possible fix for a bug that caused Comskip to exit when invoked with an empty or incorrect .txt file.
Changes in 0.79.11
- Fixed a bug causing deletion of of the EDL file when using a txt file to inspect the found commercials
- Fixed a bug causing incorrect handling of the final 1 frame commercial.
Changes in 0.79.10
- Remember to us a 4 times higher non_uniformity setting when recording HD!
- Both the edl file and the txt file have as last entry a one frame size commercial at the end of the recording
- Now you can set the cuttermaran command options in cuttermaran_options in the ini file
- The ini file is first searched for in the directory where the mpg file resides so you can automatically have multiple ini files.
- The current block is highlighted in the timeline in the debugwindow
- The csvout command ine switch is working again
Changes in 0.79.8
- Changed the numbering in the womble clip list to work around a womble bug
Changes in 0.79.7
- Improved the delete_show_before_or_after_current processing to deal with commercials up to min_commercial_size seperating the current and the previous/next show
- Solved a bug that was causing Comskip to crash on certain recordings
- Improved the display of found logo in the display bar. No more false positives
Changes in 0.79.06
- Fixex a hangup in debug window when the timeline of the mpg is corrupt
Changes in 0.79.05
- Fixed a bug causing a crash with 60 fps material.
Changes in 0.79.04
- Improvements in the heuristics
Changes in 0.79.01
- Can process mpeg video with size up to 2000 by 1200
- Moved the info bar in the debug window to above the video
- Enabled the scaling of video in debugwindow, anything above 800x600 will be halved in size
Changes in 0.78.02
- Solved a bug where setting a new 'b'eginning of a commercial the written cutpoint was 10 frames earlier
- Solved a bug where after editing the cutpoints all output formats that where in time instead of frames could be wrong.
- Change the format of the TXT file to include the framerate to solve the bug above.
- Solved a bug where the output files where cleared if no 'w' was used when reviewing the .txt file.
Changes in 0.78.01
- Writing a new cutlist writes all selected output files instead of only the txt file
- Substantial changes to heuristics.
Changes in 0.77.77
- Added the output_edl parameter to generate the edit decision list for mencoder so you can cut and convert at the same time.
Changes in 0.77.76
- Solved a bug that might cause a crash when live_tv is on on a very long recording
- Solved a bug that might cause a crash when the timeline of the input mpeg is severily corrupt
- Improved the heuristics related to Aspect Ratio mismatch
- Improved the heuristics related to logo changes
Changes in 0.77.68
- Fixed a bug in the support for cuttermaran
- Added the information required to run cuttermaran from the commandline
Changes in 0.77.66
- Added support for output for cuttermaran
Changes in 0.77.65
- Added the ticker_tape option to ignore a certain amount of pixels at the bottom. Nice if you have a continues ticker tape or fixed logo at the bottom and still want to detect black frames for the rest of the screen.
Changes in 0.77.61
- Solved a bug in sound processing causing too many black frames to be detected.
Changes in 0.77.59
- Its is now possible to review the .txt file by running comskip on it such as: comskip filename.txt
- Now it is possible to edit the begin, end and number of commercials in the debugwindow and write out an updated .txt file, also for output of older Comskip versions.
- Added the shrink_logo parameter to reduce the duration of the logo if it extends into the commercial
- The ESCAPE key can be used to exit comskip in debugwindow mode
- Corrected a bug that prevented the reviewing of .txt files of long recordings
- Corrected a bug in the CSV file causing problems with NTSC
- Corrected a bug in shrink_logo processing which could cause a crash
Changes in 0.77.52
- Fixed a buf in the application of padding
- Corrected a bug that prevented the after_logo cutpoints to be shown in the debugwindow
- Added a debug window showing in graphical way the processing and results of Comskip, enable with -w.
- Added intelligence for situations with logo only on commercials, logo everywhere and different logo's on commercials and show.
- Added intelligence for detecting the end of the previous show and the beginning of the next show
- Fps is now read from the mpg file, no need to adjust it anymore.
- If a .ref file with the same name and structure as the .txt file is present in the directory of the mpg file then it is used for comparison
- When verbose=10 a histogram of the volume is written to the log to help determin the max_volume
- Added cut_on_ar_change parameter in ini file
- live_tv=1 works again
- Added delete_show_after_last_commercial to do what is says.
- Added a commandline option to set the logo file so a learned logo can be re-used. --logo=filename.logo.txt
- Multiple ini files can be used with the --ini=filename.ini commandline switch. Use --ini=comskip%2.ini in postprocessing.bat to have a per channel ini file.
- Added the possibility to review the output of comskip in the debugwindow. When finished positioning the mpeg is possible with cursor keys and mouse.

Changes in 0.74
- Corrected a bug that prevented the use of the minimum commercial size
- Improved the comparison with the reference file

Changes in 0.73
- Added sound processing
- Improved blackframe and logo processing which makes tuning less important.
- Speed has not improved due to the added processing.
- Framerate is now automatically read from the mpeg file, no need to set in the ini.
- Did a lot on the block algorithm which may not have improved the performance for you ;-)
-

Changes in 0.72
- Fixed a bug causing noisy black frames not to be recognized, this can improve performance considerable!
- Changed the CSV file to use ';' as separator so you can load it into excel without conversion.

Changes in 0.71
- Many changes in the log file, most specific in the final block list, hope you like it!

Changes in 0.70
- Any frame with a non_uniformity below the threshold will be used as a cut point. So pur blue or pure white will work. Default is 100.
- A show_percentile below about 0.2 will be added to the logo_percentile to get the real show_percentile. Usefulle for shows that take some time before the logo appears
- Determining show/commercials for a recording without logo with many very darks scenes is not working, but then I don't know how this will ever work.
- Some improvements in the log file (mostly for myself).

Changes 0.69
- About 25% faster then comskip 0.64
- Logo learning is substantial more robust leading to much better logo recognition
- The written framearray now also shows the (aspect_ratio-1)*100
- Corrected some stupid errors that where leading to a bad performance.
- The show_percentile is automatically calculated from the logo_percentage when sufficient logo is found.
- experimental code for settings generating cutpoints before and after a logo, value is in seconds, should be 10 or bigger, 0 disables
- have installed automatic regression testing on a test set of 10 shows to prevent future stupid errors

Changes 0.68
- The written framearray can not be read back!!! You can import it in a spreadsheet to do extensive evaluation of blackframe detection, logo detection and aspect ratio detection
- The written framearray now includes the variable "uniform" that shows the quality of the black frames.
- The saved logo file combines horizontal and vertical edges in one presentation. Should be easier to read
- Ascpect Ratio changes do generate candidate cutpoints
- Logo processing shows time before logo appears and time after logo disappearing.
- Scene Change detection seems to be based on wrong assumptions, please do not use it.
- Debug level above 2 shows the Long/Short block ratio per long block to support blackframe detection tuning. The ratio should be above 50 percent.

Changes 0.67
- Fixed the bug that prevented the scene change rate to be used in the calculation
- Now logo's at the right side of the screen are also detected
- A new parameter for the ini files is added ar_split, it sets the ration above which it is assumed to be widescreen
If you see too many AR segments increase the minimum commercial length

No dictonary is included, please use your old one.
The comskip.ini included is mine, please adapt it to your settings!!!!!!

[evilpenguin]

Wow, thanks for the heads up

I'll have to check this new TS support out tonight.

[evilpenguin]

Wow, i'm impressed with the newest comskip. It didn't do too well on HDTV 24, but it nailed HDTV Gilmore girls. Great work

[erik]

Could you in return help me to improve Comskip by mailing me the .csv file and a corrected .txt file of HDTV24?

Please follow the procedure described here

[evilpenguin]

No problem, i'll get you the info tonight.

[clmolnar]

I am using this newest version of comskip along with SageMC 16x9 STV and for some reason, it is not being run by SageTV automatically. Has anything changed in the syntax for running comskip that might have caused this?

If I revert to an older version, say .54 or so, everything works fine.

Thanks for any help.

[erik]

Nothing has changed as far as I know.
Any possibility to log something?
Other peope seem to be able to run the new version.
Any particular directory path names?
Do you know chich command line parameters are being used?

[clmolnar]

It seems to have corrected itself after I updated to the beta version of SageTV. I'm not sure what the problem might have been.

[erik]

Comskip build 26 is available

There will be no support during the next couple of days.

From the changelog

Changes in 0.79.26
- Added scoring modifiers based on comparing this block to the average value of Brightness, Non-Uniformity, Volume, Non Silence and Scene change rate.
- Added parameters to controll this scoring: punish, punish_threshold and punish_modifier, see ini file for details.
- Added graphs in Debugwindow to show where parameters are above threshold for scoring, indicated by red lines.
- Warning: Do not blindly enable these because you assume commercials are louder, brighter, etc... Do check your assumptions. I have not yet found a foolproof approach to have them always enabled.
- No longer required to have Black Frame detection enabled. Certain recordings work better without them

[Fluffdaddy]

Thank You

[erik]

Originally Posted by evilpenguin

No problem, i'll get you the info tonight.

Did I receive something?

[erik]

The newest Comskip release can read also DVR-MS and MPEG TS files.
The TS files must contains only one show, possibly in multiple resolutions and with multiple audio formats.
The DVR-MS files must contain regular MPEG2 video and either MPEG2 or AC3 audio.
More information is available on http://mk.kaashoek.com/comskip

[erik]

After some serious work on the MPEG decoder I think I have solved many of the audio/video synchronisation and timeline problems.
Some serious MPEG Transport Stream and DVR-MS testing could further improve Comskip so keep sending me your feedback.

From the change log

Changes in 0.79.32
- Added the BTV Smartskip output format, set output_btv=1 in the comskip.ini file to enable
- Bug solved: AC3 sound volume is too high causing missing of black frames
- Bug solved: Recording with 3/2 pull down has wrong frame numbers leading to incorrect cut points
- Bug solved: The volume_slip default value is too low
- Added parameter in ini file: ms_audio_delay=5 set to the number of frames audio lags video in dvr-ms files. Default value of 5 should be OK for most recordings.

[erik]

made an error during building.
If your comskip.exe does not do anything but exit, download build 32 again.

[dblaine2]

Is this better than SA?

[erik]

There are people who tried both, I rather have them comment.

[blade]

Originally Posted by dblaine2

Is this better than SA?

I've used both and SA was much faster than comskip and was just as accurate. I haven't tried comskip in a long time so I don't know if it's any faster or more accurate than it used to be. I don't think SA supports HD yet, but I may be mistaken.

[dblaine2]

The reason I ask is it seems SA is getting worse at detecting commercials and it seems from this thread that comskip is adding more detection methods, etc.

So if someone is using this newest version of comskip how is it working for you? Is it fast?

The one thing I like about SA is the GUI. If I recall correctly comskip still relies on manually setting up a config file (?)

[erik]

Reboot at the GBPVR forum has been using both.
You could ask him

[Fluffdaddy]

Originally Posted by dblaine2

The reason I ask is it seems SA is getting worse at detecting commercials and it seems from this thread that comskip is adding more detection methods, etc.

So if someone is using this newest version of comskip how is it working for you? Is it fast?

The one thing I like about SA is the GUI. If I recall correctly comskip still relies on manually setting up a config file (?)

Updated and been using the latest comskip for 4 days now, and it works fine for me.

The only thing I notice is it shows maybe half a second of a commercial before it skips. Any user-end fix for this ?

What do you mean................is it fast ?

Overall I been more then happy with erik's comskip work

thx Erik