Utility: Comskip

[erik]

Take build 133
http://www.kaashoek.com/files/comskip79_133.zip

make sure you have in your comskip.ini file
skip_b_frames=0
Then look at the detect_method setting and make sure it contains the 16=closed captions method.
If you don't know how to check set
detect_method=255
but remember to set it back after testing!
and set
verbose=10

Then drag and drop a couple of recordings on comskip.exe extracted from the build 133 zip
Does not matter if its dvr-ms, mpg or ts. If You regular comskip does process the recordings this one should also be able to do so.
If you are using dvr-ms and don't want to wait for comskip to start due to its regular "process during recording" delay you may set
dvrms_live_tv_retries=1
otherwise comskip may wait 300 seconds before it starts to do some real work.

When everything is done look at the generated .log file with the same name as the recording.
Open the file and look at the end for "Closed caption transcript"
Beneath there should be the full transcript of the close caption in the recording.
The transcript should be readable and make sense. If its garbled I need a short part of the recording to do some more tests.
Then look for lines starting with XDS, they come in pairs. The first with numbers and the second letters like
XDS: 1 83 d3 4f ce 49 43 20 58 80 8f
XDS: SONIC_X y
If you see them I like you to copy all the lines with XDS into a mail (or post in this thread)
Then you may have set
output_srt=1
or
output_smi=1
and see if there have been generated .srt or .smi file and see if they have the correct content.
If you do not know what the correct content should be, don't bother, as I also don't know

If there is no CC and no XDS set
output_data=1
and see if a .dmp file gets generated when processing the recording through comskip.
If no .dmp file is generated the recording does not have a user data stream and so no CC or XDS.
If there is a dmp file but no CC or XDS in the log file I like to get the first say 10kBytes of the .dmp file.

[erik]

Build 134 is available

From the changelog
Changes in 0.79.134
- Change the extension of the XDS dump file to .data
- When comskip is started with a .csv file the .data file containing the XDS dump will also be read when available.
- In Debugwindow the 'X' key can be used to show XDS data. Currently the Program Name, V_Chips, Program duration and Program position are displayed. Decoding of the XDS data is not yet complete
- A very thin grey vertical line at the very top of the debugbar timeline shows the location of the XDS data changes

Could someone set
output_framearray=1
output_data=1
and enable the Close Caption processing and upload for the a file with a lot of XDS both the generated .csv and .data to my ftp server?

I prefer a recording with V-Chip data that changes during show/commercial and that shows program position information.
This will allow me to test additional detection rules using the XDS data

[erik]

From the changelog

Changes in 0.79.135
- Improved the CC word separation on certain broadcast channels
- (Re)added the "CC" type of CC data. All digital recorded ATSC channels now should have CC.

[erik]

Many improvements done, in particular solving some crashes

Changes in 0.79.144
- Bug solved: Comskip crashes on recordings with mono AC3 sound
Changes in 0.79.143
- Bug solved: When remove_before is set to a positive value and the first commercial starts at the beginning of the recording then the first commercial is missing in the generated list of commercials.
- It is on request again possible to set enable_mencoder_pts=1 in order to use the initial PTS as an offset in the .edl output file to allow mencoder to correctly cut TS recordings.
Changes in 0.79.142
- Made a big change to the sound detection algorithm. This should solve some crashes on certain recordings.
Changes in 0.79.141
- Bug solved: Too short commercials at the beginning or end are not deleted even when shorter then min_commercialbreak
- Substantial changes to the heuristics and to the validation algorithm. Its better on my set of test files, how about yours?
- Switched back to the old compiler
- Bug solved: Can not disable heuristics rule 7.
Changes in 0.79.140
- When the debugwindow is uncovered the windows stayed black untill you hit a key or clicked on the time bar. I tried to show the debuginfo when the window is uncovered but it is not always working.
- Switched to a new compiler, did expect all kind of problems but haven't seen one yet.
- Bug solved: Heuristics rule 7 triggers incorrectly causing loss of show at the beginning of the recording.
Changes in 0.79.139
- Output for Virtualdup shifted with 2 frames
- Added a fflush after writing to stderr
Changes in 0.79.138
- It is now possible to use the .csv file to test changed settings of maxbright
Changes in 0.79.137
- Bug solved: When reviewing the .txt file the bar at the top of the debugwindow is grey instead of black/white
Changes in 0.79.136
- Bugs solved: Comskip crashes when invoked with a .csv file and no .data file is present.
Changes in 0.79.135
- Improved the CC word separation on certain broadcast channels
- (Re)added the "CC" type of CC data.
Changes in 0.79.134
- Change the extension of the XDS dump file to .data
- When comskip is started with a .csv file the .data file containing the XDS dump will also be read when available.
- In ComskipGUI.exe the 'X' key can be used to show XDS data. Currently the Program Name, V_Chips, Program duration and Program position are displayed. Decoding of the XDS data is not yet complete
- A very thin grey vertical line at the very top of the debugbar timeline shows the location of the XDS data changes

[hemicuda]

since I have an extra core or 2 sitting idle most of the time...

is there a way to run more than one instance of comskip at a time? seems I can't get more than one going. do I need a program folder for CSM and a separate one for the cmd line in the STVi?

thanks again for the effort on this app. I'm finally getting around to trying out the CC feature, too.

[bcjenkins]

You can run more than one instance at a time. Dirmon will do this on Windows.

B

[hemicuda]

yeah, I had trouble w/ that scheduler back when I tried it. maybe I didn't tweak it enough.

my issue is w/ CSM running an instance of comskip, I can't kick off another through the Sage menu. I have it pointed to the location of the comskip.cmd file and that's edited to point to the correct location of comskip.exe.

[hemicuda]

next question...

with output_smi=1 is it normal for the CC file to have the format [filename]_1.smi? it's causing the smi file to get deleted shortly after it's created.

I remember there being a work-around using this for the auto-compress stvi, but I didn't load that in this time around and CSM/comskip don't refer to any STVi for operation.

[erik]

Build 0.80.001, available in the early access area, has h.264 decoding enabled.
Do read the readme.txt included in the distribution zip file.

[CollinR]

Awesome man! Strong work.

[erik]

Be aware that it is VERY slow and still has difficulty with some types of AAC.
And it does (often) NOT work on converted program stream h.264, only on original transport stream recordings.

[erik]

There seems to be an error in this release.
I removed it from the server

[erik]

I uploaded build 002, hopefully will solve some reported crashes

[antplugger]

Could anyone tell me if sage_minute_bug and sage_framenumber_bug are still relevent and need to be set to 1 in latest version of SageTV ?

Thanks

[gplasky]

I've run it both ways and haven't noticed a difference. Currently I have it set to 0 (zero).

Gerry

[bcjenkins]

Originally Posted by antplugger

Could anyone tell me if sage_minute_bug and sage_framenumber_bug are still relevent and need to be set to 1 in latest version of SageTV ?

Thanks

If memory serves that is for the default method of txt (frame based) output. I don't believe it is required for edl files which are better (time based)

B

[Slipshod]

Originally Posted by bcjenkins

If memory serves that is for the default method of txt (frame based) output. I don't believe it is required for edl files which are better (time based)

Yep, so if you're generating .edls you don't need to set it. I was just reading this last night while trying to tune Comskip. Unfortunately it still only uses one CPU and Erik didn't know why it wasn't using more than one correctly.

[TallMomof2]

I want to get Comskip to ignore recorded movies. In SageMC there's an option for ignored channels. Should they be separated by a "," comma or ";" semi-colon? Does this affect past recordings or only future recordings?

[Slipshod]

Originally Posted by TallMomof2

I want to get Comskip to ignore recorded movies. In SageMC there's an option for ignored channels. Should they be separated by a "," comma or ";" semi-colon? Does this affect past recordings or only future recordings?

You'd have to read up on SJQ (SageJobQueue) plugin in order to have it run comskip on only certain channels and not others. I haven't sat down to tackle this yet, and I may just wait to see if Showanalyzer gets better in a month or two since it had this capability built-in to it.

[hemicuda]

I remember a file that used to be (maybe still is) included w/ either comskip, the STVi, or CSM that would allow you to script certain channels not to be processed. It would still kick off a process, but not comskip itself.

Ok, it's in the ...\SageTV\SageTV\STVs\SageTV3\Externals folder; "comskip.cmd". Not sure if that will do what you want, but its worth a look.