Using Mythcommflag to create an EDL (ComskipPlayback-compatible) skip file on Linux

[touchstone]

This the script I use with SJQ to run mythcommflag on new recordings and turn the result into an edl suitable for ComskipPlayback. It will probably only be useful to people who are comfortable with bash scripting. It's also only been tested on a handful of recordings, for about one evening. I'm posting because it seems to work, and I haven't seen a solution that does this before. The script requires ffmpeg and awk in addition to mythcommflag.

The first part of the procedure is to calculate the FPS with FFMpeg. I'm using the one provided by Packman (SuSE repo) for that purpose. I haven't tested the one that comes with Sage, which may work just fine. The script then runs mythcommflag and uses the FPS value to calculate the elapsed times from the framerates supplied by mythcommflag for commercial start and end. The script relies on the assumption that mythcommflag will immediately follow a commercial start line (marktype: 4) with a commercial end line (marktype: 5).

Until recently I was running Sage on Vista, and used Comskip for flagging. I was very satisfied, and in fact was a donator, because Erik has done a great job developing it. This script has a disadvantage over running comskip in that nothing is written to the edl file until mythcommflag finishes, preventing 'live' commercial skipping. However, I found comskip's performance under wine to be less than impressive, and all other things being equal I would prefer not to run wine on my server.

Please post any improvements. Thanks.

mythcommflag2edl.sh

#! /bin/bash
#syntax: mythcommflag2edl.sh <video file> [<edl file for output>]
#
#Use sed to remove initial/terminal double quotes that seem to be
#passed as literals when escaped per the SJQ manual (i.e. \"%c%\")
VIDEOFILE="`echo $1 | sed 's/^\"//;s/\"$//'`"
EDLDEFAULTEXTENSION="edl"
FILEBASE="`echo $VIDEOFILE | sed "s/\(.*\)\..*$/\1/"`" 

if [[ -z "$2" ]] ; then
    EDLFILE="$FILEBASE.$EDLDEFAULTEXTENSION"
else
    EDLFILE="$2"
fi
touch "$EDLFILE" ; retval=$?
[[ "$retval" -ne 0 ]] && echo "Error: can't create EDL file $EDLFILE." && exit 1
[[ -z "$VIDEOFILE" ]] && echo "Error: no file to commflag specified." && exit 2
[[ ! -f "$VIDEOFILE" ]] || [[ ! -r "$VIDEOFILE" ]] && echo "Error: $VIDEOFILE doesn't exist or is not readable." && exit 3

echo "Mythcommflag2edl : videofile: $VIDEOFILE, output: $EDLFILE `date`"

echo -n "Calculating FPS with FFMpeg... "
FPS="`ffmpeg -i "$VIDEOFILE" 2>&1 |  grep "Video:" | sed 's/^.*, \([0-9][0-9]\.[0-9][0-9]*\) fps,.*/\1/'`"

[[ -z "$FPS" ]] && echo "Could not determine FPS of $VIDEOFILE." && exit 4

# Bash does integer math only -- convert FPS result to integer for sanity-check
INTFPS=`echo $FPS | sed "s/\..*$//"`
[[ "$INTFPS" -lt 1 ]] || [[ "$INTFPS" -gt 60 ]] && echo "invalid FPS value: $INTFPS" && exit 5
echo "$FPS"

echo "Running mythcommflag"...
# Mythcommflag command syntax
# --skipdb 	No MythTV database to write to
# --veryquiet	Becuase mythcommflag spews messages when scanning hdpvr files
# Awk command syntax: I'm using awk for parsing and converting
#   mythcommflag files have the syntax framenum: xxxx	marktype: y
#   where y is either 4 (start commercial) or 5 (end commercial). A "marktype:
#  4" line is followed by a "marktype: 5".
#
#   awk "/marktype: 4/ 		Find start commerical lines.
#	{ startframe=\$2  ; 	Save the value of the second field in that line
#				(the frame number) to the variable startframe.
#	getline ; 		Read the next line (should be "marktype: 5").
#	print startframe / $FPS	Print the start time of the commercial break,
#				which is its frame number / # frames per second
#	,"\t",\$2 / $FPS	Print a tab, then divide the value of the second
#				field in the now-current line (should be the
#				"marktype: 5" line) by the FPS to get the end
#				time.
#	"\t",0 } " > "$EDLFILE"	Add a "0" at the end per EDL syntax, & write it

mythcommflag --skipdb --very-quiet --outputfile - -f "$VIDEOFILE" | \
awk "/marktype: 4/ { startframe=\$2  ; getline ; print startframe / $FPS,"\t",\$2 / $FPS,"\t",0 } " > "$EDLFILE"
echo " done. `date`"
echo "EDL markup for $VIDEOFILE:"
cat "$EDLFILE"

The client configuration in SJQ looks like this (not including :RESOURCES, etc.):

MYTHCOMMFLAG {
     "<path to>/mythcommflag2edl.sh \"%c%\""
}

[bcjenkins]

Do you still have to have Myth installed for access to mythcommflag? That is the only reason I am using wine/comskip. I haven't found a standalone package.

B

[touchstone]

Originally Posted by bcjenkins

Do you still have to have Myth installed for access to mythcommflag? That is the only reason I am using wine/comskip. I haven't found a standalone package.

B

On Suse mythcommflag requires installation of one RPM containing six files. I'm not sure what other distributions may require.

$ rpm -qa | grep mythtv
mythtv-0_22-common-0.22-1.pm.5.1.x86_64
$ rpm -ql mythtv-0_22-common
/etc/mythtv
/etc/mythtv/config.xml
/etc/mythtv/mysql.txt
/usr/bin/mythcommflag
/usr/bin/mythtranscode
/usr/share/mythtv
/usr/share/mythtv/mythconverg_backup.pl
/usr/share/mythtv/mythconverg_restore.pl

[drewg]

At least for me, comskip is measurably faster, and just as accurate, if not more, than mythcommflag. On my machine (dual-core 2.5GHz 4850e, 4GB RAM) comskip under wine will process a one hour 1080i mpeg2 recording in 16:13, while mythcommflag (even with --hogcpu) takes 19:40 for the exact same recording. Comskip is actually a bit faster than that, but I have the live TV stuff enabled, so it pauses for a while to make sure the file is not growing before it exits. Eg:

mythcom : 1076.19user 71.60system 19:40.60elapsed 97%CPU (0avgtext+0avgdata 0maxresident)k
comskip : 730.97user 56.11system 16:13.31elapsed 80%CPU (0avgtext+0avgdata 0maxresident)k

I don't have an HD-PVR, but from what I understand, the paid version for MPEG4 should be much faster than mythcommflag.

At least in my area, where broadcasters insert repeat frame flags into broadcasts of 24FPS "film" material, mythcommflag is not accurate when used with SageTV, since it does not understand the repeat flags. Eg, from the above recording, comskip said there were 107839 frames, where as mythcommflag incorrectly said there were 90213 frames, since it doesn't understand RFF. Because of this, the mythcommflag cutlist is off by about a factor of 1.2 (30fps/24fps).

So, while it is an interesting idea, and I applaud your work, I'm sticking with comskip.

Drew

PS: Your script did not work for me on Ubuntu 8.04. Probably because it has bash-isms, and /bin/sh on Ubuntu is some minimal thing that is more-or-less posix sh compliant, but not bash compatible.

[bcjenkins]

Drew,

Interesting observations. I have recently put my HDPVR back in service thanks to a forced firmware update on the DirecTV box and could do that comparative.

Touchstone,

I am wondering if there are any libraries which are dependent on Myth being installed as well. Do you have the full Myth package?

B

[touchstone]

Originally Posted by bcjenkins

Drew,
Touchstone,

I am wondering if there are any libraries which are dependent on Myth being installed as well. Do you have the full Myth package?

B

You're right -- I should have grepped "myth". Mythtv-common does indeed depend on libmyth, which is about 22MB of libraries. But I think that's all for Myth dependencies.

Originally Posted by drewg

At least for me, comskip is measurably faster, and just as accurate, if not more, than mythcommflag. On my machine (dual-core 2.5GHz 4850e, 4GB RAM) comskip under wine will process a one hour 1080i mpeg2 recording in 16:13, while mythcommflag (even with --hogcpu) takes 19:40 for the exact same recording.

My own comparisons were limited to running comskip with a handful of files and minimal tuning, albeit with skip_b_frames=2, and observing that it "felt like" about the same speed as mythcommflag. The ~3 min/hour of video difference you observed with MPEG2 is probably within my "feels like" threshhold. But hdpvr files may be another question -- Erik has confirmed that comskip is much slower when processing h.264 files as opposed to mpeg (as is mythcommflag). There may be hardware h.264 decoding in a future comskip release, but then there's a further question of if that hardware decoding will work under wine.

Originally Posted by drewg

At least in my area, where broadcasters insert repeat frame flags into broadcasts of 24FPS "film" material, mythcommflag is not accurate when used with SageTV, since it does not understand the repeat flags.

I think variable frame rates will usually throw off the results of this script regardless of how well mythcommflag handles them. I just have awk calculate the timestamp by dividing the frame number by the frame rate returned by FFMpeg. That won't work if the frame rate changes.

Over the past couple days far it's worked for me (with two HDPVRs and one HDHomerun recording from cable) -- and I'll go back to comskip if it doesn't. But I've seen questions about using mythcommflag with Sage or other non-Myth media software before, without a solution, and thought I'd share mine.

Originally Posted by drewg

PS: Your script did not work for me on Ubuntu 8.04. Probably because it has bash-isms...

You're right: this is a bash script, not sh. I'll update the original post.

[drewg]

Originally Posted by touchstone

The ~3 min/hour of video difference you observed with MPEG2 is probably within my "feels like" threshhold.

Its much more than that -- like I said, I have comskip configured for livetv, so it waits a few minutes before exiting. If you look at the total working time spent by comskip (787 sec / hr) vs mythcommflag (1147 sec / hr), then the difference is much more pronounced (45% more work).

Originally Posted by touchstone

But hdpvr files may be another question -- Erik has confirmed that comskip is much slower when processing h.264 files as opposed to mpeg (as is mythcommflag). There may be hardware h.264 decoding in a future comskip release, but then there's a further question of if that hardware decoding will work under wine.

That's odd. I was told the main advantage to comskip was that it used some uber efficient binary-only windows codecs. I guess the open source codecs are just as good, or maybe better for mpeg4.

Originally Posted by touchstone

I think variable frame rates will usually throw off the results of this script regardless of how well mythcommflag handles them. I just have awk calculate the timestamp by dividing the frame number by the frame rate returned by FFMpeg. That won't work if the frame rate changes.

Complicated encoder settings are MythTV's achillies heel. These recordings will show up as 52:00 in MythTV, and until 0.23 would stutter horribly on playback.

Originally Posted by touchstone

Over the past couple days far it's worked for me (with two HDPVRs and one HDHomerun recording from cable) -- and I'll go back to comskip if it doesn't. But I've seen questions about using mythcommflag with Sage or other non-Myth media software before, without a solution, and thought I'd share mine.

You wouldn't see it with HDPVR, just HDHR. You're probably (un)lucky enough to live in an area where the broadcasters don't use these fancy encoder settings.

Ironically, I've often wanted to use comskip under MythTV, since it is so much faster for Mpeg2, which is all I care about

Drew

[jleonard78]

I had been trying to get this working on my 'nix server today, and kept getting errors. The script would still run, and generate an EDL file, but the EDL file was blank.

i looked at the error in the terminal output, and it was pointing to line 29. Since everything was comparisons and echos on that line, I traced back to the actual ffmpeg command on line 23.

I tried to run this command by myself in the terminal, and immediately realized the problem - the output of ffmpeg has a "Video:" entry to grep, but the sed command is looking for the 'fps' entry, and my version of ffmpeg was outputting the fps as 'tbr'. So this is the "CORRECTION" part of my post. Anyone having issues with this script should test their ffmpeg output.

FYI, I am running Mythbuntu 10.04 The first few lines of output when I run the command gives me the following version information:

FFmpeg version SVN-r0.5.1-4:0.5.1-1ubuntu1, Copyright (c) 2000-2009 Fabrice Bellard, et al.


So, now to the "HELP REQUEST" part of my post...I fixed up my script so that it detected the FPS properly, but when I ran it, it still generated an empty edl file. My edl file is literally a 0 byte file with nothing in it when I check it in a text editor. Is there anything I need to check that might explain why this isn't working? Could it really just be that mythcommflag isn't picking anything up in my file? I have to admit I have only tested 2 files so far...but do not use my DVR function that often.

As a side note, I am using most of the default options for the standard SageTV UI.

[touchstone]

What does ffmpeg display when you run it? E.g., when I run ffmpeg -i Videofile.ts, I see:

FFmpeg version SVN-r23289, Copyright (c) 2000-2010 the FFmpeg developers  built on May 24 2010 19:23:37 with gcc 4.4.1 [gcc-4_4-branch revision 150839]
.
. (compile-time options and complaints re h264 decoding)
.
Seems stream 0 codec frame rate differs from container frame rate: 59.94 (60000/1001) -> 59.94 (60000/1001)
Input #0, mpegts, from 'Videofile.ts':
  Duration: 00:29:56.26, start: 0.419044, bitrate: 8272 kb/s
  Program 1 
    Stream #0.0[0x1011]: Video: h264, yuv420p, 1920x1080 [PAR 1:1 DAR 16:9], 73.82 fps, 59.94 tbr, 90k tbn, 59.94 tbc
    Stream #0.1[0x1100]: Audio: ac3, 48000 Hz, stereo, s16, 384 kb/s
At least one output file must be specified

(As an aside, I'm not entirely sure if I should be using the "fps" vs. the "tbr" field).

[jleonard78]

Thanks for your reply. I see the 'fps' in your output, but not in mine. Let me see if I can get this all in here....

sagetv@sagetv-server:/var/media/tv$ ffmpeg -i BoyMeetsWorld-HisAnswer-111137-0.m
pg
FFmpeg version SVN-r0.5.1-4:0.5.1-1ubuntu1, Copyright (c) 2000-2009 Fabrice Bell
ard, et al.
  configuration: --extra-version=4:0.5.1-1ubuntu1 --prefix=/usr --enable-avfilte
r --enable-avfilter-lavf --enable-vdpau --enable-bzlib --enable-libgsm --enable-
libschroedinger --enable-libspeex --enable-libtheora --enable-libvorbis --enable 
-pthreads --enable-zlib --disable-stripping --disable-vhook --enable-runtime-cpu
detect --enable-gpl --enable-postproc --enable-swscale --enable-x11grab --enable
-libdc1394 --enable-shared --disable-static
  libavutil     49.15. 0 / 49.15. 0
  libavcodec    52.20. 1 / 52.20. 1
  libavformat   52.31. 0 / 52.31. 0
  libavdevice   52. 1. 0 / 52. 1. 0
  libavfilter    0. 4. 0 /  0. 4. 0
  libswscale     0. 7. 1 /  0. 7. 1
  libpostproc   51. 2. 0 / 51. 2. 0
  built on Mar  4 2010 12:35:30, gcc: 4.4.3

Seems stream 0 codec frame rate differs from container frame rate: 59.94 (60000/
1001) -> 29.97 (30000/1001)
Input #0, mpeg, from 'BoyMeetsWorld-HisAnswer-111137-0.mpg':
  Duration: 00:29:59.43, start: 0.361200, bitrate: 3336 kb/s
    Stream #0.0[0x1e0]: Video: mpeg2video, yuv420p, 480x480 [PAR 4:3 DAR 4:3], 2
800 kb/s, 29.97 tbr, 90k tbn, 59.94 tbc
    Stream #0.1[0x1c0]: Audio: mp2, 48000 Hz, stereo, s16, 256 kb/s
At least one output file must be specified

You can see that my file does not return any calculated fps entry (as I would expect with my knowledge of video editing). I suspect something is missing in the header that SageTV uses, as these are files recorded directly by SageTV 7 with a basic setup. Will Sage write the files differently for me using different quality settings? (i would think this just changes the bitrate)

Come to think of it, I think I see the problem now. The line that talks about how the stream frame rate differs from the container frame rate...that further hints at a problem with the format of the recordings.

Seems stream 0 codec frame rate differs from container frame rate: 59.94 (60000/
1001) -> 29.97 (30000/1001)

I wonder why they report differently for the file? Although, I do realize that tbr is just guessed from the file.

I wonder if I use the tbc (timebase for AVCodecContext), if things will work. In theory, it should...the mythcommflag program just needs the right frame rate from some part of ffmpeg


*** goes to his Putty session, modifies the script and runs the commands ***

Well, still nothing out of that run. Mythcommflag, as usual, runs without errors. What value can I plug in to at least get it working? (granted, I would prefer the script to calculate on each file, in case I ever change file options) Or should I just compile ffmpeg on my own? (i would hate to get away from the distro but i intend to keep the server software static)

*sigh* What a shame so far...but I refuse to pay for the feature. I can still skip 10 seconds at a time on my remote