Comskip for Sage Linux..... :)

[laurenglenn]

First, get a copy of comskip.exe (I used version 0.64) and put the files in /var/media/tv

emerge wine
cd /var/media/tv

----------------------------------------

Here is a copy of the bash prompt (keep in mind the exception error comes *after* it discovers the logo for commercial detection). Just run comskip on the file again and it will pick up the logo file.

STVMC-883 tv # wine comskip TheSimpsons-HOMR-110296-0.mpg
ComSkip 0.64, made using:
mpeg2dec-0.4.0 - by Michel Lespinasse <walken@zoy.org> and Aaron Holtzman
The commandline used was:
comskip TheSimpsons-HOMR-110296-0.mpg

Opening TheSimpsons-HOMR-110296-0.mpg
Using comskip.ini for initiation values.
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg0 sec (113.00 fps)
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
Z:\var\media\tv>comskip TheSimpsons-HOMR-110296-0.mpg
err:seh:setup_exception stack overflow 1324 bytes in thread 000b eip 7fbcc362 esp 7faa0ad4 stack 0x7faa1000-0x7fbb0000
STVMC-883 tv # wine comskip.exe TheSimpsons-HOMR-110296-0.mpg
ComSkip 0.64, made using:
mpeg2dec-0.4.0 - by Michel Lespinasse <walken@zoy.org> and Aaron Holtzman
The commandline used was:
comskip.exe TheSimpsons-HOMR-110296-0.mpg

Opening TheSimpsons-HOMR-110296-0.mpg
Using comskip.ini for initiation values.
Using TheSimpsons-HOMR-110296-0.logo.txt for logo data.
0:07:22 - 13281 frames in 95.66 sec (138.84 fps), 1.00 sec (150.00 fps)
start: 12619, end: 12930, len: 10s
0:07:52 - 14174 frames in 103.82 sec (136.52 fps), 2.12 sec (35.85 fps))
start: 12619, end: 12930, len: 10s
0:08:19 - 14973 frames in 109.94 sec (136.19 fps), 1.00 sec (135.00 fps)
start: 12619, end: 12930, len: 10s
0:08:38 - 15552 frames in 113.91 sec (136.53 fps), 1.00 sec (132.00 fps)
start: 12619, end: 12930, len: 10s
0:08:55 - 16060 frames in 117.89 sec (136.23 fps), 1.00 sec (132.00 fps)
start: 12619, end: 12930, len: 10s
0:09:10 - 16512 frames in 120.88 sec (136.60 fps), 1.00 sec (155.00 fps)
start: 12619, end: 12930, len: 10s
0:09:54 - 17849 frames in 130.92 sec (136.34 fps), 1.00 sec (133.00 fps)
start: 12619, end: 12930, len: 10s
0:10:20 - 18613 frames in 136.91 sec (135.95 fps), 1.00 sec (122.00 fps)
start: 12619, end: 12930, len: 10s
0:11:18 - 20357 frames in 150.92 sec (134.89 fps), 1.00 sec (130.00 fps)
start: 12619, end: 12930, len: 10s
0:11:32 - 20788 frames in 153.89 sec (135.08 fps), 1.00 sec (145.00 fps)
start: 12619, end: 12930, len: 10s
0:12:04 - 21732 frames in 160.94 sec (135.03 fps), 1.00 sec (140.00 fps)
start: 12619, end: 12930, len: 10s
0:12:23 - 22294 frames in 166.49 sec (133.91 fps), 1.56 sec (39.10 fps))
22294 frames decoded in 166.50 seconds (133.90 fps)
Commercials were found.

STVMC-883 tv # dir *.txt
NYPDBlue-OntheFence-78271-0.txt SportsCenter-80864-0.txt TheSimpsons-HOMR-110296-0.txt
STVMC-883 tv # dir -l *.txt
-rw-r--r-- 1 root root 13 Jan 22 15:36 NYPDBlue-OntheFence-78271-0.txt
-rw-r--r-- 1 root root 47 Jan 22 15:29 SportsCenter-80864-0.txt
-rw-r--r-- 1 root root 73 Jan 22 15:40 TheSimpsons-HOMR-110296-0.txt
STVMC-883 tv # cat TheSimpsons-HOMR-110296-0.txt
FILE PROCESSING COMPLETE
-------------------
14279 16533
18164 22071

[laurenglenn]

Here´s a batch file to run comskip at a lower priority so it doesn´t interfere with recording:

# nano comskip
#---
nice wine comskip.exe -n $1
------------------------------------------

Oh.... I almost forgot... Go to http://www.sage-community.org to download Comskip 0.64

You can also get the Comskip import from http://forums.sagetv.com/forums/showthread.php?t=12194&highlight=comskip+import . Just make sure that you create a copy of the STV file and load it before importing the STVi.

:D
Lauren

[laurenglenn]

Now.... if someone could write an automation program for KDE to run this automatically....

But until then, you can use Wine and DirMon to run comskip.exe

Lauren

[sleonard]

Comskip was originally written for Linux, for use in MythTV wasn't it? I also recall reading somewhere that some of the subsequent improvements made in the Windows port were put back into the original Linux version. Couldn't you just use the native Linux version of comskip instead of running the Windows version with the somewhat unstable Wine?

[laurenglenn]

Comskip was originally written for Linux, for use in MythTV wasn't it? I also recall reading somewhere that some of the subsequent improvements made in the Windows port were put back into the original Linux version. Couldn't you just use the native Linux version of comskip instead of running the Windows version with the somewhat unstable Wine?
I guess I could. Do you have a copy of this program? I couldn´t find a Linux version.

Lauren

[sleonard]

Unfortunately, I don't. The only reason I even know about it is that I kept up w/ everything comskip from when it was ported to Windows to when Jere came out w/ SA.

[churth]

Here is a perl script that I wrote to process all the files in a set of directories against comskip (using wine).

It will skip files that already have a txt file.

Change the $inifile to the location of the comskip.ini file (in the script, I use /usr/local/etc/comskip.ini) and change @directories to list the directories that contain the mpg files.


----

#!/usr/bin/perl

### CHANGE THESE VALUES FOR YOUR SYSTEM
# You have to use 4 \'s per one /, since perl changes \\\\ to \\
# and shells change \\ to \
$inifile = "Z:\\\\usr\\\\local\\\\etc\\\\comskip.ini";
@directories = qw (/video1 /video2 /video3 /video4 /video5 /var/media/tv);



foreach(@directories) {
$dir=$_;
opendir(DIR, "$_");
@files = grep (/\.mpg$/,readdir(DIR));
closedir(DIR);
foreach (@files) {
if ($_ =~ /mpg/) {
$filename = $_;
$filename2 = $filename;
$filename2 =~ s/\.mpg/\.txt/;
if (-e "$filename2") {
print "Comskip file $filename2 already exists\n";
} else {
system "(cd $dir ; nice wine /usr/local/bin/comskip --ini=$inifile $filename )\n";

system "(cd /$dir ; nice wine /usr/local/bin/comskip --ini=$inifile $filename)\n";
}
}
}
}

[bcjenkins]

Now.... if someone could write an automation program for KDE to run this automatically....

But until then, you can use Wine and DirMon to run comskip.exe

Lauren
Why not just add it to a cron job? On gentoo emerge cron should do it. Then add the script to your crontab

B

[laurenglenn]

Here is a perl script that I wrote to process all the files in a set of directories against comskip (using wine). Very nice and simple. It does the job well. :) Is there any way to put a pause in there in between files? This way, if it processes all of them, it doesn't go into some kind of endless loop. Thx, Lauren

[Arawak]

Hi All....

I was encouraged by Lauren's successes to take a chance on the Linux version, and got it successfully installed last night. Very pleased to report that it installed quite well.

Of course, immediately I wanted to get commercial detection and transcoding working, so I have been googling away and found this topic.

So, i've emerged wine, and have it running comskip nicely from a shell. My next step is to use the EDL to run mencoder on it and produce a mpeg4 minus commercials. Now, my question is how to make the new file visible to Sage. Do I just overwrite the old mpeg2 with the new mpeg4, complete with .mpg extension? Or is there some other way I should let Sage know about this file?

Anyone know?

Eventually I think I'll write a simple daemon that monitors the system load and the output dir, and launch this process from there.

[churth]

Very nice and simple. It does the job well. :) Is there any way to put a pause in there in between files? This way, if it processes all of them, it doesn't go into some kind of endless loop. Thx, Lauren

It shouldn't get into an endless loop, as it checks for a comskip text file before processing it. If you still want the pause, change where it says sleep 1; to sleep (some number representing the number of seconds).

#!/usr/bin/perl

### CHANGE THESE VALUES FOR YOUR SYSTEM
# You have to use 4 \'s per one /, since perl changes \\\\ to \\
# and shells change \\ to \
$inifile = "Z:\\\\usr\\\\local\\\\etc\\\\comskip.ini";
@directories = qw (/video1 /video2 /video3 /video4 /video5 /var/media/tv);



foreach(@directories) {
$dir=$_;
opendir(DIR, "$_");
@files = grep (/\.mpg$/,readdir(DIR));
closedir(DIR);
foreach (@files) {
if ($_ =~ /mpg/) {
$filename = $_;
$filename2 = $filename;
$filename2 =~ s/\.mpg/\.txt/;
if (-e "$filename2") {
print "Comskip file $filename2 already exists\n";
} else {
system "(cd $dir ; nice wine /usr/local/bin/comskip --ini=$inifile $filename )\n";

system "(cd /$dir ; nice wine /usr/local/bin/comskip --ini=$inifile $filename)\n";

sleep 1; # Change this here
}
}
}
}

[Dekard]

Lol, emerging Wine is taking forever. I'm at 2 hours so far and still cooking. Of course, the linux box is fairly low powered. This is my first time installing Wine, just didn't expect it to take this long.

ps.. emerge is so cool

[drewg]

Here is a perl script that I wrote to process all the files in a set of directories against comskip (using wine).


Thanks, it was a great help as I'm pretty weak in perl.

I modified your script somewhat to garbage collect .txt and .edl files which seem to linger. I think this happens when you change channels in live tv after you've already started comskipping a file.

Drew

#!/usr/bin/perl

### CHANGE THESE VALUES FOR YOUR SYSTEM
# You have to use 4 \'s per one /, since perl changes \\\\ to \\
# and shells change \\ to \
#$inifile = "Z:\\\\usr\\\\local\\\\etc\\\\comskip.ini";
@directories = qw (/var/media/tv);
#@directories = qw (/tmp/tv);

foreach(@directories) {
$dir=$_;
opendir(DIR, "$_");
@files = grep (/\.mpg$/,readdir(DIR));
closedir(DIR);
foreach (@files) {
if ($_ =~ /mpg/) {
$filename = $_;
$filename2 = $filename;
$filename2 =~ s/\.mpg/\.txt/;
$pathname = $dir . "/" . $filename2;
if (! -e "$pathname") {
# print "Will comskib $filename ($pathname missing)\n";
system "(cd $dir ; wine comskip $filename )\n";
}
}
}

# look for comskip .txt or .edl files left over after .mpg file has
# been deleted. These seem to mainly come from live tv.

opendir(DIR, "$_");
@files = grep (/\.edl$|\.txt$/,readdir(DIR));
closedir(DIR);
foreach (@files) {
$filename = $_;
$filename2 = $filename;
$filename2 =~ s/\.[et][xd][tl]/\.mpg/;
$pathname_mpg = $dir . "/" . $filename2;
if (! -e "$pathname_mpg") {
$pathname = $dir . "/" . $filename;
# print "rm $pathname\n";
unlink $pathname;
}
}
}

[erik]

Comskip was originally written for Linux, for use in MythTV wasn't it? I also recall reading somewhere that some of the subsequent improvements made in the Windows port were put back into the original Linux version. Couldn't you just use the native Linux version of comskip instead of running the Windows version with the somewhat unstable Wine?

There is a native Mac port of Comskip on google code.
Get the source and you should be able to compile it on Linux

[erik]

Lol, emerging Wine is taking forever. I'm at 2 hours so far and still cooking. Of course, the linux box is fairly low powered. This is my first time installing Wine, just didn't expect it to take this long.

ps.. emerge is so cool

Comskip runs under wine at the same speed as native on windows.
The speed depends on your system :D

[drewg]

Comskip runs under wine at the same speed as native on windows.
The speed depends on your system :D

Comskip under Wine runs about 2x to 3x faster on my system than the native Linux MythTV commercial skipping did.

Drew

[ntk2]

Besides running comskip on the files, what else is required? Comskip detects commercials and generates a text file with the locations. Do you have to run another program to actually create an .mpg file with the commercials removed?

Does comskip work equally well on SD, DT, or HD files? Some of my files are recorded in standard definition using a PVR-150. Others are recorded using a digital tuner (HD-5500).

[bcjenkins]

You need the comskip plugin loaded to use the marking file.

B

[chip33az]

It shouldn't get into an endless loop, as it checks for a comskip text file before processing it. If you still want the pause, change where it says sleep 1; to sleep (some number representing the number of seconds).

#!/usr/bin/perl

### CHANGE THESE VALUES FOR YOUR SYSTEM
# You have to use 4 \'s per one /, since perl changes \\\\ to \\
# and shells change \\ to \
$inifile = "Z:\\\\usr\\\\local\\\\etc\\\\comskip.ini";
@directories = qw (/video1 /video2 /video3 /video4 /video5 /var/media/tv);



foreach(@directories) {
$dir=$_;
opendir(DIR, "$_");
@files = grep (/\.mpg$/,readdir(DIR));
closedir(DIR);
foreach (@files) {
if ($_ =~ /mpg/) {
$filename = $_;
$filename2 = $filename;
$filename2 =~ s/\.mpg/\.txt/;
if (-e "$filename2") {
print "Comskip file $filename2 already exists\n";
} else {
system "(cd $dir ; nice wine /usr/local/bin/comskip --ini=$inifile $filename )\n";

system "(cd /$dir ; nice wine /usr/local/bin/comskip --ini=$inifile $filename)\n";

sleep 1; # Change this here
}
}
}
}

I'm a little lost on this one. I have Wine set up, and I modified the directories in this script to what I have. The program runs, but it seems to run on each video file, even on ones that have already been processed.

Any suggestions?

[bcjenkins]

I wrote this script for the ComChecker utility I put together for our Mac friends. It worked in my limited testing on my Mac, but it should work in linux with no issue.

http://forums.sagetv.com/forums/downloads.php?do=file&id=338

[ChePazzo]

I just keep the Sage server in the basement, turn on smb, and run comskip over the network from a windows machine..... Works fairly well. I run showanalyzer that way as well.

[lewispm]

I have a WinXP Server that only runs comskip overnight to keep the load down while we're watching tv. I have two linux laptops that are on most of the day with some cpu to spare, so I looked into using these computers to run comskip while they're on.

I modified the script I found here (http://david-latham.blogspot.com/2008/06/python-inotify-pyinotify-how-to-watch.html) to run comskip if a new .mpg file is created in the recording directory. It uses inotify in the linux kernel, so there's no polling of the directory involved. It does nothing until a new file appears, then it runs comskip.

#!/usr/bin/python

import os
import pyinotify
import time

wm = pyinotify.WatchManager()
mask = pyinotify.IN_DELETE | pyinotify.IN_CREATE

class PTmp(pyinotify.ProcessEvent):
def process_IN_CREATE(self, event):
#test for .mpg extension
if os.path.join(event.name)[-4:] == '.mpg':
# wait 3 mins before processing new file
time.sleep(180)
# run comskip on the new file
os.system("/usr/bin/wine 'C:\Program Files\comskip\comskip.exe' 'C:\\tv\%s'" % os.path.join(event.name))


notifier = pyinotify.Notifier(wm, PTmp())

wdd = wm.add_watch('/media/tv', mask, rec=True)

while True:
try:
notifier.process_events()
if notifier.check_events():
notifier.read_events()
except KeyboardInterrupt:
notifier.stop()
break

change the line:
os.system("/usr/bin/wine 'C:\Program Files\comskip\comskip.exe' 'C:\\tv\%s'" % os.path.join(event.name))
to match your setup. ie - location of wine, location wine will look for comskip.exe, and your recording directory. I have a symlink named tv in my wine drive_c to the recording directory.

[lewispm]

wdd = wm.add_watch('/media/tv', mask, rec=True)


Also change this line to match your recording directory.

and here's how to install pyinotify. (http://trac.dbzteam.org/pyinotify/wiki/InstallPyinotify)

[loonsailor]

Thanks, lewispm!

Like several other scripts that I've seen, this will run comskip on all new recordings, even those on channels with no commercials (HBO, PBS, ...). Is there any way to know, other than from inside sage, what channel a recording came from? I guess there's no problem just running comskip anyway, other than a few wasted watts, but it would be nice to be able to be more selective. Has anybody had issues with comskip finding "false positive" commercials on shows where no commercials exist?

[Slugger]

Thanks, lewispm!

Like several other scripts that I've seen, this will run comskip on all new recordings, even those on channels with no commercials (HBO, PBS, ...). Is there any way to know, other than from inside sage, what channel a recording came from? I guess there's no problem just running comskip anyway, other than a few wasted watts, but it would be nice to be able to be more selective. Has anybody had issues with comskip finding "false positive" commercials on shows where no commercials exist?

Since Sage doesn't put the channel name/number in the file name, there's no way to know which channel the recording came from other than to ask SageTV.

But if you want to be more selective about which recordings get comskip run on them, then SJQ (http://sagetv-addons.googlecode.com) can definitely help with that. The task client will also run on Linux so you can use a Linux client to run comskip tasks.

[bcjenkins]

you can also query the filename against a webservice as well.

B

[lewispm]

Thanks, lewispm!

Like several other scripts that I've seen, this will run comskip on all new recordings, even those on channels with no commercials (HBO, PBS, ...). Is there any way to know, other than from inside sage, what channel a recording came from? I guess there's no problem just running comskip anyway, other than a few wasted watts, but it would be nice to be able to be more selective. Has anybody had issues with comskip finding "false positive" commercials on shows where no commercials exist?

I have found a few false positives, which led me to disable the auto commercial skipping feature. Having said that, I'm very impressed at the accuracy of comskip. It works most of the time for me.

Aside from the suggested SJQ, the only way I can think of selectively running comskip would be recording in different directories (I don't think this is possible), or creating symlinks to a separate, monitored directory, from only those files you want to run comskip on. If you are parsing the name of the file to make the decision, that could be accomplished in this script, but it would probably be more work than its worth.

[drewg]

you can also query the filename against a webservice as well.

B

Thats what I do. FWIW, here is my script which runs from a cron job at 5 minutes past the hour, and the 1/2 hour. I'm pretty sure it was based on another script I found here. Maybe yours?

Drew


#!/usr/bin/perl

sub commercial_free
{
$file=$_[0];
$url="http://127.0.0.1:8080/sage/DetailedInfo?FileName=" . $file;
use LWP::UserAgent
$ua = LWP::UserAgent->new;
$req = HTTP::Request->new(GET => $url);
$req->authorization_basic('sage', 'frey');
$r = $ua->request($req)->as_string;

# Look for a string which identifies a show as commercial free. For
# me, I just look for shows recorded on WUNC, my local PBS affiliate

$i = index($r, "WUNC");
if ($i > 0) {
# print "$file is a PBS show\n";
return 1;
}
# print "$file is not a PBS show\n";
return 0;
}

### CHANGE THESE VALUES FOR YOUR SYSTEM
# You have to use 4 \'s per one /, since perl changes \\\\ to \\
# and shells change \\ to \
#$inifile = "Z:\\\\usr\\\\local\\\\etc\\\\comskip.ini";
@directories = qw (/var/media/tv);
#@directories = qw (/tmp/tv);

foreach(@directories) {
$dir=$_;
opendir(DIR, "$_");
@files = readdir(DIR);
closedir(DIR);
foreach $file (@files) {
if ($file =~ /mpg/) {
$comskip_file = $file;
$comskip_file =~ s/\.mpg/\.txt/;
$exists = 0;
foreach $name (@files) {
if ($name eq $comskip_file) {
$exists = 1;
break;
}
}
if (! $exists) {
$pathname_mpg = $dir . "/" . $file;
if (!commercial_free($pathname_mpg)) {
print "Will comskib $file ($comskip_file missing)\n";
system "(cd $dir ; wine comskip $file )\n";
} else {
print "Will create .txt file $comskip_file\n";
system ("cd $dir; touch $comskip_file\n");
}
}
}
else {
if ($file =~ /\.edl$|\.txt$/) {
# look for comskip .txt or .edl files left over after .mpg file has
# been deleted. These seem to mainly come from live tv.
$mpg_file = $file;
$mpg_file =~ s/\.[et][xd][tl]/\.mpg/;
$mp4_file = $file;
$mp4_file =~ s/\.[et][xd][tl]/\.mp4/;
$avi_file = $file;
$avi_file =~ s/\.[et][xd][tl]/\.avi/;
$exists = 0;
foreach $name (@files) {
if ($name eq $mpg_file ||
$name eq $mp4_file ||
$name eq $avi_file) {
$exists = 1;
break;
}
}
if (! $exists) {
$pathname = $dir . "/" . $file;
print "mp4 = $mp4_file";
print "mpg = $mpg_file";
print "rm $pathname\n";
unlink $pathname;
}
}
}
}
}

[bcjenkins]

The new script (link above) now has the ability to load the CPU based on file type. TS files require more CPU that MPEG-2 files for instance.

B

I'll have to add in the web lookup feature though. ;)

[bcjenkins]

I installed jetty and sagex.api from jreichen and stuckless. I chose curl to parse because it is standard on OS X (my script can run on that platform too)

Using that I can determine the channel id fairly easily.

curl --user user:pass --silent http://localhost:8080/sagex/api?c=GetMediaFileForFilePath\&1=$VIDEO | grep AiringChannelName | cut -d[ -f3 | cut -d] -f1

I'll put out a script update which covers this as well later. The channel ID will be in an array format.

New variables will be:

server
port
server_user
server_pass
No_scan_channels


I would be open to further enhancements as well such as being able to specify a comskip ini on a per show/channel basis which would help for tuning specific shows

Thoughts?

B

[Slugger]

I don't mean to hijack and if you tell me to I'll shut up and move on, but is there something about SJQ that is preventing you from using it? If so, I'd really like to know if it's something I could address. It seems to me you're writing scripts to parse out show info to help make decisions on what gets processed. SJQ does it already and all the hard work of having to parse out details to make decisions is done for you. You want to comskip everything, but stuff from PBS and HBO? The ruleset for SJQ can be as simple as this:


if [IsTV == true && IsScheduledRecording == true && ChannelName !% "HBO.*|WPBS" && FileExists != "%d%/%p%.edl"] {
COMSKIP
}


Basically, if the file is a tv recording, was scheduled (i.e. don't comskip live tv recordings since it's likely a waste of time), was not recorded from any HBO or WPBS (replace with your actual PBS callsign) and there's no edl already then run the COMSKIP task. Plus there are 42 other tests available meaning you can make your decision to run based on just about anything.

Then configure a task client to run comskip for the COMSKIP task:


COMSKIP {
:CPU "LOW"
/usr/local/bin/comskip "%c%"
}


Again, if SJQ just isn't what you're looking for (and there's nothing missing/can be fixed/etc.) then just say the word and I'll quietly vanish. However, if you're not using it b/c there's something missing, etc. then I'd love to hear your ideas.

[bcjenkins]

Well honestly I never looked at it. Off the top after skimming the docs (still learning to read you see)


Does SJQ support multiple executions of comskip?
Support throttling of executions based on file type?
Clean up when a media file has been deleted?
Run without SageTV?
(Until my proposed change above, there isn't an explicit need for SageTV nor will the script cease to function when I add it in if unavailable)


Given the last is only mildly useful to me, but I do post this on my blog as well.

B

[Slugger]

Well honestly I never looked at it. Off the top after skimming the docs (still learning to read you see)


Does SJQ support multiple executions of comskip?
Support throttling of executions based on file type?
Clean up when a media file has been deleted?
Run without SageTV?
(Until my proposed change above, there isn't an explicit need for SageTV nor will the script cease to function when I add it in if unavailable)


Given the last is only mildly useful to me, but I do post this on my blog as well.

B

Does SJQ support multiple executions of comskip?

Yes. SJQ supports multiple, distributed clients and clients can run multiple tasks simultaneously.

Support throttling of executions based on file type?

Yes, sort of. You can assign CPU priority to defined tasks. If you defined an MPG comskip task and a TS comskip task then you could set the CPU priority to be lower for the TS task (if that's what you're getting at?).

Clean up when a media file has been deleted?

Yes, there is a built in file cleaner thread. You can tell it to do things like "delete *.edl files when there is no corresponding *.ts or *.mpg file to go with it."

Run without SageTV?

The SJQ server must run within SageTV (and it actually runs within the Jetty plugin), but the task client does not run within Sage/Jetty, it's a completely standalone self contained package. Run it on the Sage host or any other host - it doesn't need Sage at all - the task client only talks to the SJQ server looking for work to do. Once it gets work it does it and reports back to the server.

[bcjenkins]

Can SJQ determine if preexisting tasks were running and delay execution until tasks complete? I have 7 tuners; if all 7 start recording at once or within 30 minutes, how is that handled for an external application like comskip? Would it fire off 7 comskips regardless?

You say you can affect prioritization, you're executing threads at a lower priority via what mechanism?

My script or most other scripts on this page are independent of SageTV. Granted there is little value on these forums regarding that aspect, but there is still value outside.

B

[Slugger]

Can SJQ determine if preexisting tasks were running and delay execution until tasks complete? I have 7 tuners; if all 7 start recording at once or within 30 minutes, how is that handled for an external application like comskip? Would it fire off 7 comskips regardless?

Yes, the server keeps track of what's running and honours the various client settings about max number of tasks a client can execute simultaneously. Clients can be configured to run at most x number of tasks simultaneously and you have finer control on top of that by being able to say things like, "this client can run no more than 4 total tasks simultaneously and no more than 2 h.264 comskip tasks simultaneously."


You say you can affect prioritization, you're executing threads at a lower priority via what mechanism?

You can set the priority of the thread created to run the command and that setting is passed to java.lang.Thread.setPriority() (http://java.sun.com/javase/6/docs/api/java/lang/Thread.html#setPriority(int))


My script or most other scripts on this page are independent of SageTV. Granted there is little value on these forums regarding that aspect, but there is still value outside.

B

[bcjenkins]

Download here (http://forums.sagetv.com/forums/downloads.php?do=file&id=338)
Support here (http://forums.sagetv.com/forums/showthread.php?p=380049)

Added the ability to filter based on channel id.

B

[loonsailor]

I tried running comskip with SJQ and it works, though I have a couple of questions / issues.

First, in order to help the next person, here's what I did. In SJQ, I entered the following rule:

if [IsTV == true && IsScheduledRecording == true && ChannelName !% "KQED.*|KTEH.*|KRCB.*|WORLD|CSPAN.*|CALCHAN.*|EDAC.*|KVIE.*|STARZ.*|STZ.*|HBO.*|.*MAX.*|SHO.*"] {
COMSKIP
}

I also set the file cleaner to delete files with extensions "edl|log|txt|logo.txt", if there's no .mpg or .ts present.

Then I started SJQC:

cd /opt/sagetv/sjqc
sudo WINEPREFIX=~root/.wine ./sjqc.sh &

and entered the following into it:

:MAXPROCS 2

COMSKIP {
:CPU "LOW"
:MAX 2
"/usr/bin/wine comskip %c%"
}

When I run the queue, the right stuff gets put on to the active queue, and it runs, and it works. As I specified, it runs two comskips at a time (using 2 cores of my quad-core CPU). The old .edl and other files all get nicely deleted. The ability to queue all this work is really great!

The .mpg's go really fast - about 90 seconds for a one hour show. The .ts files take much longer - 20-90 minutes for an hour show. CPU is an AMD Phenom II 905e X4. Because of the time discrepancy, I don't care about the loading that bcjenkins' script uses. The .mpg's are in and out so fast it doesn't matter.

Several issues / questions

The startup script sjqc.sh wouldn't work, because it had CR-LF, the DOS convention, instead of just LF as linux expects. I had to filter the CR's out using SED.

The startup script isn't set up to start the java process as a daemon. I ran it from my shell and it works, but it's not in the right form to start sjqc at system boot time. There needs to be a script in /etc/init.d that can start and stop sjqc properly, like the equivalent sagetv scripts. It's not a big deal to write it, but it needs to be done. Does it exist yet? It's not in the distribution.

The comskip processes, running under wine, work fine, but they almost always wind up in the "failed tasks" queue, instead of in "completed tasks". Is this a problem with the value being returned by wine, or did I configure something wrong? How does sjq decide about failure / success? Not a huge deal, but a bit annoying.

Just curious - what happens if I stop / restart sage without restarting sjqc? Will it reconnect properly? If not, there should probably be some way of restarting sjqc automatically when sage restarts.

[Slugger]

I'll briefly address this post in this thread, but after this all future questions/posts about SJQ should not be hijacking this thread, but rather should be posted in the SJQ thread (http://forums.sagetv.com/forums/showthread.php?t=25147). Maybe Opus4 will be kind enough to merge the last post and this reply into that thread?


Several issues / questions
The startup script sjqc.sh wouldn't work, because it had CR-LF, the DOS convention, instead of just LF as linux expects. I had to filter the CR's out using SED.


Can you open an issue ticket (http://code.google.com/p/sagetv-addons/issues/list) for this, please?


The startup script isn't set up to start the java process as a daemon. I ran it from my shell and it works, but it's not in the right form to start sjqc at system boot time. There needs to be a script in /etc/init.d that can start and stop sjqc properly, like the equivalent sagetv scripts. It's not a big deal to write it, but it needs to be done. Does it exist yet? It's not in the distribution.


I typically don't write init.d scripts b/c each Linux distro is a little different. An init.d script that works on Red Hat rarely works on SuSE, etc. Usually each distro has a skeleton in /etc/init.d and you can whip up one that will work on your distro in like 5-10 mins. And when I moved this summer my Linux box was decommissioned so I don't have access to a Linux box at home to play with this kind of stuff (though I'm thinking about setting up a NAS this fall).


The comskip processes, running under wine, work fine, but they almost always wind up in the "failed tasks" queue, instead of in "completed tasks". Is this a problem with the value being returned by wine, or did I configure something wrong? How does sjq decide about failure / success? Not a huge deal, but a bit annoying.


This is talked about in the SJQ thread. The solution is here (http://forums.sagetv.com/forums/showthread.php?p=374171&postcount=852). Basically, comskip returns 1 on success, but SJQ expects the standard return value of zero. You can override the expected return code as discussed in the post linked to above.


Just curious - what happens if I stop / restart sage without restarting sjqc? Will it reconnect properly? If not, there should probably be some way of restarting sjqc automatically when sage restarts.
[/LIST]

On Windows, the service recovery option is available. On Linux, I'd write the init.d script and start/stop it as necessary via that mechanism or if your distro supports an inittab type setup then I'd look into that as inittab supports respawning. With that said, the task client should reconnect just fine, but it will give up and die if the SJQ server is unreachable for too long.

[loonsailor]

Thanks, slugger!

[dshields]

First, get a copy of comskip.exe (I used version 0.64) and put the files in /var/media/tv

emerge wine
cd /var/media/tv

----------------------------------------


Please tell me Wine is not the only way to get commercial skip on Linux. There is no way I am installing wine on my sagetv server. (Nothing against Wine, and I'm running it on another computer...)

[gplasky]

Please tell me Wine is not the only way to get commercial skip on Linux. There is no way I am installing wine on my sagetv server. (Nothing against Wine, and I'm running it on another computer...)

Then your other choice is run comskip from a Windows PC. Those are your choices. If you have Wine running on another PC then run comskip there and map the recordings directory to it.

Gerry

[dshields]

Then your other choice is run comskip from a Windows PC. Those are your choices. If you have Wine running on another PC then run comskip there and map the recordings directory to it.

Gerry

Thank you. I appreciate knowing the facts. I guess my other choice is to run MythTV. I went with SageTV on Linux because it includes the EPG with no subscription fee, but this comskip issue might tip me toward MythTV. Running comskip under wine is just something I do not want to do. That's just me. I do not want a solution that depends on Windows/Wine in any way. (If I had wanted that, I would have just continued to run SageTV on my Windows server.)

[bcjenkins]

That's interesting. If you want a native Linux solution install and run Myth commflag. You can run it on independent files. Personally, I have never found an installer for a standalone version so you have to install the full Myth package.

I run Wine on a headless server without X installed if it matters.

B

[dshields]

That's interesting. If you want a native Linux solution install and run Myth commflag. You can run it on independent files. Personally, I have never found an installer for a standalone version so you have to install the full Myth package.

I run Wine on a headless server without X installed if it matters.

B

Thanks. Those are both good ideas. I would not mind installing MythTV just for its commflag, if that option would work. (I have zero experience with MythTV so far.)

And if I had to use Wine, running it on a headless server might be less objectionable due to the absence of a web browser and other desktop apps. However, it is still something I am not in favor of, primarily because it forces me to think about additional security questions. (I am familiar with the basic arguments in favor or Wine security, but the fact is that I do not even want to have to understand this subject in greater detail. I would prefer to just not install Wine on a system that I plan to leave fairly unsecured. My SageTV server might even be outside my firewall... I might want the media files accessible from my Nokia N900, for example.)

[loonsailor]

I don't know anything about wine security issues (maybe I should?) but I can tell you that sage/wine/comskip works very well.

My sage box is behind a firewall, and only three ports are forwarded it, all on "nonstandard" ports exposed to the outside world - the weird placeshifter port, http (which reaches only the sage web interface) and ssh. So, I'm not sure what security issues wine would create, since I'm not sure how it could even be reached.

I played with myth about a year ago and never managed to get it working well, though I think that part of that is because I was using an underpowered atom-based machine at the time. I remember channel mapping, in particular, being a real pain. Has it improved? In contrast, sage has been quite trouble-free for me. The biggest reason I use sage, though, is the hd200 extender. It works flawlessly, is tiny and silent, uses almost no power and even my wife loves it. I don't think there's any myth equivalent to it.

[chip33az]

I use both MythTV (via Mythbuntu) and SageTV.

I had installed SageTV at my mom's house with 3 TV's, an older version of Ubuntu (never bothered to upgrade) and 3 Media MVPs. This setup works fantastic.

At the apartment we're in, I had SageTV, but wasn't getting the performance I wanted on a very underpowered server. I upgraded the server, popped in a video card and put Mythbuntu on it. Very easy to do. I'm just having some minor issues with playback that is really annoying me (and the WAF). Tomorrow I'm planning on heading back to SageTV on a fresh install of ArchLinux and possibly run ComSkip as well.

Using both, I can tell you they both have great features that I wish the other would use.

Since this is a comskip topic, MythTV does a great job out of the box with this. Since it is built into MythTV, it is very easy to enable and you can also tell it what you want to run comskip on. For example, on our local PBS station I have removed it from running comskip since there are no commercials. My wife enjoys a show on VH1 and the commercials so I set it up not to run comskip on that show.

The two items I don't like about MythTV. 1 - I have a small PC in the living room. The wife doesn't mind, but it just drives me a little batty. I know I could build another frontend with an Atom processor, but the SageTV extenders are nicer (my opinion). 2 - Connecting to the TV was a hassle. I admit I have an older TV without S-Video or HDMI, so getting the server to connect to it correct was about a 1 hour job. This was to size, resize and manipulate the screen.

Sorry I ventured a little off topic. If you have questions about MythTV, usage, setup, etc. let me know. I haven't played with the new .22 version as it hasn't been released yet.