Plugin: Sage Job Queue (SJQ) v3

[Taddeusz]

Originally Posted by Slugger

There's a misplaced ':' on line 4 of your client config. If you can't spot it then paste your client config here and I'm sure someone else will see it.

Ah, I see what I did wrong. Thanks.

Edit:

This is so much more flexible than using dirmon! I have been having to rely on a specific set of file names (show titles) in ShowAnalyzer to determine which profile should be used. Now I can choose which ini file to use based on the channel number so that any show recorded on that channel uses the special ini rather than relying on the file name. Much better.

[zzmystique]

I'm using QFIX on all my recordings and need a little help with some rules, to get them to complete successfully.

1. Detecting if a recording is in multiple parts. (only seems to QFIX myfile-0.ts not myfile-1.ts etc..)
2. Detecting if the recording to be QFixed is currently being viewed.

here is the relevant part of the server ruleset.

// QuickStreamFix all recordings
if [IsObjMediaFile == true && $.IsTVFile{} == true && $.IsFileCurrentlyRecording{} == false && Filename =$ ".ts" && FileExists != "%d%/%p%.QFixed"]{
:SCANMEDIA "true"
:RUNQLOADER "true"
:PRIORITY 100
QSF
}

What do I add to the above ruleset to detect multiple parts and check if the file is in use?

Cheers.

[Slugger]

Originally Posted by zzmystique

I'm using QFIX on all my recordings and need a little help with some rules, to get them to complete successfully.

1. Detecting if a recording is in multiple parts. (only seems to QFIX myfile-0.ts not myfile-1.ts etc..)
2. Detecting if the recording to be QFixed is currently being viewed.

here is the relevant part of the server ruleset.

// QuickStreamFix all recordings
if [IsObjMediaFile == true && $.IsTVFile{} == true && $.IsFileCurrentlyRecording{} == false && Filename =$ ".ts" && FileExists != "%d%/%p%.QFixed"]{
:SCANMEDIA "true"
:RUNQLOADER "true"
:PRIORITY 100
QSF
}

What do I add to the above ruleset to detect multiple parts and check if the file is in use?

Cheers.

It should be running the command on all file segments, not just the first one. If it's not then that's a bug. Which version are you running?

[zzmystique]

Originally Posted by Slugger

It should be running the command on all file segments, not just the first one. If it's not then that's a bug. Which version are you running?

It is only doing the first file, not all of them.

I'm running version 3.0.2.790 with the upgraded SageTV common-logging jar files.

Edit: have updated to the latest version (version v3.0.2.792) and made the following changes to my ruleset.

// Merge multiple file segments before processing other tasks
if [IsObjMediaFile == true && $.IsTVFile{} == true && $.IsFileCurrentlyRecording{} == false && Filename =$ ".ts" && $.GetNumberOfSegments{} != "1" && FileExists == "%c%.xml"]{
:SCANMEDIA "true"
:PRIORITY 20
MERGESPLITFILES
}


// QuickStreamFix all recordings
if [IsObjMediaFile == true && $.IsTVFile{} == true && $.IsFileCurrentlyRecording{} == false && Filename =$ ".ts" && FileExists != "%d%/%p%.QFixed" && $.GetNumberOfSegments{} == "1" && (TimeOfDay < "19:00" || TimeOfDay > "23:00")]{
:SCANMEDIA "true"
:RUNQLOADER "true"
:PRIORITY 10
QSF
}

Is there a better way to check for split files? Also check whether the file to be QSFixed is currently in use?

Cheers

[razrsharpe]

Originally Posted by zzmystique

Is there a better way to check for split files?

GetNumberOfSegments will return the number of segments for the passed mediafile... so you could use that to make sure that it the qsfix only runs on single item media files...

Also check and whether the file to be QSFixed is currently in use?

you could do conceivably do this with some custom java methods that loop through each UIContext, compare the currently loaded mediafile against the passed mediafile, and return true/false. I do not know of any single Sage method or internal SJQ test that does this...

[Slugger]

I'd manually run the GetNumberOfSegments API call in Studio against a media file and ensure that Sage believes there is more than 1 segment. SJQ will only run the command against each segment if Sage API tells SJQ that there is more than one. Last time I tested this it worked and was confirmed as working by someone else. I rarely, if ever, have multi segmented recordings to test this with, but you can see if SJQ is trying to run the command because you'll see the header lines in the log for the task for each segment.

There currently is no test to determine if a specific media file is in use by a client/extender. As suggested, a custom java test would be needed (or an internal test added to SJQ). The latter is unlikely to happen anytime soon (I'm currently working on other things right now and SJQ enhancements aren't at the top of the list). With that said, I would encourage you to add a feature request ticket at the proj site if it's something you really want added.

[nyplayer]

zzmystique

You could add the following to the top of your QFIX batchfile. It will do a fake move... It will exit the job with a bad condition code if the file is in use .. task will fail. I am not sure if you have SJQ to reque failed jobs. If not you would have to manually delete the failed entry

move "%~f1" "%~d1%~p1"
set checkuse=%ERRORLEVEL%
if %checkuse% NEQ 0 exit 1

[zzmystique]

Originally Posted by nyplayer

zzmystique

You could add the following to the top of your QFIX batchfile. It will do a fake move... It will exit the job with a bad condition code if the file is in use .. task will fail. I am not sure if you have SJQ to reque failed jobs. If not you would have to manually delete the failed entry

move "%~f1" "%~d1%~p1"
set checkuse=%ERRORLEVEL%
if %checkuse% NEQ 0 exit 1

Thanks - have many .tmp files from incomplete QFIX jobs.

Yes I have SJQ setup to reque failed jobs, so will give your suggestion a try.

Cheers

[davephan]

I gave up on offloading the Comskip processing with the SJQC on a second computer. I did have a Windows firewall and Norton firewall, but I disabled both and still had the same types of errors. Changing the properties file didn't help either. Perhaps the SJQC is not able to function on a separate computer. I don't know if the SJQC has ever been tested by anyone in that configuration.

I do know if I used SJQ to re-name each file that is recorded to include the source channel name, in the file name, such as naming "RushHour3-4040927-0.ts" to "RushHour3-4040927-0-HBO2.ts", then I could easily selectively Comskip files on the remote server by detecting the channel names within each file name. If it was a non-commercial channel, the file would not be processed.

Dave

[nyplayer]

Originally Posted by davephan

I gave up on offloading the Comskip processing with the SJQC on a second computer. I did have a Windows firewall and Norton firewall, but I disabled both and still had the same types of errors. Changing the properties file didn't help either. Perhaps the SJQC is not able to function on a separate computer. I don't know if the SJQC has ever been tested by anyone in that configuration.

I do know if I used SJQ to re-name each file that is recorded to include the source channel name, in the file name, such as naming "RushHour3-4040927-0.ts" to "RushHour3-4040927-0-HBO2.ts", then I could easily selectively Comskip files on the remote server by detecting the channel names within each file name. If it was a non-commercial channel, the file would not be processed.

Dave

I only use SJQ processing on a remote client my server does not do comskip or mediashrink my clients do it.... also QFIX vrdcut etc... The only time my server does any processing is when a cleint is not connected.

[Slugger]

Using remote clients with SJQ definitely works - I do it myself all the time. The easiest way to troubleshoot this is to try and load the SJQ GUI from one of your remote client hosts. If you can load the web site then the firewall issues are resolved and that means it's a configuration problem - check the sjqc.properties file again b/c if the web gui is loading then this is likely the problem.

[zzmystique]

The $.GetNumberOfSegments{} in the ruleset and adding

move "%~f1" "%~d1%~p1"
set checkuse=%ERRORLEVEL%
if %checkuse% NEQ 0 exit 1

to the beginning of the QFix batch file is working great. It has enabled me to remove the time restriction in the ruleset - if the file is in use it just repeatedly reques the job until the file is no longer being used.

I may still submit the feature request though, as it would be a useful check for the _MOVEREC internal job.

Thanks Guys

[Slugger]

Do you have a lot of split recordings? I'm curious to get some more feedback on if the task is definitely not being run on the additional segments. I'd like to see your task log on such an instance, if possible. As I said, last time I tested this (when I had a multi-segmented recording) it worked for me and was confirmed by another user.

[broconne]

I checked the manual, and I searched the forum.. Is there a way in the SJQ client to set the working directory prior to executing a command?

[Slugger]

Originally Posted by broconne

I checked the manual, and I searched the forum.. Is there a way in the SJQ client to set the working directory prior to executing a command?

No, at least not directly. It could be achieved by wrapping your exe into a batch file and setting the working directory in the batch file prior to calling the target exe.

[broconne]

Originally Posted by Slugger

No, at least not directly. It could be achieved by wrapping your exe into a batch file and setting the working directory in the batch file prior to calling the target exe.

Thanks, a batch file works. Would be a nice feature if you are looking for feature requests :-)

[Slugger]

Originally Posted by broconne

Thanks, a batch file works. Would be a nice feature if you are looking for feature requests :-)

Open a feature request ticket at the project web site. Most feature requests will be put on the back burner as my SageTV dev time is almost over for another year, but feature requests definitely have absolutely no chance of happening without a ticket.

[tmiranda]

Slugger,

Sorry if this has been asked before, but how do I uninstall this? Do I just delete the .xml in the jetty\contexts directory and then restart Sage? Do I need to do anything with the .war file?

Tom

[Slugger]

Originally Posted by tmiranda

Slugger,

Sorry if this has been asked before, but how do I uninstall this? Do I just delete the .xml in the jetty\contexts directory and then restart Sage? Do I need to do anything with the .war file?

Tom

Delete the xml file in the contexts dir. Once that's done, SJQ will stop. You don't need to remove the war file, but you can. If you want to delete the war then you may have to restart Sage in order to delete it (especially on Windows) due to Jetty holding a lock on the war file.

[davephan]

I tried offloading the Comskip process to the second computer using SJQC again. I had the c:\sjqc\sjqc.properties configured correctly on the SageTV computer. However, I notice that the c:\sjqc\sjqc.properties file was not configured correctly on the second computer.

The second computer's sjqc.properties file parameters, "buffer" and "port" were not configured correctly. They are now configured the same as the SageTV computer's sjqc.properties file:
buffer=65000
port=8500

The SJQ web GUI works as it did before from the second computer, and the SJQC connects from the second computer, which it didn't connect before the changes made to the sjqc.properties file stored on the second computer.

I tried running Comskip from the second computer manually. I ran the Comskip against a mapped drive to the SageTV computer. Each video recording drive is mapped on the second computer to the same drive letters that are used on the SageTV computer. Comskip did work successfully when manually running Comskip on a mapped drive file.

After a recording completes, the recording appears in the SJC GUI (on the second computer) list waiting to be assigned to the client on the second computer. The file is assigned to the SJQC on the second computer. However, the Comskip process fails.

Here is a log from a failed Comskip processed file:

Sat Apr 24 08:45:43 CDT 2010: Starting task with priority 5/1/10
Executing command line: C:/comskip/comskip.exe "E:\Video\SellThisHouse-ProspectParkNJ-1706071-0.mpg"
Comskip 0.80.031, made using:
mpeg2dec-0.4.0 - by Michel Lespinasse <walken@zoy.org> and Aaron Holtzman
No such file or directory - could not open file E:\Video\SellThisHouse-ProspectParkNJ-1706071-0.mpg

The commandline used was:
C:\comskip\comskip.exe E:\Video\SellThisHouse-ProspectParkNJ-1706071-0.mpg
Opening E:\Video\SellThisHouse-ProspectParkNJ-1706071-0.mpg

I manually check to make sure the E:\Video\SellThisHouse-ProspectParkNJ-1706071-0.mpg" file exists on the mapped drive. The file exists. I then try manually running Comskip on the second computer against the E:\Video\SellThisHouse-ProspectParkNJ-1706071-0.mpg stored on the SageTV computer. Comskip processes the 30 minute program in 2 minutes, 32 seconds, creating a TXT and EDL file, finding three commercial blocks.

I don't think anything is wrong in my SJQC configuration file:

COMSKIP {
   :RESOURCES 50 # Up to 2 comskip can be run simultaneously (every client has exactly 100 resources available)

   # CPU priority to be used for this task process; one of LOW, NORMAL, HIGH (case sensitive)
   :CPU "NORMAL"   # Default is NORMAL if not specified

   # Comskip returns 1 on success instead of the universally accepted zero; tell SJQ that
   # a return code of 1 means success instead of the default value of zero
   :RETURNCODE 1

   "C:/comskip/comskip.exe \"%c%\""
}

I think my ruleset is good too:

if [IsObjMediaFile == true && $.IsTVFile{} == true && $.IsFileCurrentlyRecording{} == false && ($.GetAiringChannelName{} !% "KTCADT.*|CSPAN.*|ENCR.*|ENCORE|HBO.*|MPLEX|IFC|SUNDAE|INDIE|RETR|FLIX|TCM")] {
   COMSKIP
}

So, at least the SQJC is now connecting from the second computer to the SJQ running on the SageTV computer, but the Comskip jobs are failing for some reason.

Dave