Plugin: Sage Job Queue (SJQ) v3

[toricred]

Originally Posted by superdodge

I am having an issue running ShowAnalyzer from SJQ.

My Task is:
# Identify Commercials in HDPVR Recordings using ShowAnalyzer
SHOWANALYZERHDPVR {
:RESOURCES 34
:CPU "NORMAL"
"\"C:\\Comskip\\SA\\ShowAnalyzer.exe\" \"%c%\""
}

The Resultant Log Is:
Mon Jan 04 22:17:56 EST 2010: Starting task with priority 5/1/10
Executing command line: "C:\Comskip\SA\ShowAnalyzer.exe" "D:\Recordings\TheOffice-FrameToby-2754123-0.ts"

The Task Ends Up in the "Failed Tasks" Section.

Running "C:\Comskip\SA\ShowAnalyzer.exe" "D:\Recordings\TheOffice-FrameToby-2754123-0.ts" from the commandline works successfully.

Am i doing something wrong?

I think you've got a \ in the wrong place. I believe it should be "\"C\:\\SA\\ShowAnalyzer.exe\" \"%c%\""

[ohpleaseno]

Originally Posted by evilpenguin

That's most likely related to your CLI parameters or the videos you're trying to code rather than the bug they're talking about where mediaShrink fails to run at all.

If I'm using autoencode, wouldn't it pick the correct parameter?

[Peter_h]

Hey Slugger, I've been using SJQ for a little over a year now. I have only had a single client configured locally on the server and it has run great.

I just set up an additional client on the LAN tonight. However, I'm experiencing a problem with the it not getting sent the correct media path.

Here is the relevant ruleset.

if [IsObjMediaFile == true && $.IsTVFile{} == true && $.IsNotManualOrFavorite{} == false && FileExists != "%d%/%p%.edl" && Filename =$ ".mpg"]
{
   :PRIORITY 10
   COMSKIPSD
}

if [IsObjMediaFile == true && $.IsTVFile{} == true && $.IsNotManualOrFavorite{} == false && FileExists != "%d%/%p%.edl" && Filename =$ ".ts"]
{
   :PRIORITY 9
   COMSKIPHD
}

Here is the client config file for the machine on the LAN.

COMSKIPSD {
   :RESOURCES 50 # Up to 2 comskipsd can be run simultaneously. (every client has exactly 100 resources available)

   # CPU priority to be used for this task process; one of LOW, NORMAL, HIGH (case sensitive)
   :CPU "NORMAL"   # Default is NORMAL if not specified

   # Comskip returns 1 on success instead of the universally accepted zero; tell SJQ that
   # a return code of 1 means success instead of the default value of zero
   :RETURNCODE 1
   :MAXRUNRATIO 1

   "C:/comskip/comskip.exe \"%c%\""
}

I should give you some background. I had to rebuild my server a few weeks ago due to some major hardware failure. In the process, I changed the recording directory. I followed the guide for moving recording directories when I brought the server back to life and everything was fine.

The current recording directory is \\Server\SageTV Recordings.

The original recording directory was \\Server\SageTV.

Whenever the comskip task is assigned to the client on the LAN, I get an error in the log from comskip stating

No such file or directory - could not open file \\Server\Sagetv\recording.mpg

As you can see it is sending the incorrect recording directory. That directory does not exist on this server. I am not quite sure what is going on and whether this is a wiz.bin problem or how to fix it.

Is there a way to get around this in SJQ? If it is a wiz.bin problem, would I be able to move the file in SJQ using the internal _MOVEREC tasks and potentially fix this?

Thanks,
Peter

[Slugger]

It looks like a wiz.bin problem, but you'll want to confirm that by checking the full path of the file in your STV or on the web server, etc. If they show the same (incorrect) path then the problem is in your wiz.bin. I suspect that this is the case because the value SJQ uses is just pulled from wiz.bin. However, confirm that via another source.

You have two options to handle this. First, you can use the :MAPDIR setting in SJQ to tell the client to map one dir to another. This should work fine, but if it is indeed a wiz.bin issue then you're going to have other problems (like trying to watch the recording isn't going to work because Sage thinks the file is somewhere it's not). But as far as SJQ goes, the :MAPDIR option should work just fine. My recommended solution, however, is to get the paths right in wiz.bin since not doing so is going to cause you way more problems beyond SJQ.

I would not use _MOVEREC task here because _MOVEREC must always copy the file somewhere and even if it's a copy to the same dir, it will do a read/write of the entire physical file (as opposed to a rename operation). You definitely don't want to use _MOVEREC for this scenario.

[nyplayer]

Originally Posted by Peter_h

Hey Slugger, I've been using SJQ for a little over a year now. I have only had a single client configured locally on the server and it has run great.

I just set up an additional client on the LAN tonight. However, I'm experiencing a problem with the it not getting sent the correct media path.

Here is the relevant ruleset.

if [IsObjMediaFile == true && $.IsTVFile{} == true && $.IsNotManualOrFavorite{} == false && FileExists != "%d%/%p%.edl" && Filename =$ ".mpg"]
{
   :PRIORITY 10
   COMSKIPSD
}

if [IsObjMediaFile == true && $.IsTVFile{} == true && $.IsNotManualOrFavorite{} == false && FileExists != "%d%/%p%.edl" && Filename =$ ".ts"]
{
   :PRIORITY 9
   COMSKIPHD
}

Here is the client config file for the machine on the LAN.

COMSKIPSD {
   :RESOURCES 50 # Up to 2 comskipsd can be run simultaneously. (every client has exactly 100 resources available)

   # CPU priority to be used for this task process; one of LOW, NORMAL, HIGH (case sensitive)
   :CPU "NORMAL"   # Default is NORMAL if not specified

   # Comskip returns 1 on success instead of the universally accepted zero; tell SJQ that
   # a return code of 1 means success instead of the default value of zero
   :RETURNCODE 1
   :MAXRUNRATIO 1

   "C:/comskip/comskip.exe \"%c%\""
}

I should give you some background. I had to rebuild my server a few weeks ago due to some major hardware failure. In the process, I changed the recording directory. I followed the guide for moving recording directories when I brought the server back to life and everything was fine.

The current recording directory is \\Server\SageTV Recordings.

The original recording directory was \\Server\SageTV.

Whenever the comskip task is assigned to the client on the LAN, I get an error in the log from comskip stating

No such file or directory - could not open file \\Server\Sagetv\recording.mpg

As you can see it is sending the incorrect recording directory. That directory does not exist on this server. I am not quite sure what is going on and whether this is a wiz.bin problem or how to fix it.

Is there a way to get around this in SJQ? If it is a wiz.bin problem, would I be able to move the file in SJQ using the internal _MOVEREC tasks and potentially fix this?

Thanks,
Peter

If you look closely it appears that SJQ is not picking up the blank in your path name correctly. That is one reason why I avoid using blanks in my recording directories.

Path \\Server\SageTV Recordings appears to be picked up as \\Server\SageTV

As I stated in your other post If the file can be played in sage then the paths must be correct in sage. Look in the details of the recording.

[mguebert]

Slugger,

I tried the ruleset you suggested and I get an error in my logs, I looked at the rule and I cannot see the problem.

if [IsObjMediaFile == true && $.IsTVFile{} == true && $.IsFileCurrentlyRecording{} == false && $.IsWatchedCompletely{} == false && FileExists != "%d%/%p%.edl" && $.GetAiringChannelNumber{} !% "499|[5-7][0-9][0-9]|8(?:[0-6][0-9]|7[0-6])"]
{
:PRIORITY 10
COMSKIP

}


Here is the log output

2010-01-08 09:02:51,749 ERROR [MediaFileQueueLoader]: Unexpected exception
java.util.regex.PatternSyntaxException: Unclosed group near index 41
499|[5-7][0-9][0-9]|8(?:[0-6][0-9]|7[0-6]
^
at java.util.regex.Pattern.error(Unknown Source)
at java.util.regex.Pattern.accept(Unknown Source)
at java.util.regex.Pattern.group0(Unknown Source)
at java.util.regex.Pattern.sequence(Unknown Source)
at java.util.regex.Pattern.expr(Unknown Source)
at java.util.regex.Pattern.compile(Unknown Source)
at java.util.regex.Pattern.

[Slugger]

Originally Posted by mguebert

Slugger,

I tried the ruleset you suggested and I get an error in my logs, I looked at the rule and I cannot see the problem.

if [IsObjMediaFile == true && $.IsTVFile{} == true && $.IsFileCurrentlyRecording{} == false && $.IsWatchedCompletely{} == false && FileExists != "%d%/%p%.edl" && $.GetAiringChannelNumber{} !% "499|[5-7][0-9][0-9]|8(?:[0-6][0-9]|7[0-6])"]
{
:PRIORITY 10
COMSKIP

}


Here is the log output

2010-01-08 09:02:51,749 ERROR [MediaFileQueueLoader]: Unexpected exception
java.util.regex.PatternSyntaxException: Unclosed group near index 41
499|[5-7][0-9][0-9]|8(?:[0-6][0-9]|7[0-6]
^
at java.util.regex.Pattern.error(Unknown Source)
at java.util.regex.Pattern.accept(Unknown Source)
at java.util.regex.Pattern.group0(Unknown Source)
at java.util.regex.Pattern.sequence(Unknown Source)
at java.util.regex.Pattern.expr(Unknown Source)
at java.util.regex.Pattern.compile(Unknown Source)
at java.util.regex.Pattern.

You're missing the ending parenthesis. Did you copy/paste my post into your SJQ? Are you missing the ending parenthesis in your ruleset? I copy/pasted what's at the top here (which, I assume, is what I originally wrote?) and it works for me (syntactically, I still didn't actually test to see if it produces the desired results).

java.util.regex.PatternSyntaxException: Unclosed group near index 41
499|[5-7][0-9][0-9]|8(?:[0-6][0-9]|7[0-6]
                                         ^

Should be a ')' at the end of the string (just as there is in the rule at the top).

[Slugger]

And now that I've taken a closer look at the regex I originally provided, I'd revise it to this:

499|[5-7]\\d{2}|8(?:[0-6]\\d|7[0-6])

But I believe either one will work and unless you're quite comfortable reading regex expressions, the original one I provided is probably easier to read and understand what it's trying to match against. Either way, you need to make sure the end parenthesis is included in your ruleset and you still need to test to make sure that it actually matches the numbers you're expecting it to since I still haven't actually tested it (and won't, since, hey, you need something fun to do, right?)

[gveres]

I am trying to figure out if I can determine that a file is part of a playlist. But in doing so, I realized that I have no idea where to get the list of tests that I can perform on an object. The Guide points me to the SageTV API document, but that seems to say the MediaFileAPI only has a few Is commands. In fact the commands that are referenced in the Guide are not listed in the API docs as being methods no that object. I must be missing something.

Can somebody point me to the right list of queries we can make on an object?

Thanks
Greg

[Slugger]

If the object is a MediaFile object (tv recording, import video, music, photo, DVD, BluRay) then you can call methods (that start with Is, Has, Get) from the following Sage APIs:

MediaFileAPI,
ShowAPI,
AiringAPI,
ChannelAPI

If the object is a SystemMessage object then you can call methods from the following APIs:

SystemMessageAPI

The last paragraph in this section explains it (though I'll admit I should probably highlight it a little better in the docs).

[Slugger]

Starting with SJQ v3.0.2, I have repackaged the application such that it relies on the commons-logging-1.1.1 jar files being installed to the SageTV/JARs folder. However, SageTV ships with commons-logging v1.0 (installed in the JARs folder as commons-logging.jar and commons-logging-api.jar). Therefore, to use SJQ v3.0.2 or newer, you must upgrade SageTV's commons-logging jar files. I have written a wiki doc that explains the steps to do it (quite simple, actually).

The need to do all of this arose from an SRE user who was encountering ClassLoader problems. Turns out commons-logging prior to v1.1 is the root cause of the problem along with the fact that I was packaging the core commons-logging libs in my war files and they were also shipped with Sage in the JARs folder, causing additional collisions.

I have opened a bug report with SageTV asking them to upgrade the commons-logging packages themselves in a future release. Until that happens, you will need to manually upgrade the libraries. I have performed this upgrade myself and it has caused no ill effects (and the SRE user who hit the ClassLoader problems has also done it with no ill effects and it also got SRE working again). However, I feel obligated to warn people that changing these libs from the ones shipped with Sage could cause undesired results so if you are unsure about doing this then simply don't do it. If you choose not to then you cannot use SJQ v3.0.2 or newer as it depends on the upgraded jars being in the SageTV/JARs folder.

SJQ snapshot 748 is now available. This snapshot contains no functional changes vs. v3.0.1. The only change is that it's repackaged to properly support the use of the commons-logging-1.1.1 jars in the SageTV/JARs folder.

[gveres]

Originally Posted by Slugger

If the object is a MediaFile object (tv recording, import video, music, photo, DVD, BluRay) then you can call methods (that start with Is, Has, Get) from the following Sage APIs:

MediaFileAPI,
ShowAPI,
AiringAPI,
ChannelAPI

If the object is a SystemMessage object then you can call methods from the following APIs:

SystemMessageAPI

The last paragraph in this section explains it (though I'll admit I should probably highlight it a little better in the docs).

Ok Thanks. I see that the methods I couldn't find on the MediaFile API are coming form the other APIs you mention.

I think for what I was thinking of doing you would have to add processing of the playlists and provide access to the playlist APIs and a way to iterate over the playlist contents. I will look for another way to do what I am trying to do.

Thanks
Greg

[razrsharpe]

Originally Posted by gveres

I will look for another way to do what I am trying to do.

Write your own simple function to iterate through the playlist and determine if it is a member (something like IsPlayListMember(PlayListObject, Mediafile) ). Slugger would then need to implement support for custom java methods (i.e. - something of the form package.api.Method)... pretty please

[Slugger]

Originally Posted by razrsharpe

Write your own simple function to iterate through the playlist and determine if it is a member (something like IsPlayListMember(PlayListObject, Mediafile) ). Slugger would then need to implement support for custom java methods (i.e. - something of the form package.api.Method)... pretty please

And yet no issue ticket for this feature request exists on the project site?

As for gveres' original request, it doesn't appear that the PlaylistAPI even provides good search methods for determining if a media file is in a playlist. The only way I see being able to do it is to grab all the defined Playlists and iterate through them and look for the media file in question. I could add an internal SJQ test that does this. If that's all you need, it'd be simple enough to add a test called something like 'IsObjInAnyPlaylist'. If that's something that interests you then open an issue ticket. Won't take much effort to make it happen. I could also expose the PlaylistAPI via the $ object, but, as I say, I don't see much value in it as there really isn't a lot to be done with the API.

[razrsharpe]

Originally Posted by Slugger

And yet no issue ticket for this feature request exists on the project site?

ticket submitted... i always keep forgetting about the issue tracker

[Slugger]

Server snapshot 752 now available:

Fix issue 106: TranscodeQueueSize test now accounts for pending _TRANSCODE tasks when calculating transcode queue size

Task client snapshot 750 now available:

Fix issue 105: sjqc.properties file now accepts sleep property. Set it to an integer, which tells the task client how often to poll the server for new work. Default, if not specified, is 300 seconds (5 mins), min value is 30.

NOTE: Starting with SJQ server v3.0.2, you must upgrade SageTV's commons-logging jar files in order to use SJQ. This wiki doc explains how to do it.

[gasinger]

Originally Posted by Slugger

Task client snapshot 750 now available:

Fix issue 105: sjqc.properties file now accepts sleep property. Set it to an integer, which tells the task client how often to poll the server for new work. Default, if not specified, is 300 seconds (5 mins), min value is 30.

Thank you Slugger. I installed the updated builds of the client 750 and server 752 and it works very well.

[superdodge]

About 50% of my ShowAnalyzer jobs never come out of the active queue. They sit there and say Running although ShowAnalyzer is complete, commercial skips are fine and ShowAnalyzerEngine.exe is no longer open.

i would assume that my setup was incorrect, but 50% of the time the items do move to the Completed List.

I've tried calling SageAnalyzerEngine.exe from SJQ and I've tried using a batch file with "start /i /wait SageAnalyzerEngine.exe" in it. Either way the result is the same.

Any ideas?

[toricred]

Make sure that you're not calling ShowAnalyzer until the recording is finished. It's currently having problems with analyzing a file that is still recording. Jere is aware of the issue and it is included in the current bug list for the product.

[Slugger]

Going to need logs... check the client logs from the GUI. Are there any exceptions logged back by SJQ? Check the SJQ server logs. Any exceptions there? If no exceptions there then you may need to turn on Jetty debug logging and look for exceptions in the Jetty logs (this would be necessary if it's a runtime exception being thrown such as an unexpected NullPointerException).

Jobs get stuck in RUNNING state if the task client fails to report back the completed status of the task. Usually this would be a result of the task client crashing. If so, check the wrapper.log file in the logs dir of where you installed the task client (assuming you installed the Windows service version, if not then hopefully you're logging the output of the regular task client somewhere). When a task gets stuck in running state do other tasks loaded into the queue afterwards continue to run? If so, then the task client may not be crashing. I need logs/exception dumps to determine these kinds of things.

As you can see, the overwhelming theme here is that I need logs. Unless you can describe an exact step by step process for reproducing the error AND I can reproduce it locally, I need error logs and exception dumps to see where things are going wrong in order to fix them.

EDIT: If it's indeed the problem described by toricred, then I'd still like to get a hold of some exception dumps b/c the task client should be able to handle a native process that crashes out.