Plugin: Sage Job Queue (SJQ) v3

[bikesquid]

not seeing the leading \ in your code I just went with :

"C:\\Program Files\\ComskipMonitor\\comskip80_031\\comskip.exe \"%c%\""

Now it's showing things are running rather than waiting so an improvement!

Just for my education, is there a place where I can look up the syntax /meaning differences between:
"C:\\Program Files\\ComskipMonitor\\comskip80_031\\comskip.exe \"%c%\""
and
"\"C:\\Program Files\\ComskipMonitor\\comskip80_031\\comskip.exe\" \"%c%\""

I see the error with single quotes explained under rule set syntax - quoted here (but can't find the reference to leading with a \
:
The syntax of the ruleset file is rather simple, but does require a bit of study to completely understand. The new ruleset syntax allows for a much more powerful test definition along with some other capabilities. All input should be double quoted, especially input that contains spaces or special characters. Double quoted strings honour escaped characters such as backspace (\b), newline (\n), etc. So to print a backslash in a double quoted string you must use \\.

[bikesquid]

Also, is that what I would do in wrapper.config to point at the sjqc.properties file I want to use?

This attempt got me a similar error:
# wrapper.app.parameter.2='C:\Program Files\sjqc\'

And thanks so much for the very prompt help!

[Slugger]

Originally Posted by bikesquid

not seeing the leading \ in your code I just went with :

"C:\\Program Files\\ComskipMonitor\\comskip80_031\\comskip.exe \"%c%\""

Now it's showing things are running rather than waiting so an improvement!

Just for my education, is there a place where I can look up the syntax /meaning differences between:
"C:\\Program Files\\ComskipMonitor\\comskip80_031\\comskip.exe \"%c%\""
and
"\"C:\\Program Files\\ComskipMonitor\\comskip80_031\\comskip.exe\" \"%c%\""

I see the error with single quotes explained under rule set syntax - quoted here (but can't find the reference to leading with a \
:
The syntax of the ruleset file is rather simple, but does require a bit of study to completely understand. The new ruleset syntax allows for a much more powerful test definition along with some other capabilities. All input should be double quoted, especially input that contains spaces or special characters. Double quoted strings honour escaped characters such as backspace (\b), newline (\n), etc. So to print a backslash in a double quoted string you must use \\.

If the command line itself contains a space, you need to double quote it, so that's why you need the original " "

Then your executable name also has a space in it, so that string needs to be double quoted, but since that string is inside an already quoted string, you need to escape those quotes, so that's why you get:

"\"C:\\Progam ...\\comskip.exe\" ..."

Of course, backslashes always need to be escaped, so that's why you get the \\ all over the place as well.

Originally Posted by bikesquid

Also, is that what I would do in wrapper.config to point at the sjqc.properties file I want to use?

This attempt got me a similar error:
# wrapper.app.parameter.2='C:\Program Files\sjqc\'

And thanks so much for the very prompt help!

Yes, but the easier fix here is to just flip the slashes around to forward slashes. Since this input goes into Java and Java know how to handle it, you can do that. Doubling up the backslashes will also fix the problem.

[bikesquid]

Thanks Slugger, I think that was the nudge I needed to really get started.

Now I'll be adding comclean to the process if I can ever figure out why the .ts files won't run through nicely...

your explanations greatly appreciated.

[voorhees]

Originally Posted by Slugger

The easiest thing to do would be to assign a static IP to your client so that it uses the old one.

Thanks so much for the response and help. I will give the static IP route a go. I really should have done that along time ago, but my server has only switched IP addresses a couple of times in a year and a half. Now that I use SJQ (and love it) this is the first time it has been a little more of an issue to correct. It looks like I need to pick an IP address though that the router will not try to hand out to other things on the network, so I will just have to recreate the task client anyway. Got three things about to kick off recording, so I will play with it once I get a chance. Thanks again.

[bikesquid]

I guess I'm not done needing help quite yet...
the files (4) get to 'running' in sjq, but then sit, 16 hours, the individual file logs:

Executing command line: "C:\Program Files\ComskipMonitor\comskip80_031\comskip.exe" "J:\recorded TV\FoodTech-ChineseTakeOut-292625-0.ts"

but looking at wrapper.log I see this at the bottom:
HTTP response code: 500 for URL: http://Master-PC:8082/sjq/SJQ?proto=4&cmd=writeLog
INFO | jvm 1 | 2010/02/03 02:29:37 | at sun.net.http://www.protocol.http.HttpURLConn...Stream(Unknown Source)
INFO | jvm 1 | 2010/02/03 02:29:37 | ... 3 more
INFO | jvm 1 | 2010/02/03 02:30:52 | Exception in thread "Thread-87" java.lang.RuntimeException: java.io.IOException: Server returned HTTP response code: 500 for URL: http://Master-PC:8082/sjq/SJQ?proto=4&cmd=writeLog
INFO | jvm 1 | 2010/02/03 02:30:52 | at com.google.code.sagetvaddons.sjqc.TaskClient.log(Unknown Source)
INFO | jvm 1 | 2010/02/03 02:30:52 | at com.google.code.sagetvaddons.sjqc.IOMonster.dumpBuffer(Unknown Source)
INFO | jvm 1 | 2010/02/03 02:30:52 | at com.google.code.sagetvaddons.sjqc.IOMonster.run(Unknown Source)
INFO | jvm 1 | 2010/02/03 02:30:52 | Caused by: java.io.IOException: Server returned HTTP response code: 500 for URL: http://Master-PC:8082/sjq/SJQ?proto=4&cmd=writeLog
INFO | jvm 1 | 2010/02/03 02:30:52 | at sun.net.http://www.protocol.http.HttpURLConn...Stream(Unknown Source)
INFO | jvm 1 | 2010/02/03 02:30:52 | ... 3 more

I have no idea where to start working on that! suggestions anyone?

[Tropicaljoe]

Originally Posted by Slugger

The only thing you have to be careful of is to ensure that the job is only queued once by SJQ, which is not intuitive. SJQ was designed to process Sage objects and perform tasks on them. To perform some external task just once based on some set of conditions requires you to pick a media object that you know will ALWAYS be available and key your ruleset off of it like this:

if [IsObjMediaFile == true && $.GetShowTitle{} == "Something I've Archived and Will Always Be Around, Ideally a Movie or Something With Only 1 Episode" && <Whatever conditions you care about>] {
  MYTASK
}

If you don't include the title check on some movie or something then SJQ will queue up your task once for each object it processes. Alternatively, you could handle multiple invocations in the actual script you're calling, but that results in a rather cluttered queue.

Thanks very much Slugger, that did the trick. Thanks for all your help on this

[bela41]

Originally Posted by Slugger

The only thing you have to be careful of is to ensure that the job is only queued once by SJQ, which is not intuitive. SJQ was designed to process Sage objects and perform tasks on them. To perform some external task just once based on some set of conditions requires you to pick a media object that you know will ALWAYS be available and key your ruleset off of it like this:

if [IsObjMediaFile == true && $.GetShowTitle{} == "Something I've Archived and Will Always Be Around, Ideally a Movie or Something With Only 1 Episode" && <Whatever conditions you care about>] {
  MYTASK
}

If you don't include the title check on some movie or something then SJQ will queue up your task once for each object it processes. Alternatively, you could handle multiple invocations in the actual script you're calling, but that results in a rather cluttered queue.

Slugger, I've been trying to set up a similar external job, and I've managed to create a pretty complex ruleset, Windows Scheduled Task and two batch files. However, the problem I'm having is that once SJQ runs on the necessary file, I have to manually delete the successfully completed job from the Completed Tasks log - otherwise it won't process the same media file twice. Is there anyway to write the task so that it deletes its own completed/failed task entry or to have it reanalyze the object no matter its previous status? I suppose a possible work around would be to have the client report that the task failed and then set SJQ to automatically retry failed tasks, but this opens up other issues for legitimately failed tasks.

[Slugger]

Originally Posted by bikesquid

I guess I'm not done needing help quite yet...
the files (4) get to 'running' in sjq, but then sit, 16 hours, the individual file logs:

Executing command line: "C:\Program Files\ComskipMonitor\comskip80_031\comskip.exe" "J:\recorded TV\FoodTech-ChineseTakeOut-292625-0.ts"

but looking at wrapper.log I see this at the bottom:
HTTP response code: 500 for URL: http://Master-PC:8082/sjq/SJQ?proto=4&cmd=writeLog
INFO | jvm 1 | 2010/02/03 02:29:37 | at sun.net.http://www.protocol.http.HttpURLConn...Stream(Unknown Source)
INFO | jvm 1 | 2010/02/03 02:29:37 | ... 3 more
INFO | jvm 1 | 2010/02/03 02:30:52 | Exception in thread "Thread-87" java.lang.RuntimeException: java.io.IOException: Server returned HTTP response code: 500 for URL: http://Master-PC:8082/sjq/SJQ?proto=4&cmd=writeLog
INFO | jvm 1 | 2010/02/03 02:30:52 | at com.google.code.sagetvaddons.sjqc.TaskClient.log(Unknown Source)
INFO | jvm 1 | 2010/02/03 02:30:52 | at com.google.code.sagetvaddons.sjqc.IOMonster.dumpBuffer(Unknown Source)
INFO | jvm 1 | 2010/02/03 02:30:52 | at com.google.code.sagetvaddons.sjqc.IOMonster.run(Unknown Source)
INFO | jvm 1 | 2010/02/03 02:30:52 | Caused by: java.io.IOException: Server returned HTTP response code: 500 for URL: http://Master-PC:8082/sjq/SJQ?proto=4&cmd=writeLog
INFO | jvm 1 | 2010/02/03 02:30:52 | at sun.net.http://www.protocol.http.HttpURLConn...Stream(Unknown Source)
INFO | jvm 1 | 2010/02/03 02:30:52 | ... 3 more

I have no idea where to start working on that! suggestions anyone?

You have hit this issue.

[Slugger]

Originally Posted by bela41

Slugger, I've been trying to set up a similar external job, and I've managed to create a pretty complex ruleset, Windows Scheduled Task and two batch files. However, the problem I'm having is that once SJQ runs on the necessary file, I have to manually delete the successfully completed job from the Completed Tasks log - otherwise it won't process the same media file twice. Is there anyway to write the task so that it deletes its own completed/failed task entry or to have it reanalyze the object no matter its previous status? I suppose a possible work around would be to have the client report that the task failed and then set SJQ to automatically retry failed tasks, but this opens up other issues for legitimately failed tasks.

Not really - SJQ wasn't designed to be a generic task scheduler. It was designed to perform tasks against media objects so the way it works is get the list of media objects (now also SysMsg objects supported in v3) and process the ruleset for each object, if an object passes a rule, add the assigned tasks to the queue. To queue up generic tasks requires some work around as described above. But as you point out, there are problems with that approach as well.

I suppose a task option could be added so that a successful run doesn't record in the completed tasks list. Open an issue ticket for it if you'd like - no promises as to when it's completed, but it should be easy enough that I will say that it'll eventually get done.

[bela41]

Originally Posted by Slugger

Not really - SJQ wasn't designed to be a generic task scheduler. It was designed to perform tasks against media objects so the way it works is get the list of media objects (now also SysMsg objects supported in v3) and process the ruleset for each object, if an object passes a rule, add the assigned tasks to the queue. To queue up generic tasks requires some work around as described above. But as you point out, there are problems with that approach as well.

I suppose a task option could be added so that a successful run doesn't record in the completed tasks list. Open an issue ticket for it if you'd like - no promises as to when it's completed, but it should be easy enough that I will say that it'll eventually get done.

Thanks. I realize I'm trying to push SJQ beyond what it was built to do so I appreciate your help. I'm not certain that sacrificing the completed-task-record is the best solution just yet so before I ask you to do the work I'll keep thinking about it and perhaps I can figure out another work around. Or perhaps someone else has an idea.

If not, I'll submit a ticket and if you ever want to tackle it, I'll leave it up to you.

Basically the entire purpose for using SJQ for this task is to figure out a way to schedule a task to complete only if Sage isn't busy and doesn't look to be busy in the near future. If anyone has a way to do something like this either with SJQ or without, I'd appreciate the suggestion.

In the meantime, thanks again for all your fantastic work.

[MitchSchaft]

Just an FYI, my install crapped out again with showing the blank GUI. I'll reinstall it real quick.

[Slugger]

Originally Posted by MitchSchaft

Just an FYI, my install crapped out again with showing the blank GUI. I'll reinstall it real quick.

A few users have reported this problem, but I've never seen it. Basically, someone/something is wiping the temp dir that Jetty is exploding the war to. Using the context XML file I include with my plugins, this dir will be in your system temp directory. Nothing should play with a directory in that location, but something is and that's the problem. Find out what's wiping the directory and then stop it to the fix the problem. That's about all I can say on this one.

[MitchSchaft]

Do you have any idea which temp dir jetty uses? The ..Local Settings\Temp dir is full of stuff.

[Slugger]

Originally Posted by MitchSchaft

Do you have any idea which temp dir jetty uses? The ..Local Settings\Temp dir is full of stuff.

Should be the value of %TEMP% for the user running SageTV:

2010_02_02.stderrout.log:2010-02-02 18:49:56.396::INFO:  Extract C:\Program Files\SageTV\SageTV\jetty/webapps/sre.war to C:\DOCUME~1\dbattams\LOCALS~1\Temp\Jetty_0_0_0_0_80_sre.war__sre__vwfyq7_4622114227503637636\webapp

If you have Jetty debug logging enabled then you'll see a statement like the above every time a war is deployed. It will tell you exactly where Jetty is exploding the war to.

[razrsharpe]

Originally Posted by MitchSchaft

Do you have any idea which temp dir jetty uses? The ..Local Settings\Temp dir is full of stuff.

Originally Posted by Slugger

Should be the value of %TEMP% for the user running SageTV:

or open up a command prompt (as the sagetv server user) and type %TEMP%... it should output the temp directory.

[MitchSchaft]

Ok, I see it. The directory is definately full. I'll let you know if I come up with anything if it happens again.

[Skirge01]

I finally (and reluctantly) took the time to upgrade from v2 to v3. I was truly scared that I was going to screw something up and I'd wind up spending hours trying to figure out where I went wrong. I'm shocked to say that it was quite painless. A simple find/replace on my ruleset worked fantastically! I was even able to find the API equivalents pretty easily.

Awesome work, Slugger!

Off to upgrade SageAlert now!

[tomfisk]

On one of the systems I maintain where SJQ had been running without issues I am now having a problem with the client NT service. It is trying to talk to SJQ on port 80, rather than on 8500. The sjqc.properties file:

#Auto generated by SJQC
#Sun Feb 07 14:41:05 CST 2010
buffer=199000
port=80
host=localhost
is_ssl=false

Skype was recently installed on this system and it appears to concur with the start of the problem.

So...how does SJQC determine what port to use? I noticed that Skype looks at port 80 and 443 for incoming connections. I turned this off in the Skype options but SJQC still is trying to connect on port 80.

Thanks!

[superdodge]

I seem to have an issue where options like "DONEALERT" or "RUNQLOADER" are not honored. Is the following the correct syntax?

Sample From My Ruleset:

# ARCHIVE VIDEO FILES TO NAS FOLLOWING QSFIX, COMMERCIAL CUT, ENCODING & COPYING
if [IsObjMediaFile == true && $.IsTVFile{} == true && FileExists == "D:/Archive/5-Completed/%p%.vprj" && FileExists != "D:/Archive/5-Completed/%p%.properties"] {
   :PRIORITY 10
   :SCANMEDIA "true"
   :RUNQLOADER "true"
   :DONEALERT "true"
   :FAILALERT "true"
   ARCHIVE
}