Plugin: Sage Job Queue (SJQ) v3

[Slugger]

Originally Posted by tomfisk

On one of the systems I maintain where SJQ had been running without issues I am now having a problem with the client NT service. It is trying to talk to SJQ on port 80, rather than on 8500. The sjqc.properties file:

#Auto generated by SJQC
#Sun Feb 07 14:41:05 CST 2010
buffer=199000
port=80
host=localhost
is_ssl=false

Skype was recently installed on this system and it appears to concur with the start of the problem.

So...how does SJQC determine what port to use? I noticed that Skype looks at port 80 and 443 for incoming connections. I turned this off in the Skype options but SJQC still is trying to connect on port 80.

Thanks!

Just stop the task client, edit the port number, restart the task client. If sjqc can't find the properties file where expected then it auto generates one, which will set the port to 80 by default.

[Slugger]

Originally Posted by superdodge

I seem to have an issue where options like "DONEALERT" or "RUNQLOADER" are not honored. Is the following the correct syntax?

Sample From My Ruleset:

# ARCHIVE VIDEO FILES TO NAS FOLLOWING QSFIX, COMMERCIAL CUT, ENCODING & COPYING
if [IsObjMediaFile == true && $.IsTVFile{} == true && FileExists == "D:/Archive/5-Completed/%p%.vprj" && FileExists != "D:/Archive/5-Completed/%p%.properties"] {
   :PRIORITY 10
   :SCANMEDIA "true"
   :RUNQLOADER "true"
   :DONEALERT "true"
   :FAILALERT "true"
   ARCHIVE
}

The syntax looks right. Can you check your SJQ server logs and see if there are log messages saying the SageAlert events were fired? I need to know if SJQ recognized it should fire the alerts, but didn't or it just never realized that it needed to fire the alerts. Can you open an issue ticket for this with the results, please? I won't have time to look at this right away so an issue ticket will remind me to do so when I have some free time this week.

[superdodge]

Originally Posted by Slugger

The syntax looks right. Can you check your SJQ server logs and see if there are log messages saying the SageAlert events were fired? I need to know if SJQ recognized it should fire the alerts, but didn't or it just never realized that it needed to fire the alerts. Can you open an issue ticket for this with the results, please? I won't have time to look at this right away so an issue ticket will remind me to do so when I have some free time this week.

Slugger,

I don't know why I didn't check the logs myself...

I found this:

2010-02-08 18:34:27,040 WARN [TaskQueue]: Unable to submit SageAlert because SageAlert URL is not defined in SJQ setting

s!

So I went and deleted the SageAlert Address/User/Pass and saved the settings then entered them in again and now I get:

2010-02-08 18:44:46,172 INFO [TaskQueue]: Alert fired successfully!

So I'm not sure what was wrong, but it appears to be fixed now.

Thanks for help, I'll remember to check the logs myself next time (I forget how complete SJQ is and that I have all that info available!)

I do have an unrelated question. Would it be alot of work to allow variables in the :ALERTTXT and :FAILALERTTXT options? My current issue with the default text is that it references the Object ID instead of the file title which is meaningless to me when I'm away from the house.

Thanks again!
-RL

[Slugger]

Originally Posted by superdodge

Slugger,
I do have an unrelated question. Would it be alot of work to allow variables in the :ALERTTXT and :FAILALERTTXT options? My current issue with the default text is that it references the Object ID instead of the file title which is meaningless to me when I'm away from the house.

Thanks again!
-RL

Not really, just open an issue ticket and when I find a few free minutes I'll add it in. No issue ticket and I'll forget about it...

[bikesquid]

Had an unrelated problem and had to 'reinstall' sagetv, left everything in the directories hoping... now sjq says my password is invalid. Tried the default and what I changed it to without luck. Any help HUGELY appreciated, I just got it working the way I like...

[superdodge]

Originally Posted by Slugger

Not really, just open an issue ticket and when I find a few free minutes I'll add it in. No issue ticket and I'll forget about it...

Thanks, I just didn't want to make an unreasonable request. I have listed the issue.

-RL

[Slugger]

Originally Posted by bikesquid

Had an unrelated problem and had to 'reinstall' sagetv, left everything in the directories hoping... now sjq says my password is invalid. Tried the default and what I changed it to without luck. Any help HUGELY appreciated, I just got it working the way I like...

Take a look at the sjq.log file in your root install directory. Does it say anything as to the cause of the issue? If not, you may need to enable Jetty debug logging and get some logging information. Unfortunately, without some logs and/or an indication as to what the problem is, my hands are really tied without direct access to the system. The SJQ password is encrypted inside the sjq.sqlite db file - there is no way to recover it. I'd need to write a program to go in and reset the password - I'd rather not have to do that so please try to get some info from the logs.

If the logs yield no info, then stop SageTV, rename/move the current sjq.sqlite db file to another location. Restart SageTV. When SJQ restarts it will create a fresh db file. Does that allow you to get back in (with the default pwd, of course)? If so, then we at least know it's a problem with the db file. If that's the case, I still need logs and exception dumps to determine if the current db file can be recovered.

[bikesquid]

Thanks for the suggestions. Found the problem.

As FYI for other unlucky souls:

After looking through the logs, renaming the sjq.sqlite db file, rebooting, restarting, and other flailings, with no success I realized that I assumed if I got the sjq password screen I had the server/jetty installed correctly. I didn't. I decided to go back and retrace the steps of a clean install all the way from before jetty.

Was missing the parameter line edit from jetty installation: load_at_startup_runnable_classes=sagex.jetty.starter.Main

adding that and restarting fixed my slop. I had assumed the problem to be with sjq or the sjqnt client

Thanks Slugger for the help, the fact that no new db file was created pointed to jetty not running.

[bikesquid]

I've searched the guide and forum but don't see it...
I've got two locations, J:\Recorded TV\ for Sage to record to and J:\com free\ where I dump files after comcheck and videoredo are done with their work. Problem is sjq keeps adding those files back in to the queue and I'd like to stop that relying on the 'completed task list'. Is there a condition check i can use against a files path?

[Slugger]

Originally Posted by bikesquid

I've searched the guide and forum but don't see it...
I've got two locations, J:\Recorded TV\ for Sage to record to and J:\com free\ where I dump files after comcheck and videoredo are done with their work. Problem is sjq keeps adding those files back in to the queue and I'd like to stop that relying on the 'completed task list'. Is there a condition check i can use against a files path?

Sure, the Filename test should do what you want.

[bikesquid]

sorry to be so dense, I hadn't realized the Filename test was the complete path... obvious now. Worked light a treat. I tried using the %d% media variable as an internal check and that failed miserably as I couldn't figure out a sintax to try internally.

Using Filename =* "J:\\Recorded TV" is perfect. Thanks

Thanks again for the pointers.

[ChaOConnor]

Slugger... sorry to bring this up again, but still no logs showing on my WHS box. Any idea what could be causing this? I followed the install instructions, SJQ is running great and solid (for about 3 weeks now), but still no logs.

What could I be missing? I'm running in service mode, using my Admin account, the SJQ directory has full permissions for the Admin account (plus everyone as a test)... but still nothing. I must be missing something obvious, but it's driving me nuts, I can't figure it out.

Any ideas? Thanks!

[Slugger]

Originally Posted by ChaOConnor

Slugger... sorry to bring this up again, but still no logs showing on my WHS box. Any idea what could be causing this? I followed the install instructions, SJQ is running great and solid (for about 3 weeks now), but still no logs.

What could I be missing? I'm running in service mode, using my Admin account, the SJQ directory has full permissions for the Admin account (plus everyone as a test)... but still nothing. I must be missing something obvious, but it's driving me nuts, I can't figure it out.

Any ideas? Thanks!

Did you enable Jetty debug logging? Did it give any clues as to why it can't create/write to the log file? I have no idea what could be causing it. I'm going to need some kind of clue from the Jetty logs in order to even take a guess.

[ohpleaseno]

Hey Slugger,

I had to reinstall Win7 from the RC, so I'm back to reinstalling the ntservice on the box.

I made a sjqc.properties file in the correct folder, but the service won't start and here is the error it gives me.

STATUS | wrapper  | 2010/02/14 11:44:53 | --> Wrapper Started as Service
STATUS | wrapper  | 2010/02/14 11:44:53 | Java Service Wrapper Community Edition 3.3.1
STATUS | wrapper  | 2010/02/14 11:44:53 |   Copyright (C) 1999-2008 Tanuki Software, Inc.  All Rights Reserved.
STATUS | wrapper  | 2010/02/14 11:44:53 |     http://wrapper.tanukisoftware.org
STATUS | wrapper  | 2010/02/14 11:44:53 | 
STATUS | wrapper  | 2010/02/14 11:44:53 | Launching a JVM...
FATAL  | wrapper  | 2010/02/14 11:44:53 | Unable to execute Java command.  The system cannot find the file specified. (0x2)
FATAL  | wrapper  | 2010/02/14 11:44:53 |     "java" -Djava.library.path="../lib" -classpath "../lib/wrappertest.jar;../lib/wrapper.jar;sjqc.jar;json.jar" -Dwrapper.key="1y2iWKkITuPdQ32V" -Dwrapper.port=32000 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.pid=1520 -Dwrapper.version="3.3.1" -Dwrapper.native_library="wrapper" -Dwrapper.service="TRUE" -Dwrapper.cpu.timeout="10" -Dwrapper.jvmid=1 org.tanukisoftware.wrapper.WrapperSimpleApp com.google.code.sagetvaddons.sjqc.TaskClient
FATAL  | wrapper  | 2010/02/14 11:44:53 | Critical error: wait for JVM process failed

[nyplayer]

Slugger,

I finally figured out why I get more tasks than should be scheduled.

I get the 60 secs error but the batch job continues running even though it shows as failed in SJQ. This causes SJQ to schedule another media file even though I say resources 55% in the client.

Sun Feb 14 08:49:19 PST 2010: Starting task with priority 1/1/10
Killed because tasks cannot remain in STARTED state for more than 60 seconds!
Executing command line: S:\SJQSAGE\penguin64\shrink.bat "M:\sagem\LawOrder-Absentia-5745618-0.qfix.cut.mpg" "/outputDirectory" "T:\saget" "/cliBitrate" "1000" "/horizontalScale" "480"


Yet the job is running fine... just out of sjq control.

Job below is running fine even though it shows failed.

    - Remaining Commands (8): (/branch)
      + Command : /branch
      + Command Empty, Skipping: ""

    - Remaining Commands (7): (/insertFunction)
      + Command : /insertFunction
      + Command Empty, Skipping: ""

    - Remaining Commands (6): (/insertFunction)
      + Command : /insertFunction
      + Command Empty, Skipping: ""

    - Remaining Commands (5): (/insertFunction)
      + Command : /insertFunction
      + Command Empty, Skipping: ""

    - Remaining Commands (4): (/exe HandBrakeCLI.exe)
      + Command : /exe HandBrakeCLI.exe
        - Executing command: "S:\SJQSAGE\penguin64\mediaEngineBins\HandBrake 9.4
\HandBrakeCLI.exe" -v -i "M:/sagem/LawOrder-Absentia-5745618-0.qfix.cut.mpg" -o
"M:/sagem/LawOrder-Absentia-5745618-0.qfix.cut.workFolder/LawOrder-Absentia-5745
618-0.qfix.cut.scratch.37.MAIN.mp4"   -e ffmpeg -b 1000 -B 160 -E faac     --cro
p 16:16:13:19  -w 480  > "M:\sagem\LawOrder-Absentia-5745618-0.qfix.cut.workFold
er\LawOrder-Absentia-5745618-0.qfix.cut.scratch.37.HandBrakeCLI.log" 2>&1

[Slugger]

@ohpleaseno Windows cannot find Java in your system path. Either ensure java.exe is in your system path or edit the wrapper.conf and point it to the absolute location of the java.exe file.

@nyplayer The task client logs (probably wrapper.conf if using service) should hopefully tell us why the jobs are getting stuck in started state. Jobs get stuck in started because the assigned task client did not report back that it actually started the task successfully (thereby moving it to running state). If the task client is failing to provide that update then the cause of the error should be logged somewhere. I will need that error in order to try and fix it.

[nyplayer]

Slugger,

I am not sure if this helps but I do see this.

2010-02-14 10:46:24,625 INFO  [SJQServlet]: Servlet POST request completed [422ms]
2010-02-14 10:46:25,390 WARN  [SJQServlet]: All client connections are being accepted and processed, please consider setting up client restrictions in SJQ settings
2010-02-14 10:46:25,968 WARN  [SJQServlet]: All client connections are being accepted and processed, please consider setting up client restrictions in SJQ settings
2010-02-14 10:46:26,046 INFO  [SJQServlet]: Servlet POST request completed [93ms]
2010-02-14 10:46:26,281 WARN  [SJQServlet]: All client connections are being accepted and processed, please consider setting up client restrictions in SJQ settings
2010-02-14 10:46:26,281 WARN  [SJQServlet]: All client connections are being accepted and processed, please consider setting up client restrictions in SJQ settings
2010-02-14 10:46:26,281 INFO  [SJQServlet]: Servlet POST request completed [0ms]
2010-02-14 10:46:31,171 WARN  [SJQServlet]: All client connections are being accepted and processed, please consider setting up client restrictions in SJQ settings
2010-02-14 10:46:31,375 INFO  [SJQServlet]: Servlet POST request completed [204ms]
2010-02-14 10:46:32,796 WARN  [SJQServlet]: All client connections are being accepted and processed, please consider setting up client restrictions in SJQ settings
2010-02-14 10:46:33,187 WARN  [SJQServlet]: All client connections are being accepted and processed, please consider setting up client restrictions in SJQ settings
2010-02-14 10:46:55,390 WARN  [DataStore]: DataStore pool timeout expired! '27660371@qtp-15333782-4' did not return connection within 30 seconds!  Adding a replacement connection to the pool.
2010-02-14 10:46:56,281 WARN  [DataStore]: DataStore pool timeout expired! '26432279@qtp-15333782-12' did not return connection within 30 seconds!  Adding a replacement connection to the pool.
2010-02-14 10:47:02,796 WARN  [DataStore]: DataStore pool timeout expired! '11379265@qtp-15333782-16' did not return connection within 30 seconds!  Adding a replacement connection to the pool.
2010-02-14 10:47:03,187 WARN  [DataStore]: DataStore pool timeout expired! '9584382@qtp-15333782-13' did not return connection within 30 seconds!  Adding a replacement connection to the pool.
2010-02-14 10:47:33,250 INFO  [SystemMessageQueueLoader]: Run completed [59469ms]
2010-02-14 10:47:33,250 WARN  [DataStore]: Thread '9584382@qtp-15333782-13' attempted to return a timed out connection!  Dropping connection.
2010-02-14 10:47:33,250 INFO  [SJQServlet]: Servlet POST request completed [60063ms]
2010-02-14 10:47:33,250 WARN  [DataStore]: Thread '11379265@qtp-15333782-16' attempted to return a timed out connection!  Dropping connection.
2010-02-14 10:47:33,250 INFO  [SJQServlet]: Servlet POST request completed [60454ms]
2010-02-14 10:47:33,265 WARN  [DataStore]: Thread '26432279@qtp-15333782-12' attempted to return a timed out connection!  Dropping connection.
2010-02-14 10:47:33,265 INFO  [SJQServlet]: Servlet POST request completed [66984ms]
2010-02-14 10:47:33,265 WARN  [SJQServlet]: All client connections are being accepted and processed, please consider setting up client restrictions in SJQ settings
2010-02-14 10:47:33,265 WARN  [DataStore]: Thread '27660371@qtp-15333782-4' attempted to return a timed out connection!  Dropping connection.
2010-02-14 10:47:33,265 INFO  [SJQServlet]: Servlet POST request completed [67875ms]
2010-02-14 10:47:33,265 INFO  [SJQServlet]: Servlet POST request completed [0ms]
2010-02-14 10:47:33,281 WARN  [SJQServlet]: All client connections are being accepted and processed, please consider setting up client restrictions in SJQ settings
2010-02-14 10:47:33,281 INFO  [SJQServlet]: Servlet POST request completed [0ms]

[Slugger]

Web requests are taking 60 seconds to complete... that's not right. Though it appears everything completes (or at least tries to if not for me killing it off) successfully, just too slowly. I have no idea why web requests would be taking 60 seconds to complete on any system. Certainly not anything I could reproduce locally.

The slow web requests are the root cause of everything else. Fix the slow web requests then everything else should fall into place. Alternatively, the 60 second timeout could be increased, but I have a feeling that increasing the timeout will just end up increasing the length of those web requests. Is the task client perhaps being blocked from accessing the Jetty server? Can you load the SJQ GUI from a task client host? If not, then that's probably the real issue, but that's just speculation on my part.

[ohpleaseno]

Back to my reinstall...

Where else can I look for client logs that would tell me why jobs aren't starting. I was previously getting a "unexpected ":" in line 17" on this client ruleset

:MAPDIR "\\\\localhost\\recorded tv" "\\\\server\\recorded tv"

COMSKIP {
   :MAX 2
   :RESOURCES 25
   :RETURNCODE 1
   "D:/shares/Software/sagetv/comskip80_027/comskip.exe \"--ini=D:/shares/Software/sagetv/comskip80_027/comskip.ini\" \"%c%\""
}

# Run at most 1 mediashink job from 2am - 10am Monday - Friday  :OFFHOUR "0-2, 14-23"

MEDIASHRINKHQ 
{
   :CPU "LOW"
   :MAX 1
   :RESOURCES 50
   "C:\\encoding\\mediashrink\\mediaEngine.exe \"%c%\" /inPlace /mediaShrink /batch /saveLog /defaultProfile autoEncode /onepass /findFileRegEx \"avi|mpg|mkv|mp4|mpeg|VIDEO_TS|ts|ogm|divx|mpts|dvr-ms\""
}

But now my log is empty and no jobs are starting. Where else can I look to troubleshoot?

[Slugger]

Look in the SageTV or Jetty logs for exception dumps. I don't see a problem with the syntax as displayed, but if the logs aren't filling in then that's a sign of an uncaught exception perhaps, so I'd go looking for that.