Plugin: Sage Job Queue (SJQ) v3

[Slugger]

I'm not familiar with VideoRedo - don't use it myself - but I can tell you that SJQ runs all processes as the same user that SageTV is running as and if you're running SageTV as a service (on Windows) then this is likely the LocalSystem account, which may or may not be causing you these problems. Again, I don't use VideoRedo so just consider this as something to think about and hopefully it points you in the right direction.

I seem to recall at least one other person using VideoRedo with SJQ, perhaps they'll see this and chime in with some more specific advice.

[ChaOConnor]

Originally Posted by gveres

Here is my ruleset:

// compress adult shows
if [IsObjMediaFile == true && $.IsTVFile{} == true && $.IsFileCurrentlyRecording{} == false && FileExists == "%d%/%p%.edl" && Filename =$ ".mpg" && ($.GetShowTitle{} == "The Ellen DeGeneres Show" || $.GetShowTitle{} == "Nova scienceNOW" || $.GetShowTitle{} == "Rolex Sports Car Series Racing" || $.GetShowTitle{} == "SCCA Racing" || $.GetShowTitle{} == "New Yankee Workshop" || $.GetShowTitle{} == "Monster Jam")] 
{
    :SCANMEDIA "true"
    :PRIORITY -20
       ADULTCOMPRESS
}

and on the client side:

ADULTCOMPRESS {
   :MAX 1
   :RESOURCES 51
   :CPU "LOW"
   "C:/QFIXSAGE/AdultShrink.bat \"%c%\""
}

I think the max 1 is a hold over from old versions of SJQ that I just haven't cleaned up.

Greg

So SJQ is properly kicking stuff off now, but nothing happens. I don't see anything on the server or my other PC client. I let two shows sit all last night in "RUNNING" status, but nothing. I eventually killed the tasks. What will the program show up as in my Task Manager?

This the batch file I'm using:

set fname=%~d1%~p1%~n1

C:\sjqc\MediaShrink\mediaEngineBins\touch.exe -r "%~f1" "%fname%.timestamp"
start /I /NORMAL /WAIT C:\sjqc\MediaShink\mediashrink.exe "%~f1" /InPlace
C:\sjqc\MediaShrink\mediaEngineBins\touch.exe -r "%fname%.timestamp" "%fname%.mkv"

del "%fname%.timestamp"

I changed it to NORMAL to test, still no luck.

EDIT: Nevermind... I'm retarded. I misspelled the directory name... idiot. Thanks to everyone who helped!

[SteveW]

I've got everything set up OK, but running into a problem when running comskip against my recordings.

SJQ hands the job off to the client also running on my server. The comskip (latest donator) process starts, but never finishes according to SJQ. It sits in the active queue forever saying "running".

I can see comskip in task manager running. It eventually goes to 0% cpu, there are txt and log files for the show, but no .edl. The sjqc also keeps throwing errors (see attached). My client properties are as follows:

COMSKIP {
:RESOURCES 75 # Up to 4 comskips can be run simultaneously (every client has exactly 100 resources available)

# CPU priority to be used for this task process; one of LOW, NORMAL, HIGH (case sensitive)
:CPU "NORMAL" # Default is NORMAL if not specified

# Comskip returns 1 on success instead of the universally accepted zero; tell SJQ that
# a return code of 1 means success instead of the default value of zero
:RETURNCODE 1

# :OFFHOUR "0-1,12-23" # 8pm - 9:59pm and 7am to 9:59am

"C:\\Comskip\\comskip.exe \"%c%\""
}


My properties file looks like this:

buffer=199000
port=8500
host=localhost
is_ssl=false
sleep=300


There is nothing in the client log. Anyone have a suggestion?

[Slugger]

Hmm... seems like a possible miscalculation on my part - maybe, not sure. Let's find out.

Can you confirm that you did not increase Jetty's max post request size as described at the beginning of the User's Guide? Assuming you didn't then there are two fixes:

1) Increase Jetty's max post size as described in the installation section of the online User's Guide. This is the better choice since it allows you to keep a higher buffer value, though this would require a stop/restart of SageTV

2) Stop the task client and decrease the buffer size from 199000 to 65000 then restart the task client

Either one should fix the problem, please let me know if it does. I suspect the problem is that I'm not accounting for the encoding of the log output so even though the client is only buffering 199000 bytes of output, when it's encoded for HTTP transfer that output becomes larger (in theory up to exactly 3x larger and 3 x 199000 > 200000, which is the problem if you didn't increase Jetty's max post size). Practically speaking, the encoded buffer is probably just slightly going over the 200000 limit, but still anything over 200000 will cause the errors you're seeing.

Anyway, choose a fix and let me know how it goes.

[SteveW]

Originally Posted by Slugger

Hmm... seems like a possible miscalculation on my part - maybe, not sure. Let's find out.

Can you confirm that you did not increase Jetty's max post request size as described at the beginning of the User's Guide? Assuming you didn't then there are two fixes:

1) Increase Jetty's max post size as described in the installation section of the online User's Guide. This is the better choice since it allows you to keep a higher buffer value, though this would require a stop/restart of SageTV

2) Stop the task client and decrease the buffer size from 199000 to 65000 then restart the task client

Either one should fix the problem, please let me know if it does. I suspect the problem is that I'm not accounting for the encoding of the log output so even though the client is only buffering 199000 bytes of output, when it's encoded for HTTP transfer that output becomes larger (in theory up to exactly 3x larger and 3 x 199000 > 200000, which is the problem if you didn't increase Jetty's max post size). Practically speaking, the encoded buffer is probably just slightly going over the 200000 limit, but still anything over 200000 will cause the errors you're seeing.

Anyway, choose a fix and let me know how it goes.

That did the trick. Tnx muchly...

[Slugger]

Which fix did you go with?

[SteveW]

Originally Posted by Slugger

Which fix did you go with?

Fix #1, as per the online manual.

[Slugger]

All SJQ users who have not increased Jetty's max post request size should decrease the task client buffer from the default value of 199,000 to 66,666 or lower in order to avoid this bug. SJQ v3.0.2 will automatically reset this for you. Gory details on the bug available here. Also, star the issue if you want to be notified when a task client with the fix is available.

[superdodge]

I posted this as an issue on the code.google site.

[Slugger]

Originally Posted by superdodge

Slugger,

I've written a script now that works perfectly with VideRedo. Thanks so much for your help.

I'm still having an issue where a large percentage of jobs (~25%) never appear to be completed in SJQ Queue. The programs/scripts they are running are complete and they all return the correct return status when I run them through other interfaces. This happens with Comskip, ShowAnalyzer and with my own Batch files.

What happens to these jobs that don't finish in SJQ? You say the actual external processes complete fine? Do the tasks get moved to failed state in SJQ or do they stay in the active queue in running state for ever? If the latter, then do the processes stay listed in your Windows task manager?

The client logs don't log too much and if the cause of all of this is the task client crashing then those client logs aren't going to record that problem. What you need to do is to look in the wrapper.log file. It's found in the logs subdirectory where you installed the task client. Have a look at the wrapper.log file and see if any exceptions are being logged in there, if so, that's what I need to see.

[superdodge]

Originally Posted by Slugger

What happens to these jobs that don't finish in SJQ? You say the actual external processes complete fine? Do the tasks get moved to failed state in SJQ or do they stay in the active queue in running state for ever? If the latter, then do the processes stay listed in your Windows task manager?

The client logs don't log too much and if the cause of all of this is the task client crashing then those client logs aren't going to record that problem. What you need to do is to look in the wrapper.log file. It's found in the logs subdirectory where you installed the task client. Have a look at the wrapper.log file and see if any exceptions are being logged in there, if so, that's what I need to see.

You responded too quickly! I decided to move the issue to the google code site, but you saw the post here before I took it down.

When this happens the items stay in the active queue with a "Running" status until I manually kill them. The processes do not stay listed in the Windows Task Manager.

I am going to dig through wrapper.log right now.

Thanks again
-RL

[superdodge]

From wrapper.log:

Launching a JVM...
INFO | jvm 1 | 2010/01/26 19:54:01 | WrapperManager: Initializing...
INFO | jvm 1 | 2010/01/26 19:54:01 | SJQ Task Client v3.0.2.750
INFO | jvm 1 | 2010/01/26 19:54:01 | Using 'C:/sjqc' as location for configuration file...
INFO | jvm 1 | 2010/01/26 19:54:02 | Connection refused: connect
INFO | jvm 1 | 2010/01/26 20:01:10 | Exception in thread "Thread-6" java.lang.OutOfMemoryError: Java heap space
INFO | jvm 1 | 2010/01/26 20:01:10 | at java.nio.HeapCharBuffer.<init>(Unknown Source)
INFO | jvm 1 | 2010/01/26 20:01:10 | at java.nio.CharBuffer.allocate(Unknown Source)
INFO | jvm 1 | 2010/01/26 20:01:10 | at com.google.code.sagetvaddons.sjqc.IOMonster.<init>(Unknown Source)
INFO | jvm 1 | 2010/01/26 20:01:10 | at com.google.code.sagetvaddons.sjqc.SJQThread.run(Unknown Source)
INFO | jvm 1 | 2010/01/26 20:01:10 | Exception in thread "Thread-11" java.lang.OutOfMemoryError: Java heap space
INFO | jvm 1 | 2010/01/26 20:01:10 | at java.io.Reader.read(Unknown Source)
INFO | jvm 1 | 2010/01/26 20:01:10 | at com.google.code.sagetvaddons.sjqc.IOMonster.run(Unknown Source)
INFO | jvm 1 | 2010/01/26 20:01:10 | Exception in thread "Thread-7" java.lang.OutOfMemoryError: Java heap space
INFO | jvm 1 | 2010/01/26 20:01:10 | at java.nio.HeapCharBuffer.<init>(Unknown Source)
INFO | jvm 1 | 2010/01/26 20:01:10 | at java.nio.CharBuffer.allocate(Unknown Source)
INFO | jvm 1 | 2010/01/26 20:01:10 | at com.google.code.sagetvaddons.sjqc.IOMonster.<init>(Unknown Source)
INFO | jvm 1 | 2010/01/26 20:01:10 | at com.google.code.sagetvaddons.sjqc.SJQThread.run(Unknown Source)
INFO | jvm 1 | 2010/01/26 20:01:10 | Exception in thread "Thread-13" java.lang.OutOfMemoryError: Java heap space
INFO | jvm 1 | 2010/01/26 20:01:10 | at java.io.Reader.read(Unknown Source)
INFO | jvm 1 | 2010/01/26 20:01:10 | at com.google.code.sagetvaddons.sjqc.IOMonster.run(Unknown Source)
INFO | jvm 1 | 2010/01/26 22:53:29 | Exception in thread "Thread-44" Exception in thread "Thread-45" java.lang.OutOfMemoryError: Java heap space
INFO | jvm 1 | 2010/01/26 22:53:29 | java.lang.OutOfMemoryError: Java heap space
INFO | jvm 1 | 2010/01/26 22:53:29 | Exception in thread "Thread-43"
INFO | jvm 1 | 2010/01/26 22:53:29 | java.lang.OutOfMemoryError: Java heap space
INFO | jvm 1 | 2010/01/26 22:53:29 | Exception in thread "Thread-47" java.lang.OutOfMemoryError: Java heap space
INFO | jvm 1 | 2010/01/27 06:21:09 | Exception in thread "Thread-66" java.lang.OutOfMemoryError: Java heap space
INFO | jvm 1 | 2010/01/27 06:21:09 | Exception in thread "Thread-70" java.lang.OutOfMemoryError: Java heap space
INFO | jvm 1 | 2010/01/27 09:23:22 | Exception in thread "Thread-90" java.lang.OutOfMemoryError: Java heap space
INFO | jvm 1 | 2010/01/27 09:28:22 | Exception in thread "Thread-92" java.lang.OutOfMemoryError: Java heap space
STATUS | wrapper | 2010/01/27 10:27:14 | Machine is shutting down.
STATUS | wrapper | 2010/01/27 10:27:16 | <-- Wrapper Stopped

I currently have my SageTV JVM Max Memory set to 1,040,512 M

[Slugger]

What does your sjqc.properties file look like? FYI: The task client runs in its own JVM so it would be using the default heap size. However, unless you've jacked up the task client buffer value, I don't see any reason why you'd be exhausting the heap.

[superdodge]

sjqc.properties

#Auto generated by SJQC
#Wed Jan 27 10:28:10 EST 2010
buffer=5241880
port=8081
host=localhost
is_ssl=false
sleep=60

[Slugger]

Originally Posted by superdodge

sjqc.properties

#Auto generated by SJQC
#Wed Jan 27 10:28:10 EST 2010
buffer=5241880
port=8081
host=localhost
is_ssl=false
sleep=60

Ah ha! I love easy to fix problems. You increased the wrong buffer. With the default heap values, a 5MB task client buffer is exhausting the Java heap space. Reset that buffer back to 199,000 (unless you're not going to adjust the Jetty max post value then you should lower it to 66,000). The Jetty max post value needs to be increased to the value you have above. Make sure the JettyStarter.properties value was increased as described in the user guide and then lower this buffer value.

There are alternatives if you increase the Jetty max post value. After increasing the Jetty max post value, you could then alternatively increase the heap size for the JVM running the task client if you wanted to keep a 5MB task client buffer. However, I don't see a need for a 5MB task client buffer. My recommended solution: increase Jetty's max post setting and reset this buffer to a value between 128,000 and 200,000.

Please confirm if that gets everything working again.

[superdodge]

I see the appropriate passage in the SJQ3 User's Guide now.

Thank you SO much for your help.

Create Client Configuration File

Before attempting to start the task client for the first time, you need to create the configuration file. Not doing so before starting the task client for the first time will automatically create a default settings file, which is very likely to be wrong for almost every setup. Simply create a text file in the following location:

Windows: C:\sjqc\sjqc.properties
Linux: /etc/sjqc/sjqc.properties

Open up this file and add the following content:

host=localhost
port=8008
buffer=199000
sleep=300
is_ssl=false

Replace the values as appropriate for your Jetty plugin installation. Host is the host name or IP address where your Jetty server is running (default value is localhost). The port is the port number that Jetty is listening on (default is 80 or 443 if SSL). Buffer is the amount of data (in bytes) that the task client will buffer for each running task before flushing it to the server for logging. You should keep this value at 199,000 or lower. If you plan to increase it above this value then you need to reconfigure Jetty to accept larger POST requests (see details on how to do this in the installation section at the beginning of this document). Setting this value lower will flush the output more frequently and allow you to view your task's output while it's running, but too low a value (say, less than 10,000) will cause undesired network traffic between the client and server, which may cause performance issues with your system. The sleep setting tells the task client how frequently, in seconds, to contact the server looking for new work to be done. The default value is 300 (five minutes) and any value below 30 will be reset to 30. And finally, set the is_ssl to true if the client should only speak to Jetty via an encrypted (SSL) connection.

Save this file and then proceed with starting up the task client process. If you need to change where the client reads the configuration settings from then read up on the command line argument available to the task client application.

[jhh]

hi,

I've had a challenge with the _RELINK command - it seems that in rare cases the result is that SageTV thinks that the tvshow is a music album after linking a new file to an existing airing.

I tried to remux a .ts to a .mpg as I thought my player would understand .mpg better in combination with external subtitles. VLC gets the conversion done nicely but when I linked the new file I got the show in archived files (documented and easy to fix) but also as a music album. I have no clue why/when - I tested a show 2 airings same name different episode - one had the problem the other one did not.

In the mean time I found out that my player does behave nicely so I don't need to replace files anymore. So use with care.

Apart from that - really beautiful tool - learning curve is worth the result!

Jan

[Slugger]

If a file ends up listed as music, then it's likely a result of SageTV's import. All I do in the plugin is add the file to the SageTV library - SageTV processes the file and decides what type of file it is, etc. It seems your conversion results are confusing SageTV as to the type of file it is. I have not seen this behaviour personally and if you constantly see it then you might be inclined to open a bug report with Sage, post the file that's being listed in music for them to download and have them investigate the import issue.

If you do decide to open a bug report with Sage, you may want to point them to the code I'm using to do the relink task. It is available here:

http://code.google.com/p/sagetv-addo...gRelinker.java

[jhh]

Originally Posted by Slugger

If you do decide to open a bug report with Sage, you may want to point them to the code I'm using to do the relink task.

hm - I suspected this a bit as there are no parameters to specify which indicate the type of media - I will file a bug report.

Thanks!

[ChaOConnor]

Slugger,

SJQ is running fine, but whenever I try to get my server logs I get this:

HTTP ERROR 500

Problem accessing /sjq/SJQ. Reason:

sjq.log (The system cannot find the file specified)

Caused by:

java.io.FileNotFoundException: sjq.log (The system cannot find the file specified)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(Unknown Source)
at java.io.FileInputStream.<init>(Unknown Source)
at java.io.FileReader.<init>(Unknown Source)
at com.google.code.sagetvaddons.sjq.server.SJQServlet.doPost(SJQServlet.java:263)
at com.google.code.sagetvaddons.sjq.server.SJQServlet.doGet(SJQServlet.java:100)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:389)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.handler.rewrite.RewriteHandler.handle(RewriteHandler.java:230)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:539)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:520)


Powered by Jetty://

any ideas?