Plugin: Sage Job Queue (SJQ)

[matt91]

Originally Posted by wayner

I have been putting together some docs from the perspective of a new user of SJQ.

any chance you'll be willing to share soon?

I just installed SJQ, but am just staring at the client interface without a clue as to what to do. I see lots of code here in this thread, but I'm not sure if i'm supposed to cut and paste it into the ruleset editor to work with or what.

The wiki docs were helpful to get everything installed, but they didn't really speak to setting anything up.

thanks
Matt

[Slugger]

Originally Posted by matt91

any chance you'll be willing to share soon?

I just installed SJQ, but am just staring at the client interface without a clue as to what to do. I see lots of code here in this thread, but I'm not sure if i'm supposed to cut and paste it into the ruleset editor to work with or what.

The wiki docs were helpful to get everything installed, but they didn't really speak to setting anything up.

thanks
Matt

The client config nor the ruleset syntax wiki docs were helpful?

Being the author/coder of this, I can easily understand where the docs can become lacking rather quickly (I take a lot of this stuff for granted as I try to document it). What is it you're trying to accomplish and what are the wiki docs lacking? Hopefully wayner has got something he can share.

[wayner]

What I was concentrating on was more basic stuff like - do you need both the client and server even if you are just running SJQ on your server?

Here is the first cut without talking about the UI too much. I also try to stay away from what is already in the wiki: (Note that hyperlinks didn't copy over from Word, also note that this doesn't contain nearly as much detail as I plan to include when fully fleshed out)

Purpose
SJQ is a plugin for SageTV written by Slugger that facilitates the processing of video files, generally Recorded TV files. It is normally used to run internal Sage processes or external programs that do tasks such as marking commercials (using programs like Comskip or Showanalyzer), converting Recorded TV shows to other formats (such as Divx, iPod, etc.) or even cutting out the commercials out of the shows (using VideoReDo). There are other Sage addins that do some of this functionality, such as Comskip Monitor and Autocompress, but SJQ gives more flexibility at the cost of requiring a bit initial setup.

Background Info
SJQ uses the jetty Java based web server. Jetty is also used by several other Sage plugins including SageRecordingExtender (also written by Slugger) and the Sage Mobile Web server so you may already have jetty installed on your Sage PC.

Installation
SJQ consists of two components – a server and a client. Even if you will only be running SJQ on your server you must install both the server and client. There are two versions of the client – a windows service version and a regular version. The files are located here. Download the SJQ server software (i.e. sjq-2.1x.xxx.zip) and the client software (i.e. sjqc-2.1x.xxx.zip). You may need to also install Java on your PC if you get an error when you try to run the client software – Java is available here. Follow the instructions to install the server software here and the client software from here.

The ruleset on your server is a number of IF statement that let you select what files that you want to process. If true this IF statement will then launch a task which will be run on one of your clients. The one exception to this is Sage internal commands (such as _TRANSCODE) which can be run as part of the server ruleset.

To Start go into the Edit Rules portion of the server and put in some rules - you will find some good examples in the wiki. You might want to limit the files that will pass successfully (otherwise your PC could be running for days) - you could do this by making sure that only one file successfully passes all test - i.e. put in a really strict test like Title =="Seinfeld" && Episode =="The Contest".

SJQ contains a number of media variables that help to facilitate the IF statements. These media variables contain items such as file name, file path, etc.

Note for Comskip users (This may now be obsolete in the most current version)
When using comskip it is quite likely that SJQ will think that your jobs have failed. In order to get around this you have to use a batch file to return a successful code. Here is a sample batch file (this assumes that you have comskip in c:\comskip):
cd c:\comskip
c:
start /b /w /belownormal comskip.exe -n "%~f1"
set comskiperror=%ERRORLEVEL%
if %comskiperror% EQU 1 exit 0
if %comskiperror% NEQ 1 exit 1

Transcoding
SJQ will report Transcoded jobs as complete as soon as they are submitted to Sage. You can check the progress of these Jobs by going into Sage’s video conversions page (MyMenu|Video Conversions on SageMC)

Accessing SJQ
You access JQ via your web browser by pointing it at http://192.168.1.99/sjq/SJQ (replace the 192.168.1.99 with the IP address or name of your server PC).

You will likely have a password on SJQ. It is likely the same for all Jetty clients and is set in the realm.properties file.

[Slugger]

Originally Posted by wayner

You will likely have a password on SJQ. It is likely the same for all Jetty clients and is set in the realm.properties file.

SJQ uses its own authentication scheme. The default password is sjqadmin and the password is encrypted in the sjq database. There is no way to retrieve forgotten passwords (well there is, but will require me to write a Java program to reset the encrypted password within your database file).

[wayner]

Sorry about that - my memory must be going.

[ryansebiz]

Slugger,

Thanks for all your hard work on this.

I just installed this tonight and have a suggestion on something that tripped me up.

The JettyInstall uses port 8080.

In your install Wiki, under the "Deploy" header you say to use:

http://192.168.0.1:8000/sjq/SJQ

I think it would be more helpful if you edit this to match the Jetty port:

http://192.168.0.1:8080/sjq/SJQ

[matt91]

Originally Posted by Slugger

The client config nor the ruleset syntax wiki docs were helpful?

Originally Posted by wayner

What I was concentrating on was more basic stuff like - do you need both the client and server even if you are just running SJQ on your server?

Thanks, gents.

Slugger, they certainly provide the detail needed, but not the basics. How does one utilize (e.g., where do I paste) the rules that are in those documents? I see, in what appears to be the client interface, a screen at Tools > Ruleset Editor. I assume that's where something will go.

However, what about the server-side? I have no idea where those rules go. Into the same place? A different file?

What might be helpful is an actual example. Say running Comskip on a single series or a single channel or something. Perhaps even building on it such that:

this example will run comskip on all Seinfeld episodes

this example will run comskip on all Seinfeld episodes recorded on channel 212

this example will run comskip on all Seinfeld episodes recorded on channel 212, only 1am - 4am

etc...I think you get the idea. Providing the actual rules with instructions on where they go would be immensely helpful.

Again, thanks for all the effort.

Matt

[tomfisk]

Slugger, tell me more about these tests. It seems to me that most of the tests are associated with a media object (i.e. Recorded TV program). Are these associated with a media object or not?

From the behavior I've seen, each rule runs at most once for each media object. So...are these tests only going to run once for each media object, or are they not tied to a media object and will fire every time the QueueLoader runs?

BTW, SJQ is a great utility!

[Slugger]

Originally Posted by tomfisk

Slugger, tell me more about these tests. It seems to me that most of the tests are associated with a media object (i.e. Recorded TV program). Are these associated with a media object or not?

From the behavior I've seen, each rule runs at most once for each media object. So...are these tests only going to run once for each media object, or are they not tied to a media object and will fire every time the QueueLoader runs?

BTW, SJQ is a great utility!

Each run of the QueueLoader does the following (with gory details removed):

1. Gets list of all media objects and filters based on selected media mask (as defined in SJQ settings - default is to only scan tv recordings).

2. For each media object gathered in step 1, perform each test block on that media object. If the result of the test block is true then goto step 3, otherwise skip step 3.

3. For each task defined for the test block, register that task in the active queue if and only if that task has not been attempted for this media object. If it has then drop it and move on to the next task defined.

Exception: If the user told SJQ not to register failed jobs then the insertion into the active queue is only dropped if the task has already completed successfully for the given media object.

I'll admit, this probably isn't the greatest explanation. If you can read Java code, then tracing the QueueLoader and RulesParser classes is probably a better way to understand what's going on.

[tomfisk]

OK...I understand.

How about creating an "infrastructure" (or some such similarily named) object that is always in each QueueLoader list and is never marked as completed for any task. Then have a test named "IsInfrastructure" that tests true on this object.

What this would allow one to do is to check tests that aren't associated with a media object (like IsClientConnected) and perform tasks.

You follow where I am going?

Thanks!

[Slugger]

Originally Posted by tomfisk

OK...I understand.

How about creating an "infrastructure" (or some such similarily named) object that is always in each QueueLoader list and is never marked as completed for any task. Then have a test named "IsInfrastructure" that tests true on this object.

What this would allow one to do is to check tests that aren't associated with a media object (like IsClientConnected) and perform tasks.

You follow where I am going?

Thanks!

What do you want to do outside the context of a media object? SJQ's purpose is to allow you to perform actions on your media objects (files). I understand (I think) what you're asking for, just not sure of how you plan to use it?

[tomfisk]

I have a service that runs a Kixstart script which uses Windows Performance Metrics to see if Sage or selected activities are active, and if not hibernates the system.

I have SJQ tasks that create a flag file when IsActivelyRecording flips that lets this process know that Sage is doing something. However, when a client is connected there is the possibility that my script would signal inactivity (right now I am watching network packet rates as a proxy for client connection...but if the client is sitting on a menu there is no net packets). So, if I can test to see if a client is connected, I can flip my flag file so my script won't hibernate the system.

That's my immediate use but I could imagine other possiblities for housekeeping tasks.

[tomfisk]

Another quick one...how do I check if an object is the result of watching "Live TV" and don't want to process it because it's not going to be saved when I stop "Live TV"?

[Slugger]

Originally Posted by tomfisk

I have a service that runs a Kixstart script which uses Windows Performance Metrics to see if Sage or selected activities are active, and if not hibernates the system.

I have SJQ tasks that create a flag file when IsActivelyRecording flips that lets this process know that Sage is doing something. However, when a client is connected there is the possibility that my script would signal inactivity (right now I am watching network packet rates as a proxy for client connection...but if the client is sitting on a menu there is no net packets). So, if I can test to see if a client is connected, I can flip my flag file so my script won't hibernate the system.

That's my immediate use but I could imagine other possiblities for housekeeping tasks.

Ok, I understand what you're trying to do, just no way for it to happen in SJQ right now. Even if I added the pseudo media object for you to test against, I'd have to restrict the use of so many tests (because they'll cause null pointers when I go looking in wiz.bin) that it's just too much of a hack to add.

Alternatively, you could disable the registering of failed jobs (foregoing the other problems this could introduce) and do something like this:

if [Title == "Something You Have Archived and Will Always Be Around" && IsClientConnected == true] {
  YOURTASK
}

Replace the title with something that you have archived and will always be scanned on each QueueLoader run. Then make sure YOURTASK always fails (make it do what it's suppose to do but return a fail code to SJQ) and then you can ensure that this task will run on each run of the QueueLoader. Make sense?

Originally Posted by tomfisk

Another quick one...how do I check if an object is the result of watching "Live TV" and don't want to process it because it's not going to be saved when I stop "Live TV"?

Use the IsScheduledRecording test.

[tomfisk]

Yep...makes sense. I did think about using an archived recording along the way but that was early in my understanding of how the QueueLoader worked.

Thanks again!

[matt91]

Slugger - are you able to help me get started?

thanks
matt

[Slugger]

Originally Posted by matt91

Slugger - are you able to help me get started?

thanks
matt

The RulesetSyntax wiki doc: those examples go into the Ruleset editor - from the options menu (tools menu if using a recent snapshot version).

The ClientConfig syntax wiki doc: those examples go into the client config which you get to by clicking on Clients then clicking on a client listed.

Example rules as you requested already exist in the wiki docs, if you want some examples expanded on then let me know what you're looking to do.

[personalt]

I think I am close but nothing makes the queue. I was looking at the client logs. There is a recent error that is from the last time I ran it tonight..

--------------------
Date: Sun Mar 01 23:56:57 EST 2009

HTTP ERROR: 500
Unexpected '#' on line 1
RequestURI=/sjq/SJQ

Caused by:
java.lang.RuntimeException: Unexpected '#' on line 1
----------------

Is sjqc.properites that is the problem?

Ever time I see this error I stop the service and delete the first two lines which start with #. But after the restart of the service the error returns.

Is that the right file?
How can I erase the client logs?

My properities file just has 3 lines
port=8080
host=localhost
is_ssl=false

but when I restart the client service the sjqc.properties files grows by two lines, the first is:
#Auto Generated by SQJC
if I delete the two # lines and restart the pc the lines just re-reappear

Is that problem my error above?

[Slugger]

Originally Posted by personalt

I think I am close but nothing makes the queue. I was looking at the client logs. There is a recent error that is from the last time I ran it tonight..

--------------------
Date: Sun Mar 01 23:56:57 EST 2009

HTTP ERROR: 500
Unexpected '#' on line 1
RequestURI=/sjq/SJQ

Caused by:
java.lang.RuntimeException: Unexpected '#' on line 1
----------------

Is sjqc.properites that is the problem?

Nope, when you look at the client logs then it's the client config that the errors are referring to. So bring up the list of connected clients and then instead of looking at the logs, click on the IP address of the client, which will bring up the client config for that client. This is the input that SJQ is complaining about. If you're using a recent snapshot of SJQ (with the new GUI) then click on a client and choose 'Configure' from the context menu that appears.

Note to all: When asking for support please try to mention the version and build number you're running - makes it a lot easier for me to know how to answer. All I need is the complete version reported in the GUI (2.x.y.z). The final number (z) being most critical as it tells me exactly which SVN revision of the code you're running.

[personalt]

If I bring up the client config I have this... I dont have any # in this config but it is syntactically correct? I saw back in the earlier pages talk of calling a comskip.bat but many of the recent posts seem to call comskip directly.

I am using v2.1.1.236

Only other weird thing I see is that when via IE I go to Settings Dropdown and the settings menu it says error on page. but all the other pages, including the server rules work. I included the server rules below just in case.

Thanks... I am on hour 5 or 6 of this.. I am sure it is something stupid at this point but I just cant see it.

//Client options
:MAXPROCS 1
// TASK definition
COMSKIPMPEG {
:CPU LOW
:MAX 1
"C:\\comskip\\comskip.exe -n \"%c%\""
}


----
Server Rules

if [IsTV == "true" && IsHD == "false" && FileExists != "%d%\\%p%.edl" && Filename =$ ".mpg" && IsActivelyRecording == "false" && ChannelNumber != "0|1|1400" && IsScheduledRecording == "true"]
{
COMSKIPMPEG
}

Originally Posted by Slugger

Nope, when you look at the client logs then it's the client config that the errors are referring to. So bring up the list of connected clients and then instead of looking at the logs, click on the IP address of the client, which will bring up the client config for that client. This is the input that SJQ is complaining about. If you're using a recent snapshot of SJQ (with the new GUI) then click on a client and choose 'Configure' from the context menu that appears.

Note to all: When asking for support please try to mention the version and build number you're running - makes it a lot easier for me to know how to answer. All I need is the complete version reported in the GUI (2.x.y.z). The final number (z) being most critical as it tells me exactly which SVN revision of the code you're running.