Plugin: Sage Job Queue (SJQ) v3

[Slugger]

Given the sweeping changes between SJQ v1 and the new SJQ v3, I feel it appropriate to start a new thread specifically for SJQ v3 support.

Please post only about SJQ v3 in this thread. SJQ v2 support can be had in this thread.

SJQ v3 is now officially open to support questions.

Please visit the project download site and grab the latest version (3.0.0.640 as of this post). Be sure to read the beta upgrade wiki before upgrading from SJQ v2. New users to SJQ v3 can ignore that wiki doc.

An initial draft of the SJQ v3 User's Guide is now available. It's a draft, it's rough, but most of the examples have been updated. Please feel free to point out errors and omissions in the doc.

A quick cheat sheet for users migrating from v2 to v3 will be available in a day or two (or three or four, i.e. some time sort of soon). Most of the tips to be included in that doc are scattered throughout the old support thread ($ object, kill active processes, IsObj* tests, etc.).

Please note, SJQ v3 is still in beta!

[nyplayer]

Slugger,

Question about _RUNQLOADER can this be run more than once.

for example

Rule # 1 runs a task QFIX against a media object then _RUNQLOADER

Rule # 2 runs COMSKIP against the same media object can I schedule another _RUNQLOADER in rule 2

[tomfisk]

I just upgraded to 3.0.0.640. During the upgrade I shutdown the SageTV service, made a copy of the sjq.sqlite file, and installed the new version. The sjq.war file in the jetty/webapps directory shows a date of 11/4/09.

When I get to the login screen it isn't recognizing my password nor the default "sjqadmin". Additionally, when I click on the "About" under help it shows a dialog box with 2.1.5.484 as the version...???!!!

I did upgrade to the 1.6 Jetty release...and tested the old SJQ (coincidentally 2.1.5.484) to make sure the jetty installation was stable...everything worked fine. I also switched over to the jetty version of the webserver and that is running without problem.

So...is something being cached somewhere that needs to be flushed? Did I miss something in the upgrade?

[Slugger]

Originally Posted by nyplayer

Slugger,

Question about _RUNQLOADER can this be run more than once.

for example

Rule # 1 runs a task QFIX against a media object then _RUNQLOADER

Rule # 2 runs COMSKIP against the same media object can I schedule another _RUNQLOADER in rule 2

You're going to have to test that and let me know. I believe it's only going to run it once. Actually, it'd be a bug if it ran it more than once (for the same media file). I was aware of this when I wrote those internal tasks and am trying to think of a way to make those tasks "special" such that they don't attach themselves to a specific object, but instead you can just throw them into a rule block and have them run. My initial result was that it wouldn't be easy so I just implemented them as they are now (i.e. you can only run the tasks once per scanned object).

I can see a need/desire to be able to trigger those two tasks arbitrarily and more than once (actually the use case that triggered the addition of those tasks would probably want this behaviour as well).

Those tasks are likely to change before v3 is moved out of beta. For now, however, I think they're only going to run once (per media file). Please let me know if that's what you're seeing.

[Slugger]

Originally Posted by tomfisk

I just upgraded to 3.0.0.640. During the upgrade I shutdown the SageTV service, made a copy of the sjq.sqlite file, and installed the new version. The sjq.war file in the jetty/webapps directory shows a date of 11/4/09.

When I get to the login screen it isn't recognizing my password nor the default "sjqadmin". Additionally, when I click on the "About" under help it shows a dialog box with 2.1.5.484 as the version...???!!!

I did upgrade to the 1.6 Jetty release...and tested the old SJQ (coincidentally 2.1.5.484) to make sure the jetty installation was stable...everything worked fine. I also switched over to the jetty version of the webserver and that is running without problem.

So...is something being cached somewhere that needs to be flushed? Did I miss something in the upgrade?

Probably just your browser's cache. If you're indeed running SJQv3 then in the root install dir of Sage should be the sjq.log file. If that file's there and it's being updated periodically then SJQv3 is running on the server. And if so, then you just need to wipe the browser cache to ensure you're getting the new GUI (the 2.1.5 GUI you're seeing definitely will fail to communicate the pwd [or anything] to a SJQv3 server).

[nyplayer]

Originally Posted by Slugger

You're going to have to test that and let me know. I believe it's only going to run it once. Actually, it'd be a bug if it ran it more than once (for the same media file). I was aware of this when I wrote those internal tasks and am trying to think of a way to make those tasks "special" such that they don't attach themselves to a specific object, but instead you can just throw them into a rule block and have them run. My initial result was that it wouldn't be easy so I just implemented them as they are now (i.e. you can only run the tasks once per scanned object).

I can see a need/desire to be able to trigger those two tasks arbitrarily and more than once (actually the use case that triggered the addition of those tasks would probably want this behaviour as well).

Those tasks are likely to change before v3 is moved out of beta. For now, however, I think they're only going to run once (per media file). Please let me know if that's what you're seeing.

Yes it only runs once per media object... being able to run _SCANMEDIA and _RUNQLOADER without being attached to an object would be great.

[tomfisk]

Originally Posted by Slugger

Probably just your browser's cache. If you're indeed running SJQv3 then in the root install dir of Sage should be the sjq.log file. If that file's there and it's being updated periodically then SJQv3 is running on the server. And if so, then you just need to wipe the browser cache to ensure you're getting the new GUI (the 2.1.5 GUI you're seeing definitely will fail to communicate the pwd [or anything] to a SJQv3 server).

Yep...that was it...very strange that a CTRL/Refresh didn't even work...had to actually go and delete the cache from the settings (Firefox).

[tomfisk]

I've pre-pended all my rules with "IsObjMediaFile == true" but I'm still getting a slew of

2009-11-06 15:47:45,046 WARN [RulesParser]: Unable to invoke method 'GetShowTitle'; return false for test!

warnings.

Any ideas?

[nyplayer]

Originally Posted by tomfisk

I've pre-pended all my rules with "IsObjMediaFile == true" but I'm still getting a slew of

2009-11-06 15:47:45,046 WARN [RulesParser]: Unable to invoke method 'GetShowTitle'; return false for test!

warnings.

Any ideas?

it should look like this.

IsObjMediaFile == "true" && $.GetMediaTitle{} == "NCIS"

[Slugger]

nyplayer is correct. Hopefully this section from the new SJQv3 User Guide will help.

Unfortunately, this is once again a case of the author of the code also authoring the docs so I'm probably skipping over some details just because I consider them "minor" b/c of my familiarity with everything. The basic idea from that link above is you need to test what kind of object you're scanning (via IsObjMediaFile or IsObjSysMsg) and once you know what kind of object you have then you can call tests against it via $.MethodName{}. The table of tests below that example shows which tests have been removed in SJQv3 and how to access similar info via calls to the $ object.

EDIT: You could also call $.GetMediaTitle{} as nyplayer suggested. $.GetShowTitle{} is also valid. For media objects, you can call methods from MediaFileAPI, ShowAPI. AiringAPI, or ChannelAPI, which is why either one would be valid.

[Slugger]

Originally Posted by nyplayer

Yes it only runs once per media object... being able to run _SCANMEDIA and _RUNQLOADER without being attached to an object would be great.

Issue ticket 91 details the solution I plan on implementing for this. Hopefully it makes sense? Basically going to drop these internal tasks and make them options that the task queue will then run after the task completes successfully.

if [IsObjMediaFile == true && $.GetShowTitle{} == "Seinfeld"] {
   :SCANMEDIA "true"
   :RUNQLOADER "true"
   :PRIORITY 100
   MYTASK
}

And so if MYTASK completes successfully then the task queue will also run a media scan then the queue loader (with the media scan always occurring before the queue loader is triggered). Busy weekend so probably won't be until early next week before I get a chance to implement this, but the idea popped into my head late last night so I thought I should write it down.

[Slugger]

Snapshot 647 contains the fix to move those two internal tasks to task options; implemented exactly as described above.

[dislecksea]

Just did the install (SJQ v3.0.0.647) and I am getting an error. FYI - I am just going for the basic "only process scheduled recordings using comskip" approach, so I have copied the rule directly from the wiki:

if [IsObjMediaFile == true && $.IsTVFile{} == true && $.IsNotManualOrFavorite{} == false] {
:PRIORITY 10 # Optional, default priority is zero
COMSKIP
}

But when I run it nothing happens and the log I get says:

2009-11-09 08:27:18,453 INFO [MediaFileQueueLoader]: Thread started
2009-11-09 08:27:18,453 INFO [ClientMonitor]: Thread started
2009-11-09 08:27:18,453 INFO [InternalTaskClient]: Thread started
2009-11-09 08:27:18,468 INFO [SystemMessageQueueLoader]: Thread started
2009-11-09 08:27:19,109 INFO [DataStore]: SQLite driver implementation: 'native'
2009-11-09 08:27:19,109 INFO [DataStore]: DataStore connection pool initialized with 5 connections.
2009-11-09 08:27:19,203 ERROR [MediaFileQueueLoader]: Unexpected exception
java.lang.RuntimeException: Invalid field name, 'IsTV', on line 1
at com.google.code.sagetvaddons.sjq.server.RulesParser$Evaluator.internalTest(RulesParser.java:210)
at com.google.code.sagetvaddons.sjq.server.RulesParser$Evaluator.runTest(RulesParser.java:135)
at com.google.code.sagetvaddons.sjq.server.RulesParser$Evaluator.evalTest(RulesParser.java:122)
at com.google.code.sagetvaddons.sjq.server.RulesParser$Evaluator.(RulesParser.java:72)
at com.google.code.sagetvaddons.sjq.server.RulesParser$Evaluator.(RulesParser.java:101)
at com.google.code.sagetvaddons.sjq.server.RulesParser.loadQ(RulesParser.java:258)
at com.google.code.sagetvaddons.sjq.server.RulesParser.(RulesParser.java:232)
at com.google.code.sagetvaddons.sjq.server.MediaFileQueueLoader.run(MediaFileQueueLoader.java:71)
at java.lang.Thread.run(Unknown Source)


Any thoughts on what I screwed up?

Thanks

[Slugger]

The example works as expected when I test it here, so if you did a copy/paste from the user's guide then check over it again and ensure there's no spaces accidentally inserted (check around $.IsTVFile{} b/c that's where the parser is complaining). If you upgraded from SJQ v2, then make sure your old rulesets are either removed or updated. 'IsTV' was the name of a test in v2 that is no longer valid in v3 and would need to be replaced with $.IsTVFile{} under SJQv3.

[kfontenot]

I know that you can no longer check for the archive flag in version 3, but is there any other flag that I could set manually and check for other than "Watched"? I would like to set some sort of flag to a recording and have it moved to a different drive using the new MOVEREC task. I don't want to set any rules that check for channel number or show id.

[Slugger]

Originally Posted by kfontenot

I know that you can no longer check for the archive flag in version 3, but is there any other flag that I could set manually and check for other than "Watched"? I would like to set some sort of flag to a recording and have it moved to a different drive using the new MOVEREC task. I don't want to set any rules that check for channel number or show id.

You can still set the archive flag with the _ARCHIVE task and you can still check for the flag, it just requires two tests:

($.IsTVFile{} == true && $.IsLibraryFile{} == true)

If that test is true then the file is an archived tv recording.

[gveres]

Slugger, great work. I just upgraded to v3 tonight to try out the SCANMEDIA and RUNQLOADER features that I asked for. They work perfectly, thank you very much for putting them in. It makes the entire system run so much smoother.


The above post just gave me an idea. I have been recording shows off the HD movie network channels and I then drag them over to my mac pro to run handbrake on them and put them into one of my video directories. Since I upgraded my server to a Core i5, I now have enough horsepower on the server to have the server do the work. I will have to see if those recordings are also flagged with isTV. I think what you and kfontenot are saying is that if I mark something as archived in the GUI, the rule would then trigger and it would be moved to the videos folder. That would definitely save some effort.


Greg

[kfontenot]

Thanks Slugger!! Would I also need this test as well : IsObjMediaFile == true ? I'm thinking not. I think the other tests would not pass if if it were not a media file.

[Slugger]

You don't need it, but it's recommended. You should always check to see what kind of object you're processing before calling API methods against it. Not doing so could cause a lot of headaches under certain circumstances (SJQ can handle it, but you may end up doing some things you don't expect).

[Slugger]

To illustrate my point from above:

# Run commercial flagger against all recordings that aren't watched except episodes of "Family Guy"
if [$.GetShowTitle{} != "Family Guy" && $.IsWatchedCompletely{} == false] {
   COMSKIP
}

If you don't check IsObjMediaFile == true then SJQ will insert any object that isn't a media file into the queue and try to run comskip on it, which for now is just system messages, but there are plans to add more types of objects to be scanned in the future. So you always want to do the IsObj* == true check.