Plugin: Sage Job Queue (SJQ)

[gveres]

I added the new rule and it didn't trigger. There must be something else. I will dig in when I have better access to the machine.

Greg

[Slugger]

Turn on debug logging from the settings then run the media debugger again with that new rule in there and check the logs. The debug logging should give more details on why it didn't add that task to the queue. Report back your findings...

[gveres]

You were correct that it wasn't running the rule because SJQ thought it has run the rule.

I removed all the completed entries (by clearing the grid, which doesn't clear teh entire grid). Once I did that and re-ran the Queueloader, the jobs showed up.

Greg

[Slugger]

To any SJQ users who are also Java coders:

I've just committed r638 to the project. This commit adds the new _MOVEREC task to SJQ (v3 only). This task allows you to automate the moving of your tv recordings while SageTV is running. Use of this task eliminates the need to stop SageTV to move tv recordings around.

I've been running this code for about a week now and it's been automatically moving my syndicated recordings over to my new NAS with no issues. I was able to also test the rollback function and in the limited testing I've done, it works. I plan on trying to test more of the rollback function, but in the mean time if anyone who can would like to review the code I'd appreciate it. Given the consequences of bugs in this code, the more eyes looking at it, the better.

If you choose to help, simply focus on TvArchiver.java. Feel free to post review comments in this thread.

[Slugger]

SJQ v3 beta 639 now available for download...

This build includes the new _MOVEREC task, which allows the scripting of automatic recording moves to other directories/disks while SageTV is running.

This task should be treated as highly experimental and it could possibly cause major damage (i.e. delete/lose recordings). I take no responsibility for what happens to your recordings, wiz.bin, SageTV installation, etc. as a result of using this task in your rulesets. But with that said, I've been running it quite successfully for over a week and have successfully transferred some 150GB worth of recordings over to my new NAS without issue.

I'm now working on the SJQ v3 user guide updates and am not quite ready to open myself up to full support for v3 just yet, however, for those willing to test, here's a sample ruleset that will move recordings via the new task.

# Move all recordings of 'Seinfeld' to the specified target dir only
# if they aren't currently recording and they aren't already archived
# (this task automatically marks the recording as archived after a
# successful move)
#
# The method calls against the $ object in these rulesets come from the
# SageTV API.  When dealing with a media file object, you can call any 
# method from the MediaFileAPI, ShowAPI, AiringAPI, or ChannelAPI against
# the object; more details will be available in the new user guide, when
# ready.  NOTE: It's curly braces after a method name, not parenthesis.

if [IsObjMediaFile == true && $.IsFileActivelyRecording{} == false && $.IsTVFile{} == true && $.IsLibraryFile{} == false && $.GetShowTitle{} == "Seinfeld"] {
   # You must specify the target directory via :MOVERECOPTS option

   # Here's a local drive specification
   :MOVERECOPTS "F:/archive"

   # Here's a UNC path specfication (notice the double up on back slashes)
   # :MOVERECOPTS "\\\\mediasrv\\tv"
   _MOVEREC
}

[nyplayer]

Slugger,

Seems to be working real well... I like that you move the edls properties etc... great job

PS it run a lot faster than the way I was doing it I was using the transcoder.

I also tested by moving a file to a directory that is not in Sagetv and it added the file which I like to do.

[Slugger]

Not sure how the transcoder works, but even with no transcode it probably still runs it through some process unnecessarily. I just do a straight file read of the source and write it out byte for byte to the destination folder so I would expect it to be quite faster than tricking the transcoder into doing it.

[gveres]

After I setup v2 and watched how it behaved, I realized that I have an interesting problem that I don't know how to solve.

I have the following tasks:
QSF - this is a quick stream fix from videoredo.
COMSKIP
ADULTCOMPRESS
PODCASTCOMPRESS
BNNPODCASTCOMPRESS
KIDSCOMPRESS

In the client config I limit the number of processes to 3 and in each task I limit the number of processes to 2 or 1. 2 for QSF and comskip and 1 for the compression tasks.

The problem is that each of the compression tasks uses handbrake with various options to compress the video. Handbrake likes to use all the available cpus, which is great. But I would prefer to have a single handbrake instance running. So what I really want to do is limit all 4 compression tasks to a single process. Is that possible?


Hmm, now that I am writing this I am coming up with an idea. I guess if I run 2 instances of the client on the same machine (different ports obviously), then I could have one client for the compression tasks and one client for the other tasks. I could then tell the compression client to limit itself to 1 process.

Is that how you intended this type of requirement to be satisfied, slugger?

Thanks
Greg

[razrsharpe]

Originally Posted by gveres

In the client config I limit the number of processes to 3 and in each task I limit the number of processes to 2 or 1. 2 for QSF and comskip and 1 for the compression tasks.

The problem is that each of the compression tasks uses handbrake with various options to compress the video. Handbrake likes to use all the available cpus, which is great. But I would prefer to have a single handbrake instance running. So what I really want to do is limit all 4 compression tasks to a single process. Is that possible?

If i understand you correctly, what you really want is some notion of resource allocation that is client based (not task based) that is similar to dirmon2. I asked about this topic in this post a couple weeks ago. Here's my snippet:

Originally Posted by razrsharpe

Does SJQ have the notion of resource allocation... aka you can run multiple threads of comskip but only one video conversion thread? like in dirmon2 each "job" consumes x amount of resource out of a total y. Dirmon2 will then allocate jobs until the sum of all x is the maximum that does not exceed y.

the key being that resources are at the "client" computer level (not the task level) and each "task" consumes x percentage of the total resources available to that client. Then the task scheduler would assign tasks to that client until there is no more resources left.

My understanding is that this can't be done currently in SJQ (and is really the only thing preventing me at the moment to ditch dirmon2 and switch on over). If you read the conversation that followed my original post Slugger expands on the issue but that's my take away.

[gveres]

Originally Posted by razrsharpe

If i understand you correctly, what you really want is some notion of resource allocation that is client based (not task based) that is similar to dirmon2. I asked about this topic in this post a couple weeks ago. Here's my snippet:



the key being that resources are at the "client" computer level (not the task level) and each "task" consumes x percentage of the total resources available to that client. Then the task scheduler would assign tasks to that client until there is no more resources left.

My understanding is that this can't be done currently in SJQ (and is really the only thing preventing me at the moment to ditch dirmon2 and switch on over). If you read the conversation that followed my original post Slugger expands on the issue but that's my take away.

SJQ can have 2 threads of comskip and 1 thread of compression. But right now that thread of compression is per compression job type. So in my case above, I have 4 compression job types. What I am interested in is being able to treat the 4 compression job types as a single job type for the purpose of job scheduling.

Right now, I have SJQ successfully spawning 2 comskip jobs and 1 job for each of the compression tasks. I have the total running jobs limited to 3. The problem with this today is that 3 of those jobs can be compression jobs of the different type.

But as long as I can run multiple clients on the same machine, I am pretty sure what I wrote above will work. I am going to give it a try tonight to see if I can get it working. I will report back.

Greg

[razrsharpe]

Originally Posted by gveres

What I am interested in is being able to treat the 4 compression job types as a single job type for the purpose of job scheduling.

Right, I get that. Using a resource allocation scheme similar to dirmon2 would allow you to do that no problem. For example:

Total resources available for that client = 100

Task resources required:
Comskip = 30
Compress1 = 100
Compress2 = 100
Compress3 = 100

The scheduler would execute tasks until the total resources consumed is the maximum that does not exceed the total resources available. So for example, you could have one of the compression jobs running or 3 comskip jobs. No other combination of jobs would total less than 100 in this scenario.

But as long as I can run multiple clients on the same machine, I am pretty sure what I wrote above will work. I am going to give it a try tonight to see if I can get it working. I will report back

This was my first thought on how to solve this problem too. Unfortunately, my understanding is this cant be done per this post

[nyplayer]

Slugger,

I have a question about persons is show.

is the change below correct

Person == "Bela Lugosi" now becomes $.GetPeopleInShow{} == "Bela Lugosi"

Edit

I got it to work with $.GetPeopleInShow{} =* "Bela Lugosi" using contains

[Slugger]

Originally Posted by nyplayer

Slugger,

I have a question about persons is show.

is the change below correct

Person == "Bela Lugosi" now becomes $.GetPeopleInShow{} == "Bela Lugosi"

Edit

I got it to work with $.GetPeopleInShow{} =* "Bela Lugosi" using contains

Just saw this post now, and oddly enough I just finished the blurb in the user guide for it. You did figure it out correctly. $.GetPeopleInShow{} returns a comma separated list of all people in credits, so you'll have to use =* (contains) or =% (matches regex) to check that info now.

Alternatively, you could also use $.GetPeopleInShowInRole{Actor} =* "Whomever" to check for actor credits. Replace Actor with whatever role you want to check against.

[Slugger]

Originally Posted by razrsharpe

Right, I get that. Using a resource allocation scheme similar to dirmon2 would allow you to do that no problem. For example:

Total resources available for that client = 100

Task resources required:
Comskip = 30
Compress1 = 100
Compress2 = 100
Compress3 = 100

The scheduler would execute tasks until the total resources consumed is the maximum that does not exceed the total resources available. So for example, you could have one of the compression jobs running or 3 comskip jobs. No other combination of jobs would total less than 100 in this scenario.


This was my first thought on how to solve this problem too. Unfortunately, my understanding is this cant be done per this post

It's not possible, two separate clients running on the same machine are simply seen as one client polling more frequently than it should by the server. It would only work if you could force each client instance to connect to the server via a separate IP address (that's actually how SJQ identifies task clients, by IP).

[karljayne]

I've recently done a clean install (Win7++) and followed the guides for installing SJQ and encountered this error when running the queue loader

2009-11-05 13:38:52,763 INFO [SystemMessageQueueLoader]: Processing a system message...
2009-11-05 13:38:52,763 INFO [SystemMessageQueueLoader]: SysMsg needs to be processed...
2009-11-05 13:38:52,763 ERROR [SystemMessageQueueLoader]: Unexpected exception
java.lang.RuntimeException: Invalid field name, 'IsTV', on line 2
at com.google.code.sagetvaddons.sjq.server.RulesParser$Evaluator.internalTest(RulesParser.java:206)
at com.google.code.sagetvaddons.sjq.server.RulesParser$Evaluator.runTest(RulesParser.java:131)
at com.google.code.sagetvaddons.sjq.server.RulesParser$Evaluator.evalTest(RulesParser.java:118)
at com.google.code.sagetvaddons.sjq.server.RulesParser$Evaluator.(RulesParser.java:70)
at com.google.code.sagetvaddons.sjq.server.RulesParser$Evaluator.

I'm running SJQ 3b on Jetty 1.6 and tried upgrading JAVA to V6U17.

Any idea what's going on and why the queue fails?

Thank you in advance.

[nyplayer]

Looks like you are using Old rules with the new SJQ... you need to post your rule. The way rules are written have changed.

[Slugger]

With my announcement of SJQ v3 support earlier today, SJQ v2 is now officially out of support. I will continue to monitor this thread for a little while longer (until v3 moves out of beta), but support will be limited to replies to messages. Bugs identified in v2 will only be fixed in v3 and no further code changes will be made to the v2 code base.

All messages concerning v3 should be directed to the new v3 thread.

v3 contains many new features and bug fixes and I encourage all users to move to v3 in the very near future. The more people who do make the move and test out the beta builds, the quicker bugs are found and the quicker the beta tag is removed. I plan on moving v3 out of beta early in 2010.

[Slugger]

Originally Posted by karljayne

I've recently done a clean install (Win7++) and followed the guides for installing SJQ and encountered this error when running the queue loader



I'm running SJQ 3b on Jetty 1.6 and tried upgrading JAVA to V6U17.

Any idea what's going on and why the queue fails?

Thank you in advance.

As nyplayer pointed out, your ruleset is not valid for SJQ v3. The IsTV test was replaced by the new $.IsTVFile{} test, but only if the object being scanned is a media file. You may want to read this section of the new user guide. Read the example ruleset above that paragraph then that paragraph, then the two paragraphs following the table of test names. Hopefully, that helps you out. If not, please move over to the new SJQ v3 thread and we can continue there.

[karljayne]

Solved, thanks for the reply and the link to the new thread and user guide!

Great work BTW.

[personalt]

This is somewhat of a loose SJQ question.

When i have SJQ run comskip I use :CPU LOW and runs as a low priority task as show in the taskmgr. However if I have SJQ run ffmpeg it doesnt seem to respect the priority and runs and 'normal' prioirty in task mangager.

is there a way to set this up to run at low priority?