Plugin: Sage Job Queue (SJQ) v3

[evilpenguin]

Hey Slugger,

Is there a way for me to limit the hours that a client will take a certain job off the queue? I've got a backlog of encoding tasks (200+) and I only want to run them overnight. The only time limiting I can find is on the queuing side which doesn't work because all of my jobs will get added the first time it goes to load the queue and then the jobs will be run 24/7 after that.

-EP

[Peter_h]

Wouldn't using the DayofWeek/TimeofDay on the server ruleset and the :resources on the client config file achieve this?

You could also use the :maxrun or :maxrunratio so that anything that runs too long will be killed and marked as failed so that it does not keep going in the morning.

[Slugger]

Originally Posted by evilpenguin

Hey Slugger,

Is there a way for me to limit the hours that a client will take a certain job off the queue? I've got a backlog of encoding tasks (200+) and I only want to run them overnight. The only time limiting I can find is on the queuing side which doesn't work because all of my jobs will get added the first time it goes to load the queue and then the jobs will be run 24/7 after that.

-EP

Though missing from the docs (I'm adding them in now), the OFFDAY and OFFHOUR settings can also be used in client configs to disable a client. However, the options are only valid at the global level for a client and are used to disable the client from accepting ANY work. A client can not be disabled for specific tasks. A feature ticket for such a request could be accepted if submitted.

:OFFHOUR "18-23" # Disable this client from accepting work between 6pm and 11:59pm daily
:OFFDAY "6" # In addition, this client does not work on Fridays (who really wants to anyway?) :)
COMSKIP {
   "C:/comskip/comskip.exe \"%c%\""
}

[Peter_h]

Nice!

Going to add this when I get some free time on Turkey day.

[evilpenguin]

Excellent, issue submitted

Another question regarding your move from :max to :resources: I have my server rule set setup to create a separate task for comskip jobs on specific drives

if [IsObjMediaFile == true && $.IsTVFile{} == true && $.IsNotManualOrFavorite{} == false && FileExists != "%d%/%p%.edl" && Filename =$ ".mpg" && Filename =* "Data1"]
{
   :PRIORITY 10
   COMSKIPDATA3
}

if [IsObjMediaFile == true && $.IsTVFile{} == true && $.IsNotManualOrFavorite{} == false && FileExists != "%d%/%p%.edl" && Filename =$ ".mpg" && Filename =* "Data2"]
{
   :PRIORITY 10
   COMSKIPDATA2
}

if [IsObjMediaFile == true && $.IsTVFile{} == true && $.IsNotManualOrFavorite{} == false && FileExists != "%d%/%p%.edl" && Filename =$ ".mpg" && Filename =* "Data3"]
{
   :PRIORITY 10
   COMSKIPDATA1
}

And then my client is setup like this so that I can have have...

• Up to 4 COMSKIP's tasks running at a time.
• Never more than 2 COMSKIP's tasks on the same drive.

:MAXPROCS 4

COMSKIPDATA1 {
   :MAX 2
   :RETURNCODE 1
   "C:/Program Files/SageTV/SageTV/Comskip_orig/comskip.exe \"--ini=C:\\Program Files\\SageTV\\SageTV\\comskip_orig\\comskip.ini\" \"%c%\""
}

COMSKIPDATA2 {
   :MAX 2
   :RETURNCODE 1
   "C:/Program Files/SageTV/SageTV/Comskip_orig/comskip.exe \"--ini=C:\\Program Files\\SageTV\\SageTV\\comskip_orig\\comskip.ini\" \"%c%\""
}

COMSKIPDATA3 {
   :MAX 2
   :RETURNCODE 1
   "C:/Program Files/SageTV/SageTV/Comskip_orig/comskip.exe \"--ini=C:\\Program Files\\SageTV\\SageTV\\comskip_orig\\comskip.ini\" \"%c%\""
}

However, and correct me if i'm wrong, but I don't think I can successfully do this anymore in your :RESOURCES implementation.

[Slugger]

It seems the answer is no. I made the switch to the global client resources because it satisfied a use case I really wanted (and some others had asked for it as well). It also seemed to handle most other use cases I could think of under the old model, but obviously I didn't think of them all.

I could reintroduce the :MAX setting which would allow you to restrict the number of instances of a task that can run at once on a client, regardless of the resource setting. So setting each task like this:

:MAX 2
:RESOURCES 25

Should then accomplish what you want (up to 4 simultaneously, no more than 2 of any one task type)? Making the :MAX setting optional shouldn't break anything else, I don't think. Anyone else see a problem with this solution?

[evilpenguin]

Yep, that's exactly what I was looking for, the combination of MAX and RESOURCES should allow for some obsessively fine tuned configuration

Want me to submit another ticket?

[Slugger]

Nah, got it all... building it right now... new snapshot should be ready before I go make dinner.

[Slugger]

Snapshot 661 now available. Has everything talked about above:

:MAX setting returned for task def'ns in the client config; limit max number of a given task that can run simultaneously on a client regardless of resource settings; use -1 for option to be ignored (default val); use 0 to prevent a client from running a specific task; use any value > 0 to cap a task

:OFFDAY and :OFFHOUR settings now honoured at the task level for a client config; use 'em just as documented for other places.

I'll update the docs soon enough but all of these features exist in snapshot 661.

Very light testing on my part, but it looks good:

2009-11-23 18:49:57,305 WARN  [SJQServlet]: Client '127.0.0.1' cannot pop a task of type 'TVMETADATA': Already running :MAX instances [MAX=0]
2009-11-23 18:49:57,305 WARN  [SJQServlet]: Client '127.0.0.1' cannot pop a task of type 'MOVMETADATA': Already running :MAX instances [MAX=0]
2009-11-23 18:49:57,305 WARN  [SJQServlet]: Client '127.0.0.1' cannot pop a task of type 'SDCOMSKIP': Task disabled by :OFFHOUR setting!
2009-11-23 18:49:57,305 WARN  [SJQServlet]: Client '127.0.0.1' cannot pop a task of type 'HDCOMSKIP': Task disabled by :OFFHOUR setting!

Load up 661, test it out, let me know how it goes.

EDIT: Build is still uploading to Google Code... should be available by 19:00 EST

[evilpenguin]

Excellent, i'll get it loaded up and tested tonight

Thanks for all your hard work on SJQ! Upgrading to this from Dirmon2 has been on my todo list for *ages*, but I never quite got around to it until my server rebuild last week. Now that i'm fully up and running w/SJQ v3.0 I can't even remember how I lived without this much configurability

Edit: Looks like the restrictions are working good on my end as well:

... Client '127.0.0.1' cannot pop a task of type 'COMSKIPDATA3': Already running :MAX instances [MAX=2]
... Client '127.0.0.1' cannot pop a task of type 'MEDIASHRINKHQ': Task disabled by :OFFHOUR setting!

[Slugger]

Originally Posted by jakfech

The only major problem I am currently having is with the server, and maybe this has already been addressed. I've found that SJQ will hang if a recording-based task is queued and I delete that recording before SJQ starts the task on a client. SJQ stays at "Started" forever. I am able to delete the task successfully however.

Was able to reproduce this bug. Issue 96 created. Those interested in receiving a notification when this is fixed can star the issue. It will be fixed before beta tag removed.

[nyplayer]

Slugger,

Stuck in internal tasks again.

Tue Nov 24 07:36:14 PST 2009: SJQ Application Dump MediaQueueLoader thread is alive... java.lang.Thread.sleep(Native Method) com.google.code.sagetvaddons.sjq.server.MediaFileQueueLoader.run(MediaFileQueueLoader.java:101) java.lang.Thread.run(Unknown Source) SysMsgQueueLoader thread is alive... java.lang.Thread.sleep(Native Method) com.google.code.sagetvaddons.sjq.server.SystemMessageQueueLoader.run(SystemMessageQueueLoader.java:77) java.lang.Thread.run(Unknown Source) ClientMonitor thread is alive... java.lang.Thread.sleep(Native Method) com.google.code.sagetvaddons.sjq.server.ClientMonitor.run(ClientMonitor.java:35) java.lang.Thread.run(Unknown Source) InternalTaskClient thread is dead! Most recent LogCleaner thread is dead (this is normal behaviour)! Most recent FileCleaner thread is dead (this is normal behaviour)!

[Slugger]

Are there tasks stuck in STARTED state as well? Might you have hit the above mentioned issue 96 (for which I'm about to commit the fix)?

[nyplayer]

There were no tasks in starting just waiting. All the regular tasks kept running and completed.

[Slugger]

Hard to read that thread dump - the internal task client definitely crashed. Do you know what kind of internal task last ran? Are there any exceptions logged in sjq.log or in the SageTV/jetty logs? Unfortunately, the thread dump only tells me the internal task client died, I need the exception dumps as well to try and figure out why.

If it was a move task and the recording was deleted before the move was attempted then it falls into issue 96 and the fix I'm testing now should fix it. If that's not it, then I need exception dumps to see what's going on.

[nyplayer]

All the tasks that were waiting were _MOVEREC which I schedule to run during the night.... I will keep an eye on it.

[nyplayer]

Slugger,

2009-11-24 04:21:33,765 INFO [SystemMessageQueueLoader]: Starting run of SystemMessage queue loader...
2009-11-24 04:22:42,968 INFO [SystemMessageQueueLoader]: Done run of SystemMessage queue loader...
2009-11-24 04:22:43,187 INFO [LogCleaner]: LogCleaner instance started
2009-11-24 04:22:43,187 INFO [FileCleaner]: FileCleaner instance started
2009-11-24 04:22:43,187 ERROR [FileCleaner]: File cleaner settings cannot be empty!
2009-11-24 04:22:47,921 ERROR [DataStore]: SQL error
java.sql.SQLException: database locked
at org.sqlite.DB.execute(DB.java:270)
at org.sqlite.DB.executeUpdate(DB.java:281)
at org.sqlite.PrepStmt.executeUpdate(PrepStmt.java:77)
at com.google.code.sagetvaddons.sjq.server.DataStore.markAsProcessed(DataStore.java:790)
at com.google.code.sagetvaddons.sjq.server.TaskQueue.updateTask(TaskQueue.java:197)
at com.google.code.sagetvaddons.sjq.server.TaskQueue.updateTask(TaskQueue.java:266)
at com.google.code.sagetvaddons.sjq.server.InternalTaskClient.run(InternalTaskClient.java:58)
at java.lang.Thread.run(Unknown Source)
2009-11-24 04:22:48,453 INFO [LogCleaner]: LogCleaner instance finished [5266ms]
2009-11-24 04:23:02,968 WARN [SJQServlet]: All client connections are being accepted and processed, please consider setting up client restrictions in SJQ settings

[Slugger]

Blah!! Ok... looks like the _MOVEREC task was trying to update the database at the same time the LogCleaner was probably purging old logs. The LogCleaner is not efficient and is definitely slow, especially if there is lots of output stored from your task runs. And since the LogClearner took over 5 seconds to complete, then that's why the internal task client couldn't update the db and died off.

To confirm, I need to see the timestamp from the settings table in your sjq db. Do you have sqlite3 installed on your system? If so, can you run this query and paste back the results?

SELECT * FROM settings WHERE var LIKE 'Last%';

Should get something like this in return:

sqlite> SELECT * FROM settings WHERE var LIKE 'Last%';
LastServerClean|1253500213395
LastVacuum|1258995611931
LastClientClean|1258995611931
LastRun|1259078586278
LastTaskLogClean|1259078588169

[Slugger]

Snapshot 662 fixes issues 96 and 97.

Issue 96: Fix bug where server would return 500 HTTP error if the task it was trying to assign to a client was for a recording that had been deleted after the task was inserted into the queue, but before the task was assigned to a client

Issue 97: Server now monitors tasks that are in STARTED state. If a task remains in started state for more than 60 seconds then the task is automatically marked as failed and removed from the queue (a proper task client is expected to return an update and tell the server that the task is indeed running on the assigned client - this update is to happen within no more than 1-2 seconds after a client receives a task from the server).

[Slugger]

Originally Posted by nyplayer

Slugger,

2009-11-24 04:21:33,765 INFO [SystemMessageQueueLoader]: Starting run of SystemMessage queue loader...
2009-11-24 04:22:42,968 INFO [SystemMessageQueueLoader]: Done run of SystemMessage queue loader...
2009-11-24 04:22:43,187 INFO [LogCleaner]: LogCleaner instance started
2009-11-24 04:22:43,187 INFO [FileCleaner]: FileCleaner instance started
2009-11-24 04:22:43,187 ERROR [FileCleaner]: File cleaner settings cannot be empty!
2009-11-24 04:22:47,921 ERROR [DataStore]: SQL error
java.sql.SQLException: database locked
at org.sqlite.DB.execute(DB.java:270)
at org.sqlite.DB.executeUpdate(DB.java:281)
at org.sqlite.PrepStmt.executeUpdate(PrepStmt.java:77)
at com.google.code.sagetvaddons.sjq.server.DataStore.markAsProcessed(DataStore.java:790)
at com.google.code.sagetvaddons.sjq.server.TaskQueue.updateTask(TaskQueue.java:197)
at com.google.code.sagetvaddons.sjq.server.TaskQueue.updateTask(TaskQueue.java:266)
at com.google.code.sagetvaddons.sjq.server.InternalTaskClient.run(InternalTaskClient.java:58)
at java.lang.Thread.run(Unknown Source)
2009-11-24 04:22:48,453 INFO [LogCleaner]: LogCleaner instance finished [5266ms]
2009-11-24 04:23:02,968 WARN [SJQServlet]: All client connections are being accepted and processed, please consider setting up client restrictions in SJQ settings

Issue 98 opened. I'd really still like to see those timestamps from your db to confirm my hypothesis, but based on the evidence I currently have, issue 98 is the underlying cause. This fix is not going to be a quick one, unfortunately.