Plugin: Sage Job Queue (SJQ)

[Slugger]

Originally Posted by nyplayer

Slugger,

1.Add the ability to archive..unarchive..set manualrec status. You seem to have everything covered.
2. Ability to tell each client when they can process ... ex time of day they can process.

PS.
Is shutting the client process as easy as shutting down the batch job or does it still stay connected? I shut down the batch job on a client but it still shows connected.

1. Those internal SageTV tasks will be coming back - still figuring out how to process them, but they're definitely coming back

2. Hmm... not so sure about this one. I'm more inclined to control which jobs go into the queue using the DayOfWeek, TimeOfDay (not yet implemented, but coming soon) and/or CurrentDate (not yet implemented, but coming soon) tests. If you really want this feature, please add it to the issue tracker - probably will not debut in the first official release, but a future enhancement possibility.

And finally, yes, killing the batch job will stop the client. The server caches client connections for 5 minutes so it will take up to 5 minutes after you kill the client before the entry is removed on the server.

[nyplayer]

Slugger,
Ignore#2 I can simply shutdown the batch job and restart using task scheduler on the client.

[Slugger]

Snapshot 102 is available. Just managed to finish this before NFL kickoff!

This morning's addition: Context menus added to the active queue, failed, and completed grids. Right click on rows in the grids to get the menus.

With these menus, the following can now be done:

• Remove individual items from the active queue
• Edit priority for items in the queue
• Clear the entire active queue
• Remove a failed job; remove all failed jobs (allows failed jobs to be reran, if desired)
• Remove single completed jobs (remove all disabled here to prevent undesired removal of all completed jobs)

Note: Only queued jobs that aren't running can be edited/removed from the queue. Once a job is running on a client you cannot touch it.

It may be another few days (maybe longer) before I have time to play with the code some more, but in the meantime please keep testing the latest snapshot and keep posting bugs, ideas, feedback, etc.

[nyplayer]

Slugger,

So far so good.... one thing though I cannot seem to get the client to run more than 1 job. Also I noticed there is no TimeOfDay is this for future release?

It only schedules 1.

PASS1XVID {
:MAXPROCS 2
:CPU NORMAL
  "s:\\mysage\\encodeall\\1PassXvidEDLCut.bat \"%c%\""
}

[Slugger]

Originally Posted by nyplayer

Slugger,

So far so good.... one thing though I cannot seem to get the client to run more than 1 job. Also I noticed there is no TimeOfDay is this for future release?

It only schedules 1.

PASS1XVID {
:MAXPROCS 2
:CPU NORMAL
  "s:\\mysage\\encodeall\\1PassXvidEDLCut.bat \"%c%\""
}

:MAXPROCS is a client option, not a task option. Put :MAXPROCS at the top of the file (as in the documented example). :MAXPROCS is ignored inside a task block and so the default value is being used (which is one). Inside the task block you use :MAX to limit the number of runs for that task.

Try this instead:

:MAXPROCS 2
PASS1XVID {
:MAX 2
:CPU NORMAL
  "s:\\mysage\\encodeall\\1PassXvidEDLCut.bat \"%c%\""
}

http://code.google.com/p/sagetv-addo...ntConfigSyntax

Also, the TimeOfDay test will be coming back once I get the date and time tests into the new code base. Might be this weekend, but my alma mater is playing in the (Canadian) national football championship this weekend (go Mustangs!), so I'm heading to Hamilton to sit in the cold, and possibly snow, to watch some football this weekend then all of a sudden December and the holiday season is upon us so development may slow down on this a little bit over the next little while, but I will post updates as I deliver new features to subversion. Getting the date and time tests back in has bubbled its way to the top of the list, so it's probably the next thing I'll work on. The only question is, when that will be. I will probably find a couple hours here and there before the weekend, but not much. Same with next week.

[nyplayer]

Slugger,

Works like a charm... enjoy your football and time away from SJQ. This is one great piece of coding you have done and is much appreciated.

[CollinR]

Originally Posted by nyplayer

This is one great piece of coding you have done and is much appreciated.

Agreed IMO it's very under valued by the community, I assume do to not understanding it's power.

I would put this plugin on par with Dynamic Menus and Webserver interface as far as "must have" customizations go.

[Slugger]

Found an hour this evening to put together snapshot 110. It contains one change: Add the time and date tests back to the core. Specifically, the following tests were added with this delivery:

• TimeOfDay
• CurrentDate
• OrigAirDate

The wiki docs have been updated with these new tests and include the details (expected format, etc.).

If you go to the project issues page, you can see what items are on the todo list. Personally, my next one to tackle (probably sometime next week) is issue 27. However, if you wanted to (possibly) influence what gets worked on then feel free to star the issues that are important to you. No promises as to the order of things, but I'll certainly look at how many people have starred issues. Also, feel free to add new issue tickets for bugs, features, etc. as you play with these snapshots.

[nyplayer]

Slugger,

Some questions.


1. How do you decide in what seq your jobs are run after in que if the priorities are the same ? Oldest first alphabetical or newer first.

2. I noticed that if i delete a show that is in the active que. It still tries to run it then it hangs there... under "starting".

3. I have it setup to compress if there is less than 5% freespace. But what it does is schedule every show... and continues compressing even if it reaches the 5% limit.

[Slugger]

Originally Posted by nyplayer

Slugger,

Some questions.


1. How do you decide in what seq your jobs are run after in que if the priorities are the same ? Oldest first alphabetical or newer first.

When two tasks are waiting and both have the same priority then the order in which they're assigned is undefined. That's my official answer. If you're curious about how it works then the answer goes something like this:

Typically, when a priority tie occurs, it will be broken by the order of the task id def'n blocks in the client asking for work. So if the client connecting defines its task def'n blocks as TASK1 { } TASK2 { } TASK3 { }, etc. then in case of a tie TASK1 would be assigned before TASK2, etc. However, I can't even guarantee that because HashMap.keySet() does not guarantee the order of the returned List of hash keys. Typically, at least with the Sun JVM, the order is usually (always?) the order in which they were inserted. Assuming that to be true then what I've written will hold.

However, I may change the tie breaking formula for one reason or another at any time and my "official" statement holds. In case of a tie, the order of task assignment is unknown. One will be picked over the others, but no guarantee as to which one will be picked first and if the same tie occurs multiple times the order in which they're assigned can differ each time. The only way to ensure an ordering is to have unique priority values.

2. I noticed that if i delete a show that is in the active que. It still tries to run it then it hangs there... under "starting".

Which client version are you using? This would be a symptom of removing a task just as a client connected and the server assigned it, but before it could be updated as "STARTING" on GUI. Later snapshots of the client poll the server every 5 mins as opposed to 10 seconds, which would make this scenario less likely, though not impossible. At some point I will automatically timeout tasks stuck in starting state, but the code's not there yet. Unfortunately, the only way to remove such entries is to restart SJQ (which can be done without restarting SageTV by updating the timestamp on the sjq.context.xml file).

Client snapshots 81 or higher have the 5 min polling (as opposed to 10 seconds), which should make hitting this issue much less likely. As I say, the server will need to monitor and delete entries stuck in started state, and that will come at some point sooner than later.

[Slugger]

Originally Posted by nyplayer

3. I have it setup to compress if there is less than 5% freespace. But what it does is schedule every show... and continues compressing even if it reaches the 5% limit.

I think I understand what you're saying, but let me make sure.

SJQ runs at, say, 11:00. When it starts scanning at 11:00 you have 4% disk free, so SJQ inserts all your recordings for compression even though, say, compressing 2 or 3 only would bring your free space back up over 5%.

If I'm understanding it right then there are two solutions off the top of my head:

If you're using the internal Sage transcoder (which I'm guessing you're not since SJQ2 doesn't include those tasks yet) then incorporate the TranscodeQueueSize test. Using TranscodeQueueSize <= 1 will mean that the SJQ scan will put no more than 1 recording into the transcode queue.

Since you're probably running an external compression script, you're going to want to have that script check the disk space prior to actually running your compression program. If the space >= 5% then you'll want to return an exit status of non-zero (i.e. force the job to be marked as failed in SJQ). This way you can clear the failed list and next time it falls below 5% those shows will be eligible to reenter the queue.

[jreichen]

Originally Posted by Slugger

However, I can't even guarantee that because HashMap.keySet() does not guarantee the order of the returned List of hash keys.

Try a LinkedHashMap.

[Slugger]

Originally Posted by jreichen

Try a LinkedHashMap.

Good point. Will keep in mind should I want to be able to definitively break priority ties. For now, I'm happy with the "undetermined" sol'n provided by HashMap.

[QueOnda]

Online document down?? http://sagetv.battams.ca/

Also, does this work for SageMC and when moving to a dirctory, does it move the commercial files (srt or txt files)?

[CollinR]

Works with MC as it doesn't have a precence in the STV at all so it'll work with any.

You would need to make a batch file to do the moving as such whatever you script into it will be executed when the SJQ conditions are met.

[Slugger]

Originally Posted by QueOnda

Online document down?? http://sagetv.battams.ca/

Also, does this work for SageMC and when moving to a dirctory, does it move the commercial files (srt or txt files)?

The download page is a little outdated, but I've just updated it. The entire project now resides at http://sagetv-addons.googlecode.com.

If you're just starting out with SJQ, you'll definitely want to start with the v2.0.0 beta. New users should stay away from the 1.x versions (basically no longer supported as the v2.0.0 beta continues to mature).

[Slugger]

Server snapshot 113 and client snapshot 112 are available.

Note: Starting with server snapshot 113, the server expects a protocol negotiation to occur before processing client requests. The first client snapshot to support this is 112. Therefore, if installing server snapshot 113 then you must upgrade all task clients to at least snapshot 112. All older clients will simply die when they try to talk to server 113 or newer.

The server upgrades today are:

Internal SageTV tasks once again added to the core. With this update you can now perform SageTV tasks on your media objects via SJQ. You can archive/unarchive tv recordings, mark/unmark objects as watched, add objects to the SageTV transcode queue, and delete objects. All the details, limitations, examples, etc. are in this wiki document. If there are other internal SageTV tasks you'd think would be beneficial to add to SJQ then open up issue tickets on the project web site.

[nyplayer]

Slugger,

I notice you didn't add "SetManRecStatus" "RemoveManRecStatus" is that because it's not possible?


I added the check for freespace check in my script like you suggested that works fine. It will work even better when I can re-insert the jobs back into the que automatically. :-)

[Slugger]

Originally Posted by nyplayer

Slugger,

I notice you didn't add "SetManRecStatus" "RemoveManRecStatus" is that because it's not possible?


I added the check for freespace check in my script like you suggested that works fine. It will work even better when I can re-insert the jobs back into the que automatically. :-)

Just forgot to add them to the wiki doc. The doc has been updated to list those tasks.

Of a more important note, there is a bug in 113 that prevents the internal task client from stopping when SJQ is stopped/restarted/etc. I lose track of the thread I started and it can't be stopped. Therefore, I highly recommend upgrading the server to 118. If you're running 113 then I suggest stopping SageTV completely, unzipping the 118 zip, then restarting SageTV. This will ensure all stray threads are killed off. Snapshot 118 fixes all of these problems. The manual flag internal tasks are in 113 (and above), I just forgot to document them in the wiki. The delete task is documented as being in 113, but will not start working until 118.

To summarize, everyone should move their server to 118.

[Slugger]

Originally Posted by nyplayer

Slugger,
I added the check for freespace check in my script like you suggested that works fine. It will work even better when I can re-insert the jobs back into the que automatically. :-)

Snapshot 120 adds a new server setting: IgnoreFailedTasks

As I write this I'm not liking the name of the option so much, but I've already built it so it'll remain as is for now. Probably should be named more like DoNotMarkFailedJobs or something. The name of this option might change in a future snapshot, but the functionality will remain. When set to true, this option will prevent the server from marking failed jobs in the database, essentially allowing failed jobs to reenter the task queue. Note, however, that failed jobs will not reenter the queue until the next run of the QueueLoader; this setting does not automatically move failed jobs back into the queue, but rather prevents them from being marked as done/failed, which will allow them to reenter the queue on the next run of the QueueLoader thread.