Plugin: Sage Job Queue (SJQ) v3

[Slugger]

@Fuzzy Are you seeing the problem with the active queue or just the failed and completed lists? I just took a quick look at the code and I see that I fixed this problem on the active queue some time ago (and just confirmed with some testing). The other two definitely still have this problem (and I'm about to check in a fix for it). I'd like to know if you can reproduce the behaviour on the active queue (try as I may, I could not, which doesn't surprise me given the code I'm looking at that fixed this some time ago).

[Fuzzy]

y'know, I don't rightly remember.. I'm certain I saw it in the failed queues (as that's what I'm normally clearing out). Been a long time since I ever really tried altering the active queue.. Figured it was the same coding in them all.

[Slugger]

Should be fixed for all grids in the new snapshot that's now available (snapshot 794).

[bialio]

Hey Slugger,

I'm looking at adding a rule to move recordings from the recording drives over to a NAS automatically - but it seems like the move recording is only supported as an INTERNAL task? Is that correct? So if there are any other tasks in the Active Queue the internal task won't run - is that correct?

Internal tasks only run on an object if there are no external tasks running or waiting to run. If there are external tasks active or pending then the internal tasks you have queued will remain in waiting state until all the external tasks have completed.

I've got a lot of stuff hanging around in the Active Queue waiting to run that shouldn't interfere with moving recordings - these are items waiting to be transcoded by mediaShrink - they only run at night and there is quite a big list to get through.

I thought maybe if I put a time limit on the RuleSet for the mediashrink jobs - so that they only show up 1:00 AM and 7:00 AM this would fix it, but after they are IN the queue they don't 'disappear' at 7:00, which is probably by design based on our earlier conversation......


btl.

[Slugger]

Hmm... can you confirm that your task client(s) are still polling the server every 5 mins? (Check the clients tab in the GUI.)

External tasks should never be held up by internal tasks. Internal tasks can be held up by other internal tasks. Because the internal task queue is a single thread, serial execution, there is a known bug that if an internal task hits the top of the queue, but can't run because of pending external tasks then all the other internal tasks behind it will also be held up. There is no (easy) fix for the issue and I have no plans to fix it since eventually the internal tasks will get unstuck once all the external tasks catch up.

But if you're seeing external tasks being held up then I'd really like to see a snapshot of your task queue (plus all the details you can provide). Enabling debug logging in sjq.log4j.properties, restarting SJQ, and providing the detailed logs would also be helpful.

[Slugger]

Originally Posted by bialio

I thought maybe if I put a time limit on the RuleSet for the mediashrink jobs - so that they only show up 1:00 AM and 7:00 AM this would fix it, but after they are IN the queue they don't 'disappear' at 7:00, which is probably by design based on our earlier conversation......


btl.

Right, once a task enters the queue it's only removed when assigned to a client. If clients are being restricted by time from picking up those tasks then they sit in the queue for ever.

After rereading your post, am I to understand that _MOVEREC tasks are being held up, but there are no external tasks pending/running for those recordings waiting to be moved? If that's the case then I'd still like you to restart SJQ with the debug logging and let the clients call home for a 20-30 mins and get me the output. If this is what you're seeing then that's a lot more plausible than my original thoughts (but still a bug).

[bialio]

Let me rephrase and maybe we will begin to understand each other

I haven't actually defined a _MOVEREC task yet because I was trying to understand the internal task rules. But since my Active Queue has a lot of pending tasks in 'WAITING' state, which get whittled down by one or two each night, I'm assuming that the internal task _MOVEREC won't get to run based on the snippet I posted above from the User Guide.

There aren't any external tasks being held up, so cross that off the list. I'm just curious how to get the internal task thread to fire since I have a bit of a backlog of external tasks that will take a few weeks (at least) to clear out......

After rereading your post, am I to understand that _MOVEREC tasks are being held up, but there are no external tasks pending/running for those recordings waiting to be moved?

The question has me a bit confused tho - if there are no external tasks waiting/running for a recording, but an internal task waiting for that recording, does the presence of other external tasks for other recordings that are waiting/running preclude the internal task from running?

btl.

[Slugger]

Ok, we're on the same page now.

The docs, as quoted (presumably verbatim), are incomplete. They should read like this:

"An internal task for an object will only run if there are no pending or running external tasks for that same object."

So,

if there are no external tasks waiting/running for a recording, but an internal task waiting for that recording, does the presence of other external tasks for other recordings that are waiting/running preclude the internal task from running?

No, in general. There is that bug where if an internal task for Show A gets to the top of the queue, but can't run because Show A has other pending/running external tasks then it will hold up all other internal tasks behind it (for any and all shows). This is not as designed, but the easy fix will introduce a deadlock scenario into the queue, which is bad and the hard fix is, well, hard (basically a redesign/rewrite of the queue).

Since the internal queue will eventually continue on (when the external jobs blocking it complete), I chose not to fix it. I can see the problem if you restrict tasks to just a few hours a day and have a backlog, but eventually it'll work itself out.

[bialio]

Cool - I understand now I think. I've got some _MOVEREC tasks running now. One thing I'm seeing is that the edl files aren't getting moved with the recordings - as I see comskip tasks kicking off on all of the files once they land in the new spot.....

btl.

[Slugger]

They should be copied over... they definitely do on my system. Ensure the edls have the same prefix (i.e. abc.mpg will only move an edl if it's named exactly abc.edl). Check the task log for the _MOVEREC, are there any errors reported?

[bialio]

The log seems to indicate that they were copied over :

Wed May 12 10:02:23 CDT 2010
Source: D:\SageTV\LegendoftheSeeker-Eternity-9500843-0.mpg
Target: \\mediavault\Volume_1\sage
Renaming source file... DONE!
Copying source file to target directory... DONE!
Deleting source media object in SageTV... DONE!
Renaming target file... DONE!
Creating SageTV media object for target file... DONE!
Relinking target file to SageTV airing... DONE!
Deleting source file... DONE!
Copying artifacts... DONE!
Archive operation completed SUCCESSFULLY!
Wed May 12 10:16:55 CDT 2010

They are definitely named correctly. Is is possible that the FileCleaner is getting them before they have a chance to be moved?

btl.

[bialio]

I think that must be it - I see in the Server logs that the artifacts for the currently active _MOVEREC task were deleted.

btl.

[Slugger]

Originally Posted by bialio

The log seems to indicate that they were copied over :



They are definitely named correctly. Is is possible that the FileCleaner is getting them before they have a chance to be moved?

btl.

That's possible. Are you using SJQ only to cleanup artifacts? How frequently do you run the QueueLoader (the SJQ FileCleaner runs everytime the QueueLoader runs)? If it's only set to run when a rec starts/stops then it's extremely bad luck for the QueueLoader to run at the exact time between the mpg being deleted and the artifacts being copied. Is it possible something else is also cleaning up (dirmon2, etc.)? If it was the FileCleaner that toasted the artifacts then it will be recorded in the SJQ server logs.

[bialio]

I had the queue set to run every 10 minutes. For now I've just cleared out the file cleaner settings and will turn them back on once the initial 'bulk move' has finished.

btl.

[Fuzzy]

Would it be difficult to 'lock' the cleaner while the moverec is in progress?

[Slugger]

Originally Posted by Fuzzy

Would it be difficult to 'lock' the cleaner while the moverec is in progress?

Nope. When I get a chance, I'll add it.

[Fuzzy]

I'm actually thinking the queue builder and cleaner and stuff would be better suited being treated as internal tasks, instead of separate threads. This would prevent ANY sort of conflicts, as they'd only run one at a time.

[Slugger]

Originally Posted by Fuzzy

I'm actually thinking the queue builder and cleaner and stuff would be better suited being treated as internal tasks, instead of separate threads. This would prevent ANY sort of conflicts, as they'd only run one at a time.

True, but then I'd have to go back and fix the bug where the internal tasks can get stuck if one that can't run due to external tasks gets to the top of the queue first. I think I'll just synchronize the _MOVEREC and FileCleaner threads.

[rmac321]

First - thanks for the great tool. I've finally decided to invest the time to learn to use SJQ and am having great success automating a bunch of stuff.

The question - Is there a test for whether a specific recording is being watched? If I understand correctly, the test "IsViewingMedia" is actually a test for whether anything at all is currently being watched. It would be nice to have a quick test for whether a specific file is being watched before attempting a relink or a move. I took a quick look at the Sage API listing, but I don't see anything obvious like "IsBeingWatched"

[rmac321]

FWIW - the SJQ User's Guide on the googlecode site won't load today.