Plugin: Sage Job Queue (SJQ) v3

[ohpleaseno]

Originally Posted by Slugger

Once they restart the tasks should start being assigned, no? If not, then you may need to restart the SJQ server. If restarting the server fixes the issue then you've discovered a bug that hasn't been discovered (or at least reported) to me before now. Unfortunately, I'm pretty much committed to SJQv4 development so tracking down such a bug in the server code probably isn't going to happen at this point, unless I can easily reproduce it locally. The good news is that the v4 code base is a complete rewrite from scratch and so any such bug in the server should not carry over to v4.

Nope, they are not being reassigned.

I'll restart the server after The Good Wife finishes recording (or my Good Wife will kill me), but restarting Sage before didn't fix this problem

For the record, three things are happening.

1) The internal task client is not working. In fact, a task that I killed is perpetually stuck in the Active Queue as running, but it is not doing anything and I can't make it go away.

2) One of my clients refuses to pick up anything. I've restarted it, but it won't pick up a mediashrink task in the queue and it is the only client that has that task assigned to it.

3) I have a task for Sageaudioconvert that reports a version 0.1.15 in SJQ, but is reporting 0.1.16 in the command line on the client box itself. Weird problem, but I have no idea where to look for a solution for that one.

[Slugger]

An SJQ restart should fix it all up. 2-3 months ago I'd have worked closely with you to track this issue down. Now I'm going to have you wait for SJQv4. This sounds like a problem that isn't going to be easily fixed and now I'm far enough along in SJQv4 that taking a night or two (or longer) to try and track down this bug just means I lose that time for v4. v4 is bringing a complete overhaul to the client/server communication. Bugs like you describe (where job state updates seems to get lost, etc.) should be a thing of the past in v4. I realize this doesn't help you much now, but I simply ask for your patience. Continue to post questions here and I'll reply with fixes/workarounds where possible.

[ohpleaseno]

No problem. Just wanted to put it up there in case there was something that was a known bug that I wasn't beating my head to solve.

Although I'm a little wary of the fact that I feel like I've finally got the hang of SJQ v3 and I'm going to have to relearn everything, I'm eagerly looking forward to v4.

I'm going to give my issues above a go one more time tonight and if anything pops up I'll post it here. I understand that bug fixes aren't happening and you're not going to dig deep into the problem, I'm just making sure that something isn't obvious that I'm missing.

Thanks for all your help.

[Slugger]

If you feel comfortable with v3 then moving to v4 should be no issue at all - other than the time required to convert the setup, but that time, though tedious, will be a one time only thing. I'm really striving to remove the steepness of the learning curve for SJQv4 as much as possible. I think I've done the best I can while maintaining a high level of flexibility in what you can do with SJQ. Obviously, I won't know how successful I was until I start to make betas available.

[ohpleaseno]

Originally Posted by ohpleaseno

Nope, they are not being reassigned.

I'll restart the server after The Good Wife finishes recording (or my Good Wife will kill me), but restarting Sage before didn't fix this problem

For the record, three things are happening.

1) The internal task client is not working. In fact, a task that I killed is perpetually stuck in the Active Queue as running, but it is not doing anything and I can't make it go away.

2) One of my clients refuses to pick up anything. I've restarted it, but it won't pick up a mediashrink task in the queue and it is the only client that has that task assigned to it.

3) I have a task for Sageaudioconvert that reports a version 0.1.15 in SJQ, but is reporting 0.1.16 in the command line on the client box itself. Weird problem, but I have no idea where to look for a solution for that one.

1) Restart fixed it.
2) Restart fixed it. That and an errant { that I missed.
3) Fixed because I'm an idiot. A complete and total idiot.

Everything works as expected. As usual if I stop and think for a few minutes, it all gets fixed.

[mayamaniac]

I must've installed a long time ago but never really need to use it. It's been running I think because I see all these mailcheck.log files generated in the SageTV directory. How do I uninstall SJQ?

[Slugger]

For v7, just remove the plugin. For v6, go to SageTV/jetty/contexts/ and remove the sjq.context.xml file then go to SageTV/jetty/webapps/ and delete the sjq.war file.

[razrsharpe]

Originally Posted by mayamaniac

I must've installed a long time ago but never really need to use it. It's been running I think because I see all these mailcheck.log files generated in the SageTV directory. How do I uninstall SJQ?

I think sagemc (and earlier versions of bmt) also use the mailcheck.log file

[lenkost]

Hello Slugger,
I am happy to hear that you have decided to work on SJQv4.
I have some feature requests to propose:

1. Autodetect client after SJQ restart

2. Ability to manually remove items from job queue and place them into disabled category so they are not added back on the next run

3. Add Recorded(time) column to Active Queue view next to Added column

4. Ability to disable client as discussed in #1014

Thank you.

[Slugger]

Originally Posted by lenkost

Hello Slugger,
I am happy to hear that you have decided to work on SJQv4.
I have some feature requests to propose:

1. Autodetect client after SJQ restart

In SJQv4, you have to register task clients with the server. The server then pings the task clients periodically in order to keep them online. If a task client is stopped, the ping fails and the server marks the client as offline. Once the task client is restarted, the server will put it back online the next time it pings it. So it's not quite an immediate "auto detect", but it's pretty close.

2. Ability to manually remove items from job queue and place them into disabled category so they are not added back on the next run

SJQv4 does not scan media (or anything) to add jobs to the queue. Jobs are only added via

• An event (such as a recording started event)
• A cronjob (SJQv4 supports a generic crontab for arbitrary scheduling of any type of task)
• Manually (there will be a way to manually add items to the queue)

The MediaFileQueueLoader thread from SJQv3 is no more - SJQv4 uses the Sage 7 event model for queuing up tasks.

3. Add Recorded(time) column to Active Queue view next to Added column

This does not apply to SJQv4 because tasks in v4 are not associated directly with a Sage object (necessary to support the arbitrary crontab), but instead they have metadata attached to them, which is available in the exe environment of the task (more details when I get closer to a beta release, which I'm not close to yet ). However, I'm curious as to why you'd want/need the recorded time displayed?

4. Ability to disable client as discussed in #1014

Clients can be disabled in a very flexible way. If you're familiar with Unix crontab then basically you provide a crontab like string that describes when the client should be enabled. A client only accepts tasks when it is enabled. Tasks configured for a client can also have a schedule attached to them, which allows disabling of specific tasks for clients at specific times; again using the crontab schedule format.

[lenkost]

Thanks for clarification on items 1. and 2.

However, I'm curious as to why you'd want/need the recorded time displayed?

3. Often I end-up with a lot of jobs in Que due to lost connection to client etc.
I want to get rid of the jobs with recordings older then lets say 2 days. But all I see is the time job was added to the Que.
But since SJQv4 will add jobs by recording event only it really does not matter anymore.

4.

Clients can be disabled in a very flexible way. If you're familiar with Unix crontab then basically you provide a crontab like string that describes when the client should be enabled.

Basically I am looking for a simple Web interface to temporarily disable client.

Do you have approximate revease date for v4 beta?

Thanks.

[Slugger]

Originally Posted by lenkost

4.Basically I am looking for a simple Web interface to temporarily disable client.

Initial betas may only include an STVi interface. If that proves to be sufficient then there may not be a web interface for v4. Either way, there will be some way to disable a task client.

A simple click on/click off ability has just been added to my notes (currently not an option).

Do you have approximate revease date for v4 beta?

I just got the v4 engine to launch its first exe on command today... I'm probably a week or two away from actually uninstalling SJQv3 on my systems and moving to v4 only. If I were to guess - and it's a big time guess at this point - I'd say we're at least 3-4 weeks away from the first beta... at least. Since that starts to creep into the holiday season, that timeline might get pushed back even further. I'd like to aim for early December, but I wouldn't be surprised if the first public betas weren't ready until after the new year. But it's so early into the implementation phase so that schedule could shift one way or the other in the blink of an eye.

[Slugger]

SJQv3 end of life will be 31 Oct 2010. What does this mean? I will make no further code changes of any kind to the SJQv3 code base as of Oct 31. I'm making solid progress on SJQv4 and I expect by Oct 31 I will have moved my environment exclusively to SJQv4 and when I do, I will not have any Sage servers easily available to test code changes to SJQv3.

I will, however, continue to support SJQv3 via the forums until the first SJQv4 public beta (no ETA on that yet). Be aware that discovered bugs will not be fixed in v3 code, but I will try to help find a workaround until SJQv4 is available.

This really shouldn't affect much since the code changes in SJQv3 have been minimal over the last few months anyway, but I'm making this statement explicitly to warn users that there will be no more code changes nor new builds of SJQv3.

[lenkost]

Hello,

1. How can I add a rule to limit jobs for recordings not older then 3 days?

2. How do I shrink SJQ.sqlite database? Mine is almost 2 GB and it does not get smaller after deleteng all logs via WEB Interface.

Thank you.

[Slugger]

Originally Posted by lenkost

Hello,

1. How can I add a rule to limit jobs for recordings not older then 3 days?

Add a MediaAge < "3D" test to your ruleset

2. How do I shrink SJQ.sqlite database? Mine is almost 2 GB and it does not get smaller after deleteng all logs via WEB Interface.

Thank you.

It should shrink on its own. I believe it does it once a week, but it might be once a month - don't remember - unless you're running an older build. There were bugs with that code at one point that would have prevented it from ever running the shrink task.

In all honesty, if your db is 2GB then I'd just leave it because if you try to clean up a 2GB db from within SJQ, it's going to deadlock SJQ/crash it, unfortunately. If you aren't hurting for the space then I'd just leave it until SJQv4. Once you move to v4 then you can just delete that db file. If you really need to regain that space then PM and I can explain how to clean up the db outside of SJQ.

[doncote0]

Have you had success with fast forward after transcoding ts files?

Has anyone had success with fast forward after transcoding ts files?

Does the HD300 allow normal FF response (not skip) for H.264 recordings?

[kevine]

Why would I have a stderrout log file almost 1GB for 1 day? I am on WHS and it only gives me 20 GB for a system drive. 1 GB is a lot of space for a log. Any workarounds to this? Thanks.

[Slugger]

Originally Posted by kevine

Why would I have a stderrout log file almost 1GB for 1 day? I am on WHS and it only gives me 20 GB for a system drive. 1 GB is a lot of space for a log. Any workarounds to this? Thanks.

What seems to be written most often to this file? Jetty will dump loads of info to that file if you have DEBUG or TRACE (I believe) as the log level for Jetty. SJQ can write lots of info as well, but I've never seen 1GB - I'm usually 6-10MB per day.

EDIT: SJQv4 does not use Jetty for the server/engine and uses log4j logging only with a configured RollingFileAppender logger that will keep no more than 50MB of log output from SJQ.

[timg11]

I've been reading these threads because I wanted to install SJQ3 to use its ability to move files from my SageTV recordings drive to the "archival" drive without having to stop SageTV.

I wasn't sure which version of Jetty was best for SJQ3, but now it sounds like SJQ4 is so close I should just wait for the beta.

It this the right thread to subscribe to if I want to see any announcement of SJQ4 beta availability?

[Slugger]

Yeah, I will make an announcement here and then create a SJQv4 support thread once a public beta is released - then I'd subscribe to that thread.