Plugin: Sage Job Queue (SJQ)

[Slugger]

Originally Posted by rsagetv99

Hi Slugger, just wanted to say congratulations on another great plugin!

I have a bug to report, and a couple feature requests as well.

Running Version 2.0, no previous versions ever installed.

Bug: If you leave "When to run:" set to "Constant interval" you get the error "Errors were encountered during the update
*Delay must be an integer greater than zero!"
Workaround: Pick one of the other 2 options and set the Delay to something other than zero and update. After that you can switch back to "Constant interval" without the error.

Feature requests:
1. Add configurable job priorities.
2. Add option to create empty .edl files for skipped commflag jobs - this will allow someone to compress all .mpgd but only after the show has been commflagged even when the show was not actually commflagged.
3. Add configurable option to run multiple copies of a process.
4. Add configurable option to check CPU before running and/or to only run if CPU is X% or less.
5. Add option to only run if target drive is inactive.

Currently on vacation and will only be checking in periodically (some people would prefer I left my laptop at home, but that's a whole other story). It'd be greatly appreciated if you could submit all of these items to the (new) project issue tracker here:

http://sagetv-addons.googlecode.com

If you can, please add one issue per item from above and apply the Plugin-SJQ tag to each of them (not a big deal if you forget the tag). Plan on starting on the next SJQ release shortly after I return home. As a preview I can say that the bug will obviously get its due attention and of the feature requests, #3 is definitely on my todo list (along with some type of client/server architecture to allow multiple jobs to be processed by multiple clients). The other four are interesting and will be considered. But do please try to add them to the tracker - bug reports/feature requests easily get lost in these forums.

[Surtr]

Man, this looks excellent. I've got another request; if you're cool with it I'll add it to your issue tracker:

- Can you add ability to set/unset watched flag and delete a recording in sage?

My motivation is this: On DirecTV, the episode data for NFL Total Access is jacked. for some reason, almost all episodes show up as the exact same episode. So, I'll get one recording and it won't record again until I mark the existing recording as not watched and delete the recording. I want to set up a job that will do this for me every day just before the new episode airs.

Thanks!

-Justin

[Slugger]

I've completed work on the initial version of the new ruleset syntax for SJQ v2.0.0 and am looking for feedback from any users out there. To assist in keeping the discussion easy to following, I've started this particular discussion here:

http://groups.google.ca/group/sagetv...50238784cd42a1

Please try to keep the discussion with in that thread on Google Groups and leave this thread for general/support questions.

Thanks.

[perfessor101]

Hello,

I was curious if you could add an expiration timer for certain jobs ... (or time to live or ...)
I'm currently having fun with SageTVTranscoder and VideoRedo. Depending on which grabs the Overlay first VideoRedo will hang mid QuickStreamFix and the job won't complete. It doesn't happen all the time, so having the job terminate after 0.5 times the show length ... (or x times) ... would allow other jobs to complete. Later when I go through my processing logs I can drag and drop the files that didn't complete successfully to manually kick off the process.

Another thing that might be useful ... would there be anyway of accessing the SJQ Current Status through an RSS feed so we can check in SageTV which file is currently being processed? (ie ... I can choose to watch, etc a show or not based on whether it's currently being 'used' by SJQ)

or I guess I could start compressing everything and use SJQ to kickoff the transcodes ... but that would lead to the question :
In the past when I have compressed a show with replace original and then watched that compressed show ... the 'original' recording will not get marked watched and will re-record at a later date. (If I convert the recording without replacing the original and then watch the converted video ... both will be marked watched at the end ... but then I have an mpeg and xvid copy which doesn't save me any space)


Thanks for your time,
Bobby

[Slugger]

Originally Posted by perfessor101

Hello,

I was curious if you could add an expiration timer for certain jobs ... (or time to live or ...)

Very interesting/useful idea. I like the idea of a process timeout option. Please add an issue to the project tracker (things get lost on these threads rather quickly):

http://sagetv-addons.googlecode.com

Another thing that might be useful ... would there be anyway of accessing the SJQ Current Status through an RSS feed so we can check in SageTV which file is currently being processed? (ie ... I can choose to watch, etc a show or not based on whether it's currently being 'used' by SJQ)

Again, add another issue ticket. I'll say this one doesn't grab my attention as much as the process timeout idea, but putting it in the tracker means I won't forget about it and will probably consider it at some point down the road. With that said, the servlet does give a real time status as to what SJQ is up to (I realize an RSS feed means you can access it easier, but just in case you weren't aware).

In the past when I have compressed a show with replace original and then watched that compressed show ... the 'original' recording will not get marked watched and will re-record at a later date. (If I convert the recording without replacing the original and then watch the converted video ... both will be marked watched at the end ... but then I have an mpeg and xvid copy which doesn't save me any space)

I suspect this has something to do with the SageTV core. I believe transcoding via the core and telling SageTV to remove the source does something to cause this, but I don't remember.

While I'm here, a quick update on the progress of v2.0.0. First, I just haven't had the kind of time I was hoping to work on this so it's been a slow pace, but I am making progress. I managed to finish the initial version of the server ruleset parser and the priority queue generation on Sunday morning. So SJQ is able to process the new, more powerful syntax file (see above for those details) and it's properly adding tasks to the priority queue (priority queue is a simple weighted queue where you set the weight of jobs in the ruleset file). Now I'm about ready to start on the client code where clients will come remove jobs from the queue and do them.

As I say, I've been so busy the last little while with other things that it seems like I'm only working on this a couple days a month (usually Sunday mornings before NFL starts). So I have no time line for when an initial version might be ready for testing, but when it is, I'll definitely be looking for testers. I can tell you that v2.x will run under Jetty so if you wanted to get a head start you could install the Jetty plugin and make sure it's operational.

[Slugger]

Thought I'd post a quick update and an initial screen shot from SJQ 2.0.0. The attached screen shot shows the view of the active job queue.

As you can see, SJQ 2.0.0 supports multiple clients and clients are able to run multiple jobs in parallel and jobs can consist of more than one command line. Clients can also restrict parallelism by job type as well (i.e. a client can be configured to run up to 3 jobs in parallel, but no more than 1 transcode job at a time, for example).

Everything is working nicely as far as client/server communication, job assignments from the server to the client and the client reporting back updates as it completes jobs. Right now I'm working on having the clients actually run the assigned command lines instead of just accepting the jobs and immediately reporting back to the server.

Depending on how much free time I find over the next week or two, I might have something ready for full testing in the next couple weeks. In the meantime, I'm looking to line up some testers who are willing to try and break things, especially when it comes to the ruleset and client configuration parsers. If this is something you're able to help me out with then please let me know (might be a week or so before I'm ready for testers, but please send me a PM or reply here if I can contact you once I'm ready).

[lovingHDTV]

Could someone show me how to set the File Mangement options to delete *.edl, *.txt *.log files if there are no *.mpg matching files?

thanks
dave

[Slugger]

Originally Posted by lovingHDTV

Could someone show me how to set the File Mangement options to delete *.edl, *.txt *.log files if there are no *.mpg matching files?

thanks
dave

Those inputs use regular expressions, so to match edl, txt or log files use this:

edl|txt|log

Then just use:

mpg

to match mpg files in the other box. Don't include the period, just the extension.

[lovingHDTV]

I tried this and get an error "AJAX call failed" That is why I thought I was doing something wrong.

Any ideas?

thanks
dave

[Slugger]

Originally Posted by lovingHDTV

I tried this and get an error "AJAX call failed" That is why I thought I was doing something wrong.

Any ideas?

thanks
dave

Might be this issue? If it is, hold on for SJQ 2.0.0, I will not be investigating that issue in the 1.x stream (I've stopped all work on v1.x and am focusing on the new 2.x stream). But feel free to use the documented workaround to set the parameters up as needed.

[brandypuff]

the reference at http://code.google.com/p/sagetv-addo...SJQ2TaskClient points to a werb page with only the Official SRE v2.0.2 release
available.

[Slugger]

Originally Posted by brandypuff

the reference at http://code.google.com/p/sagetv-addo...SJQ2TaskClient points to a werb page with only the Official SRE v2.0.2 release
available.

It's actually talking about the unreleased v2.0.0 version of SJQ. If you haven't ever used SJQ 1.x then stay away until 2.0.0 is released.

There are enough differences between the two that you'll want to wait for SJQ 2.0.0.

Some users are currently helping me do some beta testing, but things aren't quite ready for full release.

[wayner]

Slugger - you work differently then when I used to do coding. Writing docs before releasing the code - What a concept! I thought you were just suppsed to code and then worry about the docs later - which meant that they never got written.

[Slugger]

Here's the latest screen shot from SJQ2, hopefully it'll grab your attention enough to join the testing effort. Snapshot builds are now being done and are being made available for download.

If upgrading from v1.x then before you start, stop SageTV and make a backup of your sjq.sqlite database file! Once SJQ2 upgrades the database it can no longer be used by SJQ 1.x!.

Snapshot builds are available here. You must install the server package (sjq) as an application within the Jetty plugin. Details are here. SJQ2 uses a client/server architecture for performing tasks on your media files, therefore you must also install the task client package (sjqc) on each machine you wish to be able to run tasks from. This includes the machine running the server (i.e. the server does not include a task client). Details on installing the task client are here.

The wiki area on the project web site has various docs for writing the ruleset file and for writing your client configuration files. Some early testers have indicated that the docs may not be 100% clear on some issues. Please, please send me messages if the docs aren't clear and tell me what the problems are. Even better, open up issue tickets on the project web site.

Please note that the current snapshots are not feature complete, but things are stable enough that I've replaced my own v1.5 installation with these snapshots and have been happily running them for a few days without issue, which is why I'm opening up the snapshots for more testers. I'm looking for people to break things, criticize the docs, tell me what's missing, suggest new features, etc. General feedback also welcome.

Features not currently implemented in v2.x (but on the todo list):

• Currently only the string and boolean tests are available - see wiki doc for list of all tests currently available; the number, date, and time tests will be added back shortly
• Ability to add failed and/or completed jobs back into the queue for reprocessing will be added
• Ability to empty out (reset) the pending items in the queue will be added
• An option to have failed jobs automatically reenter the queue for reprocessing is being considered, but currently not implemented
• The "edl" cleaner, as I call it, has not been ported over to 2.x, but is definitely going to make a return
• There is no way to cleanup the logs at this time; still deciding how best to do it (ideas?)
• Task client handles errors with server comm fine, but doesn't print out the best of messages
• Task client runs in foreground on Windows; looking for way to allow it to be registered as Windows service
• Please feel free to add to this list!

[lovingHDTV]

I'm having issues with SJQ actually running.

Here is my log output:
Processing media object: 'The Locator'
Checking if media object '9057574' has been processed by action type 'ComSkip'
'\\SAGE-SERVER\SageTV\TheLocator-AMothersLastWish-9005108-0.edl' does not exist; test failed.
Rule processing stopped: Rule 0.4 returned false

The rules says if the file .mpg exists, and .edl does not then run comskip. I just put in what was posted earlier in the thread without the channel restrictions.

Ideas why this wont work?

thanks
dave

[Slugger]

Originally Posted by lovingHDTV

I'm having issues with SJQ actually running.

Here is my log output:
Processing media object: 'The Locator'
Checking if media object '9057574' has been processed by action type 'ComSkip'
'\\SAGE-SERVER\SageTV\TheLocator-AMothersLastWish-9005108-0.edl' does not exist; test failed.
Rule processing stopped: Rule 0.4 returned false

The rules says if the file .mpg exists, and .edl does not then run comskip. I just put in what was posted earlier in the thread without the channel restrictions.

Ideas why this wont work?

thanks
dave

The output suggests you're running v1.5? Might I suggest a v2.0.0 beta?

My guess, off the top of my head, is that the Java File API is not fond of UNC paths. Just a guess. Personally, I have a headless server and HD100s for all my clients so I don't have any UNC paths at all to test this theory.

The other thing is that the file is checked for on the machine running SJQ, which is usually the server. Is there a need to use UNC paths in this case? Of course, if you're running SJQ on a client system then ignore this.

If none of the above solves the issue, then please paste in the ruleset file for inspection. If it is a Java/UNC path thing then you may have to move SJQ to the server (or to the machine that hosts the edl files). Or perhaps mapping the drive for use by SJQ will help.

[lovingHDTV]

Originally Posted by Slugger

The output suggests you're running v1.5? Might I suggest a v2.0.0 beta?

My guess, off the top of my head, is that the Java File API is not fond of UNC paths. Just a guess. Personally, I have a headless server and HD100s for all my clients so I don't have any UNC paths at all to test this theory.

The other thing is that the file is checked for on the machine running SJQ, which is usually the server. Is there a need to use UNC paths in this case? Of course, if you're running SJQ on a client system then ignore this.

If none of the above solves the issue, then please paste in the ruleset file for inspection. If it is a Java/UNC path thing then you may have to move SJQ to the server (or to the machine that hosts the edl files). Or perhaps mapping the drive for use by SJQ will help.

It ran initially, I see it in the jobs file, but I had issues with my web browser messing up the delete files stuff, I followed the work around for that and it never worked sense.

I'll upgrade to 2.0, just finished reading the documentation.

thanks
dave

[lovingHDTV]

OK got 2.0 installed, luckily already had jetty installed.

When I connect to the server it is asking for a login. I tried my jetty login password, but that did not work. I tried my neilm password that did not work.

where can I find the password for SJQ?

thanks
dave

[Slugger]

Originally Posted by lovingHDTV

OK got 2.0 installed, luckily already had jetty installed.

When I connect to the server it is asking for a login. I tried my jetty login password, but that did not work. I tried my neilm password that did not work.

where can I find the password for SJQ?

thanks
dave

The default password is 'sjqadmin' (no quotes). Updating the wiki now...

Thanks for pointing this out.

[lovingHDTV]

OK that worked.

When I try to run the client it fails to run:


C:\Documents and Settings\David\Desktop\sjqc-2.0.0.84>sjqc.bat 192.168.1.134 8088
Exception in thread "main" java.lang.NoClassDefFoundError: and
Caused by: java.lang.ClassNotFoundException: and
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClassInternal(Unknown Source)

Ideas?