SJQv4: Design Discussion

[Slugger]

Since I was introduced to the Sage v7 beta, I've been thinking about how to update SJQ to take advantage of the many new features of Sage v7. The biggest feature being the event model. After struggling for some time with too many features and not enough time to implement them all, I've finally come up with something I'm ready to proceed with.

The link below is a very high level overview of my plan for SJQv4. I encourage current users to at least read the "goals" section and comment on it. You'll notice there are a lot of changes in how SJQ will work if I proceed with this plan. For those not wanting to read that document, here's an executive summary of the key goals and changes in SJQv4:

Key Goals:

• Simplify the code base; must keep app as feature rich as possible, but must also keep it maintainable
• Where possible, remove barriers of entry for new users; where not possible, attempt to reduce the complexity of those barriers
• Take advantage of the new features provided by Sage v7, namely the event model

Key Changes:

• No more ruleset config! All jobs are queued via the core event system
• No more client configs! Restrictions/conditions handled solely in user's scripts/exe calls (see the design doc for further details)
• Add generic cron scheduler for scheduling of repeatable tasks
• Add true support for processing of any and all types of Sage objects
• STV integration (maybe, if someone would like to help with it)

I encourage interested users to read at least the "goals" section in the document below, especially the section about the "removal" of client configs. This is a very different way to queue and run jobs vs. SJQv3 and older, but I feel this design aligns itself better with the new Sage v7 event model.

http://links.battams.ca/sjq4design

There is no ETA for betas, but I plan to start coding within the next few weeks. Once I get going, I don't expect it will take more than a month or so to get a functional beta in the plugin repository.

I am looking for an STVi developer to join me, if someone's interested. I would really like to add STV integration for SJQv4, but I am not the person to do it.

Looking forward to user feedback on this...

[PLUCKYHD]

Originally Posted by Slugger

I am looking for an STVi developer to join me, if someone's interested. I would really like to add STV integration for SJQv4, but I am not the person to do it.

I would be willing to help on that part if you want my help. A good UI interface inside sagetv would be valuable to all users especially the basic ones. I don't know what all you are wanting on the UI end (and to be honest haven't used SQJ in many months), but planned on starting to use it once my ploxeetv is released to move my recordings that I keep out of recordings and into imported tv section. hit me up if you want my help if not that is fine as well

Breezed through the goals and I like the direction you are taking especially

allow the use of something much better: real programming languages

[Peter_h]

This looks great.

I have one question regarding the language support. Will you allow the use of multiple language in a config file or will it have to be declared at the top and you stick with it throughout the config?

What I'm getting at is I have a concern that since you are moving to support for multiple languages that when a new user that has no programming experience and asks for help, he gets a batch script to do one tasks from one person and later on when he needs to tweak something else is given some java or php code to do something else. I see a scenario where a user could be become very confused and does a lot of copy/paste and never quite grasps what they are actually doing.

[Slugger]

Originally Posted by Peter_h

This looks great.

I have one question regarding the language support. Will you allow the use of multiple language in a config file or will it have to be declared at the top and you stick with it throughout the config?

There won't be any config files in v4. Basically, you're going to map a job id, say "COMSKIP", to an exe on your client. The client will simply run the assigned exe any time it is assigned a task of type "COMSKIP" from the server. What that exe does is totally up to the user. How the exe is implemented is totally up to the user. It might be a Java program, a C++ program, a PHP script, Perl script, etc. The only requirement is that the program must return a special value to denote what the client did with the task:

Return code 0: Client accepted the task and completed it successfully
Return code >0: Client accepted the task and its execution failed
Return code -1: The client rejects the task, push it back on the queue for future assignment

NOTE: Above return values aren't finalized yet, just an example of the idea.

So COMSKIP might map to comskip.exe. Doing so means you don't want to do any constraint checking on this task. If the server assigns this task to this client, it will always just run comskip.exe and return the result.

And maybe your "ARCHIVE" task maps to "java -jar archiver.jar" and archiver.jar is a custom Java program that you wrote (or someone else in the community wrote and made available) and does a bunch of checks against the Sage API (is the file older than 60 days, is it not currently being used, are there 0 clients connected, etc.) before moving the recording to your NAS. If the constraints aren't met then it returns -1 and the task client tells the server we can't do this task right now so put it back on the queue. If the constraints are met then it proceeds with the task (move the recording and rescan the media library).

So users aren't actually ever going to copy/paste raw code. They may copy jars or exes or scripts that do tasks of interest, but should rarely have to actually look at them. The key is that the exe being triggered by the task client is now responsible for deciding if this task should be run immediately instead of rulesets and client configs being used for all of these checks. The power in this decision is that much more expressive, powerful languages can be used to do this scripting. And of more interest to me, I no longer have to maintain the ugliness of the "scripting" language that was used in SJQv3.

What I'm getting at is I have a concern that since you are moving to support for multiple languages that when a new user that has no programming experience and asks for help, he gets a batch script to do one tasks from one person and later on when he needs to tweak something else is given some java or php code to do something else. I see a scenario where a user could be become very confused and does a lot of copy/paste and never quite grasps what they are actually doing.

I see what you're saying, except that it's more of a black box. Users in this situation aren't going to be given java code or php code and have to copy/paste it all over the place. Instead, this user is going to say, "hey, I need SJQ to archive my recordings and I can't program. Does someone have a sol'n for this?" And in response, theoretically, they get someone to say, "yeah, here's a java program that archives recordings. Map your "ARCHIVE" task to the attached jar file and call it like so:

java -jar archiver.jar -noclients=true -minage=60days -destfolder="D:/archived/tv" -src=%SJQ_FILE%

And for another solution they might get a perl script from someone else. And maybe a php script for something else. So the user will have to install perl/php, etc. then map the script. Of course, with scripts they have access to the code, but a generic enough script should not require the user to even look at the code, but instead just configure it with the proper command line flags.

Hopefully this makes sense?

[tmiranda]

Slugger,

I quickly read over the specs and see where you are going. I like the general direction.

I enjoyed our collaboration on SRE and would also volunteer to do some Studio work on this.

Tom

[Slugger]

Originally Posted by PLUCKYHD

I would be willing to help on that part if you want my help. A good UI interface inside sagetv would be valuable to all users especially the basic ones. I don't know what all you are wanting on the UI end (and to be honest haven't used SQJ in many months), but planned on starting to use it once my ploxeetv is released to move my recordings that I keep out of recordings and into imported tv section. hit me up if you want my help if not that is fine as well

Very exciting! Definitely won't say no to STVi coders.

Basically, at the very least I just want users to be able to set their job lists for favourites, etc. Basically, a new option to the favorites editor in the stv. Also, add a job list to manual recordings as well.

I'd love to add a task queue viewer as well. And if you want to get fancy, a task queue editor (clear, stop, kill jobs from the stv as well).

I'll be providing the API, I just need someone to put the UI elements in the STV. I think tmiranda and I did quite well with the same formula for SRE (I provided a high level API and he brought it to life in the STV).

[Slugger]

Originally Posted by tmiranda

Slugger,

I quickly read over the specs and see where you are going. I like the general direction.

I enjoyed our collaboration on SRE and would also volunteer to do some Studio work on this.

Tom

Excellent! I'll definitely be in touch with you (and Plucky) when I have more details. I'll probably throw my ideas out there and you guys can tell me what is and isn't possible via the STVi.

[skiingwiz]

Originally Posted by Slugger

Return code -1: The client rejects the task, push it back on the queue for future assignment

Have you considered a way for clients to have better control over when future execution would occur? In your archive example, if the task won't be valid for 60 days, it would be nice if that task wasn't retried over and over again for those 60 days.

This could either be accomplished with a set of return values indicating the timespan of the next check. Or another option, which could add flexibility for other situations, is for the server to expose an API (maybe even through the event system) for rescheduling a task. So the client would reschedule the task and return success.

I realize your return values were just an example, but they sparked this idea for me.

[Slugger]

Originally Posted by skiingwiz

Have you considered a way for clients to have better control over when future execution would occur? In your archive example, if the task won't be valid for 60 days, it would be nice if that task wasn't retried over and over again for those 60 days.

Very interesting point. Is there a reason you wouldn't want the server to constantly retry that task assignment for 60 days? I don't disagree with you, but I'm trying to look at it from both sides. Let's assume the network traffic and CPU cycles req'd to keep retying for 60 days is irrelevant (as it should be). Any other reason not to have it retry?

Just a counter thought: Perhaps the archiver.jar task is a cron scheduled task that just runs every day looking for new recordings that meet the age criteria and then adds them to the queue and then you don't use the age criteria flag to finally archive them. This way jobs aren't retried for 60 days, but only until all the clients disconnect, etc.?

This could either be accomplished with a set of return values indicating the timespan of the next check. Or another option, which could add flexibility for other situations, is for the server to expose an API (maybe even through the event system) for rescheduling a task. So the client would reschedule the task and return success.

I realize your return values were just an example, but they sparked this idea for me.

I've thought about this, but I then had the situation where if originally you checked for 60 days old and then the client told the server, "this job can't be valid until 60 days from today so talk to me again in 60 days". Now there's two problems:

1) What if you decide you actually want 30 day archiving? That old job is hidden until 60 days. I'd still have to show it in the queue so you'd remember to delete.

2) A new client might be added before the 60 days is up that doesn't have the 60 day constraint attached to it so the server still needs to check in with that server.

I think the best thing to do is retry for 60 days. Alternatively, tasks with this kind of long term delay should probably not queue up when the fav starts recording, but instead use a cron job to poll for recordings that match the criteria and queue them up at that time.

(These ideas are great, keep them coming. This is why I want to discuss things before I dive in and start coding.)

[evilpenguin]

Read over the doc and I like the way you're thinking. I'll have to read it over a few more times to really wrap my head around it, but here are a few quick use cases that I didn't see addressed:

1. If a user installs SJQ v4 on a server that's been running for a while, will there be a way for him to run tasks on media that had already been recorded?
2. What if you have two jobs to run on a media file (comskip, compress) and the order the jobs are run matters (comskip first, compress second)? I guess in this example you could use artifacts (edl), but it'd be nice to be able to handle a use case where there are none.
3. Will a Sage Plugin be able to inject a SJQ job programatically? For example the comskip plugin could also automatically make a COMSKIP job available.

[Slugger]

Originally Posted by evilpenguin

If a user installs SJQ v4 on a server that's been running for a while, will there be a way for him to run tasks on media that had already been recorded?

A cron job would be one way. So after an initial installation of SJQv4 you might create a cron job that runs once and finds all the recordings that need to be comskipped. After that, remove the cron job and let the event handler queue up jobs as favs are recorded.

The other way, which I wanted to avoid, but I don't think I can, will be to provide a manual task loader. Either through the web UI or maybe as a cmd line tool. You provide the job id and the object id (i.e. MediaFile:123456) and it'll get added to the queue. Obviously, I'm going to have to provide some way to inject jobs into the queue outside the event listener mechanism. Making this available via the STV will be very useful as well, I'd think.

What if you have two jobs to run on a media file (comskip, compress) and the order the jobs are run matters (comskip first, compress second)? I guess in this example you could use artifacts (edl), but it'd be nice to be able to handle a use case where there are none.

My vision would be that you'd create an exe that handles the logic for you. Let's call it "comskip_n_compress.exe". This exe calls comskip, ensures it was successful, then compresses your file. If it all completes successfully then return success to the SJQ server, if not, then return a failure code and SJQ will (optionally) requeue the task. As in previous versions, SJQ tasks will not be aware of other tasks. Special logic (such as ordering of individual actions) will be left to the exe called by the task client.

My bigger vision would be that someone (perhaps me) creates a generic "run_in_order.exe" program that accepts a series of exes to run in order, ensuring that nothing runs unless all the others before it succeed. So you might map your "COMSKIP_N_COMPRESS" task id in SJQv4 to this exe:

run_in_order.exe %SJQ_FILE% comskip.exe mediashrink.exe

And run_in_order.exe calls comskip.exe %SJQ_FILE% and if it succeeds it then calls mediashrink.exe %SJQ_FILE% and if it succeeds then it tells SJQ the task ran successfully.

As you can see, one of my main goals in this version is to remove the "brains" from the SJQ code and put them into the user's scripts/exes. Two reasons: 1) Maintenance and 2) Trying to cover everyone's wants and needs is too much for one person. Instead, I'll focus on providing a distributed, multi-threaded job queue and let the community come up with the controlling logic through various scripts and programs. Even if I write a lot of these helper scripts, I'd rather write helper jars for people than try to constantly change/enhance the SJQ scripting language, especially when it's not something I particularly enjoy anyway.

Will a Sage Plugin be able to inject a SJQ job programatically? For example the comskip plugin could also automatically make a COMSKIP job available.

Most likely. There will be some type of API to inject jobs into the queue. Presumably, plugins will be able to use the API as well.

[skiingwiz]

Originally Posted by Slugger

Very interesting point. Is there a reason you wouldn't want the server to constantly retry that task assignment for 60 days? I don't disagree with you, but I'm trying to look at it from both sides. Let's assume the network traffic and CPU cycles req'd to keep retying for 60 days is irrelevant (as it should be). Any other reason not to have it retry?

With the network traffic and CPU negligible, I can't come up with a good reason not to just have it retry the task. This was just something that I thought I'd bring up in the design phase. Seems like you have it all thought through.

[Slugger]

Originally Posted by skiingwiz

With the network traffic and CPU negligible, I can't come up with a good reason not to just have it retry the task. This was just something that I thought I'd bring up in the design phase. Seems like you have it all thought through.

The only issue I can think of is that you'd have a task in the queue for up to 60 days. I don't particularly like that myself, but for an archiving task, I'd create a cron job that goes out and finds recordings that are old enough and then queue them for archiving (or just move them right away) instead of queuing an archival task for every recording that is started.

Keep thinking of issues/scenarios b/c, trust me, I definitely haven't thought of them all and this is why I'm starting this discussion now before I start coding.

[Fuzzy]

On the UI Mod discussion, I can definately see a need to be able to add a task to the queue for a given object, or group of objects. Perhaps if the Tasks themselves could be tagged with a UI_Friendly_Name property. If that tag exists, then that task will be created as a button on the options menu for media file(s). My first example would be a 'Send to Phone' task that could, for instance, compress a file, and put the results in a 'sync' folder for next device sync.

[Slugger]

Originally Posted by Fuzzy

On the UI Mod discussion, I can definately see a need to be able to add a task to the queue for a given object, or group of objects. Perhaps if the Tasks themselves could be tagged with a UI_Friendly_Name property. If that tag exists, then that task will be created as a button on the options menu for media file(s). My first example would be a 'Send to Phone' task that could, for instance, compress a file, and put the results in a 'sync' folder for next device sync.

I have to be careful because I won't be implementing this so I don't want to speak for my STVi volunteers, but my thought would be a more generic button on the recording options screen that, when clicked, would provide a list of all defined job ids and then you could click all the ones you wanted to queue up for that object then click submit and it would queue them all up for that object.

Your idea is just as good except that I'd imagine would be a lot more difficult to provide in the STV. But either way, I'm really looking forward to integrating something into the STV this time around.

[verdin77]

As I think about your desire to reduce future maintenance requirements I was wondering if you had thought about making use of one of the several open source job schedulers out there. They generally have clients already written and people who are interested in maintaining that part of the project. This would allow you to concentrate on the Sage interface and the unique logic required to prioritize operations for processing shows. Also, they tend to focus on robust job completion in cluster environments so we'd have a ready made mechanism for managing job failures.

Down side is that they have a bit of a learning curve and may be a bit heavy for this type of work. But it might be worth looking into.

[Slugger]

Originally Posted by verdin77

As I think about your desire to reduce future maintenance requirements I was wondering if you had thought about making use of one of the several open source job schedulers out there. They generally have clients already written and people who are interested in maintaining that part of the project. This would allow you to concentrate on the Sage interface and the unique logic required to prioritize operations for processing shows. Also, they tend to focus on robust job completion in cluster environments so we'd have a ready made mechanism for managing job failures.

Down side is that they have a bit of a learning curve and may be a bit heavy for this type of work. But it might be worth looking into.

Are there any in particular you're thinking of? I've looked at some (for v4 and in the past before writing v3 and v2) and I haven't ever found one that fits the needs of SJQ quite right. If you are thinking of some specific projects then let me know because I would definitely like to take a look.

[loonsailor]

I run sjq under linux, and the one little nit that bothers me is that I need to restart the client manually each time I restart sage. If I restart sage without restarting sjqc, the new incarnation of sage never connects to the existing sjqc. Some sort of more robust sage-sjqc connection would be nice.

[Slugger]

Originally Posted by loonsailor

I run sjq under linux, and the one little nit that bothers me is that I need to restart the client manually each time I restart sage. If I restart sage without restarting sjqc, the new incarnation of sage never connects to the existing sjqc. Some sort of more robust sage-sjqc connection would be nice.

This will be addressed in SJQv4.

[Slugger]

So I'm well underway with the coding of the new SJQv4 engine. I've got the new task client talking with the new server engine and vice versa. Tasks can be added to the queue and tasks are assigned to free clients, blah, blah, blah. It's all PoC/pre-alpha at this point, but it's coming a long quite nicely. My original estimates of a month or so are probably way off, it's going to be longer before betas are made public (the free hours to code this just aren't as plentiful as I was expecting).

Anyway, I'm starting to think about how people are actually going to write those custom conditions. Helper programs that I (or others) may provide will never be able to cover every conceivable scenario. For those special cases, users are going to have to write scripts themselves to do those checks and tests. Though any program/script/exe in any language can be run by the task clients (assuming you have the proper runtime environment installed on your system), I can't possibly provide a reference environment for them all. So it's time to speak up! Here's exactly what you're deciding on:

For lots of users, they'll simply say they want to run comskip.exe for task id 'COMSKIP'. They don't have any conditions, etc., they just want to run it. Great, tell the (new v4) task client that COMSKIP id maps to comskip.exe and you're done.

Other users are going to want to ignore comskipping certain channels and maybe delay running when clients/extenders are connected to the server. For these common/popular conditions, there will likely be a helper program available so you'll just map 'COMSKIP' to "intelliskip.exe" where intellitskip.exe is a program contributed by someone in the community that takes a bunch of command line args, etc. and will do the checks for you. This is also an easy case.

Now, what we're talking about here is the final scenario: You only want to run comskip on shows with specific titles, on specific channels, on specific days and you only run if there are no clients connected and there is no recording scheduled to start for at least the next 3 hours.

Obviously, that's a pretty unique/custom set of circumstances you want to test for. It's very unlikely that someone else in the community will have a helper program that can deal with all of those requirements. So you need to write you're own tester. If you can program then you can write a Java program, C#, etc. to do this and then map 'COMSKIP' to your new exe and that will work fine. But if you're not a programmer, you need a reference platform - a skeleton, if you will. And that's what I hope to provide. Basically, I'd have a skeleton of a script where there's a single function that does the prechecks as desired and if those pass then it proceeds to launch your exe (comskip.exe, in this example). All you'd have to do is replace the exe to call and fill in your test block with the conditions you want to check for.

There is but one requirement: the reference platform must be a scripting language. Why? Obviously people with this need for custom tests aren't going to have C/C++ compilers or JDKs installed and you don't want to have to require a recompile of the skeleton for each change. I want the user to be able to make their change, save it as a new script, then map their task id to the script and then forget about it.

I'm stuck deciding between PHP and Perl. My main reasons for these two are:

• Of all the scirpting languages I know, I'm most comfortable with these two.
• Both languages are supported on Windows and Linux (and I assume OS X)
• Installing the runtime for either is more than doable on both Windows and Linux - sorry OS X users, I just don't have access to a OS X env so if I'm shutting you out with these choices then speak up

With no feedback on the issue, I'm most likely to proceed with PHP5. This means users (who require the reference/skeleton scripting support) will need to install PHP5 on all systems that run a task client.