SJQv4: Technology Preview

[Slugger]

Ok, I got sick of writing docs today so I'm going to finish my Sage dev time today with a technology preview of SJQv4.

After a solid 2+ months of development, I'm pleased to announce that we've finished coding features for the initial beta of SJQv4 and are now finalizing documentation. Once the doc efforts are complete, SJQv4 public beta 1 will be made available.

First, the things you need to know about SJQv4:

1. SJQv4 requires a sagetv-addons license file to unlock concurrency features (i.e. distributed and parallel task execution) from the start of the beta. The crontab feature will be enabled initially during beta, but will be turned off without a license after a few weeks. The reasoning is the same as SageAlert and the details are here. Note that a SageAlert license will also unlock all SJQv4 features.
2. Thanks to the incredible efforts of tmiranda, SJQv4 seamlessly integrates into the STV via an STVi for the first time! There is currently no web UI for SJQv4 and as of this moment no plans to create one. Given the event based task queueing in v4, an STVi approach to the UI made much more sense.
3. SJQv4 no longer uses server rulesets to decide what gets put into the task queue. In SJQv4, all tasks are queued in response to events or schedule requests.
4. SJQv4 supports regularly scheduled task execution via the SJQ Crontab. No more wild and crazy ruleset hacks required to schedule periodic task execution.
5. Client configs are removed and replaced by the "SJQ Scripting Enivronment (ScriptEnv)". The official scripting language of SJQv4 is Groovy. Other scripting languages are possible, but not officially supported. During development I successfully (with some effort) used Sleep, Jython and JavaScript in addition to Groovy.
6. The entire client/server architecture was rewritten from scratch. The SQLite database was replaced with H2.
7. SJQv4 supports attachment of any type of object to a task through a more abstract approach by attaching metadata maps to tasks instead of the actual Sage objects themselves.
8. The task client (agent) is now installable as a SageTV plugin. The standalone Windows service and console task clients are also available for users wishing to install task clients on systems without SageTV installed on them.

For those eager to get learning about SJQv4, there are docs available online, but they aren't complete - far from it. But you can read what's there and start to get a feel of what's new in this release. In addition to the above user guide wiki, you will want to become familiar with Groovy. With the shift to a real scripting language comes a new learning curve with this release of SJQ, but a real, robust, complete scripting language also adds an infinite amount of power and flexibility to this tool! "Anything" that was possible in SJQv3 will become way easier to accomplish in SJQv4 - once you learn the basics of Groovy. But more importantly, things that weren't possible in SJQv3 are now available in SJQv4. Variables, for loops and all the things you can do with just those two things are now possible in SJQv4 via Groovy.

I'm excited about this release and look forward to the feedback once we make it available. I know this is going to present a new set of challenges for current users of SJQ and new users alike, but I also honestly believe that this iteration of SJQ provides an incredibly powerful platform to script any and all automatic task execution needs for your SageTV platform!

Have a look at the screen shots from the STVi and feel free to compliment tmiranda for his incredible efforts on bringing SJQ to the STV. ETA for public beta 1 is December 5.

The SJQ menu:
http://sagetv-addons.googlecode.com/...i_sjq_menu.jpg

Registered Task Clients:
http://sagetv-addons.googlecode.com/...st_clients.jpg

(Completed) Task List:
http://sagetv-addons.googlecode.com/...eted_queue.jpg

Assign Tasks to Favourites (Menu Injection):
http://sagetv-addons.googlecode.com/...sks_2_favs.jpg

Assign Tasks to Favourites (Assignment Menu):
http://sagetv-addons.googlecode.com/...ks_2_favs2.jpg

Assign Tasks to Events:
http://sagetv-addons.googlecode.com/...s_2_events.jpg

Manually Queue Media File (after it's already recorded/imported):
http://sagetv-addons.googlecode.com/...queue_4_mf.jpg

[tmiranda]

Slugger did all of the hard work, I just dressed it up a bit.

Here is a preview of the docs for the UI. As Slugger stated, they are not complete but there are a number of screenshots for you to look at.

http://code.google.com/p/tmiranda/wi...4UserInterface

[meisner]

Right now, I have STV v7 and SJQ v3 (beta) installed via the plugin manager. It almost works in the sense that when the system starts up, the SJQ client (installed as a service) fails to connect to the SJQ server since SageTV isn't running yet (not installed as a service, but set to autostart with the desktop).

So my questions, when this release comes out, can I just uninstall the SJQ server plugin and install the new v4 plugin via the plug in manager? Is there any other cleanup that will need to be done prior to installing v4?

Also, is it as simple as installing the v4 SJQ server and client via the plug in manager and writing the Groovy script? Nothing else needs to be done...except for the config within the SageTV GUI described in the docs?

[Slugger]

Originally Posted by meisner

Right now, I have STV v7 and SJQ v3 (beta) installed via the plugin manager. It almost works in the sense that when the system starts up, the SJQ client (installed as a service) fails to connect to the SJQ server since SageTV isn't running yet (not installed as a service, but set to autostart with the desktop).

So my questions, when this release comes out, can I just uninstall the SJQ server plugin and install the new v4 plugin via the plug in manager? Is there any other cleanup that will need to be done prior to installing v4?

Sage will actually generate an upgrade system message when the v4 engine is available. You will upgrade v3 to v4, it's all automatic and no cleanup will be necessary. You can optionally delete the old sjq.sqlite db file after the upgrade is complete, but you don't need to do that (v4 just ignores the old db file).

Also, is it as simple as installing the v4 SJQ server and client via the plug in manager and writing the Groovy script? Nothing else needs to be done...except for the config within the SageTV GUI described in the docs?

Correct. You will want to uninstall any v3 task clients as well. The new v4 task client will not collide with a v3 one, but keeping a v3 one running is a waste and unnecessary. Once you configure a task client it's just a matter of writing (or copying) your Groovy test scripts and setting them up to run as desired. Before v4 is put into the repository there will some full, complete examples of how to do common/popular tasks in SJQv4.

[meisner]

Excellent! thanks for the reply.

My SJQ setup really just runs comskip (donator version) and then runs a VB script to fixup a skew that always happens due to the video stream created by the HD-PVR. I just need to launch comskip.exe, call the MediaInfo.GetFileDuration() function, and then call the VB script (via cscript.exe). After reading some of the SJQv4 & Groovy docs, this should be an easy Groovy script to write.

[graywolf]

In V4 since there won't be rulesets/client tasks, how would you go about running comskip but using different .ini files based on channel?

At least for me, I could not get an .ini file that worked well with all channels and therefore had to make a few different ones based on the channels.

Example rulesets I use are below...the client tasks are pretty much the same except uses a differently specified .ini file

if [IsObjMediaFile == "true" && $.IsTVFile{} == "true" && $.GetAiringChannelName{} =% "TNT.*|USA.*|FX.*|DSC.*|WCCB.*" && FileExists != "%d%/%p%.edl" && (TimeOfDay < "19:40" || TimeOfDay > "23:59") ]
{
:PRIORITY 10
COMSKIP1
}

if [IsObjMediaFile == "true" && $.IsTVFile{} == "true" && $.GetAiringChannelName{} =% "BBCA.*|SYFY.*|WJZY.*" && FileExists != "%d%/%p%.edl" && (TimeOfDay < "19:40" || TimeOfDay > "23:59") ]
{
:PRIORITY 10
COMSKIP2
}

[tmiranda]

Originally Posted by graywolf

In V4 since there won't be rulesets/client tasks, how would you go about running comskip but using different .ini files based on channel?

You can do this by creating two Tasks, call them COMSKIP1 and COMSKIP2. Create a test script for COMSKIP1 that passes if the channel matches a channel that should be processed using the first .ini and create a test script for COMSKIP2 that passes if the channel matches a channel that should be processed using the second .ini.

Slugger may have a better solution.

[graywolf]

Not knowing Groovy or other options for V4,

Would you be able to do a CASE (or similiar function?). Then I could think it would be just 1 TASK and internally it would determine the .ini

Something along the lines of the below?

CASE channel
WHEN "WXYZ" then execute "C:/comskip/comskip.exe --ini=c:/comskip/comskip1.ini \"%c%\""
WHEN "WABC" then execute "C:/comskip/comskip.exe --ini=c:/comskip/comskip1.ini \"%c%\""
WHEN "BBCA" then execute "C:/comskip/comskip.exe --ini=c:/comskip/comskip2.ini \"%c%\""
WHEN "SYFY" then execute "C:/comskip/comskip.exe --ini=c:/comskip/comskip2.ini \"%c%\""
ELSE IGNORE
END CASE

[Slugger]

Originally Posted by graywolf

Not knowing Groovy or other options for V4,

Would you be able to do a CASE (or similiar function?). Then I could think it would be just 1 TASK and internally it would determine the .ini

Something along the lines of the below?

CASE channel
WHEN "WXYZ" then execute "C:/comskip/comskip.exe --ini=c:/comskip/comskip1.ini \"%c%\""
WHEN "WABC" then execute "C:/comskip/comskip.exe --ini=c:/comskip/comskip1.ini \"%c%\""
WHEN "BBCA" then execute "C:/comskip/comskip.exe --ini=c:/comskip/comskip2.ini \"%c%\""
WHEN "SYFY" then execute "C:/comskip/comskip.exe --ini=c:/comskip/comskip2.ini \"%c%\""
ELSE IGNORE
END CASE

Definitely.

Object mediaFile = MediaFileAPI.GetMediaFileForID(SJQ4_METADATA.get("SJQ4_ID").toInteger())
String channel = AiringAPI.GetAiringChannelName(mediaFile)
String file = SJQ4_METADATA.get("SJQ4_PATH") + SJQ4_METADATA.get("SJQ4_LAST_SEGMENT")

String iniFile;
if(channel ==~ /WABC|WXYZ/)
   iniFile = "comskip1.ini"
else if(channel ==~ /SYFY|BBCA/)
   iniFile = "comskip2.ini"
else
   iniFile = ""
if(iniFile.length() > 0)
   iniFile = "--ini ${iniFile}"

Process cmd = "C:/comskip/comskip.exe ${iniFile} \"${file}\"".execute()
cmd.waitFor()
return cmd.exitValue()

There may be some syntax errors in the above - it's from the top of my head only - but the point being it's relatively easy. All the SJQ4_* variables that magically appear everywhere in the script are magically provided by SJQ when it runs Groovy scripts. The Sage API calls are the same way - SJQ injects global variables that allow direct access to the Sage API (via stuckless' remote APIs). That's a complete, full example of how to do this in Groovy. Of course, it's just one way and may not even be the best way - it's just a way I thought of it off the top of my head. Hopefully this example shows just how powerful Groovy is and just how flexible the SJQ system is about to become. Of course, this new found power will require some learning for some people - it's a delicate trade off, but I think I've balanced it as best I can while providing an extremely powerful task automation environment for SageTV.

[tmiranda]

I forgot that the "executable" could actually be a Groovy script.

I've got to get my head out of the STV!

[aaronb]

This sounds great. I don't currently use SJQ3, and I don't know if I'll end up using SJQ4, but I just donated anyways because I appreciate the hard work people put in trying to make software like this even better.

[phunny]

I don't mind paying and I saw the FAQ about donations but where do you make it to? It mentions paypal but don't see an address I really like SJQ and don't want it to expire or stop working once the new version is available. Thanks

Andy

[razrsharpe]

Originally Posted by phunny

I don't mind paying and I saw the FAQ about donations but where do you make it to?

click the donate link near the top of sluggers google page

[phunny]

Thanks I made my donation to me its well worth it for the time spent on it!

[evilpenguin]

SJQ v4 is sounding very cool, let me run though my v3 tasks real fast to understand how I can convert them for v4.

1) Comskip - Having comskip queue off the recording events a lot of sense, but I worry its going to keep me from doing some of the neat tricks I can do in v3.

For example, right now I have 3 COMSKIP tasks configured, one for each recording drive, and I use the resource limits to ensure that there's never more than 2 comskips jobs on the same recording drive at the same time. Am I going to be able to do this in v4 by creating custom events for recording started/completed for each recording drive?

In addition, I also have a COMSKIP_FAILURE tasks that I run on recordings that comskip fails on. Right now its artifact based because I can detect the failure by seeing a .log but not an .edl. And since the recovery process is very harddrive/processor intensive I have it kick off in the middle of the night.

For this task i'm not clear on how I can implement it in v4. COMSKIP will be initiated by the recording started event and, while I can certainly detect the failure inside the groovy script I use to run comskip, I don't want to respond to the failure immediately rather pop it back in the queue as a new task that only runs at certain times

2) Mediashrink - In v3 I have mediashrink configured to only queue jobs if the recording is over a month old. Will it be possible to generate a custom event that fires when I recording hits a certain age?

[Slugger]

Originally Posted by evilpenguin

SJQ v4 is sounding very cool, let me run though my v3 tasks real fast to understand how I can convert them for v4.

1) Comskip - Having comskip queue off the recording events a lot of sense, but I worry its going to keep me from doing some of the neat tricks I can do in v3.

For example, right now I have 3 COMSKIP tasks configured, one for each recording drive, and I use the resource limits to ensure that there's never more than 2 comskips jobs on the same recording drive at the same time. Am I going to be able to do this in v4 by creating custom events for recording started/completed for each recording drive?

For public beta 1, SJQ only listens for and acts on RecordingStarted (RecordingSegmentAdded), RecordingStopped, SystemMessagePosted and MediaFileImported events. Adding more (core) events is rather trivial and will be done based on user demand. Adding support for custom events is also not too difficult, but SJQ (is not planned) to generate and fire custom events. If you fire the custom event, SJQ can be configured to listen for it and act on it (with some work on my end).

However, you won't necessarily need custom events to pull this off. Basically, on RecordingStarted queue up a "COMSKIP" task. It's only job (via Groovy script) will be to inspect the path of the recording and based on that path you can queue up a "COMSKIPDISK1" (or 2 or 3, etc.) task. You can load up tasks into the queue from other tasks via Groovy (or Java) since there is a common jar file for interactions with the SJQv4 task queue.

In addition, I also have a COMSKIP_FAILURE tasks that I run on recordings that comskip fails on. Right now its artifact based because I can detect the failure by seeing a .log but not an .edl. And since the recovery process is very harddrive/processor intensive I have it kick off in the middle of the night.

For this task i'm not clear on how I can implement it in v4. COMSKIP will be initiated by the recording started event and, while I can certainly detect the failure inside the groovy script I use to run comskip, I don't want to respond to the failure immediately rather pop it back in the queue as a new task that only runs at certain times

Two choices**: Detect the failure in the Groovy script and queue up a new task in response. Ensure your task client is configured to only run this new task in the middle of the night (task clients can be configured to run certain tasks only at certain times in v4).

Alternately, setup a task via cron that runs periodically and looks for .log with no .edl and then queue up tasks as appropriate. There is an SJQ crontab in v4 that allows periodic/repeated runs of arbitrary tasks.

** That I can think of right off the top of my head. The beauty of moving to Groovy is that there really is 50 different ways to do any one task now in SJQ. When I document my current comskip setup people are going to say, "why are you doing it like that instead of like this?" And in reality there are probably a half dozen better ways to do it, but I migrated to SJQv4 exclusively early in development and at the time it was actually the only way to do it and it also shows an example of tasks that check preconditions and ship tasks back to the queue.

2) Mediashrink - In v3 I have mediashrink configured to only queue jobs if the recording is over a month old. Will it be possible to generate a custom event that fires when I recording hits a certain age?

No plans for a custom event, though it's actually quite possible (not in beta 1, though). Instead, I'd recommend running a cronjob periodically that looks for recordings older than a month. I do something similar to find recordings I want to archive and will be documenting that example before launch.

[wayner]

How will the upgrade path work? In a day or two we will see a new upgrade in the Sage Plugin screens - we just upgrade and we go from the current version (which I believe is 3.1.0.93 beta) to a version 4 beta? Will we lose any functionality in the v4 beta?

FYI - I am very happy to pay for the upgrade. I personally don't mind if developers charge a small fee, or if they ask for donations to a charity of their choice.

[tmiranda]

Originally Posted by wayner

How will the upgrade path work? In a day or two we will see a new upgrade in the Sage Plugin screens - we just upgrade and we go from the current version (which I believe is 3.1.0.93 beta) to a version 4 beta? Will we lose any functionality in the v4 beta?

Yes, and no. You will see the upgrade notice appear in the Plugin Manager. If you do the upgrade things will not continue to work because the architecture of SJQv4 is completely different.

I'd suggest reading the wiki's Slugger and I have created to get a feel for what you will have to do. Most of it is straight forward, writing a new "test script" will probably be the most difficult part.

Tom

[Slugger]

The beta 1 is delayed for two reasons:

1. I was on vacation last week and despite my best intentions, no Sage dev work was done at all (but Montreal was a blast! ). I still have some examples to document. Now we're really getting close to the holiday season. I'm really going to try and get it done this week, but if it slips past the weekend then we're probably looking at the new year because holiday parties, blah, blah, blah all take over my calendar starting next week.
2. I'm investigating a possible memory leak with SJQv4. I 99% sure it's not SJQ that was leaking, but I've had to strip all plugins, etc. from my producton server and am running barebones + SJQ under my production load to verify. It has been 4-5 days to exhaust the heap memory on my production server (never happened on my dev server) so it's going to be a couple more days before I get some conclusive results. Hopefully everyone eagerly awaiting SJQv4 can appreciate the few extra days.

[Slugger]

Originally Posted by wayner

FYI - I am very happy to pay for the upgrade. I personally don't mind if developers charge a small fee, or if they ask for donations to a charity of their choice.

I believe you already have a SageAlert license file? SJQv4 uses the same one so no need to donate again. You just have to register your email address with SJQv4 after the upgrade and you'll be good to go. (A future update will centralize the sagetv-addons license registration such that you don't need to register with every plugin, but that's not done yet and won't be for SJQv4 beta 1).