Plugin: SJQv4

[Slugger]

Originally Posted by wayner

One suggestion - it would be nice if you could view both logs as an option just like in the UI.

Ticket this; it's easily doable in a future update.

Is there a maximum size for the logs? Some tasks, I am thinking comskip in particular, can be very verbose even when you try to keep the verbosity at a low level. Trying to comskip an 8 hr recording could produce a huge log but I don't have many recordings like that - Ryder Cup golf being the one that comes to mind.

There is no max, and the way SJQ currently handles logging there is a good chance you'll exhaust the memory of the JVM if a process dumps enough output. The task client currently captures all the task output in memory then dumps it back to the server after the process finishes in one shot. However, SJQ tasks are run in their own JVM so if this were to happen then you'd only crash the task client executor JVM, never the actual Sage core JVM.

Before Google joined the party, I actually had redesigned the task logging completely to address this (and other issues with H2, namely eliminating the need for CLOB columns in the database) - I still see the design notes in my Sage notebook sitting here on the office desk. Google shifted my focus to other things and this exercise got put on the back burner. Not sure if/when I'll get around to it, but if I do then the redesign addresses all these issues such that you can dump however much data you want in process output and it wouldn't cause issues (the only limitation being the available space on the disk housing the SJQ database).

With all that said, this isn't next on my Sage todo list nor is it even on the radar at all for 2012. So the current limitations and issues probably aren't going to be addressed (by me) any time soon.

[wayner]

I agree that this isn't a high priority as in almost all instances it can be fixed by turning done the verbosity in the exe that you are running.

[graywolf]

Is it possible to use Groovy in a Sage V6 environment?

[Slugger]

Originally Posted by graywolf

Is it possible to use Groovy in a Sage V6 environment?

Yes... probably. Depends on what you mean by "use"?

[graywolf]

Do up groovy scripts to work with Sage 6 airing/show info.

Trying to see about cleaning up where a bunch of files were "corrupted" because the AiringIDs got reset.

Was thinking it might be easier using Groovy like I do with SJQv4 than 1 by 1 via the old web interface.

[Slugger]

Originally Posted by graywolf

Do up groovy scripts to work with Sage 6 airing/show info.

Trying to see about cleaning up where a bunch of files were "corrupted" because the AiringIDs got reset.

Was thinking it might be easier using Groovy like I do with SJQv4 than 1 by 1 via the old web interface.

Let's take SJQ out of the equation (since SJQv3 cannot use Groovy directly)... I've been meaning to document this, but anyway here's a quick overview of my "groovy SageTV environment", which has nothing to do with SJQ.

1. Install Groovy 1.8; grab the Windows installer version, ensure Groovy is on the system path.

2. Grab the latest sagex-api.jar for your version of SageTV. Add it to the %HOME%/.groovy/lib folder. Doing this means groovy has access to the sagex APIs everytime it's started.

(NOTE: This works on Linux, too, I've done it on my test server, just do the equivalent "Linux" thing for the two steps above.)

3. Now simply write a script against the sagex-api.jar. You can now use all the Sage APIs from groovy scripts (SJQ not requred).

In my env I've written a bunch of utility scripts that I use often to set system properties, read props, etc. during plugin development. Then I wrote a wrapper around those such that now I have this program called "sageg" that allows me to run these scripts from my command line:

C:\Users\slugger>groovy --version
Groovy Version: 1.8.4 JVM: 1.6.0_20

C:\Users\slugger>sageg
sageg: Required command missing!  Valid commands are:

   getcp: Get a client property: <ui_context> <name>
   getos: Display the SageTV server's OS
   getsp: Get a server property: <name>
lnimport: Import a video and link it to a Show object
   setcp: Set a client property: <ui_context> <name> <val>
   setsp: Set a server property: <name> <val>

C:\Users\slugger>sageg getos
LOG4J: Configured Logging for: sagex-api using file: sagex-api.log4j.properties
Connecting to SageTV server at [192.168.0.42]
Windows XP

C:\Users\slugger>sageg getsp SageSync/ts
LOG4J: Configured Logging for: sagex-api using file: sagex-api.log4j.properties
Connecting to SageTV server at [192.168.0.42]
1326346500001

C:\Users\slugger>

This output is from my Windows 7 laptop, where I do a lot of Java coding, and it's talking to my Sage server on XP. With a single command line switch I can immediately start running the same commands against my Linux test server.

Each command listed above is just a plain old groovy script named after the command (i.e. getos.groovy) and are sitting in a predetermined directory. Some are just one liners (getos), others are a little longer (lnimport).

All those commands are various groovy scripts I've written to accomplish each task listed. As time goes on, I'll continue to add to that. You can do that same kind of thing once you install groovy and add the sagex-api jar to the classpath.

This will work with any version of Sage - just make sure the sagex-api jar is in the classpath and the RMI service is enabled with the sagex-services plugin on your Sage server.

NOTE: That if you then wanted to add groovy to SJQv3, you could do it by calling groovy.exe in your tasks. Remember that doing it this means you don't get any of the implicit variable injection as discussed in the SJQv4 docs (but all those auto variables are easily pulled into scripts with basic sagex API calls).

[ptzink]

Hello all. I have a licensed version of SJQ4 installed, but haven't really utilized it much. However, I am now trying to setup a SageTV full directory backup job to run once a week using groovy scripts similar to what is outlined here http://code.google.com/p/sagetv-addo...vBackupExample. At the bottom of this page, it suggests an alternative method (which seems simpler to me) using only sagegroovy and a merged test/backup groovy script. Does someone have this stand alone script they can share? I am assuming it is not just a copy/paste since I have tried that with errors. Any help is much appreciated.

[Slugger]

Originally Posted by ptzink

Hello all. I have a licensed version of SJQ4 installed, but haven't really utilized it much. However, I am now trying to setup a SageTV full directory backup job to run once a week using groovy scripts similar to what is outlined here http://code.google.com/p/sagetv-addo...vBackupExample. At the bottom of this page, it suggests an alternative method (which seems simpler to me) using only sagegroovy and a merged test/backup groovy script. Does someone have this stand alone script they can share? I am assuming it is not just a copy/paste since I have tried that with errors. Any help is much appreciated.

Basically, just take the main script and right below the import lines add your conditions for execution. For example, if you wanted to not do backups if there's a recording in progress then put this just below the import lines:

while(Global.GetCurrentlyRecordingMediaFiles().size() > 0) {
    sleep 60000 // Check every minute until there are no recordings, then proceed
}

Obviously you'll probably want to add more conditions (i.e. no clients connected, no recording scheduled in the next 30 mins, etc.). Some of those conditions are shown in the test script. Basically, just keep the script in that loop until you determine it's "safe" to proceed with the backup.

[ptzink]

Slugger, thanks. That sounds easy enough. It turns out that my problem (right now) is with my SJQ4 install on my test server. Here is an excerpt from my log file.

2012-01-24 11:37:37,022 WARN  [DataStore]: Problem connecting to database... trying again in 30 seconds...
2012-01-24 11:38:24,781 FATAL [DataStore]: SQL Error
org.h2.jdbc.JdbcSQLException: Connection is broken: "Connection refused: connect" [90067-145]
	at org.h2.message.DbException.getJdbcSQLException(DbException.java:327)
	at org.h2.message.DbException.get(DbException.java:156)
	at org.h2.engine.SessionRemote.connectServer(SessionRemote.java:380)
	at org.h2.engine.SessionRemote.connectEmbeddedOrServer(SessionRemote.java:271)
	at org.h2.engine.SessionRemote.createSession(SessionRemote.java:265)
	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:110)
	at org.h2.jdbc.JdbcConnection.<init>(JdbcConnection.java:94)
	at org.h2.Driver.connect(Driver.java:72)
	at java.sql.DriverManager.getConnection(Unknown Source)
	at java.sql.DriverManager.getConnection(Unknown Source)
	at com.google.code.sagetvaddons.sjq.server.DataStore.<init>(DataStore.java:147)
	at com.google.code.sagetvaddons.sjq.server.DataStore.<init>(DataStore.java:60)
	at com.google.code.sagetvaddons.sjq.server.DataStore$1.initialValue(DataStore.java:68)
	at com.google.code.sagetvaddons.sjq.server.DataStore$1.initialValue(DataStore.java:64)
	at java.lang.ThreadLocal.setInitialValue(Unknown Source)
	at java.lang.ThreadLocal.get(Unknown Source)
	at com.google.code.sagetvaddons.sjq.server.DataStore.get(DataStore.java:101)
	at com.google.code.sagetvaddons.sjq.server.Plugin.start(Plugin.java:270)
	at sage.plugin.a.case(Unknown Source)
	at sage.SageTV.run(Unknown Source)
	at java.lang.Thread.run(Unknown Source)
Caused by: java.net.ConnectException: Connection refused: connect
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.PlainSocketImpl.doConnect(Unknown Source)
	at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
	at java.net.PlainSocketImpl.connect(Unknown Source)
	at java.net.SocksSocketImpl.connect(Unknown Source)
	at java.net.Socket.connect(Unknown Source)
	at org.h2.util.NetUtils.createSocket(NetUtils.java:109)
	at org.h2.util.NetUtils.createSocket(NetUtils.java:91)
	at org.h2.engine.SessionRemote.initTransfer(SessionRemote.java:91)
	at org.h2.engine.SessionRemote.connectServer(SessionRemote.java:376)
	... 18 more

I have tried uninstalling the plugin, deleting the sjq folder in the plugins directory, and reinstalling to no avail. Any ideas?

[Slugger]

The H2 database plugin isn't working properly. You need to fix that. If you have squeezebox running on the system (I think or some other program), it may be using the default port that H2 tries to bind to. In any case, you need to figure out why H2 isn't running.

[ptzink]

Slugger, I am on my production server now and have decided to go the stand alone SJQ4 task client route. My question is probably a stupid one. When I follow Tom's UI guide to create a new task, do I specify the "full_backup.groovy" as the exe? I tried a task with "full_backup_test.groovy" as the test script and "full_backup.groovy" as the executable. The test script was working (I know because I tried to purposely run it during a recording) but the "full_backup.groovy" did not work. Do I need to create an .exe out of this? How? Thanks for all your help and for all of your great plugins. I owe you multiple beers at this point.

[Slugger]

If you want to use a groovy script as an executable then you must prefix the file path with 'script:' as described here:

http://code.google.com/p/sagetv-addo...sk_Executables

[ptzink]

Originally Posted by Slugger

If you want to use a groovy script as an executable then you must prefix the file path with 'script:' as described here:

http://code.google.com/p/sagetv-addo...sk_Executables

Slugger, thanks again! That did the trick. I must have missed that in my reading of the wiki. For some reason my backup failed/crashed at the "STVs" folder. I will give it another shot after my wife finishes watching AI. Which brings me to my next series of questions:

• Is there a way to be sure an extender/client isn't playing a recording in the test script?
• Can I check to see if a process (comskip.exe) is running in the test script? I am using the comskip plugin (vs. SJQ) for ease of use's sake.
• Is there a way to power on an extender after restarting the sagetvservice in the backup script?

Appreciate any pointers. Thanks as always.

[coppit]

I've set up a mediashrink task to convert mpegs to h264 mkvs. I set the Maximum Time to 10800 (3 hours), but I see jobs that run for much longer than that.

Max Time Ratio is set to 1, and Resources is 100.

Is there something else I need to do, or is this a known issue? I noticed that mediashrink spawns HandBrakeCLI.exe, and that's what is taking a long time. (I think it gets wedged if the MPEG has corruption in it.)

Does SJQ4 have trouble killing the entire process tree?

[Slugger]

Originally Posted by ptzink

Slugger, thanks again! That did the trick. I must have missed that in my reading of the wiki. For some reason my backup failed/crashed at the "STVs" folder. I will give it another shot after my wife finishes watching AI. Which brings me to my next series of questions:

• Is there a way to be sure an extender/client isn't playing a recording in the test script?
• Can I check to see if a process (comskip.exe) is running in the test script? I am using the comskip plugin (vs. SJQ) for ease of use's sake.
• Is there a way to power on an extender after restarting the sagetvservice in the backup script?

Appreciate any pointers. Thanks as always.

#1 and #3 can be done with API calls. However, just so happens I've already written utilities to check these exact types of things. Just include the referenced packages in the javadocs. Both items are dealt with in the GlobalHelpers class. You'll want to checkout the isMediaPlaying() and powerOnExtenderHost(). Read the docs carefully for the extender power on methods.

#2 can be done, but it's OS specific. For Windows, you'll want to research the pslist tool, assuming you're not going to write JNI code to query the Windows process table. That's all I'm going to say about that - it's doable, but you'll have to do the research.

[Slugger]

Originally Posted by coppit

I've set up a mediashrink task to convert mpegs to h264 mkvs. I set the Maximum Time to 10800 (3 hours), but I see jobs that run for much longer than that.

Max Time Ratio is set to 1, and Resources is 100.

Is there something else I need to do, or is this a known issue? I noticed that mediashrink spawns HandBrakeCLI.exe, and that's what is taking a long time. (I think it gets wedged if the MPEG has corruption in it.)

Does SJQ4 have trouble killing the entire process tree?

Yes, sort of. The Process API for Java is terrible. It only promises to kill the spawned process and makes no promises about any child processes it may spawn. And that's the issue here, especially on Windows. SJQ does all it can do, but the underlying JRE and/or OS calls don't do what is expected/desired. All SJQ can do is kill the original process. I think part of the problem is that mediashrink fork()s for all it's child processes it spawns, which detach from the parent process, making it even less likely Windows is going to cleanup the mess when asked to by the JRE. In other words, the killing/timing out of tasks is a best effort feature in SJQ (due to limitations in the JRE).

[coppit]

Originally Posted by Slugger

In other words, the killing/timing out of tasks is a best effort feature in SJQ (due to limitations in the JRE).

I thought as much. Do you have access to the PID? If so, this worked well:

C:\Users\Privacy>taskkill /pid 2544 /t /f
SUCCESS: The process with PID 2636 (child process of PID 2092) has been terminated.
SUCCESS: The process with PID 2092 (child process of PID 1732) has been terminated.
SUCCESS: The process with PID 1732 (child process of PID 5616) has been terminated.
SUCCESS: The process with PID 5616 (child process of PID 4996) has been terminated.
SUCCESS: The process with PID 4996 (child process of PID 2544) has been terminated.
SUCCESS: The process with PID 2544 (child process of PID 6564) has been terminated.

I ran it on MediaShrink.exe and it seemed to have nuked HandBrakeCLI.exe pretty well. I'm on Win7 64.

[ptzink]

Originally Posted by Slugger

#1 and #3 can be done with API calls. However, just so happens I've already written utilities to check these exact types of things. Just include the referenced packages in the javadocs. Both items are dealt with in the GlobalHelpers class. You'll want to checkout the isMediaPlaying() and powerOnExtenderHost(). Read the docs carefully for the extender power on methods.

#2 can be done, but it's OS specific. For Windows, you'll want to research the pslist tool, assuming you're not going to write JNI code to query the Windows process table. That's all I'm going to say about that - it's doable, but you'll have to do the research.

Thanks for pointing me in the right direction. I just wanted to make sure the were doable (assumed they were) before I spent a lot of time looking into them.
I just queued up the backup up job again and if failed once more in the STVs folder. Any ideas what is going on here? I attached the log with the most recent backup attempt at the end of the list. Sorry to pester you and thanks for the help.

[Slugger]

Originally Posted by coppit

I thought as much. Do you have access to the PID? If so

No, I don't and that's the heart of the problem. The Process API in Java does not provide access to the process PID meaning all I can do is call Process.destroy(). Whatever destroy() does is what SJQ can do as far as killing processes and in the case on Windows it seems to only ever kill the parent process that was spawned.

[Slugger]

Originally Posted by ptzink

Thanks for pointing me in the right direction. I just wanted to make sure the were doable (assumed they were) before I spent a lot of time looking into them.
I just queued up the backup up job again and if failed once more in the STVs folder. Any ideas what is going on here? I attached the log with the most recent backup attempt at the end of the list. Sorry to pester you and thanks for the help.

Are you sure it's not actually creating the full backup? The logs simply show that the SageTV server has not fully restarted before the backup task finishes sleeping. After the backup is created, the script restarts SageTV and then sleeps for 5 minutes (if not changed). It has to sleep in order to allow time for the Sage server to restart the SJQ server so that the task can report its status back. If the task doesn't sleep long enough then when it tries to report back to the SJQ server it fails because the SJQ server isn't back online. That's what the log file is showing.

Try increasing the sleep time at the top of the script from 300 seconds to something higher - try 600 or 900 seconds. Be generous, you must ensure that the SJQ server is back online before that sleep time ends otherwise you get the errors you're seeing. Now if the actual copying of the dir failed then the output should be reported back to the server as long as the server is back online to receive the response from the task client.