Plugin: Sage Job Queue (SJQ) v3

[nyplayer]

SJQ 3 not compatible with Java 7 update 21. It appears that it fails on any batch job that uses cmd.exe (Windows Command Processor ).

INFO   | jvm 1    | 2013/04/19 08:56:21 | Exception in thread "Thread-95" java.lang.IllegalArgumentException: Argument has embedded quote, use the explicit CMD.EXE call.
INFO   | jvm 1    | 2013/04/19 08:56:21 | 	at java.lang.ProcessImpl.isQuoted(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:21 | 	at java.lang.ProcessImpl.needsEscaping(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:21 | 	at java.lang.ProcessImpl.<init>(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:21 | 	at java.lang.ProcessImpl.start(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:21 | 	at java.lang.ProcessBuilder.start(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:21 | 	at java.lang.Runtime.exec(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:21 | 	at java.lang.Runtime.exec(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:21 | 	at java.lang.Runtime.exec(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:21 | 	at com.google.code.sagetvaddons.sjqc.SJQThread.run(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:21 | Exception in thread "Thread-93" java.lang.IllegalArgumentException: Argument has embedded quote, use the explicit CMD.EXE call.
INFO   | jvm 1    | 2013/04/19 08:56:21 | 	at java.lang.ProcessImpl.isQuoted(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:21 | 	at java.lang.ProcessImpl.needsEscaping(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:21 | 	at java.lang.ProcessImpl.<init>(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:21 | 	at java.lang.ProcessImpl.start(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:21 | 	at java.lang.ProcessBuilder.start(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:21 | 	at java.lang.Runtime.exec(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:21 | 	at java.lang.Runtime.exec(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:21 | 	at java.lang.Runtime.exec(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:21 | 	at com.google.code.sagetvaddons.sjqc.SJQThread.run(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:21 | Exception in thread "Thread-94" java.lang.IllegalArgumentException: Argument has embedded quote, use the explicit CMD.EXE call.
INFO   | jvm 1    | 2013/04/19 08:56:21 | 	at java.lang.ProcessImpl.isQuoted(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:21 | 	at java.lang.ProcessImpl.needsEscaping(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:21 | 	at java.lang.ProcessImpl.<init>(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:21 | 	at java.lang.ProcessImpl.start(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:21 | 	at java.lang.ProcessBuilder.start(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:21 | 	at java.lang.Runtime.exec(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:21 | 	at java.lang.Runtime.exec(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:21 | 	at java.lang.Runtime.exec(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:21 | 	at com.google.code.sagetvaddons.sjqc.SJQThread.run(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:22 | Exception in thread "Thread-97" java.lang.IllegalArgumentException: Argument has embedded quote, use the explicit CMD.EXE call.
INFO   | jvm 1    | 2013/04/19 08:56:22 | 	at java.lang.ProcessImpl.isQuoted(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:22 | 	at java.lang.ProcessImpl.needsEscaping(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:22 | 	at java.lang.ProcessImpl.<init>(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:22 | 	at java.lang.ProcessImpl.start(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:22 | 	at java.lang.ProcessBuilder.start(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:22 | 	at java.lang.Runtime.exec(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:22 | 	at java.lang.Runtime.exec(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:22 | 	at java.lang.Runtime.exec(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:22 | 	at com.google.code.sagetvaddons.sjqc.SJQThread.run(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:22 | Exception in thread "Thread-98" java.lang.IllegalArgumentException: Argument has embedded quote, use the explicit CMD.EXE call.
INFO   | jvm 1    | 2013/04/19 08:56:22 | 	at java.lang.ProcessImpl.isQuoted(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:22 | 	at java.lang.ProcessImpl.needsEscaping(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:22 | 	at java.lang.ProcessImpl.<init>(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:22 | 	at java.lang.ProcessImpl.start(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:22 | 	at java.lang.ProcessBuilder.start(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:22 | 	at java.lang.Runtime.exec(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:22 | 	at java.lang.Runtime.exec(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:22 | 	at java.lang.Runtime.exec(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:22 | 	at com.google.code.sagetvaddons.sjqc.SJQThread.run(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:22 | Exception in thread "Thread-99" java.lang.IllegalArgumentException: Argument has embedded quote, use the explicit CMD.EXE call.
INFO   | jvm 1    | 2013/04/19 08:56:22 | 	at java.lang.ProcessImpl.isQuoted(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:22 | 	at java.lang.ProcessImpl.needsEscaping(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:22 | 	at java.lang.ProcessImpl.<init>(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:22 | 	at java.lang.ProcessImpl.start(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:22 | 	at java.lang.ProcessBuilder.start(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:22 | 	at java.lang.Runtime.exec(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:22 | 	at java.lang.Runtime.exec(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:22 | 	at java.lang.Runtime.exec(Unknown Source)
INFO   | jvm 1    | 2013/04/19 08:56:22 | 	at com.google.code.sagetvaddons.sjqc.SJQThread.run(Unknown Source)

see thread.
http://forums.sagetv.com/forums/show...03&postcount=1

[Slugger]

See my reply in the linked thread. Short answer: Definitely will never be fixed in SJQv3.

[nyplayer]

Originally Posted by Slugger

See my reply in the linked thread. Short answer: Definitely will never be fixed in SJQv3.

Understood I did a work around with the batchfile for SJQ to point to java 6 even though Sage uses Java 7 ... I need Java 7 in Sage because of the Phoenix android app.

[graywolf]

Got a strange error.

When I go to Tools => Ruleset Editor, instead of seeing my rules, it say Not Authorized

Any clues?

[nyplayer]

Originally Posted by graywolf

Got a strange error.

When I go to Tools => Ruleset Editor, instead of seeing my rules, it say Not Authorized

Any clues?

You probably need to clear the browser cache or hold down shift key when refreshing the SJQ browser.

[graywolf]

Finally got a couple minutes to look further into this. Looks like I have bigger issues.

2013-06-28 17:27:31,123 ERROR [MediaFileQueueLoader]: Unexpected exception
java.lang.RuntimeException: Expected "if" to start line 1
	at com.google.code.sagetvaddons.sjq.server.RulesParser.shellChecker(RulesParser.java:301)
	at com.google.code.sagetvaddons.sjq.server.RulesParser.loadQ(RulesParser.java:264)
	at com.google.code.sagetvaddons.sjq.server.RulesParser.(RulesParser.java:239)
	at com.google.code.sagetvaddons.sjq.server.MediaFileQueueLoader.run(MediaFileQueueLoader.java:73)
	at java.lang.Thread.run(Unknown Source)

Now to try and find out where that is. Or try and find the SJQ 3.0 install instructions and do a reinstall maybe.
Just going to have to find time.
Maybe if I don't get too many work calls this weekend I might get a chance to look at it some more.

[nyplayer]

That is a ruleset error.

[graywolf]

Got it working again.

Still got the Not Authorized when I did Edit Ruleset but did a copy/paste from the file I use as a backup of the rulese.

Did a SAVE and then things are working again.

Now it has a LOT of active jobs running since it looks like it broke near the beginning of June.

Glad to have it back working.

[Skirge01]

@Slugger: I'm having an issue (discussed here) with an absolutely huge sjq.sqlite file at 1.7GB and growing. nyplayer suggested that this file keeps logs of tasks, but I'm having difficulty figuring out how to clear the file out correctly (i.e. safely). I emptied the completed tasks from the UI, but it had no effect on that file. I do have SJQ set to purge itself weekly, but it would appear that's not helping either. Any ideas?

[nyplayer]

Originally Posted by Skirge01

@Slugger: I'm having an issue (discussed here) with an absolutely huge sjq.sqlite file at 1.7GB and growing. nyplayer suggested that this file keeps logs of tasks, but I'm having difficulty figuring out how to clear the file out correctly (i.e. safely). I emptied the completed tasks from the UI, but it had no effect on that file. I do have SJQ set to purge itself weekly, but it would appear that's not helping either. Any ideas?

Deleting from a DB really does nothing to the size the empty space is still there ... it has to be compacted. The sqlitebrowser application has an option to compact the DB make a copy of it and Compact the copy in sqlitebrowser and let us know if that helps. You do not need to stop sage in order to copy it.

another way is to download sqlite3.exe. ex below. I am not sure if Slugger VACUUMS SJQ DB.

cd c:\temp
c:
sqlite3.exe "c:\temp\sjq.sqlite" "VACUUM;"
exit

[Slugger]

Yeah, I don't remember how most of this works.

I seem to remember implementing a weekly vacuum of the database file, but, honestly, I can't remember and even if I implemented it, it doesn't mean it actually worked. Or if it did work, you may have found a case where it doesn't. At this stage, I'm really no help to you with anything related to SJQ prior to v4. And, as it is, I'm not much help with v4 either as I haven't looked at that code really since Google joined the party. The only saving grace for v4 users is that I still use v4, but my setup/use of it has been static for well over a year, probably longer. It all "just runs" in the background and I don't ever touch it.

As suggested, make a copy of the db file and vacuum it yourself to see if it shrinks. If it does, apply it to the production version of the file. If that helps, you could schedule a periodic vacuum with Windows task scheduler, etc. going forward. Vacuuming the file while being used by SJQ shouldn't be a problem, but you might want to test and/or google that to be sure b/c I barely remember the details of sqlite's inner workings.

[Slugger]

Though I have to say that the size of the sqlite file shouldn't matter. If that file is being accessed heavily and seems to be the cause of your problem, I'd suggest it's more likely whatever SJQ task is running and writing to the db rather than the db itself. A 1GB SQL database is really nothing in the grand scheme of things. Of course, there could be bugs in how I'm using the db in my code, but the JDBC driver & a 1 or 2 GB database file is almost certainly not the culprit. Again, I don't remember much of how I implemented v3 so bugs in my code, poor design, etc. are all possibilities along with whatever task(s) you're triggering to run after a recording is stopped.

[Skirge01]

Thanks nyplayer & Slugger. I did a compact and it shrunk it down to just 76MB, so that's one heck of a reduction. I'll see if that makes any difference with the issue I was trying to solve.

[Skirge01]

Originally Posted by Slugger

along with whatever task(s) you're triggering to run after a recording is stopped.

That's the thing, though; I have nothing scheduled to run when a recording is stopped. I currently use SJQ3 exclusively for conditional triggering of comskip 1 minute after a recording starts.

[Slugger]

But maybe it's when the recording ends and comskip ends that all the output is dumped to the db at once? Again, I don't remember and I'd have to dig deep into my svn archives to even find the code much less start to read it. But it seems to me that in v3 I may have buffered the task output into memory and waited to dump it into the db until after the process was terminated. I don't know if I'm right or not, but I think I am and it's one of the things I changed in v4 -- maybe, or I'm just making it up... only the source code can confirm. But I tend to think I may be right because it kind of sounds like what is happening. So SJQ eats the output and buffers it until the process completes then does its cleanup, which basically means to dump the buffered output to the db. If it does, that's definitely one of those "poor design" choices I was referring to earlier today. On beefier h/w, such a poor design choice could be masked and I'd have never noticed. My server would have been beefy by such standards (it's the same system in my sig) for when v3 was written (2009ish, I believe).

If such a design is the cause, vacuuming the db file won't do anything to help other than to free up a gigabyte on your hard disk. Again, the JDBC driver for sqlite is likely more than up to the task of dealing with larger db files (and, really, 1GB isn't what I'd call large anyway). The root of the problem would be that I decided to do a massive I/O operation all at once instead of spreading it out. If you want to confirm/fix this problem, do as nyplayer suggested, redirect the output from comskip to its own file, which would then alleviate the massive db write at the end of each comskip process.

[Skirge01]

I'm fairly certain this is happening even when comskip isn't running at all, no recordings are occurring, and while watching previously recorded (or even ripped) files. I'll have to pay attention to that and ensure that truly is the scenario, but I'm confident it happens even under those circumstances. Given that most shows are on hiatus at the moment, this should be an easy one to confirm. I really do appreciate all the input you're providing (and nyplayer, too, of course!).