Plugin: SJQv4

[Slugger]

Originally Posted by wayner

But thing of the other example - your wife calls when you are at work or out of town and complains about really slow performance of the UI. You pull out your smartphone and restart the service and tell her to try again in 3 minutes and that she can thank Slugger for that capability!

Geeez... when you put it that way.

I'll probably implement something here and see what I think of it. From there, I'll decide if I want to release it.

To be honest it may make more sense to incorporate this functionality into another web server (assuming that you have one on your LAN) since sometimes when the server gets bogged down it kills the web UI as well. Then you would run a batch file that contains something like "net stop SageTV && net start SageTV" and you are done, at least for a Windows server.

Assuming your Sage box is up to the task, you could just run an additional web server on an alternate port. Even Jetty. You can just launch an additional Jetty server using the jetty jar in your SageTV/JARs folder. Just point it at alternate config files. Doing that would launch a separate JVM for the server, keeping it separate from the SageTV process. Anyway, lots of alternatives. In this Jetty scenario, you just run a single app that does exactly what you describe when you hit the url.

I do actually have a Groovy web page that can restart Sage but it requires manually entering the URL.

What are you looking for instead? All I'll be doing is adding a groovy page (probably with the exact same content as yours) and adding a link to it as a menu item somewhere. You realize you can also add new menu items to the webui right now? That link... that's all I'll be doing myself. These functions (restart & GC) will not be incorporated into the existing webui Home page. That's a legacy page written in (ugly, hard to read, hard to maintain, scary to edit) Java code that I will not touch unless absolutely necessary and at this stage of the game, nothing I can imagine will come about requiring me to touch any of the existing legacy Java code ever again. I'm just relieved every time I make edits to the groovy infrastructure of the webui and the legacy java code continues to successfully compile.

Actually I did remember seeing the SJQ webUI stuff - it is in Groovy, isn't it.

Indeed, it's all written in groovy. But I made it a separate plugin because at the time I wrote it, I thought I'd be doing more changes/updates/etc. to it and didn't want it to be a part of the main webui. Also, it shows how others could add webui functionality via isolated plugins if others wanted to continue adding to the web ui.

Regrading the JVM stuff - I am surprised that this wasn't part of the original web UI but maybe it is only an issue for people with lots of extenders, like me.

Mainly, I imagine, because a well written Java app should never rely on/require manual, explicit calls to the garbage collector. I think this is more of a function of the app slowly growing beyond the max memory abilities of the 32bit JVM, but either way it's not really that hard to add. Again, though, it's going to be a menu item pointing at a groovy file that I'll add to stock webui build.

[Taddeusz]

Originally Posted by wayner

But thing of the other example - your wife calls when you are at work or out of town and complains about really slow performance of the UI. You pull out your smartphone and restart the service and tell her to try again in 3 minutes and that she can thank Slugger for that capability!

This is the reason I have a ssh server and tunnel Remote Desktop through it. Allows me to be able to remote into any computer on my network and not have port 3389 directly exposed to the internet.

[Slugger]

Just released my update for the SJQv4 WebUI plugin. This update has to do with adding the ability to attach metadata to queued tasks from the web.

Just realized that I forgot to update the dependency for this update to require the new web ui release (from earlier today). If you don't update the web ui first, then the queue a task page from this plugin will look and act weird. Just update the web ui first before updating this plugin.

The web ui update includes the ability to call the JVM garbage collector and to restart the SageTV service from the web ui home page (as previously discussed, OT, in this thread).

[ptzink]

Hey Slugger, I have been having issues with SJQ backup task failing for the last month or so and have just got around to troubleshooting. Hopefully, this is an easy one. Here is an excerpt from the log file:

2014-01-27 11:01:26,534 FATAL [192_168_1_5-23347-160]: Failed to update Task[id=FULLBACKUP, reqRes=100, maxInst=1, sched=ON, maxTime=86400, maxTimeRatio=1.0, rc=0-0, exe=script:c:\scripts\full_backup.groovy, exeArgs=, test=c:\scripts\full_backup_test.groovy, testArgs=]
java.net.ConnectException: Connection refused: connect
	at java.net.TwoStacksPlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
	at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
	at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
	at java.net.PlainSocketImpl.connect(Unknown Source)
	at java.net.SocksSocketImpl.connect(Unknown Source)
	at java.net.Socket.connect(Unknown Source)
	at java.net.Socket.connect(Unknown Source)
	at java.net.Socket.<init>(Unknown Source)
	at java.net.Socket.<init>(Unknown Source)
	at com.google.code.sagetvaddons.sjq.listener.ListenerClient.<init>(ListenerClient.java:41)
	at com.google.code.sagetvaddons.sjq.agent.network.ServerClient.<init>(ServerClient.java:30)
	at com.google.code.sagetvaddons.sjq.agent.ProcessRunner.updateTask(ProcessRunner.java:226)
	at com.google.code.sagetvaddons.sjq.agent.ProcessRunner.run(ProcessRunner.java:173)
	at java.lang.Thread.run(Unknown Source)
2014-01-27 11:01:26,534 WARN  [ProcessRunner]: Removing task from active list: SJQ4Task-192.168.1.5-23347-160

I am running the Windows Task Client Service, so I tried adding the wrapper.exe executable and port 23347 to the firewall exceptions, but that did not resolve the issue. However, when I disabled the firewall completely, the task processed as normal. So the culprit definitely seems to be something with the firewall. What ports should I add/check for exceptions? Execuatables? Thanks for the help and as always thanks for the awesome plugins!

[ptzink]

I think I figured it out. I added port 23344 exception (server and client are on the same machine) and all seems well now. Not sure why it was working all this time and then suddenly stopped working. Oh well.

[wayner]

Apparently Windows Services, at least in XP, Vista and Win7, are not allowed to make calls to hardware encoders. But this is allowed in Windows 8. Therefore you cannot use SJQ when it is running as a service to kick off Intel QuickSync encoder jobs.

I use the nightly builds version of Handbrake that supports Intel QuickSync and when I tried to do encoding from SJQ on a client that was running the Windows Services version of SJQ client I was getting strange results - like it would crap out after 1 second or so of the file.

The solution to this problem is to use the Windows Console version of SJQ client.

[phelme]

Argh. 3 days after I leave on a snowboard trip, the H2 database engine once again fills up "sagetv/.lobs.db" with 65 GB's of temp file crap and my server crashes.

Has anyone tried a H2 jar file later than 1.2.145 that possibly fixes this?

If not, anyone ever create a script that monitors the lobs directory for excess usage, shuts down Sage, cleans the directory and restarts Sage?

[wayner]

I have had this happen occasionally but it hasn't always come back - I don't think I have had the problem in a long time.

Maybe you could try having something like Growl send you a message (to Prowl or something similar on your phone) when your hard drive is getting full. I think SageAlert used to have this functionality but I don't see it in the current version. I guess the other thing is to write an SJQ cron job that checks for free disk space every few hours and generates a System Message when it is low.

[Slugger]

Originally Posted by phelme

Argh. 3 days after I leave on a snowboard trip, the H2 database engine once again fills up "sagetv/.lobs.db" with 65 GB's of temp file crap and my server crashes.

Has anyone tried a H2 jar file later than 1.2.145 that possibly fixes this?

If not, anyone ever create a script that monitors the lobs directory for excess usage, shuts down Sage, cleans the directory and restarts Sage?

The problem is that I could never reproduce this issue locally to try and actually isolate and address the cause. I saw it happen once on one of my systems, but could never reproduce it. I did create a ticket for SJQ to rewrite the task logging to remove the use of CLOB columns in the sjq database, which would have effectively eliminated this problem but then dev on this project died when Google arrived and so I just never got around to it. The code base is so old now (last commit was sometime in 2011) that I wouldn't even be able to wrap my head around the codebase. The only remote chance of this ever being fixed is if the problem ever surfaced on my production box -- then and only then would I take a crack at trying to fix it at this stage.

You could try upgrading the H2 jar file to see if it addresses the issue. I don't know who actually maintains the H2 plugin within Sage, but maybe they'd upgrade the jar in the repo? Not sure if I'm particularly fond of that idea at this stage of the game though either.

Originally Posted by wayner

I have had this happen occasionally but it hasn't always come back - I don't think I have had the problem in a long time.

Maybe you could try having something like Growl send you a message (to Prowl or something similar on your phone) when your hard drive is getting full. I think SageAlert used to have this functionality but I don't see it in the current version. I guess the other thing is to write an SJQ cron job that checks for free disk space every few hours and generates a System Message when it is low.

Yeah, there are some creative ways one could alert themselves of this situation. I know it's not ideal, but I don't think it's going to get much better than this. Really, I'd just check the disk space on C: every 20 mins (or whatver) and if it ever falls below X GB then shutdown the SageTV service and fire off an email/txt/whatever.

[phelme]

Originally Posted by Slugger

You could try upgrading the H2 jar file to see if it addresses the issue. I don't know who actually maintains the H2 plugin within Sage, but maybe they'd upgrade the jar in the repo? Not sure if I'm particularly fond of that idea at this stage of the game though either.
...
Yeah, there are some creative ways one could alert themselves of this situation. I know it's not ideal, but I don't think it's going to get much better than this. Really, I'd just check the disk space on C: every 20 mins (or whatver) and if it ever falls below X GB then shutdown the SageTV service and fire off an email/txt/whatever.

I replaced the H2 jar file yesterday morning with the latest, I think it was 1.3.175. I had to rename it to a 1.2 version so the H2 plugin would be happy.

So far, so good, things work and nothing appears to be complaining. A number of tasks went off overnight without issue.

I'll check the .lobs.db directory everyone once in a while manually for now to see how it goes.
---
I'll take partial blame for huge LOB's because my groovy scripts output tons of debug data from the Handbrake & Mediashrink calls. I should really get around to putting a flag in to minimize that.

Edit: Link to H2 Database downloads.

[wayner]

Anyone have an idea of how to get the type of CPU from within SJQ/Groovy/Java? I use Handbrake to do video encoding and I need to determine if the CPU can support Intel Quick Sync - this is the case for Ivy Bridge, Sandy Bridge and Haswell but not for older CPUs. How can I find this out in SJQ?

[Slugger]

There's no direct way. I've pruned a script I have around for other things. This will give you a description of the CPU(s) on your Windows system. You might have to further parse and pick apart the string to do what you want. All error checking, etc. is suppressed in this example script.

def wmiInput = """
var locator = WScript.CreateObject("WbemScripting.SWbemLocator");
var services = locator.ConnectServer();
var cpus = new Enumerator(services.ExecQuery("SELECT * FROM Win32_Processor"));
while (!cpus.atEnd()) {
  var cpu = cpus.item();
  WScript.StdOut.WriteLine(cpu.Name);
  cpus.moveNext();
}
"""

def tmp = File.createTempFile('tmp', '.js')
tmp.deleteOnExit()
tmp << wmiInput

def cpuInfo = new StringBuilder()
def p = "cscript //NoLogo \"${tmp.absolutePath}\"".execute()
p.consumeProcessOutput(cpuInfo, System.err)
p.waitFor()
println "Your CPU appears to be a(n) $cpuInfo"

Output on my Sage system:

Your CPU appears to be a(n) Intel(R) Core(TM) i5-4570 CPU @ 3.20GHz

[wayner]

Thanks Slugger. There are also Windows Env Variables. How would you acess these from Groovy?

NUMBER_OF_PROCESSORS=4
PROCESSOR_ARCHITECTURE=AMD64
PROCESSOR_IDENTIFIER=Intel64 Family 6 Model 42 Stepping 7, GenuineIntel
PROCESSOR_LEVEL=6
PROCESSOR_REVISION=2a07

And I have found a Java API called SIGAR:

Class CpuInfo
 java.lang.Object
  org.hyperic.sigar.CpuInfo

getModel
 public java.lang.String getModel() Get the CPU model.
 Supported Platforms: AIX, FreeBSD, Linux, HPUX, Solaris, Win32. 

System equivalent commands:
• AIX: lsattr -El proc0
• Darwin: 
• FreeBSD: 
• HPUX: 
• Linux: cat /proc/cpuinfo
• Solaris: psrinfo -v
• Win32: 

Returns:CPU model

That should work in Groovy if I load the jar file, shouldn't it?

[Slugger]

System.getenv(<var_name>) to get a env var.

Yes on the jar, though I suspect it isn't doing anything much different than the script I posted.

[awhenry]

Originally Posted by wayner

Apparently Windows Services, at least in XP, Vista and Win7, are not allowed to make calls to hardware encoders. But this is allowed in Windows 8. Therefore you cannot use SJQ when it is running as a service to kick off Intel QuickSync encoder jobs.

I use the nightly builds version of Handbrake that supports Intel QuickSync and when I tried to do encoding from SJQ on a client that was running the Windows Services version of SJQ client I was getting strange results - like it would crap out after 1 second or so of the file.

The solution to this problem is to use the Windows Console version of SJQ client.

Just wanted to say thanks for this. I could not figure out why my slower linux server was still chugging through jobs with no problem and suddenly my Windows server was crapping out most of the time. I am not thrilled about having to run the application version of SJQ, but offhand I could not find any switch to handbrake to disable hardware encoding. It looks like you may be building up to a workaround? I am following with interest. And if I figure out a way to disable the hardware encoding and get back on to the service I'll post the details.

Thanks!

[wayner]

Originally Posted by awhenry

but offhand I could not find any switch to handbrake to disable hardware encoding. It looks like you may be building up to a workaround? I am following with interest. And if I figure out a way to disable the hardware encoding and get back on to the service I'll post the details.

Are you using the Command line version of Handbrake - HanbrakeCLI.exe? One of the paramaters is the encoder paramater -it is -e . For software encoding it is -e x264 . For Quick Sync haredware encoding it is -e qsv_h264. Then you may also need to change the deinterlace paramater. For QS you need a custom profile called qsv and it might have to be in quotes.

FYI - you can play with all of this stuff by running the Handbrake GUI. Set up all the parameters that you want and then start an encoding session. Then look at the logs and it will show you the CLI command that you need for that. Copy that command from the logs and paste it into a text file to work with.

For more info see here http://mcebuddy2x.codeplex.com/discussions/444131 and here: https://mcebuddy2x.codeplex.com/discussions/437207

[wayner]

Is it possible to assign SJQ tasks to a hotkey, like F3 or Ctrl-S or whatever? It would be very nice to be able to quickly assign tasks to shows by just scrolling through a show list and hitting a key. Is there any way this is doable?

[Slugger]

Originally Posted by wayner

Is it possible to assign SJQ tasks to a hotkey, like F3 or Ctrl-S or whatever? It would be very nice to be able to quickly assign tasks to shows by just scrolling through a show list and hitting a key. Is there any way this is doable?

No. I only deal with the web ui and in order to inject SJQ type functions into the recordings list or recordings details page, I'd have to rewrite the current legacy version of those web pages and that's just not going to happen.

But this is why I made those mods to the SJQ pages of the web ui. Now it's pretty easy to insert a task into the queue and attach metadata to the task as needed. I don't think I actually explained it, but the queue a task web page now has those inputs for media file ids. But you actually enter a recording name in the box and it searches your media for you and fills in the media id number via ajax. It's not what I'd call the best/preferred way to be able to do it, but it's certainly better than nothing.

[awhenry]

Originally Posted by wayner

Are you using the Command line version of Handbrake - HanbrakeCLI.exe? One of the paramaters is the encoder paramater -it is -e . For software encoding it is -e x264 . For Quick Sync haredware encoding it is -e qsv_h264.

I am indeed using the command line version, and I arrived at my command line in just the way you suggested, although that was a couple years ago now so I should probably revisit it. I am actually using -e x264, and I don't have the hardware for quicksync. However I think it must be trying to use some other GPU feature. I can't swear to it, but I think the only encodes that were failing were ones that I was resizing to a lower resolution (which is most of them, so most were failing). In any case, it works fine in application mode, so from that I conclude that it is the same issue you encountered, just with some other type of hardware access than yours.

So regarding your earlier note about Windows 8.1, would running that OS allow this process to run in service mode? If so I guess I could finally say I have at least one reason to want to upgrade...

Thanks for the links. Interesting.

[wayner]

I have had the Sage Groov Dev Environment on my Sage server for years. I also want to install this on other PCs on my LAN. I should be able to do this by just copying over all of the folders in my Groovy folder, including bin, lib, logs, etc. and then starting sagegroovy with the -s parameter linking to my server's IP, correct? That would be something like "C:\GroovyFiles\sagegroovy\bin\sagegroovy.exe -s 192.168.1.92 -g"

Copying over the lib folder should mean I have all of the jars that I will need. Is there anything else I should be aware of?