Plugin: SJQv4

[wayner]

Coppit - I am not sure if this is exactly the same problems you were having but I was having issues with SJQ/Groovy choking on exes that had verbose output - and HandbrakeCLI is an exact example of this as is comskip even when you have the verbosity set very low.

But my code was 100% groovy which is different from you. Let me know if you want to see my code.

[wayner]

Originally Posted by Slugger

In summary, can SJQv4 do this kind of thing? Absolutely. But it will require you to write some decent amount of scripts to do it - or ask wayner nicely for his solution.

I never finished adapting Slugger's code to do this but it is almost there. But the code is all in Groovy so you have to graduate to SJQ4.

If you look back a couple of pages you will see a script that Slugger posted that contains the code to rename all of your Recorded TV files. But this doesn't actually rename the files as he didn't want it to screw up your file names. It just posts the old and new file names. It doesn't rename the folder structure but that should be a minor revision.

[GKusnick]

I'm trying to automate some transcoding tasks and I think SJQ might be the right tool for the job. In the past I've always just coded my own solution to this sort of problem but this time around I'm particularly attracted to SJQ's ability to offload task execution from the Sage server by farming it out to other machines on my LAN. That's something I'd rather not reinvent if I don't have to.

I've read through the docs and how-tos (but not this entire thread, so apologies if I've missed something). I think I see how to set up tasks and link them to events so that new recordings automatically get queued for transcoding. The part I'm not clear on, however, is what to do with my existing backlog of recordings, which I'd also like transcoded.

I see how to trigger tasks manually or periodically, but if I'm not mistaken, those sorts of tasks do not get passed a MediaFile argument; only the ones triggered by recording events get that. What I'd like to be able to do is to select a task and say, Queue an instance of this task for each existing recording (or for each existing MediaFile that meets some specified criteria, although I suppose I can postpone that check to my task's test script).

Alternatively, if there's a way that an executing task can queue additional tasks, that would also serve my purpose. Then I could just write a one-shot manual task to scan the existing recordings and queue transcoding tasks as appropriate. But I'm not really seeing a way to do that either.

Yet another way might be for me to raise fake RecordingStopped events for the existing recordings, thereby fooling SJQ into queuing tasks for them. But that seems a bit heavy-handed and may have undesirable side-effects.

Has anybody else dealt with this issue of applying a newly defined task to a large set of existing recordings? Any suggestions on the best approach to take?

Thanks.

[graywolf]

You can use/modify Slugger's Media File Scanner/Task Queuer script to determine the logic of scanning your library and to queue a task for the files/recordings that match the criteria

An example is attached. changed .goovy to .txt in order to get past the upload filter

[wayner]

Manual tasks do get passed a MediaFile argument. When you select a recording in Sage (assuming you have SJQ installed) one of the options in the menu is to run an SJQ task which will then be performed on that specific MediaFile. The problem is that you don't want to have to do this hundreds of times to run the task on all of your recordings.

If what you want to transcode is mainly TV shows another way to do this would be to manually get SJQ to launch tasks for all episodes of a given show or all episodes in a given season. That would probably lead to a manageable number of tasks in the queue at any given time. You could then have separate shows being transcoded on separate PCs. I don't know what the limit would be but I don't know that you want to have too many transcode jobs going at once if the original files are all on the same hard drive since that drive will be getting hammered, particularly if you have multiple recordings also occurring at the same time.

You might have to change your Sage options so that you get a menu to pop up when you select a TV Show title in Sage - from that menu you can then select a task which, I am pretty sure, will then get executed on all instances of the show.

I don't sort by Season but I believe you can get Sage to sort your shows by seasons, can you not? If so then you can do the same thing I described above but on a season by season basis for each show.

@GKusnick - which UI are you using? I don't know that it matters too much but the stuff I am describing is for Diamond but I am pretty sure that it is the same for the default UI. I don't know about Phoenix.

[Slugger]

@GKusnick

The media scanner example script is probably what you want to process existing recordings. Run it once from the crontab (or manually). You can queue other tasks from a script.

Lines 65-75 in that script show how you can queue other tasks from a script (i.e. the ServerClient() object).

[GKusnick]

Thanks; that helps a lot.

Still not 100% sure if I'm going to go this route. I had a brainstorm in bed last night about how to modify my existing Jetty webapp to work as a task dispatcher. So I may experiment with that a bit before jumping into SJQ. But at least I have a clearer picture now of how to do it either way.

Wayner, I have my own custom UI that I've been using for years. But I do revert to the stock UI for maintenance and configuration chores that my UI doesn't implement.

[coppit]

Originally Posted by wayner

I am not sure if this is exactly the same problems you were having but I was having issues with SJQ/Groovy choking on exes that had verbose output

It's possible, but:

• I'm echoing all the output from HandBrake now and don't seem to have problems
• It seems like the movies HandBrake gets stuck on have signal noise

It feels more like HandBrake going off the rails when there's lots of corruption in the MPEG stream.

[GKusnick]

Originally Posted by coppit

It feels more like HandBrake going off the rails when there's lots of corruption in the MPEG stream.

The released version of Handbrake (0.9.5) crashes routinely on my Colossus recordings. From my reading of the Handbrake forums, there seems to be a known memory-management bug in this version that shows up when running on 32-bit Windows. The bug has since been fixed but hasn't yet made it into a release build. If you're adventurous, you could try one of the nightly builds from here to see if it solves your problem (without causing any new problems).

[wayner]

Originally Posted by tmiranda

Java Heap Monitor is correct. It put's a nice little bar graph in the header and lets you set trigger levels for messages.

Thanks for the heads-up - just installed this and it looks great.

[wayner]

Here is the script that I use to call Handbrake to reencode. This script uses the preset iPad settings but it should be pretty easy to change the HBParams variable to change the parameters. I have written everything in groovy and call groovy from the executable line so there is no test script.

To configure in SJQ put in your Task ID name and put "script:c:/MyFolder/HB_ipad.groovy" (replancing your folder and your actual groovy file name) in the Executable line. Nothing else has to be changed.

This is for Win7 342 bit but it should be easy to adapt to other OSes. You may have to change your Handbrake path, particularly if you are running a 64 bit version of Windows or not running Windows.

The one tricky thing with this script is that you have to process the output from Handrake since it is so verbose.

Object mf = MediaFileAPI.GetMediaFileForID(Integer.parseInt(SJQ4_METADATA.get("SJQ4_ID")))
String FileName =  MediaFileAPI.GetFileForSegment(mf,0)
FilePrefix = FileName.substring(0, FileName.lastIndexOf('.'))
FileType= FileName.substring(FileName.lastIndexOf('.')+1, FileName.size())
InputVidFile=FileName
OutputVidFile=FilePrefix+".mp4"
println FileType

if (FileType != "mpg" ) {
println("Recording is not mpg")
   return 1
}

HBCLIPath='"C:\\Program Files\\Handbrake\\HandBrakeCLI.exe"'
HBParams=" --preset iPad -v 0"

HBProc=HBCLIPath+" -i "+InputVidFile+" -o "+OutputVidFile+HBParams
println (HBProc)

def initialSize = 8192
def outStream = new ByteArrayOutputStream(initialSize)
def errStream = new ByteArrayOutputStream(initialSize)

def command=HBProc
def proc=command.execute()
proc.consumeProcessOutput(outStream, errStream)
proc.waitFor()

println "return code: ${ proc.exitValue()}"
println "stderr: ${proc.err.text}"
println "stdout: ${proc.in.text}" // *out* from the external program is *in* for groovy
println 'out:\n' + outStream
println 'err:\n' + errStream
return 0

[wayner]

Slugger and/or Tmiranda -

What is the range of return codes for SJQv4? And what colours do these correspond to in the UI?

Does it make sense to have a "failed" return code for a job that isn't necessary - like when you go to run comskip but there is already a valid .edl file for the recording? What return code should I use for such events?

[Slugger]

Originally Posted by wayner

Slugger and/or Tmiranda -

What is the range of return codes for SJQv4? And what colours do these correspond to in the UI?

Not sure what the STVi will show, but there are four categories of return codes recognized by the SJQ engine:

• The exe's return code fell between the defined min rc and max rc settings for the defined task. In this case, the task is marked as COMPLETED.
• The exe's rc falls outside of the defined min/max settings; task is marked FAILED
• The test script's rc = 1 or rc > 2; the task is set back to WAITING and will be retried later
• The test script's rc = 2; the task is set to SKIPPED and will not be executed or retried

Does it make sense to have a "failed" return code for a job that isn't necessary - like when you go to run comskip but there is already a valid .edl file for the recording? What return code should I use for such events?

For this case, you should have a test script that finds the edl and then returns rc=2, which will mark the task as SKIPPED. Marking it FAILED is also acceptable as it achieves the same end goal, just depends if you care that tasks are marked as failed.

[wayner]

Thanks Slugger - one other question. I have a task that runs when all recordings are complete that checks for bad video files that are caused by a STB being off.

I also run tasks depending on the show being recorded - for kids' shows I move to another folder and encode using handbrake. This is a destructive process in that it deletes the original TV file which will cause the other process to not run correctly.

Is it possible to determine in which order SJQ runs these tasks - in other words how do I know which one gets run first? Is there any way of specifying a priority which will cause a certain task to run first?

[Slugger]

Originally Posted by wayner

Is it possible to determine in which order SJQ runs these tasks - in other words how do I know which one gets run first? Is there any way of specifying a priority which will cause a certain task to run first?

No, all of the things you mention are non-deterministic. If you have two or more tasks that depend on each other you must either synchronize the tasks externally (through lock files or other mechanisms) or (and my preferred/recommended approach) you must combine the tasks into a single task exe script using Groovy. In this case, your groovy script is likely just a light weight wrapper that calls each individual exe, but does so in the proper/required order, etc.

[wayner]

Thanks Slugger, that's what I expected. I will either combine into one "mother" task, or I will run a cron job in the middle of the night that processes all files that were recorded yesterday.

My scripts do basically just do some processing to call EXEs like comskip and handbrake, plus Java/Groovy I/O commands, but you have convinced me to use Groovy as much as possible so I don't even use batch files anymore, everything is called from a Groovy script.

[phelme]

Crap. I'm suddenly getting failures with tasks. An example is the following:

===== Tue Feb 21 12:39:48 PST 2012 =====

Remote SageAPI is not functional.  Can't use server: rmi://x.x.x.x:1098
----- stderr -----

java.lang.RuntimeException: Remote SageAPI is not functional.  Can't use server: rmi://x.x.x.x:1098
	at sagex.remote.rmi.RMISageAPI.(RMISageAPI.java:28)
	at com.google.code.sagetvaddons.sjq.agent.ScriptRunner.main(ScriptRunner.java:154)
Caused by: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: 
	java.io.EOFException
	at sun.rmi.transport.tcp.TCPChannel.createConnection(Unknown Source)
	at sun.rmi.transport.tcp.TCPChannel.newConnection(Unknown Source)
	at sun.rmi.server.UnicastRef.newCall(Unknown Source)
	at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
	at sagex.remote.rmi.RMISageAPI.callService(RMISageAPI.java:36)
	at sagex.remote.rmi.RMISageAPI.callService(RMISageAPI.java:52)
	at sagex.remote.rmi.RMISageAPI.(RMISageAPI.java:25)
	... 1 more
Caused by: java.io.EOFException
	at java.io.DataInputStream.readByte(Unknown Source)
	... 8 more

Tried turning off the firewall and that's not the cause. Restarted the SJQ task client service. Don't think I changed anything else... though I had been messing with the NIC driver settings for SageDCT. Hmmm. What else should I look at?

[Slugger]

sagex-services plugins is no longer functional. Restart/fix it or restart the Sage server.

[phelme]

Originally Posted by Slugger

sagex-services plugins is no longer functional. Restart/fix it or restart the Sage server.

I've tried both restarting the SageTV service and resetting the sagex-services plugin and it's still failing. Do I need to uninstall the plugin itself? That will be a little bit of a pain because of the dependencies it has.

The jetty stuff is still functioning if that's any help.

[Slugger]

Not sure why the RMI service would suddenly stop working. You might want to track down stuckless and see if he has any thoughts (sagex-services is his plugin). My initial guess is that something else is latching onto port 1098? Only a guess though.