Plugin: Sage Job Queue (SJQ)

[scott2020]

Got it. I read that and this whole thread several times but I didn't put two and two together! My bad. I think for now I'll just leave well enough alone and be happy it is running, and tackle that another day. thanks!

[edbmdave]

Originally Posted by lovingHDTV

Could someone show me how to set the File Mangement options to delete *.edl, *.txt *.log files if there are no *.mpg matching files?

thanks
dave

Is there a way to do this in 2.0 yet? I would also like to be able to cleanup the files after the mpg or mp4 files are deleted.

[Slugger]

Originally Posted by edbmdave

Is there a way to do this in 2.0 yet? I would also like to be able to cleanup the files after the mpg or mp4 files are deleted.

The file cleaner is in the v2.0.0 official release (and all snapshots that have followed it). The file cleaner runs periodically and will delete the specified files the first time it's run after the corresponding mpg/mp4/etc. is deleted. Though it doesn't do it immediately, it does do it eventually and does it effectively to the point that you don't need to worry about those files clogging up your recording dir - just setup the file cleaner and forget about it. Everytime the file cleaner deletes a file it records what it deleted in the server logs for your review.

[scott2020]

I have a couple of questions. First, I tried getting the latest SVN build from the Google code site, and the link goes to battams.ca I believe, but the link didn't work for me. I am having a problem with jobs being stuck on STARTED on the client and the only way to get things flowing again is to edit the file in the CONTEXT directory.

Second, I have the job set to not do anything with LiveTV. However, it looks like when someone is watching live TV and the program changes at the top of the hour for example and moves onto the next program, a new MPG file is created for the new program. However, the old LiveTV file is still there for some period of time and SJQ picks it up for processing. Should I have a delay or something so the old LiveTV files aren't processed before Sage removes them?

Scott

[edbmdave]

how do you seperate the file extensions?

say I want to delete any txt, VPRJ or EDL files if neither a MP4 or and MPG exists?

(I don't want to goof this up).

[Slugger]

Originally Posted by edbmdave

how do you seperate the file extensions?

say I want to delete any txt, VPRJ or EDL files if neither a MP4 or and MPG exists?

(I don't want to goof this up).

Delete files with this extension: txt|vprj|edl
Where there is no matching file with this extension: mp4|mpg

The values are case sensitive...

[Slugger]

Originally Posted by scott2020

I have a couple of questions. First, I tried getting the latest SVN build from the Google code site, and the link goes to battams.ca I believe, but the link didn't work for me. I am having a problem with jobs being stuck on STARTED on the client and the only way to get things flowing again is to edit the file in the CONTEXT directory.

Please try to download a snapshot build again - there was a power outage earlier this morning that lasted longer than my UPS' power supply so my server shutdown and when everything came back up there was a problem with my gateway device, blah, blah, blah. Servers should be back online now...

For tasks to get stuck in started state usually means the task client has crashed. Try the latest snapshot and let me know.

Second, I have the job set to not do anything with LiveTV. However, it looks like when someone is watching live TV and the program changes at the top of the hour for example and moves onto the next program, a new MPG file is created for the new program. However, the old LiveTV file is still there for some period of time and SJQ picks it up for processing. Should I have a delay or something so the old LiveTV files aren't processed before Sage removes them?

Scott

Will need to see your server ruleset. To not process live tv you should be adding a test to your rules like this:

IsScheduledRecording == true

[scott2020]

I can get to the snapshots now. Would I just download and unzip them over top of what I already have?

Below is the code. I just added the IsScheduledRecording option you mentioned above.

if [IsTV == "true" && IsActivelyRecording == "false" && IsScheduledRecording == "false" && FileExists == "%d%\\%p%.mpg" && FileExists != "%d%\\%p%.edl" && ChannelName != "KMOSDT" && ChannelNumber != "1"]
{
COMSKIPMPEG
}

thank you!

[Slugger]

Originally Posted by scott2020

I can get to the snapshots now. Would I just download and unzip them over top of what I already have?

Below is the code. I just added the IsScheduledRecording option you mentioned above.

if [IsTV == "true" && IsActivelyRecording == "false" && IsScheduledRecording == "false" && FileExists == "%d%\\%p%.mpg" && FileExists != "%d%\\%p%.edl" && ChannelName != "KMOSDT" && ChannelNumber != "1"]
{
COMSKIPMPEG
}

thank you!

Yeah, just unzip as per the upgrade instructions in the install wiki doc. Also, note that I incorrectly said IsScheduledRecording == false, it should be true since you only want to process scheduled recordings (and not live tv). I edited the original post and switched it to true for others who come across the post. Be sure to change it to true or the rule will only queue up live recordings.

[scott2020]

Downloaded and upgraded everything. All is running smoothly. Thanks for your help!

[CollinR]

Need a little help, just upgraded from 1.5 and am slowly geeting it figured out.

Server Rule

//ComSkipSD
if [IsTV == "true" && IsHD == "false" && FileExists == "%d%\\%p%.mpg" && FileExists != "%d%\\%p%.edl" && IsActivelyRecording == "false" && ChannelNumber != "0|1|1400"]
{
COMSKIPSD
}

That seems to be generating tasks in the queue so I assume no troubles there.

Client Task

//Client options
MAXPROCS 3:
// TASK definition
COMSKIPSD {  
:CPU LOW
:MAX 2     
"C:\\MediaServer\\Comskip\\comskip.exe -n \"%c%\""
}

Client Log shows

Date: Thu Jan 15 11:44:24 CST 2009

HTTP ERROR: 500
Expected '{' to start task definition on line 2
RequestURI=/sjq/SJQ

Caused by:
java.lang.RuntimeException: Expected '{' to start task definition on line 2
	at com.google.code.sagetvaddons.sjq.server.ClientParser.(Unknown Source)
	at com.google.code.sagetvaddons.sjq.server.SJQServlet.getPopCandidates(Unknown Source)
	at com.google.code.sagetvaddons.sjq.server.SJQServlet.doPost(Unknown Source)
	at com.google.code.sagetvaddons.sjq.server.SJQServlet.doGet(Unknown Source)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:362)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:729)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
	at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:206)
	at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:320)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:505)
	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:829)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:513)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:211)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:380)
	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:395)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:488)

I assume once I get one task running I can figure out the rest. lol

[Slugger]

Originally Posted by CollinR

Need a little help, just upgraded from 1.5 and am slowly geeting it figured out.

Client Task

//Client options
MAXPROCS 3:
// TASK definition
COMSKIPSD {  
:CPU LOW
:MAX 2     
"C:\\MediaServer\\Comskip\\comskip.exe -n \"%c%\""
}

Try :MAXPROCS 3 instead of MAXPROCS 3: (colon is out of place).

[Slugger]

SJQ v2.1.0 is now available. This release adds a new internal task for auto rescheduling of recordings and also includes various bug fixes. It is recommended that all SJQ users upgrade to v2.1.0.

[perfessor101]

Hello, I removed SJQ 1.5 and have been trying to get SJQ 2.1 working today.

TimeUntilNextRecording seems to take fixed times now (30 seconds, 30 minutes, 30 days etc) ... urp ... shoulda helped out with the debugging ;-) or can I still use it as a percentage of the recording time?

The Jetty plugin seems to be opening random ports via SageTV (Firewall is asking me to ok two new ports for SageTV each time I run it with jetty installed) and I have two cores running maxed out (out of four) the full time SageTV is running. I have also found SageTV will no longer close.

I'll be going over the install information from start to finish again tonight when I get up (after a few times already)

What I did was ...

Removing SJQ 1.5
01 stop SageTV
02 removed a few files:
   sagetv/webserver/webroot/sjqweb/*.*
   slugger_sage-job-queue.jar
   Jars/json.jar
   Jars/sqlitejdbc.jar
   the rest weren't there for me
03 removed the reference from the webserver\servlets.properties to sjq
04 removed net.sourceforge.sagetv.sjq.SageJobQueue from start runnable classes in the sage.properties file

Installing Jetty
05 downloaded jettystarter_1_4.zip
06 extracted to my SageTV folder c:\program files\SageTV\SageTV
07 Added sagex.jetty.starter.Main to the line load_at_startup_runnable_classes in my sage.properties file
08 In JettyStarter.properties I set jetty.port=8090 (don't know why that port number) and set DEBUG=true removing the '#'

Installing SJQ 2.1 Server
09 Downloaded the latest files 
    sjqc-ntsvc-2.1.0.181.zip - windows service client
    sjq-2.1.0.176.zip           - server
10 extracted sjq-2.1.0.176.zip to c:\program files\SageTV\SageTV
11 logged in to SJQ at http://192.168.0.34:8090/sjq/SJQ with 'sjqadmin'

Installing SJQ 2.1 Client
10 extracted sjqc-ntsvc-2.1.0.181.zip to c:\sjqc (not very inventive this morning)
11 In C:\sjqc\sjqc.properties I entered :
    host=localhost
    port=8090
    is_ssl=false
12 from the command line in the c:\sjqc directory I ran 'bin\InstallApp-NT.bat'
13 under administrative tools I selected services and started 'SJQ Task Client for Windows'

Created a Rule on the Server
14 Created a rule to run VideoRedo Quick Stream fix on files
    
Created a Task on the Client
15 Created a task to run VideoRedo Quick Stream fix on files

If you can see what my lack of sleep has contributed to me missing I would greatly appreciate it ...

Thanks for your time,

and thanks for a great plugin,

Bobby

[Slugger]

What you've written sounds exactly right on as far as removing SJQ 1.x and installing SJQ 2.x. The 100% CPU on two cores obviously doesn't sound right, but nothing in your checklist jumps out at me nor does anything appear to be missing. Remove SJQ from Jetty and see if Jetty shows the same problems, if so then I'd suggest heading over to the Jetty plugin thread for support. If things only start to act up when you install/enable SJQ then that might suggest it's my department and perhaps you've found a bug.

As for the TimeUntilNextRecording test, it only takes static values for comparison. I assume it used to take a percentage of the recording length in 1.x (I don't have the source for the 1.x stream around anymore)? Feel free to open an issue ticket and I can probably add that back in (or more likely a new test) in an upcoming version.

[perfessor101]

Hello,
I removed the SJQ files from jetty\contexts\sjq.context.xml and jetty\webapps\war ... and after restarting SageTV my cpu utilization dropped to normal and SageTV will now quit.

I guess I'll try loading an older version of SJQ 2.X server and see what happens there just in case it's a problem with something on my system.

Even with SJQ 2.0.0 154 I get high CPU utilization and currently I don't seem to be able to get past the login prompt as well and SageTV doesn't want to quit in this release either.

C:\Documents and Settings\SageTV>java -version
java version "1.6.0_11"
Java(TM) SE Runtime Environment (build 1.6.0_11-b03)
Java HotSpot(TM) Client VM (build 11.0-b16, mixed mode, sharing)

I don't know if it's just Windows 2005 Media Center Edition or ...

if there are any logs I can post ...

Bobby

[Slugger]

Originally Posted by perfessor101

Hello,
I removed the SJQ files from jetty\contexts\sjq.context.xml and jetty\webapps\war ... and after restarting SageTV my cpu utilization dropped to normal and SageTV will now quit.

I guess I'll try loading an older version of SJQ 2.X server and see what happens there just in case it's a problem with something on my system.

Even with SJQ 2.0.0 154 I get high CPU utilization and currently I don't seem to be able to get past the login prompt as well and SageTV doesn't want to quit in this release either.

C:\Documents and Settings\SageTV>java -version
java version "1.6.0_11"
Java(TM) SE Runtime Environment (build 1.6.0_11-b03)
Java HotSpot(TM) Client VM (build 11.0-b16, mixed mode, sharing)

I don't know if it's just Windows 2005 Media Center Edition or ...

if there are any logs I can post ...

Bobby

Stop SJQ, backup/rename the sjq.sqlite database file in \Program Files\SageTV\SageTV\ so a fresh one will be created at restart then restart SJQ without configuring the server ruleset nor the client configuration. Do the symptoms continue? Seems like you might have stumbled across a bug in one of the parsers in SJQ.

[perfessor101]

SageTV Cpu utilization is now down to normal ... I looked at the size of sjq.sqlite and it was 275,009KB ... there might have been a possibility for a parsing error in there ;-)

Thank you ...

When I get up tonight I'll continue with configuring the rules and tasks ...

Thanks for you time and effort,
Bobby

[Slugger]

I believe the parsing error is with the ruleset and/or client config parser (the ruleset and client configs are stored in the database file). If you encounter the same symptoms please post the contents of your ruleset and client config(s) so I can (hopefully) reproduce the symptoms here and squash the bug with my parser code.

Alternatively, if you kept the original sjq.sqlite db file then I'd love to get my hands on it. PM me if you're able to send that original db file (the one causing the heavy CPU usage) to me.

[perfessor101]

Hello,
I seem to have things working fairly well ... just learning to leave it running ;-) and quit playing with it.

I have three rules and tasks defined (VRDQSF, VRDCUT and SA) which work perfectly on most recordings (complete - not split).

On Split Recordings It would seem it's running the task on the first part of a split recording (*-0.mpg) twice and not getting to the second or other parts.

It also doesn't seem to want to pick up the next stages for processing on the split recordings. (It may be detecting *-1.vrdqsf file doesn't exist and so the rule fails for the second part of the recording)


After a Video Redo Quick Stream Fix ...
I run Show Analyzer ...
and then after confirming commercial cuts ...
I run Video Redo to cut the commercials ...
move it to an unRaid Array (soon)
convert them to xvid using EP's plugin using autocrop (soon).

Sun Jan 18 09:58:53 2009 -- Starting VideoRedo Quick Stream Fix \\sagetvserver\tv4\HotelBabylon-6001862-0.mpg 

Sun Jan 18 10:00:36 2009 -- Finished QUick Stream Fix \\sagetvserver\tv4\HotelBabylon-6001862-0.mpg 

Sun Jan 18 10:00:50 2009 -- Starting VideoRedo Quick Stream Fix \\sagetvserver\tv4\HotelBabylon-6001862-0.mpg 

Sun Jan 18 10:02:38 2009 -- Finished QUick Stream Fix \\sagetvserver\tv4\HotelBabylon-6001862-0.mpg 

Sun Jan 18 10:03:53 2009 -- Starting VideoRedo Quick Stream Fix \\sagetvserver\tv4\FlashGordon-Blame-6002433-0.mpg 

Sun Jan 18 10:05:33 2009 -- Finished QUick Stream Fix \\sagetvserver\tv4\FlashGordon-Blame-6002433-0.mpg 

Sun Jan 18 10:05:47 2009 -- Starting VideoRedo Quick Stream Fix \\sagetvserver\tv4\FlashGordon-Blame-6002433-0.mpg 

Sun Jan 18 10:07:31 2009 -- Finished QUick Stream Fix \\sagetvserver\tv4\FlashGordon-Blame-6002433-0.mpg 

Sun Jan 18 10:08:53 2009 -- Starting VideoRedo Quick Stream Fix \\sagetvserver\tv4\Crusoe-Hour11TheHuntingParty-6000420-0.mpg 

Sun Jan 18 10:11:24 2009 -- Finished QUick Stream Fix \\sagetvserver\tv4\Crusoe-Hour11TheHuntingParty-6000420-0.mpg 

Sun Jan 18 10:11:38 2009 -- Starting VideoRedo Quick Stream Fix \\sagetvserver\tv4\Crusoe-Hour11TheHuntingParty-6000420-0.mpg 

Sun Jan 18 10:14:13 2009 -- Finished QUick Stream Fix \\sagetvserver\tv4\Crusoe-Hour11TheHuntingParty-6000420-0.mpg

My rules are

// Rule Definitions

// VideoRedo Quick Stream Fix
if [IsTV == "true" && IsHD == "false" && FileExists == "%d%\\%p%.mpg" && FileExists != "%d%\\%p%.done" && FileExists != "%d%\\%p%.working" && FileExists != "%d%\\%p%.vrdqsf" && IsActivelyRecording == "false"]
{
:PRIORITY 2000
VRDQSF
}

// Show Analyzer
if [IsTV == "true" && IsHD == "false" && FileExists == "%d%\\%p%.mpg" && FileExists != "%d%\\%p%.done" && FileExists != "%d%\\%p%.working" && FileExists == "%d%\\%p%.vrdqsf" && FileExists != "%d%\\%p%.vprj" && FileExists != "%d%\\%p%.Vprj"]
{
:PRIORITY 1900
SA
}

// VideoRedo Cut
if [IsTV == "true" && IsHD == "false" && FileExists == "%d%\\%p%.mpg" && FileExists != "%d%\\%p%.done" && FileExists != "%d%\\%p%.working" && FileExists == "%d%\\%p%.slice" && FileExists != "%d%\\%p%.cut"]
{
:PRIORITY 1800
VRDCUT
}

My Client Options are

//Client options
:MAXPROCS 1

// TASK definitions

// VideoRedo Quick Stream Fix
VRDQSF {  
:CPU LOW
:MAX 1     
"C:\\testing\\vrdqsf.bat \"%c%\""
}

// Show Analyzer
SA {  
:CPU LOW
:MAX 1     
"C:\\testing\\sa.bat \"%c%\""
}

// VideoRedo Quick Stream Cut
VRDCUT {  
:CPU LOW
:MAX 1     
"C:\\testing\\vrdcut.bat \"%c%\""
}

Thanks for your time,
Bobby

ps -- I still have the sjq.sqllite file that was confusing the parser