Plugin: Sage Job Queue (SJQ)

[Slugger]

Originally Posted by nyplayer

Slugger,

I create mpg.properties and mpg.xml files for all my recordings. I would like to cleanup these files. I have it setup as below.... but it deletes the properties files even if an mpg avi etc... exists?

Delete files with an extension matching this regex
vprj|edl|my|mpg.properties|avi.properties|mkv.properties|mp4.properties

When no corresponding file has an extension matching this regex:
mpg|avi|mkv|mp4

So let's say you have TheSimpsons.mpg recorded. What is the full name of the mpg.properties file created?

[Wheemer]

My problem was I had downloaded the wrong snapshot. SSL instead of normal. Thanks Slugger.

[nyplayer]

Originally Posted by Slugger

So let's say you have TheSimpsons.mpg recorded. What is the full name of the mpg.properties file created?

TheSimpsons.mpg.properties

[Slugger]

Originally Posted by nyplayer

TheSimpsons.mpg.properties

Yeah, that's what I thought. The way it works (right now) is that the FileCleaner only looks at file extensions. So it trims the file name down to everything after the last '.' then goes and compares to the settings. So it's never going to work the way you want it to. On the other hand instead of TheSimpsons.mpg.properties, if you could name the file TheSimpsons.properties then you'd be set. Then just tell FileCleaner to delete properties files when there's no mpg.

[nyplayer]

Slugger,

Thanks.. I will use Dirmon2 because that is the way Sage properties files are also the xml files created are the same way.

[Slugger]

Now that everything appears to be stable again (unless someone else tells me otherwise), I plan on marking 414 as the official v2.1.3 release. I've also managed to complete the User's Guide for v2.1.3. My original plan was to convert it to PDF and include it in the downloads section, but after reviewing the exported PDF, the formatting isn't quite perfect and it might be easier to just reference the online version. If you think a downloadable PDF would be useful just let me know and I can create it as it's not too much effort.

So unless I hear reports from someone, I will mark 414 as the official v2.1.3 probably on Sunday.

Thanks to everyone who helped with testing during this cycle. With your help I was able to identify and squash many bugs that affected SJQ server stability.

[Slugger]

Official v2.1.3 Builds Released

29 Mar 2009: The official server and task client builds for SJQ v2.1.3 are now available for download in the project downloads section.

The v2.1.3 release is a bug fix release that addresses various issues with the server and task client packages:

Server Fixes

• Fix various thread synchronization issues; v2.1.2 and older contain many problems that can lead to deadlock between various SJQ execution threads; ALL SJQ USERS ARE ENCOURAGED TO UPGRADE TO v2.1.3; though most of these issues have existed in the code since v2.0.0, users only started to experience deadlock with v2.1.2 so it is recommended that users stay away from v2.1.2 and if you're running it then move to v2.1.3.414 (or newer)
• Issue 57 : Fix case where client logs never purge regardless of settings
• Issue 58 : Fix task log purging so it honours user's settings
• Stop log viewers from loading the server data twice on initial object creation
• Add thread dumper for assisting with certain debug scenarios (see changelog for r378)
• Upgrade JDBC driver to latest available
• Cleanup finalize() methods in DataStore and Logger; cleanup flush() in Logger
• Rework logging facilities to support various thread changes
• Issue 59 : Properly purge task logs, including orphaned log entries
• Issue 60 : Security fix: Only accept server commands from an authenticated web session
• Ensure the TaskQueue is not manipulated by other threads while the QueueLoader is running
• Attempt to fill in the QueueLoader menu items in GUI only after the web session has been authenticated successfully
• Issue 56 : Return of the "Go Back" and "Clear" buttons on log viewer panels
• Issue 61 : Properly catch invalid regex compilations, log them, and report errors in FileCleaner
• Resize scroll panels when browser window is resized
• Issue 55 : Honour sorting of grid columns (where appropriate - not all cols can be sorted)
• Delay spawning of cleanup threads until after the QueueLoader is done (as opposed to before the QueueLoader starts) to minimize risk of database contention during SageTV startup

Task Client Fixes

• Address thread synchronization issues when passing JSONObject instances between threads

[HokiePerogi]

I installed SJQ on my headless server yesterday. I'm having a heck of a time getting a ruleset to run which contains a time constraint. Here is my ruleset:

if [IsTV == "true" && IsActivelyRecording == "false" && IsWatched == "false" && FileExists == "%d%/%p%.edl" && Title == "Law & Order" && (TimeOfDay < 20:00 || TimeOfDay > 22:00)]
{
   :TRANSCODEOPTS "iPhone-Widescreen" "true"
   _TRANSCODE
}

I get the following in the server log when forcing a run on the queueloader:

2009/04/01 13:02:14.884	QueueLoader awakened: sleep interrupted
2009/04/01 13:02:14.959	Application exception:
java.lang.RuntimeException: Expected time input in form of hh:mm:ss or hh:mm; 24-hour format
	at com.google.code.sagetvaddons.sjq.server.FieldTime.run(FieldTime.java:44)
	at com.google.code.sagetvaddons.sjq.server.FieldTimeOfDay.run(FieldTimeOfDay.java:31)
	at com.google.code.sagetvaddons.sjq.server.RulesParser$Evaluator.evalTest(RulesParser.java:104)
	at com.google.code.sagetvaddons.sjq.server.RulesParser$Evaluator.<init>(RulesParser.java:51)
	at com.google.code.sagetvaddons.sjq.server.RulesParser$Evaluator.evalTest(RulesParser.java:79)
	at com.google.code.sagetvaddons.sjq.server.RulesParser$Evaluator.<init>(RulesParser.java:57)
	at com.google.code.sagetvaddons.sjq.server.RulesParser$Evaluator.<init>(RulesParser.java:69)
	at com.google.code.sagetvaddons.sjq.server.RulesParser.loadQ(RulesParser.java:145)
	at com.google.code.sagetvaddons.sjq.server.RulesParser.<init>(RulesParser.java:119)
	at com.google.code.sagetvaddons.sjq.server.QueueLoader.run(QueueLoader.java:68)
	at java.lang.Thread.run(Unknown Source)

2009/04/01 13:02:15.116	LogCleaner instance started
2009/04/01 13:02:15.119	FileCleaner instance started
2009/04/01 13:02:15.124	FileCleaner instance finished [5ms]
2009/04/01 13:02:15.191	LogCleaner instance finished [74ms]

What am I doing wrong? I've tried both HH:MM:SS and HH:MM. In fact, I've copied the exact examples from the SJQ v2.1.3 User's guide and get the same error.

I'm running SJQ v2.1.3.414.

Thanks.

[Slugger]

Double quote the times. Examples in user guide updated...

[HokiePerogi]

Ta-da....and now it works. Thanks for the quick reply!

[HokiePerogi]

I think my SJQ is freezing every so often. I check this morning to make sure one of my jobs ran correctly, but the server log shows the last activity was: 2009/04/03 01:05:41.136 LogCleaner instance finished [131ms]

It's current 12:30 pm here. The time in Vista is correctly set. I have SJQ set with a maximum queue loader sleep time between runs of 30 minutes and to be run at the end of each schedule recording.

Restarting the SageTV service help correct the issue and gets SJQ processing again. Any suggestions on how I can debug this?

[phelme]

Never mind. Browser cache problem!

[Slugger]

Originally Posted by phelme

Hmm. I upgraded to 2.1.3.414 (from 2.1.2) and now I'm getting "NotAuthorized" inside my ruleset editor. And all my task history is empty. My client doesn't show up in the queue anymore either. Inside the FileCleaner, I'm again seeing "NotAuthorized" inside the first text box. This can't be good.

This is what I get from the debug dump:

Fri 4/3 10:00:03.121 Fri Apr 03 10:00:03 PDT 2009: SJQ Application Dump
QueueLoader thread is alive...
java.lang.Thread.sleep(Native Method)
com.google.code.sagetvaddons.sjq.server.QueueLoader.run(QueueLoader.java:96)
java.lang.Thread.run(Unknown Source)
ClientMonitor thread is alive...
java.lang.Thread.sleep(Native Method)
com.google.code.sagetvaddons.sjq.server.ClientMonitor.run(ClientMonitor.java:34)
java.lang.Thread.run(Unknown Source)
InteralTaskClient thread is alive...
java.lang.Thread.sleep(Native Method)
com.google.code.sagetvaddons.sjq.server.InternalTaskClient.run(InternalTaskClient.java:49)
java.lang.Thread.run(Unknown Source)
Most recent LogCleaner thread is dead (this is normal behaviour)!
Most recent FileCleaner thread is dead (this is normal behaviour)!


Not going to be happy if all my rules are gone.

Fear not, nothing's gone. That just means the web browser isn't authenticated to the SJQ server. Reload the GUI (might have to clear your browser cache to ensure the latest version of the GUI loads up; hold shift while clicking refresh [in Firefox] or hold Ctrl while clicking refresh [in IE]). Enter your password to login... all should now be good.

[Slugger]

Originally Posted by HokiePerogi

I think my SJQ is freezing every so often. I check this morning to make sure one of my jobs ran correctly, but the server log shows the last activity was: 2009/04/03 01:05:41.136 LogCleaner instance finished [131ms]

It's current 12:30 pm here. The time in Vista is correctly set. I have SJQ set with a maximum queue loader sleep time between runs of 30 minutes and to be run at the end of each schedule recording.

Restarting the SageTV service help correct the issue and gets SJQ processing again. Any suggestions on how I can debug this?

Next time this happens, get a thread dump of the SJQ server for me by following the directions in this post. Also other details are helpful such as:

• Is SageTV frozen as well? If not, is SageTV's performance affected by SJQ's problems?
• When SJQ encounters a problem has your CPU load spiked to a constant 100% (or 100% of a core for multi-core systems)?
• Ensure SageTV and Jetty debug logging are enabled. Are there any SJQ error messages in the logs?

[phelme]

Originally Posted by Slugger

Fear not, nothing's gone. That just means the web browser isn't authenticated to the SJQ server. Reload the GUI (might have to clear your browser cache to ensure the latest version of the GUI loads up; hold shift while clicking refresh [in Firefox] or hold Ctrl while clicking refresh [in IE]). Enter your password to login... all should now be good.

yep, got it. thank you!!

[nyplayer]

Slugger,

Why does the showanalyzer always run before the _UNMANUALREC?.....

2078679 See Jane Date SHOWANALYZER RUNNING 97 Apr 5, 2009 3:10:58 PM 192.168.2.170
2078679 See Jane Date _UNMANUALREC WAITING 99 Apr 5, 2009 3:10:58 PM

if [IsTV == "true" && IsScheduledRecording == "true" && Filename =$ ".mpg" && TranscodeQueueSize < "1"]{
    :PRIORITY 99
      _UNMANUALREC
    :PRIORITY 97
      SHOWANALYZER
}

[Slugger]

Details here. To summarize, internal tasks only run when there are no external tasks queued or running for the media object. Why? You likely want to process all your external scripts against a media file before flipping flags/deleting/etc. the media object within SageTV. Mainly it's a safety feature to prevent the case where you have an external task and the _DELETE task. If the delete task runs before the external task then the media object will not exist when the external task is assigned to a client. Do you have a specific use case where you'd want/need to run the internal task before the external one?

[nyplayer]

Slugger,

I have a rule that compresses any shows that I make a manual recording. What is happening is that If anyone sets up a timed recording it is marked as manual then it gets compressed and moved to my keep folder .I would like to clear this flag as soon as the recording starts. What was happening is that the 2nd rule below runs before the _UNMANUALREC


No problem I have a work around... I guess I just missed that part of the doc. I just added a mediage 1d to the 2nd rule below

//ManualRec Title
if [IsTV == "true" && IsActivelyRecording == "false" && Filename =$ ".mpg" && (Title =^ "Star Trek" || (Title == "Smallville" || (Title == "Tales From the Darkside" || (Title == "The Outer Limits" || (Title == "Charmed" || (Title == "Jake 2.0" || (Title == "Law & Order: Criminal Intent" || (Title == "Planet Earth" || (Title == "CSI: Crime Scene Investigation")))))))))]{
    :PRIORITY 50
      _MANUALREC
}

//1Pass HQ Title Manual
if [IsTV == "true" && IsActivelyRecording == "false" && Filename =$ ".mpg" && TranscodeQueueSize < "1" && (IsManualRecord == "true" || (IsArchived == "true"))]{
    :PRIORITY 1
      PASS1XVID
}

[nyplayer]

Slugger,
Something is not working right the Internal tasks are not running if any task is running. If there is no regular task running for a particular show why don't the internal tasks run for that show? If I have a sporting event etc... running showanalyzer than no transcodes or other internal tasks run for 3 to 4 hrs. Did this recently change it used to work fine before?


2078679 See Jane Date SHOWANALYZER Apr 5, 2009 3:10:58 PM Apr 5, 2009 5:03:18 PM 192.168.2.170

queloader ran at 5:20 and 5:34

2078777 NBA Basketball SHOWANALYZER RUNNING 97 Apr 5, 2009 4:10:58 PM 192.168.99.1
2078679 See Jane Date _UNMANUALREC WAITING 99 Apr 5, 2009 3:10:58 PM
2078777 NBA Basketball _UNMANUALREC WAITING 99 Apr 5, 2009 4:10:58 PM
2078678 Candy Girls _TRANSCODE WAITING 25 Apr 5, 2009 4:10:58 PM

[Slugger]

Did those internal tasks eventually run (without the need to restart SJQ)?