SageTV Community Plugin: Sage Job Queue (SJQ)

Notices

 SageTV Customizations This forums is for discussing and sharing user-created modifications for the SageTV application created by using the SageTV Studio or through the use of external plugins. Use this forum to discuss customizations for SageTV version 6 and earlier, or for the SageTV3 UI.

#1
05-02-2007, 07:36 PM
 Slugger SageTVaholic Join Date: Mar 2007 Location: Kingston, ON Posts: 4,008
Plugin: Sage Job Queue (SJQ)

[This overview is pasted from my online document. Please see the online doc for full details including system requirements and installation. You can download the plugin from here.]

Overview
Sage Job Queue (SJQ) is a loadable class plugin for the SageTV PVR system. With SJQ, you can selectively process your media files based on the contents of the file as stored in the SageTV database. For example, you can setup a ruleset that runs your commercial flagging programs on user scheduled programs only (i.e. don't commercial flag live tv recordings or IR based recordings). You can also specify rules that ignore commercial flagging for recordings from specific channels (say premium channels that don't have commercials). You can also ignore recordings of sports programming (based on program title), etc.

The example above was my motivation for this plugin. I wanted a way to ensure that live tv programs weren't being scanned, I also wanted PPV programs and other channels ignored as well.

Changelog
Code:
v0.0.1 -> v0.1.0
================

* Added new field type: Time
+ Format is: HH:MM:SS, 24 hour format
+ Valid operators: >, <, =
* Added new time field for comparisons:
+ TimeOfDay: Compares the current time of day with the given string; use for restricting time that SJQ will do specific jobs
* Added new boolean fields for comparison:
+ IsClientConnected: Returns true if there is at least one client connected to the SageTV server
+ IsViewingMedia: Returns true if media is being viewed on the Sage instance from which SJQ is being run (see online docs for further details)
* Added more details to SJQ database for completed jobs
+ For each completed action SJQ now records the exe that was run, the file(s) the action was run against and the time the action completed
+ varchar field 'exe' added to recordings table
+ New table 'files' created; records full path of the file and the time the action finished for a given media file id
* Added new table to SJQ database: settings
+ Holds various internal settings, including the schema version of the SJQ database, which is used for upgrading purposes

Last edited by Slugger; 02-11-2009 at 02:09 PM. Reason: New release of plugin
#2
05-08-2007, 07:59 PM
 mayamaniac Sage Icon Join Date: May 2004 Posts: 2,177
of course, obvious request is to make a UI for this plugin so it is configurable within the SageTV menus. This has good potential to be handy for a lot of comskip users.

Unrelated but just curious, what comskip version are you using?
__________________
Mayamaniac

- SageTV 7.1.9 Server. Win7 32bit in VMWare Fusion. HDHR (FiOS Coax). HDHR Prime 3 Tuners (FiOS Cable Card). Gemstone theme.
- SageTV HD300 - HDMI 1080p Samsung 75" LED.
#3
05-09-2007, 06:15 PM
 Slugger SageTVaholic Join Date: Mar 2007 Location: Kingston, ON Posts: 4,008
It's unlikely I'd ever create an STVi for this plugin. I personally think it's best configured from the sjq.rules text file. If anything, I may create a servlet to add to the web server plugin to allow configuration from there, but that would be some time down the road. I further hesitate to create an STVi because I don't want to support both the default STV and SageMC because I only use SageMC.

I've already hit this problem (multiple STV support) with my SRE plugin. I originally wrote the STVi for the default STV then a week later I made the switch to SageMC. Seeing as how these plugins (SRE and SJQ) don't need to tie into the STV at all to function, I think my time's better spent creating servlets for the web server plugin. The servlet approach interests me even more because I need not maintain multiple versions for each STV.

As to which comskip app I use, I'm currently using ShowAnalyzer.
#4
05-14-2007, 02:44 PM
 deria Sage Aficionado Join Date: Nov 2005 Posts: 408
This is very impressive. Up until now, I've been using the (also impressive) Cmmerical Skip Monitor.

After reading the documentation, it LOOKS like I should be able to make a rule that only runs between 11:00 pm and 9:00 am, by doing something like:

# RuleSet: Do commercial skip processing between 11:00 pm and 9:00 am, on scheduled TV recordings only,
# with names that end in .mpg, that are not currently recording. Also, omit recordings from the
# following channels: 211, 212

= IsTV: true # Only run SA against recorded TV
= IsScheduledRecording: true # Ignore live tv and IR recordings
$Filename: .mpg # Only run SA against MPEG2 recordings = IsActivelyRecording: false # Wait until program is finished before scanning !% ChannelNumber: 211|212 # Ignore recordings from PPV channels and Movie Network; notice use of ! operator > TimeOfDay: 23:00:00 # Time must be after 23:00. < TimeOfDay: 23:59:59 # Time must be before 23:59:59. COMMFLAG "C:\Program Files\comskip\comskip.exe" = IsTV: true # Only run SA against recorded TV = IsScheduledRecording: true # Ignore live tv and IR recordings$ Filename: .mpg # Only run SA against MPEG2 recordings
= IsActivelyRecording: false # Wait until program is finished before scanning
!% ChannelNumber: 211|212 # Ignore recordings from PPV channels and Movie Network; notice use of ! operator
< TimeOfDay: 09:00:00 # Time must be before 9:00.
> TimeOfDay: 00:00:00 # Time must be after midnight.
COMMFLAG "C:\Program Files\comskip\comskip.exe"

I included two rules because you don't seem to have a "between" operator. At first, I was going to have a > and a < in the same rule, but I suspect that wouldn't work becasue at 23:50 the time would only satisfy one of the conditions (which would make the whole rule fail).

Does what I've planned here make sense?

Anyway, nice idea and nice implementation. I'm planning on giving this one a try this afternoon.

EDIT: forum killed my formatting. oh well, its still relatively legible

EDIT #2: I had a logic error in my ruleset. Each rule would need to have a < and > time, otherwise the commercials could get marked twice. Not the end of the world, but why make the machien work harder then it has to?

Last edited by deria; 05-14-2007 at 03:11 PM.
#5
05-14-2007, 04:20 PM
 Slugger SageTVaholic Join Date: Mar 2007 Location: Kingston, ON Posts: 4,008
Your rule sets look good based on what you say you want them to do. Obviously, you'll know better once you try to run it. The key is to look in the generated sjq.log file if things aren't happening the way you expected. I think you'll find the log does a pretty good job of telling you why an action did or did not execute. Just remember that everything the log tells you is 0-indexed so if the log says an action didn't execute because rule 0.3 returned false then look at the 4th test of your first rule.

As to the TimeOfDay issue, I'll definitely look at addressing that in the next version. As you pointed out, running an action between 11pm and 9am requires two different rules. However, if you modified it to run between midnight and 9am then you could do it all in one rule.
#6
05-14-2007, 05:02 PM
 deria Sage Aficionado Join Date: Nov 2005 Posts: 408
Quote:
 Originally Posted by Slugger Your rule sets look good based on what you say you want them to do. Obviously, you'll know better once you try to run it. The key is to look in the generated sjq.log file if things aren't happening the way you expected. I think you'll find the log does a pretty good job of telling you why an action did or did not execute. Just remember that everything the log tells you is 0-indexed so if the log says an action didn't execute because rule 0.3 returned false then look at the 4th test of your first rule. As to the TimeOfDay issue, I'll definitely look at addressing that in the next version. As you pointed out, running an action between 11pm and 9am requires two different rules. However, if you modified it to run between midnight and 9am then you could do it all in one rule.
I spent a little bit of time working on this, and so far it seems to be working (well, I see one instance of comskip running -- its not using any cpu time apparently, so I might not have set comskip up right, which is entirely possible since its a brand new installation of it). I do have some things I should pass along though:

1. In the installation instructions, its not clear what you mean by adding the class as a loadable class. I figured out that you meant that it should be added to the load_at_startup_runnable_classes line after some poking around though.

2. Under Vista, if you open the log in notepad while SJQ is doing its thing, it seems to crash (ie: the sage UI vanished on me). At first i thought it had just run into a bad media file or something, but it didn't crash if I left the log file alone during subsequent passes.

3. I don't think that the file i/o is doing a flush when writing to the log. I see comskip running on my system, but in the log I see no indication that it launched it or what file its actually processing. The log ends in mid-word. Would be nice if it listed the command it was executing and flushed the log file prior to doing it so that I could see what it was actually on.

4. Is there a way to control the frequency of when SJQ does its scan? It seems to be doing it every 4 minutes, but to be honest I really don't need it to fire that often (since it scans every single item of media on the system). If there was a way to set it to a frequency of my choosing, it would be nice.
#7
09-11-2007, 12:35 PM
 Slugger SageTVaholic Join Date: Mar 2007 Location: Kingston, ON Posts: 4,008
SJQ v1.0.0 Released

SJQ v1.0.0 is now available.

Changelog
Code:
v0.1.1 -> v1.0.0
================

* Bug  7: Add ability to run internal tasks as actions in SJQ
* Bug  8: Update SQLite JDBC driver; move to pure Java driver
* Bug  9: Add support for interpolated strings in action lines and tests
* Bug 10: Add GKusnick's Studio Tools jar to packaged zip
* Bug 11: Add new boolean test: FileExists
* Bug 12: Schema upgrades not done in transaction
* Bug 15: 'files' table does not store job type

NOTE: Bug 16 has been deferred to the next release

You've been warned.
#8
10-19-2007, 08:01 PM
 Slugger SageTVaholic Join Date: Mar 2007 Location: Kingston, ON Posts: 4,008
SJQ v1.0.1 Released

SJQ v1.0.1 is now available.
#9
04-03-2008, 06:51 PM
 Slugger SageTVaholic Join Date: Mar 2007 Location: Kingston, ON Posts: 4,008
v1.1.0 is now available. A special thanks to cnovak for helping me test the RC builds and iron out the bugs!

v1.1.0 introduces a web configuration servlet compatible with the web server plugin. The servlet allows you to configure all of SJQ's options in real time via the web without the need to stop/restart the SageTV service. Also in this release is the web based rule editor. You no longer need to learn the ruleset syntax to setup processing rules in SJQ. Many other bug fixes and enhancements (new tests to test your media against) are also included. The online documentation has many more details.

Changelog:

Code:
v1.0.1 -> v1.1.0
================

* Bug 34: Add configuration servlet compatible with web server plugin
* Bug 44: Handle case where ruleset file does not exist at plugin startup
* Bug 45: New test: Age; allows testing of a media object's age
* Bug 47: Move all logging to database
* Bug 48: Use complete, absolute path of file names in 'Filename' test
* Bug 54: Log arguments used in transcode action lines
* Bug 55: Update online docs
* Bug 56: Add ability to run SJQ on demand (wake it up early and run it from servlet)
* Bug 59: Fix issue where some processes triggered could hang if they exhausted the OS output buffer
* Bug 61: Store plugin settings in data store instead of using Sage properties
* Bug 63: New test: TranscodeQueueSize
* Bug 64: Log all stdout and stderr output from external commands to the database
* Bug 65: Fix synchronization issue in Logger class
* Bug 67: New test: IsSomethingRecording - returns true if Sage is currently recording anything
* Bug 68: New test: FreeSpacePercentage - allows checking the amount of free space available on the disks used for recording
* Bug 71: New test: ShowID - allows testing against the unique external show id
#10
04-04-2008, 06:24 PM
 nyplayer SageTVaholic Join Date: Sep 2005 Posts: 3,982
Slugger,

Maybe you can help me by providing a sample.

Currently I run videoredo quickfix on all my files. I then run showanalyzer against the fixed files.

1. I run Videoredo Qfix it creates a file with the extension qfix to let me know it is completed.

2. I then would like to run Showanalyzer against the mpg files the are qfixed only... creating EDL and VPRJ files.

Also I do not want videredo to run if a qfix file exists.
__________________
Windows 10 I3 Dual Core 3.0 Server, SageTV9... OpenDCT with 2 HDHRPrimes and 4 HDHRconnects for a total of 14 Tuners.

Last edited by nyplayer; 04-04-2008 at 06:28 PM.
#11
04-04-2008, 06:54 PM
 Slugger SageTVaholic Join Date: Mar 2007 Location: Kingston, ON Posts: 4,008
Quote:
 Originally Posted by nyplayer Slugger, Maybe you can help me by providing a sample. Currently I run videoredo quickfix on all my files. I then run showanalyzer against the fixed files. 1. I run Videoredo Qfix it creates a file with the extension qfix to let me know it is completed. 2. I then would like to run Showanalyzer against the mpg files the are qfixed only... creating EDL and VPRJ files. Also I do not want videredo to run if a qfix file exists.
I'm not sure how familiar you are with SJQ, but I'll provide a ruleset file that should do what you want, but I also sent you a PM with details on how to grab me on MSN as it might be easier to walk through an example together.

Code:
# Run Qfix
= IsTV: true
= IsActivelyRecording: false
!= FileExists: %d%\%b%.qfix
QFIX "C:\Program Files\path\to\qfix.exe" %f%

# Run SA for files that have a qfix file
= FileExists: %d%\%b%.qfix
!= FileExists: %d%\%b%.edl   # Use this test to prevent SA running for recordings that already have an edl
= IsActivelyRecording: false # Don't run SA against actively recording files
COMMSKIP "C:\Program Files\path\to\ShowAnalyzer.exe" %f%
This ruleset file can be made with little difficulty using the web based ruleset editor. You don't need to know all this syntax when using the web. Let me know if you have any questions (may not see them until tomorrow now).

NOTE: Copy/paste of the ruleset file may not work. The action lines must start with a tab character - not sure what this BB will do to tab characters.

Last edited by Slugger; 04-04-2008 at 06:58 PM.
#12
01-24-2009, 08:39 PM
 Wheemer Sage Icon Join Date: Dec 2004 Location: Deer Lake, NL, Canada Posts: 1,493
How can I make multiple copies of comskip run on each client?

I also would like to have the captions processed at the same time as the commercials...

An example of how that might be done would be great.

This plugin rocks... can have any of the spare computers I have on my network do some of the processing work.

Also how do I get it to fire off the commands as soon as it starts recording?

Last edited by Wheemer; 01-24-2009 at 09:20 PM.
#13
01-25-2009, 11:14 AM
 Slugger SageTVaholic Join Date: Mar 2007 Location: Kingston, ON Posts: 4,008
Quote:
 Originally Posted by Wheemer How can I make multiple copies of comskip run on each client?
Configure the client to run multiple tasks simultaneously. The setup is described in the wiki doc.

Quote:
 I also would like to have the captions processed at the same time as the commercials... An example of how that might be done would be great.
No clue on this one; I assume it's a command line option to comskip or sa? If it is a command line option then simply pass that option when calling comskip or sa. Someone else may have an answer on this one.

Quote:
 This plugin rocks... can have any of the spare computers I have on my network do some of the processing work. Also how do I get it to fire off the commands as soon as it starts recording?
Ok, this one's not in the wiki... will add a new doc shortly. On the settings grid, change the RunType setting from 'AfterRec' to 'DuringRec'. When set to 'DuringRec' the QueueLoader will run 10 seconds after the start of each recording, which will queue up the jobs, but it may be up to 5 more minutes before a client comes along and starts the queued job.
__________________
Server: Intel i5-4570 Quad Core, 16GB RAM, 1 x 128GB OS SSD (Win7 Pro x64 SP1), 1 x 2TB media drive
Capture: 2 x Colossus
STB Controller: 1 x USB-UIRT
Software:Java 1.7.0_71; SageTV 7.1.9
Clients: 1 x HD300, 2 x HD200, 1 x SageClient, 1 x PlaceShifter
Plugins: Too many to list now...
#14
01-25-2009, 09:17 AM
 Grasshopper Sage Aficionado Join Date: Nov 2003 Posts: 314
Newbie assistance?

I have Sage Job Queue installed and working (I think). I have Comskip also installed and can manually run it by dragging a recorded show file onto it; I haven't tuned the commercial skipping yet, but it appears to work okay. But I'm at a loss when it comes to using Sage Job Queue to automate the comskip process. Can someone post an example of a ruleset (am I even talking about the right thing?) for a very simple comskip call without a lot of complex conditions? I have a very simple Sage server setup, running on a WHS box. I don't need to process on another pc, just the server is fine.
#15
01-25-2009, 11:28 AM
 Slugger SageTVaholic Join Date: Mar 2007 Location: Kingston, ON Posts: 4,008
Quote:
 Originally Posted by Grasshopper I have Sage Job Queue installed and working (I think). I have Comskip also installed and can manually run it by dragging a recorded show file onto it; I haven't tuned the commercial skipping yet, but it appears to work okay. But I'm at a loss when it comes to using Sage Job Queue to automate the comskip process. Can someone post an example of a ruleset (am I even talking about the right thing?) for a very simple comskip call without a lot of complex conditions? I have a very simple Sage server setup, running on a WHS box. I don't need to process on another pc, just the server is fine.
The wiki doc has some simple examples that should help you on your way. (I just added another example showing a typical rule for setting up comskip/showanalyzer.)
__________________
Server: Intel i5-4570 Quad Core, 16GB RAM, 1 x 128GB OS SSD (Win7 Pro x64 SP1), 1 x 2TB media drive
Capture: 2 x Colossus
STB Controller: 1 x USB-UIRT
Software:Java 1.7.0_71; SageTV 7.1.9
Clients: 1 x HD300, 2 x HD200, 1 x SageClient, 1 x PlaceShifter
Plugins: Too many to list now...
#16
04-12-2009, 03:32 AM
 zzmystique Sage Advanced User Join Date: May 2004 Location: New Zealand Posts: 211
Transcode based on Media Aspect Ratio (ie 16:9 or 4:3 etc..)

Hi Guys

I'm trying to configure a block to transcode based on the show Aspect Ratio...

if [IsTV == "true" && IsActivelyRecording == false && ?MediaRatio? == "16:9"]
{
:PRIORITY 15
:TRANSCODEOPTS "iPhone-Widescreen" "false"
_TRANSCODE
}

if [IsTV == "true" && IsActivelyRecording == false && ?MediaRatio? == "4:3"]
{
:PRIORITY 15
:TRANSCODEOPTS "iPhone-Standard" "false"
_TRANSCODE
}

I know mediaRatio is not a listed option, so has anyone got a suggestions on how I can do this?

Thanks
__________________
ZZMystique
My setup: Server: SageTV v7, Intel i5, 8GB, 180GB SSD, 4TB HDD, Win7HP & (DVB Viewer Pro with DVB Viewer for SageTV using 2xTBS6281 DVB-T2).

Last edited by zzmystique; 04-12-2009 at 03:34 AM.
#17
04-12-2009, 09:08 AM
 Slugger SageTVaholic Join Date: Mar 2007 Location: Kingston, ON Posts: 4,008
Open an issue ticket with a request to add the test to SJQ. I'm not even sure if I can access the video's aspect ratio through a Sage API call, but if I can then I'll add the test. Going to be at least a week and a half before I get a chance to investigate it though, as I'm getting ready to head to the airport for a trip to Las Vegas.
__________________
Server: Intel i5-4570 Quad Core, 16GB RAM, 1 x 128GB OS SSD (Win7 Pro x64 SP1), 1 x 2TB media drive
Capture: 2 x Colossus
STB Controller: 1 x USB-UIRT
Software:Java 1.7.0_71; SageTV 7.1.9
Clients: 1 x HD300, 2 x HD200, 1 x SageClient, 1 x PlaceShifter
Plugins: Too many to list now...
#18
04-12-2009, 08:44 PM
 wayner SageTVaholic Join Date: Jan 2008 Location: Toronto, ON Posts: 6,521
My Transcode jobs seem to be stuck!

For the last few days when I go into the Active Queue I see several _TRANSCODE jobs that are "waiting". I never had any issues with transcoder jobs in the past, in fact they would be complete immediately. When I look at the Sage Transcode queue these jobs do not appear to be in that queue.

Other jobs, such as Comskip jobs and Mediascrape jobs execute without issues but I don't seem to have had a successful Transcode since Apr 8 and I don't think I changed anything on that date. And I have rebooted my Sage server and that doesn't appear to have helped.
__________________
New Server - Sage9 on unRAID 2xHD-PVR, HDHR for OTA
Old Server - Sage7 on Win7Pro-i660CPU with 4.6TB, HD-PVR, HDHR OTA, HVR-1850 OTA
Clients - 2xHD-300, 8xHD-200 Extenders, Client+2xPlaceshifter and a WHS which acts as a backup Sage server
#19
04-13-2009, 08:22 AM
 nyplayer SageTVaholic Join Date: Sep 2005 Posts: 3,982
Quote:
 Originally Posted by wayner My Transcode jobs seem to be stuck! For the last few days when I go into the Active Queue I see several _TRANSCODE jobs that are "waiting". I never had any issues with transcoder jobs in the past, in fact they would be complete immediately. When I look at the Sage Transcode queue these jobs do not appear to be in that queue. Other jobs, such as Comskip jobs and Mediascrape jobs execute without issues but I don't seem to have had a successful Transcode since Apr 8 and I don't think I changed anything on that date. And I have rebooted my Sage server and that doesn't appear to have helped.
That has been discussed on this thread..
http://forums.sagetv.com/forums/show...&postcount=631. You need to go back to 343 or install snapshot 455.
__________________
Windows 10 I3 Dual Core 3.0 Server, SageTV9... OpenDCT with 2 HDHRPrimes and 4 HDHRconnects for a total of 14 Tuners.
#20
04-13-2009, 08:32 AM
 wayner SageTVaholic Join Date: Jan 2008 Location: Toronto, ON Posts: 6,521
Thanks nyplayer - I missed that.
__________________
New Server - Sage9 on unRAID 2xHD-PVR, HDHR for OTA
Old Server - Sage7 on Win7Pro-i660CPU with 4.6TB, HD-PVR, HDHR OTA, HVR-1850 OTA
Clients - 2xHD-300, 8xHD-200 Extenders, Client+2xPlaceshifter and a WHS which acts as a backup Sage server

 Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)

 Posting Rules You may not post new threads You may not post replies You may not post attachments You may not edit your posts BB code is On Smilies are On [IMG] code is On HTML code is Off Forum Rules
 Forum Jump User Control Panel Private Messages Subscriptions Who's Online Search Forums Forums Home Information & Announcements     View the Forum Terms of Service     Announcements     SageTV Downloads & Instructions SageTV Products     SageTV Software     SageTV EPG Service     SageTV Placeshifter     SageTV Media Extender     SageTV HD Theater - Media Player     SageTV Linux         SageTV for unRAID/Docker     SageTV Mac Edition     SageTV Recorder Software SageTV Development and Customizations     SageTV Github Development     SageTV Clients Development     SageTV Studio     SageTV Customizations         SageMC Custom Interface     SageTV v7 Customizations         Batch Metadata Tools         Diamond         Sage My Movies         Phoenix         Gemstone for v7     SageTV v9 Customizations         Gemstone for v9         Phoenix 3.0     Customization Announcements     SageTV Downloads SageTV BETA Release Products     SageTV Beta Test Software Hardware Support     Hardware Support SageTV International User Forums     SageTV Australia/New Zealand     SageTV Canada     SageTV Germany     SageTV France     SageTV Netherlands     SageTV South America     SageTV United Kingdom     SageTV Europe (Miscellaneous) General Discussion     General Discussion     The SageTV Community     SageTV Marketplace

 Similar Threads Thread Thread Starter Forum Replies Last Post Slugger SageTV Customizations 1355 07-25-2013 07:44 AM Slugger Customization Announcements 3 12-17-2009 09:59 AM raffmanlt SageTV Customizations 2 08-18-2009 07:34 PM personalt SageTV Customizations 6 03-02-2009 10:27 AM Slugger SageTV Customizations 35 04-21-2008 08:12 AM

All times are GMT -6. The time now is 10:47 AM.

 Contact Us - SageTV, LLC - Archive - Privacy Statement - Top