Plugin: Sage Job Queue (SJQ) v3

[gveres]

Originally Posted by davephan

Greg,

Could you post your code for processing files with the VideoReDo Quick Stream Fix (QSF)?

I'm also interested in your code for video compression.

Dave

Here is my ruleset:

// QuickStreamFix all recordings
if [IsObjMediaFile == true && $.IsTVFile{} == true && $.IsFileCurrentlyRecording{} == false && Filename =$ ".ts"] 
{
    :SCANMEDIA "true"
    :RUNQLOADER "true"
    :PRIORITY 100
	QSF
}

// Comskip non-movie channels
if [IsObjMediaFile == true && $.IsTVFile{} == true && $.IsFileCurrentlyRecording{} == false && FileExists != "%d%/%p%.edl" && Filename =$ ".mpg" && $.GetShowTitle{} != "The Colbert Report" && $.GetShowTitle{} != "The Daily Show With Jon Stewart" && $.GetShowTitle{} != "Squeeze Play" && $.GetShowTitle{} != "Berman's Call" || $.GetShowTitle{} != "Trading Day"] 
{
    :RUNQLOADER "true"
    :PRIORITY 10
       COMSKIP
}

// compress for podcast
if [IsObjMediaFile == true && $.IsTVFile{} == true && $.IsFileCurrentlyRecording{} == false && Filename =$ ".mpg" && ($.GetShowTitle{} == "The Colbert Report" || $.GetShowTitle{} == "The Daily Show With Jon Stewart")] 
{
    :SCANMEDIA "true"
    :PRIORITY -10
       PODCASTCOMPRESS
}

// compress for podcast BNN
if [IsObjMediaFile == true && $.IsTVFile{} == true && $.IsFileCurrentlyRecording{} == false && Filename =$ ".mpg" && ($.GetShowTitle{} == "Squeeze Play" || $.GetShowTitle{} == "Berman's Call" || $.GetShowTitle{} == "Trading Day")] 
{
    :SCANMEDIA "true"
    :PRIORITY -10
       BNNPODCASTCOMPRESS
}

// compress adult shows
if [IsObjMediaFile == true && $.IsTVFile{} == true && $.IsFileCurrentlyRecording{} == false && FileExists == "%d%/%p%.edl" && Filename =$ ".mpg" && ($.GetShowTitle{} == "The Ellen DeGeneres Show" || $.GetShowTitle{} == "Nova scienceNOW" || $.GetShowTitle{} == "Rolex Sports Car Series Racing" || $.GetShowTitle{} == "SCCA Racing" || $.GetShowTitle{} == "New Yankee Workshop" || $.GetShowTitle{} == "Monster Jam")] 
{
    :SCANMEDIA "true"
    :PRIORITY -20
       ADULTCOMPRESS
}

// compress children's shows
if [IsObjMediaFile == true && $.IsTVFile{} == true && $.IsFileCurrentlyRecording{} == false && Filename =$ ".mpg" && ($.GetShowTitle{} == "Back at the Barnyard" || $.GetShowTitle{} == "Danny Phantom" || $.GetShowTitle{} == "Drake & Josh" || $.GetShowTitle{} == "Hannah Montana" || $.GetShowTitle{} == "Imagination Movers" || $.GetShowTitle{} == "Johnny Test" || $.GetShowTitle{} == "Jonas" || $.GetShowTitle{} == "Kim Possible" || $.GetShowTitle{} == "Life With Derek" || $.GetShowTitle{} == "Lizzie McGuire" || $.GetShowTitle{} == "Phineas and Ferb" || $.GetShowTitle{} == "Sonny With a Chance" || $.GetShowTitle{} == "SpongeBob SquarePants" || $.GetShowTitle{} == "Tak and the Power of Juju" || $.GetShowTitle{} == "The Emperor's New School" || $.GetShowTitle{} == "The Fairly OddParents" || $.GetShowTitle{} == "The Latest Buzz" || $.GetShowTitle{} == "The Penguins of Madagascar" || $.GetShowTitle{} == "The Replacements" || $.GetShowTitle{} == "The Suite Life of Zack & Cody" || $.GetShowTitle{} == "The Suite Life on Deck" || $.GetShowTitle{} == "Wizards of Waverly Place" || $.GetShowTitle{} == "Total Drama Action" || $.GetShowTitle{} == "Zoey 101" || $.GetShowTitle{} == "iCarly" ) ] 
{
    :SCANMEDIA "true"
    :PRIORITY -10
        KIDSCOMPRESS
}

All compression is using Media Shrink. The BNN podcast is different because it is a financial news network in canada and it has both a ticker on the bottom and a bar of info down the right side. So I had to adjust the comskip parameters to get comskip to properly find the commercials. But those parameters are not useful for regular channels.


Here is my client config

QSF {
   :MAX 2
   :RESOURCES 49
   :CPU "NORMAL"
   "C:/QFIXSAGE/qfixsage.bat \"%c%\""
}

COMSKIP {
   :MAX 2
   :RESOURCES 49
   :CPU "NORMAL"
   "C:/QFIXSAGE/cs.bat \"%c%\""
}

KIDSCOMPRESS {
   :MAX 1
   :RESOURCES 51
   :CPU "LOW"
   "C:/QFIXSAGE/KidsShrink.bat \"%c%\""
}

ADULTCOMPRESS {
   :MAX 1
   :RESOURCES 51
   :CPU "LOW"
   "C:/QFIXSAGE/AdultShrink.bat \"%c%\""
}

PODCASTCOMPRESS {
   :MAX 1
   :RESOURCES 40
   :CPU "LOW"
   "C:/QFIXSAGE/DailyShow.bat \"%c%\""
}

BNNPODCASTCOMPRESS {
   :MAX 1
   :RESOURCES 40
   :CPU "LOW"
   "C:/QFIXSAGE/squeezeplay.bat \"%c%\""
}

I think some of the options like Max might be outdated because of the resources option but I didn't go back and clean it up. Notice that each of these call a batch file. There was a lot of discussion in this thread about getting SJQ to properly call mediashrink. I have always used a batch file and not had a problem. I have zipped up all the batch files. Here is what each one does:

qfixsage.bat - this one does the QuickStreamFix. I have the R5000 create .ts files and QSF converts them to mpg files.

cs.bat - runs comskip using the normal profile

AdultShrink.bat - runs mediashrink to do a high quality h.264 mkv compression

DailyShow.bat - calls comskip and handbrake directly to create an ipod version of the file. It replaces the mpg file with the mp4 file. I also have it insert chapter markers so that when I am viewing on the ipod I can still skip the commercials

KidsShrink.bat - similar to AdultShrink but it compresses to a lower bitrate. Sorry kids, but you have too many shows eating up my disk space.

squeezeplay.bat - this is a varient of dailyshow.bat but it uses a special profile for comskip to properly detect the commercials.

You should probably PM me if you have any questions.

Hope that helps
Greg

[HuMan321]

Hi,
I just upgraded to version 3 from 2. My active queue is not populating and I have tried all scenarios I have read. Here is a paste of the server files when I manually try and populate.

2010-04-10 12:00:15,343 INFO [MediaFileQueueLoader]: Run completed [265ms]
2010-04-10 12:00:15,343 INFO [DataStore]: Thread 'SJQ-LogCleaner' is seeking an exclusive DataStore connection!
2010-04-10 12:00:15,343 INFO [DataStore]: Thread 'SJQ-LogCleaner' has received an exclusive DataStore connection! [WAIT=0ms]
2010-04-10 12:00:15,406 INFO [DataStore]: 'SJQ-LogCleaner' has returned its exclusive DataStore connection!
2010-04-10 12:00:15,406 INFO [LogCleaner]: Thread finished [63ms]
2010-04-10 12:00:15,406 INFO [VacuumCleaner]: Thread finished [63ms]
2010-04-10 12:00:15,421 INFO [FileCleaner]: Thread finished [78ms]
2010-04-10 12:00:29,875 INFO [SystemMessageQueueLoader]: Run completed [63ms]
2010-04-10 12:00:50,921 INFO [InternalTaskClient]: Run completed [0ms]
2010-04-10 12:00:57,078 WARN [SJQServlet]: All client connections are being accepted and processed, please consider setting up client restrictions in SJQ settings
2010-04-10 12:00:57,078 INFO [SJQServlet]: Servlet POST request completed [0ms]
2010-04-10 12:01:09,046 WARN [SJQServlet]: All client connections are being accepted and processed, please consider setting up client restrictions in SJQ settings
2010-04-10 12:01:09,078 INFO [SJQServlet]: Servlet POST request completed [47ms]
2010-04-10 12:01:30,078 INFO [SystemMessageQueueLoader]: Run completed [47ms]
2010-04-10 12:01:50,921 INFO [ClientMonitor]: Run completed [0ms]
2010-04-10 12:01:54,031 WARN [SJQServlet]: All client connections are being accepted and processed, please consider setting up client restrictions in SJQ settings

Does this show as to what may be going wrong?

Here is my client configuration:

COMSKIP {
"C:/Comskip/comskip.exe \"%c%\""
}
COMSKIP {
:RESOURCES 50 # Up to 4 comskips can be run simultaneously (every client has exactly 100 resources available)

# CPU priority to be used for this task process; one of LOW, NORMAL, HIGH (case sensitive)
:CPU "NORMAL" # Default is NORMAL if not specified

# Comskip returns 1 on success instead of the universally accepted zero; tell SJQ that
# a return code of 1 means success instead of the default value of zero
:RETURNCODE 1

"C:/comskip/comskip.exe \"%c%\""
}


Here is my Ruleset editor configuration:

if [IsObjMediaFile == true && $.IsTVFile{} == true && $.IsNotManualOrFavorite{} == false] {
:PRIORITY 10 # Optional, default priority is zero
COMSKIP
}


Media Debug:

2010-04-10 12:11:42,531 INFO [MediaFileQueueLoader]: sleep interrupted
2010-04-10 12:11:42,531 INFO [SJQServlet]: Servlet POST request completed [16ms]
2010-04-10 12:11:42,593 WARN [MediaFileQueueLoader]: Processing 'Kitchen Nightmares'...
2010-04-10 12:11:42,593 WARN [RulesParser]: Line 1: Evaluating 'IsObjMediaFile == "true"' == true
2010-04-10 12:11:42,593 WARN [RulesParser]: Line 1: Evaluating '$.IsTVFile{} == "true"' == true
2010-04-10 12:11:42,593 WARN [RulesParser]: Line 1: Evaluating '$.IsNotManualOrFavorite{} == "false"' == true
2010-04-10 12:11:42,671 INFO [MediaFileQueueLoader]: Run completed [140ms]
2010-04-10 12:11:42,671 INFO [VacuumCleaner]: Thread finished [0ms]
2010-04-10 12:11:42,671 INFO [DataStore]: Thread 'SJQ-LogCleaner' is seeking an exclusive DataStore connection!
2010-04-10 12:11:42,671 INFO [DataStore]: Thread 'SJQ-LogCleaner' is waiting for an exclusive DataStore connection! [AVAIL=4; MAX=5; WAIT=0ms]
2010-04-10 12:11:42,671 INFO [DataStore]: Thread 'SJQ-LogCleaner' has received an exclusive DataStore connection! [WAIT=0ms]
2010-04-10 12:11:42,687 INFO [FileCleaner]: Thread finished [16ms]
2010-04-10 12:11:42,750 INFO [DataStore]: 'SJQ-LogCleaner' has returned its exclusive DataStore connection!
2010-04-10 12:11:42,750 INFO [LogCleaner]: Thread finished [79ms]
2010-04-10 12:11:47,234 WARN [SJQServlet]: All client connections are being accepted and processed, please consider setting up client restrictions in SJQ settings

Thanks

[Fuzzy]

Originally Posted by HuMan321

Hi,
I just upgraded to version 3 from 2. My active queue is not populating and I have tried all scenarios I have read. Here is a paste of the server files when I manually try and populate.

2010-04-10 12:00:15,343 INFO [MediaFileQueueLoader]: Run completed [265ms]
2010-04-10 12:00:15,343 INFO [DataStore]: Thread 'SJQ-LogCleaner' is seeking an exclusive DataStore connection!
2010-04-10 12:00:15,343 INFO [DataStore]: Thread 'SJQ-LogCleaner' has received an exclusive DataStore connection! [WAIT=0ms]
2010-04-10 12:00:15,406 INFO [DataStore]: 'SJQ-LogCleaner' has returned its exclusive DataStore connection!
2010-04-10 12:00:15,406 INFO [LogCleaner]: Thread finished [63ms]
2010-04-10 12:00:15,406 INFO [VacuumCleaner]: Thread finished [63ms]
2010-04-10 12:00:15,421 INFO [FileCleaner]: Thread finished [78ms]
2010-04-10 12:00:29,875 INFO [SystemMessageQueueLoader]: Run completed [63ms]
2010-04-10 12:00:50,921 INFO [InternalTaskClient]: Run completed [0ms]
2010-04-10 12:00:57,078 WARN [SJQServlet]: All client connections are being accepted and processed, please consider setting up client restrictions in SJQ settings
2010-04-10 12:00:57,078 INFO [SJQServlet]: Servlet POST request completed [0ms]
2010-04-10 12:01:09,046 WARN [SJQServlet]: All client connections are being accepted and processed, please consider setting up client restrictions in SJQ settings
2010-04-10 12:01:09,078 INFO [SJQServlet]: Servlet POST request completed [47ms]
2010-04-10 12:01:30,078 INFO [SystemMessageQueueLoader]: Run completed [47ms]
2010-04-10 12:01:50,921 INFO [ClientMonitor]: Run completed [0ms]
2010-04-10 12:01:54,031 WARN [SJQServlet]: All client connections are being accepted and processed, please consider setting up client restrictions in SJQ settings

Does this show as to what may be going wrong?

Here is my client configuration:

COMSKIP {
"C:/Comskip/comskip.exe \"%c%\""
}
COMSKIP {
:RESOURCES 50 # Up to 4 comskips can be run simultaneously (every client has exactly 100 resources available)

# CPU priority to be used for this task process; one of LOW, NORMAL, HIGH (case sensitive)
:CPU "NORMAL" # Default is NORMAL if not specified

# Comskip returns 1 on success instead of the universally accepted zero; tell SJQ that
# a return code of 1 means success instead of the default value of zero
:RETURNCODE 1

"C:/comskip/comskip.exe \"%c%\""
}


Here is my Ruleset editor configuration:

if [IsObjMediaFile == true && $.IsTVFile{} == true && $.IsNotManualOrFavorite{} == false] {
:PRIORITY 10 # Optional, default priority is zero
COMSKIP
}


Media Debug:

2010-04-10 12:11:42,531 INFO [MediaFileQueueLoader]: sleep interrupted
2010-04-10 12:11:42,531 INFO [SJQServlet]: Servlet POST request completed [16ms]
2010-04-10 12:11:42,593 WARN [MediaFileQueueLoader]: Processing 'Kitchen Nightmares'...
2010-04-10 12:11:42,593 WARN [RulesParser]: Line 1: Evaluating 'IsObjMediaFile == "true"' == true
2010-04-10 12:11:42,593 WARN [RulesParser]: Line 1: Evaluating '$.IsTVFile{} == "true"' == true
2010-04-10 12:11:42,593 WARN [RulesParser]: Line 1: Evaluating '$.IsNotManualOrFavorite{} == "false"' == true
2010-04-10 12:11:42,671 INFO [MediaFileQueueLoader]: Run completed [140ms]
2010-04-10 12:11:42,671 INFO [VacuumCleaner]: Thread finished [0ms]
2010-04-10 12:11:42,671 INFO [DataStore]: Thread 'SJQ-LogCleaner' is seeking an exclusive DataStore connection!
2010-04-10 12:11:42,671 INFO [DataStore]: Thread 'SJQ-LogCleaner' is waiting for an exclusive DataStore connection! [AVAIL=4; MAX=5; WAIT=0ms]
2010-04-10 12:11:42,671 INFO [DataStore]: Thread 'SJQ-LogCleaner' has received an exclusive DataStore connection! [WAIT=0ms]
2010-04-10 12:11:42,687 INFO [FileCleaner]: Thread finished [16ms]
2010-04-10 12:11:42,750 INFO [DataStore]: 'SJQ-LogCleaner' has returned its exclusive DataStore connection!
2010-04-10 12:11:42,750 INFO [LogCleaner]: Thread finished [79ms]
2010-04-10 12:11:47,234 WARN [SJQServlet]: All client connections are being accepted and processed, please consider setting up client restrictions in SJQ settings

Thanks

For starters, it looks like you've got 2 COMSKIP tasks in that client config. A simple one with just the comskip launch, and another with some configuration settings. Get rid of the simple one at the top, and see what happens.

[davephan]

Gveres,

Thanks for posting your ruleset and client files. It will take me awhile to go through all the information. I have already improved my ruleset file by adding:

&& $.IsTVFile{} == true && $.IsFileCurrentlyRecording{} == false

to the file, which was from your ruleset file. My ruleset file currently lists 25 non-commercial channels not to process with Comskip. There are 27 non-commercial channels not processed due to the wildcard for CSPAN. I think I could reduce the ruleset file size by using more wildcards in the channel list.

My ruleset file is currently:

if [IsObjMediaFile == true && $.IsTVFile{} == true && $.IsFileCurrentlyRecording{} == false && $.IsTVFile{} == true && ($.GetAiringChannelName{} !% "KTCADT|KTCADT2|KTCADT3|KTCADT4|CSPAN*|ENCRWA|ENCORE|ENCRA|ENCRM|ENCRL|ENCRD|ENCRWS|HBOHD|HBO2|HBOSIG|HBOF|HBOC|HBOZ|MPLEX|IFC|SUNDAE|INDIE|RETR|FLIX|TMC")] {
   COMSKIP
}

My client file is currently:

if [IsObjMediaFile == true && $.IsTVFile{} == true && $.IsFileCurrentlyRecording{} == false && $.IsTVFile{} == true && ($.GetAiringChannelName{} !% "KTCADT|KTCADT2|KTCADT3|KTCADT4|CSPAN*|ENCRWA|ENCORE|ENCRA|ENCRM|ENCRL|ENCRD|ENCRWS|HBOHD|HBO2|HBOSIG|HBOF|HBOC|HBOZ|MPLEX|IFC|SUNDAE|INDIE|RETR|FLIX|TMC")] {
   COMSKIP
}

Dave

[Fuzzy]

I'm thinking that AIN'T your client file... copy/paste mixup... either way, you could simplify your rule quite a bit by wildcarding a few more of those channel groups. KTCA*, ENC* and HBO* would eliminate most of your list.

[Slugger]

I encourage you to read the Java docs on regular expressions as linked to in the SJQv3 User's Guide. A simple * is not a wild card in regex, and is, therefore, not a valid regex value. Which means your ruleset will not work as you've currently posted it.

CSPAN* is not correct. CSPAN.* (notice the dot) would be valid. As Fuzzy says, that list could be cut down to something much shorter:

$.GetAiringChannelName{} !% "KTCADT.*|CSPAN.*|ENCR.*|ENCORE|HBO.*|MPLEX|IFC|SUNDAE|INDIE|RETR|FLIX|TMC"

[davephan]

Thanks for the correction! That was a copy/paste error. I did not realize that a wildcard needed to be a ".*" instead of a "*". I fixed the scripts now.

The ruleset is much shorter now:

if [IsObjMediaFile == true && $.IsTVFile{} == true && $.IsFileCurrentlyRecording{} == false && $.IsTVFile{} == true && ($.GetAiringChannelName{} !% "KTCADT.*|CSPAN.*|ENCR.*|ENCORE|HBO.*|MPLEX|IFC|SUNDAE|INDIE|RETR|FLIX|TMC")] {
COMSKIP
}

The client configuration file is:

COMSKIP {
:RESOURCES 50 # Up to 2 comskips can be run simultaneously (every client has exactly 100 resources available)

# CPU priority to be used for this task process; one of LOW, NORMAL, HIGH (case sensitive)
:CPU "NORMAL" # Default is NORMAL if not specified

# Comskip returns 1 on success instead of the universally accepted zero; tell SJQ that
# a return code of 1 means success instead of the default value of zero
:RETURNCODE 1

"C:/comskip/comskip.exe \"%c%\""
}

Dave

[Fuzzy]

Only other thing I'd recommend is changing the :CPU setting to "LOW". This will prevent comskip from messing with more 'important' tasks.

[davephan]

Originally Posted by Fuzzy

Only other thing I'd recommend is changing the :CPU setting to "LOW". This will prevent comskip from messing with more 'important' tasks.

I tried experimenting with the CPU setting with SJQ and Comskip. It seems that the donor version of Comskip must be set elsewhere to Low priority, since when I tried other values, such as Normal priority, Comskip still ran at Low priority. Otherwise, I would have set the priority to Low.

Dave

[HuMan321]

Can anyone tell me why this Chelsea Handler file is not being comskipped with my settings?

Thanks

2010-04-14 19:00:12,928 WARN [MediaFileQueueLoader]: Processing 'Chelsea Lately'...
2010-04-14 19:00:12,928 WARN [RulesParser]: Line 1: Evaluating 'IsObjMediaFile == "true"' == true
2010-04-14 19:00:12,928 WARN [RulesParser]: Line 1: Evaluating '$.IsTVFile{} == "true"' == true
2010-04-14 19:00:12,928 WARN [RulesParser]: Line 1: Evaluating '$.IsNotManualOrFavorite{} == "false"' == true
2010-04-14 19:00:12,928 WARN [RulesParser]: Line 1: Evaluating '$.GetAiringChannelName{} !% "CSPAN.*|ACMAX.*|KAIDDT.*|STARZ.*|STZ.*|HBO.*|.*MAX.*|SHO.*|TMC.*|PPV\d+"' == true
2010-04-14 19:00:13,022 INFO [MediaFileQueueLoader]: Run completed [156ms]
2010-04-14 19:00:13,022 INFO [DataStore]: Thread 'SJQ-LogCleaner' is seeking an exclusive DataStore connection!
2010-04-14 19:00:13,022 INFO [DataStore]: Thread 'SJQ-LogCleaner' is waiting for an exclusive DataStore connection! [AVAIL=3; MAX=5; WAIT=0ms]

[Slugger]

Will need to see the ruleset. Also, looks like maybe we need to see the client config as well. Looks like something is crashing based on the last line of the given output. Check the client logs for exception dumps as well.

[HuMan321]

Originally Posted by Slugger

Will need to see the ruleset. Also, looks like maybe we need to see the client config as well. Looks like something is crashing based on the last line of the given output. Check the client logs for exception dumps as well.

Thanks for looking
Ruleset:
if [IsObjMediaFile == true && $.IsTVFile{} == true && $.IsNotManualOrFavorite{} == false && $.GetAiringChannelName{} !% "CSPAN.*|ACMAX.*|KAIDDT.*|STARZ.*|STZ.*|HBO.*|.*MAX.*|SHO.*|TMC.*|PPV\\d+"] {

COMSKIP
}

Client Config:
COMSKIP {
:RESOURCES 50 # Up to 4 comskips can be run simultaneously (every client has exactly 100 resources available)

# CPU priority to be used for this task process; one of LOW, NORMAL, HIGH (case sensitive)
:CPU "NORMAL" # Default is NORMAL if not specified

# Comskip returns 1 on success instead of the universally accepted zero; tell SJQ that
# a return code of 1 means success instead of the default value of zero
:RETURNCODE 1

"C:/Comskip/comskip.exe \"%c%\""
}


Client logs always appear empty as they do now.

[davephan]

Here's my working ruleset code. I discovered I have the code

$.IsTVFile{} == true

is in there twice. I will remove one of them.

if [IsObjMediaFile == true && $.IsTVFile{} == true && $.IsFileCurrentlyRecording{} == false && $.IsTVFile{} == true && ($.GetAiringChannelName{} !% "KTCADT.*|CSPAN.*|ENCR.*|ENCORE|HBO.*|MPLEX|IFC|SUNDAE|INDIE|RETR|FLIX|TMC")] {
COMSKIP
}


When I compared your code to my code, this is missing from the ruleset:

$.IsFileCurrentlyRecording{} == false

which will cause Comskip to try running on programs that are not completed recording.

That the end of this code

$.GetAiringChannelName{} !% "CSPAN.*|ACMAX.*|KAIDDT.*|STARZ.*|STZ.*|HBO.*|.*MAX.*|SHO.*|TMC.*|PPV\\d+"] {

you have:

\\d+"]

instead of

")]

I don't know what effect this will have.

The client configuration file has a minor problem where it says "4" Comskips at a time, when it will really be 2 Comskips at a time with the 50%. It would have to be 25% to have 4 Comskips at a time.

I've found the

:CPU "NORMAL"

does not make any difference. Comskip seems to always run at LOW priority, which is OK. It will take a little longer to run, but the Comskip processes will yield to other processes, most of which are running at NORMAL priority. You can verify that in Windows Task Manager, Control-Alt-ESC, Process tab, right click, hover over "set priority", you will see the current priority that is set while the process is running.

Dave

[Slugger]

Originally Posted by HuMan321

Thanks for looking
Ruleset:
if [IsObjMediaFile == true && $.IsTVFile{} == true && $.IsNotManualOrFavorite{} == false && $.GetAiringChannelName{} !% "CSPAN.*|ACMAX.*|KAIDDT.*|STARZ.*|STZ.*|HBO.*|.*MAX.*|SHO.*|TMC.*|PPV\\d+"] {

COMSKIP
}

Client Config:
COMSKIP {
:RESOURCES 50 # Up to 4 comskips can be run simultaneously (every client has exactly 100 resources available)

# CPU priority to be used for this task process; one of LOW, NORMAL, HIGH (case sensitive)
:CPU "NORMAL" # Default is NORMAL if not specified

# Comskip returns 1 on success instead of the universally accepted zero; tell SJQ that
# a return code of 1 means success instead of the default value of zero
:RETURNCODE 1

"C:/Comskip/comskip.exe \"%c%\""
}


Client logs always appear empty as they do now.

Please stop SJQ, edit sjq.log4j.properties and increase the log level to DEBUG and try again. As far as I can tell, the task should be added to the queue and I have no guess as to why it's not being added.

[HuMan321]

Originally Posted by Slugger

Please stop SJQ, edit sjq.log4j.properties and increase the log level to DEBUG and try again. As far as I can tell, the task should be added to the queue and I have no guess as to why it's not being added.

2010-04-15 11:55:16,147 WARN [SJQServlet]: All client connections are being accepted and processed, please consider setting up client restrictions in SJQ settings
2010-04-15 11:55:16,147 INFO [SJQServlet]: Servlet POST request completed [16ms]
2010-04-15 11:55:16,147 INFO [MediaFileQueueLoader]: sleep interrupted
2010-04-15 11:55:16,209 WARN [MediaFileQueueLoader]: Processing 'Chelsea Lately'...
2010-04-15 11:55:16,209 WARN [RulesParser]: Line 1: Evaluating 'IsObjMediaFile == "true"' == true
2010-04-15 11:55:16,209 WARN [RulesParser]: Line 1: Evaluating '$.IsTVFile{} == "true"' == true
2010-04-15 11:55:16,209 WARN [RulesParser]: Line 1: Evaluating '$.IsNotManualOrFavorite{} == "false"' == true
2010-04-15 11:55:16,209 WARN [RulesParser]: Line 1: Evaluating '$.GetAiringChannelName{} !% "CSPAN.*|ACMAX.*|KAIDDT.*|STARZ.*|STZ.*|HBO.*|.*MAX.*|SHO.*|TMC.*|PPV\d+"' == true
2010-04-15 11:55:16,287 INFO [MediaFileQueueLoader]: Run completed [140ms]
2010-04-15 11:55:16,287 INFO [DataStore]: Thread 'SJQ-LogCleaner' is seeking an exclusive DataStore connection!
2010-04-15 11:55:16,287 INFO [DataStore]: Thread 'SJQ-LogCleaner' is waiting for an exclusive DataStore connection! [AVAIL=3; MAX=5; WAIT=0ms]
2010-04-15 11:55:16,287 INFO [DataStore]: Thread 'SJQ-LogCleaner' has received an exclusive DataStore connection! [WAIT=0ms]
2010-04-15 11:55:16,287 INFO [VacuumCleaner]: Thread finished [0ms]
2010-04-15 11:55:16,319 WARN [FileCleaner]: Deleted '\\SAGE\Sage Recordings\RachaelRay-DIYHairDisastersWithKyan-4037724-0.logo.txt'
2010-04-15 11:55:16,319 INFO [FileCleaner]: Thread finished [32ms]
2010-04-15 11:55:16,366 INFO [DataStore]: 'SJQ-LogCleaner' has returned its exclusive DataStore connection!
2010-04-15 11:55:16,366 INFO [LogCleaner]: Thread finished [79ms]
2010-04-15 11:55:20,553 WARN [SJQServlet]: All client connections are being accepted and processed, please consider setting up client restrictions in SJQ settings


Not sure I edited the properties file correctly because the server logs are still empty. Here it is:
log4j.logger.com.google.code.sagetvaddons.sjq=DEBUG, sjqApp
#log4j.logger.com.google.code.sagetvaddons.sjq.server.SQLLogger=FATAL

log4j.appender.sjqApp=org.apache.log4j.RollingFileAppender
log4j.appender.sjqApp.File=sjq.log
log4j.appender.sjqApp.MaxFileSize=1MB
log4j.appender.sjqApp.MaxBackupIndex=5
log4j.appender.sjqApp.layout=org.apache.log4j.PatternLayout
log4j.appender.sjqApp.layout.ConversionPattern=%d %-5p [%c{1}]: %m%n


After PC Reboot and media debugger:

2010-04-15 12:12:09,109 DEBUG [DataStore]: Thread '26085267@qtp-31130885-6' has received a non-exclusive DataStore connection! [WAIT=0ms]
2010-04-15 12:12:09,109 WARN [SJQServlet]: All client connections are being accepted and processed, please consider setting up client restrictions in SJQ settings
2010-04-15 12:12:09,109 DEBUG [DataStore]: Thread '26085267@qtp-31130885-6' returned a connection; now 5 in the pool.
2010-04-15 12:12:09,109 INFO [SJQServlet]: Servlet POST request completed [0ms]
2010-04-15 12:12:10,703 DEBUG [InternalTaskClient]: Run started
2010-04-15 12:12:10,703 INFO [InternalTaskClient]: Run completed [0ms]
2010-04-15 12:12:14,828 DEBUG [DataStore]: Thread 'SJQ-SystemMessageQueueLoader' has received a non-exclusive DataStore connection! [WAIT=0ms]
2010-04-15 12:12:14,875 DEBUG [DataStore]: Thread 'SJQ-SystemMessageQueueLoader' returned a connection; now 5 in the pool.
2010-04-15 12:12:14,875 DEBUG [SystemMessageQueueLoader]: Run started
2010-04-15 12:12:14,875 DEBUG [DataStore]: Thread 'SJQ-SystemMessageQueueLoader' has received a non-exclusive DataStore connection! [WAIT=0ms]
2010-04-15 12:12:14,953 INFO [SystemMessageQueueLoader]: Run completed [78ms]
2010-04-15 12:12:15,015 DEBUG [DataStore]: Thread 'SJQ-SystemMessageQueueLoader' returned a connection; now 5 in the pool.
2010-04-15 12:12:24,109 DEBUG [DataStore]: Thread '26085267@qtp-31130885-6' has received a non-exclusive DataStore connection! [WAIT=0ms]
2010-04-15 12:12:24,109 WARN [SJQServlet]: All client connections are being accepted and processed, please consider setting up client restrictions in SJQ settings
2010-04-15 12:12:24,109 DEBUG [DataStore]: Thread '26085267@qtp-31130885-6' returned a connection; now 5 in the pool.
2010-04-15 12:12:24,109 INFO [SJQServlet]: Servlet POST request completed [0ms]
2010-04-15 12:12:28,765 DEBUG [DataStore]: Thread '26085267@qtp-31130885-6' has received a non-exclusive DataStore connection! [WAIT=0ms]
2010-04-15 12:12:28,765 WARN [SJQServlet]: All client connections are being accepted and processed, please consider setting up client restrictions in SJQ settings
2010-04-15 12:12:28,765 INFO [MediaFileQueueLoader]: sleep interrupted
2010-04-15 12:12:28,765 DEBUG [MediaFileQueueLoader]: Run started
2010-04-15 12:12:28,765 DEBUG [DataStore]: Thread '26085267@qtp-31130885-6' returned a connection; now 5 in the pool.
2010-04-15 12:12:28,765 INFO [SJQServlet]: Servlet POST request completed [0ms]
2010-04-15 12:12:28,765 DEBUG [DataStore]: Thread 'SJQ-MediaFileQueueLoader' has received a non-exclusive DataStore connection! [WAIT=0ms]
2010-04-15 12:12:28,828 DEBUG [DataStore]: Thread 'SJQ-MediaFileQueueLoader' returned a connection; now 5 in the pool.
2010-04-15 12:12:28,828 DEBUG [MediaFileQueueLoader]: Using MediaMask: 'T'
2010-04-15 12:12:28,828 WARN [MediaFileQueueLoader]: Processing 'Chelsea Lately'...
2010-04-15 12:12:28,828 WARN [RulesParser]: Line 1: Evaluating 'IsObjMediaFile == "true"' == true
2010-04-15 12:12:28,828 DEBUG [RulesParser]: Want to run: 'IsTVFile' with arg: ''
2010-04-15 12:12:28,828 DEBUG [RulesParser]: Invoking 'IsTVFile' on type 'gkusnick.sagetv.api.MediaFileAPI.MediaFile' with return type: 'boolean'
2010-04-15 12:12:28,828 WARN [RulesParser]: Line 1: Evaluating '$.IsTVFile{} == "true"' == true
2010-04-15 12:12:28,828 DEBUG [RulesParser]: Want to run: 'IsNotManualOrFavorite' with arg: ''
2010-04-15 12:12:28,828 DEBUG [RulesParser]: Invoking 'IsNotManualOrFavorite' on type 'gkusnick.sagetv.api.AiringAPI.Airing' with return type: 'boolean'
2010-04-15 12:12:28,828 WARN [RulesParser]: Line 1: Evaluating '$.IsNotManualOrFavorite{} == "false"' == true
2010-04-15 12:12:28,828 DEBUG [RulesParser]: Want to run: 'GetAiringChannelName' with arg: ''
2010-04-15 12:12:28,828 DEBUG [RulesParser]: Invoking 'GetAiringChannelName' on type 'gkusnick.sagetv.api.AiringAPI.Airing' with return type: 'java.lang.String'
2010-04-15 12:12:28,828 WARN [RulesParser]: Line 1: Evaluating '$.GetAiringChannelName{} !% "CSPAN.*|ACMAX.*|KAIDDT.*|STARZ.*|STZ.*|HBO.*|.*MAX.*|SHO.*|TMC.*|PPV\d+"' == true
2010-04-15 12:12:28,828 DEBUG [DataStore]: Thread 'SJQ-MediaFileQueueLoader' has received a non-exclusive DataStore connection! [WAIT=0ms]
2010-04-15 12:12:28,828 DEBUG [TaskQueue]: Skipping: COMSKIP::4057276::media::0::CREATED, already been processed
2010-04-15 12:12:28,828 DEBUG [DataStore]: Thread 'SJQ-MediaFileQueueLoader' returned a connection; now 5 in the pool.
2010-04-15 12:12:28,828 DEBUG [DataStore]: Thread 'SJQ-MediaFileQueueLoader' has received a non-exclusive DataStore connection! [WAIT=0ms]
2010-04-15 12:12:28,828 DEBUG [DataStore]: Thread 'SJQ-MediaFileQueueLoader' returned a connection; now 5 in the pool.
2010-04-15 12:12:28,828 DEBUG [DataStore]: Thread 'SJQ-MediaFileQueueLoader' has received a non-exclusive DataStore connection! [WAIT=0ms]
2010-04-15 12:12:28,921 DEBUG [DataStore]: Thread 'SJQ-MediaFileQueueLoader' returned a connection; now 5 in the pool.
2010-04-15 12:12:28,921 INFO [MediaFileQueueLoader]: Run completed [156ms]
2010-04-15 12:12:28,921 DEBUG [VacuumCleaner]: Thread started
2010-04-15 12:12:28,921 DEBUG [LogCleaner]: Thread started
2010-04-15 12:12:28,921 DEBUG [FileCleaner]: Thread started
2010-04-15 12:12:28,921 DEBUG [DataStore]: Thread 'SJQ-VacuumCleaner' has received a non-exclusive DataStore connection! [WAIT=0ms]
2010-04-15 12:12:28,921 DEBUG [DataStore]: Thread 'SJQ-FileCleaner' has received a non-exclusive DataStore connection! [WAIT=0ms]
2010-04-15 12:12:28,921 INFO [DataStore]: Thread 'SJQ-LogCleaner' is seeking an exclusive DataStore connection!
2010-04-15 12:12:28,921 INFO [DataStore]: Thread 'SJQ-LogCleaner' is waiting for an exclusive DataStore connection! [AVAIL=3; MAX=5; WAIT=0ms]
2010-04-15 12:12:28,921 DEBUG [DataStore]: Thread 'SJQ-VacuumCleaner' returned a connection; now 4 in the pool.
2010-04-15 12:12:28,921 INFO [VacuumCleaner]: Thread finished [0ms]
2010-04-15 12:12:28,921 INFO [DataStore]: Thread 'SJQ-LogCleaner' is waiting for an exclusive DataStore connection! [AVAIL=4; MAX=5; WAIT=0ms]
2010-04-15 12:12:28,921 DEBUG [DataStore]: Thread 'SJQ-FileCleaner' returned a connection; now 5 in the pool.
2010-04-15 12:12:28,921 INFO [DataStore]: Thread 'SJQ-LogCleaner' has received an exclusive DataStore connection! [WAIT=0ms]
2010-04-15 12:12:28,953 INFO [FileCleaner]: Thread finished [32ms]
2010-04-15 12:12:29,000 INFO [DataStore]: 'SJQ-LogCleaner' has returned its exclusive DataStore connection!
2010-04-15 12:12:29,000 DEBUG [DataStore]: Thread 'SJQ-LogCleaner' returned a connection; now 5 in the pool.
2010-04-15 12:12:29,000 INFO [LogCleaner]: Thread finished [79ms]
2010-04-15 12:12:35,078 DEBUG [DataStore]: Thread '26085267@qtp-31130885-6' has received a non-exclusive DataStore connection! [WAIT=0ms]
2010-04-15 12:12:35,078 WARN [SJQServlet]: All client connections are being accepted and processed, please consider setting up client restrictions in SJQ settings
2010-04-15 12:12:35,156 DEBUG [DataStore]: Thread '26085267@qtp-31130885-6' returned a connection; now 5 in the pool.
2010-04-15 12:12:35,156 INFO [SJQServlet]: Servlet POST request completed [78ms]
2010-04-15 12:12:39,109 DEBUG [DataStore]: Thread '26085267@qtp-31130885-6' has received a non-exclusive DataStore connection! [WAIT=0ms]
2010-04-15 12:12:39,109 WARN [SJQServlet]: All client connections are being accepted and processed, please consider setting up client restrictions in SJQ settings
2010-04-15 12:12:39,109 DEBUG [DataStore]: Thread '26085267@qtp-31130885-6' returned a connection; now 5 in the pool.
2010-04-15 12:12:39,109 INFO [SJQServlet]: Servlet POST request completed [0ms]
2010-04-15 12:12:54,109 DEBUG [DataStore]: Thread '26085267@qtp-31130885-6' has received a non-exclusive DataStore connection! [WAIT=0ms]
2010-04-15 12:12:54,109 WARN [SJQServlet]: All client connections are being accepted and processed, please consider setting up client restrictions in SJQ settings
2010-04-15 12:12:54,109 DEBUG [DataStore]: Thread '26085267@qtp-31130885-6' returned a connection; now 5 in the pool.
2010-04-15 12:12:54,109 INFO [SJQServlet]: Servlet POST request completed [0ms]
2010-04-15 12:13:09,109 DEBUG [DataStore]: Thread '26085267@qtp-31130885-6' has received a non-exclusive DataStore connection! [WAIT=0ms]
2010-04-15 12:13:09,109 WARN [SJQServlet]: All client connections are being accepted and processed, please consider setting up client restrictions in SJQ settings
2010-04-15 12:13:09,109 DEBUG [DataStore]: Thread '26085267@qtp-31130885-6' returned a connection; now 5 in the pool.
2010-04-15 12:13:09,109 INFO [SJQServlet]: Servlet POST request completed [0ms]
2010-04-15 12:13:10,703 DEBUG [ClientMonitor]: Run started
2010-04-15 12:13:10,703 INFO [ClientMonitor]: Run completed [0ms]
2010-04-15 12:13:15,015 DEBUG [DataStore]: Thread 'SJQ-SystemMessageQueueLoader' has received a non-exclusive DataStore connection! [WAIT=0ms]
2010-04-15 12:13:15,078 DEBUG [DataStore]: Thread 'SJQ-SystemMessageQueueLoader' returned a connection; now 5 in the pool.
2010-04-15 12:13:15,078 DEBUG [SystemMessageQueueLoader]: Run started
2010-04-15 12:13:15,078 DEBUG [DataStore]: Thread 'SJQ-SystemMessageQueueLoader' has received a non-exclusive DataStore connection! [WAIT=0ms]
2010-04-15 12:13:15,156 INFO [SystemMessageQueueLoader]: Run completed [78ms]
2010-04-15 12:13:15,234 DEBUG [DataStore]: Thread 'SJQ-SystemMessageQueueLoader' returned a connection; now 5 in the pool.
2010-04-15 12:13:20,078 DEBUG [DataStore]: Thread '26085267@qtp-31130885-6' has received a non-exclusive DataStore connection! [WAIT=0ms]
2010-04-15 12:13:20,078 WARN [SJQServlet]: All client connections are being accepted and processed, please consider setting up client restrictions in SJQ settings

[Slugger]

And there's our answer...

2010-04-15 12:12:28,828 DEBUG [TaskQueue]: Skipping: COMSKIP::4057276::media::0::CREATED, already been processed

SJQ already believes that it has performed the COMSKIP task on that media object. If you want it to rerun it then you will have to find it in the completed or failed task list and delete it from there. Until then, SJQ will refuse to perform the same task more than once on the same media object.

[HuMan321]

Originally Posted by Slugger

And there's our answer...

2010-04-15 12:12:28,828 DEBUG [TaskQueue]: Skipping: COMSKIP::4057276::media::0::CREATED, already been processed

SJQ already believes that it has performed the COMSKIP task on that media object. If you want it to rerun it then you will have to find it in the completed or failed task list and delete it from there. Until then, SJQ will refuse to perform the same task more than once on the same media object.

Nice work!!!

That was it even though the file did not have an edl attached to it I found it in the completed file list. If SJQ has a file in the active directory queue with an associated edl file will it skip it? I hope so because I ended up dumping the whole completed file section.

Thanks again!

[Fuzzy]

No, not by itself.. it will only skip a file if it has already been processed by that task. You can add a check for the .edl file to the rule though, to prevent it from even adding those shows to the queue.

[HuMan321]

Originally Posted by Fuzzy

No, not by itself.. it will only skip a file if it has already been processed by that task. You can add a check for the .edl file to the rule though, to prevent it from even adding those shows to the queue.

I modified the ruleset. Does this look accurate?

if [IsObjMediaFile == true && $.IsTVFile{} == true && FileExists != "%d%/%p%.edl" && $.IsNotManualOrFavorite{} == false && $.GetAiringChannelName{} !% "CSPAN.*|ACMAX.*|KAIDDT.*|STARZ.*|STZ.*|HBO.*|.*MAX.*|SHO.*|TMC.*|PPV\\d+"] {

COMSKIP
}

[Slugger]

That looks good to me.