Plugin: Sage Job Queue (SJQ) v3

[Slugger]

Instead of writing a batch file, just add the :RETURNCODE 1 option to the client where you call comskip. That will tell SJQ that it should expect a return code of 1 instead of the default 0. No need to write a batch file anymore to work around this issue.

[doc]

Originally Posted by Slugger

Instead of writing a batch file, just add the :RETURNCODE 1 option to the client where you call comskip. That will tell SJQ that it should expect a return code of 1 instead of the default 0. No need to write a batch file anymore to work around this issue.

that one slipped through my radar!

[mikep]

Hi,

First, thanks for a great program!

Also , sorry for any typos, I am french and english is not my first language.

Is it possible to include an options for _MOVEREC task in order to specified a login and password to use when moving the file. Let me explain. My sagetv service (WinXP) is running with an account that have only read access to my nas, I don't want kids to be able to delete files from my nas within sage with the delete media button. The SJQ web is also running within the same security context as sagetv service (it is loaded by jetty that is loaded by sagetv). In order to use the _MOVEREC task I need a way to enable temporary write access to my nas. Maybe something like MOVERECLOGIN "user", "password".

The only other way is with a batch file, that launch psexec, move the file and maybe a _RELINK task?

Any idea?

Mike

[Slugger]

You're going to have to write a batch file as the Java API does not provide any means to connect to a network share with credentials other than those used by the process (unless you were to connect via FTP or something, but I won't be adding that kind of support to the _MOVEREC task). So you'll need to use a batch file to move the recording to the NAS then use the _RELINK task to relink the airing info after you move it. Something like this maybe:

# Decide which to shows to copy to the NAS as needed;
# you should only move recordings with exactly one file segment because
# that's all that the _RELINK task can handle (due to Sage API limitations)
if [IsObjMediaFile == true && $.GetShowTitle{} == "Show to Archive" && $.GetNumberOfSegments{} == 1] {
   COPYREC
   :RELINKOPTS "$.GetAiringID{}" "\\\\nas\\tv\\%p%"
   # Replace the path above as needed
   _RELINK
}

NOTE: Above ruleset completely untested, you will want to test accordingly before running it against all your recordings!

Then just write a batch file or whatever that copies the recording to your NAS. As long as that task returns a zero code back to SJQ then _RELINK task will run after it to relink the airing info to the file in the new location (and it will also delete the original recording).

Another idea just came to me: I'm not a Windows guru so this is just a thought that popped into my head, but couldn't you also map your NAS to a different drive letter, say Z: or something, and create that mapping as a different user with write permission to the NAS then use _MOVEREC and copy everything via the Z: and that should work as well? Don't know, just throwing it out there.

[Slugger]

SJQv3 is complete. Thanks to all who helped test during the beta cycle. I'll be hanging around the forums during the holidays, but am taking a holiday break from coding, tweaking, etc. so any bugs, feature requests, etc. should be added to the project issue tracker. Major issues would probably be addressed in a timely fashion, but hopefully we've squashed all of those types of problems during the beta cycle.

I plan on starting up with Sage coding again sometime after the new year (late January 2010).

[loonsailor]

Thanks for the work on this!

I upgraded yesterday from v2 to what is, I guess, the final beta of v3. Works fine, except that my system now runs only one task at a time, instead of running two parallel tasks. Here's the client configuration, which I haven't changed in v3.

:MAXPROCS 2

COMSKIP {
   :MAX 2
   :CPU "LOW"
   :RETURNCODE 1

   "/usr/bin/wine comskip %c%"
}

Two questions. First, what is MAXPROCS? I did this a while ago, and don't remember why it's there. I can't find any reference to it in the v2 or v3 manuals. Should I just delete it? Second, is a RESOURCES tag required, and how does it interact with multiple cores? If I want one task to run per core, should I specify (or default to) RESOURCES == 100 or, in my case, should I use 50 to allow two tasks to run in parallel, each using one core of my 4-core CPU?

[razrsharpe]

by default (when RESOURCES is not specified) all tasks will run serially because a value of RESOURCES = 100 is assigned to it. SJQ is not aware of how many processors you have (it is system based) so it is up to the user to do the math to figure out how many resources a given task should consume.

Not sure what MAXPROCS was but i dont think its there anymore (was not a SJQ v2 user). Use MAX to specify the number of instances of this task a client will run concurrently, regardless of how many RESOURCES it consumes (must still add up to less then 100).

Between RESOURCES and MAX you should be able to configure your tasks to do exactly what you want (btw thanks again to Slugger for implementing this )


SJQ Manual - MAX
SJQ Manual - RESOURCES

[loonsailor]

Thanks,razr. So, aren't MAX and RESOURCES kind of redundant? It seems like I could use either (MAX=2 RESOURCES=0), or (MAX undefined RESOURCES=50) to make sure that no more than two instances would run. Right?

[razrsharpe]

RESOURCES are much more configurable when running multiple different types of job. MAX doesnt really need to be used except in some very limited use cases. Here is is an example of when you want to use both MAX and RESOURCES (and the reason why Slugger added MAX back into the mix).

I would default to always using RESOURCES unless a specific use case required you to use both.

[mikep]

Hi,
I have update to the latest release and my active queue is waiting forever.
My active queue contain 8 jobs waiting and my client have nothing to do.
Any help?

2009-12-16 13:41:19,431 INFO  [DataStore]: Thread 'SJQ-LogCleaner' is waiting for an exclusive DataStore connection! [AVAIL=1; MAX=25; WAIT=923313ms]
2009-12-16 13:41:27,197 INFO  [SJQServlet]: Servlet POST request completed [16ms]
2009-12-16 13:41:27,197 INFO  [DataStore]: Thread 'SJQ-LogCleaner' is waiting for an exclusive DataStore connection! [AVAIL=1; MAX=25; WAIT=931079ms]
2009-12-16 13:41:27,197 INFO  [SJQServlet]: Servlet POST request completed [0ms]
2009-12-16 13:41:27,197 INFO  [DataStore]: Thread 'SJQ-LogCleaner' is waiting for an exclusive DataStore connection! [AVAIL=1; MAX=25; WAIT=931079ms]
2009-12-16 13:41:29,415 INFO  [SJQServlet]: Servlet POST request completed [0ms]
2009-12-16 13:41:29,415 INFO  [DataStore]: Thread 'SJQ-LogCleaner' is waiting for an exclusive DataStore connection! [AVAIL=1; MAX=25; WAIT=933297ms]
2009-12-16 13:41:31,181 INFO  [InternalTaskClient]: Run completed [0ms]
2009-12-16 13:41:39,431 INFO  [SJQServlet]: Servlet POST request completed [0ms]
2009-12-16 13:41:39,431 INFO  [DataStore]: Thread 'SJQ-LogCleaner' is waiting for an exclusive DataStore connection! [AVAIL=1; MAX=25; WAIT=943313ms]

[mikep]

Originally Posted by Slugger

Another idea just came to me: I'm not a Windows guru so this is just a thought that popped into my head, but couldn't you also map your NAS to a different drive letter, say Z: or something, and create that mapping as a different user with write permission to the NAS then use _MOVEREC and copy everything via the Z: and that should work as well? Don't know, just throwing it out there.

It is a good idea BUT is the resulting media in sage will be linked to drive Z:? If so, my kids will be able to delete those, they will have write access to the nas.

Am I right?

Mike

[Slugger]

Originally Posted by mikep

Hi,
I have update to the latest release and my active queue is waiting forever.
My active queue contain 8 jobs waiting and my client have nothing to do.
Any help?

2009-12-16 13:41:19,431 INFO  [DataStore]: Thread 'SJQ-LogCleaner' is waiting for an exclusive DataStore connection! [AVAIL=1; MAX=25; WAIT=923313ms]
2009-12-16 13:41:27,197 INFO  [SJQServlet]: Servlet POST request completed [16ms]
2009-12-16 13:41:27,197 INFO  [DataStore]: Thread 'SJQ-LogCleaner' is waiting for an exclusive DataStore connection! [AVAIL=1; MAX=25; WAIT=931079ms]
2009-12-16 13:41:27,197 INFO  [SJQServlet]: Servlet POST request completed [0ms]
2009-12-16 13:41:27,197 INFO  [DataStore]: Thread 'SJQ-LogCleaner' is waiting for an exclusive DataStore connection! [AVAIL=1; MAX=25; WAIT=931079ms]
2009-12-16 13:41:29,415 INFO  [SJQServlet]: Servlet POST request completed [0ms]
2009-12-16 13:41:29,415 INFO  [DataStore]: Thread 'SJQ-LogCleaner' is waiting for an exclusive DataStore connection! [AVAIL=1; MAX=25; WAIT=933297ms]
2009-12-16 13:41:31,181 INFO  [InternalTaskClient]: Run completed [0ms]
2009-12-16 13:41:39,431 INFO  [SJQServlet]: Servlet POST request completed [0ms]
2009-12-16 13:41:39,431 INFO  [DataStore]: Thread 'SJQ-LogCleaner' is waiting for an exclusive DataStore connection! [AVAIL=1; MAX=25; WAIT=943313ms]

Restart SJQ. It seems like you have threads that are dying off for some reason because the db pool has added 20 connections and yet only 1 is free. Check the rest of the logs to see if there's an exception dump. Check your task client logs to see if it's receiving or generating exceptions. Something is crashing that's causing the db pool to be depleted.

[Slugger]

Originally Posted by mikep

It is a good idea BUT is the resulting media in sage will be linked to drive Z:? If so, my kids will be able to delete those, they will have write access to the nas.

Am I right?

Mike

Shouldn't be, if Sage already links your media to a UNC path then you should just be able to map a drive that only SJQ uses. I'm not 100% sure that this will work, but it seems like it should. If not, then you'll have to write the batch file and do the move manually then use the _RELINK task to relink the airing info as needed.

[mikep]

Originally Posted by Slugger

Restart SJQ. It seems like you have threads that are dying off for some reason because the db pool has added 20 connections and yet only 1 is free. Check the rest of the logs to see if there's an exception dump. Check your task client logs to see if it's receiving or generating exceptions. Something is crashing that's causing the db pool to be depleted.

I found my error, in one of my client config rule, I had a line without a double quoted . It was written on your manual.... All input should be double quoted... I found the error by looking at the client view log menu....

Mike

[mikep]

Do you have an idea how can I write a rule that will be trigger only once a day, say like at 3:00AM?

Also is it possible to have one trigger after savetv has done is media scan? I had mine configure to run once a day (in sagetv: Automatic library rescan frequency). I want to create a task that will run after sagetv has done it rescan job.

Thanks in advance.

Mike

[Slugger]

Originally Posted by mikep

I found my error, in one of my client config rule, I had a line without a double quoted . It was written on your manual.... All input should be double quoted... I found the error by looking at the client view log menu....

Mike

Can you provide a sample client config that was causing the problem? It seems SJQ wasn't properly recovering the db pool as intended so I'd like to reproduce this and have a closer look the next time I dig into the code.

[Slugger]

Originally Posted by mikep

Do you have an idea how can I write a rule that will be trigger only once a day, say like at 3:00AM?

You can't tell a task to run only at exactly 3am, but you could tell a client to only run a task between 3am and 4am and if that task were queued up and waiting in the task queue before 3am then I could guarantee the task would run between 3am and 3:05am. That's about the best you can do in SJQ because SJQ isn't crond.

Also is it possible to have one trigger after savetv has done is media scan? I had mine configure to run once a day (in sagetv: Automatic library rescan frequency). I want to create a task that will run after sagetv has done it rescan job.

I wish this were possible, but it's not. The Sage API does not provide a way to listen for and act upon key (or any) Sage events. I've submitted feature requests for this in the past, perhaps if you did the same then there might be a better chance to get this kind of feature added in the future.

[mikep]

Originally Posted by Slugger

Can you provide a sample client config that was causing the problem? It seems SJQ wasn't properly recovering the db pool as intended so I'd like to reproduce this and have a closer look the next time I dig into the code.

If you look carefully you will see the last line of TESTMEDIA is missing the double quoted, also on the last line of MEDIASHRINKSD. The other two were ok.

Also, there was 8 tasks waiting in the queue for TESTMEDIA at that moment

/* Task block for comskip */
COMSKIP
{
   :CPU NORMAL
   :RESOURCES 50 
   :MAX 2 // max 2 instance of this task
   :MAXRUNRATIO 4.0 // Up to 4 length of recording.
   :RETURNCODE 1
   "C:/comskip/comskip.exe \"%c%\""
}

/* Task block for MediaShrink HD resolution */
MEDIASHRINKHD
{   
   :NORECORDING 4.0
   :NOCLIENT "true"
   :MAX 1 // max 1 instance of this task
   :CPU LOW
   :RESOURCES 50 
   "C:\\MediaShrink\\mediaShrink.exe \"%c%\" /batch /InPlace /x264 /onePass /highProfile /cliBitrate 3250 /ac3 /mkv"
}

/* Task block for MediaShrink SD resolution*/
MEDIASHRINKSD
{   
   :NORECORDING 2.5
   :NOCLIENT "true"
   :CPU LOW
   :MAX 1 // max 1 instance of this task
   :RESOURCES 50 
   #C:/MediaShrink/mediaShrink.exe \"%c%\" /batch /InPlace /x264 /onePass /highProfile /cliBitrate 1500 /ac3 /mkv
   C:/MediaShrink/mediaEngine.exe /saveLog /defaultProfile autoEncode /findFileRegEx \"avi|mpg|mkv|mp4|mpeg|VIDEO_TS|ts|ogm|divx|mpts|dvr-ms\" /mediaShrink fixBug \"%c%\" /batch /InPlace /x264 /onePass /highProfile /cliBitrate 1500 /ac3 /mkv
}

TESTMEDIA
{
  :MAX 1
  :RESOURCES 5 
  R:/temp/testmedia.cmd \"%c%\"
}

[mikep]

Originally Posted by Slugger

I wish this were possible, but it's not. The Sage API does not provide a way to listen for and act upon key (or any) Sage events. I've submitted feature requests for this in the past, perhaps if you did the same then there might be a better chance to get this kind of feature added in the future.

I will submit a feature requests too.

[toricred]

I just installed SJQ 3.0 and I'm impressed. I think I can use it to replace by delicate DTB install. I've run into three problems. One is that I set my password and logged off and back on a couple of times so I'm certain that I know what the password is, but now it is saying that the password is invalid. Second I've set it up with resources at 33 for SAV1 and it's only queueing up one recording. Third is that as it was just about to finish the first SAV1 the Active queue suddenly went blank. That was made me log off and try to log back in causing the first problem. Obviously the password issue is first and foremost.

Edit: I don't know what happened, but I noticed that the client service was stopped and started it again. Then I could log in and the jobs are in the queue. Now I just have this problem with only launching 1 job.