Plugin: Sage Job Queue (SJQ) v3

[Slugger]

The server logs should tell you why it's only running 1 job at a time (typically it'll be because of lack of resources available or :MAX tasks already running). What do the server logs say?

[toricred]

OK, I've figured it out and the problem is more in the way I'm calling SAV1. It is calling them with the ShowAnalyzer.exe program. This adds them to the queue and SAV1 takes over the management and tells SJQ that it's done its job. Now I have every recording set in SAV1's queue while it works through them seeing which ones already have edl files and skipping those. I had forgotten about how SAV1 works when it is called via Showanalyzer.exe.

[Slugger]

Originally Posted by Slugger

Can you provide a sample client config that was causing the problem? It seems SJQ wasn't properly recovering the db pool as intended so I'd like to reproduce this and have a closer look the next time I dig into the code.

Originally Posted by mikep

If you look carefully you will see the last line of TESTMEDIA is missing the double quoted, also on the last line of MEDIASHRINKSD. The other two were ok.

Also, there was 8 tasks waiting in the queue for TESTMEDIA at that moment

/* Task block for comskip */
COMSKIP
{
   :CPU NORMAL
   :RESOURCES 50 
   :MAX 2 // max 2 instance of this task
   :MAXRUNRATIO 4.0 // Up to 4 length of recording.
   :RETURNCODE 1
   "C:/comskip/comskip.exe \"%c%\""
}

/* Task block for MediaShrink HD resolution */
MEDIASHRINKHD
{   
   :NORECORDING 4.0
   :NOCLIENT "true"
   :MAX 1 // max 1 instance of this task
   :CPU LOW
   :RESOURCES 50 
   "C:\\MediaShrink\\mediaShrink.exe \"%c%\" /batch /InPlace /x264 /onePass /highProfile /cliBitrate 3250 /ac3 /mkv"
}

/* Task block for MediaShrink SD resolution*/
MEDIASHRINKSD
{   
   :NORECORDING 2.5
   :NOCLIENT "true"
   :CPU LOW
   :MAX 1 // max 1 instance of this task
   :RESOURCES 50 
   #C:/MediaShrink/mediaShrink.exe \"%c%\" /batch /InPlace /x264 /onePass /highProfile /cliBitrate 1500 /ac3 /mkv
   C:/MediaShrink/mediaEngine.exe /saveLog /defaultProfile autoEncode /findFileRegEx \"avi|mpg|mkv|mp4|mpeg|VIDEO_TS|ts|ogm|divx|mpts|dvr-ms\" /mediaShrink fixBug \"%c%\" /batch /InPlace /x264 /onePass /highProfile /cliBitrate 1500 /ac3 /mkv
}

TESTMEDIA
{
  :MAX 1
  :RESOURCES 5 
  R:/temp/testmedia.cmd \"%c%\"
}

Ok, I am able to reproduce this issue locally. The problem is that I implemented the "DataStore connection replenisher" scheme completely wrong and basically under certain conditions, a client config syntax error being one of them, a timed out DataStore connection will get lost and not replenished. And if the pool is not ever replenished to 100% at some point then those worker threads requiring exclusive connections will never get them and starvation occurs (i.e. those worker threads will sleep for ever).

This is a big enough problem to warrant a 3.0.1 release before my Christmas hiatus. I'm testing the fix now and will make the release hopefully sometime today.

[Slugger]

Originally Posted by Slugger

Ok, I am able to reproduce this issue locally. The problem is that I implemented the "DataStore connection replenisher" scheme completely wrong and basically under certain conditions, a client config syntax error being one of them, a timed out DataStore connection will get lost and not replenished. And if the pool is not ever replenished to 100% at some point then those worker threads requiring exclusive connections will never get them and starvation occurs (i.e. those worker threads will sleep for ever).

This is a big enough problem to warrant a 3.0.1 release before my Christmas hiatus. I'm testing the fix now and will make the release hopefully sometime today.

SJQ v3.0.1 released. This release contains just one change: the fix for the above mentioned bug. However, this is a big enough bug that I highly recommend everyone upgrade immediately b/c if you trigger one of the conditions that causes this bug (client config syntax error is NOT the only way this bug can be triggered), your SJQ server will not function properly (it won't do any damage, the CPU won't spike, etc, but threads will just go to sleep and never wake up).

[ehfortin]

Hi team,

I" having a problem with a rule and I would like to know if there is a way to actually print the content of getAiringChannelNumber and getAiringChannelName? I"m under the impression those fields are not populated for some shows and want to make sure of this. Right now, I"m trying to just create a rule for station that have adds but... it is returning the whole list of recordings back as apparently the channel number are always out of the range I"ve specified (also tried by name) while in the reality, most of the recordings come from the same few stations so... I should be able to reach them easily.

So, I just want to know how to print the value of the methods (like getAiring...) in the log or anywhere else.

I just went back in SageTV remembering that I should be able to see the channel name in there. I have recordings that shows a "-". Anybody has an idea why the channel has been replaced/erased? I"m using BMT which is the only one I think could have change metadata but I would have expected channel recorded from to be a fixed field (at least, I can"t change it manually in BMT as far as I know).

Thank you.

ehfortin

[razrsharpe]

bmt might overwrite that bit of data... what happens on shows that are recorded but have not have their metadata updated by bmt?

getAiringChannelNumber and Name should exist for all channels... definitely the number not 100% positive on the Name. If you go to your epg that will show you what the ChannelNumber and ChannelName are.

[Slugger]

I see the same thing for some of my recordings. The channel info is lost on any recordings that I've manually updated the fanart for via BMT web UI. I believe because doing it through the web UI overwrites the metadata as well as downloads the fanart so when the metadata is overwritten then the channel info is lost. I mostly do this for movies I've recorded because the automatic call I do via a command line in SJQ never seems to work for movies recorded from tv.

[ehfortin]

I just did a quick check. I know I have not used BMT to update metadata for a few days. I have a few shows that still have the channelnumber (those that have no fanart/background so, basically, that have not been found in an external DB) and I have one show that have been recorded two times (once per day) where the title and description is all there but where the channelnumber is now a "-".

I have the automatic metadata activated but the option for overwrite fanart and overwrite metadata are unchecked. Should not change the channelnumber based on this as it is already existing metadata coming from SageTV, right?

I"ll deactivate the automatic plugin to see if the same thing happens again for that specific show at the next recording.

Outside of this, any idea? Can I find back the information somewhere else or recreate it?

Thanks.

ehfortin

[Slugger]

You might be able to recreate it by importing the info back in via the web server plugin. Alternatively, if you happen to know the airing id for the show you could relink it via Studio calls, if Sage hasn't already purged the airing info (if airing info isn't linked to an available media file and the show is not marked as watched or don't like then Sage will purge the airing info on a future EPG update).

[lovingHDTV]

Originally Posted by Slugger

Instead of writing a batch file, just add the :RETURNCODE 1 option to the client where you call comskip. That will tell SJQ that it should expect a return code of 1 instead of the default 0. No need to write a batch file anymore to work around this issue.

Any possibility to get this example added to the documentation where you teach us how to build a comskip client? Seems a perfect place to showcase this feature

I just came to the forum to learn this very thing after following the User's Guide to setup SJQ and comskip.

Very nice work, much improved since the last time I played with it a year or so ago.

dave

[Slugger]

Originally Posted by lovingHDTV

Any possibility to get this example added to the documentation where you teach us how to build a comskip client? Seems a perfect place to showcase this feature

I just came to the forum to learn this very thing after following the User's Guide to setup SJQ and comskip.

Very nice work, much improved since the last time I played with it a year or so ago.

dave

Example updated in docs...

[gasinger]

Is there a way to configure the amount of time the client tasks are started from the queue? I'm able to create the task, but it may take up to 5 minutes before the task process is started. I' trying to respond to a system message that an encoder failed and want to run a batch script as soon as possible.

Thanks,
Gregg

[Tropicaljoe]

Can anyone tell me if a restore of the client or sever configs is possible? I had my configs setup perfectly and re-formatted the drive. I do have a full backup copy of the drive though.

[Slugger]

Originally Posted by gasinger

Is there a way to configure the amount of time the client tasks are started from the queue? I'm able to create the task, but it may take up to 5 minutes before the task process is started. I' trying to respond to a system message that an encoder failed and want to run a batch script as soon as possible.

Thanks,
Gregg

No, unfortunately. Task clients call into the server every five minutes to check if there's new work for them to do. I suppose a feature request to make this configurable might be addressed, if created, but not until after the new year (I'm on a holiday hiatus from Sage tweaking/coding/anything other than using SageTV ).

[Slugger]

Originally Posted by Tropicaljoe

Can anyone tell me if a restore of the client or sever configs is possible? I had my configs setup perfectly and re-formatted the drive. I do have a full backup copy of the drive though.

Restore the sjq.sqlite file - that's the file with all the settings, rulesets, client configs etc. Make sure to stop SageTV before replacing that file.

[Tropicaljoe]

Originally Posted by Slugger

Restore the sjq.sqlite file - that's the file with all the settings, rulesets, client configs etc. Make sure to stop SageTV before replacing that file.

Thanks for the answer. One more question for you, I have a SJQ command to find all files older than 30 days and transcode them ( adds them to the native sage que fine), the issue is that all files it finds get dumped into Sages que, which is a ton. Can the command be changed to allow it to find the first file and only process that file until the SJQ process que runs again ( which will find the next, and so on and so on) this would avoid having to transcode 50 shows all at once as it would eventually stop processing when the time on the original command is past its run time ( I have it set to run between 2am and 8am ). Or this there something I can put in the command to wait until it gets an "all clear" from sage letting it know that the file is done and then send the next file to it

[Slugger]

Originally Posted by Tropicaljoe

Thanks for the answer. One more question for you, I have a SJQ command to find all files older than 30 days and transcode them ( adds them to the native sage que fine), the issue is that all files it finds get dumped into Sages que, which is a ton. Can the command be changed to allow it to find the first file and only process that file until the SJQ process que runs again ( which will find the next, and so on and so on) this would avoid having to transcode 50 shows all at once as it would eventually stop processing when the time on the original command is past its run time ( I have it set to run between 2am and 8am ). Or this there something I can put in the command to wait until it gets an "all clear" from sage letting it know that the file is done and then send the next file to it

Use the TranscodeQueueSize test:

# Only load one transcode job into the queue at a time...
if [IsObjMediaFile == true && $.GetShowTitle{} == "Whatever" && TranscodeQueueSize < 1] {
   _TRANSCODE
}

[Tropicaljoe]

Originally Posted by Slugger

Use the TranscodeQueueSize test:

# Only load one transcode job into the queue at a time...
if [IsObjMediaFile == true && $.GetShowTitle{} == "Whatever" && TranscodeQueueSize < 1] {
   _TRANSCODE
}

Thanks very much, this is the greatest thing since sliced bread! as a former beyondTv user, this convinced me to make the switch, good work!

[Tropicaljoe]

Originally Posted by Slugger

Use the TranscodeQueueSize test:

# Only load one transcode job into the queue at a time...
if [IsObjMediaFile == true && $.GetShowTitle{} == "Whatever" && TranscodeQueueSize < 1] {
   _TRANSCODE
}

Slugger, I use the following:
# Convert files older than 2 days
if [IsObjMediaFile == true && $.IsTVFile{} == true && MediaAge >= "10D" && Filename =$ ".mpg" && TranscodeQueueSize < 1 && DayOfWeek == 1 && TimeOfDay >= "2:00" && TimeOfDay < "9:00"] { # Only transcode on Sundays between 2am and 9am


:TRANSCODEOPTS "MPEG4-Good Quality AVI" "false"
_TRANSCODE
}

This still seems to find all files older than 10 days and dump them into the internal Que for transcoding, I was hoping to dump one file at a time to the internal Que, this would allow it to follow the time constraints I have, otherwise it just dumps all fifty files into the Que, which may run for 24 hours ( well outside the time constraints setup). Is there any way to have it submit one file at a time to the Que? My idea behind this is to transcode any file over the age limit, but only in the time constraints setup, even if it means that it has to que items up and run them several nights in a row or every night to get it completed

[Slugger]

Open an issue ticket asking for the TranscodeQueueSize test to account for the number of pending jobs queued up in SJQ. I won't be coding again until mid January, but could add this feature when I get back into it.

Basically the test could be changed to look at the Sage queue (as it currently does), but then also add any jobs pending in the SJQ queue so basically it'd only queue up one at a time as you want. You'll need to add this ticket or else I'll likely forget about this come mid January.