SageTV Community

SageTV Community (http://forums.sagetv.com/forums/index.php)
-   SageTV Customizations (http://forums.sagetv.com/forums/forumdisplay.php?f=27)
-   -   Plugin: Sage Job Queue (SJQ) v3 (http://forums.sagetv.com/forums/showthread.php?t=45174)

Slugger 12-16-2009 09:30 PM

The server logs should tell you why it's only running 1 job at a time (typically it'll be because of lack of resources available or :MAX tasks already running). What do the server logs say?

toricred 12-16-2009 09:45 PM

OK, I've figured it out and the problem is more in the way I'm calling SAV1. It is calling them with the ShowAnalyzer.exe program. This adds them to the queue and SAV1 takes over the management and tells SJQ that it's done its job. Now I have every recording set in SAV1's queue while it works through them seeing which ones already have edl files and skipping those. I had forgotten about how SAV1 works when it is called via Showanalyzer.exe.

Slugger 12-17-2009 08:05 AM

Quote:

Originally Posted by Slugger (Post 398208)
Can you provide a sample client config that was causing the problem? It seems SJQ wasn't properly recovering the db pool as intended so I'd like to reproduce this and have a closer look the next time I dig into the code.

Quote:

Originally Posted by mikep (Post 398211)
If you look carefully you will see the last line of TESTMEDIA is missing the double quoted, also on the last line of MEDIASHRINKSD. The other two were ok.

Also, there was 8 tasks waiting in the queue for TESTMEDIA at that moment

Code:

/* Task block for comskip */
COMSKIP
{
  :CPU NORMAL
  :RESOURCES 50
  :MAX 2 // max 2 instance of this task
  :MAXRUNRATIO 4.0 // Up to 4 length of recording.
  :RETURNCODE 1
  "C:/comskip/comskip.exe \"%c%\""
}

/* Task block for MediaShrink HD resolution */
MEDIASHRINKHD

  :NORECORDING 4.0
  :NOCLIENT "true"
  :MAX 1 // max 1 instance of this task
  :CPU LOW
  :RESOURCES 50
  "C:\\MediaShrink\\mediaShrink.exe \"%c%\" /batch /InPlace /x264 /onePass /highProfile /cliBitrate 3250 /ac3 /mkv"
}

/* Task block for MediaShrink SD resolution*/
MEDIASHRINKSD

  :NORECORDING 2.5
  :NOCLIENT "true"
  :CPU LOW
  :MAX 1 // max 1 instance of this task
  :RESOURCES 50
  #C:/MediaShrink/mediaShrink.exe \"%c%\" /batch /InPlace /x264 /onePass /highProfile /cliBitrate 1500 /ac3 /mkv
  C:/MediaShrink/mediaEngine.exe /saveLog /defaultProfile autoEncode /findFileRegEx \"avi|mpg|mkv|mp4|mpeg|VIDEO_TS|ts|ogm|divx|mpts|dvr-ms\" /mediaShrink fixBug \"%c%\" /batch /InPlace /x264 /onePass /highProfile /cliBitrate 1500 /ac3 /mkv
}

TESTMEDIA
{
  :MAX 1
  :RESOURCES 5
  R:/temp/testmedia.cmd \"%c%\"
}


Ok, I am able to reproduce this issue locally. The problem is that I implemented the "DataStore connection replenisher" scheme completely wrong and basically under certain conditions, a client config syntax error being one of them, a timed out DataStore connection will get lost and not replenished. And if the pool is not ever replenished to 100% at some point then those worker threads requiring exclusive connections will never get them and starvation occurs (i.e. those worker threads will sleep for ever).

This is a big enough problem to warrant a 3.0.1 release before my Christmas hiatus. I'm testing the fix now and will make the release hopefully sometime today.

Slugger 12-17-2009 10:05 AM

Quote:

Originally Posted by Slugger (Post 398268)
Ok, I am able to reproduce this issue locally. The problem is that I implemented the "DataStore connection replenisher" scheme completely wrong and basically under certain conditions, a client config syntax error being one of them, a timed out DataStore connection will get lost and not replenished. And if the pool is not ever replenished to 100% at some point then those worker threads requiring exclusive connections will never get them and starvation occurs (i.e. those worker threads will sleep for ever).

This is a big enough problem to warrant a 3.0.1 release before my Christmas hiatus. I'm testing the fix now and will make the release hopefully sometime today.

SJQ v3.0.1 released. This release contains just one change: the fix for the above mentioned bug. However, this is a big enough bug that I highly recommend everyone upgrade immediately b/c if you trigger one of the conditions that causes this bug (client config syntax error is NOT the only way this bug can be triggered), your SJQ server will not function properly (it won't do any damage, the CPU won't spike, etc, but threads will just go to sleep and never wake up).

ehfortin 12-17-2009 03:12 PM

Hi team,

I" having a problem with a rule and I would like to know if there is a way to actually print the content of getAiringChannelNumber and getAiringChannelName? I"m under the impression those fields are not populated for some shows and want to make sure of this. Right now, I"m trying to just create a rule for station that have adds but... it is returning the whole list of recordings back as apparently the channel number are always out of the range I"ve specified (also tried by name) while in the reality, most of the recordings come from the same few stations so... I should be able to reach them easily.

So, I just want to know how to print the value of the methods (like getAiring...) in the log or anywhere else.

I just went back in SageTV remembering that I should be able to see the channel name in there. I have recordings that shows a "-". Anybody has an idea why the channel has been replaced/erased? I"m using BMT which is the only one I think could have change metadata but I would have expected channel recorded from to be a fixed field (at least, I can"t change it manually in BMT as far as I know).

Thank you.

ehfortin

razrsharpe 12-17-2009 03:15 PM

bmt might overwrite that bit of data... what happens on shows that are recorded but have not have their metadata updated by bmt?

getAiringChannelNumber and Name should exist for all channels... definitely the number not 100% positive on the Name. If you go to your epg that will show you what the ChannelNumber and ChannelName are.

Slugger 12-17-2009 04:03 PM

I see the same thing for some of my recordings. The channel info is lost on any recordings that I've manually updated the fanart for via BMT web UI. I believe because doing it through the web UI overwrites the metadata as well as downloads the fanart so when the metadata is overwritten then the channel info is lost. I mostly do this for movies I've recorded because the automatic call I do via a command line in SJQ never seems to work for movies recorded from tv.

ehfortin 12-17-2009 04:05 PM

I just did a quick check. I know I have not used BMT to update metadata for a few days. I have a few shows that still have the channelnumber (those that have no fanart/background so, basically, that have not been found in an external DB) and I have one show that have been recorded two times (once per day) where the title and description is all there but where the channelnumber is now a "-".

I have the automatic metadata activated but the option for overwrite fanart and overwrite metadata are unchecked. Should not change the channelnumber based on this as it is already existing metadata coming from SageTV, right?

I"ll deactivate the automatic plugin to see if the same thing happens again for that specific show at the next recording.

Outside of this, any idea? Can I find back the information somewhere else or recreate it?

Thanks.

ehfortin

Slugger 12-17-2009 04:14 PM

You might be able to recreate it by importing the info back in via the web server plugin. Alternatively, if you happen to know the airing id for the show you could relink it via Studio calls, if Sage hasn't already purged the airing info (if airing info isn't linked to an available media file and the show is not marked as watched or don't like then Sage will purge the airing info on a future EPG update).

lovingHDTV 12-17-2009 06:55 PM

Quote:

Originally Posted by Slugger (Post 397974)
Instead of writing a batch file, just add the :RETURNCODE 1 option to the client where you call comskip. That will tell SJQ that it should expect a return code of 1 instead of the default 0. No need to write a batch file anymore to work around this issue.

Any possibility to get this example added to the documentation where you teach us how to build a comskip client? Seems a perfect place to showcase this feature :D

I just came to the forum to learn this very thing after following the User's Guide to setup SJQ and comskip.

Very nice work, much improved since the last time I played with it a year or so ago.

dave

Slugger 12-18-2009 01:15 PM

Quote:

Originally Posted by lovingHDTV (Post 398368)
Any possibility to get this example added to the documentation where you teach us how to build a comskip client? Seems a perfect place to showcase this feature :D

I just came to the forum to learn this very thing after following the User's Guide to setup SJQ and comskip.

Very nice work, much improved since the last time I played with it a year or so ago.

dave

Example updated in docs...

gasinger 12-18-2009 02:20 PM

Is there a way to configure the amount of time the client tasks are started from the queue? I'm able to create the task, but it may take up to 5 minutes before the task process is started. I' trying to respond to a system message that an encoder failed and want to run a batch script as soon as possible.

Thanks,
Gregg

Tropicaljoe 12-18-2009 02:30 PM

Restore
 
Can anyone tell me if a restore of the client or sever configs is possible? I had my configs setup perfectly and re-formatted the drive. I do have a full backup copy of the drive though.:)

Slugger 12-18-2009 03:22 PM

Quote:

Originally Posted by gasinger (Post 398455)
Is there a way to configure the amount of time the client tasks are started from the queue? I'm able to create the task, but it may take up to 5 minutes before the task process is started. I' trying to respond to a system message that an encoder failed and want to run a batch script as soon as possible.

Thanks,
Gregg

No, unfortunately. Task clients call into the server every five minutes to check if there's new work for them to do. I suppose a feature request to make this configurable might be addressed, if created, but not until after the new year (I'm on a holiday hiatus from Sage tweaking/coding/anything other than using SageTV :) ).

Slugger 12-18-2009 03:24 PM

Quote:

Originally Posted by Tropicaljoe (Post 398457)
Can anyone tell me if a restore of the client or sever configs is possible? I had my configs setup perfectly and re-formatted the drive. I do have a full backup copy of the drive though.:)

Restore the sjq.sqlite file - that's the file with all the settings, rulesets, client configs etc. Make sure to stop SageTV before replacing that file.

Tropicaljoe 12-27-2009 01:35 AM

_transcode que
 
Quote:

Originally Posted by Slugger (Post 398468)
Restore the sjq.sqlite file - that's the file with all the settings, rulesets, client configs etc. Make sure to stop SageTV before replacing that file.

Thanks for the answer.:) One more question for you, I have a SJQ command to find all files older than 30 days and transcode them ( adds them to the native sage que fine), the issue is that all files it finds get dumped into Sages que, which is a ton. Can the command be changed to allow it to find the first file and only process that file until the SJQ process que runs again ( which will find the next, and so on and so on) this would avoid having to transcode 50 shows all at once as it would eventually stop processing when the time on the original command is past its run time ( I have it set to run between 2am and 8am ). Or this there something I can put in the command to wait until it gets an "all clear" from sage letting it know that the file is done and then send the next file to it

Slugger 12-27-2009 09:55 AM

Quote:

Originally Posted by Tropicaljoe (Post 399632)
Thanks for the answer.:) One more question for you, I have a SJQ command to find all files older than 30 days and transcode them ( adds them to the native sage que fine), the issue is that all files it finds get dumped into Sages que, which is a ton. Can the command be changed to allow it to find the first file and only process that file until the SJQ process que runs again ( which will find the next, and so on and so on) this would avoid having to transcode 50 shows all at once as it would eventually stop processing when the time on the original command is past its run time ( I have it set to run between 2am and 8am ). Or this there something I can put in the command to wait until it gets an "all clear" from sage letting it know that the file is done and then send the next file to it

Use the TranscodeQueueSize test:

Code:

# Only load one transcode job into the queue at a time...
if [IsObjMediaFile == true && $.GetShowTitle{} == "Whatever" && TranscodeQueueSize < 1] {
  _TRANSCODE
}


Tropicaljoe 12-27-2009 11:30 AM

Quote:

Originally Posted by Slugger (Post 399654)
Use the TranscodeQueueSize test:

Code:

# Only load one transcode job into the queue at a time...
if [IsObjMediaFile == true && $.GetShowTitle{} == "Whatever" && TranscodeQueueSize < 1] {
  _TRANSCODE
}


Thanks very much, this is the greatest thing since sliced bread! as a former beyondTv user, this convinced me to make the switch, good work!
:clap:

Tropicaljoe 12-27-2009 11:10 PM

Quote:

Originally Posted by Slugger (Post 399654)
Use the TranscodeQueueSize test:

Code:

# Only load one transcode job into the queue at a time...
if [IsObjMediaFile == true && $.GetShowTitle{} == "Whatever" && TranscodeQueueSize < 1] {
  _TRANSCODE
}


Slugger, I use the following:
# Convert files older than 2 days
if [IsObjMediaFile == true && $.IsTVFile{} == true && MediaAge >= "10D" && Filename =$ ".mpg" && TranscodeQueueSize < 1 && DayOfWeek == 1 && TimeOfDay >= "2:00" && TimeOfDay < "9:00"] { # Only transcode on Sundays between 2am and 9am


:TRANSCODEOPTS "MPEG4-Good Quality AVI" "false"
_TRANSCODE
}

This still seems to find all files older than 10 days and dump them into the internal Que for transcoding, I was hoping to dump one file at a time to the internal Que, this would allow it to follow the time constraints I have, otherwise it just dumps all fifty files into the Que, which may run for 24 hours ( well outside the time constraints setup). Is there any way to have it submit one file at a time to the Que? My idea behind this is to transcode any file over the age limit, but only in the time constraints setup, even if it means that it has to que items up and run them several nights in a row or every night to get it completed

Slugger 12-28-2009 12:39 AM

Open an issue ticket asking for the TranscodeQueueSize test to account for the number of pending jobs queued up in SJQ. I won't be coding again until mid January, but could add this feature when I get back into it.

Basically the test could be changed to look at the Sage queue (as it currently does), but then also add any jobs pending in the SJQ queue so basically it'd only queue up one at a time as you want. You'll need to add this ticket or else I'll likely forget about this come mid January.


All times are GMT -6. The time now is 12:12 PM.

Powered by vBulletin® Version 3.8.7
Copyright ©2000 - 2019, vBulletin Solutions, Inc.
Copyright 2003-2005 SageTV, LLC. All rights reserved.