Plugin: Sage Job Queue (SJQ) v3

[Slugger]

Originally Posted by nixter80

Hi,

I was wondering if it is possible to get a task to re-run at regular intervals.

I have a situation where I have multiple sagetv clients at different locations, and if one location watches a show it becomes maked as watched and then is deleted before the older unwatched shows.

I was hoping to run job that every hour removes the watched status from all shows but alas if the job has already queued once for a show it will not again unless i remove it manually from the job log.

Any ideas would be great.

Thanks

There is no way to do this for internal tasks at this time. You may want to open a ticket at the project site as a reminder for me to look at this.

[razrsharpe]

Originally Posted by nixter80

Hi,

I was wondering if it is possible to get a task to re-run at regular intervals.

I have a situation where I have multiple sagetv clients at different locations, and if one location watches a show it becomes maked as watched and then is deleted before the older unwatched shows.

I was hoping to run job that every hour removes the watched status from all shows but alas if the job has already queued once for a show it will not again unless i remove it manually from the job log.

Any ideas would be great.

Thanks

you could also set up the favorite to never auto-delete those recordings...

[doncote0]

Hi,

Is it possible to run multiple filters for kicking of a transcode job?

I am trying to do this...

# Convert 1hr or longer HD files
if [IsObjMediaFile == true && IsViewingMedia == false && Filename =$ ".ts" && MediaSize > 4000000000 && $.IsFileCurrentlyRecording{} == false]
{ # Use timeOfDay with quoted times in if statement above
:PRIORITY 300
:TRANSCODEOPTS "AppleTV-High Quality" "false"
_TRANSCODE
}

but also allow this

# Convert files older than 30 days
if [IsObjMediaFile == true && MediaAge >= "30D" && Filename =$ ".mpg" && FileExists = "%d%/%p%.edl"]
{ # Use timeOfDay with quoted times in if statement above
:PRIORITY 30
:TRANSCODEOPTS "iPhone-Standard" "false"
_TRANSCODE

and perhaps this

# Convert shorter HD files
if [IsObjMediaFile == true && IsViewingMedia == false && Filename =$ ".ts" && MediaSize <= 4000000000 && $.IsFileCurrentlyRecording{} == false]
{ # Use timeOfDay with quoted times in if statement above
:PRIORITY 200
:TRANSCODEOPTS "iPhone-Widescreen" "false"
_TRANSCODE
}

If not, what would you suggest to accomplish the same tasks?

Thank you.

[Slugger]

You can have all three of those rules, but any one recording is only ever going to run one of those transcode tasks unless the transcoded output retains the same file extension, but you don't want to do that otherwise SJQ will constantly retranscode the same file over and over again. If you want to handle different files with those different rules then that's fine and should work as intended.

One note, though. Change the value of your MediaSize test to "4G" as that test expects a string with a multiplier instead of an actual number value so:

MediaSize > "4G"

instead of:

MediaSize > 4000000000

Where G is gigabyte, B is byte, M is megabyte, and T is terabyte.

[doncote0]

I'll give it a try with "4G" instead of 4000000000.

[phelme]

So I've never noticed this before, but running files through MediaShrink via SJQ causes the "Encoded by: (tuner)" information to be lost on the media. (Kind of a bummer, especially down the road if one needs to debug a particular tuner.) If I use Sage's built-in transcoder, with _TRANSCODE, it's retained.

I don't think I'm doing anything out of the ordinary, passing /inPlace /batch /onePass on the command line. It probably has to do with the way MediaShrink disposes of the original file. Is this a known issue?

[Slugger]

It's probably more to do with how the core handles finding the replacement file. I suspect when you use the Sage transcoder it does things to ensure all metadata for the recording is maintained. When you transcode via an external process the core probably first loses the original media file, deletes it from wiz.bin, then finds the new one and replaces it. I suspect that process loses some of the metadata about the original recording, namely the tuner used to record (probably among other things).

[phelme]

Originally Posted by Slugger

It's probably more to do with how the core handles finding the replacement file. I suspect when you use the Sage transcoder it does things to ensure all metadata for the recording is maintained. When you transcode via an external process the core probably first loses the original media file, deletes it from wiz.bin, then finds the new one and replaces it. I suspect that process loses some of the metadata about the original recording, namely the tuner used to record (probably among other things).

I'm sure you're exactly right.

Now the question is can one set the metadata on the MediaShrink created replacement file with the original encoder name from within SJQ?

[Slugger]

Originally Posted by phelme

I'm sure you're exactly right.

Now the question is can one set the metadata on the MediaShrink created replacement file with the original encoder name from within SJQ?

Yes, but it would have to be done in your mediashrink process. Let's assume you call a batch file that wraps the call to mediashrink. In that same batch file you'd have to call other scripts (you create) that extract the metadata from the original and then add it into the newly created mediafile. These scripts can be written using the sagex remote APIs, etc. It's certainly not easy, but doable, I think.

[lenkost]

I am using a dedicated quad core computer to run Comskip and AirVideo Server. Even with the quad core having 4 instances of Comskip consume almost 100% CPU. And the is no space for AirVideo which is also CPU hungry process.
I would go to SageAlert WEB UI via my iPhone, change the client Resources to 50 (2 Comskips running). Then I can use AirVideo Live Conversion.

I would like to request an additional option for Client: Disable/Enable.
Disabling client should kill all current jobs. Enabling client should run MediaFileQueueLoader to add them back.

Or if possible to add drop-down to quickly specify number of processes for this client killing extra jobs if any.

Editing Resources on a tiny iPhone screen is very inconvinient.
Thanks.

[Taddeusz]

You could just configure comskip to throttle the usage down so it's not so CPU hungry. Do the comskip processes really need to get done as quickly as possible?

[Slugger]

Originally Posted by lenkost

I am using a dedicated quad core computer to run Comskip and AirVideo Server. Even with the quad core having 4 instances of Comskip consume almost 100% CPU. And the is no space for AirVideo which is also CPU hungry process.
I would go to SageAlert WEB UI via my iPhone, change the client Resources to 50 (2 Comskips running). Then I can use AirVideo Live Conversion.

Why don't the comskip and airvideo tasks both use 25 resources? That way, up to 4 processes (of any combination of comskip and airvideo) will run simultaneously and the rest will remain queued until resources become available.

I would like to request an additional option for Client: Disable/Enable.
Disabling client should kill all current jobs. Enabling client should run MediaFileQueueLoader to add them back.

Hmm... ok, but unless you config SJQ to ignore failed jobs, auto killing the jobs is going to mark them as failed and SJQ will not requeue them. I still go back to the above comment: if comskip and airvideo are set to both use 25 resources then you'll always have 4 processes running and the rest queued up. You shouldn't need to manually kill in-progress tasks to make room for others - that's what the task queue is for as it handles this for you.

Or if possible to add drop-down to quickly specify number of processes for this client killing extra jobs if any.

This is definitely not going to happen. Why? This just screams of feature creep. Because once I say yes to this, then the next request from someone is, "well now I need to be able to prioritize which tasks get killed first when I decide it's time to kill off some processes." And it just snowballs from there so, sorry, this won't happen. Feature creep is already killing my SJQv4 effort - so much I'd like to do and every time I sit down and think about it, the feature list grows a little more and now it's at the point where it's so big I really don't want to tackle it. On top of that, your inspiration that got me to implement the process executor in SageAlert has really got me wondering if I even need SJQ myself anymore. With a process executor assigned to the RecordingStarted/RecordingCompleted events in SageAlert, I can basically do everything I do today with SJQ. The only thing I lose is the advanced queuing and multi-client support - both things I don't actually use other than when I'm testing SJQ. You're brilliant process executor idea for SageAlert may have just killed SJQ.

Editing Resources on a tiny iPhone screen is very inconvinient.
Thanks.

Send me an iPhone 4 (black, 32GB ) and I'll see what I can do to make the UI for iPhone friendly!

[Slugger]

I think I misunderstood. AirVideo isn't a task triggered by SJQ, but just something else running on the same server? In that case, you'll have to throttle back the number of processes you allow the SJQ client to run on that system. Such resource management is definitely required on the user's part. You have to play the sysadmin role and take a look at the big picture of what your system is doing and then plan accordingly when configuring things like SJQ, etc. to ensure there are always enough actual resources available on the system to do everything you need the system to do.

[lenkost]

ok, but unless you config SJQ to ignore failed jobs, auto killing the jobs is going to mark them as failed and SJQ will not requeue them

you'll have to throttle back the number of processes you allow the SJQ client to run on that system. Such resource management is definitely required on the user's part. You have to play the sysadmin role and take a look at the big picture of what your system is doing and then plan accordingly when configuring things like SJQ, etc. to ensure there are always enough actual resources available on the system to do everything you need the system to do.

I don't mind if some shows will not get comskip when I use AirVideo.

But most of the time I don't use it and would like to have all computer resources to be used for comskip.

So it is a resource management just remotely via SJQ WEB GUI.

There may be more reasons to stop client.
Like if one needs to use client computer for something else.
The ability to stop client ad-hoc should be useful.

I get the point about endless new features. Lets forget about drop-down.
I just suggest to have this:

Configure
View Log
Disable/Enable

If Disable is selected, warning "All current jobs will be marked as failed" displays and if confirmed, all jobs are killed and Client is disabled.

You're brilliant process executor idea for SageAlert may have just killed SJQ.

The Road To Hell Is Paved With Good Intentions

[Slugger]

Originally Posted by lenkost

I get the point about endless new features. Lets forget about drop-down.
I just suggest to have this:

Configure
View Log
Disable/Enable

This is manageable, but as I say, the future of SJQ dev is up in the air at this time. I see the ticket in the project tracker so that's about all you can do.

[phelme]

I'm trying to prevent the SageTVTranscoder and MediaShrink (i.e. Handbrake) from running at the same time to conserve resources. Theres no win to have them going concurrently, even on a quad core. Especially if the files reside on the same disc.

I'm probably mistaken, but there doesn't appear to be a way to cross-check resource availability. I can check within each compression type i.e TranscodeQueueSize for the SageTVTranscoder and RESOURCES within the client script for MediaShrink to see if the same type process is running. What I want to do is check if the OTHER type is running and only run if it isn't. Make sense? Is there an obvious way to do that which I'm missing?

I could call the SageTVTranscoder directly myself I guess in a client script, which could use the SJQ RESOURCE mechanism. But then I'd have to build up all the cmd line options for the transcoder myself. But I'm lazy!

BTW I'm only doing this because for some reason, MediaShrink is having deinterlacing problems of late on 1080i MPEGs from the HDHomeRun. Not sure what's going on there but I've had to split the compression duties for MPEGs because of that.

[Slugger]

There is no way to do this directly with SJQ. You'd have to wrap your call to mediashrink in another script and in that script check the status of the Sage transcode queue before deciding to run. The SJQ internal task for transcoding is not capable of checking external conditions before deciding to run (other than the conditions that were checked in the ruleset before queuing up the task itself). SJQ tasks are not aware of other SJQ tasks (either running or in the queue). There is no way for a task to view the state of the queue before making any decision to launch.

[doncote0]

I am running videoredo 4 and the process works fine when run manually.

However, when it is kicked off by SJQ it never completes.

The command line is

START /B /Wait /BELOWNORMAL cscript //nologo C:\Progra~2\VideoReDoTVSuite4\vp.vbs "%~d1%~p1%~n1.ts" "%~d1\%~n1.ts" /t4 /q /e

I ran an echo of this to a bat file for a test file and got this:

START /B /Wait /BELOWNORMAL cscript //nologo C:\Progra~2\VideoReDoTVSuite4\vp.vbs "E:\SageTV\ThePinkPantherStrikesAgain-4123592-0.ts" "E:\ThePinkPantherStrikesAgain-4123592-0.ts" /t4 /q /e

Perfect. Exactly what I want. When manually executed, the bat file does exactly what it should.

That same command will not work in my qsf.bat when called via SJQ (it executes forever [when SJQ executes a bat file from a bat file], or it just doesn't run correctly [from qsf.bat]) and completes without error. I am using Windows 7 64 bit.

Any ideas?

[Slugger]

"... it executes forever..."

What exactly does this mean? The program starts, but never finishes? The program gets in some kind of endless loop? The program seems to hang? The program never starts? You need to log the output of the program and figure out what it's doing. What happens when it gets "stuck"? This is usually one of two things:

1. The program cannot find expected config files, gets confused, and as a result it refuses to run/does weird things. All processes kicked off by SJQ have a working directory of the base install dir of SageTV. You need output from the program itself to debug. If this is the problem then it's not an SJQ problem.

2. The process I/O buffers are filling, causing deadlock. SJQ should be eating the I/O streams and therefore this shouldn't be the cause (a lot more people would be complaining if this were a problem). However, I won't say my code is perfect so to test this theory, you need to redirect all output from any script/exe/etc. called by SJQ to a file. If this solves your problem then it's likely an SJQ problem.

Either way, the first step in debugging this will require you to capture the output of the program you're calling and figuring out exactly what's happening when it "gets stuck"/"runs forever". Also, look in the sjq.log file, sqjc logs (wrapper.log and the GUI log for the client), sagetv debug logs, and jetty debug logs (if applicable) for any stack traces dumped by SJQ. Such stack traces are priceless to me when trying to find problems in my code.

[wayner]

I am having a problem where I have no audio on some recordings. This problems seem to be due to my STBs are dropping audio for some channels when the box sits for a long time (one day or more?) on the same channel. This problem is fixed if you change the channel and change it back.

Now I could fix this by just running a Scheduled Task on a daily basis that changes the channel. But this could potentially screwup a recording as it could change the channel during a recording.

Is there anyway to use SJQ to determine if a tuner is currently recording? If so then I could just use SJQ to run this batch file that changes the channel every once in a while.

If not SJQ then does anyone have any other ideas to fix this problem?