Plugin: Sage Job Queue (SJQ) v3

[wayner]

I use it for Comskip - separate processes for SD and HD files. I use it to automatically compress Kids' shows to MPEG4, rename the file to episode name.avi and move to the relevant folder i.e. \Kids Shows\Dora.

I also use it to run MediaScrape for movies but I don't know that this is still required since BMT came along.

[gveres]

I just use it for standard stuff:

* I use the R5000 so I use it to process every single recording with VideoReDo's QuickStreamFix so that the recordings are watchable
* When that is done, I have it start comskip for shows not recorded on movie channels
* then I have it do various compression profiles
* some of the shows I want to compress for my iPod
* I have another compression profile for kids shows that don't need to be high quality
* I have another compression profile for parent's shows that we want HQ.


I can share my ruleset if anybody wants it

Greg

[Brent]

In the SJQv3 User's Guide it says:

Failure to make this change will result is the task client failing to flush output back to the server and will also likely result in the hanging of your task processes. On the other hand, if your tasks do not produce much output then you may not have a problem despite not making this change. Users who plan to run comskip via SJQ should DEFINITELY make this change as ShowAnalyzer is known to produce more than enough output (especially when processing H.264 files) to cause problems; however, it is recommended that ALL users of SJQ make this change prior to your first use.

Is first references comskip and then references ShowAnalyzer in the same sentence. Is that a mistake?

[Slugger]

It should reference comskip in both instances. I've just corrected the online version of the doc and republished.

[davephan]

Originally Posted by nyplayer

I use it to move diff favorites to other folders and drives... kids wife etc... archive to NAS.

Reboot server once a week when nothing is recording... also reboot clients.

Run Videoredo qfix and cut commercials.

Could you post your code that you are using? How can you make sure the recordings are done before the files are attempted to be moved to a different folder?

I was thinking about moving recordings that need to be processed by Comskip to a different folder. After the files are in the different folder, then the files would be Comskipped by a another computer using DirMon2 to monitor the mapped drive from the second computer with a mapped drive. For that process to work, then SJQ would have to 'know' when the files are done being written to by the recording process.

Dave

[nyplayer]

Dave,

This rule checks the recording is done

&& $.IsFileCurrentlyRecording{} == "false"

//Move Wife's
if [IsObjMediaFile == "true" && Filename =$ ".mpg" && $.IsFileCurrentlyRecording{} == "false" && $.IsTVFile{} == "true" && $.GetShowTitle{} =% "Little People, Big World|17 Kids and Counting|Extreme Makeover: Home Edition|Jon & Kate Plus 8|18 Kids and Counting|Table for 12|The Little Couple|Cake Boss|Raising Sextuplets|19 Kids and Counting|Forensic Files"] {
:MOVERECOPTS "J:\\sagej"
_MOVEREC
}

the reboot is here

http://forums.sagetv.com/forums/show...44&postcount=6

[Fuzzy]

Originally Posted by davephan

Could you post your code that you are using? How can you make sure the recordings are done before the files are attempted to be moved to a different folder?

I was thinking about moving recordings that need to be processed by Comskip to a different folder. After the files are in the different folder, then the files would be Comskipped by a another computer using DirMon2 to monitor the mapped drive from the second computer with a mapped drive. For that process to work, then SJQ would have to 'know' when the files are done being written to by the recording process.

Dave

You shouldn't have to move them to another folder to have comskip process them on another computer. SJQ allows you to run tasks on different client machines transparently. Just set up SJQC on the computer you want to run the task, and set it up in that config.

[davephan]

Originally Posted by nyplayer

Dave,

This rule checks the recording is done

&& $.IsFileCurrentlyRecording{} == "false"

//Move Wife's
if [IsObjMediaFile == "true" && Filename =$ ".mpg" && $.IsFileCurrentlyRecording{} == "false" && $.IsTVFile{} == "true" && $.GetShowTitle{} =% "Little People, Big World|17 Kids and Counting|Extreme Makeover: Home Edition|Jon & Kate Plus 8|18 Kids and Counting|Table for 12|The Little Couple|Cake Boss|Raising Sextuplets|19 Kids and Counting|Forensic Files"] {
:MOVERECOPTS "J:\\sagej"
_MOVEREC
}

the reboot is here

http://forums.sagetv.com/forums/show...44&postcount=6

I'm trying to understand your code. As I understand it, the code is for the SJQ ruleset. The file is a media file with an mpg extension, the show is a TV file that is no longer recording, and matches one of the show name titles that is listed.

I don't understand this part of the code:
{
:MOVERECOPTS "J:\\sagej"
_MOVEREC
}

What code is used for the client configuration?

Dave

[Fuzzy]

_MOVEREC is an internal task in SJQ3. It moves the file to the path specified by the :MOVERECOPTS option. Check the Internal Tasks section of the SJQv3 user guide for a description, as well as many other built-in tasks. This task is actually carried out by the server itself, not on any particular client, in fact, I don't think you even NEED a client if using JUST internal tasks.

[wayner]

The other way to move a file is as part of a Transcode operation. Although I have never done it you may be possible to use the Transcode operation to only move and to not Transcode, or to Transcode to the same destination type as source so effectively not transcoding. I believe the Move part of Transcode is safe as I have used it hundreds of times whereas Slugger has a disclaimer on the _MOVEREC task.

[nyplayer]

There is no transcode move operation basically what you set up is a copy profile... but this does not do a bit by bit copy as the _MOVEREC command... If you have multiple audio tracks transcoding only copies the first audio track. I have never had a problem with the _MOVREC and I have done hundreds. Also the _MOVEREC will move over edls, my and other metdata files.

[wayner]

Thanks for the info - moving over edls is an issue for me. Any advice on how to do a Move, Comskip and Transcode? Should I move the file first and then do the Comskip and finally the transcode? If I tried to do the Transcode first and then Move I am worried that SJQ does not know when the Transcode is finished and it might try to do the Move while the Transcoding is in progress.

[Slugger]

I concur with nyplayer in that I've personally moved over 1000 recordings (and counting) to my NAS via the _MOVEREC task. Everyday more get copied and I have yet to have a problem with it. However, wayner is correct that I've placed an obligatory disclaimer on the _MOVEREC task more as a precaution/cover my ass kind of statement in case something does go wrong. In other words, I'm quite confident in its use - I use it every day - but I'm not crazy enough to say that it is 100% bug free. It just so happens if you discover a bug with this particular feature then it's more likely to cause undesired results to your system than a regular, run of the mill bug. It's definitely a "use at your own risk" kind of feature (just as SJQ in general and any 3rd party plugin, for that matter, would be), but for me it works exactly as designed (YMMV).

[Slugger]

Originally Posted by wayner

Thanks for the info - moving over edls is an issue for me. Any advice on how to do a Move, Comskip and Transcode? Should I move the file first and then do the Comskip and finally the transcode? If I tried to do the Transcode first and then Move I am worried that SJQ does not know when the Transcode is finished and it might try to do the Move while the Transcoding is in progress.

Are you triggering the transcode via SJQ? If so, add a constraint to the rule that doesn't add the transcode job until you know the file's been moved to its final destination (probably via the Filename test). You are correct that trying to transcode first will most likely confuse SJQ since SJQ knows nothing about the status of files in the SageTV transcode queue (SJQ only knows how to put them into the queue. Once in the queue, SJQ cannot learn anything more about it). So the correct order should be:

* Move via _MOVEREC task
* Comskip (add constraint to rule to ensure the move is completed)
* Transcode (same constraint)

#2 and #3 could be queued up at the same time since SJQ will never start an internal task (transcode queue) while an external task (comskip, in this case) is pending or running.

[davephan]

Comskip works for several hours, then quits working. I have SJQ setup to process 3 Comskips at the same time. I've tried varying the number from 1 to 3, but always receive the same result. SJQ will successfully process Comskip jobs for awhile, the Comskip processes all show CPU load after I start SJQ. There are jobs that get completed. However, after several hours, each Comskip process goes down to consuming zero CPU, and each Comskip log file looks like Comskip gets stuck on a frame number for each of the three items being Comskipped. Each of the three jobs do not get stuck at the same time, so it takes awhile for all three to get stuck.

If I try running Comskip with just a simple command line for the same TS or MPG files that were stuck, Comskip does not fail, and completes processing each file.

I am suspecting that the SJQC service might be the problem, where it just dies out after awhile for some reason. I did not try running the client as an application rather than a service. I was thinking that I could simply move files that should be processed by Comskip to another directory, which SJQ could do since it 'knows' which channel names that are commercial and non-commercial, and just have DirMon2 control the Comskipping.

However, if others are able to Comskip reliably with SJQ, I should be able to Comskip with SJQ too.

Here is the ruleset code I am using:

if [IsObjMediaFile == true && $.IsTVFile{} == true && ($.GetAiringChannelName{} !% "KTCADT|KTCADT2|KTCADT3|KTCADT4|CSPAN*|ENCRWA|ENCORE|ENCRA|ENCRM|ENCRL|ENCRD|ENCRWS|HBOHD|HBO2|HBOSIG|HBOF|HBOC|HBOZ|MPLEX|IFC|SUNDAE|INDIE|RETR|FLIX")] {
   COMSKIP
}

Here is the client code:

COMSKIP {
   :RESOURCES 33# Up to 3 comskips can be run simultaneously (every client has exactly 100 resources available)

   # CPU priority to be used for this task process; one of LOW, NORMAL, HIGH (case sensitive)
   :CPU "NORMAL"   # Default is NORMAL if not specified

   # Comskip returns 1 on success instead of the universally accepted zero; tell SJQ that
   # a return code of 1 means success instead of the default value of zero
   :RETURNCODE 1

   "C:/comskip/comskip.exe \"%c%\""
}

Dave

[Slugger]

Stop your task client, open the sjqc.properties file. If the buffer is higher than 65000 (it's probably 199000) then reset it to 65000. Restart task client - all should be fixed. (This bug is also fixed in the latest task client snapshot build: v3.0.2.771. But if you installed 3.0.1.705 first then installing 3.0.2.771 will not fix the bug automatically (you must fix it manually by resetting the buffer value yourself).

[davephan]

Originally Posted by Slugger

Stop your task client, open the sjqc.properties file. If the buffer is higher than 65000 (it's probably 199000) then reset it to 65000. Restart task client - all should be fixed. (This bug is also fixed in the latest task client snapshot build: v3.0.2.771. But if you installed 3.0.1.705 first then installing 3.0.2.771 will not fix the bug automatically (you must fix it manually by resetting the buffer value yourself).

Thanks Slugger! I am very happy that it is a known bug. That problem where Comskip seemed to be dying after several hours was driving me crazy. I have a big backlog of files to process. I don't know if I'll ever have time to watch many of them, but at least they will be Comskipped.

I was running the SJQC version 3.0.1.705, I stopped the SJQC service., and the buffer was set to 199000. I started up the SJQC service. I modified the client file for up to 4 Comskips at once.

I also deleted the extra files that were created for the files that were attempted to be processed, but failed.

Here's my sjqc.properties file before changing 199000 to 65000.

#Auto generated by SJQC
#Sun Apr 04 12:10:15 CDT 2010
buffer=199000
port=8500
host=localhost
sleep=300
is_ssl=false

It's still running 4 processes (not the same files) after about two hours. I think that fixed the problem.


Dave

[davephan]

The SJQ is still running after running all night, so the change absolutely fixed the problem. I did notice when viewing the log files of the jobs that are currently running, sometimes only two lines of the log are visible, other times many lines are visible. The end of the log does not have the completed time, like the log file that is stored near the recorded file.

Is there a way that the total number of jobs in the queue could be displayed on the main SJQ page? Although, I can get a rough idea of the number by clicking on the scroll bar many times between the screen refreshes.

Dave

[Slugger]

Originally Posted by davephan

I did notice when viewing the log files of the jobs that are currently running, sometimes only two lines of the log are visible, other times many lines are visible. The end of the log does not have the completed time, like the log file that is stored near the recorded file.

The task client only flushes the output after the buffer is full or the task has completed. So when you only see the two lines while a task is running it means the task client's buffer hasn't filled yet. If you want a real time dump of the output then lower the buffer to an even smaller number, but doing so means there's going to be constant network traffic between the task client and the server, which will affect performance.

Is there a way that the total number of jobs in the queue could be displayed on the main SJQ page? Although, I can get a rough idea of the number by clicking on the scroll bar many times between the screen refreshes.

Dave

Open an issue ticket on the project site and I can probably add it in.

[davephan]

Originally Posted by gveres

I just use it for standard stuff:

* I use the R5000 so I use it to process every single recording with VideoReDo's QuickStreamFix so that the recordings are watchable
* When that is done, I have it start comskip for shows not recorded on movie channels
* then I have it do various compression profiles
* some of the shows I want to compress for my iPod
* I have another compression profile for kids shows that don't need to be high quality
* I have another compression profile for parent's shows that we want HQ.


I can share my ruleset if anybody wants it

Greg

Greg,

Could you post your code for processing files with the VideoReDo Quick Stream Fix (QSF)?

I'm also interested in your code for video compression.

Dave