Plugin: Sage Job Queue (SJQ) v3

[Slugger]

Originally Posted by kevine

I know you just put this out there but I have a request for a change. If this came with options before and after a specified time it would be more useful. For example, Handbrake sometimes takes 2-3 times the media length to convert some files. It would be great if it could limit it by:{No recordings happening from now until 3 x target media length}. This way if it begins transcoding a 30 minute program, 30-60 minutes later it would not interfere with another recording just starting.

It's funny, I actually thought of the same thing just before I committed the code changes, but I was hungry and wanted to make dinner. Your posting confirms the need for the option to be able to do this. Some night this week I'll get it added in... but I think I'm actually going to spend a few nights this week using SageTV and get caught up on some of my shows.

[evilpenguin]

Originally Posted by kevine

I was wondering if anybody had ideas on how to use SJQ for the various stages of recording conversion I wish to accomplish.

I have comskip running and am looking to:
Manually verify the cuts comskip has found with Videoredo and save the commercial cut video to a subdirectory.

Automatically:
copy file back over original (here is where I think SJQ cannot help me)
handbrake to ipod format on a client machine leaving cut MPG
then mediashrink to it's final format in place and replace MPG

How about this process?

1) Manually go through and generate .vprj files, but don't save the actual video.
2) Set a MediaShrink task that picks up on .vprj files and does the cut/encode/replace. This way you can queue up the hard drive intensive task until the night time.
3) Make a Handbrake task that looks for the encoded videos (.mkv) but not (.mp4) and converts the encoded .mkv's into ipod format. (I don't think you'll notice any difference in quality on the iPod)

[kevine]

Originally Posted by Slugger

It's funny, I actually thought of the same thing just before I committed the code changes, but I was hungry and wanted to make dinner. Your posting confirms the need for the option to be able to do this. Some night this week I'll get it added in... but I think I'm actually going to spend a few nights this week using SageTV and get caught up on some of my shows.

I hear ya!

[kevine]

Originally Posted by evilpenguin

How about this process?

1) Manually go through and generate .vprj files, but don't save the actual video.
2) Set a MediaShrink task that picks up on .vprj files and does the cut/encode/replace. This way you can queue up the hard drive intensive task until the night time.
3) Make a Handbrake task that looks for the encoded videos (.mkv) but not (.mp4) and converts the encoded .mkv's into ipod format. (I don't think you'll notice any difference in quality on the iPod)

Yeah, I know that is the way you do it but I have vprj file for every recording I have already. They get created with comskip. However, like you said in your mediashrink thread, nothing automated for commercial cutting is perfect. So I go and touch it up. It is much faster doing the touch up than a vprj from scratch. If I could get comskip to use a different extension. I guess I could just manually create a text file to test for or manually delete the txt file that also gets created. Open to alternates here.

I did not realize you can reencode an mkv to mp4 using handbrake. I like it! That way, I do not have to wait to convert to mp4 before shrinking my entire library. Thanks.

[Slugger]

Originally Posted by kevine

How does SJQ perform actions on file objects not in the medial library? Are you talking about blindly copying the entire directory back with a batch file? How would you do the copy back?

Ok, one last thinking exercise then I really am going to go and use SageTV for awhile!

You got to think about it sort of backwards. Basically, let's say you've got a SageTV recording named Seinfeld-12345-0.mpg. You've run comskip on it and done your manual verification and created a cut version of the recording and let's say it's called Seinfeld-12345-0.cut.mkv (you'll need to pick some kind of naming convention) and the file is in C:\cut\.

So now you want to copy this back over and replace the original. (Side tanget: I don't think Sage is going to like the fact that this new file is a shorter length than the original; I think it's going to mess the time line up, but I'm just ignoring that for now).

Server ruleset:

if [IsObjMediaFile == true && $.IsTVFile{} == true && FileExists == "C:/cut/%p%.cut.mkv"] {
   # Ok, a cut version of the recording exists so let's replace the orig
   REPLACEORIG
}

Client config:

REPLACEORIG {
   /*
      This bat file takes in the source dir, source file, dest dir, dest file and 
      replaces the dest with the src; I ain't writing the batch file! ;)
   */
   "C:/mytools/replace.bat \"C:/cuts\" \"%p%.cut.mkv\" \"%d%\" \"%f%\"
}

The REPLACEORIG task has got to be a batch file that accepts the source dir, source name, target dir and target name and does the actual replacement. If it succeeds touch a file and then check that file in a rule that does the next step.

See this is where I thought SJQ could do this task. Because if it failed, the next line of code does not run. So:

pseudocode in the client:

copy xyz.mpg to sagetvDirectory
create ipod text file

Ipod text file would not actually happen if the copy did not work. No?

Nope, SJQ does not, nor will it ever, perform tasks on arbitrary files. It will only ever work on SageTV objects. But you can check for the existence of arbitrary files and perform tasks based on what you find.

NOTE: All of these SJQ examples are untested, off the top of my head. I wouldn't use them verbatim without some heavy testing first. Use them more as a guide.

[evilpenguin]

Originally Posted by kevine

Yeah, I know that is the way you do it but I have vprj file for every recording I have already. They get created with comskip. However, like you said in your mediashrink thread, nothing automated for commercial cutting is perfect. So I go and touch it up.

I wrote a tool for converting edl files to .vprj that's handy for this. That way I can just generate the .edl files and if I want, convert them to .vprj for clean-up and cutting at a later time.

http://upnp2podcast.googlecode.com/s...r/edl2vprj.exe

Its just drag and drop.

[kevine]

Thank you all for excellent suggestions. Once I have it implemented, I will definitely share my configuration.

[Slugger]

Originally Posted by kevine

I know you just put this out there but I have a request for a change. If this came with options before and after a specified time it would be more useful. For example, Handbrake sometimes takes 2-3 times the media length to convert some files. It would be great if it could limit it by:{No recordings happening from now until 3 x target media length}. This way if it begins transcoding a 30 minute program, 30-60 minutes later it would not interfere with another recording just starting.

Originally Posted by Slugger

It's funny, I actually thought of the same thing just before I committed the code changes, but I was hungry and wanted to make dinner. Your posting confirms the need for the option to be able to do this. Some night this week I'll get it added in... but I think I'm actually going to spend a few nights this week using SageTV and get caught up on some of my shows.

Snapshot 675 contains this enhancement. The :NORECORDING option goes from taking a boolean string to a number value. The number is the amount of time required before the start of the next scheduled recording. So a value of 2.0 on a 30 min recording would mean that there must be at least 60 mins before the next recording for the task to be assigned. Using 0.2 instead means 6 mins required (0.2 x 30) and so on. The user guide has been updated with this change and has more details on its usage.

[kevine]

Originally Posted by Slugger

Snapshot 675 contains this enhancement. The :NORECORDING option goes from taking a boolean string to a number value. The number is the amount of time required before the start of the next scheduled recording. So a value of 2.0 on a 30 min recording would mean that there must be at least 60 mins before the next recording for the task to be assigned. Using 0.2 instead means 6 mins required (0.2 x 30) and so on. The user guide has been updated with this change and has more details on its usage.

Awesome. Thanks, Slugger.

[tomfisk]

Hi Slugger,

This morning I noticed 2 programs that didn't get Comskipped last nite. Seems that sometime yesterday morning the available resources got decreased when nothing was happening:

2009-12-02 09:29:37,031 INFO  [SJQServlet]: Checking client '127.0.0.1': 100 resources free.
2009-12-02 09:29:37,093 INFO  [DataStore]: Flushed 1 log message(s) in 62ms
2009-12-02 09:29:37,234 INFO  [DataStore]: Flushed 1 log message(s) in 203ms
2009-12-02 09:29:37,234 INFO  [SJQServlet]: Checking client '127.0.0.1': 67 resources free.
2009-12-02 09:29:37,328 INFO  [DataStore]: Flushed 1 log message(s) in 78ms
2009-12-02 09:29:37,531 INFO  [DataStore]: Flushed 1 log message(s) in 281ms
2009-12-02 09:29:37,546 INFO  [SJQServlet]: Checking client '127.0.0.1': 1 resources free.
2009-12-02 09:29:37,546 INFO  [SJQServlet]: Client '127.0.0.1' cannot pop a task of type 'SHOWANALYZER': Requires 33 resources, but only 1 available.
2009-12-02 09:29:37,546 INFO  [SJQServlet]: Client '127.0.0.1' cannot pop a task of type 'COMSKIP': Requires 33 resources, but only 1 available.
2009-12-02 09:29:37,546 INFO  [SJQServlet]: Client '127.0.0.1' cannot pop a task of type 'COMSKIPTS': Requires 66 resources, but only 1 available.
2009-12-02 09:29:37,656 INFO  [DataStore]: Flushed 1 log message(s) in 110ms

At 37.093 there were 100 resources then it dropped to 67 free resources at 37.234. Two tasks would take 33 resources, SHOWANALYZER or COMSKIP, but there was nothing to process at the time. Even more interesting is that the SHOWANALYZER code is completely commented out in the Ruleset, but yet it looks like SJQ is trying to pop it.

From the Ruleset:

//Process SHOWANALYZER on all H.264 recordings unless recording was from KTCADT/KSMQDT or the result of Live TV
//if [IsObjMediaFile == true && $.IsTVFile{} == true && $.IsNotManualOrFavorite{} == false && $.IsFileCurrentlyRecording{} == false && $.IsWatchedCompletely{} == false && IsSomethingRecording == false && $.GetAiringChannelName{} !% "KTCA.*|KSMQ.*" && Filename =$ ".ts" && FileExists != "%d%/%p%.edl"]
//{
//  :PRIORITY 5
//  SHOWANALYZER
//}

Milliseconds later in the log, and the resources are down to 33 free and it wants to pop the SHOWANALYZER task for some reason:

2009-12-02 09:29:37,546 INFO  [SJQServlet]: Checking client '127.0.0.1': 1 resources free.
2009-12-02 09:29:37,546 INFO  [SJQServlet]: Client '127.0.0.1' cannot pop a task of type 'SHOWANALYZER': Requires 33 resources, but only 1 available.
2009-12-02 09:29:37,546 INFO  [SJQServlet]: Client '127.0.0.1' cannot pop a task of type 'COMSKIP': Requires 33 resources, but only 1 available.
2009-12-02 09:29:37,546 INFO  [SJQServlet]: Client '127.0.0.1' cannot pop a task of type 'COMSKIPTS': Requires 66 resources, but only 1 available.
2009-12-02 09:29:37,656 INFO  [DataStore]: Flushed 1 log message(s) in 110ms
2009-12-02 09:29:37,671 INFO  [SJQServlet]: Checking client '127.0.0.1': 1 resources free.
2009-12-02 09:29:37,671 INFO  [SJQServlet]: Client '127.0.0.1' cannot pop a task of type 'SHOWANALYZER': Requires 33 resources, but only 1 available.
2009-12-02 09:29:37,671 INFO  [SJQServlet]: Client '127.0.0.1' cannot pop a task of type 'COMSKIP': Requires 33 resources, but only 1 available.
2009-12-02 09:29:37,671 INFO  [SJQServlet]: Client '127.0.0.1' cannot pop a task of type 'COMSKIPTS': Requires 66 resources, but only 1 available.
2009-12-02 09:29:37,859 INFO  [DataStore]: Flushed 121 log message(s) in 906ms

And a couple of hours later there is this error:

2009-12-02 12:14:31,125 ERROR [DataStore]: SQL error
java.sql.SQLException: database locked
	at org.sqlite.DB.execute(DB.java:270)
	at org.sqlite.DB.executeUpdate(DB.java:281)
	at org.sqlite.PrepStmt.executeUpdate(PrepStmt.java:77)
	at com.google.code.sagetvaddons.sjq.server.DataStore.setSetting(DataStore.java:930)
	at com.google.code.sagetvaddons.sjq.server.MediaFileQueueLoader.run(MediaFileQueueLoader.java:48)
	at java.lang.Thread.run(Unknown Source)

This was from build 671, I installed build 675 this morning.

Tom

[awhenry]

Originally Posted by evilpenguin

Time for some alternate debugging...

I'm wondering if you have had any success resolving this? I am encountering the same issue. I have reverted to having SJQ call a batch file which launches comskip, and that resolves the issue but it doesn't seem like a very clean solution. Incidentally, I see the issue with anything of two hours in length or over.

Thanks.

[awhenry]

Hello all, I have just migrated over from Comskip Monitor to SJQ. I did love CSM, and it was humming along just fine, but I wanted to try recompressing some programs automatically, and it isn't well suited to that task.

Anyway, I am a little afraid that I joined the SJQ ranks at an inopportune time, with the new version barely out of the gates and all. I have encountered some strange problems in getting everything implemented, and sometimes I am not sure if it is me doing something wrong, or a bug somewhere in SJQ. I especially wonder when I don't see posts from anyone else reporting the same problems .

So, regarding the issue at hand, I have a couple of questions. First, does task logging work in the current version? It doesn't appear to work for me. Clicking any active task and selecting "view log" presents me with a page that says "Task log:<show name>" at the top, and nothing else. Completed or failed tasks show "Task log:<some ID number>/<task name>" and nothing else. Additionally, from the "clients" page, viewing the logs of any client also shows a blank page. Should something be appearing in these logs? Is there a config option I am missing somewhere? I know that some of the tasks I run produce output on stdout, so even if that was the only thing logged I should still be getting that I would think.

My second question involves CPU usage of SJQ itself during a media file update. I have SJQ set to scan when a scheduled recording starts, as well as every ten minutes in order to try to catch live shows for commercial marking and closed caption extraction (ideally it would be even more frequent...actually, ideally SJQ would notice that a live program was initiated and I wouldn't need to scan periodically at all ). The problem is that when the scan fires off, the SageTV service suddenly jumps to 100% CPU usage, and viewing of programs and manipulating the UI becomes choppy (via an HD200 extender). I understand that SJQ runs "inside" SageTV. Is there a way to make it more resource-friendly when it does its media file scans?

I have some more questions, but I'll leave it at that for now.

Thanks for any insight anyone can provide on this!

[Slugger]

Originally Posted by awhenry

I'm wondering if you have had any success resolving this? I am encountering the same issue. I have reverted to having SJQ call a batch file which launches comskip, and that resolves the issue but it doesn't seem like a very clean solution. Incidentally, I see the issue with anything of two hours in length or over.

Thanks.

What exactly is happening after two hours?

[Slugger]

Originally Posted by tomfisk

Hi Slugger,

This morning I noticed 2 programs that didn't get Comskipped last nite. Seems that sometime yesterday morning the available resources got decreased when nothing was happening:

2009-12-02 09:29:37,031 INFO  [SJQServlet]: Checking client '127.0.0.1': 100 resources free.
2009-12-02 09:29:37,093 INFO  [DataStore]: Flushed 1 log message(s) in 62ms
2009-12-02 09:29:37,234 INFO  [DataStore]: Flushed 1 log message(s) in 203ms
2009-12-02 09:29:37,234 INFO  [SJQServlet]: Checking client '127.0.0.1': 67 resources free.
2009-12-02 09:29:37,328 INFO  [DataStore]: Flushed 1 log message(s) in 78ms
2009-12-02 09:29:37,531 INFO  [DataStore]: Flushed 1 log message(s) in 281ms
2009-12-02 09:29:37,546 INFO  [SJQServlet]: Checking client '127.0.0.1': 1 resources free.
2009-12-02 09:29:37,546 INFO  [SJQServlet]: Client '127.0.0.1' cannot pop a task of type 'SHOWANALYZER': Requires 33 resources, but only 1 available.
2009-12-02 09:29:37,546 INFO  [SJQServlet]: Client '127.0.0.1' cannot pop a task of type 'COMSKIP': Requires 33 resources, but only 1 available.
2009-12-02 09:29:37,546 INFO  [SJQServlet]: Client '127.0.0.1' cannot pop a task of type 'COMSKIPTS': Requires 66 resources, but only 1 available.
2009-12-02 09:29:37,656 INFO  [DataStore]: Flushed 1 log message(s) in 110ms

This output is a clear marker that something was in the queue then popped off. Those log flushes are the header messages being written to the log for the task ("Starting task ... with priority ..." and "Executing command line..."). What the output is actually saying is that at 37.031 you had 100 resources and then it popped off a task using the 33 resources. The next two lines are the log headers being written to the db. Then it checked again at 37.234 and had 67 resources free and popped off a task using 66. Again, the next two lines are flushing the task log headers to the database.

The warnings about not being able to pop because of low resources: That's always going to happen every time the client connects. It doesn't mean that there is a task that it can't pop it means that it won't try to pop a task of that type, should there be one, because there are not enough free resources.

At 37.093 there were 100 resources then it dropped to 67 free resources at 37.234. Two tasks would take 33 resources, SHOWANALYZER or COMSKIP, but there was nothing to process at the time. Even more interesting is that the SHOWANALYZER code is completely commented out in the Ruleset, but yet it looks like SJQ is trying to pop it.

From the Ruleset:

//Process SHOWANALYZER on all H.264 recordings unless recording was from KTCADT/KSMQDT or the result of Live TV
//if [IsObjMediaFile == true && $.IsTVFile{} == true && $.IsNotManualOrFavorite{} == false && $.IsFileCurrentlyRecording{} == false && $.IsWatchedCompletely{} == false && IsSomethingRecording == false && $.GetAiringChannelName{} !% "KTCA.*|KSMQ.*" && Filename =$ ".ts" && FileExists != "%d%/%p%.edl"]
//{
//  :PRIORITY 5
//  SHOWANALYZER
//}

Milliseconds later in the log, and the resources are down to 33 free and it wants to pop the SHOWANALYZER task for some reason:

2009-12-02 09:29:37,546 INFO  [SJQServlet]: Checking client '127.0.0.1': 1 resources free.
2009-12-02 09:29:37,546 INFO  [SJQServlet]: Client '127.0.0.1' cannot pop a task of type 'SHOWANALYZER': Requires 33 resources, but only 1 available.
2009-12-02 09:29:37,546 INFO  [SJQServlet]: Client '127.0.0.1' cannot pop a task of type 'COMSKIP': Requires 33 resources, but only 1 available.
2009-12-02 09:29:37,546 INFO  [SJQServlet]: Client '127.0.0.1' cannot pop a task of type 'COMSKIPTS': Requires 66 resources, but only 1 available.
2009-12-02 09:29:37,656 INFO  [DataStore]: Flushed 1 log message(s) in 110ms
2009-12-02 09:29:37,671 INFO  [SJQServlet]: Checking client '127.0.0.1': 1 resources free.
2009-12-02 09:29:37,671 INFO  [SJQServlet]: Client '127.0.0.1' cannot pop a task of type 'SHOWANALYZER': Requires 33 resources, but only 1 available.
2009-12-02 09:29:37,671 INFO  [SJQServlet]: Client '127.0.0.1' cannot pop a task of type 'COMSKIP': Requires 33 resources, but only 1 available.
2009-12-02 09:29:37,671 INFO  [SJQServlet]: Client '127.0.0.1' cannot pop a task of type 'COMSKIPTS': Requires 66 resources, but only 1 available.
2009-12-02 09:29:37,859 INFO  [DataStore]: Flushed 121 log message(s) in 906ms

Are you sure it was commented out at the time this client connected? If so, then completely cut the block from the ruleset and save it. See if it keeps seeing that task type - I can't believe it will. Again, the 'cannot pop' info msgs are not saying that it's trying to pop a task of that type, it's saying that it won't if there are any (i.e. it's a precheck before actually looking at the queue). The last line, where it flushed 121 log messages: that's a task flushing its output back to the database; so there was definitely a task running and logging back output - no SJQ log message consists of 121 lines, so it's definitely something from a task client. The fact that it took those 121 lines 906ms to flush tells me your system is bogged down. I routinely see 5000+ lines be committed to my logs in less than 50ms.

And a couple of hours later there is this error:

2009-12-02 12:14:31,125 ERROR [DataStore]: SQL error
java.sql.SQLException: database locked
	at org.sqlite.DB.execute(DB.java:270)
	at org.sqlite.DB.executeUpdate(DB.java:281)
	at org.sqlite.PrepStmt.executeUpdate(PrepStmt.java:77)
	at com.google.code.sagetvaddons.sjq.server.DataStore.setSetting(DataStore.java:930)
	at com.google.code.sagetvaddons.sjq.server.MediaFileQueueLoader.run(MediaFileQueueLoader.java:48)
	at java.lang.Thread.run(Unknown Source)

This was from build 671, I installed build 675 this morning.

Tom

Please let me know if you can reproduce this on 675. Also, if you see this error again, please take a snapshot of the server threads and provide that output in addition (take that snapshot from the tools menu). Also, please increase the logging to DEBUG level in sjq.log4j.properties. It's going to produce tonnes of output, but it will be more helpful in hopefully identifying the problem.

[awhenry]

Originally Posted by Slugger

What exactly is happening after two hours?

Sorry. The issue is the same as EvilPenguin reported. For long programs (in my case I see it generally with programs two hours long) comskip never completes when launched by SJQ. Eventually the task times out and SJQ kills it, but that is two hours after the end of the program when comskip typically completes within a few seconds of the end time. If I instead have SJQ launch comskip indirectly by executing a batch file that in turn executes comskip, it runs for the normal duration and terminates on its own.

Unfortunately, due to my logging issue (mentioned in a separate post), I have no idea if comskip complains about anything under these circumstances. It does seem to actually do some commercial marking when this happens, so it is at least starting off okay.

[Slugger]

Originally Posted by awhenry

Hello all, I have just migrated over from Comskip Monitor to SJQ. I did love CSM, and it was humming along just fine, but I wanted to try recompressing some programs automatically, and it isn't well suited to that task.

Anyway, I am a little afraid that I joined the SJQ ranks at an inopportune time, with the new version barely out of the gates and all. I have encountered some strange problems in getting everything implemented, and sometimes I am not sure if it is me doing something wrong, or a bug somewhere in SJQ. I especially wonder when I don't see posts from anyone else reporting the same problems .

So, regarding the issue at hand, I have a couple of questions. First, does task logging work in the current version? It doesn't appear to work for me. Clicking any active task and selecting "view log" presents me with a page that says "Task log:<show name>" at the top, and nothing else.

Active tasks: the task client actually buffers the output and will only flush it to the server every 5000 lines or so. If your task doesn't produce more than 5000 lines of output then it will all just be flushed in batch at the end of the task. External tasks should, however, always have two lines visible b/c the server writes a couple lines showing the time the task started, it's priority, and the command line that was executed. Something like this:

Thu Dec 03 12:02:44 EST 2009: Starting task with priority 5/1/10
Executing command line: "C:/Program Files/SageTV/SageTV/bmt.bat" "D:\tv\TheFreshPrinceofBelAir-TheBabyComesOut-1188853-0.mpg"

Completed or failed tasks show "Task log:<some ID number>/<task name>" and nothing else. Additionally, from the "clients" page, viewing the logs of any client also shows a blank page. Should something be appearing in these logs? Is there a config option I am missing somewhere? I know that some of the tasks I run produce output on stdout, so even if that was the only thing logged I should still be getting that I would think.

Completed and failed tasks should show all the output produced. At the very least, it should show those two header lines like above. If those lines aren't being produced for external tasks (tasks run on a task client) then something's wrong. Some programs won't write to stderr/stdout and if they don't then SJQ won't log the output. For example, I use ShowAnalyzer and it never produces any output (other than the two header lines written by SJQ) b/c its output is written to a log file by ShowAnalyzer instead.

The client logs will be blank unless an error is encountered then it will log the error, but that's about all the client logs are used for. You must be running the latest client build (655) as there was a bug causing problems with this functionality prior to 655.

My second question involves CPU usage of SJQ itself during a media file update. I have SJQ set to scan when a scheduled recording starts, as well as every ten minutes in order to try to catch live shows for commercial marking and closed caption extraction (ideally it would be even more frequent...actually, ideally SJQ would notice that a live program was initiated and I wouldn't need to scan periodically at all ).

Ideally, I'd like to be able to do it based on events, but SageTV does not provide an event listening mechanism to Java programs, so you have to poll all the time.

The problem is that when the scan fires off, the SageTV service suddenly jumps to 100% CPU usage, and viewing of programs and manipulating the UI becomes choppy (via an HD200 extender). I understand that SJQ runs "inside" SageTV. Is there a way to make it more resource-friendly when it does its media file scans?

Just tested this, I guess you're right. For me, it spikes my CPU to 25%, but I have a quad core, so I guess it is pegging a core completely for the 1 second or so that it runs. This is definitely a function of your server specs and the number of the media objects being scanned. I mean I don't have any performance issues while SJQ is running, but if you're on a single core then I suppose you might see issues. All I can do is loop through each media object and run it against the ruleset. Are there some optimizations that could be made to reduce the CPU load? Probably, but I'm not going to be spending much time looking for them. The one big thing I'd suggest is to rearrange your rules so that it can short circuit out as quick as possible and therefore make as few Sage API calls as possible. For example,

if [IsObjMediaFile == true && $.IsFileCurrentlyRecording{} == false && $.GetShowTitle{} == "Seinfeld" ] {
   BLAH
}

It is very likely that the bulk of your media objects are going to result in 2 API calls if your rule set is written like above. Why? IsFileCurrentlyRecording{} is going to always return false for all but a few of your files, which means that test will pass and it will then always call the GetShowTitle{} test. However, if you rearrange it to this:

if [IsObjMediaFile == true && $.GetShowTitle{} == "Seinfeld" && $.IsFileCurrentlyRecording{} == false ] {
   BLAH
}

Now all of a sudden most of your media objects are only going to make one Sage API call since most of your files are going to fail on the Seinfeld test and then bail out.

Imagine a much more complicated rule block with lots of tests and you can start to see the gain that can be made.

[evilpenguin]

Originally Posted by awhenry

Unfortunately, due to my logging issue (mentioned in a separate post), I have no idea if comskip complains about anything under these circumstances. It does seem to actually do some commercial marking when this happens, so it is at least starting off okay.

I cranked up comskip's logs to max and it doesn't give any indication why its hanging. Its working normally for about two hours and then all of the sudden it just sits there doing nothing until you kill it.

[nyplayer]

Slugger,

I have complicated rule sets... after IsObjMediaFile == true should i then put all my fileexists after IsObjMediaFile ?

if [IsObjMediaFile == true && $.IsTVFile{} == "true" && $.IsNotManualOrFavorite{} == "false" && $.IsFileCurrentlyRecording{} == "false" && $.GetNumberOfSegments{} == "1" && $.GetMediaTitle{} != "Not Recording" && $.GetAiringChannelNumber{} !="35" && $.GetAiringChannelNumber{} !="36" && FileExists != "%d%/%p%.qfix" && Filename =$ ".mpg" && FileExists != "c:\\oktoreboot.txt" && FileExists != "c:\\Stoprunning.txt"] {
   :RUNQLOADER "true"    
    :PRIORITY 998
     SJQFIXShow
}

//Set Manual Record Title
if [IsObjMediaFile == true && $.IsTVFile{} == "true" && $.IsFileCurrentlyRecording{} == "false" && $.GetShowEpisode{} != "" && FileExists == "%d%/%p%.qfix" && Filename =$ ".mpg" && ($.GetMediaTitle{} =^ "Star Trek" || ($.GetMediaTitle{} == "Smallville" || ($.GetMediaTitle{} == "Law & Order" || ($.GetMediaTitle{} == "NCIS" || ($.GetMediaTitle{} == "Law & Order: Criminal Intent" || ($.GetMediaTitle{} == "The Dead Zone" || ($.GetMediaTitle{} == "Dark Angel" || ($.GetMediaTitle{} == "Heroes" || ($.GetMediaTitle{} == "Monk" || ($.GetMediaTitle{} == "Friday the 13th: The Series" || ($.GetMediaTitle{} == "Tales From the Darkside" || ($.GetMediaTitle{} == "The Outer Limits" || ($.GetMediaTitle{} == "Jake 2.0" || ($.GetMediaTitle{} == "Eureka" || ($.GetMediaTitle{} == "Criminal Minds" || ($.GetMediaTitle{} == "Law & Order: Special Victims Unit" || ($.GetMediaTitle{} == "Dancing With the Stars")))))))))))))))))] {
   :RUNQLOADER "true"
    :PRIORITY 248
      _MANUALREC
}

[Slugger]

Originally Posted by evilpenguin

I cranked up comskip's logs to max and it doesn't give any indication why its hanging. Its working normally for about two hours and then all of the sudden it just sits there doing nothing until you kill it.

Hmm... sounds like the task client threads created for chewing up the output died off. Let me try something... can you test a new task client build?

[Slugger]

Originally Posted by nyplayer

Slugger,

I have complicated rule sets... after IsObjMediaFile == true should i then put all my fileexists after IsObjMediaFile ?

if [IsObjMediaFile == true && $.IsTVFile{} == "true" && $.IsNotManualOrFavorite{} == "false" && $.IsFileCurrentlyRecording{} == "false" && $.GetNumberOfSegments{} == "1" && $.GetMediaTitle{} != "Not Recording" && $.GetAiringChannelNumber{} !="35" && $.GetAiringChannelNumber{} !="36" && FileExists != "%d%/%p%.qfix" && Filename =$ ".mpg" && FileExists != "c:\\oktoreboot.txt" && FileExists != "c:\\Stoprunning.txt"] {
   :RUNQLOADER "true"    
    :PRIORITY 998
     SJQFIXShow
}

//Set Manual Record Title
if [IsObjMediaFile == true && $.IsTVFile{} == "true" && $.IsFileCurrentlyRecording{} == "false" && $.GetShowEpisode{} != "" && FileExists == "%d%/%p%.qfix" && Filename =$ ".mpg" && ($.GetMediaTitle{} =^ "Star Trek" || ($.GetMediaTitle{} == "Smallville" || ($.GetMediaTitle{} == "Law & Order" || ($.GetMediaTitle{} == "NCIS" || ($.GetMediaTitle{} == "Law & Order: Criminal Intent" || ($.GetMediaTitle{} == "The Dead Zone" || ($.GetMediaTitle{} == "Dark Angel" || ($.GetMediaTitle{} == "Heroes" || ($.GetMediaTitle{} == "Monk" || ($.GetMediaTitle{} == "Friday the 13th: The Series" || ($.GetMediaTitle{} == "Tales From the Darkside" || ($.GetMediaTitle{} == "The Outer Limits" || ($.GetMediaTitle{} == "Jake 2.0" || ($.GetMediaTitle{} == "Eureka" || ($.GetMediaTitle{} == "Criminal Minds" || ($.GetMediaTitle{} == "Law & Order: Special Victims Unit" || ($.GetMediaTitle{} == "Dancing With the Stars")))))))))))))))))] {
   :RUNQLOADER "true"
    :PRIORITY 248
      _MANUALREC
}

Yeah, I'd move the FileExists tests up front. Also, your ORs of all those titles, I'd turn it into a single regex test, which would reduce it to one API call. For example:

if [IsObjMediaFile == true && ($.GetShowTitle{} == "X" || $.GetShowTitle{} == "Y" || $.GetShowTitle{} == "Z")] {

}

Could be rewritten as :

if [IsObjMediaFile == true && $.GetShowTitle{} =% "X|Y|Z"] {

}

How much improvement is this going to get you? Not really sure. As I say, I don't experience the UI sluggishness, etc. on my quad core so I really haven't studied this too much, but it certainly can't hurt to try.