Plugin: Sage Job Queue (SJQ)

[Slugger]

Originally Posted by nyplayer

Slugger,

I seem to have run into a problem is there a limit on the amount of data a rules can contain?

[Fri May 02 13:41:21 PDT 2008] /sage/SJQ: init
[Fri May 02 13:43:10 PDT 2008] WARNING: POSTed data from Form content truncated to 10000 bytes
[Fri May 02 13:43:10 PDT 2008] unexpected problem running servlet net.sourceforge.sagetv.sjq.SJQServlet: java.lang.ClassCastException: java.lang.String cannot be cast to java.util.Collection
[Fri May 02 13:43:19 PDT 2008] unexpected problem running servlet net.sourceforge.sagetv.sjq.SJQServlet: java.lang.ClassCastException: java.lang.String cannot be cast to java.util.Collection
[Fri May 02 13:43:49 PDT 2008] unexpected problem running servlet net.sourceforge.sagetv.sjq.SJQServlet: java.lang.ClassCastException: java.lang.String cannot be cast to java.util.Collection
[Fri May 02 13:44:19 PDT 2008] unexpected problem running servlet net.sourceforge.sagetv.sjq.SJQServlet: java.lang.ClassCastException: java.lang.String cannot be cast to java.util.Collection
[Fri May 02 13:44:28 PDT 2008] /sage/SJQ: destroy

This is a limitation of the web server plugin. I've taken a quick look at the web server source code and it appears as though this limitation is arbitrary and unnecessary. I've posted a message in the web server plugin thread to see about at least increasing the value or completely removing the limit all together. Unfortunately, there's nothing I can do within SJQ to work around this. It also appears as though that this limitation has corrupted your ruleset file. Since the web server truncates the form data at 10,000 bytes and proceeds, the data passed along to SJQ is incomplete/corrupted so the result of SJQ's processing of the truncated data is unknown. Your best bet is to delete the rules.sjq file and recreate what you can without going over the 10K limit.

Alternatively, you could compile a custom version of the web server plugin. The constant that needs to be increased is in the Serve.java file. The const name is MAX_FORM_LENGTH. I'd increase it to 100,000. If this isn't an option for you, then hopefully the next release of the web server plugin will include a fix for this.

[nyplayer]

Slugger,

Thanks for looking into to this.... Fortunately it does not corrupt the Rules file. It will just not save anything to it. As Far as compiling a webserver appl.. I am not really a programmer so that is not an option. I have to edit the rules file manually once it reaches the max .

[gveres]

Hi Slugger,

I am trying to do the same thing that nyplayer is doing. I need to run VideoReDo qfix on my mpg files.

I am trying to do it with the web UI. Here is some feedback for you:

1) how do I enter something like ! FileExist "%d\%b\.qfix" ? The options I get are if fileExists / does not equal / %d\%b\.qfix

This doesn't seem to imply the same thing as file doesn't exist. I am hoping that it means that but I can't succesfully enter a rule so I can't try it out.

2) I wanted to look at the Log. I clicked on one of the + signs to expand the log and it locked up the browser for a few minutes loading in a ton of log entries - one for each photo in my photo gallery. I suspect there was also one for each music file and one for each recording. This kills the browser.

3) I can't enter a rule with the ruleset editor because I have zero idea what the "of type ___________" field is. this doesn't seem to correspond to anything in your example rule you provided nyplayer earlier in the thread. There also seems to be no reference to it in the documentation. And once I hit Save, it tells me this field needs to be filled in

I think it should work great once I get it going but i am stuck right now.

Oh, the other thing is ... is there an example of calling comskip? I am not a comskip expert and I have been using the built in SageMC comskip mechanism, but I really need to run videoReDo on all my mpg files first so now I have to switch to having SJQ call comskip. Any pointer would be appreciated. I just don't know what to put in the command line. Did your installer install comskip? Or are you expecting that it is already installed and we should know how to invoke it and therefore we should know what to put in the command line field?

Thanks
Greg

[Slugger]

Originally Posted by gveres

Hi Slugger,

I am trying to do the same thing that nyplayer is doing. I need to run VideoReDo qfix on my mpg files.

I am trying to do it with the web UI. Here is some feedback for you:

1) how do I enter something like ! FileExist "%d\%b\.qfix" ? The options I get are if fileExists / does not equal / %d\%b\.qfix

This doesn't seem to imply the same thing as file doesn't exist. I am hoping that it means that but I can't succesfully enter a rule so I can't try it out.

Yes, it does mean file does not exist. Unfortunately, the way this test is implemented leads to the unclear test syntax within the GUI. However, note that the test is not a regular expression, but rather a simple check for equality (or inequality) after the media variables are expanded (i.e. don't escape anything). Also, the media variables require a trailing '%' character (i.e. %d% not %d).

Originally Posted by gveres

2) I wanted to look at the Log. I clicked on one of the + signs to expand the log and it locked up the browser for a few minutes loading in a ton of log entries - one for each photo in my photo gallery. I suspect there was also one for each music file and one for each recording. This kills the browser.

This is a known issue with the Yahoo! User Interface library used in the GUI. I keep trying to remember to investigate a work around. If you'd add a bug to the project tracker I will get to it next time I sit down to work on the code (which may be sooner than later as I'm lying in bed with my laptop reading the forums this evening with a 100F fever instead of on vacation). :-(

Originally Posted by gveres

3) I can't enter a rule with the ruleset editor because I have zero idea what the "of type ___________" field is. this doesn't seem to correspond to anything in your example rule you provided nyplayer earlier in the thread. There also seems to be no reference to it in the documentation. And once I hit Save, it tells me this field needs to be filled in

From the online docs:

Action lines always start with a tab character followed by a unique ID used to describe the action being performed followed by an executable name along with any command line arguments.

When the action completes successfully it will be registered in the database such that the action ID will never be ran for the media object again. An action that fails will not be registered in the database and therefore will possibly be attempted again on the next run of SJQ.

The type is simply a unique identifier telling SJQ what this action is doing. For example, you might use 'COMM' if running commskip. I require this id string so that if you change the program used to skip commercials then SJQ won't run commercial skipping a second time for a given recording as long as you always use the string 'COMM' to identify your commercial skipping rules. Hope that makes sense?

Originally Posted by gveres

I think it should work great once I get it going but i am stuck right now.

Oh, the other thing is ... is there an example of calling comskip? I am not a comskip expert and I have been using the built in SageMC comskip mechanism, but I really need to run videoReDo on all my mpg files first so now I have to switch to having SJQ call comskip. Any pointer would be appreciated. I just don't know what to put in the command line. Did your installer install comskip? Or are you expecting that it is already installed and we should know how to invoke it and therefore we should know what to put in the command line field?

Thanks
Greg

I personally use ShowAnalyzer so I can't give an example of calling commskip as I don't use it, but I'd be surprised if it was any different than calling ShowAnalyzer, which is simply calling the exe with the name of the mpg as the argument. Within SJQ, use the %f% variable to pass the full path name of the file being processed. So if you load the online docs then any of the ShowAnalyzer examples I have in there should work just fine with commskip by simply replacing the path to ShowAnalyzer with that of commskip.

[nyplayer]

Your Rule file should look like this. i added the working file because I run my jobs on a Client and the server.

###__QFIX
$ Filename: .mpg
= IsTV: true
!= FileExists: %d%\%b%.QFixed
!= FileExists: %d%\%b%.working
= IsActivelyRecording: false
= IsScheduledRecording: true
RUNQFIX "C:\QFIXSAGE\qfixsage.bat" "%f%"

###__Comskip
$ Filename: .mpg
= IsTV: true
!= FileExists: %d%\%b%.working
!= FileExists: %d%\%b%.edl
= FileExists: %d%\%b%.QFixed
RunComskip "c:\comskip\comskip.bat" "%f%"

Then my comskip batch file looks like this.

IF EXIST "%~d1%~p1%~n1.working" goto end
echo "Commercial Skip">>"%~d1%~p1%~n1.working"
cd "C:\comskip"
c:
start /w /low cimpskip.exe "%~f1"
del "%~d1%~p1%~n1.working"
:end
exit

Originally Posted by gveres

Hi Slugger,

I am trying to do the same thing that nyplayer is doing. I need to run VideoReDo qfix on my mpg files.

I am trying to do it with the web UI. Here is some feedback for you:

1) how do I enter something like ! FileExist "%d\%b\.qfix" ? The options I get are if fileExists / does not equal / %d\%b\.qfix

This doesn't seem to imply the same thing as file doesn't exist. I am hoping that it means that but I can't succesfully enter a rule so I can't try it out.

2) I wanted to look at the Log. I clicked on one of the + signs to expand the log and it locked up the browser for a few minutes loading in a ton of log entries - one for each photo in my photo gallery. I suspect there was also one for each music file and one for each recording. This kills the browser.

3) I can't enter a rule with the ruleset editor because I have zero idea what the "of type ___________" field is. this doesn't seem to correspond to anything in your example rule you provided nyplayer earlier in the thread. There also seems to be no reference to it in the documentation. And once I hit Save, it tells me this field needs to be filled in

I think it should work great once I get it going but i am stuck right now.

Oh, the other thing is ... is there an example of calling comskip? I am not a comskip expert and I have been using the built in SageMC comskip mechanism, but I really need to run videoReDo on all my mpg files first so now I have to switch to having SJQ call comskip. Any pointer would be appreciated. I just don't know what to put in the command line. Did your installer install comskip? Or are you expecting that it is already installed and we should know how to invoke it and therefore we should know what to put in the command line field?

Thanks
Greg

[Slugger]

Originally Posted by nyplayer

Your Rule file should look like this.

###__QFIX
$ Filename: .mpg
= IsTV: true
!= FileExists: %d%\%b%.QFixed
!= FileExists: %d%\%b%.working
= IsActivelyRecording: false
= IsScheduledRecording: true
RUNQFIX "C:\QFIXSAGE\qfixsage.bat" "%f%"

###__Comskip
$ Filename: .mpg
= IsTV: true
!= FileExists: %d%\%b%.working
!= FileExists: %d%\%b%.edl
= FileExists: %d%\%b%.QFixed
RunComskip "c:\comskip\comskip.exe" "%f%"

Just a quick note, if one were to cut and paste this that you'd need to ensure that the action lines start with a tab character instead of spaces. I believe this message board app will replace tabs with spaces, so just be careful of that.

[gveres]

Thanks guys.

Slugger - sorry to hear you are laid up in bed instead of on vacation.

One suggestion: "of type" could be relabeled to "Action Unique ID". That makes sense and matches the docs.

nyplayer - thanks for the help. I am using your qfixsage.bat file (well modified because it was missing some quotes and it didn't clean up some files in error conditions.

I will modify my rule and then tape another show. BTW, because canceling a recording deletes it immediately, how do you guys test this stuff?

Thanks again for your help.
Greg

[nyplayer]

Sage will not delete partial files if you set this property. I believe the default is set to true. Also if you plan on cutting commercials make sure the property is set to false as below.

seeker/clear_partials_and_unwanted_when_ir_off=false.

You can set this property using the UI its under Detailed Setup... Customize...Delete LiveTV files when not using IR.

[gveres]

Is there any harm in having the comskip bat file delete the .QFixed file? If I understand SJQ properly, then the fact that this mpg file was QFixed is stored in the database and does not need to be tracked with the .QFixed file anymore.

In fact, do we even need the .QFixed or the .working file at all? I guess we need it in case Sage crashes and restarts.

And I think from the comskip run, all I really need to keep around is the .edl file, right. Today it leaves around a .txt, .logo.txt, .edl and .log. This really makes finding recordings in explorer hard.

your thoughts?

[nyplayer]

Originally Posted by gveres

Is there any harm in having the comskip bat file delete the .QFixed file? If I understand SJQ properly, then the fact that this mpg file was QFixed is stored in the database and does not need to be tracked with the .QFixed file anymore.

In fact, do we even need the .QFixed or the .working file at all? I guess we need it in case Sage crashes and restarts.

your thoughts?

I use the QFIX and Working files because I run th SJQ both in the server and a client. I run it on the Server When nothing is recording and on the Client when nothing is being watched. If you are running it in only 1 Place You do not need it.

[gveres]

Great. Thanks.

I will keep them there for now, and once I get it working I will remove them.

right now it just doesn't seem to be scheduling anything. I am waiting the 10 minutes for the browser to load the log file so that I can see what is going on. Hmmm, I guess the log file is on disk somewhere, I'll just go look for it.

Greg

[gveres]

Originally Posted by nyplayer

I use the QFIX and Working files because I run th SJQ both in the server and a client. I run it on the Server When nothing is recording and on the Client when nothing is being watched. If you are running it in only 1 Place You do not need it.

I just had a thought. You can use the existance of the .edl instead of the .Qfixed file. So the Comskip.bat file could delete the .QFixed file once the .edl file was there. Then you could modify the rules to not do the Qfix operation if the .edl already exists.

But then again, maybe you don't care about what it looks like in explorer.

Greg

[Slugger]

Originally Posted by gveres

Great. Thanks.

I will keep them there for now, and once I get it working I will remove them.

right now it just doesn't seem to be scheduling anything. I am waiting the 10 minutes for the browser to load the log file so that I can see what is going on. Hmmm, I guess the log file is on disk somewhere, I'll just go look for it.

Greg

The log entries are stored in the SJQ database. Writing a program that will extract the log from the database and write it to file is on the todo list, but I haven't even started to think about it yet. If you have a relatively large media library then I do understand your pain in trying to read the log via the servlet. I spent a few minutes investigating possible fixes I can try, but it seems the YUI library has no solution right now. I think the next release of SJQ will dump the pretty tree view of the log and revert back to plain text, which will solve this problem.

[gveres]

Thanks, my library is probably 400+ recordings and 15,000 photos.

I thought purging the log would help but it does not. I am guessing because when I wake up the service and ask it to run now, it processes each recording again trying to find the one that needs to be processed and each time it does, it adds an entry to the log. So I am destined to always have a huge log file.

The strange thing that I found is that I woke it up, it processed the library for a minute or so looking for the recording to work on. Then it said that it was running Qfix.bat but nothing happened. I was watching to see if the tray icon of VideoReDo would show up. It did not. Then after a while, SJQ said it was going back to sleep.

I guess I need to get a look at the log to see what might have happened. I know the bat file works because I have it working with DirMon2.

Greg

[gveres]

I figured out the problem: I used %F% instead of %f%.

It still isn't working but I am making progress.

I think it has to do with working directory, I don't know what the working directory is.

[Slugger]

Originally Posted by gveres

I figured out the problem: I used %F% instead of %f%.

It still isn't working but I am making progress.

I think it has to do with working directory, I don't know what the working directory is.

The working directory is the SageTV home dir (typically C:\Program Files\SageTV\SageTV).

I'm working on fixing the log display to use plain text instead of the tree view. I'm just about done, but a recording started so I can't test anything further until the recordings stop. By dinner I will hopefully have a v1.3.0 build that will allow you to at least view the logs without locking up the browser.

[Slugger]

SJQ v1.3.0 is now available.

v1.2.0 -> v1.3.0
================

* 1947915: New test: TimeUntilNextRecording
* 1961517: Catch all exceptions; prevent plugin from terminating early
* 1961766: Display logs in plain text; dump YUI tree widget

[gveres]

This version makes it easier to get the monster list of jobs (I was trying to get it to run more frequently while debugging and had it run every 2 minutes), but I have not successfully pulled up the details of a single run with this new version. It hangs and pegs the cpu. Remember, I have 15000 photos in my library and another 400 or so recordings.

You might want to consider having a flag that controls whether the user wants you to process music and photos. I am sure it would save some cpu time if you didn't have to look at my photos, and I don't want them touched anyway. This would then cut the log files down significantly.

BTW, it wasn't processing and then I restarted Sage this morning and it started processsing all the files from yesterday.

[Slugger]

Originally Posted by gveres

This version makes it easier to get the monster list of jobs (I was trying to get it to run more frequently while debugging and had it run every 2 minutes), but I have not successfully pulled up the details of a single run with this new version. It hangs and pegs the cpu. Remember, I have 15000 photos in my library and another 400 or so recordings.

You might want to consider having a flag that controls whether the user wants you to process music and photos. I am sure it would save some cpu time if you didn't have to look at my photos, and I don't want them touched anyway. This would then cut the log files down significantly.

BTW, it wasn't processing and then I restarted Sage this morning and it started processsing all the files from yesterday.

Give this build a try. After instaling, the servlet should report version '1.3.0a'. This build includes a new feature that allows you to tell SJQ to ignore certain types of media files. By default it will only process tv files, but all the others can be turned on via the servlet config settings. I only built the Windows installer version of this build, if you need the zip file then just let me know and I can build it. This build will become 1.4.0, but I just want to make sure that this log file issue is solved before making yet another 'official' release so your feedback is appreciated.

[nyplayer]

Slugger,

I just tried your latest build '1.3.0a' what a difference. The Scan was quicker and I can browse the logs in no time. Great job!!!!!