Plugin: Sage Job Queue (SJQ)

[wayner]

Originally Posted by Slugger

For example, you know HBOHD is commercial free so there's no need to run comskip and waste hours processing a recording that is known not to have any commercials. Such a rule might look like this:

if [IsTV == true && IsActivelyRecording == false && ChannelName != "HBOHD"] { # Ignore HBOHD recordings
   :PRIORITY 5 # optional
   HDCOMMSKIP
}

# Use ChannelName !^ "HBO" to ignore any HBO channels (since none of them would have commercials); the !^ operator means "does not start with".  All of the operators are documented in the wiki.

How would you do this same function, ie. don't run Comskip for a block of channels rather than just HBOHD? Do you need separate ifs or can you put ORs, i.e. ChannelName != ("HBOHD" or "TMNHD" or "MXSHD")

[vexhold]

correct me if I am wrong but this should work for ignoring multiple channels


if [IsTV == true && IsActivelyRecording == false && ChannelName != "HBOHD" && ChannelName != "TMNHD" && ChannelName != "MXSHD"] {
:PRIORITY 5 # optional
HDCOMMSKIP
}

[Slugger]

Originally Posted by vexhold

correct me if I am wrong but this should work for ignoring multiple channels


if [IsTV == true && IsActivelyRecording == false && ChannelName != "HBOHD" && ChannelName != "TMNHD" && ChannelName != "MXSHD"] {
:PRIORITY 5 # optional
HDCOMMSKIP
}

You've pretty much got it. The above will work just fine. You could also use a single regular expression:

if [IsTV == true && IsActivelyRecording == false && ChannelName !% "HBOHD|TMNHD|MXSHD"] {
   :PRIORITY 5
   HDCOMMSKIP
}

The important thing here is that the && (AND) and || (OR) operators can only be used between complete tests and not within the value of a test.

[gasinger]

I believe that I'm having a problem with my SJQ client getting stuck in the "STARTED" state when the file in the active queue no longer exists when the task starts. I'm seeing it when the tasks get backed up trying to process comskip on H264 recordings. I have intelligent suggestions turned on, so it is continually cycling through recordings. My only way to fix it is to stop the client, touch the sjq context file and then start the client back up.
Any thoughts on how I could remove tasks from the active list when the file gets deleted before the task has started?

Thanks,
Gregg

[Slugger]

Originally Posted by gasinger

I believe that I'm having a problem with my SJQ client getting stuck in the "STARTED" state when the file in the active queue no longer exists when the task starts. I'm seeing it when the tasks get backed up trying to process comskip on H264 recordings. I have intelligent suggestions turned on, so it is continually cycling through recordings. My only way to fix it is to stop the client, touch the sjq context file and then start the client back up.
Any thoughts on how I could remove tasks from the active list when the file gets deleted before the task has started?

Thanks,
Gregg

This is a bug. I just checked in a quick temp fix that will identify these tasks and mark them as done without assigning them to a client. The proper fix (make them vanish from the queue) will be done up when I return home next week.

Since I don't have access to an env to test the fix right now you'll have to trust my 1am java coding abilities. However, I'm fairly confident that it will do what it's suppose to or at the very least won't do any more harm than the current version. There is no way around this bug in the current version of SJQ other than to ensure that the media file exists for a queued task at the time the task is assigned to a client.

The auto build scripts won't run until 6am so the build with this fix (snapshot 165) won't be available until shortly after 6am EST this morning. If you do give the snapshot a try please let me know if it works so I can use it as the basis for the complete fix next week.

[gasinger]

Snapshot 165 has fixed the client task getting stuck in "STARTED" state. However, I have also had a couple of random "no response" from the SJQ server since then that I can't quite put my finger on that requires a restart of the SageTV service and not just a touch of the SJQ context. The SRE & SageTV Mobile web still work, so it doesn't seem that the jetty plug-in is crashing.

Thanks,
Gregg

[Slugger]

Originally Posted by gasinger

Snapshot 165 has fixed the client task getting stuck in "STARTED" state. However, I have also had a couple of random "no response" from the SJQ server since then that I can't quite put my finger on that requires a restart of the SageTV service and not just a touch of the SJQ context. The SRE & SageTV Mobile web still work, so it doesn't seem that the jetty plug-in is crashing.

Thanks,
Gregg

This sounds like another null pointer exception bug, probably caused by the same issue (recording file deleted between the time a job is queued and assigned by the server). Unfortunately, I don't see where it might be happening.

Next time this happens can you try to load the GUI? Do you get a 404 error from Jetty? This would tell me that SJQ has crashed. If that does happen, can you enable debug logging in Jetty and look for a stack trace? A stack trace would really help me locate the problem area.

In the mean time, I'm going to rework the fix so that these "problem" tasks simply vanish from the queue instead of being marked as done, which may fix this new issue as well. Hopefully I'll find a couple hours some evening this week to do it.

Thanks.

[gasinger]

Originally Posted by Slugger

Next time this happens can you try to load the GUI? Do you get a 404 error from Jetty? This would tell me that SJQ has crashed. If that does happen, can you enable debug logging in Jetty and look for a stack trace?

The GUI is what I'm getting the no response error. I believe that it is the browser that is returning the timeout and not Jetty. I'll try to enable logging in the jetty plug-in to see if I can get any more details for you.

Thanks,
Gregg

[Slugger]

Originally Posted by gasinger

The GUI is what I'm getting the no response error. I believe that it is the browser that is returning the timeout and not Jetty. I'll try to enable logging in the jetty plug-in to see if I can get any more details for you.

Thanks,
Gregg

Please try snapshot 166. It contains the "proper" fix for the original bug, that is to simply drop queued tasks that have no valid media file object available at the time of assignment to a client. Keep the debugging on in case these symptoms persist as I haven't been able to reproduce them myself yet with 165 nor 166.

[gasinger]

Originally Posted by Slugger

Please try snapshot 166. It contains the "proper" fix for the original bug, that is to simply drop queued tasks that have no valid media file object available at the time of assignment to a client. Keep the debugging on in case these symptoms persist as I haven't been able to reproduce them myself yet with 165 nor 166.

Okay, I've downloaded and installed snapshot 166. As for my other problem, it did happen again after I thought I had turned on the jetty debug, but I don't see any additional lines in the sagetv log file. And when I cleared the browser cache, I am getting a 404 error from jetty. However, I think it may be related to one of my recording directory/drives is failing.

[Slugger]

Originally Posted by gasinger

Okay, I've downloaded and installed snapshot 166. As for my other problem, it did happen again after I thought I had turned on the jetty debug, but I don't see any additional lines in the sagetv log file. And when I cleared the browser cache, I am getting a 404 error from jetty. However, I think it may be related to one of my recording directory/drives is failing.

When you enable debug logging in jetty you should have a file in SageTV\jetty\logs\ named date.stderr.log. Check there for any output.

[gasinger]

Originally Posted by Slugger

When you enable debug logging in jetty you should have a file in SageTV\jetty\logs\ named date.stderr.log. Check there for any output.

I just noticed the time out problem again. I attached the most current jetty log file. The behavior is a bit different with snapshot 166, the logon window appears in the GUI, but I get a message that login failed "A request timeout has expired after 5000 ms" and the status in the lower right corner shows the text Loading... Loading... SJQ v2.0.1.166

[Slugger]

Originally Posted by gasinger

I just noticed the time out problem again. I attached the most current jetty log file. The behavior is a bit different with snapshot 166, the logon window appears in the GUI, but I get a message that login failed "A request timeout has expired after 5000 ms" and the status in the lower right corner shows the text Loading... Loading... SJQ v2.0.1.166

Do you have the stderr.txt file? The timeout problem suggests that there's a deadlock in my code. Again, my problem is I really don't see where this problem lies and really need a backtrace or a very specific set of instructions on how to reproduce the problem.

Is it just a matter of the media file not being available at the time the corresponding task is to be assigned to a client that triggers all of these problems? If so, is it constantly reproducible?

[gasinger]

Where would I find the stderr.txt? I don't see it in the jetty log directory. However, I think this problem is not related to the missing media file, but when I switch back and forth between the completed and active pages too fast. I was just looking at the status of the active jobs and trying to determine how long the current task has been running and that is when it hung and now it's sitting at the same spot as before and I have to stop the sagetv service and restart to clear it.

thanks.

Edit: Maybe I don't have debug enabled properly for jetty. Should it be DEBUG=true in the jettystarter.properties found in the sagtv directory?

[Slugger]

Originally Posted by gasinger

Where would I find the stderr.txt? I don't see it in the jetty log directory. However, I think this problem is not related to the missing media file, but when I switch back and forth between the completed and active pages too fast. I was just looking at the status of the active jobs and trying to determine how long the current task has been running and that is when it hung and now it's sitting at the same spot as before and I have to stop the sagetv service and restart to clear it.

thanks.

Edit: Maybe I don't have debug enabled properly for jetty. Should it be DEBUG=true in the jettystarter.properties found in the sagtv directory?

Yes, you have to enable logging in that file then the stderr file should show up. Are you able to easily reproduce the hang? Quickly changing between views of the queue shouldn't cause this type of deadlock, but I can be convinced otherwise if you're able to easily reproduce the scenario. I've tried for the last 10 mins by constantly flipping between the two views and am unable to reproduce what you're describing.

I'm not trying to say a bug doesn't exist, instead I mention my inability to reproduce because if I can't reproduce it and I have no backtrace to work with then I have no idea where to start looking to fix it. So as you try to reproduce it please be sure to keep track, with as much detail as possible, exactly how to reproduce the problem.

[gasinger]

No, I can't reproduce it easily. Plus, I can't seem to get the jetty debug stderr to be created. The text in the jettystarter.properties indicates that it may be in the sagetv_0.log file, since that is where sagetv logs also. I am getting the sagtv logs in that file, but I don't find anything related to jetty in there. Although I don't know what to look for other that the word jetty.

[Slugger]

Originally Posted by gasinger

No, I can't reproduce it easily. Plus, I can't seem to get the jetty debug stderr to be created. The text in the jettystarter.properties indicates that it may be in the sagetv_0.log file, since that is where sagetv logs also. I am getting the sagtv logs in that file, but I don't find anything related to jetty in there. Although I don't know what to look for other that the word jetty.

You should see debug output from SJQ. At the very least, output like the following should start to appear.

=== GLOBAL OPTIONS ===
MAXPROCS => 3.0
=== GLOBAL OPTIONS ===
This client has 2 ClientTask object(s) attached to it.
=== ClientTask 'HDCOMM' ===
MAX => 1.0
MAXRUNRATIO => true
MAXRUN => 1.75
CPU => NORMAL
Run: C:/comskip/comskip_sjq.bat "%c%"
=== ClientTask 'HDCOMM' ===

=== ClientTask 'SDCOMM' ===
MAX => 2.0
MAXRUNRATIO => true
MAXRUN => 0.25
CPU => NORMAL
Run: C:/comskip/comskip_sjq.bat "%c%"
=== ClientTask 'SDCOMM' ===

The output will be slightly different based on the contents of your client config(s), but it'll start spitting this type of output on each run of the TaskQueue. If you're seeing that then you're looking for stack traces from crashes (standard Java stack trace):

Exception in the MiniUIClientReceiver of: java.io.EOFException
java.io.EOFException
        at sage.bk$a.run(Unknown Source)

Except ones for SJQ. Search for the string 'com.google.code.sjq' as that's the package containing all of the SJQ code.

[gasinger]

Thanks. I do see the SJQ logs and then they just stop showing up ~ 8 hours after the last task that was started and that task was still running comskip. No Java Exception errors that I can find. I'll keep monitoring this problem and report back if I can narrow it down.

[scott2020]

I'd like to share briefly my experience so far. First, I downloaded, extracted, and installed to the letter of the directions, but I could not get SJQ to answer my web requests. Jetty was working fine, but I constantly received SERVICE_UNAVAILABLE accessing SJQ. I couldn't find anyone having this problem! I almost gave up, but I pressed on. I re-downloaded and extracted on a Windows XP machine, and everything started working perfectly! I was doing the extraction, etc, from my Vista machine. Lousy Vista! Anyway, after re-doing everything from an XP machine SJQ is running great. I have my Vista client running SJQ client to start Comskip, which runs against an XP machine which is my SageTV server. I am mapping the drive on the client to the server, Comskip runs great, and everything appears to be working well.

The only issue I see now is in the SJQ task history, everything is in the "Failed Tasks" list. Even Comskip jobs that seem to have run properly, created TXT, EDL, and good Log files and processed the MPG file, they still show up as failed tasks in the history.

Thanks for a great program! It really is incredible. I have been wanting to run Comskip on my Vista Home machine to take advantage of the quad core processor, but had many authentication problems trying to connect back to my XP machine with Comskip Monitor or Dirmon2 running as a service. Since I don't have a domain I could not get the services to properly authenticate and Vista does not like to run things interactive with the desktop. The SJQ client running in a batch file after establishing a connection to my server works great!

thanks again.
Scott

[Slugger]

Originally Posted by scott2020

The only issue I see now is in the SJQ task history, everything is in the "Failed Tasks" list. Even Comskip jobs that seem to have run properly, created TXT, EDL, and good Log files and processed the MPG file, they still show up as failed tasks in the history.

See this post for details. Read the replies that follow it for an example batch file that will start marking completed comskip jobs properly.