Plugin: Sage Job Queue (SJQ) v3

[Slugger]

Just to provide an example of what I'm up to, here's my current rule for deciding what gets archived to my NAS.

if [IsObjMediaFile == true && $.IsTVFile{} == true && (TimeUntilNextRecording >= "6H" || FreeSpacePercentage < 7) && $.IsFileCurrentlyRecording{} == false && (((($.GetShowCategory{} == "Movie" && $.GetShowSubCategory{} != "Documentary") || $.GetShowTitle{} =% "Friends|Seinfeld|Night Court|Three's Company|Family Guy|Married \\.\\.\\. With Children") && $.IsLibraryFile{} == false) || ($.IsLibraryFile{} == true && Filename !^ "\\\\nas"))] {
   :MOVERECOPTS "\\\\nas\\tv"
   :FAILALERT "true"
   _MOVEREC
}

Basically, this test says to archive any tv recording as long as it's not actively recording AND there's at least 6 hours until the next recording OR there's less than 7% free space on my recording drives AND any one of the following conditions are met:

• The recording is a movie and not a documentary
• The title matches any of the titles in the given regex (current syndicated shows I'm recording)
• For both of the above, the recording is only moved to the NAS if it's not marked as archived already (prevent endless loop of trying to archive the same recordings over and over; things archived with _MOVEREC task are automatically marked as archived by SageTV after they are moved)
• Move a recording to the NAS if it's archived, but isn't already on the NAS (this allows me to mark something for the NAS without having to use titles or categories, etc)

:MOVERECOPTS tells SJQ which directory I'm archiving to (currently the UNC path "\\nas\tv"). The :FAILALERT option tells SJQ to create an alert with SageAlert should the archive attempt fail. Use SageAlert to notify yourself via email, Twitter, SMS, etc. (SageAlert options require SJQ v3.0.0.650 or newer).

Hopefully, this example shows that the sky's the limit as far as automating archival to network storage while SageTV is running.

Obligatory Disclaimer: Use of the _MOVEREC task in SJQ can cause irreversible damage to your SageTV system (lost/orphaned recordings, corrupted wiz.bin, some other unknown type of damage). Please consider the _MOVEREC task in SJQv3 to be EXPERIMENTAL. I take no responsibility for any damage caused to your SageTV through its use.

With that said, I've been using it quite happily for a few weeks now with no issues. The only known issue I'm aware of is that if you lose power or SageTV is shutdown unexpectedly during a _MOVEREC task then the recording being copied may end up being orphaned (the original recording should still be safe on your hard drive, but the link between it and its airing info in SageTV may have been lost). In such a case, the state of the operation is actually saved in SJQ and can be continued/recovered, but that continuation step has not been coded yet.

[zzmystique]

I've been plagued recently with losing my guide data.

Can anyone suggest a way I can get SJQ to trigger a sagealert to warn me it has happened?

Thanks

[Slugger]

Originally Posted by zzmystique

I've been plagued recently with losing my guide data.

Can anyone suggest a way I can get SJQ to trigger a sagealert to warn me it has happened?

Thanks

Does SageTV generate a system message when this happens? If so, then you could have SJQ process the system message and then trigger an alert based on the content of the system message. Of course, if system messages are being generated then SageAlert itself already processes system messages on its own. That's about the best SJQ can do since it's only able to process media objects and system message objects - it knows nothing about the EPG itself.

[nyplayer]

Slugger,

It appears that sometimes the internal tasks just hang out there for some reason and not run. I remember in SJQ 2 you would check to make sure internal task scheduler is running.... are you doing the same in SJQ 3?

[Slugger]

Originally Posted by nyplayer

Slugger,

It appears that sometimes the internal tasks just hang out there for some reason and not run. I remember in SJQ 2 you would check to make sure internal task scheduler is running.... are you doing the same in SJQ 3?

No, I removed that "feature", shouldn't be necessary. If tasks aren't running then get a dump of the app by visiting the /sjq/SJQ?debug=TRUE and paste the contents of sjq.thread.dump (in root SageTV install dir) - also look for any exception dumps in sjq.log (same location) and paste those in. That watchdog process shouldn't be necessary and it just masks bugs that need to be squashed.

[nyplayer]

Slugger,

I have it running in Debug Mode now it Appears that this happens after a _MOVEREC fails... no more internal tasks run until I reset SJQ.

[tomfisk]

Originally Posted by nyplayer

Slugger,

It appears that sometimes the internal tasks just hang out there for some reason and not run. I remember in SJQ 2 you would check to make sure internal task scheduler is running.... are you doing the same in SJQ 3?

I've experienced this "feature" also Haven't done any debugging to this point to try to figure out why...

[nyplayer]

Slugger,

On the way you do a move I see you rename the good file and add .tmp extension. Before actually copying it over to the destination. In the event of a sage crash etc... you would lose the original file because it is now named .tmp.

The way sagetvranscoder does it is save the file to the destination with .tmp at the end. Before deleting the original and renaming the .tmp to the original name and relinking.

[zzmystique]

Originally Posted by Slugger

Does SageTV generate a system message when this happens? If so, then you could have SJQ process the system message and then trigger an alert based on the content of the system message. Of course, if system messages are being generated then SageAlert itself already processes system messages on its own. That's about the best SJQ can do since it's only able to process media objects and system message objects - it knows nothing about the EPG itself.

I checked the system messages and there is no message generated when there is no guide data.

Will email sagetv support and see if they can suggest a way of monitoring for no guide data.

Thanks

[Slugger]

Originally Posted by nyplayer

Slugger,

I have it running in Debug Mode now it Appears that this happens after a _MOVEREC fails... no more internal tasks run until I reset SJQ.

Ok, wouldn't be totally surprised to hear that. Next time internal tasks stop running get that thread dump by visiting /sjq/SJQ?debug=TRUE - that will tell us if the InternalTaskClient indeed crashed.

[Slugger]

Originally Posted by nyplayer

Slugger,

On the way you do a move I see you rename the good file and add .tmp extension. Before actually copying it over to the destination. In the event of a sage crash etc... you would lose the original file because it is now named .tmp.

The way sagetvranscoder does it is save the file to the destination with .tmp at the end. Before deleting the original and renaming the .tmp to the original name and relinking.

The file's not lost - it's still on the HD, just renamed. I know it's not the ideal way to do it and I had originally coded it the other way, but changed it b/c I didn't want to waste time copying the file unless I knew I could delete the original so that's why I rename the original right away. I figured if the file is in use by a client (being viewed, for example) then I'll know right away if the rename of the file fails and then I can just abort the entire operation without wasting time trying to copy multiple GBs across the network (or where ever the destination of the move is). I'm still very open to changes in the _MOVEREC algorithm.

[Slugger]

@nyplayer - I'm also curious to know why your _MOVEREC tasks are failing? Is the rollback working when they do fail?

[nyplayer]

Yes the roll back works. It said that the file was in use.. for some reason.

[nyplayer]

Originally Posted by Slugger

@nyplayer - I'm also curious to know why your _MOVEREC tasks are failing? Is the rollback working when they do fail?

I believe the file was in use... that is the reason why it failed... and this appears to cause all other scheduled tasks to not run. I fully understand why you check to make sure file is not in use first.

PS I did test the rollback and it worked. Took a usb drive offline while a file was moving just to test and it worked fine.

Source: N:\sagen\LawOrder-Humiliation-4664989-0.mp4
Target: T:\saget
Renaming source file... DONE!
Copying source file to target directory... FAILED!
Rolling back archival attempt due to a step failing!
Deleting target file... ALREADY DONE!
Renaming source file... DONE!
Deleting target media object in SageTV... ALREADY DONE!
Recreating source media object in SageTV... ALREADY DONE!
Relinking source media object to airing object... DONE!
Rollback completed SUCCESSFULLY!

[nyplayer]

Slugger,

I tracked my problem to this paricular move.

java.io.IOException: Deletion of source failed!
at com.google.code.sagetvaddons.sjq.server.TvArchiver.delSourceMedia(TvArchiver.java:645)
at com.google.code.sagetvaddons.sjq.server.TvArchiver.run(TvArchiver.java:360)
at com.google.code.sagetvaddons.sjq.server.InternalTaskClient.moveRec(InternalTaskClient.java:115)
at com.google.code.sagetvaddons.sjq.server.InternalTaskClient.performTask(InternalTaskClient.java:97)
at com.google.code.sagetvaddons.sjq.server.InternalTaskClient.run(InternalTaskClient.java:53)
at java.lang.Thread.run(Unknown Source)
Sun Nov 15 16:01:12 PST 2009
Source: N:\sagen\LawOrderSpecialVictimsUnit-Users-4595432-0.mp4
Target: T:\saget
Renaming source file... DONE!
Copying source file to target directory... DONE!
Deleting source media object in SageTV... FAILED!
Rolling back archival attempt due to a step failing!
Deleting target file... DONE!
Renaming source file... DONE!
Deleting target media object in SageTV... ALREADY DONE!
Recreating source media object in SageTV... ALREADY DONE!
Relinking source media object to airing object... DONE!
Rollback completed SUCCESSFULLY!

[Slugger]

I need to know which thread(s) are crashing when a _MOVEREC task fails. Can you reproduce, then when the queue stops being processed, load the /sjq/SJQ?debug=TRUE URL then paste in the output from sjq.thread.dump, please?

The exception is being thrown b/c even though the file seems to have been renamed, Sage still refuses to delete the media file from wiz.bin (DeleteFileWithoutPrejudice() API call is failing). So the archiver task is throwing an exception and it seems to be caught, handled and rolled back properly. Looking at the code, I don't see why any thread would be dying off as a result of the TvArchiver task throwing an exception. So if you can get that thread dump output for me, it should answer all my questions.

Thanks.

[Slugger]

New task client snapshot 655 available: fixes a JSONException generated when the client tries to log task client errors back to the server; when the client tried to log this info back to the server, the client would crash as a result of this exception being thrown

[gveres]

Any ideas? This was working until I rebooted. I upgraded to v3 a few days ago.

Never mind. build 655 above fixed this issue. It's like Slugger read my mind.

wrapper  | --> Wrapper Started as Console
wrapper  | Java Service Wrapper Community Edition 3.3.1
wrapper  |   Copyright (C) 1999-2008 Tanuki Software, Inc.  All Rights Reserved.
wrapper  |     http://wrapper.tanukisoftware.org
wrapper  | 
wrapper  | Launching a JVM...
jvm 1    | WrapperManager: Initializing...
jvm 1    | SJQ Task Client v2.1.5.556
jvm 1    | Using 'C:/sjqc' as location for configuration file...
jvm 1    | WrapperSimpleApp: 
jvm 1    | WrapperSimpleApp: Encountered an error running main:
jvm 1    | WrapperSimpleApp: java.lang.RuntimeException: java.io.IOException: Server returned HTTP response code: 500 for URL: http://localhost:8080/sjq/SJQ?proto=2&cmd=writeLog
jvm 1    | WrapperSimpleApp: 	at com.google.code.sagetvaddons.sjqc.TaskClient.log(Unknown Source)
jvm 1    | WrapperSimpleApp: 	at com.google.code.sagetvaddons.sjqc.TaskClient.log(Unknown Source)
jvm 1    | WrapperSimpleApp: 	at com.google.code.sagetvaddons.sjqc.TaskClient.pop(Unknown Source)
jvm 1    | WrapperSimpleApp: 	at com.google.code.sagetvaddons.sjqc.TaskClient.run(Unknown Source)
jvm 1    | WrapperSimpleApp: 	at com.google.code.sagetvaddons.sjqc.TaskClient.main(Unknown Source)
jvm 1    | WrapperSimpleApp: 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
jvm 1    | WrapperSimpleApp: 	at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
jvm 1    | WrapperSimpleApp: 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
jvm 1    | WrapperSimpleApp: 	at java.lang.reflect.Method.invoke(Unknown Source)
jvm 1    | WrapperSimpleApp: 	at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:236)
jvm 1    | WrapperSimpleApp: 	at java.lang.Thread.run(Unknown Source)
jvm 1    | WrapperSimpleApp: Caused by: java.io.IOException: Server returned HTTP response code: 500 for URL: http://localhost:8080/sjq/SJQ?proto=2&cmd=writeLog
jvm 1    | WrapperSimpleApp: 	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
jvm 1    | WrapperSimpleApp: 	... 11 more
wrapper  | <-- Wrapper Stopped
Press any key to continue . . .

Thanks
Greg

[Slugger]

Originally Posted by gveres

Any ideas? This was working until I rebooted. I upgraded to v3 a few days ago.

Never mind. build 655 above fixed this issue. It's like Slugger read my mind.

655 didn't fix this issue, directly. The issue here was that you were running task client v2.1.5.556 with a v3.0.0 server. The HTTP 500 errors come about because a 2.x client can't talk with a 3.x server. Of course, upgrading the task client to a 3.x version would immediately fix the problem.

[nyplayer]

Slugger,

Nothing is processing.. its just hung.

Mon Nov 16 23:57:39 PST 2009: SJQ Application Dump
	QueueLoader thread is alive...
		java.lang.Thread.sleep(Native Method)
		com.google.code.sagetvaddons.sjq.server.MediaFileQueueLoader.run(MediaFileQueueLoader.java:101)
		java.lang.Thread.run(Unknown Source)
	ClientMonitor thread is alive...
		java.lang.Thread.sleep(Native Method)
		com.google.code.sagetvaddons.sjq.server.ClientMonitor.run(ClientMonitor.java:35)
		java.lang.Thread.run(Unknown Source)
	InteralTaskClient thread is alive...
		java.lang.Thread.sleep(Native Method)
		com.google.code.sagetvaddons.sjq.server.InternalTaskClient.run(InternalTaskClient.java:59)
		java.lang.Thread.run(Unknown Source)
	Most recent LogCleaner thread is dead (this is normal behaviour)!
	Most recent FileCleaner thread is dead (this is normal behaviour)!