Plugin: SJQv4

[lenkost]

Hello,

1. I do not have anything in "Log for the executable" in SJQ UI for completed comskip tasks.

Here is my sjqagent.properties file:

#Generated by SJQv4 agent
#Thu Jan 20 20:00:33 EST 2011
task.comskip.exe=C\:\\Comskip\\Comskip.exe
task.comskip.schedule=ON
task.comskip.args="\\\\LenTV\\SageTV1\\$SJQ4_LAST_SEGMENT"
task.comskip.test=\\\\LenTV\\SJQScripts\\comskip_test.groovy
agent.schedule=ON
task.comskip.testargs=ts
task.comskip.rcmin=0
task.comskip.maxprocs=4
task.comskip.maxtimeratio=1.0
agent.resources=100
task.comskip.maxtime=14400
task.comskip.rcmax=1
agent.port=23344
task.comskip.resources=25

2. I had to hard code network path in executable arguments and disable my second drive temporarily. See post #89. I know there is a ticket for "MapDir" option. Any update on a progress on this issue?

Thank you.

[Slugger]

@lenkost

1. I believe you're hitting issue 274. I believe comskip is known to dump tonnes of output depending on what log level you run it as (default is rather high, I believe, and will generate more than 64K of output). You'll want to star that issue on the project site so you'll receive updates to to the ticket. That is the only high priority bug I have right now and it is the next thing to be fixed, but as I discuss briefly in the ticket, it's a PITA to fix and requires some serious consideration before just going crazy in the code.

2. Star issue 268. There is no progress made on it to this point other than I learned that trying to do a straight port of how I did it in SJQv3 will not work in v4 so it's back to the drawing board on this one. With that said, you can implement your own dir mapper in a Groovy script (and even my sol'n for issue 268 will only work in Groovy scripts). Here's a quick example:

// So let's map any recordings from D:\tv\ to \\server\tv\
def dirMap = ["D:\\tv\\":"\\\\server\\tv\\"]
for(Object mediaFile : MediaFileAPI.GetMediaFiles("T")) {
   File[] files = MediaFileAPI.GetSegmentFiles(mediaFile)
   for(int i = 0; i < files.length; ++i) {
      String path = files[i].getAbsolutePath()
      print path
      for(String dir : dirMap.keySet()) {
         if(path.startsWith(dir)) {
            path = path.replace(dir, dirMap.get(dir))
            break
         }
      }
      files[i] = new File(path)
      println " > ${files[i]}"
   }
}

This little script will convert the file paths of a MediaFile object as defined in the given map at the top of the script. Ultimately, this is what I would be providing in the fix for the ticket. But I have to worry about how/when I will get the map and process it. If you implement this yourself, you don't have to worry about it. Make this code a function in your script and then just call the function prior to processing the media object. In the code above, the files array is replaced with the remapped file paths. I also print out the mapping so I can see it working. This script produced the following (partial) output on my server:

D:\tv\TheBigBangTheory-TheLoveCarDisplacement-4884934-0.ts > \\server\tv\TheBigBangTheory-TheLoveCarDisplacement-4884934-0.ts
D:\tv\AmericanIdol-AuditionsNo2NewOrleans-4986030-0.mpg > \\server\tv\AmericanIdol-AuditionsNo2NewOrleans-4986030-0.mpg
D:\tv\MyDadSays-CornStar-4884915-0.ts > \\server\tv\MyDadSays-CornStar-4884915-0.ts
D:\tv\Community-AsianPopulationStudies-4880822-0.mpg > \\server\tv\Community-AsianPopulationStudies-4880822-0.mpg
D:\tv\PerfectCouples-Pilot-4880825-0.mpg > \\server\tv\PerfectCouples-Pilot-4880825-0.mpg
D:\tv\TheColbertReport-4883574-0.ts > \\server\tv\TheColbertReport-4883574-0.ts
D:\tv\TheOffice-Ultimatum-4884756-0.ts > \\server\tv\TheOffice-Ultimatum-4884756-0.ts
D:\tv\ParksandRecreation-GoBigorGoHome-4884748-0.ts > \\server\tv\ParksandRecreation-GoBigorGoHome-4884748-0.ts
D:\tv\30Rock-MrsDonaghy-4884757-0.ts > \\server\tv\30Rock-MrsDonaghy-4884757-0.ts

[lenkost]

I have no idea how to use this. I will rather wait until you come up with the final solution.
I prefer to have it as simple as some additional options in Task Edit.

Thank you.

[craig2012]

Greetings,

I recently (2 weeks ago) upgraded to Sage7 and purchased the required SJQv4 sagetv addon license so I could automate Comskip. Everything worked great for a bit but a week ago Comskip quit working and that is when these errors started logging and now my system crashed from the .lobs.db folder filling up with log files that pretty much are filled with a zillion entries like below.

==============================

===== Fri Jan 14 12:04:14 CST 2011 =====

LOG4J: Configured Logging for: sagex-api using file: sagex-api.log4j.properties
----- stdout -----

ERROR: Invalid environment data passed to scritpt!
------------------

there are occasional successful entries such as:

----- stdout -----

Recording in progress, waiting...
------------------

but they are the exception.

How can I fix this invalid environment so it doesn't log anymore or how can I stop it from being logged at least. Or perhaps put a cap on the directory size so it doesn't grow to is current 35GB+ size again. Perhaps even have it logg to the 2TB files disk versus the 40GB OS disk?

Thanks,

Craig

[Slugger]

I don't know how you get 35GB worth of data, but there's no way for me to limit that. However, I just checked in a change to comskip_test.groovy that should prevent multiple reruns of a task with invalid data.

Download the latest comskip_test.groovy and update it on your system (check the comment at the top of the file and ensure the last edit date reads 21 Jan 2011).

To get 35GB worth of data would require 100s of thousands of runs of tasks... if you see that many tasks being rerun constantly then you need to take some preventative action. At the very least, simply stop SJQ, gather some logs/debug info/etc. and post a message here (as you did, but I'd probably do it before your db grows to 35GB ).

[Slugger]

Hmm... some research has shown that there is an issue with the H2 network client where it caches query results for CLOB/BLOB columns and may never properly delete them. This is why we see a .lobs.db directory in the root SageTV dir - it's created by the H2 client, which is used by the SJQ task client.

Good news: Toasting off the SageTV\.lobs.db\ directory (and all subdirs and files in it) is absolutely fine as long as Sage is not running. Do NOT toast off anything in SageTV\plugins\sjq\ - this is where the actual database is stored. Toasting off the clob files in there will corrupt the database. SJQ automatically cleans up the clob files under this dir anyways (by periodically deleting task logs).

I will be looking for a way to automatically toast off the client cache dir at SageTV startup, not sure if I'll be able to since H2 will always be started before any of my plugins that use it. I'll have to think hard about this one. In the mean time, stopping Sage and toasting off SageTV\.lobs.db\ is the best advice I can give.

The script fix above should also reduce the amount of output produced. But the H2 client cache still needs to be wiped periodically, but it can't be done while H2 is running.

[craig2012]

Thanks. I followed this comskip guide when setting it all up using the GUI:
http://code.google.com/p/sagetv-addo...i/Sjq4Comskip1

I just did the SDCompskip part and attached a compskip task to each of my many favorites (over 150 I'd say). That is all I have done with SJQ, I don't have it doing anything else but to automate comskip.

I did delete all but the latest set of .db files from the SageTV\SageTV\.lobs.db directory last night when I made this post. As of now (some 12 hours later) its back up to 170MB already.

I just got the 21 Jan version of the .groovy file and post back tomorrow on the results.

That said, how would I debug the "ERROR: Invalid environment data passed to scritpt!" part to remove what is causing this error in the firs place? Are there any logfiles or configurations I could look at that might shed some light?

Best Regards and thank you for your assistance.

Craig

[Slugger]

Basically, you'd have to debug the script. In this case, load comksip_test.groovy in an editor (try the new SageGroovy IDE I posted yesterday). You'll see the error message in that script that's being generated. You'll have to add a bunch of println() debug statements around that error to figure out exactly what value is causing the problem, etc. The IDE provides some ability to view variables as well, which may be of assistance. println statements are probably the easiest (and most tedious) way to debug it.

I can tell you the usual culprit is a recording that was inserted into the queue and then was deleted or otherwise removed from wiz.bin before the task was assigned to a client. This will cause the endless loop of trying to run the test, the test failing because the media file is no longer valid, and then the task being put back in the queue. The update I made to the test script is to mark tasks as SKIPPED on the error instead of returning them to the queue. It will prevent an endless loop of reruns of the task, but will also mean some tasks may not get comskipped. It's a tradeoff. With some more thought, one may be able to come up with a better solution.

The cache dir is back to 170MB. How big is the sjq.lobs.db directory under plugins/sjq? The cache dir should never be any larger than the lobs dir under plugins/sjq (during a single run). The cache dir can (and should be) cleaned up on Sage startup... I just have to figure out how to ensure that I do that cache cleanup before the H2 plugin is started.

[craig2012]

The SageTV\SageTV\plugins\sjq\sjq4.lobs.db is 13.7MB with 113 files in 46 folders. I did not know it existed and never deleted anything out of it manually before installing the new groovy script.

I did delete everything out of the SageTV\SageTV\.lobs.db directory before running with the new groovy script and as of now the directory is still empty. So that is some progress.

Also when I use the GUI to show the clients it says there are some 38 Active Tasks. Previous to the new groovy script I saw that say 140+ or so if I recall. I'm not sure if that information is relevant.

I'll let you know if shows start getting not getting Comskipped.

Craig

[Slugger]

Originally Posted by craig2012

The SageTV\SageTV\plugins\sjq\sjq4.lobs.db is 13.7MB with 113 files in 46 folders. I did not know it existed and never deleted anything out of it manually before installing the new groovy script.

13MB is reasonable, depending on the kind of task load you're putting on SJQ. But most importantly, never delete anything from that directory. Doing so will corrupt the SJQ database. SJQ does actively cleanup the lob files in this directory.

I did delete everything out of the SageTV\SageTV\.lobs.db directory before running with the new groovy script and as of now the directory is still empty. So that is some progress.

Yes, I'm probably going to have to ask the H2 maintainer to add the cleanup as part of the H2 start up. I just have no way to ensure that H2 hasn't started before I try to do it in any of my plugin code. With this new knowledge, I at least now know that in the mean time, users should be periodically wiping the client cache dir only (SageTV\.lobs.db\).

Also when I use the GUI to show the clients it says there are some 38 Active Tasks. Previous to the new groovy script I saw that say 140+ or so if I recall. I'm not sure if that information is relevant.

This is good, it means it's starting to remove the tasks with invalid media ids from the queue.

I'll let you know if shows start getting not getting Comskipped.

Craig

Please do.

[Slugger]

A new task client has just been added to the repository. v1350 adds a mapdir equivalent to Groovy scripts. Not as automatic as SJQv3, but still relatively user friendly. Details are in the user guide, but basically you add a new option to sjqagent.properties for each remote task client:

agent.mapdir = C:\\tv\\,\\\\server\\tv\\;C:\\music\\,\\\\server\\music\\

That setting basically maps C:\tv\ > \\server\tv\ and C:\music\ > \\server\music\

Then in your groovy scripts you filter file paths through Tools.mapDir():

Object mf; // This is a MediaFile object, pulled from somewhere earlier in this script
File[] files = Tools.mapDir(MediaFileAPI.GetSegmentFiles(mf))

Now the files[] contains the file paths for all segments of the media file, remapped using the configured agent.mapdir setting from the client's sjqagent.properties file.

You must upgrade your task clients to v1350 (or newer) to use this feature.

Happy scripting!!

[bikesquid]

What worked till a day or so ago isn't working now... specifically the client service is being assigned jobs, starts them and almost immediately they're returned. the gui log for the job says no log data available so I'm at a loss. I don't think anything has changed other than updating to the latest build on both server and client service sides. Any idea how to start addressing this?

Also, a feature request, maybe sjq could check for x number of returned jobs of same task name to the same client before adding more to that client? I've got one client that's working fine, but not getting many jobs as the other client is always idle due to the vast # of returned jobs assigned to it... and therefore available....

[wayner]

This isn't strictly a SJQ question but this problem is causing some of my jobs to fail.

In Groovy how do I strip a question mark out of a string? I am trying to rename some files to an EpisdodeName and the occasional Episode Name has a ? in it which is verboten for a Windows filename.

[Slugger]

Originally Posted by bikesquid

What worked till a day or so ago isn't working now... specifically the client service is being assigned jobs, starts them and almost immediately they're returned. the gui log for the job says no log data available so I'm at a loss. I don't think anything has changed other than updating to the latest build on both server and client service sides. Any idea how to start addressing this?

When tasks fail and there are no logs in the GUI then look at the sjqagent.log file on the client that was assigned the task.

Also, a feature request, maybe sjq could check for x number of returned jobs of same task name to the same client before adding more to that client? I've got one client that's working fine, but not getting many jobs as the other client is always idle due to the vast # of returned jobs assigned to it... and therefore available....

Hmm... this behaviour is usually caused by another problem that should be fixed. If I implemented this it would only mask the underlying problem (whether it be an SJQ bug or a bug with your groovy scripts and/or task setup). In either case, the underlying problem is the real issue that needs to the fixed.

[Slugger]

Originally Posted by wayner

This isn't strictly a SJQ question but this problem is causing some of my jobs to fail.

In Groovy how do I strip a question mark out of a string? I am trying to rename some files to an EpisdodeName and the occasional Episode Name has a ? in it which is verboten for a Windows filename.

// Let's strip out special characters from a string and print the result
def txt = "ABCD?!^^#&@EFGH??IJ@KLMNOPQ?^*^%?R??ST@@#UVWXY!Z%!^&*"
println "Before: ${txt}"
txt = (txt =~ /[?!*^&#$%@]/).replaceAll("")
println "After: ${txt}"

Output:

Before: ABCD?!^^#&@EFGH??IJ@KLMNOPQ?^*^%?R??ST@@#UVWXY!Z%!^&*
After: ABCDEFGHIJKLMNOPQRSTUVWXYZ

[bikesquid]

[quote=Slugger;477219]When tasks fail and there are no logs in the GUI then look at the sjqagent.log file on the client that was assigned the task.

Here's the line from the log and a couple before when it does seem to have worked....:

2011-01-19 17:24:05,106 INFO  [192_168_1_110-23347-900]: Starting process runner for: Task[id=HDCOMSKIP, reqRes=100, maxInst=1, sched=ON, maxTime=86400, maxTimeRatio=1.0, rc=0-1, exe=C:/Program Files/comskip/comskip.exe, exeArgs="\\Master-pc\Recorded TV\$SJQ4_LAST_SEGMENT", test=//Master-pc/c/Program Files (x86)/SageTV/scripts/comskip_test.groovy, testArgs=ts]
2011-01-19 17:24:05,107 INFO  [ProcessRunner]: Converted '"\\Master-pc\Recorded TV\$SJQ4_LAST_SEGMENT"' to '"\\Master-pc\Recorded TV\7Newsat5-3904211-0.ts"'
2011-01-19 17:24:07,040 INFO  [ListenerClient]: Disconnected from 192.168.1.110:23347
2011-01-19 17:24:07,851 INFO  [ListenerClient]: Disconnected from 192.168.1.110:23347
2011-01-19 17:24:07,851 WARN  [ProcessRunner]: Removing task from active list: SJQ4Task-192.168.1.110-23347-900
2011-01-19 17:25:05,094 INFO  [Exe]: Received task 900 of type 'HDCOMSKIP' from 192.168.1.110:23347...
2011-01-19 17:25:05,095 INFO  [192_168_1_110-23347-900]: Starting process runner for: Task[id=HDCOMSKIP, reqRes=100, maxInst=1, sched=ON, maxTime=86400, maxTimeRatio=1.0, rc=0-1, exe=C:/Program Files/comskip/comskip.exe, exeArgs="\\Master-pc\Recorded TV\$SJQ4_LAST_SEGMENT", test=//Master-pc/c/Program Files (x86)/SageTV/scripts/comskip_test.groovy, testArgs=ts]
2011-01-19 17:25:05,095 INFO  [ProcessRunner]: Converted '"\\Master-pc\Recorded TV\$SJQ4_LAST_SEGMENT"' to '"\\Master-pc\Recorded TV\7Newsat5-3904211-0.ts"'
2011-01-19 17:25:06,964 INFO  [ListenerClient]: Disconnected from 192.168.1.110:23347
2011-01-19 17:25:07,785 INFO  [ListenerClient]: Disconnected from 192.168.1.110:23347
2011-01-19 17:25:07,785 WARN  [ProcessRunner]: Removing task from active list: SJQ4Task-192.168.1.110-23347-900
2011-01-19 19:41:20,485 INFO  [Config]: Watching 'c:\Program Files (x86)\SageTV\sjqagent\bin\..\conf'
2011-01-19 19:42:19,126 INFO  [Config]: Watching 'c:\Program Files (x86)\SageTV\sjqagent\bin\..\conf'
2011-01-19 19:42:38,283 INFO  [Config]: Watching 'C:\Program Files (x86)\SageTV\sjqagent\bin\..\conf'
2011-01-19 19:44:44,831 INFO  [Config]: Watching 'c:\Program Files (x86)\SageTV\sjqagent\bin\..\conf'
2011-01-19 19:46:36,153 INFO  [Config]: Watching 'C:\Program Files (x86)\SageTV\sjqagent\bin\..\conf'
2011-01-21 12:40:58,422 INFO  [Config]: Watching 'C:\Program Files (x86)\SageTV\sjqagent\bin\..\conf'
2011-01-24 20:15:06,583 INFO  [Config]: Watching 'C:\Program Files (x86)\SageTV\sjqagent\bin\..\conf'
2011-01-24 20:16:06,355 INFO  [Config]: Watching 'c:\Program Files (x86)\SageTV\sjqagent\bin\..\conf'
2011-01-24 20:16:10,525 WARN  [ProcessRunner]: isActive(SJQ4Task-192.168.1.110-23347-988) = false; []
2011-01-24 20:16:19,728 INFO  [Exe]: Received task 985 of type 'HDCOMSKIP' from 192.168.1.110:23347...
2011-01-24 20:16:19,754 INFO  [192_168_1_110-23347-985]: Starting process runner for: Task[id=HDCOMSKIP, reqRes=100, maxInst=1, sched=ON, maxTime=86400, maxTimeRatio=1.0, rc=0-1, exe=C:/Program Files/comskip/comskip.exe, exeArgs="\\Master-pc\Recorded TV\$SJQ4_LAST_SEGMENT", test=//Master-pc/c/Program Files (x86)/SageTV/scripts/comskip_test.groovy, testArgs=ts]
2011-01-24 20:16:19,759 INFO  [ProcessRunner]: Converted '"\\Master-pc\Recorded TV\$SJQ4_LAST_SEGMENT"' to '"\\Master-pc\Recorded TV\Fringe-Firefly-3943375-0.ts"'
2011-01-24 20:16:19,813 ERROR [192_168_1_110-23347-985]: '\\Master-pc\c\Program Files (x86)\SageTV\scripts\comskip_test.groovy' does not exist or is not readable!  Test considered FAILED!

But that path does work in a browser... as it did before when it also worked with sjq....

[Slugger]

Does it work from a browser running on that machine? Which version of the task client are you using on that machine?

If the Sage plugin version, is Sage running as a privileged user capable of accessing the network? Sage cannot be running as LocalSystem account.

If running the Windows service version, did you change the user the service runs as? By default, it will run as LocalSystem, which cannot access network paths.

If neither of those apply then check the file permissions. For whatever reason, the task client cannot read the file and that's why it's failing.

[bikesquid]

Originally Posted by Slugger

Does it work from a browser running on that machine? Which version of the task client are you using on that machine?

If the Sage plugin version, is Sage running as a privileged user capable of accessing the network? Sage cannot be running as LocalSystem account.

If running the Windows service version, did you change the user the service runs as? By default, it will run as LocalSystem, which cannot access network paths.

If neither of those apply then check the file permissions. For whatever reason, the task client cannot read the file and that's why it's failing.

Was just starting to reply that I'd solved it myself for once!!

Yes it was the local user that was the problem. The issue, I now realize, is that when an update comes, I download the service version and have to uninstall the old version, reinstall the new version and that resets the user account I believe... is there a way to stop the service, copy the new version over the old, restart the service that doesn't involve also resetting up the service user account? It's getting to be a PITA!
Thanks again for the quick response!

[Slugger]

I don't recommend this, but it will work:

When upgrading the Windows service version, simply stop the service, overwrite the current install with the new download and restart the service. You do not need to uninstall the service with the uninstall bat. Doing it this way will:

1) Keep the user for the service in tact
2) Will not update the version info for the task client service properties screens. In other words, the service will report the wrong version number even though it's running the newer code.

Short of writing a complete Windows installer to handle this situation, which I don't plan on doing, this is the only way I can think of saving yourself from having to reset the user. You might be able to write a batch file or something to do this as well and make that a part of your upgrade process.

[wayner]

Originally Posted by Slugger

[code]
// Let's strip out special characters from a string and print the result
def txt = "ABCD?!^^#&@EFGH??IJ@KLMNOPQ?^*^%?R??ST@@#UVWXY!Z%!^&*"
println "Before: ${txt}"
txt = (txt =~ /[?!*^&#$%@]/).replaceAll("")

Thanks, I made a slight alteration as some of the characters above, such as the !, are legal for file names. Apparently the only illegal characters are ?*\/:<>|

Here is my final code:

avifile = (avifile =~ /[?*\/:<>|]/).replaceAll("")