Plugin: SJQv4

[ptzink]

Originally Posted by Slugger

Are you sure it's not actually creating the full backup? The logs simply show that the SageTV server has not fully restarted before the backup task finishes sleeping. After the backup is created, the script restarts SageTV and then sleeps for 5 minutes (if not changed). It has to sleep in order to allow time for the Sage server to restart the SJQ server so that the task can report its status back. If the task doesn't sleep long enough then when it tries to report back to the SJQ server it fails because the SJQ server isn't back online. That's what the log file is showing.

Try increasing the sleep time at the top of the script from 300 seconds to something higher - try 600 or 900 seconds. Be generous, you must ensure that the SJQ server is back online before that sleep time ends otherwise you get the errors you're seeing. Now if the actual copying of the dir failed then the output should be reported back to the server as long as the server is back online to receive the response from the task client.

Well, here is what is happening: After I queue the task, the sagetv service stops as expected. Moments later, I watch the backup folder and sub directories get created. Since it copies files/folders in alphabetical order, it is pretty easy to see that it gets "hung up" in the STVs folder. It chugs along nicely until it gets to that folder and then just stops half way through the directory. After the backup hangs up, the sagetv service never restarts and I have to start it manually (after waiting 10 minutes or so). When I get back into the SJQ UI, the job is marked as failed.
As far as I can tell, the copy seems to stop in the STVs\Pheonix\Fanart folder. Not sure why. If there is a different log I can provide to help let me know.

[Slugger]

Originally Posted by ptzink

Well, here is what is happening: After I queue the task, the sagetv service stops as expected. Moments later, I watch the backup folder and sub directories get created. Since it copies files/folders in alphabetical order, it is pretty easy to see that it gets "hung up" in the STVs folder. It chugs along nicely until it gets to that folder and then just stops half way through the directory. After the backup hangs up, the sagetv service never restarts and I have to start it manually (after waiting 10 minutes or so). When I get back into the SJQ UI, the job is marked as failed.
As far as I can tell, the copy seems to stop in the STVs\Pheonix\Fanart folder. Not sure why. If there is a different log I can provide to help let me know.

I'd have to whip up a debug version that dumps the output to a text file or something. I'll see what I can do, or if you're able, mod the script to dump the IO exception to a text file when it does the dir copy - that's likely what's failing.

[ptzink]

Originally Posted by Slugger

I'd have to whip up a debug version that dumps the output to a text file or something. I'll see what I can do, or if you're able, mod the script to dump the IO exception to a text file when it does the dir copy - that's likely what's failing.

My scripting chops are pretty (read: very) limited. Not asking you to do it for me necessarily, just wanting a nudge in the right direction.

[coppit]

Originally Posted by Slugger

The Process API in Java does not provide access to the process PID meaning all I can do is call Process.destroy().

What about using a little helper script that can do the killing for you? Here's one in powershell:

param(
  [Parameter(Mandatory=$true)]
  [int] $timeout,
  [string] $program,
  [string] $arguments = ""
)

if ( $arguments -eq "" ) {
  $p = Start-Process $program -passthru
} else {
  $p = Start-Process $program $arguments -passthru
}

if ( ! $p.WaitForExit($timeout * 1000) ) {
  echo "Killing $program due to timeout of $timeout seconds"
  $p.kill()
}

You can invoke it as:

powershell runwithlimit.ps1 3600 notepad

This would kill notepad after 1 hour. Note the above requires PowerShell 2.

I saw a straight-up .bat version of this on the web. I can work on it for you if you want to go down this path.

[Slugger]

That's OS specific code that won't run on Linux, which is also a supported platform for SJQ. Forking off the code into OS specific execution paths is not something I will do.

But... you could call an exe/script that launches your mediashrink via that powershell script, which would then time itself out and clean itself up.

[coppit]

Originally Posted by Slugger

you could call an exe/script that launches your mediashrink via that powershell script, which would then time itself out and clean itself up.

I can get it to run in a command shell after setting "Set-ExecutionPolicy RemoteSigned". PowerShell is pretty fussy about escaping special characters and such. Here's the full command line:

C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe "& C:\Program` Files` `(x86`)\SageTV\SageTV\MediaShrink\timer.ps1" 10800 'C:\Program Files (x86)\SageTV\SageTV\MediaShrink\MediaShrink.bat' d:\SageTV\video_to_convert.mpeg

So next I tried it in SJQ4. I set powershell as the Executable and the rest as the Executable Args, with "$SJQ4_PATH\$SJQ4_LAST_SEGMENT" as the file at the end. Unfortunately powershell refuses to run the script due to the script execution prevention stuff. I'm running the Sage service as LocalSystem, and I don't know how to get past this. I tried running powershell as Administrator and setting the execution policy but it didn't work.

I'll probably next look at a plain .bat way to do this, or maybe try signing the script.

By the way, the previous version of the script doesn't kill child processes. You have to use taskkill as per my first post. Here's the full script:

# Run powershell then type "Set-ExecutionPolicy RemoteSigned" to allow this
# script to be executed, as well as any other local unsigned script. Or sign
# it yourself.

param(
  [Parameter(Mandatory=$true)]
  [int] $timeout,
  [string] $program,
  [string] $arguments = ""
)

if ( $arguments -eq "" ) {
  $p = Start-Process $program -passthru
} else {
  $p = Start-Process $program $arguments -passthru
}

if ( ! $p.WaitForExit($timeout * 1000) ) {
  echo "Killing $program due to timeout of $timeout seconds"
  taskkill /pid $p.Id /t /f
}

[Slugger]

Originally Posted by coppit

I'm running the Sage service as LocalSystem, and I don't know how to get past this. I tried running powershell as Administrator and setting the execution policy but it didn't work.

Run the Sage service as Administrator?

By the way, the previous version of the script doesn't kill child processes. You have to use taskkill as per my first post. Here's the full script:

And this is probably why Java doesn't kill child processes when it kills the parent process.

[ptzink]

Originally Posted by Slugger

I'd have to whip up a debug version that dumps the output to a text file or something. I'll see what I can do, or if you're able, mod the script to dump the IO exception to a text file when it does the dir copy - that's likely what's failing.

Just a follow up. I found out the file that the backup job was hanging on in the Fanart directory. Once I renamed the files in that directory (to 01.jpg, 02.jpg, etc.), the backup ran fine. This was strange to me, since the file names weren't exotic to start with (they were called something like "9876543-01.jpg"). Anyway, I guess whatever works.
Next on the list, figure out how to power on the extender any time the Sage server starts/restarts.

[routerunner]

Originally Posted by ptzink

Next on the list, figure out how to power on the extender any time the Sage server starts/restarts.

I might help you on that... there is a library plugin "Power On/Off" that is normally downloaded when using "Automatic Power Off" that implements what you want and works with all extenders including the MVP. The library is very simple to use and here a simple code snippet in Java:

Power power = new Power();
power.off(IP address);
power.on(IP address);

You could implement your own plugin that listens to SageTV server events to wake up the extender you want.

Let me know if you need more details.

[wayner]

Originally Posted by ptzink

Next on the list, figure out how to power on the extender any time the Sage server starts/restarts.

One of the plugins has an option to do this - I believe that it is Fonceur's SageTCPServer. I actually had to disable this functionality as I have about 7 extenders and Sage can't handle that many extenders as it runs out of JVM space.

[wayner]

Originally Posted by routerunner

Ipower.off(IP address);
power.on(IP address);

Why do you use IP address rather than MACID or a friendly name like "Kitchen Extender"? Don't you run the risk of your IP names switching if you don't use static IPs for your extenders? I use static IPs for my PCs but I never bothered with my Sage Extenders.

[routerunner]

Originally Posted by wayner

Why do you use IP address rather than MACID or a friendly name like "Kitchen Extender"? Don't you run the risk of your IP names switching if you don't use static IPs for your extenders? I use static IPs for my PCs but I never bothered with my Sage Extenders.

Simply because the IP address/hostname is what you normally use to open a telnet session to a client...

Personally I always use static IP's or if the router allows a MAC->IP correlation via pre-assigned DHCP.

[wayner]

According to the Java docs Runtime.totalMemory() Returns the total amount of memory availalble in the Java virtual machine. Will this show me the amount of memory available and help me to diagnose when I am running low and may run into trouble?

If so then how do I call this from SageGroovy? Do I have to import a certain library? Is this included with the Java or do I have to download it from somewhere?

When I try the command "println (Runtime.totalMemory())" I get the error:

groovy.lang.MissingMethodException: No signature of method: static java.lang.Runtime.totalMemory() is applicable for argument types: () values: []
Possible solutions: totalMemory(), maxMemory(), freeMemory()

[Slugger]

Originally Posted by wayner

According to the Java docs Runtime.totalMemory() Returns the total amount of memory availalble in the Java virtual machine. Will this show me the amount of memory available and help me to diagnose when I am running low and may run into trouble?

If so then how do I call this from SageGroovy? Do I have to import a certain library? Is this included with the Java or do I have to download it from somewhere?

When I try the command "println (Runtime.totalMemory())" I get the error:

groovy.lang.MissingMethodException: No signature of method: static java.lang.Runtime.totalMemory() is applicable for argument types: () values: []
Possible solutions: totalMemory(), maxMemory(), freeMemory()

It's an instance method, not a static method. Therefore, you'd have to call it like this:

Runtime.getRuntime().totalMemory()

[wayner]

OK - I assume that when I run this from the SageGroovy environment I am not getting the values for the Sage JVM but for a separate JVM for SageGroovy, is that correct?

If I call this from within SJQ that is initiated by Sage am I getting the memory values for the Sage JVM? Or is SJQ running in a different JVM instance?

Can I use this method to write a Groovy script to monitor my free JVM memory and have it throw a System Message when I start to get low?

One other question - can I use the Java runtime Garbage Collector to clean up in Sage? Or is this dangerous or will this not make a difference if Sage cleans up the garbage on its own?

[Slugger]

Originally Posted by wayner

OK - I assume that when I run this from the SageGroovy environment I am not getting the values for the Sage JVM but for a separate JVM for SageGroovy, is that correct?

Correct.

If I call this from within SJQ that is initiated by Sage am I getting the memory values for the Sage JVM? Or is SJQ running in a different JVM instance?

All SJQ Groovy scripts run in a separate JVM. Calling the mem functions from Groovy within SJQ won't provide details about the Sage JVM.

Can I use this method to write a Groovy script to monitor my free JVM memory and have it throw a System Message when I start to get low?

If you added a groovy script to the web server and called it periodically then you could get the memstats from the Sage JVM since groovy scripts in the web server run in the Sage JVM.

But there is a plugin that monitors this stuff and generates system messages if the free mem gets too low. ("Heap monitor" or something like that is the name of it, I think.)

One other question - can I use the Java runtime Garbage Collector to clean up in Sage? Or is this dangerous or will this not make a difference if Sage cleans up the garbage on its own?

It's not recommended that you call it yourself. The JVM will garbage collect when it needs to. Calling it yourself isn't dangerous, but it can affect system performance. It can cause Sage to completely pause while GC is running. Call it with caution if you decide to call it at all.

[tmiranda]

Originally Posted by Slugger

But there is a plugin that monitors this stuff and generates system messages if the free mem gets too low. ("Heap monitor" or something like that is the name of it, I think.)

Java Heap Monitor is correct. It put's a nice little bar graph in the header and lets you set trigger levels for messages.

[Skirge01]

Slugger, I'm still running SJQv3, since it has met my--admittedly limited--needs up until now. I'm still toying with the idea of using XBMC as my front-end and was looking at bradvido's script for syncing things up. Since he isn't developing it any further, something else came to mind.

Would it be possible to use SJQv4 to save my recordings in a more XBMC-friendly naming convention? For example, XBMC would prefer a structure like this:

TV Shows
     |----TV Show 1
     |       |___ TV Show 1 Media files
     |
     |----TV Show 2
     |       |___ TV Show 2 Media files
     |
      ----TV Show 3
             |___ TV Show 3 Media files

The filenames are preferred like this:

foo.s01e01.* 
    foo.s01.e01.* 
    foo.s01_e01.* 
    foo_[s01]_[e01]_* 
    foo.1x01.* 
    foo.101.*

Movies are preferred to be in a separate folder, such as "Movies", using a format of:

foo (year).*

Not sure if something like this might assist you with your SageSync project and I'm certainly not looking for you to write this for me, but I would like to get your input on whether it's even possible. If it sounds feasible, I'll start digging and come back here when I get hung up. One thing that immediately came mind is that this would probably require having Sage "forget" the recording and then add it back as an imported video.

Thanks for any insight.

[Slugger]

Originally Posted by Skirge01

Slugger, I'm still running SJQv3, since it has met my--admittedly limited--needs up until now. I'm still toying with the idea of using XBMC as my front-end and was looking at bradvido's script for syncing things up. Since he isn't developing it any further, something else came to mind.

Would it be possible to use SJQv4 to save my recordings in a more XBMC-friendly naming convention? For example, XBMC would prefer a structure like this:

TV Shows
     |----TV Show 1
     |       |___ TV Show 1 Media files
     |
     |----TV Show 2
     |       |___ TV Show 2 Media files
     |
      ----TV Show 3
             |___ TV Show 3 Media files

The filenames are preferred like this:

foo.s01e01.* 
    foo.s01.e01.* 
    foo.s01_e01.* 
    foo_[s01]_[e01]_* 
    foo.1x01.* 
    foo.101.*

Movies are preferred to be in a separate folder, such as "Movies", using a format of:

foo (year).*

Not sure if something like this might assist you with your SageSync project and I'm certainly not looking for you to write this for me, but I would like to get your input on whether it's even possible. If it sounds feasible, I'll start digging and come back here when I get hung up. One thing that immediately came mind is that this would probably require having Sage "forget" the recording and then add it back as an imported video.

Thanks for any insight.

The SageSync plugin basically does this, but it presents a bunch of playlist files instead of trying to rename your current recording collection. If you actually want to go move/rename your recording files then grab wayner's attention - he's written a lot of Groovy scripts for SJQv4 to do this exact thing.

In summary, can SJQv4 do this kind of thing? Absolutely. But it will require you to write some decent amount of scripts to do it - or ask wayner nicely for his solution.

[coppit]

Following up on my question about enforcing a timeout for tasks...

Below is a script I've been using successfully for a couple weeks now to constrain how long mediaShrink can run. (It's actually HandBrake that gets stuck.)

I've hard-coded the command in the script, but it wouldn't be hard for one to take the arguments to the script as the command to run. Note that this script is a little complicated because I wanted the feature to stream standard output and standard error as the script runs. In a SJQ context that doesn't matter a lot, but I was trying to make the script generally useful.

My SJQ task has:

Executable = C:\Windows\System32\cscript.exe
Executable args = "C:\path\to\script\below.vbs" "$SJQ4_PATH\$SJQ4_LAST_SEGMENT"
Maximum Time = 3 hours

' In seconds
timeout = 1 * 60 * 60
checkInterval = 60

command = """C:\Program Files (x86)\SageTV\SageTV\MediaShrink\MediaShrink.bat"" " & WScript.Arguments.Item(0)

Set fso = CreateObject("Scripting.FileSystemObject")
tempOutFilePath = fso.GetSpecialFolder(2) & "\" & fso.GetTempName
tempErrFilePath = fso.GetSpecialFolder(2) & "\" & fso.GetTempName

Set shell = CreateObject("WScript.Shell")

redirectedCommand = command & " >" & tempOutFilePath & " 2>" & tempErrFilePath
Set oExec = shell.Exec(redirectedCommand)

WScript.StdOut.Writeline "Process has PID " & oExec.ProcessID

' Sleep a little to give the process a time to create the output files
WScript.Sleep 100

' For checking the file size
Set tempOutSizeFile = fso.GetFile(tempOutFilePath)
Set tempErrSizeFile = fso.GetFile(tempErrFilePath)
' For reading the file
Set tempOutReadFile = fso.OpenTextfile(tempOutFilePath)
Set tempErrReadFile = fso.OpenTextfile(tempErrFilePath)

oldTempOutFileSize = 0
oldTempErrFileSize = 0
startTime = Timer
timeoutExpired = False

While oExec.Status = 0 and timeoutExpired = False
  if Timer - startTime > timeout then
'    WScript.StdOut.Writeline "Timeout of " & timeout & " exceeded. Killing process and all its children"

    shell.Run "taskkill /t /f /pid " & oExec.ProcessID, 0, True

    timeoutExpired = True
  end if

  if (tempOutSizeFile.Size > oldTempOutFileSize) then
'    WScript.StdOut.Writeline "Passing through " & (tempOutSizeFile.Size - oldTempOutFileSize) & " more bytes of STDOUT"
    Wscript.StdOut.Write(tempOutReadFile.Read(tempOutSizeFile.Size - oldTempOutFileSize))
    oldTempOutFileSize = tempOutSizeFile.Size
  end if

  if (tempErrSizeFile.Size > oldTempErrFileSize) then
'    WScript.StdOut.Writeline "Passing through " & (tempErrSizeFile.Size - oldTempErrFileSize) & " more bytes of STDERR"
    Wscript.StdErr.Write(tempErrReadFile.Read(tempErrSizeFile.Size - oldTempErrFileSize))
    oldTempErrFileSize = tempErrSizeFile.Size
  end if

'  WScript.StdOut.Writeline "Waiting " & checkInterval & " more seconds. " & (timeout - (Timer - startTime)) & _
'    " seconds remain."

  WScript.Sleep checkInterval * 1000
Wend

tempOutReadFile.Close
fso.DeleteFile tempOutFilePath

tempErrReadFile.Close
fso.DeleteFile tempErrFilePath

if timeoutExpired = True then
  WScript.Quit 1
else
  WScript.Quit oExec.ExitCode
end if