Plugin: SJQv4

[Slugger]

So the error is this:

Invalid args for FileUtils.moveFile() of type (String, File); possible solutions moveFile(File, File)

So that error is telling you that the call to FileUtils.moveFile() requires two args, each of type File, but you passed it a String and a File. So to fix the error, make sure arg 1 is of type file:

FileUtils.moveFile(OldFileNameTS, new File(parentDir, targetFileNameTS))
FileUtils.moveFile(OldFileNameVPrj, new File(parentDir, targetFileNameVPrj))

becomes...

FileUtils.moveFile(new File(OldFileNameTS), new File(parentDir, targetFileNameTS))
FileUtils.moveFile(new File(OldFileNameVPrj), new File(parentDir, targetFileNameVPrj))

Here's a groovy script that will rename all matching files and artifacts.

NOTE: I didn't actually let it rename any files, I only ran it in test mode.

NOTE: I didn't test the multi-segment rename

NOTE: I really didn't do much testing, but it should be a solid starting point

import org.apache.commons.io.FilenameUtils
import org.apache.commons.io.FileUtils

private class Settings {
    static public final boolean TEST_MODE = true
}

def mf = MediaFileAPI.GetMediaFileForID(SJQ4_METADATA["SJQ4_ID"].toInteger())
if(mf == null) {
    println "Invalid media file id! [${SJQ4_METADATA['SJQ4_ID']}]"
    return 1
}

def title = ShowAPI.GetShowTitle(mf)
def sNum = ShowAPI.GetShowSeasonNumber(mf)
def eNum = ShowAPI.GetShowEpisodeNumber(mf)
def seNum = sNum > 0 && eNum > 0 ? String.format("S%02dE%02d", sNum, eNum) : null
def subtitle = ShowAPI.GetShowEpisode(mf)
def origAirDate = ShowAPI.GetOriginalAiringDate(mf) > 0 ? new Date(ShowAPI.GetOriginalAiringDate(mf)).format("yyyy-MM-dd") : null
def newPrefix = "${title} - "
if(seNum != null)
    newPrefix += seNum
else if(subtitle != "")
    newPrefix += subtitle
else if(origAirDate != null)
    newPrefix += origAirDate
else
    newPrefix += "UNKNOWN"

def numSegments = MediaFileAPI.GetNumberOfSegments(mf)
if(numSegments == 1) {
    def prefix = FilenameUtils.getBaseName(MediaFileAPI.GetFileForSegment(mf, 0).getAbsolutePath())
    println "Renaming files that look like '${prefix}.*' to '${newPrefix}.*'..."
    renameMatches(MediaFileAPI.GetParentDirectory(mf), prefix, null, newPrefix)
} else if(numSegments > 1) {
    for(def i = 0; i < numSegments; ++i) {
        def prefix = FilenameUtils.getBaseName(MediaFileAPI.GetFileForSegment(mf, i).getAbsolutePath())
        def segPrefix = "$newPrefix-$i"
        println "Renaming files that look like '${prefix}.*' to '${segPrefix}.*'..."
        renameMatches(MediaFileAPI.GetParentDirecotry(mf), prefix, null, segPrefix)
    }
} else {
   println "No file segments for given media file!"
   return 1
}
return 0

// Pass null for newDir to keep renamed file in same directory
def renameMatches(def oldDir, def prefix, def newDir, def newPrefix) {
    if(newDir == null)
        newDir = oldDir
    Utility.DirectoryListing(oldDir).each {
        if(FilenameUtils.wildcardMatchOnSystem(it.getName(), "${prefix}.*")) {
            def newName = (newPrefix + it.getName().substring(it.getName().indexOf('.')) =~ /[\/\\:*?<>]/).replaceAll("")
            if(!Settings.TEST_MODE) {
                FileUtils.moveFile(it, new File(newDir, newName))
            } else {
                println "Would rename '$it' to '$newName' if test mode were disabled!"
            }
        }
    }
}

Output of my test run:

Renaming files that look like 'TheColbertReport-6197132-0.*' to 'The Colbert Report - S07E43.*'...
Would rename 'D:\tv\TheColbertReport-6197132-0.edl' to 'The Colbert Report - S07E43.edl' if test mode were disabled!
Would rename 'D:\tv\TheColbertReport-6197132-0.log' to 'The Colbert Report - S07E43.log' if test mode were disabled!
Would rename 'D:\tv\TheColbertReport-6197132-0.ts' to 'The Colbert Report - S07E43.ts' if test mode were disabled!
Would rename 'D:\tv\TheColbertReport-6197132-0.ts.properties' to 'The Colbert Report - S07E43.ts.properties' if test mode were disabled!

[bikesquid]

wow!

I 'think' that if the error had read that way I'd have had some clue what to do, but the whole 'missing method exception' thing, to this mortal, is unclear at best.... regardless, never would have thought to put 'new File' in front of the old file... I can't seem to find any code reference material that provides examples of syntax AND exactly what each item means/does so this is hugely helpful to understanding the whole groovy thing. I assumed 'new file' was referring to the string for the new file...

Since you already wrote most of it in trying to help me along piece by piece I understand what's happening in the code for the most part. I seem to be struggling with the whole arg vs string vs value etc. I keep assuming as long as what would print println can be substituted anywhere for anything. I'm clearly mistaken.

I'm guessing that the .each creates the 'loop for each result of the directory list' and the 'it' refers to the current item from that list? What's the purpose of the $ which seems necessary at sometimes but not others?

And HUGE thank you!

[Slugger]

Originally Posted by bikesquid

wow!

I 'think' that if the error had read that way I'd have had some clue what to do, but the whole 'missing method exception' thing, to this mortal, is unclear at best.... regardless, never would have thought to put 'new File' in front of the old file... I can't seem to find any code reference material that provides examples of syntax AND exactly what each item means/does so this is hugely helpful to understanding the whole groovy thing. I assumed 'new file' was referring to the string for the new file...

You'll want to create some bookmarks. Anything referencing the Sage API is available here:

http://download.sage.tv/api/

So in a Groovy script, if you see a call to MediaFileAPI.GetMediaTitle() then go to the above site and lookup the GetMediaTitle method in the MediaFileAPI. You'll see it takes one argument, the media file whose title you want to grab. All the SageAPI docs are there for all the Sage API calls you'll see in groovy scripts.

You'll also notice in the script I wrote some imports at the top:

import org.apache.commons.io.FileUtils

Usually, if you google that package name, it'll popup the javadocs for that package. So any references to FileUtils.* or FilenameUtils.* in the Groovy script comes from there. Load up those javadocs and search for the methods I'm using to see how they're used. You'll also find a lot of other useful filesystem functions in that library.

Everything else is Java/Groovy. Groovy is just Java, with "enhancements" (others might argue Groovy is Java with pitfalls added, but I digress). So the best place to learn all things Groovy is the Groovy website. I'll admit the docs aren't organized all that well, but all the info you need is there. The nice thing is, you don't have to do things the Groovy way. You can just write pure Java and it'll work.

String foo = "foo";
String bar = "bar";

if(!foo.equals(bar))
   System.out.println("The strings are not equal!");
else
   System.out.println("The strings are equal!");

Or the Groovy way...

def foo = "foo"
def bar = "bar"

if(foo != bar)
   println "The strings are not equal!"
else
   println "The strings are equal!"

There's a lot of subtle things going on between the two. For example, in Java you cannot compare strings using the == operator, they'll never be equal (* unless you make both references point to the same object), but in Groovy you can (one of the convenience factors added by Groovy).

Groovy also lets you interpolate variables inside strings (i.e. where the $ comes from):

def foo = "A string."
println foo // This is ok and you can't use a $ because it's not in a ""
println "The value is: $foo" // Here you use the $ because you're inside a ""
println "The value is: " + foo // Same output as line above

Again, this is just more convenience kind of things... anyway works.

I never knew anything about Groovy until I discovered it while developing SJQv4 so you'll notice a lot of my early Groovy script examples look a lot like Java code and don't use any of the Groovy shortcuts, then as I started to learn Groovy you'll see some of my later examples start to do things, as they say, the "Groovy" way. With that said, I'm hardly a Groovy expert, but am learning on the fly as needed.

Since you already wrote most of it in trying to help me along piece by piece I understand what's happening in the code for the most part. I seem to be struggling with the whole arg vs string vs value etc. I keep assuming as long as what would print println can be substituted anywhere for anything. I'm clearly mistaken.

Since all of the methods you call are Java, they expect and require specific data types. So even though Groovy doesn't care what type a variable is, Java does so when you call Java code from Groovy it has to care and if they don't match, you get the errors you're seeing. Bookmark the javadocs I've referenced, they'll tell you what types are needed where.

I'm guessing that the .each creates the 'loop for each result of the directory list' and the 'it' refers to the current item from that list? What's the purpose of the $ which seems necessary at sometimes but not others?

And HUGE thank you!

Yes, .each is a "Groovy" thing. Read up on "closures" in the Groovy docs. I could have done it in a plain old Java for loop as well and achieved the same results. The 'it' var is the default name inside a closure, you could also rename it for clarity - that's discussed in the Groovy docs.

See above for the $ operator. It's used to interpolate variables inside String constants. It, too, is also a "groovy" thing.

[rsagetv99]

Originally Posted by Slugger

Ok, switch testMode to true and then run it again. It appears one of your media files is returning null for a value that is unexpected, which is causing the attempt to set null in the database. Run it in test mode, look at the output in the UI and report back the list of media files and ids it's attempting to queue up.

When I switch to Test mode it works fine, and I can see a bunch of shows that would have been processed. When I take it out of Test mode I get the following error for the media_scan script:

javax.scipt.ScriptException:java.io.EOFException

It seems to bomb after queuing up one show, which is a .AVI file and is therefore skipped by both my COMSKIP and COMSKIPSD scripts.

Any ideas?

[Slugger]

Without the actual data to run against, I can only guess (or start slapping println statements all over the place).

My guess is that .avi file was imported into Sage as a tv recording and does not have any (or all) regular metadata for a tv recording. So when the file is passed to the metadata factory for registration in SJQ, it's getting an unexpected null somewhere and failing.

Add a condition to the needsProcessing() function that excludes .avi files right away instead of trying to register them. If that fixes the problem then we at least know the cause - fixing it is a whole other problem.

[SteveW]

Forgive me, but I can't find anything in your tutorial or manual for sjqv4 when I search for regex...

In version 3, with your help I was able to set up a simple statement to define what channels I didn't want to comskip on...

&& ($.GetAiringChannelNumber{} !% "30[0-7]" && $.GetAiringChannelNumber{} !% "839" && $.GetAiringChannelNumber{} !% "840" && $.GetAiringChannelNumber{} !% "125[1|3-4|6|8-9]")]

In your groovy script for media file scanner in v4, you have the following:

if(AiringAPI.GetAiringChannelName(mediaFile) =~ /HBO.*|M(?:HD){0,1}|WPBS|.*PPV.*/)

I'm a little lost on how the substitutions work. The Wiki discussion on regex gets into all sorts of different forms of regex (posix basic, posix extended, perl derivative, etc, etc). I'm not clear on how your line would expand... For the TMN channels... M(?:HD) mean your're only looking for the HD versions of the TMN channels? Where does MPIX fit in there if standard def? I like the idea of putting in names so I don't have to select out all the numbers but am having a hard time understanding the channel name expansions... Does =~ mean "not eq to" ??

Tnx

-S

[Slugger]

Originally Posted by SteveW

Forgive me, but I can't find anything in your tutorial or manual for sjqv4 when I search for regex...

In version 3, with your help I was able to set up a simple statement to define what channels I didn't want to comskip on...

&& ($.GetAiringChannelNumber{} !% "30[0-7]" && $.GetAiringChannelNumber{} !% "839" && $.GetAiringChannelNumber{} !% "840" && $.GetAiringChannelNumber{} !% "125[1|3-4|6|8-9]")]

In your groovy script for media file scanner in v4, you have the following:

if(AiringAPI.GetAiringChannelName(mediaFile) =~ /HBO.*|M(?:HD){0,1}|WPBS|.*PPV.*/)

I'm a little lost on how the substitutions work. The Wiki discussion on regex gets into all sorts of different forms of regex (posix basic, posix extended, perl derivative, etc, etc). I'm not clear on how your line would expand... For the TMN channels... M(?:HD) mean your're only looking for the HD versions of the TMN channels? Where does MPIX fit in there if standard def? I like the idea of putting in names so I don't have to select out all the numbers but am having a hard time understanding the channel name expansions... Does =~ mean "not eq to" ??

Tnx

-S

http://download.oracle.com/javase/1....x/Pattern.html
http://groovy.codehaus.org/Regular+Expressions

M(?:HD){0,1} means "M followed by HD zero or one time" In other words, that matches M and MHD. The example from the script is incomplete (i.e. you'd have to add to it to catch MPIX, etc. You might be able to do it real fancy with some thought, but I'd keep it simple. And looking at that, it should be written as M(?:HD)? (same thing, just a little more condensed).

To add MPIX, I'd go with

if(AiringAPI.GetAiringChannelNumber(mediaFile) =~ /M(?:HD)?|MPIX(?:HD)?|.*PPV.*|HBO.*|HDNET.*/)

The =~ in Groovy means the same as it does in Perl (if that helps). It is the "string contains regex" operator. It returns true if the argument contains the regex somewhere within it.

"MHD" =~ /M(?:HD)?/ // true
"FOOMHDjfsdlsd" =~ /M(?:HD)?/ // true

But be aware that in Groovy/Java:

"MHDHD" ==~ /M(?:HD)?/ // FALSE!!

The ==~ is the "string exactly matches operator", which acts like the String.matches() method in Java.

I won't doc much about the details of the scripting language other than how to use it within the SJQv4 environment. So details like regex usage, etc. should be referenced from the Groovy and/or Java docs and Google is your friend on that kind of stuff. With that said, I don't mind doing quick replies like this, but usually I'm just going to point you to something in my bookmarks (as above).

[SteveW]

http://download.oracle.com/javase/1....x/Pattern.html
http://groovy.codehaus.org/Regular+Expressions


Sigh... Sorry Slugger, I don't find either of those links particularly helpful... You're talking to a guy who despised programming when doing his computing science degree at university... That's why I deal with networking, servers, and storage arrays now... Needless to say I thought "oh lord, no..." when I heard v4 was going to be a lot more scripting based...

Maybe there's a "Regex for Dummies" website...

[Slugger]

I mean you don't have to use regex to do any of this stuff. The code gets a lot bigger to do the same thing:

With regex:

if(AiringAPI.GetAiringChannelName(mf) =~ /M(?:HD)?|HBO.*|.*PPV.*/) {

}

Without regex:

def chan = AiringAPI.GetAiringChannelName(mf)
if(chan == "M" || chan == "MHD" || chan.startsWith("HBO") || chan.contains("PPV")) // And so on... {

}

You can google for other regex docs, some might be more clear than others. The ones I linked to are the ones I use for Java and Groovy. The perl regex docs might be clearer, but they don't perfectly align with how Java regex works so be careful.

Regex is one of those powerful tools that once you get it, it's a tool you'll use forever. But, as I say, you don't really need it for this stuff. Most anything you'd use regex for in SJQv4 can be done the long way as I've shown above. Really, it's what works for you to get the results you're looking for.

[bikesquid]

related to the rename script - I'm working to add the comskip step as a .execute call from the script using:

// comskip attempt

def command = ("c:\\program files\\comskip\\comskip.exe " + newPrefix +".ts")
println "full command will be '$command'"
def proc = command.execute()
if(!Settings.TEST_MODE) {
              proc.waitFor()
            } else {
                println "Would run '$command' if test mode were disabled!"
            }
}

the comskip exec 'seems' to be running when looking in task manager, but there's no output.... and the task isn't showing any cpu usage so wonder if I've got something buggered up again with syntax or variables/expressions??

[gplasky]

You might need to add the switch to comskip that gives it the path to the comskip.ini file it is suoppose to use. It can be an issue if it can't find the .ini file. Since it's producing no output that would be my guess. I think the switch is like --ini=C:\Comskip\comskip.ini

Gerry

[bikesquid]

Originally Posted by gplasky

You might need to add the switch to comskip that gives it the path to the comskip.ini file it is suoppose to use. It can be an issue if it can't find the .ini file. Since it's producing no output that would be my guess. I think the switch is like --ini=C:\Comskip\comskip.ini

Gerry

Good suggestion, and I'd tried that with this variant:

def command = ("c:\\program files\\comskip\\comskip.exe --ini=C:\\program files\\comskip\\comskip.ini" + newPrefix +".ts")

It still just sits there like a lump.... figuratively speaking of coarse...

[Slugger]

Originally Posted by bikesquid

related to the rename script - I'm working to add the comskip step as a .execute call from the script using:

// comskip attempt

def command = ("c:\\program files\\comskip\\comskip.exe " + newPrefix +".ts")
println "full command will be '$command'"
def proc = command.execute()
if(!Settings.TEST_MODE) {
              proc.waitFor()
            } else {
                println "Would run '$command' if test mode were disabled!"
            }
}

the comskip exec 'seems' to be running when looking in task manager, but there's no output.... and the task isn't showing any cpu usage so wonder if I've got something buggered up again with syntax or variables/expressions??

Originally Posted by gplasky

You might need to add the switch to comskip that gives it the path to the comskip.ini file it is suoppose to use. It can be an issue if it can't find the .ini file. Since it's producing no output that would be my guess. I think the switch is like --ini=C:\Comskip\comskip.ini

Gerry

The problem is that the OS buffers are filling with output and not being consumed, causing the process to deadlock. This is discussed here and here.

If you don't care about the output, simply call proc.consumeOutput() just before your proc.waitFor() call. If you want the output captured, use proc.comsumeOutput(System.out, System.err) (or pass ByteArrayOutputStream instances instead if you want to capture the output for processing within the script).

Also, I recommend changing your command variable to an array so your command line isn't ambiguous (to the Java command interpreter).

def command = ["C:/Program Files/comskip/comskip.exe", newPrefix + ".ts"]

If you keep it as a string then you need to double quote the exe name and the file name. Trust me, just use an array. You can call .execute() on an array just like you do the string.

[bikesquid]

I had the command variable set up exactly like that at first (using the example on the groovy site) and couldn't get it working... just skipped the executing. put it back that way and yup, it just doesn't do it, but shows complete rather than failed.
Used

def command = ["c:/program files/comskip/comskip.exe", newPrefix +".ts"]

println "full command will be '$command'"
def proc = command.execute()
if(!Settings.TEST_MODE) {
              proc.consumeProcessOutput()
              proc.waitFor()

[Slugger]

Capture the output from comskip to see what it has to say. Really, you should always capture output of external commands and log it somewhere, especially when you're not getting the expected results from that command.

[rsagetv99]

I tried adding an if statement to exclude non-mpg files, but I could get it to work. Any idea what is wrong? I added the parts in red.

// Returns true if any segment of the given media file has at least one artifact of any of the given artifact extensions; false otherwise
def hasArtifacts(Object mf, List exts) {
for(def it : MediaFileAPI.GetSegmentFiles(mf)) {
def absPath = it.getAbsolutePath()
def dir = FilenameUtils.getFullPath(absPath)
def base = FilenameUtils.getBaseName(absPath)
def extension = FilenameUtils.getExtension(absPath)
for(def ext : exts) {
def artifact = "${dir}${base}.$ext"
if(Utility.IsFilePath(artifact))
if(${extension} = "mpg")
return true
}
}
return false
}

[Slugger]

Originally Posted by rsagetv99

if(${extension} = "mpg")

For future reference, when saying "my script isn't working, here are my changes", please include the error message(s) you're getting and as much detail as possible of what's "not working". Is it an error with the script? If so, include the error messages from the Groovy interpreter. The script is running, but it's not doing what you expect it to? If so, what is it doing and what do you expect it to do?

This looks like a compile error with the script. The line above is not valid.

Try:

if(extension == "mpg")
   return true

${var} is only valid inside a string constant, typically a println:

println "Some text with a variable ${var}"

That statement prints out the text plus interpolates the ${var} into the actual value of the variable "var".

Also, to check equality you use '==' not '='. Try those fixes and see what happens.

[rsagetv99]

Originally Posted by Slugger

For future reference, when saying "my script isn't working, here are my changes", please include the error message(s) you're getting and as much detail as possible of what's "not working". Is it an error with the script? If so, include the error messages from the Groovy interpreter. The script is running, but it's not doing what you expect it to? If so, what is it doing and what do you expect it to do?

Thanks that worked. Normally I would include all of that, but I figured it was a stupid syntax error that you could figure out just by looking at my code (which it seems like it was). Do the same OR operators work in that portion of the script? E.G. can I do the following to exclude everything but mpg or ts files?

if(extension == "mpg" || extension == "ts")
return true

[Slugger]

That's valid.

[bikesquid]

Originally Posted by Slugger

Capture the output from comskip to see what it has to say. Really, you should always capture output of external commands and log it somewhere, especially when you're not getting the expected results from that command.

Good suggestion. and obvious if stop to consider it....

figured out that error was an incomplete path for the target file, so now I'm trying to figure out the syntax for the newDir\newName.ts combo and failing badly Have tried all sorts of variations. Here's the current:

def fullpathfile = (newDir + "\\" + newName + ".ts")
def command = ["c:/program files/comskip/comskip.exe", fullpathfile]



println "full command will be '$command'"
def proc = command.execute()
if(!Settings.TEST_MODE) {
          def initialSize = 4096
          def outStream = new ByteArrayOutputStream(initialSize)
          def errStream = new ByteArrayOutputStream(initialSize)

              proc.consumeProcessOutput(outStream, errStream)
              proc.waitFor()
              println 'out:\n' + outStream
              println 'err:\n' + errStream

but of coarse it doesn't work and I'm getting the missing method exception which I just have no idea how to interperate... any help available appreciated. I think if I can just get the full path of the new file name with correct extension for the target (.ts) it should work fine.... famous last words....