Plugin: SJQv4

[Slugger]

Output isn't dumped until the process ends. All output is captured within the task client then sent to the server for logging after the process completes. There will be no logs available until the task is completed/failed. Metadata, however, should be showing up. Though this may be a bug with the UI because if the file is renaming, etc. then the metadata must be available in the script otherwise it'd just fail.

[bikesquid]

Originally Posted by Slugger

Output isn't dumped until the process ends. All output is captured within the task client then sent to the server for logging after the process completes. There will be no logs available until the task is completed/failed. Metadata, however, should be showing up. Though this may be a bug with the UI because if the file is renaming, etc. then the metadata must be available in the script otherwise it'd just fail.

Once the file get's renamed it's no longer associated with the recording's metadata. Just a guess but I suspect sjq (or the ui) is actively looking up the metadata rather it being a static datapoint based on 'where' it all started. The processes all run appropriately but wonder if it's a dangerous proposition without re associating the metadata to the few file.... I'm really relying on SageTV's refresh of recordings to not be coincidental to a critical part of the process. I've seen some reference to this metadata reassignment in all my document spelunking....

[Slugger]

Originally Posted by bikesquid

Once the file get's renamed it's no longer associated with the recording's metadata. Just a guess but I suspect sjq (or the ui) is actively looking up the metadata rather it being a static datapoint based on 'where' it all started. The processes all run appropriately but wonder if it's a dangerous proposition without re associating the metadata to the few file.... I'm really relying on SageTV's refresh of recordings to not be coincidental to a critical part of the process. I've seen some reference to this metadata reassignment in all my document spelunking....

Hmm... not sure if that's it. SJQ grabs the metadata it stores at the time the task is inserted into the queue. It is not changed ever again so the metadata for the task should always be available. The UI reads the metadata from the SJQ database and the SJQ database is populated at task creation time and then never modified again. Can you confirm that the metadata is present when the task is inserted, but then disappears during task execution? If so then that's a bug that needs to be investigated, but I'm not able to reproduce it locally. I'd be surprised if this was actually happening, but I'm not crazy enough to say it's "impossible" either.

[madas]

Having a bit of an issue. Here is my job process

After recording:

add comskip job which runs after recording is complete
add transcode job that waits for the .edl to be created before it proceeds

The problem is occasionally the comskip job doesn't complete, so the transcode job just goes around and around and never starts. It seems to hold up the queue so that nothing else transcodes either.

What is the best solution to this?

Thanks

[Slugger]

Originally Posted by madas

Having a bit of an issue. Here is my job process

After recording:

add comskip job which runs after recording is complete
add transcode job that waits for the .edl to be created before it proceeds

The problem is occasionally the comskip job doesn't complete, so the transcode job just goes around and around and never starts. It seems to hold up the queue so that nothing else transcodes either.

What is the best solution to this?

Thanks

Not sure if this the best solution, but it's what came to the top of my head...

I'm assuming all of these tasks are for a specific media file object you've identified?

If so, just add a counter to the metadata of that media file and once the counter exceeds a max, just return SKIPPED for the task.


So each time the check for an edl is done and it fails, before returning 1 in the test script check a metadata value of that media file. If it's greater than the max then return 2 instead of 1 (meaning to skip the task) else increment the counter. You could also generate a system message to notify you that the transcode task never started because an edl wasn't found after x number of tries. Hopefully that makes sense?

[rsagetv99]

Can someone save me from myself?

I have over 1000 tasks in the Active queue and I cannot clear them using the kill all tasks. Is there a quick way to do it? I fixed the cause of all of the jobs queuing up (i think), but I need to clear out what is there. I even tried uninstalling the SJQ plugin, but that did not work.

[bikesquid]

Originally Posted by Slugger

Can you confirm that the metadata is present when the task is inserted, but then disappears during task execution?

Today it seems to be working, though I changed nothing. I did restart sagetv, so maybe something was 'confused' after all the testing and resolved with the all healing gesture.... the UI behavior is normal as well so... Elvis has left the building.... Will let you know if it 'happens' again. Thx.

[Slugger]

Originally Posted by rsagetv99

Can someone save me from myself?

I have over 1000 tasks in the Active queue and I cannot clear them using the kill all tasks. Is there a quick way to do it? I fixed the cause of all of the jobs queuing up (i think), but I need to clear out what is there. I even tried uninstalling the SJQ plugin, but that did not work.

Follow these instructions to connect directly to the database and then run the following query:

DELETE FROM queue WHERE state NOT IN ('STARTED', 'RUNNING')

NOTE: This will wipe everything in the queue (pending, completed, etc.; everything) except for stuff that's actively running. So you'll lose the task logs of completed tasks, etc. This is probably the best thing to do, but if you want the completed tasks to stick around then try this:

DELETE FROM queue WHERE state IN ('WAITING')

[rsagetv99]

That worked, thanks!

[bikesquid]

I'm lost... a bit and could some guidance out of the woods.
There's what looks to me like an error in my script, it works, but I think that's coincidence. I'm trying to make changes to the routine to check that type does not = "Movie" and move things around a bit, such as remove the special characters from the filename earlier on in the script so I can use the same variable later without having to do it again, etc. , in doing so I 'fixed' what looks to me like an open "{" in this block:

if (type == "Movie") {     
    def newPrefix = title + " /(/" + year + "/)/"
    
} else {
    

    def newPrefix = "${title} - "
    if(seNum != null)
        newPrefix += seNum
    else if(subtitle != "")
        newPrefix += subtitle
    else if(origAirDate != null)
        newPrefix += origAirDate
    else
        newPrefix += "UNKNOWN"

def file = new File(newPrefix + ".ts")
def dir = file.getParent()
def base = FilenameUtils.getBaseName(file.getName())
def ext = FilenameUtils.getExtension(file.getName())
def i = 1
while(file.exists())
    file = new File(dir, "$base-${i++}.$ext")
println "New file is: $file"
newPrefix = FilenameUtils.getBaseName(file.getName())

by adding a "}" after the line ending in "UNKNOWN" and removing a seeming random "}" further down and I end up getting a number of exceptions related to the newPrefix variable not being defined much lower down...

The whole original script:

// def SJQ4_METADATA = ["SJQ4_ID":"4548571", "SJQ4_TYPE":"MediaFile"]

// And then do whatever you were doing as normal; just remember to
// remove the above line before using the scirpt in SJQv4.


import org.apache.commons.io.FilenameUtils
import org.apache.commons.io.FileUtils

private class Settings {
    static public final boolean TEST_MODE = false
}

def mf = MediaFileAPI.GetMediaFileForID(SJQ4_METADATA["SJQ4_ID"].toInteger())

if(mf == null) {  //make sure it's a valid file id
    println "Invalid media file id! [${SJQ4_METADATA['SJQ4_ID']}]"
    return 1
}

AiringAPI.SetWatched(mf); // Lets set the show as watched before we go further

// def cats = ShowAPI.GetShowCatagoriesList(mf)  // show catagogries used to decide if its a move or tv series
def type = MediaFileAPI.GetMediaFileMetadata(mf, "MediaType")
def title = ShowAPI.GetShowTitle(mf)
def sNum = ShowAPI.GetShowSeasonNumber(mf)
def eNum = ShowAPI.GetShowEpisodeNumber(mf)
def seNum = sNum > 0 && eNum > 0 ? String.format("S%02dE%02d", sNum, eNum) : null
def subtitle = ShowAPI.GetShowEpisode(mf)
def origAirDate = ShowAPI.GetOriginalAiringDate(mf) > 0 ? new Date(ShowAPI.GetOriginalAiringDate(mf)).format("yyyy-MM-dd") : null
def year = ShowAPI.GetShowYear(mf)

// if (cats.contains("Movie") || cats.contains("Film")) {
if (type == "Movie") {     
    def newPrefix = title + " /(/" + year + "/)/"
    
} else {
    

    def newPrefix = "${title} - "
    if(seNum != null)
        newPrefix += seNum
    else if(subtitle != "")
        newPrefix += subtitle
    else if(origAirDate != null)
        newPrefix += origAirDate
    else
        newPrefix += "UNKNOWN"

def file = new File(newPrefix + ".ts")
def dir = file.getParent()
def base = FilenameUtils.getBaseName(file.getName())
def ext = FilenameUtils.getExtension(file.getName())
def i = 1
while(file.exists())
    file = new File(dir, "$base-${i++}.$ext")
println "New file is: $file"
newPrefix = FilenameUtils.getBaseName(file.getName())

    def numSegments = MediaFileAPI.GetNumberOfSegments(mf)
    if(numSegments == 1) {
        def prefix = FilenameUtils.getBaseName(MediaFileAPI.GetFileForSegment(mf, 0).getAbsolutePath())
        println "Renaming files that look like '${prefix}.*' to '${newPrefix}.*'..."
        renameMatches(MediaFileAPI.GetParentDirectory(mf), prefix, null, newPrefix)
    } else if(numSegments > 1) {
        for(def i2 = 0; i < numSegments; ++i2) {
            def prefix = FilenameUtils.getBaseName(MediaFileAPI.GetFileForSegment(mf, i2).getAbsolutePath())
            def segPrefix = "$newPrefix-$i2"
            println "Renaming files that look like '${prefix}.*' to '${segPrefix}.*'..."
            renameMatches(MediaFileAPI.GetParentDirecotry(mf), prefix, null, segPrefix)
        }
    } else {
       println "No file segments for given media file!"
       return 1
    }

    
    return 0
}

// Pass null for newDir to keep renamed file in same directory

def renameMatches(def oldDir, def prefix, def newDir, def newPrefix) {
    if(newDir == null)
        newDir = oldDir
    Utility.DirectoryListing(oldDir).each {
        if(FilenameUtils.wildcardMatchOnSystem(it.getName(), "${prefix}.*")) {
            def newName = (newPrefix + it.getName().substring(it.getName().indexOf('.')) =~ /[\/\\:*?<>]/).replaceAll("")
              if(!Settings.TEST_MODE) {
                try {
                  FileUtils.moveFile(it, new File(newDir, newName))
                }  catch(IOException e) {
                   e.printStackTrace()
                   println("Failed to move file to destination!")
                   FileUtils.copyFile(it, new File(newDir, newName))
                   }
                } else {
                      println "Would rename '$it' to '$newName' if test mode were disabled!"
                }
               
        }
    }

// comskip attempt

def newName2 = (newPrefix =~ /[\/\\:*?<>]/).replaceAll("") // remove special characthers from newPrefix
def fullpathfile = newDir.toString() + "/" + newName2 + ".ts" // Convert newDir from file type to string variable and add .ts to filename, combine.

def command = ["c:/program files/comskip/comskip.exe", fullpathfile] //define command as an array

println "full command will be '$command'"
def proc = command.execute()
if(!Settings.TEST_MODE) { // Are we running test mode, if not do this
          def initialSize = 4096
          def outStream = new ByteArrayOutputStream(initialSize)
          def errStream = new ByteArrayOutputStream(initialSize)

              proc.consumeProcessOutput(outStream, errStream)
              proc.waitFor()
              println 'out:\n' + outStream
              println 'err:\n' + errStream

            } else {  // if we are in test mode do this
                println "Would run '$command' if test mode were disabled!"
            }
}

return 0;

And the modified that's not working:

def SJQ4_METADATA = ["SJQ4_ID":"4586791", "SJQ4_TYPE":"MediaFile"]

// And then do whatever you were doing as normal; just remember to
// remove the above line before using the scirpt in SJQv4.


import org.apache.commons.io.FilenameUtils
import org.apache.commons.io.FileUtils

private class Settings {
    static public final boolean TEST_MODE = false
}

def mf = MediaFileAPI.GetMediaFileForID(SJQ4_METADATA["SJQ4_ID"].toInteger())

if(mf == null) {  //make sure it's a valid file id
    println "Invalid media file id! [${SJQ4_METADATA['SJQ4_ID']}]"
    return 1
}

AiringAPI.SetWatched(mf); // Lets set the show as watched before we go further

def type = MediaFileAPI.GetMediaFileMetadata(mf, "MediaType")
def title = ShowAPI.GetShowTitle(mf)
def sNum = ShowAPI.GetShowSeasonNumber(mf)
def eNum = ShowAPI.GetShowEpisodeNumber(mf)
def seNum = sNum > 0 && eNum > 0 ? String.format("S%02dE%02d", sNum, eNum) : null
def subtitle = ShowAPI.GetShowEpisode(mf)
def origAirDate = ShowAPI.GetOriginalAiringDate(mf) > 0 ? new Date(ShowAPI.GetOriginalAiringDate(mf)).format("yyyy-MM-dd") : null
def year = ShowAPI.GetShowYear(mf)
def newPrefix = "${title}"


if (type != "Movie") {    
    newPrefix += " - "
    if(seNum != null)
        newPrefix += seNum
    else if(subtitle != "")
        newPrefix += subtitle
    else if(origAirDate != null)
        newPrefix += origAirDate
    else
        newPrefix += "UNKNOWN"
}
else {
    println "we've got us a movie!"
    if(year != null)
        newPrefix += "/(/ ${year})"
    else if(origAirDate != null)
        newPrefix += origAirDate
}


newPrefix = (newPrefix =~ /[\/\\:*?<>]/).replaceAll("")

def file = new File(newPrefix + ".ts")
def dir = file.getParent()
def base = FilenameUtils.getBaseName(file.getName())
def ext = FilenameUtils.getExtension(file.getName())
def i = 1
while(file.exists())
    file = new File(dir, "$base-${i++}.$ext")
println "New file is: $file"

newPrefix = FilenameUtils.getBaseName(file.getName())

def fullfilename = file.toString()
println "fullfilename is: $fullfilename"

def numSegments = MediaFileAPI.GetNumberOfSegments(mf)
    if(numSegments == 1) {
        def prefix = FilenameUtils.getBaseName(MediaFileAPI.GetFileForSegment(mf, 0).getAbsolutePath())
        println "Renaming files that look like '${prefix}.*' to '${newPrefix}.*'..."
        renameMatches(MediaFileAPI.GetParentDirectory(mf), prefix, null, newPrefix)
    } else if(numSegments > 1) {
        for(def i2 = 0; i < numSegments; ++i2) {
            def prefix = FilenameUtils.getBaseName(MediaFileAPI.GetFileForSegment(mf, i2).getAbsolutePath())
            def segPrefix = "$newPrefix-$i2"
            println "Renaming files that look like '${prefix}.*' to '${segPrefix}.*'..."
            renameMatches(MediaFileAPI.GetParentDirecotry(mf), prefix, null, segPrefix)
        }
    } else {
       println "No file segments for given media file!"
       return 1
    }
    return 0
}

// Pass null for newDir to keep renamed file in same directory

def renameMatches(def oldDir, def prefix, def newDir, def newPrefix) {
    if(newDir == null)
        newDir = oldDir
    Utility.DirectoryListing(oldDir).each {
        if(FilenameUtils.wildcardMatchOnSystem(it.getName(), "${prefix}.*")) {
            def newName = (newPrefix + it.getName().substring(it.getName().indexOf('.')))  // =~ /[\/\\:*?<>]/).replaceAll("")
              if(!Settings.TEST_MODE) {
                try {
                  FileUtils.moveFile(it, new File(newDir, newName))
                }  catch(IOException e) {
                   e.printStackTrace()
                   println("Failed to move file to destination!")
                   FileUtils.copyFile(it, new File(newDir, newName))
                   }
                } else {
                      println "Would rename '$it' to '$newName' if test mode were disabled!"
                }
               
        }
    }

// comskip attempt

//def newName2 = (newPrefix =~ /[\/\\:*?<>]/).replaceAll("") // remove special characthers from newPrefix
// def fullpathfile = "${newDir}.toString()/${newfilename}.toString)" // Convert newDir from file type to string variable and add .ts to filename, combine.
def fullpathfile = newDir.toString() + "\\$fullfilename" 
println "fullpathfile is=$fullpathfile"
// fullpathfile += fullfilename.toString()
// def fullpathfile = "${newDir}.toString()/${newfilename}.toString()"

def command = ["c:/program files/comskip/comskip.exe", fullpathfile] //define command as an array

println "full command will be '$command'"
def proc = command.execute()
if(!Settings.TEST_MODE) { // Are we running test mode, if not do this
          def initialSize = 4096
          def outStream = new ByteArrayOutputStream(initialSize)
          def errStream = new ByteArrayOutputStream(initialSize)

              proc.consumeProcessOutput(outStream, errStream)
              proc.waitFor()
              println 'out:\n' + outStream
              println 'err:\n' + errStream

            } else {  // if we are in test mode do this
                println "Would run '$command' if test mode were disabled!"
            }
return 0
}

I don't have the interim steps where I change the name of the variable, but it was all in attempts to correct the problem. Among other things I'm wondering if the "}" will close a variable from use further down the script?

[razrsharpe]

@bikesquid - so i have no idea if this is your problem or not... but i loaded up your two original scripts into notepad++ and did a difference on them so i could easily see what changed and the one thing i notice is in the original script the final "return 0" is outside the last } and in the modified one its inside...

like i said i havent examined the code in depth to see if that is really the problem but it was one difference i noticed... if you havent already you may want to fire up the scripts in a difference editor (notepad++ works great for this) and then it makes it very easy to see the differences.

[rsagetv99]

Can someone tell me if this should work? It is the media_file_scanner script with lines added to only process .mpg and .ts files. I tried it and I am getting EOF Exception errors. I highlighted my changes in red.

/*
    Media File Scanner/Task Queuer
    
    Last Modified: 15 Feb 2011
           Author: Derek Battams <derek AT battams DOT ca>
    
    Use this script to periodically scan your media objects and check to see if
    any need to have tasks queued on them.  Basically, set up your media mask
    and then modify the needsProcessing() function to do the checks you want
    against each object.  If needsProcessing() returns true for an object then
    it will queue up each task listed in the taskIds list for that object.
    
    Typically, you would run this script periodically via the SJQv4 crontab.
    
    PLEASE RUN THIS SCRIPT WITH testMode = true BEFORE ALLOWING IT TO ACTUALLY
    QUEUE UP TASKS!!
*/

/***** CONFIGURE BELOW *****/

def testMode = false // If true, only print out which media files would be queued up, don't actually add the tasks to the queue
def mediaMask = "T" // What types of media should be scanned? (T = TV, M = Music, V = Imported Video, D = DVD, B = BluRay, P = Pictures; TMV = TV + Music + Imports, etc.)
def taskIds = ["COMSKIP","COMSKIPSD"] // Multiple tasks can be listed, separated by commas

/*
    Returns true if the argument needs to be queued or false if it should be skipped
    
    Modify this function to determine which media files get queued up and which don't
*/
def needsProcessing(Object mediaFile) {
    // This function could be written in a much more condensed manner, but I'm breaking it up for the sake of readability
    
    // So let's skip queuing this media file if it's live tv or an IR recording
    if(AiringAPI.IsNotManualOrFavorite(mediaFile))
        return false
    
    // Personally, I don't comskip until the recording is done, so don't queue up recordings in progress
    if(MediaFileAPI.IsFileCurrentlyRecording(mediaFile))
        return false

    // Let's also skip it if it's from a channel known not to have commercials (adjust the regex accordingly)
    if(AiringAPI.GetAiringChannelName(mediaFile) =~ /HBO.*|M(?:HD){0,1}|WPBS|.*PPV.*/)
        return false

    // Let's also skip it if there is already an edl file for the media file; adjust the extensions, if necessary        
    if(hasArtifacts(mediaFile, ["edl"])) // This function is defined at the bottom of the file
        return false

    // Let's skip it if the file is not a supported media file type
    if(isSupportedExt(mediaFile, ["mpg","ts"])) // This function is defined at the bottom of the file
        return true
        
    // I don't like comskipping hockey and football games, I just have the skip ahead set properly on my remote so don't bother
    if(MediaFileAPI.GetMediaTitle(mediaFile) =~ /NFL Football|MLB Baseball|NHL Hockey|College Football|College Basketball/)
        return false
    
    // All our tests have passed so return true
    return true
}

/***** END CONFIG BLOCK *****/

/***** DO NOT MODIFY BELOW THIS LINE *****/

import com.google.code.sagetvaddons.sjq.network.ServerClient
import com.google.code.sagetvaddons.metadata.Factory
import org.apache.commons.io.FilenameUtils

def sc = !testMode ? new ServerClient() : null
MediaFileAPI.GetMediaFiles(mediaMask).each { mf ->
    if(needsProcessing(mf)) {
        if(!testMode)
            taskIds.each { id ->
                sc.addTask(id, Factory.getMap(mf))
            }
        else
            println "Would queue up '${MediaFileAPI.GetMediaTitle(mf)}' (${MediaFileAPI.GetMediaFileID(mf)}); skipped because test mode is TRUE"
    }
}
if(sc != null)
    sc.close()
return 0

// Returns true if any segment of the given media file has at least one artifact of any of the given artifact extensions; false otherwise
def hasArtifacts(Object mf, List exts) {
    for(def it : MediaFileAPI.GetSegmentFiles(mf)) {
        def absPath = it.getAbsolutePath()
        def dir = FilenameUtils.getFullPath(absPath)
        def base = FilenameUtils.getBaseName(absPath)
        for(def ext : exts) {
            def artifact = "${dir}${base}.$ext"
            if(Utility.IsFilePath(artifact))
                {return true}
	}
    }
    return false
}

// Returns false if the extension is mpg or ts
def isSupportedExt(Object mf, List exts2) {
    for(def it : MediaFileAPI.GetSegmentFiles(mf)) {
        def absPath2 = it.getAbsolutePath()
	def extension = FilenameUtils.getExtension(absPath2)
	for(def ext2 : exts2) {
	    if(extension==ext2)
			{return true}
        }
    }
    return false
}

[bikesquid]

Originally Posted by razrsharpe

@bikesquid - so i have no idea if this is your problem or not... but i loaded up your two original scripts into notepad++ and did a difference on them so i could easily see what changed and the one thing i notice is in the original script the final "return 0" is outside the last } and in the modified one its inside...

like i said i havent examined the code in depth to see if that is really the problem but it was one difference i noticed... if you havent already you may want to fire up the scripts in a difference editor (notepad++ works great for this) and then it makes it very easy to see the differences.

a good find, but doesn't seem critical to the script running or not. (have done both ways.)
The errors I'm getting are (using xxx for the script file name):
MissingPropertyException: No such property:fullfilename for class: xxx
@ xxx$renamaneMatches(xxx:116) -this is the line #
@ xxx$renameMatches.callCurrent(Unknown Source)
@ xxx.run(xxx:74) - again the line #
Problem is that though I've seen MissingPropertyException errors before and slugger has explained it to me, I don't see how this is the same issue and don't know how to interpret it.

[Slugger]

Originally Posted by bikesquid

a good find, but doesn't seem critical to the script running or not. (have done both ways.)
The errors I'm getting are (using xxx for the script file name):
MissingPropertyException: No such property:fullfilename for class: xxx
@ xxx$renamaneMatches(xxx:116) -this is the line #
@ xxx$renameMatches.callCurrent(Unknown Source)
@ xxx.run(xxx:74) - again the line #
Problem is that though I've seen MissingPropertyException errors before and slugger has explained it to me, I don't see how this is the same issue and don't know how to interpret it.

I would need to see the full, unedited script you tried to run along with the full stack trace of the errors. It's important to include the script unedited so that the line numbers in the error message lineup properly with the line numbers in the script.

[bikesquid]

Originally Posted by Slugger

I would need to see the full, unedited script you tried to run along with the full stack trace of the errors. It's important to include the script unedited so that the line numbers in the error message lineup properly with the line numbers in the script.

I'm using sagegroovy for testing is there a way to output the full stack trace, it's a heck of a lot of typing otherwise...? Normal cut/paste from the output window doesn't seem to be doing anything

[bikesquid]

Originally Posted by bikesquid

I'm using sagegroovy for testing is there a way to output the full stack trace, it's a heck of a lot of typing otherwise...? Normal cut/paste from the output window doesn't seem to be doing anything

Well figured something out, not sure it's the right info, please let me know if not.
thx.

script:

def SJQ4_METADATA = ["SJQ4_ID":"4581185", "SJQ4_TYPE":"MediaFile"]

// And then do whatever you were doing as normal; just remember to
// remove the above line before using the scirpt in SJQv4.


import org.apache.commons.io.FilenameUtils
import org.apache.commons.io.FileUtils

private class Settings {
    static public final boolean TEST_MODE = false
}

def mf = MediaFileAPI.GetMediaFileForID(SJQ4_METADATA["SJQ4_ID"].toInteger())

if(mf == null) {  //make sure it's a valid file id
    println "Invalid media file id! [${SJQ4_METADATA['SJQ4_ID']}]"
    return 1
}

AiringAPI.SetWatched(mf); // Lets set the show as watched before we go further

def type = MediaFileAPI.GetMediaFileMetadata(mf, "MediaType")
def title = ShowAPI.GetShowTitle(mf)
def sNum = ShowAPI.GetShowSeasonNumber(mf)
def eNum = ShowAPI.GetShowEpisodeNumber(mf)
def seNum = sNum > 0 && eNum > 0 ? String.format("S%02dE%02d", sNum, eNum) : null
def subtitle = ShowAPI.GetShowEpisode(mf)
def origAirDate = ShowAPI.GetOriginalAiringDate(mf) > 0 ? new Date(ShowAPI.GetOriginalAiringDate(mf)).format("yyyy-MM-dd") : null
def year = ShowAPI.GetShowYear(mf)
def newPrefix = "${title}"


if (type != "Movie") {    
    newPrefix += " - "
    if(seNum != null)
        newPrefix += seNum
    else if(subtitle != "")
        newPrefix += subtitle
    else if(origAirDate != null)
        newPrefix += origAirDate
    else
        newPrefix += "UNKNOWN"
}
else {
    println "we've got us a movie!"
    if(year != null)
        newPrefix += "/(/ ${year})"
    else if(origAirDate != null)
        newPrefix += origAirDate
}


newPrefix = (newPrefix =~ /[\/\\:*?<>]/).replaceAll("")

def file = new File(newPrefix + ".ts")
def dir = file.getParent()
def base = FilenameUtils.getBaseName(file.getName())
def ext = FilenameUtils.getExtension(file.getName())
def i = 1
while(file.exists())
    file = new File(dir, "$base-${i++}.$ext")
println "New file is: $file"

newPrefix = FilenameUtils.getBaseName(file.getName())

def fullfilename = file.toString()
println "fullfilename is: $fullfilename"

def numSegments = MediaFileAPI.GetNumberOfSegments(mf)
    if(numSegments == 1) {
        def prefix = FilenameUtils.getBaseName(MediaFileAPI.GetFileForSegment(mf, 0).getAbsolutePath())
        println "Renaming files that look like '${prefix}.*' to '${newPrefix}.*'..."
        renameMatches(MediaFileAPI.GetParentDirectory(mf), prefix, null, newPrefix)
    } else if(numSegments > 1) {
        for(def i2 = 0; i < numSegments; ++i2) {
            def prefix = FilenameUtils.getBaseName(MediaFileAPI.GetFileForSegment(mf, i2).getAbsolutePath())
            def segPrefix = "$newPrefix-$i2"
            println "Renaming files that look like '${prefix}.*' to '${segPrefix}.*'..."
            renameMatches(MediaFileAPI.GetParentDirecotry(mf), prefix, null, segPrefix)
        }
    } else {
       println "No file segments for given media file!"
       return 1
    }
    return 0


// Pass null for newDir to keep renamed file in same directory

def renameMatches(def oldDir, def prefix, def newDir, def newPrefix) {
    if(newDir == null)
        newDir = oldDir
    Utility.DirectoryListing(oldDir).each {
        if(FilenameUtils.wildcardMatchOnSystem(it.getName(), "${prefix}.*")) {
            def newName = (newPrefix + it.getName().substring(it.getName().indexOf('.')))  // =~ /[\/\\:*?<>]/).replaceAll("")
              if(!Settings.TEST_MODE) {
                try {
                  FileUtils.moveFile(it, new File(newDir, newName))
                }  catch(IOException e) {
                   e.printStackTrace()
                   println("Failed to move file to destination!")
                   FileUtils.copyFile(it, new File(newDir, newName))
                   }
                } else {
                      println "Would rename '$it' to '$newName' if test mode were disabled!"
                }
               
        }
    }

// comskip attempt

//def newName2 = (newPrefix =~ /[\/\\:*?<>]/).replaceAll("") // remove special characthers from newPrefix
// def fullpathfile = "${newDir}.toString()/${newfilename}.toString)" // Convert newDir from file type to string variable and add .ts to filename, combine.
def fullpathfile = newDir.toString() + "\\$fullfilename" 
println "fullpathfile is=$fullpathfile"
// fullpathfile += fullfilename.toString()
// def fullpathfile = "${newDir}.toString()/${newfilename}.toString()"

def command = ["c:/program files/comskip/comskip.exe", fullpathfile] //define command as an array

println "full command will be '$command'"
def proc = command.execute()
if(!Settings.TEST_MODE) { // Are we running test mode, if not do this
          def initialSize = 4096
          def outStream = new ByteArrayOutputStream(initialSize)
          def errStream = new ByteArrayOutputStream(initialSize)

              proc.consumeProcessOutput(outStream, errStream)
              proc.waitFor()
              println 'out:\n' + outStream
              println 'err:\n' + errStream

            } else {  // if we are in test mode do this
                println "Would run '$command' if test mode were disabled!"
            }
            
return 0
}

output window

New file is: Nikita - S01E17.ts
fullfilename is: Nikita - S01E17.ts
Renaming files that look like 'Nikita-Covenant-4500262-0.*' to 'Nikita - S01E17.*'...
Exception thrown

groovy.lang.MissingPropertyException: No such property: fullfilename for class: file_rename_experiment

    at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:49)

    at org.codehaus.groovy.runtime.callsite.PogoGetPropertySite.getProperty(PogoGetPropertySite.java:49)

    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callGroovyObjectGetProperty(AbstractCallSite.java:239)

    at file_rename_experiment.renameMatches(file_rename_experiment.groovy:116)

    at file_rename_experiment$renameMatches.callCurrent(Unknown Source)

    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:44)

    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141)

    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:161)

    at file_rename_experiment.run(file_rename_experiment.groovy:74)

    at groovy.lang.GroovyShell.runScriptOrMainOrTestOrRunnable(GroovyShell.java:264)

    at groovy.lang.GroovyShell.run(GroovyShell.java:513)

    at groovy.lang.GroovyShell.run(GroovyShell.java:170)

    at groovy.lang.GroovyShell$run$6.call(Unknown Source)

    at groovy.ui.Console$_runScriptImpl_closure16.doCall(Console.groovy:890)

    at sun.reflect.GeneratedMethodAccessor875.invoke(Unknown Source)

    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

    at java.lang.reflect.Method.invoke(Unknown Source)

    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:88)

    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)

    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:273)

    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:886)

    at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66)

    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:149)

    at groovy.ui.Console$_runScriptImpl_closure16.doCall(Console.groovy)

    at sun.reflect.GeneratedMethodAccessor874.invoke(Unknown Source)

    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

    at java.lang.reflect.Method.invoke(Unknown Source)

    at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:88)

    at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)

    at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:273)

    at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:886)

    at groovy.lang.Closure.call(Closure.java:276)

    at groovy.lang.Closure.call(Closure.java:271)

    at groovy.lang.Closure.run(Closure.java:354)

    at java.lang.Thread.run(Unknown Source)

[Slugger]

@rsagetv99 I copied/pasted the script as you have it here into SageGroovy and it ran fine for me with no edits at all. Did you edit it on Windows and try to run it in Linux or vice versa? That may cause problems with the Groovy interpreter, not sure.

However, it doesn't quite do what you want it to so I made a few tweaks...

/*
    Media File Scanner/Task Queuer
    
    Last Modified: 15 Feb 2011
           Author: Derek Battams <derek AT battams DOT ca>
    
    Use this script to periodically scan your media objects and check to see if
    any need to have tasks queued on them.  Basically, set up your media mask
    and then modify the needsProcessing() function to do the checks you want
    against each object.  If needsProcessing() returns true for an object then
    it will queue up each task listed in the taskIds list for that object.
    
    Typically, you would run this script periodically via the SJQv4 crontab.
    
    PLEASE RUN THIS SCRIPT WITH testMode = true BEFORE ALLOWING IT TO ACTUALLY
    QUEUE UP TASKS!!
*/

/***** CONFIGURE BELOW *****/

def testMode = false // If true, only print out which media files would be queued up, don't actually add the tasks to the queue
def mediaMask = "T" // What types of media should be scanned? (T = TV, M = Music, V = Imported Video, D = DVD, B = BluRay, P = Pictures; TMV = TV + Music + Imports, etc.)
def taskIds = ["COMSKIP","COMSKIPSD"] // Multiple tasks can be listed, separated by commas

/*
    Returns true if the argument needs to be queued or false if it should be skipped
    
    Modify this function to determine which media files get queued up and which don't
*/
def needsProcessing(Object mediaFile) {
    // This function could be written in a much more condensed manner, but I'm breaking it up for the sake of readability
    
    // So let's skip queuing this media file if it's live tv or an IR recording
    if(AiringAPI.IsNotManualOrFavorite(mediaFile))
        return false
    
    // Personally, I don't comskip until the recording is done, so don't queue up recordings in progress
    if(MediaFileAPI.IsFileCurrentlyRecording(mediaFile))
        return false

    // Let's also skip it if it's from a channel known not to have commercials (adjust the regex accordingly)
    if(AiringAPI.GetAiringChannelName(mediaFile) =~ /HBO.*|M(?:HD){0,1}|WPBS|.*PPV.*/)
        return false

    // Let's also skip it if there is already an edl file for the media file; adjust the extensions, if necessary        
    if(hasArtifacts(mediaFile, ["edl"])) // This function is defined at the bottom of the file
        return false

    // Let's skip it if the file is not a supported media file type
    if(!isSupportedExt(mediaFile, ["mpg","ts"])) // This function is defined at the bottom of the file
        return false
        
    // I don't like comskipping hockey and football games, I just have the skip ahead set properly on my remote so don't bother
    if(MediaFileAPI.GetMediaTitle(mediaFile) =~ /NFL Football|MLB Baseball|NHL Hockey|College Football|College Basketball/)
        return false
    
    // All our tests have passed so return true
    return true
}

/***** END CONFIG BLOCK *****/

/***** DO NOT MODIFY BELOW THIS LINE *****/

import com.google.code.sagetvaddons.sjq.network.ServerClient
import com.google.code.sagetvaddons.metadata.Factory
import org.apache.commons.io.FilenameUtils

def sc = !testMode ? new ServerClient() : null
MediaFileAPI.GetMediaFiles(mediaMask).each { mf ->
    if(needsProcessing(mf)) {
        if(!testMode)
            taskIds.each { id ->
                sc.addTask(id, Factory.getMap(mf))
            }
        else
            println "Would queue up '${MediaFileAPI.GetMediaTitle(mf)}' (${MediaFileAPI.GetMediaFileID(mf)}); skipped because test mode is TRUE"
    }
}
if(sc != null)
    sc.close()
return 0

// Returns true if any segment of the given media file has at least one artifact of any of the given artifact extensions; false otherwise
def hasArtifacts(Object mf, List exts) {
    for(def it : MediaFileAPI.GetSegmentFiles(mf)) {
        def absPath = it.getAbsolutePath()
        def dir = FilenameUtils.getFullPath(absPath)
        def base = FilenameUtils.getBaseName(absPath)
        for(def ext : exts) {
            def artifact = "${dir}${base}.$ext"
            if(Utility.IsFilePath(artifact))
                {return true}
    }
    }
    return false
}

// Returns false if the extension is mpg or ts
def isSupportedExt(Object mf, List supported) {
    def mfExts = []
    MediaFileAPI.GetSegmentFiles(mf).each {
        mfExts += FilenameUtils.getExtension(it.getName())
    }
    if(mfExts.size() == 0)
        return false
    for(def ext : supported)
        if(mfExts.contains(ext))
            return true
    return false
}

The biggest change is in needsProcessing(). You were doing a positive check against the file extension, but you have to do negative checks and return false when a check fails instead of returning true if it passes because failed files will continue on through and eventually return true anyway and that's not what you want.

I also rewrote your isSupportedExt() function because it was hard for me to follow your logic, but it turns out that may not have been necessary since the real problem was in needsProcessing()

[Slugger]

@bikesquid Time for some code reorg.

You've got a function defined in the middle of your main execution path. It seems you've gotten the bottom half of main stuck inside the function, which is why the fullfilename var isn't accessible on line 116...

That } you thought was random really wasn't. Here's my edits... you didn't have newDir reassigned to anything - look for my big BLOCK letter comment for where you need to reassign that var if you want to move the renamed files to a different dir.

Some notes:

• Notice how I moved the function to the bottom of the file; typically you'll define your functions at the top or the bottom then write your execution path. You should never mix your main code with functions in the middle, very hard to read.
• Cleanup your indentations; this is a little anal on my part, but not really. I know you're not a programmer, but you'll want to improve upon the readability of your code, especially when posting it for review/assistance. I don't care how your code works or if it's efficient, etc.; that's (usually) irrelevant to me, but what is relevant is how easy is it to read the code. Properly indented code reads 10,000x easier than code with random indentation levels all over the place. Again, I had intentions of helping you out here, but, again, it's Friday and there's a ball game recording right now that's calling my name.
• You might want to start breaking things down into functions. You've really added a lot to the original script. Instead of trying to do it all in one big main block, you might want to investigate the use of functions. I was going to assist you there, but it's Friday and I don't want to refactor Groovy script all night so you're on your own there.
• One step further: Is there a reason you're running comskip in the same script? Why not do your rename/moves and then queue up separate comskip jobs as needed. Not necessary, but seems logical to me.

def SJQ4_METADATA = ["SJQ4_ID":"6346252", "SJQ4_TYPE":"MediaFile"]

// And then do whatever you were doing as normal; just remember to
// remove the above line before using the scirpt in SJQv4.


import org.apache.commons.io.FilenameUtils
import org.apache.commons.io.FileUtils

private class Settings {
    static public final boolean TEST_MODE = true
}

def mf = MediaFileAPI.GetMediaFileForID(SJQ4_METADATA["SJQ4_ID"].toInteger())

if(mf == null) {  //make sure it's a valid file id
    println "Invalid media file id! [${SJQ4_METADATA['SJQ4_ID']}]"
    return 1
}

AiringAPI.SetWatched(mf); // Lets set the show as watched before we go further

def type = MediaFileAPI.GetMediaFileMetadata(mf, "MediaType")
def title = ShowAPI.GetShowTitle(mf)
def sNum = ShowAPI.GetShowSeasonNumber(mf)
def eNum = ShowAPI.GetShowEpisodeNumber(mf)
def seNum = sNum > 0 && eNum > 0 ? String.format("S%02dE%02d", sNum, eNum) : null
def subtitle = ShowAPI.GetShowEpisode(mf)
def origAirDate = ShowAPI.GetOriginalAiringDate(mf) > 0 ? new Date(ShowAPI.GetOriginalAiringDate(mf)).format("yyyy-MM-dd") : null
def year = ShowAPI.GetShowYear(mf)
def newPrefix = "${title}"


if (type != "Movie") {    
    newPrefix += " - "
    if(seNum != null)
        newPrefix += seNum
    else if(subtitle != "")
        newPrefix += subtitle
    else if(origAirDate != null)
        newPrefix += origAirDate
    else
        newPrefix += "UNKNOWN"
}
else {
    println "we've got us a movie!"
    if(year != null)
        newPrefix += "/(/ ${year})"
    else if(origAirDate != null)
        newPrefix += origAirDate
}


newPrefix = (newPrefix =~ /[\/\\:*?<>]/).replaceAll("")

def file = new File(newPrefix + ".ts")
def dir = file.getParent()
def base = FilenameUtils.getBaseName(file.getName())
def ext = FilenameUtils.getExtension(file.getName())
def i = 1
while(file.exists())
    file = new File(dir, "$base-${i++}.$ext")
println "New file is: $file"

newPrefix = FilenameUtils.getBaseName(file.getName())

def fullfilename = file.toString()
println "fullfilename is: $fullfilename"

def numSegments = MediaFileAPI.GetNumberOfSegments(mf)
    if(numSegments == 1) {
        def prefix = FilenameUtils.getBaseName(MediaFileAPI.GetFileForSegment(mf, 0).getAbsolutePath())
        println "Renaming files that look like '${prefix}.*' to '${newPrefix}.*'..."
        renameMatches(MediaFileAPI.GetParentDirectory(mf), prefix, null, newPrefix)
    } else if(numSegments > 1) {
        for(def i2 = 0; i < numSegments; ++i2) {
            def prefix = FilenameUtils.getBaseName(MediaFileAPI.GetFileForSegment(mf, i2).getAbsolutePath())
            def segPrefix = "$newPrefix-$i2"
            println "Renaming files that look like '${prefix}.*' to '${segPrefix}.*'..."
            renameMatches(MediaFileAPI.GetParentDirecotry(mf), prefix, null, segPrefix)
        }
    } else {
       println "No file segments for given media file!"
       return 1
    }

// comskip attempt

//def newName2 = (newPrefix =~ /[\/\\:*?<>]/).replaceAll("") // remove special characthers from newPrefix
// def fullpathfile = "${newDir}.toString()/${newfilename}.toString)" // Convert newDir from file type to string variable and add .ts to filename, combine.
// YOU WILL PROBABLY WANT TO REASSIGN newDir HERE!!
def newDir = MediaFileAPI.GetParentDirectory(mf)
def fullpathfile = newDir.toString() + "\\$fullfilename" 
println "fullpathfile is=$fullpathfile"
// fullpathfile += fullfilename.toString()
// def fullpathfile = "${newDir}.toString()/${newfilename}.toString()"

def command = ["c:/program files/comskip/comskip.exe", fullpathfile] //define command as an array

println "full command will be '$command'"
if(!Settings.TEST_MODE) { // Are we running test mode, if not do this
          def proc = command.execute()
          def initialSize = 4096
          def outStream = new ByteArrayOutputStream(initialSize)
          def errStream = new ByteArrayOutputStream(initialSize)

              proc.consumeProcessOutput(outStream, errStream)
              proc.waitFor()
              println 'out:\n' + outStream
              println 'err:\n' + errStream

            } else {  // if we are in test mode do this
                println "Would run '$command' if test mode were disabled!"
            }
            
return 0

// Pass null for newDir to keep renamed file in same directory
def renameMatches(def oldDir, def prefix, def newDir, def newPrefix) {
    if(newDir == null)
        newDir = oldDir
    Utility.DirectoryListing(oldDir).each {
        if(FilenameUtils.wildcardMatchOnSystem(it.getName(), "${prefix}.*")) {
            def newName = (newPrefix + it.getName().substring(it.getName().indexOf('.')))  // =~ /[\/\\:*?<>]/).replaceAll("")
              if(!Settings.TEST_MODE) {
                try {
                  FileUtils.moveFile(it, new File(newDir, newName))
                }  catch(IOException e) {
                   e.printStackTrace()
                   println("Failed to move file to destination!")
                   FileUtils.copyFile(it, new File(newDir, newName))
                   }
                } else {
                      println "Would rename '$it' to '$newName' if test mode were disabled!"
                }
               
        }
    }
}

[rsagetv99]

Originally Posted by Slugger

@rsagetv99 I copied/pasted the script as you have it here into SageGroovy and it ran fine for me with no edits at all. Did you edit it on Windows and try to run it in Linux or vice versa? That may cause problems with the Groovy interpreter, not sure.

However, it doesn't quite do what you want it to so I made a few tweaks...

I noticed the misuse of negation too after I posted, but I tried fixing it and couldn't get it to work.

I tried your script and it looks like it is working perfectly.

Thanks alot.

[bikesquid]

Originally Posted by Slugger

you didn't have newDir reassigned to anything - look for my big BLOCK letter comment for where you need to reassign that var if you want to move the renamed files to a different dir.

For the time being same location is what's intended, but once it's built out I'll modify that, thank you for pointing it out.

Originally Posted by Slugger

Notice how I moved the function to the bottom of the file; typically you'll define your functions at the top or the bottom then write your execution path. You should never mix your main code with functions in the middle, very hard to read.

I'll have to take some time to digest and understand this by breaking down how you changed it and try to figure out why, but in concept it makes sense.

Originally Posted by Slugger

Cleanup your indentations

Damn! I thought I had....

Originally Posted by Slugger

You might want to start breaking things down into functions. You've really added a lot to the original script. Instead of trying to do it all in one big main block, you might want to investigate the use of functions.

I really wanted to do it that way, but have no idea how to do function calls within a script, which is why my version is ooo linear. Have been looking at the noob section of the groovy website but it's all written in geekeze and though I'm fluent in three languages, 'dat ain't one-o-dem. Comparing your rejiggering w/my flaccid variant will help pick apart some of that concept, again, thanks.

Originally Posted by Slugger

Is there a reason you're running comskip in the same script? Why not do your rename/moves and then queue up separate comskip jobs as needed.

Ummm, mostly because when I tried and it bombed I didn't want to ask yet another question... so I ran away.... After-all, if ain't "completely" broke, don't fix it. I assumed it would be as simple as "(path/scriptxxx.groovy).execute" but then I couldn't figure out how to pass variables back and forth between scripts.

I do have a follow-up "under the hood" SJQ question though. Is there a way to inject a job into the queue from outside sage? I.E. once comskip is done I'd like to review the edits and then put the job back in to run a few more steps, such as cutting the comskip output from the file, handbrake etc. But as another process(s) and as the object file isn't a sagetv recording any longer it's difficult to find from within sagetv.