Can anyone host a mirror of SageTV hosted plugins?

[rnadeau]

I have made the change to this script to downlaod all files which is about 797meg. I use sjq to run it daily. It will fail every time after the first time it is run. If I delete the folder , it will run properly and then fail every time again till I delete the folder it creates. I have looked at teh log and it seems to be a java memory error. I don't know how to fix this short of creating a file that deletes the folder that is created and redownloading all 797 meg of files, which I don't what to do if at all possible. I am including the log file, if someone could help with this issue, it would be much appreciated.

[osx-addict]

Stupid question -- coming in a bit late in the discussion.. What is the goal of the plugin -- ? Is it to just mirror a remote site and snarf up whatever it finds? Any particular reason it's written in Java? Seems like another scripting language would be better suited -- any reason it's being run on Windows? Can 'the script' be run on Unix (e.g. Linux, OSX, etc) if there's no actual requirement to run on Windows?

[darcilicious]

I don't know the particulars of the script itself but one reason for it being done in Java might be is that Java is cross-plaform compatible -- it should run on Linxu, OS X, and Windows....

[Slugger]

Originally Posted by BobbyDing

Hi Folks,
I discovered an issue with this I think (maybe it's just my setup). The first execution of the script creates a good repo.xml with the proper local links. Any subsequent executions where a new file is not downloaded ("latest version already downloaded") will leave the original web link in the repo.xml.

I don't know groovy or java, but a made a quick repair that seems to work. I copied and pasted the full "it.location" line to within the ELSE so that it does the "it.location" line for either case. So far all executions of the script have been creating the repo.xml with the proper local links even when no updates are downloaded.

I'm including a copy of the repo.groovy below.

Hopefully I didn't screw up something else.

Bobby

I'll have to take a look, but what you describe sounds reasonable. Such a bug is certainly not impossible.

Originally Posted by rnadeau

I have made the change to this script to downlaod all files which is about 797meg. I use sjq to run it daily. It will fail every time after the first time it is run. If I delete the folder , it will run properly and then fail every time again till I delete the folder it creates. I have looked at teh log and it seems to be a java memory error. I don't know how to fix this short of creating a file that deletes the folder that is created and redownloading all 797 meg of files, which I don't what to do if at all possible. I am including the log file, if someone could help with this issue, it would be much appreciated.

In the script that you're running, look for this text:

DigestUtils.md5Hex(

Show me the whole line that contains that text.

There is a memory issue where the original version of the script used an older version of DigestUtils that could only compute the md5 sum of files by first loading the entire file into memory. On very large files, which exist when you start downloading the entire repository, you would run out of memory. I made a post about that in this thread somewhere and the fix is to upgrade commons-codec to the latest version and use the updated md5Hex() method that computes from an IO stream instead of from memory.

You avoid this error when deleting the entire contents of the downloads folder because in that case, the script never has to compute the md5 of existing files to compare because there are no existing files.

Originally Posted by osx-addict

Stupid question -- coming in a bit late in the discussion.. What is the goal of the plugin -- ? Is it to just mirror a remote site and snarf up whatever it finds? Any particular reason it's written in Java? Seems like another scripting language would be better suited -- any reason it's being run on Windows? Can 'the script' be run on Unix (e.g. Linux, OSX, etc) if there's no actual requirement to run on Windows?

Goal: Provide a mirror of the repository plugin in case Sage ever shuts down their repository hosting server.

Why Java (Groovy, actually)? Because I run the script from SJQv4, which is a Java based environment.

Which OS does the script run on? Anywhere you have a JVM installed with the required prereq jars. The SJQv4 environment provides such an env.

Windows only? Nope. I run it on Linux; most run it on Windows (only because Windows is the dominant OS for Sage users).

Runs on Unix? Yup, well Linux, at least. Any Unix flavour with a JRE + the prereq jars should be fine. YMMV.

Originally Posted by darcilicious

I don't know the particulars of the script itself but one reason for it being done in Java might be is that Java is cross-plaform compatible -- it should run on Linxu, OS X, and Windows....

Correct, but I didn't write it in groovy because of cross-platform concerns, though that's a bonus of using groovy/java in this case. I went with groovy because I planned to execute it via SJQ crontab, which, of course, is a java/groovy environment.

I actually ported the script to PHP for someone to run on a hosted server out on the internet somewhere. I don't even remember for whom it was for, but I do know I still get the email notifications everytime it updates a zip file from the sagetv servers.

[osx-addict]

Cool.. The only reason I asked is that I've written a number of Korn shell scripts that do stuff similar to that and it would seem fairly straightforward to write one that uses wget and friends to download the driver tree and fire it off with a nightly/weekly cron job -- particularly if the java code is occasionally being difficult.. Anyway, just throwing that out there..

[Slugger]

Originally Posted by osx-addict

Cool.. The only reason I asked is that I've written a number of Korn shell scripts that do stuff similar to that and it would seem fairly straightforward to write one that uses wget and friends to download the driver tree and fire it off with a nightly/weekly cron job -- particularly if the java code is occasionally being difficult.. Anyway, just throwing that out there..

Sure... shell, perl, php... pick your poison. If you're running it outside of Sage altogether then whatever you want will suffice. But I'm not sure if I want to be processing xml in shell script. Perl, maybe, but perl is so old school, I'd probably go python before perl these days, but I'll stop there before a script language flame war ensues!

The java code is being difficult likely because of bugs in the code and not Java itself. If I get the rest of the details then I can probably fix the code rather easily. But, definitely, you could write a similar script in some other language and run it daily/weekly/etc. from cron on linux and get the same result.

[rnadeau]

if(!TEST_MODE && (!localFile.exists() || DigestUtils.md5Hex((localStream = new FileInputStream(localFile)).getBytes()).toLowerCase() != it.MD5.text().toLowerCase())) {

[Slugger]

Originally Posted by rnadeau

if(!TEST_MODE && (!localFile.exists() || DigestUtils.md5Hex((localStream = new FileInputStream(localFile)).getBytes()).toLowerCase() != it.MD5.text().toLowerCase())) {

Yeah, that's the old version of the script. To avoid the OOM errors, you need to upgrade commons-codec jar to the latest version and change the line to read:

if(!TEST_MODE && (!localFile.exists() || DigestUtils.md5Hex((localStream = new FileInputStream(localFile))).toLowerCase() != it.MD5.text().toLowerCase()))

If you're using the Sage plugin version of the task client to run this script then you're kind of stuck unless you want to start manipulating the plugin repo manually on your system, which I don't recommend. In that case, you could grab an md5sum generator tool from the web and use it in place of DigestUtils. Alternatively, you could nudge stuckless to upgrade the common-codec plugin to the latest version then mod the script as outlined above.

[osx-addict]

Oooh.. XML.. Yumm.. I've been working lately with XML in Smalltalk (GST) -- works great if you like that OO environment (which I do!)