Can anyone host a mirror of SageTV hosted plugins?

[jptheripper]

Originally Posted by KJake

If this actually becomes necessary, I have a linux server with plenty of bandwidth and space. Notice the upload speed (aka how fast people can download from me)...

you suck

[nyplayer]

Thanks Slugger for the info I got it working.... but it appears that plugin creators have files stored all over the place... so if they leave or decide to move on we have to download all their files wherever they may be. So I guess I will be limiting the plugins I install for now.


ex
<Location>http://www.seanet.com/~kusnick/sage/plugins/GKusnickDARSwitcher-STVI-1.4.4.zip</Location>

[Slugger]

Originally Posted by nyplayer

Thanks Slugger for the info I got it working.... but it appears that plugin creators have files stored all over the place... so if they leave or decide to move on we have to download all their files wherever they may be. So I guess I will be limiting the plugins I install for now.


ex
<Location>http://www.seanet.com/~kusnick/sage/plugins/GKusnickDARSwitcher-STVI-1.4.4.zip</Location>

If you want to mirror every plugin file then it's just a matter of changing this:

// Links matching this regex will be mirrored
def regex = /^http:\/\/download.sage/

to this:

// Links matching this regex will be mirrored
def regex = /.*/

The Sage hosted downloads are 36MB or so. Everything is probably on the order of 100-200MB (but that's just a guess). Of course the script only downloads files that match the regex and haven't been downloaded before/have a different MD5 checksum (i.e. they've been updated since last downloaded) so after the initial sync, you wouldn't be downloading everything every time you ran it. Note, however, that if new versions of a plugin are submitted with different file names then this script keeps the old file along side the new one (meaning your mirror can grow in size as new files are added since old ones aren't deleted).

[nyplayer]

Thanks,

What I want to do is take a snapshot of my current system and all the plugins as it stands today. Have plenty of space so really not a problem.

OOPS error... First few worked died on this one.

ID: twit.tv
Location: http://forums.sagetv.com/forums/down...=file&id=389&a
ct=down
Download: C:\Inetpub\wwwroot\repository\twit.tv\downloads.php?do=file&id
=389&act=down
MD5: b0f5ed7e434fc6d90c5354f7262c3feb
Downloading... Caught: java.io.FileNotFoundException: C:\Inetpub\wwwroot
\repository\twit.tv\downloads.php?do=file&id=389&act=down (The filename, directo
ry name, or volume label syntax is incorrect)
at mirror_repo$_run_closure2_closure5.doCall(mirror_repo.groovy:48)
at mirror_repo$_run_closure2.doCall(mirror_repo.groovy:38)
at mirror_repo.run(mirror_repo.groovy:36)

C:\Program Files\Groovy\groovy-1.8.0\bin>

[Slugger]

Originally Posted by nyplayer

Thanks,

What I want to do is take a snapshot of my current system and all the plugins as it stands today. Have plenty of space so really not a problem.

OOPS error... First few worked died on this one.

ID: twit.tv
Location: http://forums.sagetv.com/forums/down...=file&id=389&a
ct=down
Download: C:\Inetpub\wwwroot\repository\twit.tv\downloads.php?do=file&id
=389&act=down
MD5: b0f5ed7e434fc6d90c5354f7262c3feb
Downloading... Caught: java.io.FileNotFoundException: C:\Inetpub\wwwroot
\repository\twit.tv\downloads.php?do=file&id=389&act=down (The filename, directo
ry name, or volume label syntax is incorrect)
at mirror_repo$_run_closure2_closure5.doCall(mirror_repo.groovy:48)
at mirror_repo$_run_closure2.doCall(mirror_repo.groovy:38)
at mirror_repo.run(mirror_repo.groovy:36)

C:\Program Files\Groovy\groovy-1.8.0\bin>

Hmm... didn't know the Sage core would follow redirects for file links. They should disallow that. No fix coming for that this weekend. It's Canada Day north of the 49th and our holiday is tomorrow instead of Monday (but I get Fri/Mon anyway ) so I'll be honest, if it takes more than 2 minutes, it ain't getting done this weekend.

If you're up for a challenge, the quick and dirty fix is to identify invalid chars in the file name (such as '?'), assume it's a redirect link, hash the file name into something valid and use that as the file name instead. The "full, proper" fix would be to mod the code to follow the redirect, get the actual file name, and carry on. When I fix it, I'll be doing the former. Either fix, as you can see, is more than two mins of my time so... maybe Monday. And honestly, you might want to post a nudge reply here to remind me about this, if you're still needing the fix.

Happy Canada Day!

[Slugger]

Just as I sat down to watch a movie, the 2 min fix came to me, so here it is...

http://sagetv-addons.googlecode.com/...or_repo.groovy

Happy Canada Day!!

[sleonard]

I'm not sure you want to use your Jetty webserver for mirroring your plugins. Jetty is itself a plugin, what if you need to do a complete reinstall and need Jetty running in order to install it?

S

[nyplayer]

Originally Posted by Slugger

Just as I sat down to watch a movie, the 2 min fix came to me, so here it is...

http://sagetv-addons.googlecode.com/...or_repo.groovy

Happy Canada Day!!

Error

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\Administrator>cd C:\Program Files\Groovy\groovy-1.8.0\
bin

C:\Program Files\Groovy\groovy-1.8.0\bin>groovy mirror_repo.groovy
ID: myjavatools-lib
        Location: http://download.sage.tv/plugins/Slugger/mjlib-5.0.zip
        Download: C:\Inetpub\wwwroot\repository\myjavatools-lib\mjlib-5.0.zip
        MD5: a4301478a82e84a066656c34c37ab3bf
        Latest version already downloaded!
ID: ant
        Location: http://download.sage.tv/plugins/Slugger/ant-1.8.2.zip
        Download: C:\Inetpub\wwwroot\repository\ant\ant-1.8.2.zip
        MD5: b8dcebcc9dd77d09e6aed7501281fc17
        Latest version already downloaded!
ID: last.fm-bindings
        Location: http://sagetv-scrobbler.googlecode.com/files/last.fm-bindings-
2009.07.03.zip
        Download: C:\Inetpub\wwwroot\repository\last.fm-bindings\last.fm-binding
s-2009.07.03.zip
        MD5: 16b6e5bf0669ea791851adafce5d0a90
        Latest version already downloaded!
ID: EncoderNames
        Location: http://download.sagetv.com/plugins/JREkiwi/EncoderNames/Encode
r_Names_0.8.zip
        Download: C:\Inetpub\wwwroot\repository\EncoderNames\Encoder_Names_0.8.z
ip
        MD5: FB1B27F4EF8B92166618CF3ECDFB6373
        Latest version already downloaded!
ID: twit.tv
        Location: http://forums.sagetv.com/forums/downloads.php?do=file&id=389&a
ct=down
        WARNING: File name appears to be a redirect link!
        Download: 64ce319153de5e67a78affe5cffd512b.zip
        MD5: b0f5ed7e434fc6d90c5354f7262c3feb
Caught: groovy.lang.MissingMethodException: No signature of method: java.lang.St
ring.exists() is applicable for argument types: () values: []
Possible solutions: wait(), expand(), execute(), toList(), next(), equals(java.l
ang.Object)
        at mirror_repo$_run_closure2_closure5.doCall(mirror_repo.groovy:50)
        at mirror_repo$_run_closure2.doCall(mirror_repo.groovy:40)
        at mirror_repo.run(mirror_repo.groovy:38)

C:\Program Files\Groovy\groovy-1.8.0\bin>

[nyplayer]

Originally Posted by sleonard

I'm not sure you want to use your Jetty webserver for mirroring your plugins. Jetty is itself a plugin, what if you need to do a complete reinstall and need Jetty running in order to install it?

S

I used the Built in windows server worked fine.

[Slugger]

Originally Posted by nyplayer

Error

Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.

C:\Documents and Settings\Administrator>cd C:\Program Files\Groovy\groovy-1.8.0\
bin

C:\Program Files\Groovy\groovy-1.8.0\bin>groovy mirror_repo.groovy
ID: myjavatools-lib
        Location: http://download.sage.tv/plugins/Slugger/mjlib-5.0.zip
        Download: C:\Inetpub\wwwroot\repository\myjavatools-lib\mjlib-5.0.zip
        MD5: a4301478a82e84a066656c34c37ab3bf
        Latest version already downloaded!
ID: ant
        Location: http://download.sage.tv/plugins/Slugger/ant-1.8.2.zip
        Download: C:\Inetpub\wwwroot\repository\ant\ant-1.8.2.zip
        MD5: b8dcebcc9dd77d09e6aed7501281fc17
        Latest version already downloaded!
ID: last.fm-bindings
        Location: http://sagetv-scrobbler.googlecode.com/files/last.fm-bindings-
2009.07.03.zip
        Download: C:\Inetpub\wwwroot\repository\last.fm-bindings\last.fm-binding
s-2009.07.03.zip
        MD5: 16b6e5bf0669ea791851adafce5d0a90
        Latest version already downloaded!
ID: EncoderNames
        Location: http://download.sagetv.com/plugins/JREkiwi/EncoderNames/Encode
r_Names_0.8.zip
        Download: C:\Inetpub\wwwroot\repository\EncoderNames\Encoder_Names_0.8.z
ip
        MD5: FB1B27F4EF8B92166618CF3ECDFB6373
        Latest version already downloaded!
ID: twit.tv
        Location: http://forums.sagetv.com/forums/downloads.php?do=file&id=389&a
ct=down
        WARNING: File name appears to be a redirect link!
        Download: 64ce319153de5e67a78affe5cffd512b.zip
        MD5: b0f5ed7e434fc6d90c5354f7262c3feb
Caught: groovy.lang.MissingMethodException: No signature of method: java.lang.St
ring.exists() is applicable for argument types: () values: []
Possible solutions: wait(), expand(), execute(), toList(), next(), equals(java.l
ang.Object)
        at mirror_repo$_run_closure2_closure5.doCall(mirror_repo.groovy:50)
        at mirror_repo$_run_closure2.doCall(mirror_repo.groovy:40)
        at mirror_repo.run(mirror_repo.groovy:38)

C:\Program Files\Groovy\groovy-1.8.0\bin>

Fixed in svn. I'm heading for the river very shortly, if you test and find more errors before 11:15, I might be able to fix them. After that, won't be any more coding today, which, trust me, will be a good thing for all SageTV users!

[nyplayer]

Slugger,

Sorry it failed again. No Problem if you cannot fix it.

C:\Program Files\Groovy\groovy-1.8.0\bin>groovy mirror_repo.groovy
ID: myjavatools-lib
        Location: http://download.sage.tv/plugins/Slugger/mjlib-5.0.zip
        Download: C:\Inetpub\wwwroot\repository\myjavatools-lib\mjlib-5.0.zip
        MD5: a4301478a82e84a066656c34c37ab3bf
        Latest version already downloaded!
ID: ant
        Location: http://download.sage.tv/plugins/Slugger/ant-1.8.2.zip
        Download: C:\Inetpub\wwwroot\repository\ant\ant-1.8.2.zip
        MD5: b8dcebcc9dd77d09e6aed7501281fc17
        Latest version already downloaded!
ID: last.fm-bindings
        Location: http://sagetv-scrobbler.googlecode.com/files/last.fm-bindings-
2009.07.03.zip
        Download: C:\Inetpub\wwwroot\repository\last.fm-bindings\last.fm-binding
s-2009.07.03.zip
        MD5: 16b6e5bf0669ea791851adafce5d0a90
        Latest version already downloaded!
ID: EncoderNames
        Location: http://download.sagetv.com/plugins/JREkiwi/EncoderNames/Encode
r_Names_0.8.zip
        Download: C:\Inetpub\wwwroot\repository\EncoderNames\Encoder_Names_0.8.z
ip
        MD5: FB1B27F4EF8B92166618CF3ECDFB6373
        Latest version already downloaded!
ID: twit.tv
        Location: http://forums.sagetv.com/forums/downloads.php?do=file&id=389&a
ct=down
        WARNING: File name appears to be a redirect link!
        Download: 64ce319153de5e67a78affe5cffd512b.zip
        MD5: b0f5ed7e434fc6d90c5354f7262c3feb
        Downloading... Caught: java.lang.NullPointerException: Cannot invoke met
hod mkdirs() on null object
        at mirror_repo$_run_closure2_closure5.doCall(mirror_repo.groovy:52)
        at mirror_repo$_run_closure2.doCall(mirror_repo.groovy:40)
        at mirror_repo.run(mirror_repo.groovy:38)

C:\Program Files\Groovy\groovy-1.8.0\bin>

[Slugger]

Originally Posted by nyplayer

Slugger,

Sorry it failed again. No Problem if you cannot fix it.

C:\Program Files\Groovy\groovy-1.8.0\bin>groovy mirror_repo.groovy
ID: myjavatools-lib
        Location: http://download.sage.tv/plugins/Slugger/mjlib-5.0.zip
        Download: C:\Inetpub\wwwroot\repository\myjavatools-lib\mjlib-5.0.zip
        MD5: a4301478a82e84a066656c34c37ab3bf
        Latest version already downloaded!
ID: ant
        Location: http://download.sage.tv/plugins/Slugger/ant-1.8.2.zip
        Download: C:\Inetpub\wwwroot\repository\ant\ant-1.8.2.zip
        MD5: b8dcebcc9dd77d09e6aed7501281fc17
        Latest version already downloaded!
ID: last.fm-bindings
        Location: http://sagetv-scrobbler.googlecode.com/files/last.fm-bindings-
2009.07.03.zip
        Download: C:\Inetpub\wwwroot\repository\last.fm-bindings\last.fm-binding
s-2009.07.03.zip
        MD5: 16b6e5bf0669ea791851adafce5d0a90
        Latest version already downloaded!
ID: EncoderNames
        Location: http://download.sagetv.com/plugins/JREkiwi/EncoderNames/Encode
r_Names_0.8.zip
        Download: C:\Inetpub\wwwroot\repository\EncoderNames\Encoder_Names_0.8.z
ip
        MD5: FB1B27F4EF8B92166618CF3ECDFB6373
        Latest version already downloaded!
ID: twit.tv
        Location: http://forums.sagetv.com/forums/downloads.php?do=file&id=389&a
ct=down
        WARNING: File name appears to be a redirect link!
        Download: 64ce319153de5e67a78affe5cffd512b.zip
        MD5: b0f5ed7e434fc6d90c5354f7262c3feb
        Downloading... Caught: java.lang.NullPointerException: Cannot invoke met
hod mkdirs() on null object
        at mirror_repo$_run_closure2_closure5.doCall(mirror_repo.groovy:52)
        at mirror_repo$_run_closure2.doCall(mirror_repo.groovy:40)
        at mirror_repo.run(mirror_repo.groovy:38)

C:\Program Files\Groovy\groovy-1.8.0\bin>

That problem is fixed in svn, with one foot out the door. I'm done for the day. Hopefully that fixes it all, if not, post the next error and I'll tackle it tomorrow.

[nyplayer]

Thanks much appreciated enjoy.

It worked was able to backup everything

[hugenbdd]

Originally Posted by nyplayer

Thanks much appreciated enjoy.

It worked was able to backup everything

Hi nyplayer

How large was the full download? Was it the 200 megs as predicted?

Thanks
Dave

[nyplayer]

Originally Posted by hugenbdd

Hi nyplayer

How large was the full download? Was it the 200 megs as predicted?

Thanks
Dave

close to 500 meg.

[hugenbdd]

Originally Posted by Slugger

That problem is fixed in svn, with one foot out the door. I'm done for the day. Hopefully that fixes it all, if not, post the next error and I'll tackle it tomorrow.

Anybody getting any errors when running the script with the /.*/ regex (everything)?

I'm getting this error below. I am running on windows, but the modified regex worked fine. It seems to be crashing on the diamond plug in.... not sure if it's a size issue with that particular plug in, or just a size issue there.

No error thrown when running on the normal regex.

Error.
------------------
ID: DiamondSTVi
	Location: http://sagediamond.googlecode.com/fi...amondB3.30.zip
	Download: C:\SagePlugins\DiamondSTVi\DiamondB3.30.zip
	MD5: bf157a7e2ee3607b4426c465e34c9520
Exception thrown
Jul 11, 2011 10:52:33 PM org.codehaus.groovy.runtime.StackTraceUtils sanitize

WARNING: Sanitizing stacktrace:

java.lang.OutOfMemoryError: Java heap space

	at java.util.Arrays.copyOf(Unknown Source)

	at java.io.ByteArrayOutputStream.write(Unknown Source)

	at org.codehaus.groovy.runtime.DefaultGroovyMethods.getBytes(DefaultGroovyMethods.java:12767)

	at org.codehaus.groovy.runtime.dgm$301.invoke(Unknown Source)

	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:271)

	at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)

	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112)

Thanks
Dave

[Slugger]

Yeah, it's a function of the size of the file. You'll have to increase the java heap for the Groovy JVM (and you'll have to head to the groovy web site for those details b/c I don't remember how to specify that on the command line).

Alternatively, you can upgrade the commons-codec jar to 1.4 or newer and change this line:

        if(!TEST_MODE && (!localFile.exists() || DigestUtils.md5Hex((localStream = new FileInputStream(localFile)).getBytes()).toLowerCase() != it.MD5.text().toLowerCase())) {

To this:

        if(!TEST_MODE && (!localFile.exists() || DigestUtils.md5Hex((localStream = new FileInputStream(localFile))).toLowerCase() != it.MD5.text().toLowerCase())) {

And that should also eliminate the OOM error (though untested because I don't have a recent enough version of commons-codec and can't upgrade it for other reasons).

[hugenbdd]

Originally Posted by Slugger

Yeah, it's a function of the size of the file. You'll have to increase the java heap for the Groovy JVM (and you'll have to head to the groovy web site for those details b/c I don't remember how to specify that on the command line).

Alternatively, you can upgrade the commons-codec jar to 1.4 or newer and change this line:

        if(!TEST_MODE && (!localFile.exists() || DigestUtils.md5Hex((localStream = new FileInputStream(localFile)).getBytes()).toLowerCase() != it.MD5.text().toLowerCase())) {

To this:

        if(!TEST_MODE && (!localFile.exists() || DigestUtils.md5Hex((localStream = new FileInputStream(localFile))).toLowerCase() != it.MD5.text().toLowerCase())) {

And that should also eliminate the OOM error (though untested because I don't have a recent enough version of commons-codec and can't upgrade it for other reasons).

Thanks
That worked,and I have gotten farther.

I did the following.
In the New System Variable dialog box, enter:
Variable name: JAVA_OPTS.

Variable value -Xms256m -Xmx512m


However, now I'm getting to a file that is no longer hosted. It's the phoenix-api. the location is returning a 404.

http://sagephoenix.googlecode.com/fi...api-2.3.35.zip

I tried playing around with a "try/catch" in java but haven't had much success in getting the response code. I will keep trying to see what I can come up with to make the script more robust.

------Error Below ------
ID: phoenix-api
Location: http://sagephoenix.googlecode.com/fi...api-2.3.35.zip
Download: C:\SagePlugins\phoenix-api\phoenix-api-2.3.35.zip
MD5: 1053d2cc00b7ba4fa157fb12643e3819
Downloading...
Exception thrown
Jul 12, 2011 10:08:29 AM org.codehaus.groovy.runtime.StackTraceUtils sanitize
WARNING: Sanitizing stacktrace:
java.io.FileNotFoundException: http://sagephoenix.googlecode.com/fi...api-2.3.35.zip
at sun.net.http://www.protocol.http.HttpURLConn...Stream(Unknown Source)
at java.net.URL.openStream(Unknown Source)
at java_net_URL$openStream.call(Unknown Source)


Thanks
Dave

[Slugger]

I added a try/catch block that should properly skip files that can't be downloaded. I just committed the update to svn.

[BobbyDing]

Hi Folks,
I discovered an issue with this I think (maybe it's just my setup). The first execution of the script creates a good repo.xml with the proper local links. Any subsequent executions where a new file is not downloaded ("latest version already downloaded") will leave the original web link in the repo.xml.

I don't know groovy or java, but a made a quick repair that seems to work. I copied and pasted the full "it.location" line to within the ELSE so that it does the "it.location" line for either case. So far all executions of the script have been creating the repo.xml with the proper local links even when no updates are downloaded.

I'm including a copy of the repo.groovy below.

Hopefully I didn't screw up something else.

Bobby