GroovyPlugins: A scripting platform for plugins (design discussion)

[Slugger]

I talked the other day about writing a plugin entirely in Groovy. Groovy plugins are nice, but they still have to be compiled into bytecode in order for the core to load them.

Today, I'm ready to announce the bigger picture: GroovyPlugins. This will provide the ability to write SageTV plugins in Groovy script without the need to compile them down into bytecode! Furthermore, if properly written, such plugins can be modified while they're running and the code changes are immediately picked up and deployed. Even if you choose not to put the effort into supporting dynamic code mods, most code changes to these Groovy script based plugins will not require a SageTV restart!

I'm running a build on my test server and it's actually running way better than I ever expected, but it's going to be a little while before I'm comfortable putting something out there for people. However, I've started my documentation (albeit a little thin right now) and I'm looking for two things:

1) Is there any interest in such a platform?
2) If you are interested, please read the linked to docs above for a few more details and let me know if there are any features/functionality you'd like to see offered?

[wayner]

I am not a developer but I must say that this is a great concept. The one good thing in Sage6 was the Jetty apps that you wrote, as well as jreichen's, etc were so easy to implement since you didn't have to restart anything. I like the plugin manager in Sage7 but it seems like 99% of plugins require a Sage restart which is a pain in the ass - I rarely have the luxury of being able to shut down Sage until the family are in bed.

[stuckless]

Originally Posted by wayner

I am not a developer but I must say that this is a great concept. The one good thing in Sage6 was the Jetty apps that you wrote, as well as jreichen's, etc were so easy to implement since you didn't have to restart anything. I like the plugin manager in Sage7 but it seems like 99% of plugins require a Sage restart which is a pain in the ass - I rarely have the luxury of being able to shut down Sage until the family are in bed.

The side effect is that if you don't restart, and you update often, then you'll eventually run out of memeory

I agree that installing plugins and restarting is not convenient, which is why I only install update plugins once a week, and then I only do it when the system is not in use. It's like applying updates to Windows or installing software on Windows... you know it's going to require a restart, so you don't do it while you absolutely need the machine to remain up.

Keep in mind... Just because a plugin has been updated... doesn't mean that you need the update. If you are running fine, without issues, then I'd probably not install a plugin update until your scheduled update cycle.

[tmiranda]

Slugger,

I think this is a very interesting idea. Saving me from restarts while I am developing plugins would be welcome. My only request is that it's Linux friendly. Let me know if you want testers.

Tom

[Slugger]

Originally Posted by stuckless

The side effect is that if you don't restart, and you update often, then you'll eventually run out of memeory

True only if the GroovyScriptEngine (see section: The GroovyScriptEngine about 2/3 down the page) doesn't clean itself up properly, which is possible. Honestly, I don't know exactly what's going on under the hood with this thing, but I do know it works. You run your groovy script app within one of these objects and change a groovy script file while it's running, it automatically updates your code. I know it has its own class loader to handle parsing/compiling groovy scripts into bytecode. Whether that class loader plays nice, I don't know. Time will tell. As I say on my wiki, changes to the bootstrap class will require a restart of the plugin, which will recycle the GroovyScriptEngine object. And, of course, changes to jars in the classpath will require a SageTV restart. I suspect, but, again, I have no numbers to back this up, that for the common use case a user is more likely to upgrade a regular plugin requiring a SageTV restart long before one of these plugins exhausts available memory.

I agree that installing plugins and restarting is not convenient, which is why I only install update plugins once a week, and then I only do it when the system is not in use. It's like applying updates to Windows or installing software on Windows... you know it's going to require a restart, so you don't do it while you absolutely need the machine to remain up.

One of my motivations for pursuing this is the ability to develop plugins directly in the Sage JVM and make code changes that are immediate and don't require a Sage restart. The added bonus is that users will be able to upgrade such plugins without the need to restart SageTV.

The other bonus is that the barrier to entry for plugin development is lowered. More people should be able to write plugins in Groovy as opposed to Java. Especially when they start with your AbstractPlugin as the base.

Keep in mind... Just because a plugin has been updated... doesn't mean that you need the update. If you are running fine, without issues, then I'd probably not install a plugin update until your scheduled update cycle.

Absolutely, but then there are times when I've needed to make a critical fix for something and couldn't apply it on production because recordings were scheduled for the next 6 hours. Though not totally necessary, this "hot deploy" is at least another possibility.

Ultimately, the motivation here is to provide two key elements for myself:

1) A platform to write plugins solely in Groovy script, without the need to compile to bytecode for the Sage core to consume; I can turnover a plugin in Groovy way faster than an equivalent effort in Java.

2) Provide a "hot deploy" environment that can be used to modify plugins as they're running in the same JVM as Sage. This, for example, allows me to use GKusnick's strongly typed API, if so desired, in an env that doesn't require a constant rebuild/redeploy/restart of SageTV on every code change. Also allows access to parts of the API that just aren't available via the remote APIs (via RMI).

[Slugger]

Originally Posted by tmiranda

Slugger,

I think this is a very interesting idea. Saving me from restarts while I am developing plugins would be welcome. My only request is that it's Linux friendly. Let me know if you want testers.

Tom

My test server is Linux so it'll definitely work on Linux. If anything, it may not be Windows friendly at the moment. Obviously, if that's the case it will be fixed long before a build is made public. Especially because my production server is WinXP.

[tmiranda]

Originally Posted by Slugger

My test server is Linux so it'll definitely work on Linux. If anything, it may not be Windows friendly at the moment. Obviously, if that's the case it will be fixed long before a build is made public. Especially because my production server is WinXP.

I've got the same setup: Ubuntu for my test server, WinXP Home for my production server.

[stuckless]

Slugger, while isn't aimed at myself... I did notice in the docs that you are going to require a license for this... I'm just curious.. .is this a developer license, ie a developer can write X number of plugins or does this mean a user that installs a plugin that is using your groovy scripting may be required to buy a license. I hope it's first and not the latter... if it's the latter, then it'll be like playing Plugin Roulette

Also, about a couple years back, I looked at adding Scripting (Javascript at the time) to the sagex apis for the purpose of allowing STV development without needing a restart. This turned out to be more a pipe dream for me, since which i could write code in script, there was no way to actually invoke that code from the STV This was due to the fact that the sagetv classloader could not see anything in the scripting classloader. There were work arounds... ie, I could use factories and interfaces, but that still required a restart whenever you added a new api (ie, updated the interface), and it required that you package part of your code as a jar, etc. You could also use messaging or a "generic" api, but that was ugly. In the end, I abandoned the scripting idea, but i'm curious if you've solved these issues for Groovy?

[Slugger]

Originally Posted by stuckless

Slugger, while isn't aimed at myself... I did notice in the docs that you are going to require a license for this... I'm just curious.. .is this a developer license, ie a developer can write X number of plugins or does this mean a user that installs a plugin that is using your groovy scripting may be required to buy a license. I hope it's first and not the latter... if it's the latter, then it'll be like playing Plugin Roulette

Yeah... I have no idea. As it stands, the licensing requirement isn't even applied to the plugin yet. And it may not ever end up being applied. If it were, it would follow the latter scenario you describe. To do it the former way (dev licensing) would require me to write a new license server, which is not in the plans. Users would be able to configure which plugins run and which don't so if a limit were imposed, the user could choose which ones start and which ones don't.

I'm leaning towards not applying the license to the platform plugin. Doing so might possibly require(**) a user to donate for a license to run a plugin from another dev who doesn't want donations applied. I don't like that scenario at all and it would also most likely discourage devs from trying Groovy as a plugin platform. Most likely, there will no license required for this platform plugin.

** I hate saying "require's a license" because it really isn't the case. Any and all plugins that are restricted functionally without a sagetv-addons license can be fully unlocked without a license by anyone willing to download and build the source code themselves. It's all open source, it's all openly available.

Also, about a couple years back, I looked at adding Scripting (Javascript at the time) to the sagex apis for the purpose of allowing STV development without needing a restart. This turned out to be more a pipe dream for me, since which i could write code in script, there was no way to actually invoke that code from the STV This was due to the fact that the sagetv classloader could not see anything in the scripting classloader. There were work arounds... ie, I could use factories and interfaces, but that still required a restart whenever you added a new api (ie, updated the interface), and it required that you package part of your code as a jar, etc. You could also use messaging or a "generic" api, but that was ugly. In the end, I abandoned the scripting idea, but i'm curious if you've solved these issues for Groovy?

Testing shows that these "GroovyPlugins" have access to everything in the Sage classpath plus anything in their application hierarchy (i.e. the various .groovy files that make up the app). Definitely, the core does not have access to any classes defined in the .groovy files. If one of these plugins wanted to provide public resources (i.e. classes) for others (i.e. the core, other plugins) to consume then they'd have to be compiled to bytecode, packaged into a jar and installed into the Sage classpath ahead of time (like any other plugin).

Alternatively, one could actually dynamically compile these .groovy files at runtime and add them to the Sage classpath. (* See EDIT below) I won't be doing that, but a plugin could do it. My intent is that each plugin is a standalone thread of execution and common/public classes are precompiled and installed in the JARs folder, if necessary.

EDIT: Don't quote me on that. I speak only in theory. I really haven't investigated that claim too thoroughly, but in theory it should be possible. In practice, however, I have no intention of trying to implement it in the slightest way.

[stuckless]

That's been my experience with using scripting... ie, the script has access to everything in the sagetv classpath, but the STV or other plugins can't access the script.

[GKusnick]

Originally Posted by stuckless

Keep in mind... Just because a plugin has been updated... doesn't mean that you need the update.

I would add that just because a plugin has been updated doesn't mean the update works better than the version you have. Any update can potentially destabilize your system, so if you have recordings in progress that you don't want to interrupt, you really shouldn't be downloading plugin updates anyway until the recordings are done and you have time to test the updates. So I don't see hot deployment as a big boon to end users (at least not to sensibly cautious end users). Its primary value (in my view) is to speed up the plugin development cycle.

[GKusnick]

Originally Posted by Slugger

Definitely, the core does not have access to any classes defined in the .groovy files. If one of these plugins wanted to provide public resources (i.e. classes) for others (i.e. the core, other plugins) to consume then they'd have to be compiled to bytecode, packaged into a jar and installed into the Sage classpath ahead of time (like any other plugin).

That's too bad, since it seems to me that one appealing use for this would be to define function libraries for use by STVIs. Since STVI code can be edited and tested interactively, without a compile/reload cycle, if would be handy if the same could be true for library functions called from STVI code.

[Slugger]

Originally Posted by GKusnick

That's too bad, since it seems to me that one appealing use for this would be to define function libraries for use by STVIs. Since STVI code can be edited and tested interactively, without a compile/reload cycle, if would be handy if the same could be true for library functions called from STVI code.

To support this, the Sage classloader would have to support the recycling/replacing of classes at runtime, much like a J2EE container does. Even if it did, some kind of app restart (or more like an app recycle) would probably be necessary.

However, if one were to write a proxy server to dispatch the function calls into the .groovy files then you could achieve this such that a restart would only be required if the dispatcher itself was modified. But then we're talking about layers on top of layers.

It's interesting you question the desire of end users to have plugins that update without a required restart. I actually got thinking about that this afternoon while I was out. I kind of came to the same conclusion. And as I review what I've created so far, it seems I've just added a level of indirection/abstraction to the plugin model that will only cause headaches if it were found to be buggy in anyway.

The benefit of having the ability to redeploy without a compile/restart is good, but to support that feature requires writing your plugin in such away that the .groovy files can be monitored and reloaded at runtime and those .groovy files aren't directly referenced in your SageTVPlugin implementation. There's just enough indirection required in the approach that anything more than a simple/basic plugin would almost become tied to the platform, which I thought was ok, but now I realize that I probably don't want to support the platform.

Eclipse + Remote APIs provides a sufficient enough environment for most of what I do and what it doesn't provide I can probably live without in the dev env.

Think I might shelf this idea for awhile and come back to it later, unless there's an overwhelming desire from other devs in the community. The actual launcher of the plugins is written and working. I'm at the point where I'm trying to massage the plugin config options to dynamically load the plugin config options for the GroovyPlugins. Basically, it is a sort of plugin manager on it's own where you install GroovyPlugins into a special dir and my plugin finds them, starts them, stops them and updates them as needed. The more I realize just what I'm creating, the more I realize I don't really want to create a separate plugin manager. But the PoC, which is running on my test server, proves that it's at least possible to write Sage plugins in a way that allows for updating without a Sage restart.

[GKusnick]

Well, I wasn't trying to talk you out of it, just pointing out that the direct benefit of hot deployment to end users is marginal at best. However there could be considerable indirect benefit from a wide variety of lightweight plugins enabled by this sort of technology.

[Slugger]

Originally Posted by GKusnick

Well, I wasn't trying to talk you out of it, just pointing out that the direct benefit of hot deployment to end users is marginal at best. However there could be considerable indirect benefit from a wide variety of lightweight plugins enabled by this sort of technology.

You didn't talk me out of it, I was already there before I got home earlier and read your comments. I think I was more interested in seeing if it could be done than anything else. Once I got a PoC running I was excited. Then when I walked away from it for a couple of days, did that write up about it and thought some more about what it is I've actually created, I realized I've created a separate plugin manager ecosystem that would be reliant on this Groovy platform plugin of mine in order to operate (assuming devs followed the "best practices" guide to allow for dynamic code changes during runtime).

And before seeing your thoughts, I already thought that most users don't really care if they're plugins update "hot". Theoretically, plugin updates are so infrequent, relatively speaking, that the burden of a required restart is probably only really felt by developers during development. If we assume that's true then the restrictions/limitations (and work still to be done) to support plugin development in this env isn't really worth it when Eclipse + remote APIs pretty much gives the same thing (in a much more stable, proven env).

Hot deploy Groovy plugins was actually really easy compared to where I am right now with the code: How do I trick my plugin's config STV screen into loading up each GroovyPlugin's config screen on demand? Because these GroovyPlugins people would write would not be installed within the Sage plugin manager. Instead, if you were to put them in the Sage repository you'd have to label it a General plugin with no implementation class so that Sage itself doesn't ever try to start it, but instead leave that management to my platform. Well, originally I was going to just say manually config these plugins via props files, etc. I didn't like that so then I started down the road of manipulating the parent plugin (mine) into actually reading and displaying each GroovyPlugin's config opts from the bootstrap class. I was able to find them, but trying to get the config screen to dynamically load was becoming a headache. That's when I realized I was actually getting myself into the "Another Plugin Manager" business, which I don't want to do. This, combined with the fact that most users probably don't care about hot deploy anyways, is what has lead me to probably shelf this for a little while.