Discussion: Plugin architecture changes

[skiingwiz]

My thoughts:

Originally Posted by stuckless

jars added as dependencies not plugins

I like the idea of being able to pull dependencies right from Maven Central.

Originally Posted by stuckless

plugins get their own plugin directory
plugins jars loaded OUTSIDE of sagetv JARs

I worry that this is going to inevitably lead (as you've mentioned) to an OSGi-like system. Unless there is a compelling reason to do this, I'd vote no. I work in OSGi in my day-job and I don't think it's something we need here.

Originally Posted by stuckless

Simplified plugin manifest
Ability to publish using non-interactive means

I don't find the manifest to be overly complex, but I see both of these things as something that can be solved with an ant or gradle plugin. Using gradle, it would be easy to create a DSL for building the XML plugin manifest. (Which could leverage the dependency list that is already in the build.gradle file.)
These would be candidates for a sagetv-dev github repo.

Originally Posted by stuckless

Ability to turn off MD5 checksum

I don't think we should allow this. I like that the plugin manager has an idea that the download went correctly.

[GKusnick]

My $0.02 on some of these proposals:

Plugins get their own plugin directory:

They should also have a well-defined place to store their configuration settings. This could be a separate .properties file, or a subtree of Sage.properties, but in either case all of a plugin's settings should be in one place where they can be exported or deleted as a unit by the plugin manager. This should be enforced by the API so that calls to Get/SetProperty are transparently redirected to the appropriate plugin property space.

Multiple plugin repositories:

I don't think this needs to be in the core. It should be straightforward to extend the plugin UI to gather manifests from multiple sources and merge them on the fly into SageTVPluginsDev.xml. I've been doing this for years with dev builds of my own plugins.

Simplified plugin manifest:

Generating manifests doesn't have to be done manually. I have Ant tasks that generate manifests from XML fragments embedded in my build scripts. So I don't see a pressing need to simplify the manifest specification.

Ability to publish using non-interactive means:

This worries me, since it seems to invite abuse by spambots. My feeling is that any plugin release ought to require confirmation from a human being. Maybe things could be batched up better when releasing a system of interdependent plugins, so that one confirmation suffices for all of them.

Scriptable plugins:

Again, maybe I'm missing something, but I don't see why this needs to be in the core. If a scripting host can be implemented as a plugin, then script plugins just need to declare a dependency on that host. It would be the host's responsibility to notice when a script plugin has been loaded and take the appropriate steps to start it.

Backward compatibility:

If V9 can't run plugins built for V7, then I'm guessing a lot of plugins will be left behind, and a lot of useful functionality lost unless somebody adopts those orphaned plugins and ports them to V9. It also creates extra work for plugin devs who aren't particularly eager to issue new releases of all their V7 plugins.

[stuckless]

Originally Posted by skiingwiz

I worry that this is going to inevitably lead (as you've mentioned) to an OSGi-like system. Unless there is a compelling reason to do this, I'd vote no. I work in OSGi in my day-job and I don't think it's something we need here.

Yeah, I don't want a re-implementation of OSGi, either I think this is more of an organization thing, whereby, as Greg pointed out, that plugin properties, etc, could be stored there as well... and perhaps could be "picked up and moved" to another system, etc.

Originally Posted by skiingwiz

I don't find the manifest to be overly complex, but I see both of these things as something that can be solved with an ant or gradle plugin. Using gradle, it would be easy to create a DSL for building the XML plugin manifest. (Which could leverage the dependency list that is already in the build.gradle file.)

Agreed, I use ant tasks now (much like greg pointed out), but I like the gradle DSL approach as well. And to be honest, in reflection, that the manifest doesn't need to simplified, just the process of one creates one scratch, etc. For example, I was playing with automake lately, and I was quite impressed with the tools they have for autogenerating configure.am files form analyzing the sources, since creating a configure.am scratch is tedious. This is how I feel about manifests So perhaps a combination of ant/gradle and maybe even Annotations could help with this.... and none of this has to be in core.

Originally Posted by skiingwiz

I don't think we should allow this. I like that the plugin manager has an idea that the download went correctly.

I'm on the fence... for example every package manager that I've used, allows for an override at the user's descretion.

Originally Posted by GKusnick

They should also have a well-defined place to store their configuration settings. This could be a separate .properties file, or a subtree of Sage.properties, but in either case all of a plugin's settings should be in one place where they can be exported or deleted as a unit by the plugin manager. This should be enforced by the API so that calls to Get/SetProperty are transparently redirected to the appropriate plugin property space.

Agreed. I've add this to main post.

Originally Posted by GKusnick

Multiple plugin repositories:
I don't think this needs to be in the core. It should be straightforward to extend the plugin UI to gather manifests from multiple sources and merge them on the fly into SageTVPluginsDev.xml. I've been doing this for years with dev builds of my own plugins.

The core has to be modified to load SageTVPlugins.xml from another primary source anyways... with the existing sagetv source being an alternate. I don't think it's a stretch to modify the core plugin manager to look for the repos in multiple places (probably configured in the Sage.properties)

Originally Posted by GKusnick

Generating manifests doesn't have to be done manually. I have Ant tasks that generate manifests from XML fragments embedded in my build scripts. So I don't see a pressing need to simplify the manifest specification.

Originally Posted by GKusnick

Ability to publish using non-interactive means:
This worries me, since it seems to invite abuse by spambots. My feeling is that any plugin release ought to require confirmation from a human being. Maybe things could be batched up better when releasing a system of interdependent plugins, so that one confirmation suffices for all of them.

I'm not suggesting things can't be moderated, but, having a human push the button, is unnecessary. Much bigger repositories, such as Maven Central, exist, and work quite well, even without a person pushing the button.

Originally Posted by GKusnick

Scriptable plugins:
Again, maybe I'm missing something, but I don't see why this needs to be in the core. If a scripting host can be implemented as a plugin, then script plugins just need to declare a dependency on that host. It would be the host's responsibility to notice when a script plugin has been loaded and take the appropriate steps to start it.

This amounts to very few lines of code in the core system, and allows for scripts to be used as implementation classes. What you are suggesting is basically building another plugin manager, as a plugin, that can load scripts, and that I think is overkill, for such a minor change to the core. ie, the JVM already has support for loading and compiling scripts, this is just looking at the implementation class as defined in the manifest, and going, "oh, you're a script, lets load you into the JVM so that the rest of the system thinks your a class". All this does is facilitate it, and not in any way, include extra dependencies, etc. For example, if a plugin that uses a script is loaded, then it also must depend on the scripting engine in order to work. So, from a core point of view, if the script is loaded, and the dependency is not there, then it will fail, in the same way that if you create a plugin .class and depend on a jar that doesn't exist.

Originally Posted by GKusnick

Backward compatibility:
If V9 can't run plugins built for V7, then I'm guessing a lot of plugins will be left behind, and a lot of useful functionality lost unless somebody adopts those orphaned plugins and ports them to V9. It also creates extra work for plugin devs who aren't particularly eager to issue new releases of all their V7 plugins.

[/quote]

Yeah, my comment was more about do we care if the change the plugin system in V9 such that a V9 plugin is not compatible with V7. Not the other way around. An example being... if we introduce new manifest attributes or elements, those will no longer be valid for V7 (not sure if v7 uses a dtd or not), and as such, the developer will need to be sure they add the Min sage version and the plugin will not be compatible with V7. Do we care? I don't.

[jptheripper]

Just the thought. The SageMyMovies plugin is a very heavily used plugin based on the forum traffic.

Its author has abandoned it. MyMovies also has abandoned the version that supports it (4, now on 5 and cannot upgrade to 5 without breaking plugin).

So what do we do in this situation? The moviewall type interface is the primary reason I use Sagetv. With 2000+ movies, its essential.

[PLUCKYHD]

Originally Posted by jptheripper

Just the thought. The SageMyMovies plugin is a very heavily used plugin based on the forum traffic.

Its author has abandoned it. MyMovies also has abandoned the version that supports it (4, now on 5 and cannot upgrade to 5 without breaking plugin).

So what do we do in this situation? The moviewall type interface is the primary reason I use Sagetv. With 2000+ movies, its essential.

It is true I did abandon it but in all fairness MyMovies developers killed third party DB access with version 5 so even if I still developed on it there is no way it would ever work with 5.

As for alternates there is phoenix,diamond and gemstone all offer great views.

[jptheripper]

but but but.. .


i like your plugin

[Fuzzy]

Okay, so how about a discussion of near-term solutions to the plugin issue. Currently, the biggest hurdle I see with the plugin system and people moving to v7 is the lucene plugin needing to be deprecated only on v9 installs. I see two ways this can be fixed.
1. Remove lucene from the core and make it a 'mandatory' plugin, and replacing the current lucene plugin in the repository.
2. Make a new repository that v9 pulls down, and in that repository, remove all lucene dependencies.

#1 would keep the plugin manager pointing to the same sage.tv server to get its information, but would limit it's utility to the single lucene issue
#2 would provide a way to separate v9 plugins from v7 plugins, as well as moving the repository into community control. This would require a location to put the SageTVPlugins.xml (and SageTVPlugins.ver) file, in a place that the community could easily access it, as well as changing the plugin source location within CorePluginManager.java.

I think #2 may be the more forward thinking solution, and if the plugin source location is made to be a property (it is currently hard-coded in), would allow changing the source in the future, as well as perhaps adding the ability to parse out multiple repository locations from the property and have the software grab each and then parse them all together (as it currently does with SageTVPlugins.xml and SageTVPluginsDev.xml).

For starters, the xml can be hosted on the git. This would provide a level of version control that would help to avoid it becoming broken at some point, as well as move towards taking it out of Google control (of course, meaning it would have to be in a non-google fork).

[Slugger]

fwiw, I vote #2. I just envision massive headaches ahead if v7 & v9 plugins are forced to coexist in one repo.

Especially when you consider that the majority of devs are going to be writing for v9 anyway. If you duplicate the repo and separate them then v7 is v7 and those who decide to stay there don't have to worry about their plugins being updated with a MinVersion=9.0 manifest meaning they can't ever reinstall their v7 plugin from the repo again (at least not without some very bothersome SageTVPluginsDev.xml hacks).

One repo for v7, one for v9. To ease dev tedium, perhaps some additional tooling to make it possible to submit plugins for inclusion in both repos, just v7 or just v9 (but I don't really think that's going to be necessary -- how many devs plan on continuing dev on v7 and not v9? Very few, I'd imagine).

[Narflex]

Another option...just remove the newly added Lucene stuff from SageTV. All the prior code is still there (and if not, I can fix that)...Lucene doesn't buy all that much on PC systems because even the horrible linear search SageTV used to do executed pretty quickly. The ONLY thing Lucene is used for is when you're doing full text searches on the database (and also for matching for keyword Favorites).

If Lucene is going to cause a fracture of the plugins between V7/V9; then it's probably best to just kill it.

[Slugger]

I think the lucene issue just foreshadows other problems that can/will arise.

The big one being what I've already alluded to. Someone updates a plugin today and slaps a MinVersion=9.0 in the manifest. Well v7 users are done, they can never install the plugin again. I suspect this will be an issue in a year from now or maybe sooner depending how quickly dev revs up on the OSS version.

Off the top of my head, the Jetty plugin is in desperate need of an upgrade. Presumably whomever does that is going to move to Jetty 9.x (we're at 6.x today). Well that implies Java 7+. When the update adds a JRE MinVersion=1.7 to the manifest how many Sage 7 users still on Java 6 (or <shuttering> Java 5) does that prevent from being able to reinstall their v7 setup down the road?

To me, it just seems logical to create a new repo specifically for v9. Especially when v9 starts to introduce things that make it impossible for (some) plugins to maintain Sage 7 support going forward.

I'm assuming, of course, that there is a desire to keep Sage 7 owners operational long term. If that's not in the cards then forget everything I say with the understanding that when a dev decides their plugin update can only run on v9 then that automatically implies v7 users will never be able to reinstall it again.

[stuckless]

Originally Posted by Narflex

Another option...just remove the newly added Lucene stuff from SageTV. All the prior code is still there (and if not, I can fix that)...Lucene doesn't buy all that much on PC systems because even the horrible linear search SageTV used to do executed pretty quickly. The ONLY thing Lucene is used for is when you're doing full text searches on the database (and also for matching for keyword Favorites).

If Lucene is going to cause a fracture of the plugins between V7/V9; then it's probably best to just kill it.

I've actually gone through the effort of removing Lucene from phoenix, since it was only used in dvd profiler and mymovies lookups, and no one uses those.

I spent the last hour trying to re-host the files, since, I haven't re-submitted phoenix since google code died I'll likely get it out sometime this week.

But, even shipping a phoenix without the lucene dependency doesn't fix things, because if I recall, savetv doesn't remove a dependency when you update a plugin... so at the end of the day, removing lucene from phoenix does nothing to fix the problem

Maybe disabling lucene from the core is the easiest solution.

[skiingwiz]

Originally Posted by Fuzzy

2. Make a new repository that v9 pulls down, and in that repository, remove all lucene dependencies.

If we're going to do this, I think we ought to put some thought into changes that prevent this issue from happening between version 9 and10 (or any two versions really).

The complicated, dynamic way would probably involve the server keeping track of all submitted versions, rather than the latest (which it may already for all I know) and the client sending its version with the request so the server could send it a manifest that is compatible with the version.

The easy way would be maybe just one repository per major version of SageTV?

[Slugger]

Originally Posted by skiingwiz

The easy way would be maybe just one repository per major version of SageTV?

I think there are just too many gotchas to think about (including many we probably haven't even thought of yet) when the easy solution is just so easy in this case. New major version number, new isolated plugin repo. Especially when the added benefit is that we aren't touching the older version's repo and therefore can never mess up the older version's ability to reinstall plugins as needed. Plus you alleviate the need for devs to worry about if their new manifest is going to break users 1, 2, 3, etc. releases back.

[Fuzzy]

So what would be the best hosting and validating system to use for the SageTV9Plugins.xml?

[tmiranda]

Originally Posted by Slugger

I think the lucene issue just foreshadows other problems that can/will arise.

The big one being what I've already alluded to. Someone updates a plugin today and slaps a MinVersion=9.0 in the manifest. Well v7 users are done, they can never install the plugin again. I suspect this will be an issue in a year from now or maybe sooner depending how quickly dev revs up on the OSS version.

Off the top of my head, the Jetty plugin is in desperate need of an upgrade. Presumably whomever does that is going to move to Jetty 9.x (we're at 6.x today). Well that implies Java 7+. When the update adds a JRE MinVersion=1.7 to the manifest how many Sage 7 users still on Java 6 (or <shuttering> Java 5) does that prevent from being able to reinstall their v7 setup down the road?

To me, it just seems logical to create a new repo specifically for v9. Especially when v9 starts to introduce things that make it impossible for (some) plugins to maintain Sage 7 support going forward.

I'm assuming, of course, that there is a desire to keep Sage 7 owners operational long term. If that's not in the cards then forget everything I say with the understanding that when a dev decides their plugin update can only run on v9 then that automatically implies v7 users will never be able to reinstall it again.

I disagree. Many of my plugins (and others) will work in v7 or v9 and I do not want to maintain two manifests for separate repositories.

[stuckless]

Originally Posted by Fuzzy

So what would be the best hosting and validating system to use for the SageTV9Plugins.xml?

Hosting won't be much of an issue, but the "validating system" part might imply a server running somewhere, and I'm not sure who wants to foot the bill for that. Technically anybody, including myself, that has a server in their house, could do the "validation" part of this, and then git commit the new/updated plugin xml if it passes.

As for hosting, I did create an OpenSageTV organization on GitHub a few days ago.
https://github.com/OpenSageTV
Organizations are similar to users, except they can have multiple admins. Users can have specific roles, etc.

I think that using a separate plugin repo per sagetv major version is probably a good idea, but still doesn't fix Slugger's comment about when a developer decides to upgrade to Java 8 and set the Min java version to 8 whether it be saget 7 or 9. So do we start doing separate plugin repos per java version as well??

So, perhaps the bigger flaw here is the fact that SageTV only knows about a single version of a plugin (as skiingwiz pointed out), and not about the history of the plugin, where they can be multiple instances of the plugin based on jvm requirements, sage version requirements, etc. This obviously makes the server processing more complicated, but I'd favour that over forcing a developer to re-publish plugins to several different repositories. (For all I know the plugin system might already do some of this)

Something else to consider as well... If we used multiple repositories, could we have repositories "inherit" from another repository. So, S9 repo could inherit from S7, meaning, initially the S9 repo is basically blank. When a developer publishes a plugin, if their Min sage version is 9, then it gets added to the S9 repo, and S7 is unaffected. If the Min version is left out then it goes to the base S7 repo, and then it is picked up by S7 and S9 and the plugin is only published once. Still doesn't solve the Java version issue. And "inherit" in this sense could be a simple as the SageTVPlugins.xml having one or more <inherit> elements that simply points to other SageTVPlugins.xml URLs that need to be loaded before this one. I'm sure the existing plugin system already does the work to resolve the latest version when processing multiple sources, so this would be easy to implement.

[Slugger]

Originally Posted by Fuzzy

So what would be the best hosting and validating system to use for the SageTV9Plugins.xml?

Not that we're anywhere close to there yet, but at some point maintaining this project is going to cost money. Hosting a plugin validation site will be one of those expenses. I think at some point, donations are going to have to be solicited to deal with said expenses. With that, an AWS node or whatever can be utilized.

Originally Posted by tmiranda

I disagree. Many of my plugins (and others) will work in v7 or v9 and I do not want to maintain two manifests for separate repositories.

Yeah, I kind of assumed that devs would quickly abandon v7. If I'm way off base on that then so be it. I still think there will come a time when you just won't (be able to) support v7 anymore and under the current system, when you slap the MinVersion=9.0 in your manifest, you immediately end the ability for a v7 user to reinstall that plugin. With separate repos per major version, you don't have to kill the v7 user when your latest update relies on new v9 capabilities.

But if devs do want to maintain v7 support long term then my thoughts on extra tooling to ease the dual maintenance becomes a little more relevant than I originally thought.

Originally Posted by stuckless

Hosting won't be much of an issue, but the "validating system" part might imply a server running somewhere, and I'm not sure who wants to foot the bill for that. Technically anybody, including myself, that has a server in their house, could do the "validation" part of this, and then git commit the new/updated plugin xml if it passes.

As for hosting, I did create an OpenSageTV organization on GitHub a few days ago.
https://github.com/OpenSageTV
Organizations are similar to users, except they can have multiple admins. Users can have specific roles, etc.

I think that using a separate plugin repo per sagetv major version is probably a good idea, but still doesn't fix Slugger's comment about when a developer decides to upgrade to Java 8 and set the Min java version to 8 whether it be saget 7 or 9. So do we start doing separate plugin repos per java version as well??

Yeah my JRE version example wasn't the greatest because users of any Sage version can just upgrade their JRE (usually). My main point is that when devs start slapping MinVersion tags in their manifests to deal with v9 (be it Sage or JRE min versions) then they are messing with v7 users' abilities to reinstall older versions of those plugins. The MinVer JRE tag is easier for v7 users to overcome, but the Sage MinVer=9.0 tag simply cannot be overcome by v7 users.

Originally Posted by stuckless

So, perhaps the bigger flaw here is the fact that SageTV only knows about a single version of a plugin (as skiingwiz pointed out), and not about the history of the plugin, where they can be multiple instances of the plugin based on jvm requirements, sage version requirements, etc. This obviously makes the server processing more complicated, but I'd favour that over forcing a developer to re-publish plugins to several different repositories. (For all I know the plugin system might already do some of this)

Yeah, I think this would be the ultimate solution, I just wonder how feasible it is. Meaning I tend to think Jeff would have supported this originally if it were "easy" b/c iirc, many of us brought this up during the v7 beta when we first saw the plugin manager in action. Assuming it wasn't easy then and there were technical reasons for not making it happen originally, you have to think it would be that much harder to replace what's there now with something like this. But if that could happen, then that pretty much solves everything, I think.

Originally Posted by stuckless

Something else to consider as well... If we used multiple repositories, could we have repositories "inherit" from another repository. So, S9 repo could inherit from S7, meaning, initially the S9 repo is basically blank. When a developer publishes a plugin, if their Min sage version is 9, then it gets added to the S9 repo, and S7 is unaffected. If the Min version is left out then it goes to the base S7 repo, and then it is picked up by S7 and S9 and the plugin is only published once. Still doesn't solve the Java version issue. And "inherit" in this sense could be a simple as the SageTVPlugins.xml having one or more <inherit> elements that simply points to other SageTVPlugins.xml URLs that need to be loaded before this one. I'm sure the existing plugin system already does the work to resolve the latest version when processing multiple sources, so this would be easy to implement.

Yeah, if devs plan on maintaining their plugins in v7 long term then some kind of tooling to ease the dual maintenance will definitely want to be looked at if we go the route of separate repos per major release.

[Fuzzy]

Sadly, the most 'robust' solution would end up being the most complicated. There could be a way to 'overload' the manifest for different versions of a plugin with the same name based on the min and max version information, we'd just have to figure out the best way for sage to handle the situation where min and max from different 'versions' of a plugin overlap.

I'd say that when parsing the repository, sage would have to find the highest version of a plugin that has it's prerequisites met, and that is the only version of it that makes it past the initial phase of reading. However, it would also have to check if all the dependencies for a given plugin also passed this test, so it would be a nested process. Likely not 'hard' to code, especially if each plugin object has it's own validation method in it, that would only show valid if all the plugin's dependencies also show valid.

[Fuzzy]

Managing this might not be totally difficult if there was a better database editor for the repository. Currently, there is the fire-and-forget web form, which doesn't allow for minor changes. If there was instead a web based manifest editor, this would provide the ability to easily visualize and make changes to an existing plugin manifest. This is, of course, a whole other project that needs to be handled, but would be rather simple database manipulation.

[stuckless]

Originally Posted by Fuzzy

Sadly, the most 'robust' solution would end up being the most complicated. There could be a way to 'overload' the manifest for different versions of a plugin with the same name based on the min and max version information, we'd just have to figure out the best way for sage to handle the situation where min and max from different 'versions' of a plugin overlap.

I'd say that when parsing the repository, sage would have to find the highest version of a plugin that has it's prerequisites met, and that is the only version of it that makes it past the initial phase of reading. However, it would also have to check if all the dependencies for a given plugin also passed this test, so it would be a nested process. Likely not 'hard' to code, especially if each plugin object has it's own validation method in it, that would only show valid if all the plugin's dependencies also show valid.

I'm pretty sure the "inherit" model solves this entirely, since sagetv does all the rest. ie, it already can read multiple plugin sources, and it figures out the "latest" as it does it. (ie, SageTVPlugins.xml and SageTVPluginsDev.xml). So, making a small change to sagetv to load multiple repositories would solve all those issues of min/max, etc.

Originally Posted by Fuzzy

Managing this might not be totally difficult if there was a better database editor for the repository. Currently, there is the fire-and-forget web form, which doesn't allow for minor changes. If there was instead a web based manifest editor, this would provide the ability to easily visualize and make changes to an existing plugin manifest. This is, of course, a whole other project that needs to be handled, but would be rather simple database manipulation.

You already know where I stand, but I'd like to ditch the web form totally, and automate that flow so that people are not involved. Preferably, a submit and forget where you submit the .zip and the manfiest together, and a process reads them both. That way, automation tools (ant or gradle or maven) can be used to update plugins.