V9 of various plugins are coming... but...

[stuckless]

Since SageTV has been opened sourced, there has been more interest (at least from me) to update some of the plugins that I work on (or use).

I've deprecated Phoenix UI because the UI maintainers have moved on, and I switched to Gemstone. I'm been pleased with Gemstone.

Jusjoken has been actively updating the Phoenix core code and he's recently added new views for Missing TV Episodes and he's added Collection support. In doing this we needed to move to the latest version of the TheMovieDB apis, which required newer versions of other jars, etc, etc. This plugin update is likely to break some other plugins that depend on http client.

Because of this, the new Phoenix Core will be a V9 ONLY plugin. ie, you can't install it on V7, and the V7 version is no longer exist.

Because of how SageTV manages plugins, there can only be a single version for a plugin id. So, once the new Phoenix Core plugin is released, you will no longer be able to update/install Phoenix in V7. In my mind, this isn't a big deal, because either you have Phoenix installed on V7 and you'll continue to use that version, or if you want to move to Gemstone (or use Phoenix features), you'll move to V9.

That being said, I've mentioned before that I really do think that the Open Source version of SageTV SHOULD have a separate SageTVPlugins.xml from the V7 version and the that the V9 version would "inherit" V7 plugins, and override the ones are updated in V9. This would require Jeff to update the plugin submit process to separate V7 and V9 submissions or the open source community would need to build a plugin repository that handles submissions, etc.

We could create new plugin ids for the V9 phoenix (and all dependent libraries) but that's just crazy... and we'd end up with the possibilty that people would install both phoenix-core and phoenix-core-sagetv9 and they'd clobber each other, or end up with duplicate jars in the JARs area, etc.

Jusjoken has stated that he'll be making the new Gemstone a V9 only plugin as well, which, he'd have to do, if it depends on Phoenix (which it does).

I'm also working on updating the Jetty Plugin to Version 9, which will be a SageTV 9 only update as well. (BMT is dead, but not sure if the mobile web and the main web will be updated to work with the V9 server, or if we just start over and build a single web app that encompases, mobile, web and bmt applications -- I'm leaning towards the latter)

And I have a new plugin, PushBullet for SageTV, that will be a V9 only plugin, since it uses some APIs that are not available in V7. This plugin basically pushes SageTV System Messages (Halt Detected, etc) to your phone/desktop, if you have the PushBullet app (or extension) installed.

I'm also working on updating the sagex-api plugin and it move to a V9 version as well.

So, I guess the good news is that there are new and updated plugins coming, but, because of how intertwined V7 and the Open Source versions are (ie, sharing the same plugins), then V7 is about to get a lot less functional.

This really isn't a question of not wanting to support V7... it's just that once you move a plugin to depend on V9, V7 can no longer access the V7 version of that plugin, and anything that depended on that V7 version can not be installled, etc.

I guess I'm wondering if it's time to simply change the V9 code to support additional plugin repositories (something I've suggested in the past), and then maintain a separate SageTVPluginsV9.xml in a github repository for V9 plugin updates. This allows V7 to carry on, unbroken, while V9 can evolve without impacting V7.

[Fuzzy]

Re: open source plugin repository, I think we can likely go away completely from the web form, and simply edit the SageTVPluginsV9.xml directly in github. This would allow community management, and allow us to make global changes in the future (like say we implement a way to have auto-mirroring of the plugins, for instance) without needing the original dev to be involved.

First step should be easy - adding a loader for SageTVPluginsV9.xml to the code. It essentially would be a duplicate of how SageTVPluginsDev.xml is loaded in, but done before the Dev one.

SageTVPlugins.xml is loaded.
SageTVPluginsV9.xml is loaded and merged in, replacing any elements with the same ID.
SageTVPluginsDev.xml is loaded and merged in, replacing and elements with the same ID.

Distributing and updating the V9.xml is a separate issue, and can be handled separately after the plugin core is made to look at it.

[stuckless]

Originally Posted by Fuzzy

Re: open source plugin repository, I think we can likely go away completely from the web form, and simply edit the SageTVPlugins9.xml directly in github. This would allow community management, and allow us to make global changes in the future (like say we implement a way to have auto-mirroring of the plugins, for instance) without needing the original dev to be involved.

First step should be easy - adding a loader for SageTVPlugins9.xml to the code. It essentially would be a duplicate of how SageTVPluginsDev.xml is loaded in, but done before the Dev one.

SageTVPlugins.xml is loaded.
SageTVPlugins9.xml is loaded and merged in, replacing any elements with the same ID.
SageTVPluginsDev.xml is loaded and merged in, replacing and elements with the same ID.

Distributing and updating the 9.xml is a separate issue, and can be handled separately after the plugin core is made to look at it.

That is my thinking as well, since the core code is already doing this when it loads SageTVPluginsDev.xml.

Let's see what Jeff has to say

[Fuzzy]

Originally Posted by stuckless

That is my thinking as well, since the core code is already doing this when it loads SageTVPluginsDev.xml.

Let's see what Jeff has to say

Well, I just looked, and haven't tested yet, but this looks rather easy (for step 1 at least).

https://github.com/CraziFuzzy/sagetv...01da9931aa46d3

[stuckless]

Originally Posted by Fuzzy

Well, I just looked, and haven't tested yet, but this looks rather easy (for step 1 at least).

https://github.com/CraziFuzzy/sagetv...01da9931aa46d3

Yeah that's good for a local file, but we to implement a hosted version, like the SageTVPlugins.xml where it checks the md5 to determine if we need to update etc.

I'll spend some time and make the changes, and see if Jeff agrees.

[stuckless]

So, I made the core changes to support additional repos... I basically just refactored the core code that does the checking and downloading of the SageTVPlugins.xml to accept parameters for the url, md5 url, and the local filename.

The repo configuration is stored in the Sage.properties, and this is the documentation in the file.

  // The Repo Properties contains a set of Plugin Repositories that SageTV will periodically check for new
  // and updated SageTV Plugins.
  // The main key is sagetv_repos
  // Each Repo has a unique id, and contains the URL for the REPO, md5, and the local filename
  // sagetv_repos/xxxx/url=Plugin Repo URL
  // sagetv_repos/xxxx/md5=Plugin Repo MD5
  // sagetv_repos/xxxx/local=The Local Filename to use for this repo
  //
  // There are reserved REPO IDs, v7, and v9, with future reserved ids of v10, v11, etc.
  // sagetv_repos/v7/url=Plugin Repo URL
  //
  // When SageTV loads the REPO plugins, it will read v7, then v9, and then any alternate repos
  // listed in the sagetv_additional_repo_list, which is a comma separated list of alternate repos and
  // the order in which to read them.
  // Finally, SageTVPluginsDev.xml is read last, as means to allow devs to forcefully overrade the plugins
  // listed in the plugin system.

  // Comma separated list of alternate repos.  Should not contain, v7, v9, etc, but can contain additional
  // configured repos.  Configured repos would be configured under the sagetv_repos parent key.

This allows, that in the future, we can change the location of the v7 (and v9, etc) repo urls, should be ever need to. It also allows for custom repos that developers can add for beta testing, etc.

The default values for v7 and v9 repos (in the code) is

sagetv_repos/v7/local=SageTVPlugins.xml
sagetv_repos/v7/md5=http\://download.sagetv.com/SageTVPlugins.md5.txt
sagetv_repos/v7/url=http\://download.sagetv.com/SageTVPlugins.xml
sagetv_repos/v9/local=SageTVPluginsV9.xml
sagetv_repos/v9/md5=https\://raw.githubusercontent.com/OpenSageTV/sagetv-plugin-repo/master/SageTVPluginsV9.md5
sagetv_repos/v9/url=https\://raw.githubusercontent.com/OpenSageTV/sagetv-plugin-repo/master/SageTVPluginsV9.xml

I've testing this locally on my development machine, and, it downloaded the V9 repo file, and loaded the plugins from there (which is only 1, right now, which is my pushbullet plugin) and I was able to install it.

I did some testing to update the md5 url, and sagetv appears to noticed that it changed, and updated the V9 plugins.

The code changes are pretty minimal, because as I said, I just refactored Jeff's code to accept 3 parameters, and then I just call it for each repo configuration.

One last note, the repo order is.. v7, v9, additional repos, then the local SageTVPluginsDev.xml

[stuckless]

FYI... I created a pull request, so we'll see where this goes.

I've also creates a v9 plugin repository with instructions on how to submit new or updated plugins.

https://github.com/OpenSageTV/sagetv-plugin-repo

[troycheek]

I'm not a plugin developer, but as a plugin user I vote for separate... something. I don't even know what to call it as I don't understand most of what's going on in this thread. I tend to think that software developers should be like doctors. First do no harm. I look forward to whatever wonderful things you can do with plugins for v9, but please don't take away a single bit of functionality of v7 in the process.

[tvmaster2]

Originally Posted by troycheek

I'm not a plugin developer, but as a plugin user I vote for separate... something. I don't even know what to call it as I don't understand most of what's going on in this thread. I tend to think that software developers should be like doctors. First do no harm. I look forward to whatever wonderful things you can do with plugins for v9, but please don't take away a single bit of functionality of v7 in the process.

+1, at least until we, the coding-handicapped, can make sense of things

[Fuzzy]

Originally Posted by stuckless

Yeah that's good for a local file, but we to implement a hosted version, like the SageTVPlugins.xml where it checks the md5 to determine if we need to update etc.

I'll spend some time and make the changes, and see if Jeff agrees.

Yeah, my little commit, as mentioned in it's comment, was just about the reading of the file, didn't deal with actually fetching a new version, because I wasn't really sure which way we want to go for that. If it's something to be generated with nightly builds (which aren't happening yet either) from the it, or what.

[Fuzzy]

Just read through your implementation, seems great. I really like the ability for each dev to simply maintain their own .XML and just have to 'register' a url to the repo. Hopefully Jeff will pull it in, so that 'hopefully' it will be live and well before someone (jusjoken) gets an installer working.. :-)

[Fuzzy]

One thing. What sort of validation is going on before items are added to the repository? I worry about a typo in someone's individual manifest resulting in malformed XML that hoses up the whole repo. On my first, rather uneducated, glance, it looks like you composite all the various manifests into a single temporary repository, verify it with xmlint, and if that passes it's tests, replace the main one. So, what would happen here, is that one bad manifest would stop the whole repository from updating. Would it be possible to use xmlint on each manifest instead, before concatenating it into the whole? Would save one mistake from holding up all other plugin updates until it's found.

[stuckless]

Originally Posted by Fuzzy

One thing. What sort of validation is going on before items are added to the repository? I worry about a typo in someone's individual manifest resulting in malformed XML that hoses up the whole repo. On my first, rather uneducated, glance, it looks like you composite all the various manifests into a single temporary repository, verify it with xmlint, and if that passes it's tests, replace the main one. So, what would happen here, is that one bad manifest would stop the whole repository from updating. Would it be possible to use xmlint on each manifest instead, before concatenating it into the whole? Would save one mistake from holding up all other plugin updates until it's found.

I'll add the xmllint check to each manifest, since that part is easy.

I think that in addition to just xml validation, I know jeff actually validates each package to make sure it matches the md5, etc. I might do that eventually, but for now, I'm OK with just verifying the integrity of the xml file.

This script took about 30 minutes to create... it's amazing what you can do with simple linux commands . Eventually we'll need something more robust, either written in java or maybe python, that does better validation, etc.

[Fuzzy]

The only other thing that might be nice, and possible depending on how you intend on running the script, would be for it to write the script's console output to logfile to the git as well, so it can easily be viewed to see that everything is working without depending on you having to intervene.

[stuckless]

Originally Posted by Fuzzy

The only other thing that might be nice, and possible depending on how you intend on running the script, would be for it to write the script's console output to logfile to the git as well, so it can easily be viewed to see that everything is working without depending on you having to intervene.

I agree... not sure how this is going work at the moment. Using something like TravisCI I can kick off a build everytime the git repo changes, but, for remote url references, I'm not going to know if the remote url is updated. (ie someone updates their plugin manifest in their git repo, then how do we tell this repo to rebuild the plugins repo xml file).

So, I think we are going to have a few kinks to work out, but definately, I agree, I should not be the bottleneck here. We might have to have several custodians that can in charge to kicking off a new build of the repo xml file, until we can fully automate it.

[Fuzzy]

Originally Posted by stuckless

I agree... not sure how this is going work at the moment. Using something like TravisCI I can kick off a build everytime the git repo changes, but, for remote url references, I'm not going to know if the remote url is updated. (ie someone updates their plugin manifest in their git repo, then how do we tell this repo to rebuild the plugins repo xml file).

So, I think we are going to have a few kinks to work out, but definately, I agree, I should not be the bottleneck here. We might have to have several custodians that can in charge to kicking off a new build of the repo xml file, until we can fully automate it.

I have some ideas on that, but know too little about linux scripting to visualize what exactly needs to be done. My thought are we could actually run the CI script relatively often, but we have to make sure that the actual file and MD5 are only updated when something changes. To do this, we'd have to - after the temp repository file is made - compare it to the current file. Since it is built with the date in the first tag, however, we'd have to ignore that tag. I'm not sure if grep is able to do this (can grep pass through lines that DON'T match?). If it can, though, we could compare the two files, grep'ed to remove the "<PluginRepository" line, and if they do NOT match, then we make the MD5 and replace the .XML file.

[Fuzzy]

That didn't turn out to be that hard to find the answers to. Yes, grep does allow reversing the match, and yes i can pipe the output of two greps into a diff. this command should do what is needed:

diff <(grep -v "<PluginRepository" $PLUGINFILE) <(grep -v "<PluginRepository" $TMPFILE)

The -v on the grep inverts the match, so all lines that do NOT contain <PluginRepository are passed into the diff. Diff will then compare them. The $? will be 0 if the files match, and 1 if they are different. I tested it in a temp environment. Changing the date in the <PluginRepository> tag still results in a match, changing anything down in the 'meat' of the repository does not.

EDIT: Should probably also pass diff's output to nul, so it doesn't end up in clogging up any logfile that ends up being created.

[stuckless]

Originally Posted by Fuzzy

That didn't turn out to be that hard to find the answers to. Yes, grep does allow reversing the match, and yes i can pipe the output of two greps into a diff. this command should do what is needed:

diff <(grep -v "<PluginRepository" $PLUGINFILE) <(grep -v "<PluginRepository" $TMPFILE)

The -v on the grep inverts the match, so all lines that do NOT contain <PluginRepository are passed into the diff. Diff will then compare them. The $? will be 0 if the files match, and 1 if they are different. I tested it in a temp environment. Changing the date in the <PluginRepository> tag still results in a match, changing anything down in the 'meat' of the repository does not.

EDIT: Should probably also pass diff's output to nul, so it doesn't end up in clogging up any logfile that ends up being created.

I added your logic to the existing script
https://github.com/OpenSageTV/sagetv...ginsxml.sh#L58

It will now abort early if there are no changes.

I still need to add logic to "commit" the new SageTVPluginsV9.xml/md5 files. The process is simple enough, just a little trickier when using CI, since, I need to
1. git clone the repo
2. git copy the modifed files into the repo
3. git commit the changes

This is easy enough, but, I need to do this as a user that has access to the git repo, and I'm sure that CI does not have write access to re-commit the files.

I'll have to play with it and see.

[Fuzzy]

Originally Posted by stuckless

I added your logic to the existing script
https://github.com/OpenSageTV/sagetv...ginsxml.sh#L58

It will now abort early if there are no changes.

I still need to add logic to "commit" the new SageTVPluginsV9.xml/md5 files. The process is simple enough, just a little trickier when using CI, since, I need to
1. git clone the repo
2. git copy the modifed files into the repo
3. git commit the changes

This is easy enough, but, I need to do this as a user that has access to the git repo, and I'm sure that CI does not have write access to re-commit the files.

I'll have to play with it and see.

Looks good. I also like the flagging at the end of any omitted plugins. (Not that I envision a lot of problems with this, but it would certainly be at more risk of happening than with the current repository).

[stuckless]

So, I added a .travis.yml file so that whenever something is pushed to the repo, it will rebuild and re-publish the xml and md5 files (whithout causing an infinite loop )

It works. (added a new "test" plugin and within a few seconds I can see the repo is updated with the new SageTVPluginsV9.xml)