Automatic removal of bad Lucene JAR file

[Narflex]

What do people think about doing the following:

In Sage.java when it starts up, look in the JARs folder and see if there are 2 Lucene jar files, one of them of course must be the new one we are requiring. If it finds 2 of them, then it uses the plugin update feature to indicate that it should delete the bad lucene JAR file. It then forces a SageTV restart, which will then delete the bad Lucene jar file and then let SageTV startup normally.

This will avoid people having the problem with the Lucene JAR file when they upgrade. (yeah, I should have thought of this sooner)

I have no problem writing this myself, it should be fairly easy to code.

[stuckless]

Originally Posted by Narflex

What do people think about doing the following:

In Sage.java when it starts up, look in the JARs folder and see if there are 2 Lucene jar files, one of them of course must be the new one we are requiring. If it finds 2 of them, then it uses the plugin update feature to indicate that it should delete the bad lucene JAR file. It then forces a SageTV restart, which will then delete the bad Lucene jar file and then let SageTV startup normally.

This will avoid people having the problem with the Lucene JAR file when they upgrade. (yeah, I should have thought of this sooner)

I have no problem writing this myself, it should be fairly easy to code.

Make it so I have thought about doing this myself.... but I likely won't have time, so, yeah I think it's a good idea.

[stuckless]

Longer term, I think that all SageTV core jars should be in a SystemJARs folder (or just called "lib"), separate from the JARs folder (used by plugins), so that we can put Sage.jar, SystemJARs, and then JARs in the classpath, in that order. That would prevent plugins from overriding System Jars (or in this case, a new System Jar that conclicts with the plugins).

But I think a good, quick first step, is to simply do the check on startup for the lucene jar as you suggested.

[Tiki]

Originally Posted by Narflex

What do people think about doing the following:

In Sage.java when it starts up, look in the JARs folder and see if there are 2 Lucene jar files, one of them of course must be the new one we are requiring. If it finds 2 of them, then it uses the plugin update feature to indicate that it should delete the bad lucene JAR file. It then forces a SageTV restart, which will then delete the bad Lucene jar file and then let SageTV startup normally.

This will avoid people having the problem with the Lucene JAR file when they upgrade. (yeah, I should have thought of this sooner)

I have no problem writing this myself, it should be fairly easy to code.

I think it's a good idea. One comment though... When I went through the upgrade process last week, I got caught by this issue. I knew what the problem was right away when it hung on the splash screen because I'd seen enough posts about it on the forums. After I manually deleted the old Lucene jars and restarted Sage everything worked, but I got a warning that Sage was unable to find the now missing old Lucene jar, because I still had some plugin installed that was looking for it.

[stuckless]

Originally Posted by Tiki

I think it's a good idea. One comment though... When I went through the upgrade process last week, I got caught by this issue. I knew what the problem was right away when it hung on the splash screen because I'd seen enough posts about it on the forums. After I manually deleted the old Lucene jars and restarted Sage everything worked, but I got a warning that Sage was unable to find the now missing old Lucene jar, because I still had some plugin installed that was looking for it.

After doing the upgrade, and then updating the plugins (there should have been a bunch of updates to various plugins), did the message go away? Phoenix used to have a dependency on lucene, but I removed it when this issue started happening.. but likely the lucene library plugin was still installed, so maybe that's where it's coming from.

[tmiranda]

As I was the first person bitten by this bug, I agree it should be fixed

I like Sean's idea of a System JARs folder. That seems to make more sense than putting lucene (and maybe others in the future) into the "plugin" JARs folder.

[Fuzzy]

Originally Posted by Narflex

What do people think about doing the following:

In Sage.java when it starts up, look in the JARs folder and see if there are 2 Lucene jar files, one of them of course must be the new one we are requiring. If it finds 2 of them, then it uses the plugin update feature to indicate that it should delete the bad lucene JAR file. It then forces a SageTV restart, which will then delete the bad Lucene jar file and then let SageTV startup normally.

This will avoid people having the problem with the Lucene JAR file when they upgrade. (yeah, I should have thought of this sooner)

I have no problem writing this myself, it should be fairly easy to code.

I had thought about this before, but was unsure if it would even get to the point of getting the JVM up and running SageTV.java before the classpath issue causes a problem (I don't know enough about java to know this). I had assumed we'd need to do a check for this in the launcher, before building the classpath and passing it to java, which would be much clunkier.

[Tiki]

Originally Posted by stuckless

After doing the upgrade, and then updating the plugins (there should have been a bunch of updates to various plugins), did the message go away? Phoenix used to have a dependency on lucene, but I removed it when this issue started happening.. but likely the lucene library plugin was still installed, so maybe that's where it's coming from.

Actually, I think there was only one update message, which surprised me, but then I don't have all that many plugins installed.

Once I saw the error about the missing Lucene.jar, I went to the installed plugins section and looked for ones that had notices about updates (there were 3 or 4). However, when I tried to update, I ran into the problem where it could not locate commons.io on sourceforge. This would cause the update to fail and the plugin I was attempting to update would then disappear from the installed plugins list. I know this happened with 2 or 3 plugins (one was the CMT plugin and I think another was Phoenix Fanart for default STV - both of these depend on Phoenix core).

I remembered seeing forum posts that there was a Java issue related to accessing some plugin locations (I think it was that some of the plug repositories had gone to HTTPS and you needed at least Java 8 to work access these locations). So, I upgraded from Java 1.7 to 1.8 and then tried to re-install my plugins - this time they worked fine.

This was all last week, before you moved some of the libraries to a new host. So, hopefully, the commons.io problem is fixed now. Though I do wonder why failed plugin updates would cause the plugins to be removed from the installed plugins list (as far as I could tell, they weren't actually removed).

[EnterNoEscape]

Originally Posted by Tiki

I remembered seeing forum posts that there was a Java issue related to accessing some plugin locations (I think it was that some of the plug repositories had gone to HTTPS and you needed at least Java 8 to work access these locations). So, I upgraded from Java 1.7 to 1.8 and then tried to re-install my plugins - this time they worked fine.

1.7 will work with HTTPS. I verified all of this prior to enabling it's usage in v9. It's more likely the plugins are being built against 1.8.

[AlphaCrew]

As I'm starting to move toward V9, reading through all the great work everyone has done, I wondered why V9 didn't automatically remove it.

So yeah, I think it's a great idea.

[Thomas Wischgol]

Originally Posted by stuckless

After doing the upgrade, and then updating the plugins (there should have been a bunch of updates to various plugins), did the message go away? Phoenix used to have a dependency on lucene, but I removed it when this issue started happening.. but likely the lucene library plugin was still installed, so maybe that's where it's coming from.

I ran into the exact same issue when upgrading to v9 a week ago. After deleting the old lucene jar it complained in the log file and would not start up at all. In order to delete the plugin, I rolled back to v7 and then updated the "Complementary Metadata Tools". This then allowed me to delete the old lucene plugin. After that I reinstalled v9 over it and everything went fine from there (except that I could not install v9 clients on XP so I had to upgrade all clients to Windows 7 first).

[JREkiwi]

Originally Posted by Thomas Wischgol

I ran into the exact same issue when upgrading to v9 a week ago. After deleting the old lucene jar it complained in the log file and would not start up at all. In order to delete the plugin, I rolled back to v7 and then updated the "Complementary Metadata Tools". This then allowed me to delete the old lucene plugin. After that I reinstalled v9 over it and everything went fine from there (except that I could not install v9 clients on XP so I had to upgrade all clients to Windows 7 first).

That would have fixed it because the newer version of CMT has the later versions of Phoenix as a pre-req.

John

[Narflex]

After some thought...I think there's an even better way to do this. Ideally, we won't remove the bad lucene JAR file so plugins do not complain. From some reading, it does appear that the JVM will search the classpath in the order it is specified on the command line. So if we just put the included lucene JAR file as the first entry in the classpath (after Sage.jar), then it will always use that one instead of the one that may come from a plugin. I also wrote a test and it was working this way (at least with OpenJDK on Linux). But logically, I see no reason it would behave any other way than searching the classpath in the order specified.

This would then just require changing the startsagecore script on Linux and launcher.cpp on Windows so that it forcibly puts the included lucene JAR file as the first entry.

Anybody see any problems with that approach?

[stuckless]

Originally Posted by Narflex

After some thought...I think there's an even better way to do this. Ideally, we won't remove the bad lucene JAR file so plugins do not complain. From some reading, it does appear that the JVM will search the classpath in the order it is specified on the command line. So if we just put the included lucene JAR file as the first entry in the classpath (after Sage.jar), then it will always use that one instead of the one that may come from a plugin. I also wrote a test and it was working this way (at least with OpenJDK on Linux). But logically, I see no reason it would behave any other way than searching the classpath in the order specified.

This would then just require changing the startsagecore script on Linux and launcher.cpp on Windows so that it forcibly puts the included lucene JAR file as the first entry.

Anybody see any problems with that approach?

That will work. Java does search the classpath in order, which is why I said it would a nice future enhancement to separate the jars and put the sage system ones in a separate folder. But yeah, we could add lucene (and the other sagetv core jars) explicitly to the start of the classpath. Also if we do this change, I'd like to see "." removed from the classpath... I'm not sure why it gets added, but, it caused me some grief some months ago

I assume you are still volunteering to do this

[Narflex]

Originally Posted by stuckless

That will work. Java does search the classpath in order, which is why I said it would a nice future enhancement to separate the jars and put the sage system ones in a separate folder. But yeah, we could add lucene (and the other sagetv core jars) explicitly to the start of the classpath. Also if we do this change, I'd like to see "." removed from the classpath... I'm not sure why it gets added, but, it caused me some grief some months ago

I assume you are still volunteering to do this

Yup, I'll get it done right now.

[Narflex]

OK, commit is in...hopefully it'll compile fine on Windows since I couldn't verify that...lol.

[sflamm]

The only issue with that is if any of the old plugins rely on methods in the old Lucene interface that are no longer present in the new version... otherwise it is fine.

Probably the methods are deprecated and not removed..

I believe the best answer is to:

1. create a System Jar
2. issue the classpath on the command line to include the System Jar after SageTV
3. update all the plugins to use the proper version of Lucene

[jusjoken]

Originally Posted by Narflex

OK, commit is in...hopefully it'll compile fine on Windows since I couldn't verify that...lol.

if nobody does it first i will try a windows build and installer release this weekend.

k

[Narflex]

Originally Posted by jusjoken

if nobody does it first i will try a windows build and installer release this weekend.

k

Thanks, please do!

[jusjoken]

Originally Posted by Narflex

Thanks, please do!

Done.

I created a test on a V7 install that had the older lucene jar and installed the last 9.0.12.506 version and it failed...as expected.

I built a new installer for 9.0.13.536 and that install went smooth and V9 ran fine with the old and new lucene jars in place

I am in the midst of uploading the new Windows Installer versions and will be done later today.

I know stuckless is busy so not sure when the Linux one will be released.

k