Discussion around Possible Open Source Project restructuring/refactoring

[stuckless]

Jeff is now scared

I've looked through the GitHub project and sucessfully built it, but I think the project files needs some "tweaking" to make it more developer friendly.

At the very least, I'd like to see the "java" dir expanded to a more Maven/Gradle approach such that you have

src/main/java
src/main/resources
src/test/java/
src/test/resources

But to take it even further, I'd like to see,

SageCore/src/main/java
SageStdio/src/main/java
SageMiniClient/src/main/java

Where SageCore is the main SageTV stuff, SageStudio is the studio developer, and MiniClient are the sources for a miniclient.

This separation listed above would make it easier to integrate with Maven/Gradle (but not a requirement), but more importantly it would provide a logical separation be source code, test code, and resources (ie, things you put in the jar that is not code).

The project separate would simply provide a more logical view of the code, by separating out what is a MiniClient vs what is Core, etc. (I'm not proposing changing package namespaces, at this point).

I also think that the project could benefit from using Gradle as a cross platform build engine (especially on the Java building/packaging).

With a build.gradle, you can easily have gradle setup the Eclipse or IntelliJ projects based on the build configuration. That way, you an build either using gradle, or you can easily bring the Java projects into an IDE. (All generated IDE specific files would be added to .gitignore so that they never get checked into source). The only gradle specific things that would be required to be checked in are the build.gradle, settings.gradle, and the gradle-wrapper (which means you don't need to install gradle yourself).

Once gradle has been added, it would be trivial to checkout and build the Java portion of the project in Windows or Linux, which is probably what 99% of developers will care about.


In addition to the project layout changes (proposed) I'd also like to see some subtle changes to the SageTV installation dir. Currently SageTV uses a single JARs folder for all jars, and like to use a separate JARs folder for plugins. Ideally the SageTV startup would build a classpath from the JARs area first, and then add jars from the PluginJARs/ folder. This ensures that even if Plugins do try to provide a jar that sagetv core depends on, then the "core" one, being first in the classpath, will get loaded first. Today, it's basically a game or Jar roulette if you have same jar in there twice.


Obviously, I can make any of these changes in my own REPO, but doing so, would make it harder for me to accept upstream changes. Ideally, I'd like to see a discussion around this, and maybe, if Jeff sees the value in it, then the main REPO can be changed. Although I don't mind building it out in my own REPO, but only if there's a change it would get accepted in as Pull Request.

[MrD]

I second.

I would also like to see more partitioning of the code into packages (e.g. db code).

Maybe some refactoring of the 9000 line classes

And at the risk of starting a jihad.... a standard coding style.

[tmiranda]

Above my pay grade. I'll go with whatever you and Narflex decide.

[Narflex]

I've never heard of Maven/Gradle before now...so I don't really have any comments regrading that. A lot of the code is highly interdependent..so breaking it up into separate source trees doesn't seem to be all that worthwhile since you could never build one of those trees independently. I totally agree with that it should be separated into much more logical Java packages...I had some really bad habits regarding that initially...and in some of the later code started to actually use subpackages underneath sage.

The idea about multiple JARs folders sounds pretty good as well, and wouldn't require much changes to support (although I *think* it would require plugins to change their manifests since I think they specify the JARs location for files...or maybe the core does that...I really don't remember )

[stuckless]

Originally Posted by Narflex

I've never heard of Maven/Gradle before now...so I don't really have any comments regrading that. A lot of the code is highly interdependent..so breaking it up into separate source trees doesn't seem to be all that worthwhile since you could never build one of those trees independently. I totally agree with that it should be separated into much more logical Java packages...I had some really bad habits regarding that initially...and in some of the later code started to actually use subpackages underneath sage.

The idea about multiple JARs folders sounds pretty good as well, and wouldn't require much changes to support (although I *think* it would require plugins to change their manifests since I think they specify the JARs location for files...or maybe the core does that...I really don't remember )

IF the plugin specify their jar folders (I didn't think they did), then it might be easier to simply create a "SAGEJARs" folder and put "core" jars in there.

Gradle/Maven is build tool (like make) but it supports lots of other things as well, such as downloading dependencies, creating project files, building, packaging, etc. FWIW Gradle is the build tool of choice for Android.

I don't really see any immediate need to create project specific folders, especially, if things are really intertwined... but using the first approach for src/main/java and src/test/java etc, would better facilite a "standard" java project structure and have a separation of code and test code.

[stuckless]

Here's a link to the Gradle Doc.... you can see it's quite extensive.

https://docs.gradle.org/current/userguide/userguide

[Narflex]

Can this be put on top of the existing build stuff that is there? (i.e. it's an overlay that for those who don't want it, can just ignore it)

[MrD]

I would suggest "beside" versus "on-top-of". The native build items could be possibly be re-used. The re-factoring of the directory structure would of course impact the existing build scripts.

Long term It would be additional effort to maintain two sets of build scripts. The current build scripts could be phased out over time as a new build becomes bullet proof.

I've worked with makefiles, ant files, maven pom files. Each evolution is better than the next, I believe gradle is a good next step with the build system.

[tmiranda]

Originally Posted by stuckless

IF the plugin specify their jar folders (I didn't think they did), ...

They don't. All you do in the manifest is indicate that it's a .jar and Sage puts it in the correct place.

[jreichen]

I haven't had much time to look over the code, but I agree with the ideas.

Originally Posted by stuckless

In addition to the project layout changes (proposed) I'd also like to see some subtle changes to the SageTV installation dir. Currently SageTV uses a single JARs folder for all jars, and like to use a separate JARs folder for plugins. Ideally the SageTV startup would build a classpath from the JARs area first, and then add jars from the PluginJARs/ folder. This ensures that even if Plugins do try to provide a jar that sagetv core depends on, then the "core" one, being first in the classpath, will get loaded first. Today, it's basically a game or Jar roulette if you have same jar in there twice.

This could be taken a step further with a classloader for each plugin, then plugins wouldn't be playing Jar roulette with each other either.

[stuckless]

Jeff, so I started to create the gradle files, and I have generating the eclipse project, and I import into into eclipse.

I get 79 Errors in eclipse, and I've included all the JAR files

A couple of these files look like they can never compile, and so I re-checked your buildsage.sh and I can see why it works there. You basically put all the source dirs as a source path, and then you pass the list of files to compile, so, you never hit these files

Here's the errors, in 4 files. (I'm only sending a sampling)

Description	Resource	Path	Location	Type
The method drawVideo(GL2, Rectangle, Rectangle) in the type OpenGLVideoRenderer is not applicable for the arguments (GL, Rectangle, Rectangle)	JOGLSageRenderer.java	/sagetv/java/sage	line 1768	Java Problem
GL_COLOR_ATTACHMENT0 cannot be resolved or is not a field	OpenGLGFXCMD.java	/sagetv/java/sage/miniclient	line 1900	Java Problem
Unreachable catch block for IOException. This exception is never thrown from the try statement body	QuartzGFXCMD.java	/sagetv/java/sage/miniclient	line 1144	Java Problem
renderXform cannot be resolved or is not a field	QuartzSageRenderer.java	/sagetv/java/sage	line 548	Java Problem

So my question is are these 4 files even used, anywhere, in SageTV given that they do not compile??

QuartzSageRenderer.java
QuartzGFXCMD.java
JOGLSageRenderer.java
OpenGLGFXCMD.java

If these files are being used, then I need to figure out why they are not compiling. If they are not being used, then we can probably delete them (we can recover them later if we need them).

[MrD]

Some of those files are platform dependent (i.e. OSX)

QuartzSageRenderer.java
QuartzGFXCMD.java

Quartz is the OSX graphics engine.

[MrD]

So I started looking at using maven to build the java components.

I was using a multi-module approach. I was building third_party jars separately and installing those into the local repo and then using those as dependencies in the main module (i.e. Sage).

That is when I hit the issue. Third party libs (MetaDataExtractor, RSSLIB4J, SingularSys and Ogle) have dependencies on the main line code (i.e. the Sage folder).

The current build works because all the code is presented to the compiler at once and is bundled in the Sage.jar. This will not be a trivial task to unwind, for example, SingularSys pulls in Catbert and Pooler and Sage classes.

Thoughts?

[Fuzzy]

As far as I can tell, there's two types of third party java stuff with sage. There's the standalone JAR's that are in the classpath, but remain in their own packages, and are called by sage.jar. Then there are the 'core uses us' modules that are third party sourced routines that are a part of the core, and become a part of the sage.jar package. All of the second part should remain in the same module, partly for the issue you've found out, and also because there'd be no point in compiling them separately.

[stuckless]

Originally Posted by MrD

So I started looking at using maven to build the java components.

I was using a multi-module approach. I was building third_party jars separately and installing those into the local repo and then using those as dependencies in the main module (i.e. Sage).

That is when I hit the issue. Third party libs (MetaDataExtractor, RSSLIB4J, SingularSys and Ogle) have dependencies on the main line code (i.e. the Sage folder).

The current build works because all the code is presented to the compiler at once and is bundled in the Sage.jar. This will not be a trivial task to unwind, for example, SingularSys pulls in Catbert and Pooler and Sage classes.

Thoughts?

Originally Posted by Fuzzy

As far as I can tell, there's two types of third party java stuff with sage. There's the standalone JAR's that are in the classpath, but remain in their own packages, and are called by sage.jar. Then there are the 'core uses us' modules that are third party sourced routines that are a part of the core, and become a part of the sage.jar package. All of the second part should remain in the same module, partly for the issue you've found out, and also because there'd be no point in compiling them separately.

To be honest, I'm less concerned about it being that modular (at least from day one). My original thoughts were to split it out by "purpose", so there'd be a "Server", "Client", and "Studio" projects, but as Jeff said, all that code is tightly dependent.

As of now, I have a gradle srcript that will pull in all the SageTV sources, from all the source locations as defined in the buildsage.sh. On Windows/Linux I can compile all code an rebuild the Sage.jar. I think that is really step 1, since most people will need that level of functionality.

The gradle script can optionally create the Eclipse and IntelliJ project files well, so that you can use an IDE to edit the code (which is where I found the 4 files that won't compile).

I'm going to try and deliver this to my repo tonight or tomorrow, if people want to take a look (it's pretty minimal stuff at this point).

Jeff, for now, I'm not replacing the existing /build/*.sh files, since this can co-exist. Over time if the crossplatform gradle scripts get enough functionality to fully replace the build/*.sh stuff, then we can look as removing them.

Over the years, I've used Ant, Ant+Ivy, Maven, Make, and Gradle. And to be honest, I probably like Ant+Ivy the best, but Gradle is growing on me. I have a love/hate relationship with Maven, mainly because it's great a building projects, but it's way too restrictive. Maven is great if you start out with it, but retrofitting Maven has been painful (for me) over the years. I do like gradle, because it contains 100s of plugins (like Maven) that help from everything from building jars, zip, downloading dependencies, etc, but you a also customize each part of this pipeline, which is real benefit. ie, you can probably make it do whatever you need it to do, which is something I can usually do with Ant, but was something that I could rarely do with Maven

Gradle also has an incubating plugin for creating Visual Studio projects, so maybe once that is ready, people can use Visual Studio (on windows) if they so desire.

[rsteele]

A very enthusiastic yes to your proposals, both the reorganization of the code and the use of Gradle.

Another possible benefit of using Gradle is its cross-platform build capabilities. I'm not sure how important that will be long-term, given that it seems most of the meat is in the sage.jar, but some might appreciate the option.

[stuckless]

Originally Posted by jreichen

This could be taken a step further with a classloader for each plugin, then plugins wouldn't be playing Jar roulette with each other either.

Agreed, it would be nice to get to that point where each plugin has it's own classloader (now we are approaching OSGi -- which I talked about during the original implmentation of the plugin system )

At the very least it would be a good idea to put ALL plugins in a single classloader (sort of like how a web app does), and that way you can restart the plugins (all of them) without restarting the main system, at the expense of leaking memory.

[MrD]

Originally Posted by MrD

The current build works because all the code is presented to the compiler at once and is bundled in the Sage.jar. This will not be a trivial task to unwind, for example, SingularSys pulls in Catbert and Pooler and Sage classes.

I figured this out using the appropriate maven plugin.

My main motivation here is not modularization, but to be able to introduce standardized unit test cases and reporting. Test cases (if used) will be instrumental to keeping regressions out of the code base and it is another way to isolate code and learn how it works.