Tests? Continuous Integration?

[ptoal]

Being a TDD type of guy, I was wondering if there were any plans to create a CI server and/or create a test suite?

At the moment, I think we're all just hacking, trying to get things working for our needs, but at some point, it would be nice to start "cleaning up" a lot of the code and infrastructure of the project.

I was thinking of having my jenkins server do builds on Sage, but I don't want to duplicate efforts, if someone is already planning this. Also, I'm not sure how to do the windows builds, since I got off the Windows Train over a decade ago.

Comments?

[stuckless]

I've been looking into Shippable and setting up a CI build from there. My goal wasn't so much a build for every check-in approach (which is valuable, just don't want to have 1000s of downloads on BinTray). So, I'm looking at Shippable to be used to fire off release builds.

I think someone else has done it as well for every checkin, using Shippable.

Certainly the long term vision includes a CI server and unit testing. I think that given that there are no existing unit tests, unit tests will likely be something that is built up over time for modified code. I know I'd love to see a unit test with every check-in, but I work with enough enterprise software to know that unless code is architected to facilitate unit tests, then it becomes hard to write effective tests. In many cases code needs to be refactored, etc, to allow for easier testing.

[JustFred]

I think there's real value in some kind of regular build (either nightly or pull-triggered). I'm willing to help with getting the windows native code to build from the command line.

I don't have any opinion about the choice of TDD vs BDD vs Unit test, but strongly suspect that the original Sage team had some sort of test suite that they used. It might even have included canned transport streams that could be fed into Sage. Maybe we could get access to that work as a starting point.

[ptoal]

Originally Posted by stuckless

I know I'd love to see a unit test with every check-in, but I work with enough enterprise software to know that unless code is architected to facilitate unit tests, then it becomes hard to write effective tests. In many cases code needs to be refactored, etc, to allow for easier testing.

<sarcasm>Really? I've never seen that happen.</sarcasm>

I haven't even looked at the Java code yet, since I've spent most of my time getting natives to compile on FreeBSD. I suspect that there's a lot of modularization and refactoring that would have to be done to support good tests. That said, even having some basic smoke tests would be helpful at this point.

Originally Posted by JustFred

I don't have any opinion about the choice of TDD vs BDD vs Unit test, but strongly suspect that the original Sage team had some sort of test suite that they used. It might even have included canned transport streams that could be fed into Sage. Maybe we could get access to that work as a starting point.

Hm... Based on my experience with the product over the years, I'm not so sure. I seem to recall a number of builds that seemed to be "code, compile and ship". These guys were good, but they weren't big. Also, the whole "Agile Movement" didn't really get legs until well after SageTV was established.

If I have to be honest, I don't have a ton of time to dedicate to developing for SageTV. I just want to be effective in anything I develop. I have this FreeBSD branch, for example, that I'm nervous about submitting a PR for, because I don't know if it will break any other builds. I don't think we have a big enough development community at this point to start writing Community Contribution Guidelines, and managing things more formally. It would be nice, though, if we all knew we were rowing the boat in the same direction.

[JustFred]

My comment about the possibility of an existing test suite was based on some things I've seen in the native code.

The work that I did to move native code from vs2005 to vs2015 made it pretty clear that there had been some fairly recent changes to that code that managed to skip the compile step of "code, compile and ship". So I'm in favor of an automated build.

Just because we can't do **everything** doesn't mean we shouldn't do **something**.

Like you, I've got some specific bugs that I'd like to fix (without introducing regression issues). Currently they involve the transport stream, in native code, so the ability to feed a canned TS into the beast would give a higher confidence that a fix for cable clear QAM didn't break Sat.

[Narflex]

Originally Posted by JustFred

I don't have any opinion about the choice of TDD vs BDD vs Unit test, but strongly suspect that the original Sage team had some sort of test suite that they used. It might even have included canned transport streams that could be fed into Sage. Maybe we could get access to that work as a starting point.

Sorry, we did not have any kind of test suite at SageTV. It was basically build and then internal manual tests (or more extensive in certain components depending upon what changed). Then a couple of us would run it at home ourselves for a day or two, and then feed it to the beta testers after that. It was mainly due to the small number of devs (4 for the majority of the timeframe); and clear isolation of who worked on what (Java, STV, Linux Native, Windows Native) that we could ensure quality this way. There were also quick releases of stuff when we were highly confident in a minor fix being pushed out directly.

There were a few test things in the code; such as for FFMPEG format detection (where it can redetect everything to be sure it got the same results as last time)...but you need a highly diverse content library (which we used to have, but Google's opinion on fair use is a little more cautious than ours was so we had to leave it behind) to make that effective.