Dev Environment

[Slugger]

Ok, so I'm doing some planning for the "next big thing" I plan on working on. One of my big concerns is testing - unit and regression testing. Today my plugins don't have any automated tests that interface with SageTV because there is no way to get consistent, reliable, reproducible results from the Sage API calls, plus there are some API calls that you just don't want to be calling in a test scenario if you only have one SageTV server (which is, of course, my production server).

For example, I'd like to write an automated (Junit) test for SJQ that thoroughly tests the queue loader for loading tasks into the queue. The queue loader parses a ruleset then loops through all media file objects and decides which objects should be inserted into the queue. The problem is that on any given day if I were to run such a unit test the set of objects being scanned by the test will be different (based on what I've watched, deleted, recorded, archived, etc. since the last time I ran the test) and therefore the test results will be different and so I could never reliably verify the results of the test. So I end up very carefully unit testing the parts of my plugins that interact with Sage API manually and once I'm satisfied I cross my fingers and hope for the best after I release it out into the wild. To date, this has worked and since none of my current plugins are big/complex I've been able to get away with it.

My next big idea is, well, big and some form of automated testing is imperative, I believe, to its success (or at least to maintain my sanity when it comes time to maintain a released version of the code). In an ideal world, Sage has a client/server env simulator they're willing to share with us ( ) that redirects API calls to talk to, say, an SQLite or MySQL db that has a well defined set of data (tuners, EPG data, media files, etc.) such that a call to GetAiringsOnViewableChannelsAtTime(), for example, will be known to always return the same array of Airing objects and of course this data can be reset to a well known state at any time.

Assuming Sage would not be willing to share such tools with us outsiders (if they exist), my next thought is to create the simulation environment. I use gkusnick's API wrappers in most everything I do so extending all of his wrapper objects to then load and save these objects from a DB instead of calling Sage API would be possible (I think, though a quick look at his API docs suggest that I may be wrong - let's assume I'm right until he tells me otherwise), but it really 1) adds a lot of overhead to this project, so much so that it's really turning me off the whole idea and 2) I also need to develop, test, and maintain this SageTV simulator. Bugs found in my plugin regression testing (at least for a little while) need to be investigated in the plugin and possibly in the simulator to ensure the bug isn't with the simulator.

Alternatively, what are others doing for test automation when dealing with the Sage API? Maybe I'm just over thinking this and need to hear other perspectives.

[razrsharpe]

how about running a dev server (either on dedicated hardware or in a virtual image)? I've been thinking about doing this too so I don't hose the server during testing and can take it down/restart it at will without worrying about. Unfortunately, this would require a second server license.

re making a library static: EP shared at one point basically a bunch of small null video files that you can load as an import directory. I am sure we could do something similar with TV files, DVDs, and BDs... basically the idea would be to make a test wiz.bin and import directory structure for testing that is always static....

PS. - If sage has such dev tools that would be fantastic if they were to share them

[jreichen]

With stuckless' remote APIs you can create a custom API provider. You might look into that. Instead of forwarding calls to Sage's APIs, your provider could redirect them to your test data set.

He has a few different providers in his code, in fact there's a StubSageAPI, and it's pretty simple to override the provider wtih something like this:

SageAPI.setProvider(ISageAPIProvider provider)

After calling this API, any subsequent calls go through your provider. I think I have an example in my dev environment somewhere that I could dig up if you want to see it.

[Slugger]

I'd be willing to fork over the cost of an additional server license, but then the concern becomes being able to reset wiz.bin to a known state before each test were to run. Also, I'd want to be able to test things like scheduling recordings, play back on a given context, etc. Some things I could possibly test with the idea of mock objects, such as invoking play back on a given UI context. In that scenario, just knowing that I was going to call the playback API call to trigger the playback is probably good enough for me, so mock objects should work there. But for things like testing an airing object for some value requires that I have an airing object to test against and it also requires that I know what values are going to be in the airing object that I'm testing with. Just calling GetAiringTitle() isn't really good enough in some cases, I'll actually want to be able to ensure the value that's returned is the value I'm expecting so I either need to be able to reset the state of wiz.bin on demand or I need to simulate the environment so it pulls its data from a source I can control (SQLite or MySQL db for example).

Then there's a case of hardware. Even with the additional license, I don't have the extra hardware sitting around anymore. I mean I might be able to throw something together, but a lightweight simulator daemon would be much more desirable to test against. I don't think my current server has enough RAM to be running a VM instance and that also means tinkering with my production box (though a VMware instance should be fairly self-contained/stable). Of course, a VMware/2nd server would also require some kind of tuner in order to get EPG data for test purposes? And again, how do you ensure the EPG data is consistent?

[GKusnick]

I have to confess I don't do much in the way of automated regression testing. I manually unit-test all new code, beta-test on my own system for a few days, and then rely on users to report any regression bugs that make it through to release.

That said, it doesn't seem like it would be that difficult to set up an isolated test server with a fixed Wiz.bin and media library as razr suggests. You could leave it disconnected from the Internet to avoid EPG updates, or simply restore Wiz.bin from a known backup at the start of every test cycle. This does obviously require some investment in hardware and licenses.

Another alternative might be to use your production server for testing by using disk images to restore it to a known state at the start of a test cycle and then bring it back current again when done testing. The drawbacks here are that it requires you to take your system offline during testing, plus there's some (hopefully small) risk of hosing everything with a failed restore.

I don't quite follow your point about using my API wrappers to redirect the APIs to a database. Do you mean replacing Wiz.bin with some other media database? That would be possible, but it would take a fair amount of work to emulate all the functions of the SageTV API using another data source.

Or are you talking about having the wrapper functions themselves journal their inputs and outputs to a file so that a session can be played back exactly without reference to Wiz.bin (or any other media database)? It seems to me this can only work so long as you never change your test suite or your pattern of API calls, in which case why bother re-testing?

[Slugger]

Originally Posted by jreichen

With stuckless' remote APIs you can create a custom API provider. You might look into that. Instead of forwarding calls to Sage's APIs, your provider could redirect them to your test data set.

He has a few different providers in his code, in fact there's a StubSageAPI, and it's pretty simple to override the provider wtih something like this:

SageAPI.setProvider(ISageAPIProvider provider)

After calling this API, any subsequent calls go through your provider. I think I have an example in my dev environment somewhere that I could dig up if you want to see it.

Sure, can't hurt to look at it. I doubt it handles the simulation of things though? Or I guess I'd have to implement it all, right? For example, if I made a redirected call to Record(Airing a) then I'd need to ensure that GetCurrentlyRecordingMediaFiles() returned one media file object when Airing a is scheduled to record. Ultimately, though, I prefer to use gkusnick's API wrapper as I really like the strongly typed objects.

It seems as though the best solution would be to have a separate server that I could reset the wiz.bin with. But this also poses a problem because as soon as you start up Sage with an older wiz.bin I imagine one of the first things it will do is try to update the EPG and then there goes your known data set out the window.

[jreichen]

Originally Posted by Slugger

Sure, can't hurt to look at it. I doubt it handles the simulation of things though? Or I guess I'd have to implement it all, right? For example, if I made a redirected call to Record(Airing a) then I'd need to ensure that GetCurrentlyRecordingMediaFiles() returned one media file object when Airing a is scheduled to record. Ultimately, though, I prefer to use gkusnick's API wrapper as I really like the strongly typed objects.

Correct. There's no implementation for something like that yet. Maybe some combination of mock objects and the custom provider would be the way to do that. I'm just thinking high level right now but the best unit testing setup minimizes external dependencies such as another server. If somebody else would want to contribute to your project it would be difficult for them to use your unit tests.

I'm not as familiar with gkusnick's APIs and whether it would be much effort to adopt a provider framework.

[Slugger]

Originally Posted by GKusnick

Another alternative might be to use your production server for testing by using disk images to restore it to a known state at the start of a test cycle and then bring it back current again when done testing. The drawbacks here are that it requires you to take your system offline during testing, plus there's some (hopefully small) risk of hosing everything with a failed restore.

Some people in this house would kill me if the worst case scenario occurred. Also, there's usually only a few hours of downtime at any point of any given day - some days it seems like there's always something recording. This definitely won't work.

I don't quite follow your point about using my API wrappers to redirect the APIs to a database. Do you mean replacing Wiz.bin with some other media database? That would be possible, but it would take a fair amount of work to emulate all the functions of the SageTV API using another data source.

Thinking kind of like this as opposed to the next paragraph. Basically, I'd like to extend AiringAPI.Airing and override all the methods to basically load/save the object's values from an alternative data store (most likely some SQL db). Then I'd just have to write the simulators I'm interested in (simulate recording so if I called Record() on an airing it would simulate the recording of that airing such that GetCurrentlyRecordingMediaFiles() would return back the object, etc.). But a quick glance at your API docs suggests that the constructor for AiringAPI.Airing is private so I couldn't extend the class to read/write to an external data source. Agreed, it'd be a fair amount of work, but when completed you really would have the ultimate simulator for regression testing.

Note that I'm not committing to this, just kind of thinking out loud as I start to ponder my "big idea." And the first big road block I've encountered is how to consistently and automatically test this beast of an idea.

Or are you talking about having the wrapper functions themselves journal their inputs and outputs to a file so that a session can be played back exactly without reference to Wiz.bin (or any other media database)? It seems to me this can only work so long as you never change your test suite or your pattern of API calls, in which case why bother re-testing?

[stuckless]

@Slugger - As jreichen noted, my wrappers includes a StubAPI implemenation, specifically for junit testing. You can take a look at this simple junit test...

Another approach, which I use in some of my test cases (especailly for phoenix and bmt) is easymock. Easymock can be quite overwhelming at first, but it's extremely powerful for doing unit testing.

So, in my tests, I eather use the StubAPI or easymock, and they both work quite well for unit testing.

I also tend to use my wrappers because I develop on my local desktop while connected back to my sage server (on another box). I rarely test anything directly inside of sage until I'm close to deployment. ie, last night I finally pushed a bmt 4.0 build to my server, but I've been developing and testing it since november

It is actually quite trivial to convert greg's wrappers to use my wrappers. But it would require rebuilding gregs library to use sagex.SageAPI.call() instead of the core sage call method. I've done this already for nielm's util library, which I use in testing.

[stuckless]

Here's another unit test that shows how i'm using easymock to create a new provider for my sage apis that are guaranteed to return the results that I need. (just search for testMediaBrowserAPI to see easymock)

I personally would avoid the whole separate "mock" server thing and just embed your inputs in your test cases. That way, they are guaranteed to work, even if your "server" is not there.

[Slugger]

Hmm... EasyMock might just do the trick. Will have to do some reading and experimenting.