SageTV architecture overview

[wnjj]

With all of the discussion about native code, JARs, STV's, core vs plugins, miniclients, etc., I came to realize how well I don't understand the whole system.

Is there somewhere that describes how this all works?

What would be nice is an overall description about what happens on the server versus an extender or PC client, including which pieces are running from native (C++) code and which are java.

Maybe this is asking too much if there are too many options but with at least a basic understanding I'd be able to poke through the code to figure out more of the details. I already spent some time looking at Wizard.java to start to build an understanding of that age-old "mystery" wiz.bin file.

I have some experience with C/C++ in Visual Studio and have 2005 Pro, 2013 Pro and 2015 community editions available and may find myself wanting to play with this stuff. I've never written any java but it looks close enough to C++ to pick up.

[Fuzzy]

Most of what sage does is Java, and most of that is in a single file, sage.jar. That file is launched differently depending on what task is is supposed to be doing. It can be a headless server (service mode), it can be a UI (client mode), it can do both at the same time (standalone mode), it can be an IDE (studio mode). On windows, there are launchers written in c++ to fire off sage.jar in whatever mode that particular .exe is written for. sage.jar also pulls in other java .jar's to extend it's functionality (mostly plugins, some core libraries). sage.jar also makes calls to native libraries to handle system specific calls (like rendering the UI, receiving ui inputs, recording from local tuners, etc).

Extenders/placeshifters are a different beast. They use a different .jar called miniclient. miniclient connects to the server's sage.jar, and renders the UI that is built in the server's process, plays back media, and recieves control input. It uses various amounts of native calls, depending on the system it is running on.

[stuckless]

I do agree that an achitecture diagram, from a high level would be a good idea. I also think that having a lower level architecture diagram would be good as well.

The SageTV devs seems to have some fun in their Class naming, so we have classes, like Catbert, BigBrother, Carny, Ministry, Wizzard, etc... I'm sure their meanings are relevant to what they do, but I haven't really dug deep enough to figure them out

Either way I think it would be good to get some extra documentation around things. For example we alll know at a high level how some things work, but it would also be nice to see a flow diagram of how a client connection is handled, and how a video request is processed, ie, what classes/methods handle a video request, which parts delegate to native calls, etc. Hopefully over time, all this stuff will become clearer, to everyone... but until then, we just post questions here, and hope that Jeff answers them (which he usually does )

[Narflex]

I wish I had something I could share that outlines the whole architecture better, but I don't currently. It'd take me quite awhile to come up with something....so at this point I'm leaning towards just answering all the questions people ask. Of course feel free to ask me specific or even somewhat general questions and I'll do my best to answer them....such as if Sean wants explanations for the few things he mentioned in his post.

Fuzzy,

One correction...the miniclient on the extenders is not a JAR file. It's a C program. There is also a Sage.jar on the HD200/HD300 which is built from the same codebase as the x86 version and is used for everything but actual UI rendering and media playback (and as you probably know, the miniclient uses the same protocol as the placeshifter, and of course the placeshifter is Java based...and of course the miniclient can placeshift too )

[Fuzzy]

yeah, I had mentioned that various amounts of the individual miniclients were native code.. didn't actually clarify that in some cases, it's 100% of it.. :-)

[tmiranda]

Originally Posted by stuckless

The SageTV devs seems to have some fun in their Class naming, so we have classes, like Catbert, BigBrother, Carny, Ministry, Wizzard, etc... I'm sure their meanings are relevant to what they do, but I haven't really dug deep enough to figure them out

I think we can figure this one out:

    // Clear out all of the negative energy, this includes marked waste
    Wasted[] stoners = wiz.getWasted();

[Narflex]

Originally Posted by tmiranda

I think we can figure this one out:

    // Clear out all of the negative energy, this includes marked waste
    Wasted[] stoners = wiz.getWasted();

Lol...I guess I missed that one when scrubbing the code.

Wasted is the equivalent of 'Don't Like' in the system...while it's a funny name, conceptually it actually refers to it being a 'Wasted' recording because the user didn't watch it. SageTV will create a Wasted object when you mark something Don't Like...AND it will create one if it records something and then deletes it before it gets watched (so this is negative feedback in the intelligent recording system). Wasted has a flag for 'manual' waste (which means the user marked it don't like) so we can treat those more strongly than when the system automatically infers Wasted.

[tmiranda]

Originally Posted by Narflex

Lol...I guess I missed that one when scrubbing the code.

Wasted is the equivalent of 'Don't Like' in the system...while it's a funny name, conceptually it actually refers to it being a 'Wasted' recording because the user didn't watch it. SageTV will create a Wasted object when you mark something Don't Like...AND it will create one if it records something and then deletes it before it gets watched (so this is negative feedback in the intelligent recording system). Wasted has a flag for 'manual' waste (which means the user marked it don't like) so we can treat those more strongly than when the system automatically infers Wasted.

Thanks. I was digging through the code in that area because I'm interested in adding a "never record" option. A wrote a plugin for that but still think it may be something that can go in the core.

[wayner]

On a somewhat related note it would be nice if there was a way of not recording an airing that is a Favorite that is about to record or is actually in the process of recording. Or maybe I am missing something and you can do this today.

[KryptoNyte]

You can, wayner, but you currently have to mark the show as watched. Someone will likely come up with a better way to do this.

[wayner]

Originally Posted by KryptoNyte

You can, wayner, but you currently have to mark the show as watched. Someone will likely come up with a better way to do this.

I think I have done that in the past and it doesn't always seem to work.

[david1234]

Originally Posted by stuckless

The SageTV devs seems to have some fun in their Class naming, so we have classes, like Catbert, BigBrother, Carny, Ministry, Wizzard, etc... I'm sure their meanings are relevant to what they do, but I haven't really dug deep enough to figure them out

This is funny... reminds me of my last job where they named all the classes in a distributed search algorithm metaphorically as a fishing tournament. On my first day at work, I'm a gazillion layers deep in the debugger, and then all of a sudden I fall through the front end code into a BassBoat with a couple of GameWardens.

It was maddening, and I couldn't make any sense of what was going on. Good times

[KryptoNyte]

Originally Posted by wayner

I think I have done that in the past and it doesn't always seem to work.

I use it a few times a week, never had a problem. It does take Sage a short bit (20 seconds I suppose) to realize that it will not record the show and reflect that status.

That said, it's a good suggestion to try and get in a better method if possible.

I have to believe that the scheduling system is probably some of the most complex parts of the code. Seems like it would be easy to break something in there.

[Tiki]

Originally Posted by KryptoNyte

I use it a few times a week, never had a problem. It does take Sage a short bit (20 seconds I suppose) to realize that it will not record the show and reflect that status.

That said, it's a good suggestion to try and get in a better method if possible.

I have to believe that the scheduling system is probably some of the most complex parts of the code. Seems like it would be easy to break something in there.

I often do this as well. Another trick is to schedule a manual recording for some time in the future to force a favorite to record at a more convenient time. However, it often takes quite a while (10-20 minutes) before the "unwanted" favorite will get removed from the schedule.

[wayner]

Originally Posted by KryptoNyte

I use it a few times a week, never had a problem. It does take Sage a short bit (20 seconds I suppose) to realize that it will not record the show and reflect that status.

Maybe it is the delay that gets me. And I really don't used the Watched status at all since most of our recordings are Kids' Shows that get watched repeatedly. Or movies that we want to keep and watch more than once.

[Fuzzy]

Yeah, the issue is that once it gets flagged as watched, or you mark a future recording as manual, for instance, it still takes a whole run through the scheduler to reprocess before the originally scheduled airing is cleared from the schedule. This can take quite a while, depending on your scheduling lookahead, number of favorites, number of channels, and server processor speed.
Even if you gave the airing a super-mega-wasted tag, it still wouldn't be reflected immediately in the schedule, until a new schedule is rehashed.

[BobPhoenix]

It depends on your number of favorites or so I believe. It often takes 10 minutes to stop for me and I do that often as well. The fastest way I've found to stop it is to make it a manual recording then make it watched and then cancel the manual recording. I can usually get it under 5 minutes that way depending on the server. On my oldest server it can take more than 10 minutes to stop something once the recording as started upto 20 minutes (which if the show is 30 minutes and you don't notice it right away means you still get the whole thing anyway). On my newest server that only records OTA it usually stops in a minute or less. Another way I've done it is to still mark it watched but also start a recording of something else so that the only other available tuner (HD-PVR I have two on one server) is being used then start a show live to get the original favorite to change to the new show that is being watched live. Then once it switches I cancel both manual recordings and they usually stop in the 20 second window talked about here.

So I would like a way to kill a favorite that is recording myself. So much so I was going to write my own favorites processor and it would make the recordings manual recordings in SageTV so that I can cancel them easier. I also was going to add the ability to record the same episode or show multiple times up to a set number to allow me to get a good recording off of bad reception or around weather warnings. And a few other things like that. Problem is I never seem to have the time.

[Taddeusz]

Originally Posted by david1234

This is funny... reminds me of my last job where they named all the classes in a distributed search algorithm metaphorically as a fishing tournament. On my first day at work, I'm a gazillion layers deep in the debugger, and then all of a sudden I fall through the front end code into a BassBoat with a couple of GameWardens.

It was maddening, and I couldn't make any sense of what was going on. Good times

Wow, someone there must have wanted to ensure their job security.

[Narflex]

Originally Posted by KryptoNyte

I have to believe that the scheduling system is probably some of the most complex parts of the code. Seems like it would be easy to break something in there.

Yes, you would be correct...it's the support for multiple different lineups that made the problem very difficult to solve. It's straightforward if the lineups are all the same..but when they are not...the problem becomes NP hard.

And a lot of the variance in delay is because the Scheduler probably just needs to get kicked. When you mark something watched; that'll kick Carny (which does all the Favorite/intelligent recording analysis, and when that's done it'll kick the Scheduler). There is code in the Scheduler to exclude Watched items from the schedule, so Carny doesn't need to complete for that to get enforced...but the Scheduler does need to get kicked by some means.

Carny can take many hours to complete (mine takes almost a day because I have 13 years of history in my DB)...the Scheduler should complete it's calculations in seconds generally though.

[KryptoNyte]

Jeff, is your SageTV server installation currently running on a solid state drive or a conventional hard drive? ... or anyone else for that matter ...