If I were in charge of re-designing SageTV.....

[stuckless]

WARNING: Posting contains technical jargon not suitable for all readers!

I like SageTV. I've used many other Media Centers, and SageTV really does take the prize, especially when you factor in the cost effective media extenders. I've seen many posting on what people would like to see in the next SageTV, but my post is about what you can't see in the next SageTV.... ie, it's about the architecture and not the functionality. I started to write a media center some time ago, but I abandoned the project, once I stumbled upon SageTV. Most of what I've outlined here are ideas that I have implemented or started to implement in my own Media Center project. I've provided it here in hopes that Sage developers can look at this and maybe take some of these ideas for their next rewrite of SageTV.

What I would keep....

• Java (cross platform compatibility with Linux/Windows/Mac)
• Client/Server model (extenders, sage client, placeshifter, etc)
• All current features (pvr, recording, media, epg, etc)

What I would change....

• Use OSGI to better enable a plugin model for extending sage. OSGI would also enable better Linux/Windows/Mac bundles, enabling certain bundles to be enabled/disabled depending on the platform. OSGI could also expose things like the Player, Database, etc, as pluggable services, so that you could potentially change the player, database, etc, by using a different plugin.
  
• Dynamically loadable modules. Using the OSGI model, modules (media center, tv, pvr, online feeds, epg, etc) could be loaded and unloaded without restarting the sage service. Also dynamically loadabled modules would contain service registration, ui enhancements, etc. Everything in Sage would be a plugin. Sage would provide a TV plugin, and several implemenations, but the base TV plugin would simply be set of interfaces. This would allows other developers to add new TV plugins to support newer/different hardware tv tuner hardware. The DVD Player, Media Player, EPG, Online Videos, etc, would all be plugins with a base set of interfaces and implementations. This would also allow sage to publish a rich set of interfaces for each module without exposing their implementations. And they can continue to obfuscate their implementation code. Dynamically loadable modules would also enable people to customize thier SageTV so that they could have just the MediaCenter without requiring TV, Recordings, EPG, etc.
  
• Services. Sage is already a fairly well defined set of services. This is simply an extension to the Sage Services model and aligning it with the OSGI services model. In addition to this, it means enabling the http server bundle in OSGI, and exposing the Sage services as external services. This would allow for other user interfaces, such as web, flash, etc. An example of this, would include.... Let's say the current recordings are available as a services.... http://sageserver/CurrentRecordings/ , this could return a RSS feed, or custom XML that can be consumed by either a web application, flash application, another java application, etc.
  
• Bye-Bye STV and STVi. I'd replace the XML STV with a more modular xml UI layout. Potentially moving each “menu” out to a separate xml. I'd consider using xml, css, javascript to produce the functionality that STVs do today. By using xml, css, and javascript, you can have a clean separation of UI artifacts (xml), layout and style (css) and event handling (javascript). The side effect of choosing those technologies would be that there are many people that very proficient in xml, css, and javascript to produce applications. As an alternative to Javascript, I might choose to use Groovy.
  
• UI Extension Points and Overlays. Plugins could choose to add overlays to existing screens to add more functionality. In essence this would behave similar to firefox extensions. ie, Your overaly file could reference the “vidoes.xml” menu and add add a new button the “menu” container. UI would be built by loading the initial UI xml and the applying all overlays that are referenced in the plugins. This newly built UI would be cached. By applying the overlays on top of the original UI xml, it allows for easier loading and unloading of the plugin. (ie, Much like how FireFox and Eclipse handles plugin enhancements to the UI)
  
• Concrete UI Widgets. There are Panels, Tables, etc, but I think that there needs to be more concrete UI Widgets like CheckBoxes, Radio Boxes, Lists, etc. Creating a checkbox in a UI should be as simple as <checkbox value=”${stateVar}” checkedStyle=”” uncheckedStyle=”” label=”” onclick=”someJsOrGroovyFunction()”/>
  
• Bye-Bye Wiz.bin. I'd replace Wiz.bin with either Derby or Lucene. Lucene would be a good fit since it would double as storage and be indexed. This would make for very fast searches in Sage and would also enable 3rd party plugins to work against the open database. Ie, if you wanted to clean out your “database”, you could use an open tool to inspect your data, manipulate it, etc. I would probably provide the Database handling as a pluggable service. So that you could potentially override the default provided data plugin with a new one, such as derby, mysql, Wiz.bin, etc.
  
• Hello UpnP. I'd add UpnP support, both as a MediaRenderer (ie, remotely controlling another UpnP MediaServer) and a MediaServer (ie, enable it's content, recording, movies, etc, to be accessed by other MediaRenderers (XBMC, Xbox 360, PS3, etc)). There is development being done to add UpnP to some OSGI frameworks, so in theory, this could be done as a user extension, where the extension/plug-in provides an implementation for MediaRenderer and MediaServer.
  
• User Profiles. I'd add the ability to support User Profiles. There would be a "default" profile for people that didn't want User Profiles. User Profiles would also include User Access Controls (ACL). ACLs would prevent (in a superficial way) some users from accessing content that is not appropriate. User Profiles would enable different menu options for different user profiles. It would also track actions per profile. ie, if I pressed the watched button in my profile, then it would NOT mark it watched in my wife's profile. Optionally,pressing watch could provide a menu for you to mark it watched for other user profiles.
  
• Applets. Would be very cool to have embedded applet support. This would potentially allow you to play java games on your TV.

You can probaby see a theme here... OSGI. OSGI is an excellent choice on which to build a dynamic pluggable system, such as a PVR/Media Center.

Sage Developers.... I would love to see OSGI in your next platform release.

Good Luck,

[mickp]

Interesting. I have to admit that I'd not heard of OSGi until you brought it up. Are there any OSGi implementations in the htpc arena that would provide a compelling reason to move this way? A quick look at http://en.wikipedia.org/wiki/OSGi and http://www.osgi.org/About/Members didn't show up any current applications that made me think that OSGi would provide a quantum leap in capabilities. Admittedly I haven't yet looked at any of the detail.

Nice ideas re: splitting the stv into multiple files. This sounds similar to the way that some standards such as IMS content packaging works with submanifests referred to as required. This might simplify the process of loading stvi's in that if you installed three plugins a, b and c then decided to remove b you may not need to revert and reload. Someone who actually develops for sage/sagemc would be better placed to offer an opinion though.

I think that opening the database is generally a terrible idea. API's are a good idea in that they can be independent of the underlying technology. If you've given 3rd party developers direct access to the database then you've locked yourself into considering the impact that schema changes will have on existing plugins. As it stands Sage could decide to implement a different database technology and schema without affecting any 3rd party plugins and tools (afaik).

Mick.

[stuckless]

I have to admit that I'd not heard of OSGi...

I'm not surprised. It's a technology, while not new, has been mainly used in embedded devices.

Are there any OSGi implementations in the htpc arena that would provide a compelling reason to move this way?

NOt that i'm aware of.... but there are many OSGI enabled applications, some of which you are probably aware of, that are using OSGI in their space before anyone else. Take Eclipse, for example, the core of it's architecture is OSGI. I think they were probably the first to use OSGI in this way, and it's been hugely sucessful for them.

I think that if you ask yourself these questions in a project, then OSGI is a potentental good fit.
1. Do I want a modular system
2. Do I want to support a dynamically loadable plugin system
3. Do I want to enables services
4. Do I want a clean separation between services and their implemenations
5. Do I want to enable user contributed modules and or extensions
...

Using an OSGI framework to support this would be beneficial since it provides the framework and implemenation to add these abilities. Anyone that has ever written the code to support dynamically loadable/unloadable plugins, and dynamic classpath loaders, etc, will welcome the fact that this has been done for you.

The problem with OSGI is that it requires you to build software in a different way than we normally would think (or have been trained to think).

I think that opening the database is generally a terrible idea.

In retrospect of what I have written, I'd have to agree But my intention was not so much related to "opening" the database to enable direct access, but to allow for different persistent technologies to be used other than a closed Wiz.bin model. I think that all data access would be abstacted by the services (ie EPG, Media, Recordings, etc), so that the end user would not know or care about the persistent details. But, after reading it again, I'd have to agree that using a closed persistence model, does potentially solve some of the issues you have during upgrades, where you have schema changes, etc. On the flip side, I guess if you had a well defined set of interfaces for data access, you could enable 3rd party plugins to read/write data in a way that they don't need to rely on underlying persistence technology. This is similar to how Eclipse handles it's Preference Store.

[GKusnick]

I'm not sure why you put such a high premium on dynamic loading. My impression is that most Sage users (as opposed to plugin writers) don't do a lot of tinkering with their systems. They install a few plugins they like and then they mostly leave it alone. So I'm not seeing what practical benefit these users will get from dynamic loading.

Once you let go of the requirement for dynamic loading, then it seems to me a lot of what you want can be done in the existing architecture. Modular, layered UI designs using XML and CSS could be coded outside of Studio using browser-based apps, and then transformed or compiled into Studio-compatible XML for execution by Sage. Again, end users aren't really going to care what the underlying technology is or how the various modules get combined into a functioning UI, so long as it's relatively painless and automatic.

Similarly, nothing in the existing architecture prevents third-party devs from creating alternate UIs that aren't based on Studio at all (e.g. Neil's web server).

I also think it's rather unlikely that the Sage devs will consider it a good use of their time to completely re-engineer a mature product that's already highly extensible as is.

By the way, count me among those who had never heard of OSGI until you mentioned it. Without having delved too deeply into it, I'm not really seeing what sets it apart from other modularity frameworks like COM, CORBA, or loadable Java classes for that matter.

[stuckless]

Greg,

My posting was more of a blog than a "Hey Sage... Do this...", that's why it starts with "IF I..." I honestly don't expect Sage to re-engineer a product based on my personal experiences, especially a product as mature as SageTV. Keep in mind, if I were a Windows Dot Net evangelist, then my article would have been very different, and probably would have including things like COM, ActiveX, and C#

I totally agree that most, if not all, Sage users do not care about the actual technology, they just want it to work. Do Tivo users care that their box runs linux? Probably not. And the same is true of Sage users.... But my article was meant to look at the architecture, in a hypothetical way, and suggest an alternative. In a way, it's not much different than how SageMC compares to SageTV. SageMC looks as Sage and says, "hey, if I were to redo the Sage User Experience.... what would I do differently?" and before you know... SageMC is born

Similarly, nothing in the existing architecture prevents third-party devs from creating alternate UIs that aren't based on Studio at all

I agree... and that's where I mentioned that Sage is already a set of well defined services, and the addition of a http server in the core, is simply a logical extension of that. Personally I think that the Sage devlopers are very good as what they do and they are obviously open to new ideas. After all they chose to create a Java based media center when most people in the industry would have said that it was impossible to build a media center using java because "java is slow" or java isn't efficient, etc. Also the Sage developers chose to use an Xml layout at a time when XML was popular, but not really being used in that way. Also, the Sage developers basically implemented some CSS concepts into the STV when leads me to believe that they must understand the value of using such a content separation model. I posted about OSGI because I know the Sage developers are very competent developers and they may not know about OSGI.... but they do now And they are certainly not afraid to push the envelope when it comes to technology.

I wouldn't dismiss the OSGI technology, just because you haven't heard about it. You have heard about Eclipse and RCP and it's foundation is OSGI. At some point in time, you didn't know about Java and Xml, but they are key technologies the Sage platform. Just think how your programming experience would be different if you dismissed Java and Xml just because you hadn't heard about it before.

Maybe you interpreted my email as a criticism to Sage developers because they didn't choose OSGI. That is not the case. I can't expect them to use a technology that didn't really exist when they started out. I simply posted an article about what I would do differently. I'm willing to bet that most of the core Sage developers would do something different given the state of technology today, than when they started the project. Because they would choose to do something different doesn't mean it was done wrong in the first place.

Sage will continue to innovate, and I wouldn't be surprised to see changes in thier core technology over time. In fact I'd probably be a bit disappointed if in 5 years, SageTV was still the exact same product.

[jsonnabend]

The OP was a good read. Add me to the list of those who hadn't heard of OSGi (then again, I'm mostly a Windows Delphi programmer, very old school indeed).

The open database is not only a good idea, it's a great idea. The closed nature of wiz.bin always scares me, and it makes data manipulation a pain in the back side. Open the DB back end and programmers will come up with some neat add-ins, no doubt. (and for that matter, name one other application of this sort that runs its own database engine).

- Jeff

[stuckless]

I think that most people, unless you spend your time time in Java/Eclipse and/or embedded devices probably haven't heard about OSGi. But, in case people think that OSGi is not a "real" technology

There is Eclipse
http://www.eclipse.org/equinox/

And Apache
http://felix.apache.org/site/index.html

And Google (Well not quite Google, but OSGi can run on Android )
http://www.infoworld.com/article/08/...Android_1.html

And Sun (Glassfish at least)
http://blogs.sun.com/dochez/entry/gl...3_runs_on_osgi

And IEEE has a paper that you can't read (unless you are member, I'm not)
http://ieeexplore.ieee.org/Xplore/lo...number=4341564

And yes, surprisingly enough there is actually a media center (go figure) using OSGI (not that I was aware of it before writing this article )
http://mediacenterlinux.org/

The list goes on... and on.... especially if you start including embedded solutions....

[davephan]

I would port parts or the entire application to a much faster Language. Java is very very slow.

Dave

[stuckless]

Not biting..... Must.... resist.... the.... urge... to... defend.... Java....

[GKusnick]

If this is just a hypothetical here's-how-I-would-have-done-it, then fine; I don't see much to object to in the original post. However your closing line ("I would love to see OSGI in your next platform release") gave me the impression that you're lobbying for a major architectural change in the actual product, and that's what I was reacting to.

If I had free rein to redesign Sage (and get someone else to do the coding), there are a lot of things about Studio, widget language, and the STV file format that I'd do differently. (I'd probably use XSL templates to transform abstract menu structures into concrete UI widgets at STV load time, to name just one idea.) But I've found it more productive to focus on how far I can go within the existing framework, and quite often that turns out to be a lot farther than I thought.

So I guess my point, to the extent that I have one, is don't let new tech envy stop you from doing cool stuff with the existing tech.

[sleonard]

Originally Posted by davephan

I would port parts or the entire application to a much faster Language. Java is very very slow.

Dave

Originally Posted by stuckless

Not biting..... Must.... resist.... the.... urge... to... defend.... Java....

The SageTV core is actually C++ not Java. Only the UI and Studio are Java.

S

[mickp]

Originally Posted by jsonnabend

Open the DB back end and programmers will come up with some neat add-ins, no doubt. (and for that matter, name one other application of this sort that runs its own database engine).

- Jeff

So far as I can see that would include Snapstream BTV (which used xml files last I checked) and Microsoft MCE (it's probably jet based though I guess) which someone appears to have written a tool for.

I agree that it would be nice to have this data available in a known format. I disagree that it would be a good thing for developers to access the data directly.

Would you write an Exchange Server application that accesses the .edb files (or whatever they're using in Exchange 2007 now) directly? I wouldn't. I'd make sure that the supplied API was used to retrieve and manipulate data. This would ensure that I could expect not to need to rewrite my application/plugin with each update of the platform.

As a software vendor would I be happy to provide support to customers who have been bypassing my business objects and writing directly to the db? Probably not.

Mick.

[davephan]

Originally Posted by sleonard

The SageTV core is actually C++ not Java. Only the UI and Studio are Java.

S

That's surprising to me...

The part of SageTV that I would fix is the part that seems to have bottlenecks in the code. For example, when searching for programs. As each letter is typed, the result field narrows. The search results are displayed very slowly compared to they way it performed a couple of years ago even when using a much slower computer at that time. I also notice a lot of spinning circle waiting symbols in other program areas, even with a fast dual-core. If the user interface is causing the slow down, maybe that should be replaced with a faster language than the very slow Java.

Dave

[mickp]

Originally Posted by davephan

the very slow Java.

Hehe.

Mick.

[jsonnabend]

Originally Posted by mickp

So far as I can see that would include Snapstream BTV (which used xml files last I checked) and Microsoft MCE (it's probably jet based though I guess) which someone appears to have written a tool for.

I agree that it would be nice to have this data available in a known format. I disagree that it would be a good thing for developers to access the data directly.

Would you write an Exchange Server application that accesses the .edb files (or whatever they're using in Exchange 2007 now) directly? I wouldn't. I'd make sure that the supplied API was used to retrieve and manipulate data. This would ensure that I could expect not to need to rewrite my application/plugin with each update of the platform.

As a software vendor would I be happy to provide support to customers who have been bypassing my business objects and writing directly to the db? Probably not.

Mick.

I hear you, and in many cases, you're probably right, but . . .

We're not talking about enterprise level email systems, so I think that example is besides the point. On the other two, they both sound open to me, no?

I agree that directly accessing the underlying db can be dangerous and must be done carefully, but we're talking about tv program/recording (meta) data here, not exactly the stuff of rocket science.

- Jeff

[stuckless]

If the user interface is causing the slow down, maybe that should be replaced with a faster language than the very slow Java

Maybe the same natively coded UI that Vista uses... I hear that's quite fast

Seriously though... I would think that the slowdown in the search screen is more likely related to a poor index in the Wiz.bin data model. But since it's a closed database, we'll never know. Lucene could probably help with fast indexing and it could still be used in conjunction of the Wiz.bin. I've done searches across ~6.5 million records using Lucene, and I've seen results come back in less than 20ms.... So I think the folks in the Lucene project have optimized it fairly well. If OSGi was used, and the data persistence engine was an open set of services, then someone could/would have written a faster data search/persistence engine.

As for the comment about about the only parts of Sage being Java is the UI and Studio.... I'm not sure if Wiz.bin is being handled natively or in Java. I would think that all performance intensive parts like decoding video for playback, and transcoding would be done natively, but there's no reason for the data model to be handled natively. Handling Tuners would obviously be done natively as well.... Which is a similar model as JMF and JTV. In my original post I mentioned Java as being a part that I would not change... I probably should have put Java/Native since I would continue to use mplayer as the movie engine and native tuner handling such as linuxdvb.

As a software vendor would I be happy to provide support to customers who have been bypassing my business objects and writing directly to the db? Probably not.

Support them? No. But enabling a user to do something and supporting them are 2 different areas. If sage used an embedded DB, like berkly db (not that I'd recommend it personally), then it would enable people to gain access to the data and enable existing tools to work with the data, but I wouldn't support them. Again, my comment about opening the data persistence is more about providing different persistence options and not really about enabling end users to "backdoor" data. I think all data inputs and outputs should go through standard apis.

[m1abrams]

Surprised no one mentioned a solution for the filesystem. Yes currently 64K NTFS gets by, but really it just barely gets by when handling 4 or more tuners. Faster raid arrays help but a file system with larger clusters would help even more. Sure having a non-standard for windows filesystem means that you would have to dedicate a partition for your recordings, but we pretty much have to do that anyhow so not sure requiring that is too big a leap. With so many good free tools for resizing partitions this should not be that difficult to even add to the installer.

Other thing that needs work, the model used for tuner guide service. I need to be able to have custom channels for each tuner, as it stands I have to hunt for local "similar" cities guides to use for each tuner. Also a system to NOTIFY me when my channel lineup changes would be nice too.

[Opus4]

Originally Posted by davephan

For example, when searching for programs. As each letter is typed, the result field narrows. The search results are displayed very slowly compared to they way it performed a couple of years ago even when using a much slower computer at that time.

You may be mistaking the delay waiting for input to cease before doing the search as if that were an actual slowdown during the search -- it waits for you to stop typing before doing the search. The amount of time it waits decreases the longer the search string becomes. The purpose of this is to allow the user to continue entering the search text w/o having to wait for the search to update. And, yes, that could be an issue on the old search menus if there were a lot of things to search, but on the other hand, the old search menus didn't let you see the results for 'description' and 'all fields' searches on the text-entry screen, both of which take longer to search.

But, if you are seeing a long delay after the timeout period, then disable the one-touch NTE search, since that uses regular expressions and is slower.

And searching can be slower depending on what type of search you are doing -- searching 'all fields' is slower than just searching 'title', for example.

If you really want to see what searching is like w/o the input timeouts, then check out the search/delayXXX properties, though I don't really suggest changing them.

- Andy

[stuckless]

Greg,

If I had free rein to redesign Sage (and get someone else to do the coding), there are a lot of things about Studio....

I think that as a Studio Guru, you could offer a lot of great ideas to the SageTV folks. And the thing that I've learned in this posting is that you can't always assume that other people are currently looking at the same great technologies that you know.... so it's important to share good ideas to encourage others to look at newer technologies, newer ideas, etc. I think your XSL idea is a great Idea, and personally, that and other xml enhancements is something I'd like to see in the toolset. I think that in some cases, using a simplified XML + XSLT to produce a "menu" page faster than using studio or by doing it by hand. I doubt that the STV xml will every go away, but that doesn't mean that other simpler tools can't be expanded to make working with the STV a lot easier. Bottom line, suggesting an enhancement, or different way of doing something should not be seen as criticism.

So I guess my point, to the extent that I have one, is don't let new tech envy stop you from doing cool stuff with the existing tech.

I agree... that why I'm in there, almost daily, looking at the STV xml and trying to figure things out. I'm working on a couple of enhancements.

[Slipshod]

Originally Posted by stuckless

Not biting..... Must.... resist.... the.... urge... to... defend.... Java....

I'm weak.... I can't resist the call. I know I should, but I can't...

Davephan - When Java was originally introduced it was severely slow. It's matured significantly since then, and is very competitive with other languages on speed. C# is largely a wash, and while C and C++ can be faster in certain ways the difference is not terribly significant for a large interactive application (as opposed to a small number-crunching benchmark). You could even argue that it frequently can be faster than C/C++ because you don't have to be as skilled a programmer to have fast, optimized code.

Cheers,
Slipshod