Library: Remote APIs for SageTV

[Slugger]

Any chance that the JSON calls could be auto serialized and sent back in one big JSON array with everything encoded in one request? For example, this call:

http://server/sagex/rpcJson/Global/GetScheduledRecordings

What I was hoping for in response was a giant JSON array of JSON objects, where each JSON object was an encoding of an underlying Airing object (title, start time, end time, etc.).

My web apps currently do AJAX requests to my servlet, which sends back a giant JSON encoding as described above and displays it or whatever I do with it. I'm starting to see a recurring pattern across my apps and so am going to refactor all of this JSON data passing into a common, standalone servlet, but thought I'd investigate your package a little closer. Admittedly, I shouldn't be doing such investigations at 1:30am, but here I am.

I'm able to get the data, but to get, say, all the entries from the scheduled recordings array would require hundreds of calls to your remote API - just to construct all of the data of a single airing for display in an HTML table would be 8-10 calls - so to try and construct a displayable view of the scheduled recording array from the Global API would probably be way too slow. On the other hand, if your REST call from above did all the work in one shot and returned a giant JSON encoding of the array then I'd be set. Is this something possible? I tend to think it's not really what this API is trying to accomplish, but figured I'd ask before starting my own refactoring exercise.

[stuckless]

Out of the box... it will not return all data as you've indicated. But.... The core of the remote-api is a generator tool, and the implemenations so far have taken the easy road, which does the simple 1-1 mapping of the api.

You can probably auto-generate an entire json library that does what you need, and I've considered doing it for xml and json in the past. It's not something that I have the time for now unfortunately.

One of the biggest challeneges for the json stuff is how'd you handle arrays. For example, if you call the GetMediaFiles, then you could get back several thousand items that will each contain show and airing entries.... This json string would be very large, and would probably take some time to create and proably just as long to evaluate on the reciever.

If you'd like to do this via a generator in the remote apis project, then I'd be ok with that. I think it would be a good fit. I don't mind helping out, but I can't really dedicate much time to doing the actual work

Good luck.

[stuckless]

This is a new release that refreshes the api to the lastest 6.5. It also adds a new remote api handler, simply called, /sagex/api, that aims to be the "version 2.0" of the remote apis.

If you invoke the handler /sagex/api without any parameters, you'll get a help page for the api usage, and a searchable index for all sage commands.

The significant things about this new handler is that it internally provides support to return the results in one of 3 formats. The first is a simple xml serialization, the second is json/jsonp, and the third is Niel's Xml format.

Sage Objects are NOT returned as references using this Api. ie, if you call GetMediaFiles, you'll get an Array back of All Sage Media Files (and children) serializes as Xml or Json.

For this reason, a couple of new parameters were added. 'start' and 'size'. start indicates the array position that you want to start at, default 0, and size indicates how many items to return, 50 is default. So, if you call GetMediaFiles without any parameters, then you'll get back the first 50.

I think this new handler is probably better suited for web applications that need to get access to a "complete" media file, including it's show and airing objects in a single call.

As a side note, if you want to do complex operations, such as sort the data on the server side before send it back as xml/json, then consider using the Sage Command 'EvaluateExpression' and nest your calls together...
ie, something like,

http://mediaserver:8081/sagex/api?c=EvaluateExpression&1=Sort(GetMediaFiles("T"),false,"Intelligent")

I haven't removed any of the existing handlers, but in the future, I'll probably remove all other xml/json handlers in favour of the this single multi purpose handler. (the java handlers would remain)

Enjoy

[babgvant]

Thank you! This change makes it much easier to consume data.

I just started playing with it and found an issue; it's definitely one of my files (narrowing the scope of items returned works around it).

Calling /sagex/api?c=GetMediaFiles&start=15&size=1 (/sagex/api?c=GetMediaFiles&start=1&size=1 works fine)

<error>
−
<message>
Failed while Calling MediaFile Method: GetAlbumForFile
</message>
−
<exception>
java.lang.Exception: Failed while Calling MediaFile Method: GetAlbumForFile
at sagex.remote.builder.SimpleXmlBuilder.handleError(SimpleXmlBuilder.java:38)
at sagex.remote.builder.SageAPIBuilder.buildObject(SageAPIBuilder.java:97)
at sagex.remote.builder.SageAPIBuilder.buildMediaFile(SageAPIBuilder.java:86)
at sagex.remote.builder.SageAPIBuilder.build(SageAPIBuilder.java:43)
at sagex.remote.builder.SageAPIBuilder.buildArray(SageAPIBuilder.java:151)
at sagex.remote.builder.SageAPIBuilder.build(SageAPIBuilder.java:41)
at sagex.remote.api.XmlReplyEncoder.encodeReply(XmlReplyEncoder.java:26)
at sagex.remote.api.ApiHandler.hanleRequest(ApiHandler.java:151)
at sagex.remote.SagexServlet.doPost(SagexServlet.java:71)
at sagex.remote.SagexServlet.doGet(SagexServlet.java:49)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:362)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:729)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:206)
at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:505)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:829)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:513)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:211)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:380)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:395)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:488)
Caused by: java.lang.Exception: Failed while Calling Album Method: GetAlbumArt
at sagex.remote.builder.SimpleXmlBuilder.handleError(SimpleXmlBuilder.java:38)
at sagex.remote.builder.SageAPIBuilder.buildObject(SageAPIBuilder.java:97)
at sagex.remote.builder.SageAPIBuilder.buildAlbum(SageAPIBuilder.java:74)
at sagex.remote.builder.SageAPIBuilder.build(SageAPIBuilder.java:49)
at sagex.remote.builder.SageAPIBuilder.buildObject(SageAPIBuilder.java:95)
... 24 more
Caused by: java.lang.Exception: Unknown Object Type: sage.cq
at sagex.remote.builder.SimpleXmlBuilder.handleError(SimpleXmlBuilder.java:38)
at sagex.remote.builder.SageAPIBuilder.build(SageAPIBuilder.java:53)
at sagex.remote.builder.SageAPIBuilder.buildObject(SageAPIBuilder.java:95)
... 27 more
Caused by: java.lang.Exception: Unknown Object Type: sage.cq
... 29 more
</exception>
</error>

[stuckless]

Thanks, babgvant. While I'm walking the Sage Tree, there are a number of Sage Methods (especially related to Images) that I had to exclude. Obviously I missed this one. I'll do a quick fix tonight and post -2 version.

[babgvant]

Perfect, thanks.

[stuckless]

I fixed the serialization bug.... I was able to reproduce and confirm the fix in my environment, so hopefully it should be ok now.

[babgvant]

Fixed. Thanks again!

[rickw]

I'm a .Net developer and don't have the time to learn Java, so this framework really opens a new world. So I really appreciate your work in this regard.

I like the new style of the api and especially the fact that it returns the richer xml instead of the ArrayRef. But I have a few questions.

1) XML returned starts with a <result> as the root and not the <response> with <header> and <body> as documented on your wiki. Is this how you see it staying going forward? I don't have any opinion either way, but I'll have to write a bunch of deserialization code and would prefer not to write it twice. :-)

2) Are the element names pretty well set? Specifically I'm wondering about the "Get" prefix on most of the elements returned and whether you plan to keep the prefix. (Again, no real opinion on my part, just a question)

3) In a related question: Is the &encoder=nielm usage complete? I'd already written some code to deserialize the <sageShowInfo> using nielm's webserver so I'm thinking I'll use it.



But, in testing from the browser if I do this query:

/sagex/api?c=GetMediaFiles&encoder=nielm

I get the <results> XML

If I add the filter parameter to GetMediaFiles

/sagex/api?c=GetMediaFiles&1=T&encoder=nielm

I get the <sageShowInfo> XML.

Again, thanks for the opening up the world of Sage API.

Rick...

[stuckless]

1) XML returned starts with a <result> as the root and not the <response> with <header> and <body> as documented on your wiki. Is this how you see it staying going forward? I don't have any opinion either way, but I'll have to write a bunch of deserialization code and would prefer not to write it twice. :-)

My apologies for not updating the documentation yet.... I will... but it may take a few days....

The sagex/rpcXml and this new sagex/api are different types of apis. The new api is more of a serialization api, whereas the original api was a mix of serialization (for primitive types) and object references. I learned alot doing the first approach, but I think this new approach hold many other benefits. That being said, I'll leave in the original rpcXml for now, but moving forward, the xml that is generated from the new api should not change.

The response is designed so that the Xml and Json serializers return the same structure. Using the old style, you'd need to traverse a couple of levels to access a "real" object, which I found annoying, so I flattened the replies. Consider this the 2.0 approach... which I will document at some point.... for now, I included a minimal documentation page as the return page when you call /sagex/api without any parameters.

2) Are the element names pretty well set? Specifically I'm wondering about the "Get" prefix on most of the elements returned and whether you plan to keep the prefix. (Again, no real opinion on my part, just a question)

I think that this goes back to #1, where I said I wouldn't change the xml But I can see where removing "Get" and "Is" would be appropriate. I had thought about removing the "get" as well... so maybe I'll just drop that in the next build (for both json and xml)

3) In a related question: Is the &encoder=nielm usage complete? I'd already written some code to deserialize the <sageShowInfo> using nielm's webserver so I'm thinking I'll use it.

We'll I'll be honest... I have not idea. It's as compelte as Niel made it. It works much like the default xml/json serializers.... I pass in an Object and I get back a result. I have no idea if the result returned is correct or not. I can tell you that if you specify the encoder as 'nielm' and you get back the default Xml format, then the Nielm encoder failed while building the output. In the case of failure, I decided to reply with some serialized data as opposed to nothing at all..... I can change the behaviour of the nielm encoder to return an error if it encouters one.

[rickw]

My apologies for not updating the documentation yet.... I will... but it may take a few days....

No apology neccesary. I completely understand.

but moving forward, the xml that is generated from the new api should not change.

Works for me.

I think that this goes back to #1, where I said I wouldn't change the xml But I can see where removing "Get" and "Is" would be appropriate. I had thought about removing the "get" as well... so maybe I'll just drop that in the next build (for both json and xml)

I think the xml would look cleaner without the 'Get'. Don't have a clue if it works/looks better in json. But, I kinda like the 'Is' for boolean flags.

I can tell you that if you specify the encoder as 'nielm' and you get back the default Xml format, then the Nielm encoder failed while building the output. In the case of failure, I decided to reply with some serialized data as opposed to nothing at all..... I can change the behaviour of the nielm encoder to return an error if it encouters one.

Returning the error makes sense, because if a client app expects the nielm encoder format and gets back another format then the client app will fail anyway.

Thanks,
Rick...

[stuckless]

Change Summary

• Xml/Json Node Name changes
• Nielm Encoder will now return an error if it encounters an error

This release includes a major xml/json change. I don't anticipate any other xml changes, but I felt this change was worth doing. The Change, means that all xml elements that started with "Get" not have the "Get" dropped. So a field like <GetMediaFileTitle> will now look like <MediaFileTitle>

Something important to note about the nielm encoder. It can only encoder sage object (mediafile, show, airing, etc) and collections of those objects, but it cannot encode strings, ints, booleans, etc. So if you call, GetOS with the encoder as nielm, then you will get an error back stating that the encoder cannot handle "string" types. So, only use the nielm encoder on Sage Commands that you know will return a valid type for the encoder.

[rickw]

I'm guessing the answer is "No" to this question, but asking anyway. Is it possible to call the REST /sagex/api methods that require a object for a parameter, like GetMediaFileForAiring which takes a sage.Airing object? If so then how are the object encoded?

Right now I'm working through different ways generate a .Net library from the /sagex/api help. I've parsed the returned HTML to get the list of methods, so now I'm just trying to figure out if some will have to be ignored.

Thanks,
Rick...

[stuckless]

If you were willing to learn a bit of java.... you could write a Generator within the remote apis... The core of the project is based on a generator that parses the SageTV documentation and builds the various outputs. One developer used it to generate a set of tag libraries for jsp development.

But... on to your question.....
If you know the Object id of the object you are passing as a parameter, you can pass it as, mediafile:### or airing:###, it's been a while since i've been in that part of the code, but I think that's how it works. If not.. then that's how it should work, so I'll fix it if it's broken

[MarkV]

I plan to go thru the whole thread tonight, but is there information available from this api that isn't available via nielm's webserver? I currently access nielm's webserver remotely and am able to pull the entire media library via xml. The data gets processed using javascript on a pronto remote That allows me to build a remote list of every media asset and associated MediaFileID. Via nielm's webserver, I think I can control about every aspect of sage. Am I missing out on some must-have features and am just too ignorant to know any better?

[stuckless]

Originally Posted by MarkV

I plan to go thru the whole thread tonight, but is there information available from this api that isn't available via nielm's webserver? I currently access nielm's webserver remotely and am able to pull the entire media library via xml. The data gets processed using javascript on a pronto remote That allows me to build a remote list of every media asset and associated MediaFileID. Via nielm's webserver, I think I can control about every aspect of sage. Am I missing out on some must-have features and am just too ignorant to know any better?

I'll save you some time.... you are not missing anything . If you can currently call every sage api using neilm's web server, and you are happy in how that is working for you, then I see no reason the dig into this api.

Basically I wanted to call the sage tv api remotely from java, so I created this api. That being said, since the entire API is autogenerated from the SageTV documentation, then I also added in remote api calls that work for Xml and Json as well. Json, probably more so than the Xml format, holds some promise for someone wanting to make the calls via javascript.

And, the api also supports sending back the data in nielm's format as well, which is redundant to the webserver, but may be helpful in some cases.

I wish there were of mystery here... but in the end, it's just a remote api, similar to the web server api.

[Slugger]

Just a friendly security reminder to those who've installed this API in Jetty using the Jetty context file posted on the project wiki...

The posted context file does not require any authentication and is wide open to anyone able to connect to the servlet. If your Jetty installation is exposed to the Internet (perhaps for the mobile web server or other plugins running from Jetty) then anyone who stumbles upon the remote API servet has complete, unregulated access to all aspects of your SageTV system. Not hard to imagine the kind of damage this could cause:

http://your_jetty_server:jetty_port/sagex/api?c=EvaluateExpression&1=DeleteFile...

Easy to see that filling in the details on the above URL against an Internet-exposed, unprotected remote API installation can cause a lot of damage rather quickly.

Below is an alternative context file that requires all calls to the API to be authenticated against the configured Jetty user (as defined at Jetty plugin installation time). If you're not connecting to the servlet via SSL then the id/pwd are transmitted in plain text, but that's at least a lot better than nothing at all. If your Jetty installation is protected by a firewall then this isn't as important, though I'd still recommend it. For those programming against their installed remote API, you will need to add basic authentication to your remote calls, but most HTTP APIs make basic authentication relatively straightfoward so I really do recommend this added level of protection whether or not your Jetty installation is exposed to the net.

<?xml version="1.0"  encoding="ISO-8859-1"?>
<!DOCTYPE Configure PUBLIC "-//Mort Bay Consulting//DTD Configure//EN"
"http://jetty.mortbay.org/configure.dtd">
<Configure class="org.mortbay.jetty.servlet.Context">
   <Set name="contextPath">/sagex</Set>
   <Call name="addServlet">
      <Arg>sagex.remote.SagexServlet</Arg>
      <Arg>/*</Arg>
   </Call>

   <!-- Let's attach a security handler to prevent wide open access to the SageTV system -->
   <Set name="securityHandler">
      <New class="org.mortbay.jetty.security.SecurityHandler">
         <Set name="userRealm">
            <New class="org.mortbay.jetty.security.HashUserRealm">
               <Set name="name">SageTV Web Interface</Set>
               <Set name="config"><SystemProperty name="jetty.home" default="."/>/etc/realm.properties</Set>
            </New>
         </Set>
         <Set name="checkWelcomeFiles">true</Set>
         <Set name="constraintMappings">
            <Array type="org.mortbay.jetty.security.ConstraintMapping">
               <Item>
                  <New class="org.mortbay.jetty.security.ConstraintMapping">
                     <Set name="constraint">
                        <New class="org.mortbay.jetty.security.Constraint">
                           <Set name="name">BASIC</Set>
                           <Set name="roles">
                              <Array type="java.lang.String">
                                 <Item>user</Item>
                                 <Item>admin</Item>
                                 <Item>moderator</Item>
                              </Array>
                           </Set>
                           <Set name="authenticate">true</Set>
                           <!-- Below line is optional and will forward all requests to SSL port (via 302) -->
                           <Set name="dataConstraint">2</Set>
                        </New>
                     </Set>
                     <Set name="pathSpec">/*</Set>
                  </New>
               </Item>
            </Array>
         </Set>
      </New>
   </Set>
</Configure>

[stuckless]

Slugger, thanks for this... I'll try to update the wiki with the new secure context file for those that are running directly exposed to the internet.

[Slugger]

Please take note of my edit to the context file in my previous post. The edit adds the line necessary to force all traffic to SSL port.

[iolaus]

Could someone please provide a sample of the REST command that would be used to start playback of a file/recording? I've been trying to use the MediaFileAPI Watch command but haven't had any luck thus far. Thanks!