Library: Remote APIs for SageTV

[stuckless]

I've been working with the remote Java RPC API and ran into an issue. It looks like the payload on the request is being sent as part of the URL, and once that exceeds a certain length (a few kbytes) it fails.

Jason,
I've uploaded a new version 6.4.8-4 which resolves this issue. You will need to update both your server and client projects with the new jar file.

I discovered, as you noted, that if the request was larger than a few K in size then the server would through an IO exception when reading the request data. This really surprised me since i've never had an issue posting very large documents to a server url. I tried, post and I also tried using post with multipart/form-data and still no luck. This may be an issue with the servlet container in the tiny webserver that neilm uses, or it may have been an issue in how the post was contructed, either way, it didn't work

So, against my better judgement, I changed the Java RPC to use RMI instead of HTTP. It now works, and I was able to run your example and sort the 2800 titles from my database (remotely).

On another note....
Shadeblue, please note that the api call to force a particular server no longer exists...

sagex.SageAPI.setProvider(new SageAPIRemote(YOUR_SERVER_URL));

You can now use

sagex.SageAPI.setProvider(new RMISageAPI(host,port));

The default port for the rmi server is 1098, and you should see a startup line in the sagetv log file telling you the RMI host and port.

[shadeblue.com]

Originally Posted by stuckless

...
So, against my better judgement, I changed the Java RPC to use RMI instead of HTTP. It now works, and I was able to run your example and sort the 2800 titles from my database (remotely).
...

In switching to RMI for remoting, does the API library still need to be hosted by a servlet container? Or can I somehow start it independently of webserver/container?

Thanks,
shadeBlue

[shadeblue.com]

Originally Posted by stuckless

So, against my better judgement, I changed the Java RPC to use RMI instead of HTTP. It now works, and I was able to run your example and sort the 2800 titles from my database (remotely).

I just realized this is a bummer because I was creating web based solutions hosted across HTTP. Communication via a RMI port won't traverse the commonly exposed HTTP firewall exception in routers.

Before starting development with your plugin, I was looking into Hessian as a remoting solution for my needs, have you looked into Hessian as an alternative? I tried getting it started up as a servlet using the Tiny web server, but I was not successful, probably because I did not know what I was doing :-)

Thanks,
shadeBlue

[stuckless]

I just realized this is a bummer because I was creating web based solutions hosted across HTTP. Communication via a RMI port won't traverse the commonly exposed HTTP firewall exception in routers.

Yeah, I'm not happy about it either. RMI is probably my most hated feature in java. In this case, i think the issue is with the http server implementation in how it handles very large blocks of data. I'm not not sure the Hessian solution would work any better since it will still rely on the underlying server for transferring large blocks of data. Also, i'm trying to keep this library clean from any other external dependencies.

I'll continue to look into a http solution, but in the mean time, you can attempt to use the Hessian solution if you write your own SageAPIProvider. Typically, I create a RemoteRequest object, fill it with the request data, and then send it to a server where it get handled by the AbstractRPCHandler and sends back a RemoteResponse. Check out the RMISageAPI and the SageRemoteCommandServer for how the RMI piece works.

[stuckless]

You can download from here

Just a note about upgrading.... because I use versioned jar names, when you install a new jar, you MUST remove the other jar.

shadeblue,
I added back the java over http (but rmi is still the default unless you force the remote url). Keep in mind that it will fail for very large parameters using neilm's webserver. I've tested under jetty, and it doesn not have the same issue. So, if you know you are going to be passing around large arrays (not array refs), then make sure it's running under jetty.

This instructions for installing under jetty are in wiki. If you are going install it under jetty and you already have it installed under niem's server, then you only need to run it in one place.

This release also add a generic way of forcing a remote url. The api will check for the existence of a property called, sagex.SageAPI.remoteUrl.

So you can set it via a commandline or by using System.setProperty()

# java -Dsagex.SageAPI.remoteUrl=http://host:port/sagex/rpcJava
# java -Dsagex.SageAPI.remoteUrl=rmi://host:port

or

{
   System.setProperty("sagex.SageAPI.remoteUrl", remoteUrl);
   ....
   String titles[] = Database.GetAllTitles();
}

NOTE: you not not need to set a remoteUrl. Only set the remoteUrl if you are trying to force the API to use a particular type of remote connection (ie, http instead of rmi) or if you want to force a particular server, in the event that you have more than one sage server configured using the remote api.

[shadeblue.com]

Originally Posted by stuckless

...

shadeblue,
I added back the java over http (but rmi is still the default unless you force the remote url). Keep in mind that it will fail for very large parameters using neilm's webserver. I've tested under jetty, and it doesn not have the same issue. So, if you know you are going to be passing around large arrays (not array refs), then make sure it's running under jetty.

I'm happy you decided to add the HTTP transport back. I was forced to rethink my strategy for the Flash based SageTV Remote Control app that I just put out, because I wanted it to communicate over the already exposed web port. With the support for HTTP added back, I can continue using your API for that project ... which mean less work for me ... yea!

Thank You,
shadeBlue

[jreichen]

I prefer using HTTP also. Thanks for adding it back.

I changed it over to using Jetty. The sorting now returns without error, but when I try to call another Sage API on an item in the sorted array, it doesn't return anything. If I leave out the call to sort, it works fine.

Here's the code:

Object[] airings = (Object[]) sagex.SageAPI.call("SearchByTitle", new Object[] {"Seinfeld"});
Object[] params = new Object[] {airings, true, null};
airings = (Object[]) sagex.SageAPI.call("Sort", params);

for (int i = 0; i < airings.length; i++)
{
    Object airing = airings[i];
    String title = (String) sagex.SageAPI.call("GetAiringTitle", new Object[] {airing});
    System.out.println("Title: " + title);
}

[stuckless]

jreichen, i'll look into this api issue tonight. Curious as to why you are using the SageAPI.call() directly and not the wrappers? Not that it would fix this issue, but it would reduce the chance for typos, and the API wrappers would negate the need to create the object arrays.

[stuckless]

I've also been considering adding the Remote API functionality into Gregs typed API. It would probably work something like this...

// Api api = new Api();
Api api = APIFactory.newInstance();
MediaFileAPI.List files = api.mediaFileAPI.GetMediaFiles();

APIFactory.newInstance() would either return Greg's API object or a subclassed RemoteAPI object. There would also be a newInstance(context) method as well.

Any thoughts?

[jreichen]

Originally Posted by stuckless

jreichen, i'll look into this api issue tonight. Curious as to why you are using the SageAPI.call() directly and not the wrappers? Not that it would fix this issue, but it would reduce the chance for typos, and the API wrappers would negate the need to create the object arrays.

I'm just experimenting with your APIs at all levels. I'm not writing anything serious yet.

[jreichen]

Originally Posted by stuckless

I've also been considering adding the Remote API functionality into Gregs typed API. It would probably work something like this...

// Api api = new Api();
Api api = APIFactory.newInstance();
MediaFileAPI.List files = api.mediaFileAPI.GetMediaFiles();

APIFactory.newInstance() would either return Greg's API object or a subclassed RemoteAPI object. There would also be a newInstance(context) method as well.

Any thoughts?

How would the factory know which to create? Would it use the command line parameters you added? Or maybe a url parameter to newInstance()? Both?

The factory is a good idea, but I also like how I can directly create a provider (embedded, XML RPC, Java RMI, etc) in the current remote API code. Combine those creation patterns with the object model of Greg's API and that would be a good thing.

Question about the API class and its subclasses: would the API class be like your EmbeddedSageAPIProvider, then subclasses for the various remote implementations would implement their own constructors (and/or getters/setters) and call() methods? Another way would be to make API an interface then all implementations implement that interface, unless that would break too much existing code that uses Greg's API.

Keep up the great work

[stuckless]

How would the factory know which to create? Would it use the command line parameters you added? Or maybe a url parameter to newInstance()? Both?

Yeah, basically the factory would create Greg's Api instance if it were running embedded inside SageTV, or it would create my subclass RemoteAPI if it were not running inside of SageTV. My RemoteAPI would simply wrap my exsisting SageAPI.call() so at that point it would auto choose the remote api provider or use a fixed provider if the SageAPI.remoteUrl is set in the environment. (or maybe add a second parameter to newInstance() to pass the url). I haven't done anything yet... these are just ideas.

Question about the API class and its subclasses: would the API class be like your EmbeddedSageAPIProvider, then subclasses for the various remote implementations would implement their own constructors (and/or getters/setters) and call() methods? Another way would be to make API an interface then all implementations implement that interface, unless that would break too much existing code that uses Greg's API.

The great thing about the SageTV API is that everything eventually filters down to calling sage services. Greg's API is very clean in that there is only one place it actually makes the SageTV service call, which is in the API() class. So, adding a RemoteAPI is as simple as extending Greg's API() class and overriding his call() to sagetv with a call to my sage apis. Greg's API class does have 2 static final instance fields that I would need to make non-final in order for this to work, but in theory, it would be very easy and totally transparent.

[stuckless]

Fixed Sort/Array bug that jreichen found. You can down from the download from here.

[jreichen]

Originally Posted by stuckless

Fixed Sort/Array bug that jreichen found. You can down from the download from here.

Great! Thanks for the fix. Now I found a problem with DataUnion

Object[] airings = (Object[]) sagex.SageAPI.call("SearchByTitle", new Object[] {"Seinfeld"});
Object[] airings2 = (Object[]) sagex.SageAPI.call("SearchByTitle", new Object[] {"Dirty Jobs"});
Object[] params = new Object[] {airings, airings2};
airings = (Object[]) sagex.SageAPI.call("DataUnion", params);

for (int i = 0; i < airings.length; i++)
{
    Object airing = airings[i];
    String title = (String) sagex.SageAPI.call("GetAiringTitle", new Object[] {airing});
    System.out.println("Title: " + title);
}

Sage Provider Implementation: sagex.remote.javarpc.SageAPIRemote; http://192.168.25.102:8080/sagex/rpcJava
java.io.StreamCorruptedException: invalid stream header: 6E756C6C
    at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:783)
    at java.io.ObjectInputStream.<init>(ObjectInputStream.java:280)
    at sagex.remote.MarshalUtils.unmarshal(MarshalUtils.java:26)
    at sagex.remote.javarpc.SageAPIRemote.callService(SageAPIRemote.java:60)
    at sagex.remote.javarpc.SageAPIRemote.callService(SageAPIRemote.java:87)
    at sagex.SageAPI.call(SageAPI.java:52)
    at test.TestSageRemoteAPI.main(TestSageRemoteAPI.java:43)
Exception in thread "main" java.lang.RuntimeException: Failed to call command: DataUnion
    at sagex.remote.javarpc.SageAPIRemote.callService(SageAPIRemote.java:73)
    at sagex.remote.javarpc.SageAPIRemote.callService(SageAPIRemote.java:87)
    at sagex.SageAPI.call(SageAPI.java:52)
    at test.TestSageRemoteAPI.main(TestSageRemoteAPI.java:43)
Caused by: java.io.StreamCorruptedException: invalid stream header: 6E756C6C
    at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:783)
    at java.io.ObjectInputStream.<init>(ObjectInputStream.java:280)
    at sagex.remote.MarshalUtils.unmarshal(MarshalUtils.java:26)
    at sagex.remote.javarpc.SageAPIRemote.callService(SageAPIRemote.java:60)
    ... 3 more

[stuckless]

You didn't by chance try this using the RMI provider (default) as well? I'm wondering if this is a similar issue to the issue that found when running under nielms webserver. I'll take a look

[jreichen]

I forgot to mention the RMI provider doesn't work for me. Even if I specify the IP address:

ISageAPIProvider provider = new sagex.remote.rmi.RMISageAPI("192.168.25.102", 1098);
sagex.SageAPI.setProvider(provider);

I get an UnknownHostException. Somewhere down in the RMI code it's calling a Sage API to get the machine name and ends up using that instead. I'm running Linux and don't have any name resolution set up on my network. It would be nice if it would use the IP I specify, but it's no big deal. I need to get the name resolution set up anyway. Plus I prefer the XML RPC

[stuckless]

yeah... the sage api on the server does use the configured Sage Server from the sage configuration file. If you don't set any provider on the client side, then it will autodiscover the correct server based on your sage configuration.

As for your latest bug It's related to the fact that I don't do anything special to serialize Vectors. So i think that the vector comming back contains a bunch of Sage Airing objects which cannot be resolved on the client side.

I'll fix this... but it may be a couple of weeks (unless it becomes a real issue for you). I want to focus my attention on the web based batch metadata updater.

[jreichen]

Originally Posted by stuckless

I'll fix this... but it may be a couple of weeks (unless it becomes a real issue for you). I want to focus my attention on the web based batch metadata updater.

Not a problem. Thanks for the help.

[stuckless]

Jreichen recently provided me with JSON support for the Remote API. You can now request that your HTTP responses come back as JSON replies (javascript).

You can read the wiki entry for more information. It contains an example that gets a list of recorded shows from the server using javascript and JSONP.

There are no other changes in this release, so there is no need to upgrade unless you want try out the JSON support.

You can download from the usual place.

[babgvant]

Does the service expose a WSDL? Or something like it that could use to autogenerate a proxy class?

I'd like to integrate with a c# application, and it would save a ton of time...