Best way to communicate between threads in different UI contexts?

[tmiranda]

I'm developing an STVi that spawns several threads to do various tasks in the background. I'm using Fork() in the ApplicationStarted Hook to start the threads.

I want to make sure that if the user installs the STVi on several extenders, placeshifters and/or SageClients the threads only get spawned only once. What's the best way to check if the thread is already running? I was thinking I could use SetServerProperty() and GetServerProperty() but that gets messy since there is no way to easily do an atomic "test and set" operation on the property. (I'm assuming GetServerProperty() could be interrupted at any time.)

I could clear the property in the ApplicationExiting Hook, but I'd then need some way to recover from a situation where the UI context terminated abnormally and failed to clear the property.

Thoughts?

[GKusnick]

What do these threads do? If it's truly background processing that's independent of any UI instance, then they probably shouldn't be forked from the STVI at all; they should be started by a Runnable class running on the server in the SageTVService JVM.

[tmiranda]

I understand and see how that would be desirable. Of course I have no experience doing things that way..

I am assuming this means that the code would be run by listing it in the java_runnable_classes property? (I probably don't have the name exactly right.)

Can you point me in the right direction of where to look for examples on how do do this? I have a little experience now with NetBeans and can create .JAR files so hopefully that will give me a bit of a start. Being a java dummy I'd really like to see some existing code that works so I can use it as a template.

As usual, thank you very much for your insight and help.

Tom

[GKusnick]

The property is called load_at_startup_runnable_classes. What you do is write a Java class that implements the Runnable interface and add its fully-qualified class name to that list. Sage will then call your run() method in a thread of its own on startup. You can then use Java threading APIs to fork off additional threads if you need to.

Any of the popular plugins that run in the background and for which source is available (such as nielm's webserver or my Studio Tools) should show you the basic mechanics of setting up a Runnable class (although obviously the contents of the run() method will be different for different plugins).

[PLUCKYHD]

There was a thread a while back by me when I started implementing my own runnable classes. I simply made it a deamon thread and had it sleep a given amount of time and then wake up and do it's thing every so often. May not be what you are looking for but not a hard class to grasp really. Just make sure you don't do anything to keep the sage service from being stopped when it needs to be is what I learned.

What exactly are you doing? that would help me offer you some advice (although not as advanced us others but I will help where I can)

[tmiranda]

Thanks to both of you. If I implement things this way I will then need a way to communicate between the code that runs in this class and the various UI contexts that may be running.

Here are some more details of what I am doing. I'd appreciate advice.

I'm implementing the ability to automatically record (download) podcasts that have been selected as Favorites in my Malore Online Browser. The basic design has two threads that run when the UI is started.

Thread 1, which I call the Recording Manager, runs in the background. It wakes up every so often to see if any new episodes for Favorite podcasts are available. As it finds new episodes it queues them up to be recorded, which is done by Thread 2.

Thread 2, waits for items to show up in a download queue. If items are found it downloads them and imports them into the sage database.

Items can be put into the download queue by either Thread 1 or the UI (when a user selects to record a podcast.)

This is a simplification of things but you get the general idea. There is a lot of interaction between the UI, Thread 1 and Thread 2. Right now some fo the communication is done via properties but a lot of the communication is also done via data added to the global context via AddGlobalContext().

Ideally there should only be one instance of thread 1 and thread 2 running and they should be able to interact with multiple UI contexts (from placeshifters, extenders and SageClients.)

Ideas welcome.

[PLUCKYHD]

Definatey need a runnable class for this much easier and better. Running it ui code like you do mean it is only checking for new podcast when the ui is running. Having it in startup as a runnable class will allow it to check periodically whenever the service is running regardless if a ui session is running online.

I am going to guess when a user adds a recording manually you are adding a line to a property file?

That being the case I would just have a runnable class the first checks for any new podcast then adds those to any the user may have selected manually. Then have it download the ones it needs to download. Then cleanup your properties file or whatever is storing that and sleep the thread for a given time(however Long you want to wait in between checks/downloads preferbly defined by a property so user can adjust.)

then whatever those calls are two check and download you can make those classes/calls callable in the stv so a user could initiate a on demand scan.

As for clients/servers talking to each other I would use set and get server properties so all will be in synch

hopefully that makes sense and may not be how others suggest to do it but I is how I would do it.

If you want some example pluesebo code let me know and I will type it up for you to start with.

[tmiranda]

Plucky,

It's definitely more complicated than adding a line to a properties file. There is A LOT of data needed to determine how to download a podcast and then actually download it. Take a look at the code in the ApplicationStarted hook in the latest beta I posted in the Malore Online Browser thread.

There are probably 3 or 4 Maps that are used as well as several Lists. This data would be needed both in the UI (client) and the runnable class (server). And the data is updated frequently so each time it changed I would need to synchronize it between the client and the server.

The runnable class is certainly the way to go, but it's going to be a large learning curve for me to figure out how to do it properly. I did a little googling and started looking at RMI. Yikes!

[PLUCKYHD]

Okay now worries the details of how to fetch it can be worked out the biggest issue is going to be the client and synching things up.

I am not sure but I would think Sagetv clients would have access over the network to the sagetv server folders by default but could be completely wrong on this, if so you could store you data in a folder under sagetv to access from the client.

It will be a learning curve but like everything else worth the effort. You will find things so much easier in this instance then coding a fork and running it all in your stv.

[stuckless]

Originally Posted by tmiranda

Plucky,

It's definitely more complicated than adding a line to a properties file. There is A LOT of data needed to determine how to download a podcast and then actually download it. Take a look at the code in the ApplicationStarted hook in the latest beta I posted in the Malore Online Browser thread.

There are probably 3 or 4 Maps that are used as well as several Lists. This data would be needed both in the UI (client) and the runnable class (server). And the data is updated frequently so each time it changed I would need to synchronize it between the client and the server.

The runnable class is certainly the way to go, but it's going to be a large learning curve for me to figure out how to do it properly. I did a little googling and started looking at RMI. Yikes!

Tom, on the server (ie, for extenders), it's pretty trivial to share a single instance to a queue (singleton pattern).

But the client/server communications is a little trickier. Using the SetServerProperty is probably the easier way to send information from the client to the server, but it's really just hack, and it's using the the properties for something that was never meant to be used that way.

I think your best bets are to either use rmi, http, or multi-cast sockets. (sagex api actually uses all 3). RMI is actually quite trivial, especially since java5. You no longer have to worry about rmi stubs and rmic, etc. You simply create an interface that is shared with the client and server, register the implementation using jndi to server and connect to rmi registry from the client. Sagex-api does this for it's "remote api" implementation.

Using multicast sockets is another way to send messages over an environment where you don't want to configure the client and server to specifically connect to each other. On the server you can create a socket listener that listens for multicast udp packets that contain a message. You then act on the message. The client simply "broadcasts" the message and anything listening for it handles it. This is basically non-reliable messages and it's typically used for short messages (ie, you try to keep your message to fit inside a single packet). sagex-api also uses this as well, for remote server discovery. It sends out a message like "hey tell me who's online", then it wait for all the server replies to flood back (usually only 1).

The third option (probably the easiest is many respects) is to implement a servlet under the jetty server. This server could be called from the client with a set of parameters that add the entry to the download queue. The drawback here is that the communication is really one way. but the client could poll another servlet (or the same servlet) for status information. This option, much like the rmi option, would require the clients to be configured with the http serverort information in order to communicate.

Hope these options help.

[tmiranda]

Plucky - Thanks for the input. I really don't think I want to get into storing data in folders because the data is complex. It's not simple strings that can be stored in a flat file. It's Maps and Lists and other abstract data types.

Sean,

I think I will look into RMI. From the hour or so of reading I've done I think it can do what I need. It's all still Greek to me, but now I can understand a few of the words at least.... (It reminds me of when I moved to Europe and had to learn Dutch. At first it's just a bunch of strange sounds, then you can start to pick out words, and finally you can hear sentences. Right now I can just "hear" a few Java words.)

I had not even considered the jetty approach. Is jetty sage-specific or is it a general Java concept? Where do I find info on how to write a jetty servlet?

No matter how I look at it, it will be a major re-design of what I've already done.

Tom

[GKusnick]

Originally Posted by stuckless

The drawback here is that the communication is really one way. but the client could poll another servlet (or the same servlet) for status information.

Actually I don't consider that a drawback, I consider it smart design. Whether you use Jetty or some other method, I think you'll be much happier in the long run if you structure your code so that the UI packages up requests, hands them over to the backend for processing, and doesn't try to access them anymore after that. Status updates can be handled the same way in reverse: the backend prepares status messages and ships them off to clients for them to own and do as they please with. This is a much more robust approach (in my opinion) than trying to have UI and backend share data structures dynamically, with all the synchronization issues that entails. It also forces you to be really clear about the division of labor between UI code and backend code.

[PLUCKYHD]

Originally Posted by tmiranda

Plucky - Thanks for the input. I really don't think I want to get into storing data in folders because the data is complex. It's not simple strings that can be stored in a flat file. It's Maps and Lists and other abstract data types.
Tom

Actually you can store anything you want like that in a flat file as long as it is serializable, trust me I do it allot in MyMovies implementation, but seans method and suggestion is a good one as well more of a learning curve but probably a better way. I was trying to give you an easier way

Yes you are right it will be a huge overhaul, but if it is worth it and better the payoff will be. I can't tell you how many times Somone on here have suggested something completely different than the way I was doing it, but needless to say their ideas are usually worth the effort

[stuckless]

Originally Posted by GKusnick

Actually I don't consider that a drawback, I consider it smart design. Whether you use Jetty or some other method, I think you'll be much happier in the long run if you structure your code so that the UI packages up requests, hands them over to the backend for processing, and doesn't try to access them anymore after that. Status updates can be handled the same way in reverse: the backend prepares status messages and ships them off to clients for them to own and do as they please with. This is a much more robust approach (in my opinion) than trying to have UI and backend share data structures dynamically, with all the synchronization issues that entails. It also forces you to be really clear about the division of labor between UI code and backend code.

I totally agree, Greg. My comment was really that the web server communication is one way, so the server has no way to push updates to a client, in an event driven, asynchronous way. The client HAS to poll the server for updates. I see that as a drawback of the HTTP model. Unlike the multicast socket approach, where messages can flow either way, and it doesn't matter if anything is listening. It's actually slightly more complex to write purely async code, but it certainly makes for a better design. Under http (without getting into http push technology) you can't really have async code, and you certainly can't have messages pushed to the client, unless they opt to install their own web server as well.

[Fuzzy]

I also think this is the type of thing that perfectly suited to running within a Jetty app. In this way it could easily be accessed, without sharing problems, from many sources (those being other server threads, clients, even web threads).. Imagine making an IE plugin where you could add a feed to your queue by right clicking on it and showing an 'add to sage favorites' context option...

[tmiranda]

Originally Posted by GKusnick

Actually I don't consider that a drawback, I consider it smart design. Whether you use Jetty or some other method, I think you'll be much happier in the long run if you structure your code so that the UI packages up requests, hands them over to the backend for processing, and doesn't try to access them anymore after that. Status updates can be handled the same way in reverse: the backend prepares status messages and ships them off to clients for them to own and do as they please with. This is a much more robust approach (in my opinion) than trying to have UI and backend share data structures dynamically, with all the synchronization issues that entails. It also forces you to be really clear about the division of labor between UI code and backend code.

This is what the code does now, the UI populates some data structures and then puts an item in a queue that tells the "server" something needs to be recorded. The server waits until it find something in the queue, gets the data it needs from various data structures, does its work and then posts the results in another data structure.

So which way should I go, jetty or RMI? I don't know how to do either so I have a learning curve either way. Knowing nothing about either I like the idea of RMI so I do not have to have the added requirement on the user to have jetty up and running. But if RMI is a lot harder to learn than jetty, I'll use jetty.

Another question about the runnable classes. Do they run in a different JVM than the UI context, even if the UI is on the same physical machine (like an extender)? The reason I ask this is that some of the Sage core functions are not reentrant but (if I understand things right) if the runnable class and the UI run in different JVM's, I do not have to worry about that. StartFileDownload is a good example. In the current implementation I have to make sure that nothing else is being downloaded (STV update, online properties update, another online video etc.) before I start downloading a podcast.

[PLUCKYHD]

Extenders,servers and placeshifters all run in the same jvm. Clietns will each run in their own jvm local to that machine.

As for your own class and downloads and such those will run in their own threads if that is how you code it.

So clients is the only real reason you need to use something like jetty.

[stuckless]

Tom, personally i'd look at using servlets. If you are using eclipse or netbeans as your ide, then creating webapps is pretty trivial. (ie probably as simple as File -> New -> ...)

Creating a servlet is fairly straight forward, just create a class that extends http servlet, and implement the doGet (and/or the doPost) method. The method will accept a request and a response object... hopefully those as self explanatory.

Calling the servlet from your client code is pretty easy, just use URLConnection to open a connection to the servlet. You can use http response codes to indicate success or failure (ie, 200 ok - item was accepted, 500 error - item was rejected), etc.

Once you create the servlet, you can create a .war file (File -> Export -> ...). deploying to jetty is pretty straight forward, and I think that even nielms webserver supports war files as well (i think).

And as Fuzzy noted, using http, allow for other external integrations, should you want to allow for that.

[tmiranda]

Plucky,

Thanks for the info. Too bad, if extenders and placeshifters used different JVM's than the server I could have saved some coding time.

Sean,

I'm using the NetBeans IDE. I'll look at some servlet code to get a feel for how complicated it is.

Tom

[PLUCKYHD]

Originally Posted by tmiranda

Plucky,

Thanks for the info. Too bad, if extenders and placeshifters used different JVM's than the server I could have saved some coding time.

Tom

Why do you say that? All of them being in the same jvm is make it easy to communicat between each one or treat them as seperates. It is the clients that are difficult.