Memory / Java Heap

[wayner]

Given that there is a Diamond Legacy UI can I use Gemstone on some clients and Diamond Legacy on others?

Thanks for adding the Import/Export - that will be hugely beneficial for me as I have lots of clients.

With Diamond my system would start running out of JVM with about 4- cleints running. I believe you said that Gemstone was more efficient in terms of memory usage. What is the reduction in memory usage per client compared to Diamond?

[Fuzzy]

Originally Posted by wayner

What is the reduction in memory usage per client compared to Diamond?

That's likely going to be hard to quantify. It should be quite different depending on usage. About the only thing that could be done is make more use of shared resources between contexts, so the savings would only be based on sharing assets between extenders. I know on my clients (which don't have to share a common memory space on the server) use about the same memory footprint between diamond and gemstone.

[jusjoken]

Your mileage may vary

We have had 3 testers report that they could not use Gemstone as it fully consumed their heap... other have not reported issues.

The dev team have also had mixed results and we have been working on reducing the memory usage so we need testers to report their results and experience including what other plugins they are running.

I can report that I have 2 HD300 running all the time and the server heap is around 400MB typically. however, over 4-7 days the heap gets used up and I need to restart the service. This is not an issue for me as once the testing is over I can easily setup a scheduled task to restart the service even daily but other may not want to do that.

I can also say that we have done much to reduce the memory usage while increasing the functionality and performance... or at least that has been the goal. I can run Gemstone on my HD100 with pretty good performance as long as I don't get carried away with the more image intensive Flows. Diamond was not really able to do that.

We are still early in the testing cycle so as we get more feedback and find more issues we hope to improve the memory usage further including adding options to control the usage if needed.

PS: yes... the theory is that you could run Diamond Legacy on some clients and Gemstone on others. I am not sure anyone has tried it so please report your results if you do. Just remember.... Gemstone will REPLACE ALL your Diamond installs on ALL extenders on the same server and you would then need to disable Gemstone and INSTALL the Diamond Legacy plugin on the ones where you want to go back to Diamond.

k

[jusjoken]

Originally Posted by Fuzzy

That's likely going to be hard to quantify. It should be quite different depending on usage. About the only thing that could be done is make more use of shared resources between contexts, so the savings would only be based on sharing assets between extenders. I know on my clients (which don't have to share a common memory space on the server) use about the same memory footprint between diamond and gemstone.

Agreed. Gemstone does offer some shared solutions across extenders on the same server.

For example...

Non-Optional - All extenders share the same lists for the widget lists.

Non-Optional - All extenders share the same View Cache (if enabled) for Flows that have the same View setup (same View/Filters/etc).

Optional - Menu Manager can be set to use a single Menu File... if you do this then that menu definition is stored only once on the Server Heap.

We may find a few more that make sense to reduce the Memory usage on extenders.

k

[wayner]

OK, so given the sounds of that it may be wise to stick to my backup/test server for now.

I have 2 HD-300s and about 4-5 HD-200s currently deployed. It is rare that the are all on at once but it seems that even once an extender is turned off it retains some Memory.

Power blips also cause an issue as my server and some extenders are on UPS'. But those that are not boot up on when the power comes back and connect to the server and then I get an out of JVM error.

[jusjoken]

One user is seeing a lot of errors in the log regarding "MiniUI client" as below.

MiniUI client error9:java.io.IOException: An established connection was aborted by the software in your host machine

Anyone know what this is?

k

[Dargason]

I was digging through the log some more last night and I'm convinced it's part of Sage and not a plugin. There are references to MiniUI, MiniMVP, MiniServer, MiniClient, MiniUIServerConnection, etc. And there are references in error logs posted in the forums as far back as 2006:

http://forums.sagetv.com/forums/show...=MiniUI&page=2

Of course that's not to say that the errors aren't caused by a plugin... I'm still doing tests and I made that error go away.

[hoep]

seems that 2 hd300 are consuming most of my heap (i have 1024 by default). switching on a third hd300 makes it unstable sometimes shuts the service down.

[jusjoken]

Thanks for the feedback. We have 2 testers assisting with narrowing this issue and there is an issue on the issues list for this so feel free to check that out as we work through this and see what the problem is.

k

[jusjoken]

Originally Posted by jusjoken

One user is seeing a lot of errors in the log regarding "MiniUI client" as below.

MiniUI client error9:java.io.IOException: An established connection was aborted by the software in your host machine

Anyone know what this is?

k

From the logs I have received I believe this is related specifically to the Placeshifter and Extender connections. We have the error cleaned up now.

I also believe I know what is happening with the Placeshifter consuming more memory on the server with each restart.

It seems with each connection of the placeshifter Sage treats it as a NEW client with perhaps a different UIContext. Gemstone stores some items in memory by UIContext to keep settings for different clients/extenders separate... but it does not remove references to these settings as I am not sure I could know that that client/extender/placeshifter is no longer connected.... so we consume more memory each time.

I don't think the extenders do this as the UIContext is related to the mac address of the extender so it will be the same each time so this is not the same issue... I think.

More testing required.

I also plan to work on a manual CLEAR function so the testers having these issues can see if this helps.

k

[tmiranda]

Originally Posted by jusjoken

It seems with each connection of the placeshifter Sage treats it as a NEW client with perhaps a different UIContext. Gemstone stores some items in memory by UIContext to keep settings for different clients/extenders separate... but it does not remove references to these settings as I am not sure I could know that that client/extender/placeshifter is no longer connected.... so we consume more memory each time.

You could create a General plugin as part of Gemstone and then listen for the ClientConnected and ClientDisconnected events.

[jphipps]

There is a client disconnect event you could use to clear the objects associated with that UIContext so they wont remain after a client is turned off...

Thanks,
Jeff

[jusjoken]

I have discovered a few areas where I can improve memory handling for multiple extenders so I will work on that for the next test release.

k

[nyplayer]

I am having memory problems with the latest release I will turn debugging on. The only plugin I have is CMT andd Comskip ... I had no problems with the prior release with memory.

I had several crashes on my client and no video when memory usage gets real high.


PS I have no extenders just clients.

[jorton]

This is just weird, I'm mostly clients (x3) and I have no problems...

Hopefully we can get this sorted out.

[jusjoken]

We will need to see the gemstone.log and the sage log. You can post it on the issues site... Create a new issue for client memory.

k

[nyplayer]

Originally Posted by jusjoken

We will need to see the gemstone.log and the sage log. You can post it on the issues site... Create a new issue for client memory.

k

Jorton what Java Version are you using?

[jorton]

6 r33 on my server.

7 r5 on my clients.

I had issues with 7 r5 on my server giving my the system memory messages and had to revert. All good since total java uninstall and 6.33 install. Just haven't had time to push the clients back but haven't had any problems with them so may leave them on 7 r5.

J

[nyplayer]

Exactly what I am running I was running at Java 1280 cut it back to 1024. All logging on. Will open incident and post logs if I crash.

[jorton]

I have never used the java heap set to more that 1024MB (400 hex?).

That may have been your issue.

Good luck.