Maxing out Java Heap at 799mb

[craigap]

Java heap is at 539/799/799 right now. I run vista business 32bit and have 2gb of RAM. it would appear I need to bump up my Java heap, but doing so would seem to leave me with only about half my RAM for OS stuff. Not sure if that how it works or not, but if so then should I look into adding another Gig of ram to my HTPC?

[Fuzzy]

If you are running STVi's that are that extensive graphically, then yes, your option for better performance would be to get more ram on there. You can still bump up the heap larger, and windows will just have to swap stuff a bit more, so you'll get hurt on overall system performance, but might get a bit smoother fanart and such... just try it out and see...

That said.. RAM is actually pretty cheap...

[stanger89]

Here's a question, anybody figured a way around Java's sometime "weird" limiting of Heap Size?

[Fuzzy]

there is nothing weird about java memory management... it's just poor to begin with.. never expect 'good' behavior from any JVM, memory wise... if only sage was .net.. hehe

[stuckless]

Originally Posted by stanger89

Here's a question, anybody figured a way around Java's sometime "weird" limiting of Heap Size?

I guess my question is what is "weird" about the heap size?

Originally Posted by Fuzzy

there is nothing weird about java memory management... it's just poor to begin with.. never expect 'good' behavior from any JVM, memory wise... if only sage was .net.. hehe

You must be a very hard core java developer/engineer (engineer, if I had to guess) that has done extensive research on the jvm memory sub system. Perhaps you can explain why I should never expect to see 'good' behaviour from the jvm. I'm always trying to better understand the inner workings of java.

[stanger89]

Originally Posted by stuckless

I guess my question is what is "weird" about the heap size?

Mainly that it's very unpredictable what the actual maximum is. It seems to vary from PC to PC from one moment to the next.

[stuckless]

Originally Posted by stanger89

Mainly that it's very unpredictable what the actual maximum is. It seems to vary from PC to PC from one moment to the next.

I see. I'm not sure the "real" default heap size is actually documented anywhere, so yeah, if an application doesn't specify it's memory requirements then i'm not sure what it would be...and it probably would differ from java version (not necessarily machine to machine). But, if we are talking about sage, then I can say that on linux, they set the default memory size to be 256mb (which I usually increase to 1 gig).

[stanger89]

Supposedly it's 1906MB or something, but I've never been able to actually start a VM with that. I was playing a little while ago with BMT and "accidentally" set that test install to import way too much stuff and ran out of heap, so I remember trying to just set Sage to the max I could and I think at that time I couldn't manage over 1100MB, today I tried, same machine and could do 1600MB....

[Fuzzy]

Two main complaints I have with JVM is that:
1. The arbitrary fixed max size has to be declared at runtime.
2. The JVM will never release memory back to the operating system.

Admittedly, this is probably based on the original incarnations of Java being designed primarily for smaller applets, that usually have fixed memory footprints, it doesn't scale well to large dynamic apps like sage.

[stuckless]

Originally Posted by stanger89

Supposedly it's 1906MB or something, but I've never been able to actually start a VM with that. I was playing a little while ago with BMT and "accidentally" set that test install to import way too much stuff and ran out of heap, so I remember trying to just set Sage to the max I could and I think at that time I couldn't manage over 1100MB, today I tried, same machine and could do 1600MB....

Ok... now i really see

I guess i've never hit the upper limit, but, just to test this on my machine, I could create a machine at 2600mb... but not at 2700... and I think you are right, that the limit will vary from machine to machine depending on resources, etc. I think that java fails if it cannot allocate a contiguous block of memory.

I'm guessing the bmt issue was related to the dvd profiler xml... that's a very large file, and bmt uses a DOM parser (not smart) and so it has to cram it all into memory.

A think to note about java memory, is that if you allocate memory to the jvm, then the jvm will hold onto that memory and not let other system resources use the memory. Within the JVM it will continue to allocate objects into that space and it won't really try to free those objects until it has to. Depending on your jvm version, java may just let them accumulate until it determines it can't actually create another object, and if so it will run the garbage collector, and then try to allocate the object again. If it can't free enough contiguous heap space to create the object, then it will fail with an Out Of Memory error. Keep in the mind the actual process is alot more complex than my simple explanation, but the bottom line is really... unless you are seeing out of memory errors, or you are experiences freezes in the application related to the garbage collector running, then something like 539/799/799 is probably ok.

This differs from .net (at least my understanding of it) in that in .net, an application can access the complete memory on the system. But like java, it also uses a garbage collector to reclaim objects. Both Java and .net use very similar models for garbage collection, (including when to run and how to run), but I personally like the "constrained" memory model of java, since I can control how much memory i'm willing to allow the application to abuse.

[stuckless]

Originally Posted by Fuzzy

Two main complaints I have with JVM is that:
1. The arbitrary fixed max size has to be declared at runtime.
2. The JVM will never release memory back to the operating system.

Admittedly, this is probably based on the original incarnations of Java being designed primarily for smaller applets, that usually have fixed memory footprints, it doesn't scale well to large dynamic apps like sage.

See... isn't that a lot more constructive than your original... "Java sucks... I wish sage were .net" (i paraphrased)

BTW... i agree with your complaints (now that I know what they are)... but I'd still rather have the fixed memory constraints of Java than an open upper limit. Imagine how unresponsive your machine would get if Sage could allocate the entire system resources... including swap... because of some memory leak a plugin that some developer wrote as an add on

[Fuzzy]

You can specify memory limits programatically in .NET, and even localized to certain 'jobs'. More importantly, this can be done dynamically while the program is running, instead of having to be determined prior to any code execution.

[craigap]

Holy crap, you guys. You turned my brain into a soup sandwich. I'm not this technically inclined when it comes to computer stuff. It's taken me 3 years just to learn how to use imports.

I was thinking about moving to Win7 x64 and throwing in a 2x2 or 2x3 or something memory so I wouldn't run into a JVM issue. Souns like 2600mb is the limit regardless though?

[Fuzzy]

Originally Posted by stuckless

See... isn't that a lot more constructive than your original... "Java sucks... I wish sage were .net" (i paraphrased)

I wasn't trying to start some conversation about JVM vs. CLR, as it's really moot in regards to sage... i was merely trying to answer Stanger's question accurately, with 'no-one seems to be able to figure out where the JVM max comes from'.

[stuckless]

Originally Posted by craigap

Holy crap, you guys. You turned my brain into a soup sandwich. I'm not this technically inclined when it comes to computer stuff. It's taken me 3 years just to learn how to use imports.

I was thinking about moving to Win7 x64 and throwing in a 2x2 or 2x3 or something memory so I wouldn't run into a JVM issue. Souns like 2600mb is the limit regardless though?

I seriously doubt you need to bump it to 2600 (I only did it to see if i could).... personally I would start with 768m or 1024m and only increase it if you need absolutely need to. If you set it to the max that your system can allocate, then you may cause some unwanted side effects in other areas.

As a rule of thumb, I tend to set the Sage max memory to 256mb (which is their base) * the number of of concurrent extenders. (that calculation may be a bit aggressive, but it also doesn't account for plugins, etc, so it may all work well in the end)

[david1234]

Originally Posted by Fuzzy

2. The JVM will never release memory back to the operating system.

Admittedly, this is probably based on the original incarnations of Java being designed primarily for smaller applets, that usually have fixed memory footprints, it doesn't scale well to large dynamic apps like sage.

This could be a religious discussion about memory management

The idea is that if the app needs that much memory it will need that much again, and so it holds onto the memory that it assumes it will need later on. The the application is allowed to grow to the size it needs, and over time it will sort of hit equilibrium.

[Fuzzy]

Originally Posted by david1234

This could be a religious discussion about memory management

The idea is that if the app needs that much memory it will need that much again, and so it holds onto the memory that it assumes it will need later on. The the application is allowed to grow to the size it needs, and over time it will sort of hit equilibrium.

right, I agree that was the IDEA.. but in practice, with an app like sage, it's not the case. For basic sage UI functions, and such, yes, it would use a fairly static set of memory, however, with the newer graphically intensive imports, it simply doesn't work that way. The memory is only needed when the import is being displayed, and is garbaged out when done... however, since java doens't release it to the OS, it takes up unusable space in the memory pool. Yes, it DOES get swapped out to disc after it's been unused for a bit, however, this just means that to re-use that already allocated heap space, windows has to swap IN the effectively empty pages before it can be 'used' again. In a more flexible paradigm (CLR), when more memory was needed, it would simply allocate it from the 'standby' portion of physical RAM, so it would be immediately available to the application, instead of waiting on the pointless swapping.

[stanger89]

Originally Posted by stuckless

Ok... now i really see

I guess i've never hit the upper limit, but, just to test this on my machine, I could create a machine at 2600mb... but not at 2700... and I think you are right, that the limit will vary from machine to machine depending on resources, etc. I think that java fails if it cannot allocate a contiguous block of memory.

Yes exactly. How did you get 2600MB? I tested using a for loop in a cmd prompt: for /l %i in (1000,100,2000) do java -Xmx%im -version

That will print out the version info if successful and fail if it can't create the VM. I've never been able to get anything over about 1600MB, even though the test machine is W7 x64 with 6GB of ram.

I'm guessing the bmt issue was related to the dvd profiler xml... that's a very large file, and bmt uses a DOM parser (not smart) and so it has to cram it all into memory.

You don't need to rub it in It's also probably got something to do that I errantly pointed Sage to import a few hundred thousand media items

And the DOM parser made sense for the way it was used originally. Any interest in a SAX parser?

[stuckless]

Originally Posted by Fuzzy

Yes, it DOES get swapped out to disc after it's been unused for a bit, however, this just means that to re-use that already allocated heap space, windows has to swap IN the effectively empty pages before it can be 'used' again. In a more flexible paradigm (CLR), when more memory was needed, it would simply allocate it from the 'standby' portion of physical RAM, so it would be immediately available to the application, instead of waiting on the pointless swapping.

I've never fully understood the flaw in windows that makes it swap out "aged" memory, just because it hasn't been accessed for a bit. I have a windows machine 4 gigs of memory and it's always put stuff out to swap. In linux, I have 4 gigs of memory and I never have anything in swap and the machine is always super responsive, even when I minimize an app for a few days and then finally focus it. Can the windows swap be optimized to for better performance?

Originally Posted by stanger89

Yes exactly. How did you get 2600MB? I tested using a for loop in a cmd prompt: for /l %i in (1000,100,2000) do java -Xmx%im -version

That will print out the version info if successful and fail if it can't create the VM. I've never been able to get anything over about 1600MB, even though the test machine is W7 x64 with 6GB of ram.

I didn't anything quite a fancy... I just used the command line and keep doubling memory with each new command. I just use the up arrow and change the mem arg until it fails. Once it fails, I just half the difference and then try again... usually in about 8 tries, I have the approx value, which for me was ~2600 (I didn't try 2601, etc)

And the DOM parser made sense for the way it was used originally. Any interest in a SAX parser?

That why I used the DOM parser as well... I think eventually I'm going to change to using dom4j and take advantage of their sax/dom hybrid parsing... or just switch to using sax. I tend to use sax for most things.

[stanger89]

Originally Posted by stuckless

I've never fully understood the flaw in windows that makes it swap out "aged" memory, just because it hasn't been accessed for a bit. I have a windows machine 4 gigs of memory and it's always put stuff out to swap. In linux, I have 4 gigs of memory and I never have anything in swap and the machine is always super responsive, even when I minimize an app for a few days and then finally focus it. Can the windows swap be optimized to for better performance?

I think, but I don't have a clue where I came up with this, that it probably comes from the days of systems with very little memory, and the idea being to "proactively" move data to swap so that you don't get stuck waiting for the disk write (swap) to happen when/if you start a new app/process that needs more memory.

Of course this makes no sense anymore when 12GB of memory costs only $200

I didn't anything quite a fancy... I just used the command line and keep doubling memory with each new command. I just use the up arrow and change the mem arg until it fails. Once it fails, I just half the difference and then try again... usually in about 8 tries, I have the approx value, which for me was ~2600 (I didn't try 2601, etc)

But are you running standard 32bit Java? Which version? What OS? How much memory? Mainly curious, I've read the max VM size possible is theoretically ~1900MB.

That why I used the DOM parser as well... I think eventually I'm going to change to using dom4j and take advantage of their sax/dom hybrid parsing... or just switch to using sax. I tend to use sax for most things.

I forget, are you using DVDPro2Sage or did you roll your own DOM parser? I've got a working but not thoroughly tested, and not worked on for a while SAX parser.