Bug 190336 - Adjust default memory settings
Adjust default memory settings
Status: RESOLVED FIXED
Product: platform
Classification: Unclassified
Component: Launchers&CLI
7.0
All All
: P2 (vote)
: 7.0
Assigned To: Antonin Nebuzelsky
issues@platform
:
Depends on: 189905
Blocks:
  Show dependency treegraph
 
Reported: 2010-09-13 13:39 UTC by Tomas Pavek
Modified: 2010-10-27 02:49 UTC (History)
3 users (show)

See Also:
Issue Type: ENHANCEMENT
:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Tomas Pavek 2010-09-13 13:39:10 UTC
There's increasing need in adjusting the default memory settings the launcher uses for the started JVM for the IDE. The current state was set a few years ago, and since then computers have more memory, people more often use 64bit systems, the IDE itself is bigger, and there are more users working on large projects. I.e. more and more reasons for memory.

Specifically, it seems useful to increase the limit of memory that is given to max heap size (currently 512MB), keeping the algorithm that calculates it from physical memory size.

Also the PermGen size could be bigger sometimes. Currently it's max is fixed at 200MB, it would be more appropriate to also calculate it based on real memory size, with some limit, so if there is enough memory it can be used, and if not it does not take too much from the address space.

Not sure if it is necessary to differentiate between 32 and 64 bit JVM, perhaps the memory size itself is sufficient indicator. With too high limits there could be a problem if running 32bit JVM on 64bit system with a lot of memory. But I think there is no need to set the limits too high, e.g. over 1GB for heap.
Comment 1 Antonin Nebuzelsky 2010-10-08 16:09:32 UTC
> There's increasing need in adjusting the default memory settings
> the launcher uses for the started JVM for the IDE.

Any feedback from real users what they are setting as Xmx and XX:MaxPermSize when they have the need for the increase?

Or any measurements by perf team what should be the reasonable increase?

For the record, the current algorithm is

-J-XX:MaxPermSize=200m (in netbeans.conf)

-J-Xmx(max(96,min(512,RAM/5)))m (computed by launcher)

this maps to

Xmx512m for RAM >= 2.5gb
Xmx409m for 2gb
Xmx204m for 1gb
Comment 2 Petr Jiricka 2010-10-12 08:54:18 UTC
BTW, recently I got a PermGen error while running Ant build - could there be a leak in the Ant area again?
Comment 3 Antonin Nebuzelsky 2010-10-12 12:55:56 UTC
> could there be a leak in the Ant area

see http://netbeans.org/bugzilla/show_bug.cgi?id=189905#c5

Question here is if leaks are the problem for permgen or if some users really go through the roof by using too much functionality in the IDE. But, why should that be the case now. 6.10 is rather a bit smaller than e.g. 6.0.

For heap increase there may be more reasons, mainly users using NetBeans with larger projects... But it's not obvious what is the right increase then.
Comment 4 Tomas Pavek 2010-10-12 17:11:40 UTC
I think current NetBeans is actually noticeably bigger than 6.0 when comparing the same feature areas (i.e. for the same area there is now more features, more classes). Another thing is the mentioned increase of use of 64bit systems which were not considered years ago. They need more memory (OTOH people usually have more physical memory on these systems).

The idea was that we can improve the memory settings even if we have no exact proof about how many people need more memory legitimately and when it is just result of memory leaks. The default memory settings could scale better with the available memory, they don't really have to be bigger for all physical memory configs.

For example:
* Heap size - people really may need more when working on something extremely large, it is strange we give them only 512M even if they have corresponding HW like 8GB, 64bit. I think with 8GB we can afford to set more than 512m for max heap...
* PermGen - there is no big overhead of bigger PermGen except taking address space - so why not to compute it based on available memory as well? With 8GB the address space is not a problem, OTOH with small memory (which can be the case when running in virtualized environment) this may deserve even less than current 200MB.

Some adjustment in this direction could make the IDE work better for the users, regardless if there are memory/class leaks or not...
Comment 5 Antonin Nebuzelsky 2010-10-20 14:39:04 UTC
> Heap - people really may need more when working on something extremely large

Agreed that heap is related to the size of projects and I think we should do one change in the computation - increase the upper boundary from 512m to 768m and keep the rest of the formula untouched, i.e.

   -J-Xmx(max(96,min(768,RAM/5)))m (computed by launcher)

which will map to

   Xmx768m for >=4gb (increased from 512m)
   Xmx614m for 3gb (increased from 512m)
   Xmx409m for 2gb (as before)
   Xmx204m for 1gb (as before)

The 768m is still reasonable size of heap for GC to not be very time-consuming which could be the case for even larger heaps. Also keep in mind that on 32-bit systems the following rule applies - "(Managed Heap + native heap + (thread stack size * number of threads)) cannot exceed 2GB on 32bit x86"

> PermGen - why not to compute it based on available memory as well?

PermGen usage is not related to any size of user data, it is related to number of loaded classes and does not make too much sense to compute flexibly based on user's RAM

I suggest we increase the max permgen size from 200m to 256m, i.e.

   -J-XX:MaxPermSize=256m (in netbeans.conf)

It may help to eliminate increase of size of code in some areas since 6.0 (as you pointed out) and also for x64 users (the increase for them is not 2x but "only" a few tens of percents more (e.g. +20%) compared to 32-bits)
Comment 6 Antonin Nebuzelsky 2010-10-22 11:41:52 UTC
ValidateClassLinkageTest in o.n.core tries to load all classes of modules in clusters currently available in nbbuild/netbeans and allows measuring permgen requirement in the case user would use all the functionality in his IDE instance. I got the following results on my 32-bit Ubuntu with JDK 6u22:

cluster.config=basic
140m of permgen needed

cluster.config=full
190m of permgen needed

(full cluster config includes apisupport, cnd, dlight, enterprise, harness, ide, java, nb, php, platform, profiler, ruby, websvccommon)

So, with some additional room reserved for 64-bit increase, I am going to set the max. permgen to the 256m. When not all of that is actually used, it eats the 256m chunk only from the virtual address space of the process. On the 32-bit the 256m for permgen plus 768m for heap (the upper boundary) plus thread stacks plus JVM overhead should still fit well in the 2gb virtual address space of the process.
Comment 7 Antonin Nebuzelsky 2010-10-22 13:14:48 UTC
http://hg.netbeans.org/core-main/rev/697fffb525b5
Comment 8 Vladimir Voskresensky 2010-10-22 15:12:11 UTC
info about run on my 64 bit Linux system:
cluster.config=basic

----------------------------
  Product Version         = NetBeans Platform Dev (Build 101022-0e82dc0d19ad) (#0e82dc0d19ad)
  Operating System        = Linux version 2.6.32-24-generic running on amd64
  Java; VM; Vendor        = 1.6.0_20; Java HotSpot(TM) 64-Bit Server VM 16.3-b01; Sun Microsystems Inc.
  Runtime                 = Java(TM) SE Runtime Environment 1.6.0_20-b02
Non-heap memory usage: 207/348M

------------------------------------
  Operating System        = Linux version 2.6.32-24-generic running on i386
  Java; VM; Vendor        = 1.6.0_21; Java HotSpot(TM) Server VM 17.0-b16; Sun Microsystems Inc.
  Runtime                 = Java(TM) SE Runtime Environment 1.6.0_21-b06
  Java Home               = /opt/jdk/jdk1.6.0_21/jre
Non-heap memory usage: 141/348M
Comment 9 Vladimir Voskresensky 2010-10-22 16:03:20 UTC
cluster.config=full
failed with OOM:PermGen on 64 bit system in 64 bit mode 

btw, on 64 bit system in 32 bit java mode I have more memory consumption than in your scenario:
Non-heap memory usage: 240/348M

Please, tell me how to give more PermGen for the test to see memory consumption
Comment 10 Vladimir Voskresensky 2010-10-22 16:16:34 UTC
Ok. I got it:
  Operating System        = Linux version 2.6.32-24-generic running on amd64
  Java; VM; Vendor        = 1.6.0_20; Java HotSpot(TM) 64-Bit Server VM 16.3-b01; Sun Microsystems Inc.
  Runtime                 = Java(TM) SE Runtime Environment 1.6.0_20-b02
  Java Home               = /usr/lib/jvm/java-6-sun-1.6.0.20/jre
Non-heap memory usage: 352/648M
Comment 11 Antonin Nebuzelsky 2010-10-25 11:52:26 UTC
To sum up, permgen if all classes are loaded:

cluster.config=basic
(Java / OS bits)
32/32 - 140m
64/64 - 207m

cluster.config=full
(Java / OS bits)
32/32 - 190m
32/64 - 240m
64/64 - 352m

So, for the full config on 64-bit JVM all classes don't fit into the 256m permgen. Whatever probability there is for a user to really load all classes, I think it is a good idea to have the room for them reserved.

I see two options

- keeping MaxPermSize at 256m for 32-bit JVM and increasing to 384m for 64-bit JVM 
  or
- increasing MaxPermSize to 384m regardless of JVM

Figuring out in the launcher if the Java is 64-bit or not means running 'java -version' and grepping "64-bit" from the output which can take from hundreds of miliseconds up to second(s). On the other hand increasing MaxPermSize to 384m even for a 32-bit process should still be OK, so I suggest to set

-J-XX:MaxPermSize=384m statically in netbeans.conf.

Unless anyone has arguments or different measurements against it, I will integrate this change.
Comment 12 Vladimir Voskresensky 2010-10-25 14:26:26 UTC
I agree.
Thanks,
Vladimir.
Comment 13 Quality Engineering 2010-10-25 15:04:42 UTC
Integrated into 'main-golden', will be available in build *201010251131* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/2c5a6d3c4472
User: Jesse Glick <jglick@netbeans.org>
Log: Resurrected. Not currently run by default, but useful for investigating e.g. #190336.
Comment 14 Antonin Nebuzelsky 2010-10-26 07:54:53 UTC
-J-XX:MaxPermSize=384m
core-main #86f3f94551e6
Comment 15 Quality Engineering 2010-10-27 02:49:02 UTC
Integrated into 'main-golden', will be available in build *201010270000* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/86f3f94551e6
User: Antonin Nebuzelsky <anebuzelsky@netbeans.org>
Log: #190336: increase MaxPermSize to 384m for 7.0.


By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo