We think we improved the memory usage in build 040519. We did this by
not referencing individual elements from semantic infos directly, but
via unique IDs. Semantic info is a map that is populated by the parser
when the source code is attributed. It stores information about what
element a given element resolves to (e.g. for "System.out" in the code
it resolves to the field "out" in the class "java.lang.System").
By doing this improvement all the elements are now better collected,
also the elements representing method bodies (expressions/statements)
are now GC'd as soon as they are not referenced. Here are other memory
usage improvements we plan to make:
1) ASTs are now held by JMI elements using soft references. We plan to
change these to weak references and add a small hard-referencing cache
for the few most recently accessed ASTs. This will result in more
"polite" behavior of the javacore module and eager garbagecollection
of unnecessary data.
2) With any feature (resource (representing a java or class file,
class, field, method) all of the other features in a given resource
and the resource itself is kept in memory. I don't consider this to be
too critical by itself, however, MDR has a cache that caches JMI
objects which can currently hold 2048 objects per classpath root. If
every one of those object happens to be from a different resource,
this could result in keeping multiple times more objects in memory -
thus will of course never happen, but still can be a problem. There
are two possible solutions to this: a) make the MDR cache for storage
partitions representing classpath roots significantly smaller (we need
to measure the impact on performance), or, b) make sure the features
do not need to hardreference the resource and that the resource does
not need to hardreference the features
3) Split semantic infos into more "per-method" maps - this way the
sematic info associated with all elements in a given method can be
garbagecollected together with the method body once not referenced.

ad 1) I agree. We should switch soft refs to weak one soon and
evaluate the impact.

ad 2) Currnetly we do not have knowledge how effective the caches are.
It means it is uncertain how it will affect memory usage. 

Anyway we have to try these proposals.

Moved to new subcomponent java/javacore.

*** Issue 42475 has been marked as a duplicate of this issue. ***

I have replaced the soft references to the ASTrees by weak references
and added a very simple LRU cache which we can replace/improve later.
The cache size is now set to 16 by default and can be controlled by
"org.netbeans.ASTCache.size" system property. IMO the memory
consumption was improved significantly by this change. After doing
usages of jEdit class the memory usage was:

when max. heap set to 160MB:
- used heap: 91MB, 
- used heap after repeated gc.: 55MB

when max. heap set to 96MB:
- used heap: 80MB
- used heap after repeated gc.: 64MB

With both max heap settings the operation took about the same time.

Refactorings however still consume a lot of memory since the ASTrees
for all changed files are hard-referenced till the transaction is
commited. We have ideas how to fix even that, so hopefuly we can
resolve this soon.

I have just commited some changes that should significantly improve
the memory usage of refactoring operations. I am now able to do a
rename of jEdit class with -Xmx96m. I suggest to close this issue.

Checking in
src/org/netbeans/modules/javacore/jmiimpl/javamodel/JavaClassImpl.java;
/cvs/java/javacore/src/org/netbeans/modules/javacore/jmiimpl/javamodel/JavaClassImpl.java,v
 <--  JavaClassImpl.java
new revision: 1.22; previous revision: 1.21
done
Checking in
src/org/netbeans/modules/javacore/jmiimpl/javamodel/ResourceImpl.java;
/cvs/java/javacore/src/org/netbeans/modules/javacore/jmiimpl/javamodel/ResourceImpl.java,v
 <--  ResourceImpl.java
new revision: 1.19; previous revision: 1.18
done
Checking in
src/org/netbeans/modules/javacore/jmiimpl/javamodel/SemiPersistentElement.java;
/cvs/java/javacore/src/org/netbeans/modules/javacore/jmiimpl/javamodel/SemiPersistentElement.java,v
 <--  SemiPersistentElement.java
new revision: 1.21; previous revision: 1.20
done
Processing log script arguments...
More commits to come...
Checking in src/org/netbeans/modules/javacore/parser/ElementInfo.java;
/cvs/java/javacore/src/org/netbeans/modules/javacore/parser/ElementInfo.java,v
 <--  ElementInfo.java
new revision: 1.11; previous revision: 1.10
done
Checking in src/org/netbeans/modules/javacore/parser/FeatureInfo.java;
/cvs/java/javacore/src/org/netbeans/modules/javacore/parser/FeatureInfo.java,v
 <--  FeatureInfo.java
new revision: 1.7; previous revision: 1.6
done
Checking in src/org/netbeans/modules/javacore/parser/MDRParser.java;
/cvs/java/javacore/src/org/netbeans/modules/javacore/parser/MDRParser.java,v
 <--  MDRParser.java
new revision: 1.30; previous revision: 1.29
done
Checking in src/org/netbeans/modules/javacore/parser/ResourceInfo.java;
/cvs/java/javacore/src/org/netbeans/modules/javacore/parser/ResourceInfo.java,v
 <--  ResourceInfo.java
new revision: 1.7; previous revision: 1.6
done

The improvement is very significant. Refactoring now really works with
mx96m (for the jEdit test cases).

1) JDK sources parsed after a new project creation:
trunk 200407191800: 27/52MB

2) project (jEdit) parsed after project open:
trunk 200407191800: 27/52MB

3) IDE started with a project open (jEdit):
trunk 200407191800: 23/31MB

4) platform change from 1.4.2_04 to 1.5.0-beta2:
trunk 200407191800: 31/66MB

5) Find Usages of a class name (jEdit):
trunk 200407191800:
   70/95MB if mx96m   (took 148s)
   71/127MB if mx128m (took 96s)
   71/129MB if mx160m (took 95s)

6) Rename of a class (jEdit->jEditor):
trunk 200407191800:
   77/95MB if mx96m   (took 3:40)
   77/127MB if mx128m (took 2:08)
   79/157MB if mx160m (took 2:05)

OK, seems that the memory consumption issue is fixed -> closing.

Reorganization of java component