Refactoring also requires more memory in permanent area. Some results
from May21 build on JDK1.4.2, Linux obtained by visualgc (part of
jvmstat tools)

Empty IDE start with 20-21MB of permarea both in trunk and mdr
(obviously MDR is not initialized)

With one simple project it is 21.250 vs 22.
The same project and opened editor shows eats again more with
refactoring - 23.5 vs 25.5.

With 20 projects (jEdit, new java app, rest from netbeans -
mdr+java+deps) it needs 24.750MB in trunk and 27MB with refactoring.

Anyway I do not know if we can save something here quickly.

Moved to new subcomponent java/javacore.

*** Issue 46164 has been marked as a duplicate of this issue. ***

In refactoring builds, there are nearly 100.000 more String instances
(and also 100.000 char[] instances) than before refactoring merge.

Only the char[] instances themself account for 12MB of heap usage
icrease (char[] from ~3MB to ~15MB)

All of these numbers are just after start with quite trivial project
opened.

I can provide you more info about the increase, but few initial
observations:
There are tons of String representations of MOFIDs (where String if
much more memory expensive than the MOFID it represents),
most of them are held by TreeMaps referenced by instances of
javacore.ClassIndex.
Here is the distribution of memory consumed by all the treemaps
used from ClassIndex-es:
java.util.TreeMap: 100/4000B
[Ljava.util.HashMap$Entry;: 4088/142960B
java.util.HashMap: 4088/163520B
java.util.HashSet: 4088/65408B
java.util.HashMap$Entry: 10090/242160B
java.util.TreeMap$Entry: 35004/1120128B
java.lang.String: 76009/1824216B
[C: 76009/7403608B
Total: 209477/10966008B

Correction: The project wasn't *that* simple, it contained ant as a
library (ant means ~20 jars).
Anyway, most of the memory was apparently consumed by the data from
rt.jar and src.zip.
If I also tracked HashMaps (identifierMap), I got >14MB of my 40MB
heap   and 90.000 Strings covered just by  ClassIndex.

Suggestions:
1. All MofIDs in gived ClassIndex instance have the same storageID.
Don't waste precious heap space (5MB) on it. No need to use String at
all, Long should be enough

2. why fqnMap and simpleMap? Wouldn't it be possible to cover it with
single map? (Hint: keep the package prefix separate and only once, use
smarter search)

3. Strings are expensive, especially for such a big database. Try to
avoid using them. Use e.g. byte arrays instead.

4. For read-only resources, you can use other means of reducing memory
usage at the cost of runtime modification. Using e.g. single large
byte array instead of a lot of small ones can save 9B on each in average.

It also seems that each identifier is twice on the heap (two String
instances that are != but are eqauls())
This is probably because of src.zip / rt.jar
It is surely worth finding a mechanism for sharing the String instances
(better NOT using intern())

Also note that those String MOFID representations are also duplicated,
probably because of two maps in the ClassIndex.

Thanks for your measurements.

Re 1 - yes, we are planning to implement this (i.e. store just longs).
Hopefuly the performance will not be hurt too much (especially the
"new |" case in code completion could be affected - we will see)
Re 2 - I am affraid this would have huge performance implications
since fqn index is used very heavily and simple name index has to be
case insensitive. Moreover implementing this would be quite risky at
this point of time. And the gain would be maybe not even 1MB for 20
classpath elements after 1) is implemented.
Re 3,4 - I am not sure if I understad. Could you give some examples?

Remeasured again for really simple project (start with clean userdir,
create project (let it parse JDK), restart, measure), the results are:
JDK1.4.2_04: ~11.5MB, 7+1.7MB in char[]+String
JDK1.5.0: ~17MB, 10.5+2.5 in char[]+String

re 3:
String "TreeTableModel" consumes 24+40=64B, while its byte array
representation consumes just 28B, including array object overhead.

re 4:
If you merge more byte arrays together \0 separated and keep just
indexed, you have 5 bytes overhead instead of average 14.
That makes additional >250KB difference on 30000 strings
You should be familiar with this technique, it is used e.g. by javac
for keeping symbol tables.
What we're talking here about is the reduction of heap space used
solely by ClassIndex from 15MB to less than 5MB while keeping the same
or better performance.

The classindex now stores longs instead of Strings for MOFIDs ->
lowering priority back to P2.
Surprisingly the memory consumption after initial scanning did not get
any better (or at least not better than about 1MB), which is
definitely strange. Petr N., could you please try to measure it again
if there is something else we need to fix?
-------
Checking in javacore/src/org/netbeans/modules/javacore/ClassIndex.java;
/cvs/java/javacore/src/org/netbeans/modules/javacore/ClassIndex.java,v
 <--  ClassIndex.java
new revision: 1.10; previous revision: 1.9
done

I really see less cahr arrays containing MOFIDs. There are ~1500 of
them in my IDE with two simple projects and 3 short files opened in
editor. There were tens thousands of them before. 

I don't have numbers from GC toolbar (now it is 27M of 45M). Tonda
will add them.

I can confirm saving of over 8MB just by using this fix.
Mesaured after restart, not after the initial scan. I have not yet
traced memory after initial scan, as it is not _that_ important.

Distribution of ClassIndex objects with this fix:
[Ljava.util.HashMap$Entry;: 12/133952B
java.util.HashMap: 12/480B
java.util.TreeMap: 24/960B
[I: 14289/1673840B
java.lang.Long: 14289/228624B
java.util.HashMap$Entry: 14289/342936B
java.lang.String: 35480/851520B
[J: 35480/865584B
[C: 35480/2685392B
java.util.TreeMap$Entry: 35480/1135360B
Total: 184835/7918648B

That means the rest of Strings consume 3.55MB, which can probably
still be halved by using temporal String pool to eliminate duplicates.

I can also confirm a significant saving, measuring using the memory
toolbar.

Comparing a build with the fix and before the fix running on jdk
1.5.0-b58 on W2K I see a gain of 4.4-7.4MB.


Empty start:  7.7/23.9MB (used to be 8.6/23.9)
Open jEdit, initial scan:  24.8/54.2MB (used to be 29.2/73.5)
Open 3 files and restart:  29.2/63.3MB (used to be 36.6/70.5MB)

Thanks for verifying this. I know there are definitely less char
arrays. There is no doubt. According to our computations from the
number of classes in index and length of the MOFID string we should
really save about 8MB. We did see this difference after IDE restart.
What surprised me was that the heap size after the initial scan was
not affected much so I was wondering what might be wrong.
Btw. the numbers that Tonda attached may also be affected by the fact
that we are no longer scanning sun.* and sunw.* packages - that should
also save us some memory. Decrease in memory consumption after the
very first start should have nothing to do with the changes in javacore.

> Btw. the numbers that Tonda attached may also be affected by the fact
> that we are no longer scanning sun.* and sunw.* packages - that should
> also save us some memory

No, my numbers are not affected by that fix. I've used the build
20040722-0713 for measuring.

FYI:
The numbers with wholesale interning of all Strings comming through
IOUtils.readString:

java.lang.String: 19500/468000B
[C: 19500/1471896B
Total: 152875/6322528B

Another 1.6MB spared on this structure - nearly half of the Strings.

The general savig over the IDE was bigger, as other duplicate Strings
got merged by this change (issue 46347)

(For anybody doing measurements: Watching memory meter for this change
is not enough anymore, part of the strings were shifted to perm.
generation, which is not covered by memory meter)
 

So, should I intern those strings?

Talked about this with Petr today. Yes, please, do intern the strings.

All strings deserialized using IOUtils.readString are now interned.

Checking in org/netbeans/mdr/util/IOUtils.java;
/cvs/mdr/src/org/netbeans/mdr/util/IOUtils.java,v  <--  IOUtils.java
new revision: 1.29; previous revision: 1.28
done

Petre, please, add a note here if you find anything interesting in the
insane dump you talked about on today's meeting.

Otherwise, are we out of ideas?

Hard to process such a big dataset...
Preliminary findings:
My 92MB heap (after closing all editors and collapsing all trees)
consumed about 15MB just in byte arrays, several MB each in char
arrays (163.000 instances), object arrays(77.000 instances), String
instances (170.000), HashMap$Entry instances (180.000), ArrayList
instances (56.000), MOFIDs (42.000).

I've (a bit) analyzed only byte arrays so far, here over 8MB is
consumed by 4273 instances of byte[2048], mostly referenced by
CachedPage (3712) instances. The rest of byte arrays are mostly small
(byte[] representation of MOFIDs, not consuming significat amount of
memory. The rest is few large arrays of (generally) different sizes
used by name tables (com.sun.tools.javac.util.Name$Table)

I haven't yet followed the incomming references to individual kinds of
arrays...

Re. name tables:
There are 16 of them in my dump and they cover following memory:
[B: 16/2097408B
[Lcom.sun.tools.javac.util.Name;: 16/2097408B
com.sun.tools.javac.util.Name$Table: 16/6528B
com.sun.tools.javac.util.Name: 5753/138072B
Total: 5801/4339416B
I don't know why they are held so much and what they are used for,
but from the numbers, it is obvious that nearly 2MB of all 4MB is
wasted on oversized Name[] tables (2MB for 16 tables means 32k slots
for each table - 512k total slots - while there are only 5753 slots
really used!)

They are referenced through a (strong) cache of MDR parsers:

org.netbeans.modules.javacore.parser.ElementInfo$CachedReference.CACHE->
org.netbeans.modules.javacore.parser.ElementInfo$1@a26->
[Ljava.util.HashMap$Entry;@2e0f->
java.util.LinkedHashMap$Entry@630b->
org.netbeans.modules.javacore.parser.MDRParser@aefe->
org.netbeans.lib.gjast.ASParser@173c9->
com.sun.tools.javac.util.Name$Table@2b0a1

CachedPages are referenced from FileCaches (org.netbeans.mdr.persistence)

There are 58 FileCaches in my dump, each holding 64 pages, 2KB a piece.
They are all strongly referenced and probably never freed from memory,
which is very poor caching strategy (for my few opened projects, this
means 7.5MB just in byte arrays)

Hm. This may be it. I haven't realized there is one more cache in the
b-tree (besides MDRCache), so the optimizations I tried were done in
MDRCache only. (I cannot get OptimizeIt to work on my Linux :( )
I think we also can do something about the name tables Petr mentioned
in the previous comment.

Note: I've (for verification) remeasured memory consumed by class
indexes, and only ~6MB (e.g. 20.000 ot of 180.000 Strings) was
consumed by them, so we'll need to look for other String consumers as
well.
More to come...

Some numbers taken on build from Aug 23 running for two days. I will
try to analyze the dump. It is IDE with 38 projects (mostly NB modules).
 
[Lorg.netbeans.lib.java.parser.Token;: 27914, 460888B
org.netbeans.mdr.persistence.btreeimpl.btreestorage.CachedPage: 10752,
516096B
org.netbeans.mdr.NBMDRepositoryImpl$FacilityCache$HandlerReference:
13142, 525680B
java.lang.Integer: 43657, 698512B
java.util.ArrayList: 33887, 813288B
java.util.HashMap: 24926, 997040B
org.netbeans.lib.java.parser.ScannerToken: 34416, 1376640B
org.netbeans.mdr.persistence.MOFID: 70550, 1693200B
[J: 66605, 1697600B
java.util.TreeMap$Entry: 67233, 2151456B
[Ljava.lang.Object;: 46591, 2865336B
[Ljava.util.HashMap$Entry;: 25183, 3657704B
java.lang.String: 179297, 4303128B
java.util.HashMap$Entry: 221695, 5320680B
[I: 33093, 7943104B
[C: 173818, 16640344B
[B: 33228, 31869408B
Total size of all found objects: 101322152

Hm, my instances of CachedPage are held by FileCache or
IntrusiveList$Member (cool name) and I have 168 instances of FileCache
objects. 

It means each FileCache has 64 CachedPages -> 0.5MB of CachedPages +
keeping 2kB per object gives more than 20MB. So we have >128kB per
FileCache and several of them per project (/platform/classpath item?). 

Guys, I think I finally found the main cause of the memory problems.
It is the partitioning of TransactionCache and FileCache. I will try
to fix this tomorrow. I've already spent about 3 hours experimenting
with the TransactionCache and it is looking very promissing - I fixed
growing of the TransactionCache with the number of classpath elements.
I still get OOME if I do Find Usages on FileObject when web/project
and required projects open. I hope this will be fixed by sharing the
FileCache between storage partitions which I will focus on tomorrow.
Thanks to all for your continuous help with memory profiling.

Per discussion, -> P1

re name tables again:

Petr wrote that we have 2MB in Name[] kept by cache of 16 parsed
trees. There are also 16 byte buffers each of them keeping 64kB kept
by this chain - ASParser -> ASEnter -> ASClassReader -> byte array
64kB (more precisely kept by com.sun.tools.javac.jvm.ClassReader.buff
field). This is another 1MB of memory.

Perhaps we can split this issue into smaller pieces - name tables,
Btree storage (file caches, transaction caches).

The ASParser is not held by the cache anymore - so the name tables and
ASEnter stuff should disappear. This was fixed by Tom Ball yesterday.

Those tons of ArrayLists are comming from:
org.netbeans.modules.javacore.jmiimpl.javamodel.DeferredAttrList: 13457
org.netbeans.mdr.storagemodel.AttrList: 11905
org.netbeans.modules.java.bridge.CallableImpl$ExceptionsListener: 7710
org.netbeans.modules.java.bridge.CallableImpl$ParametersListener: 7710

AttrLists are held by different paths from different $Impl classes,
but there are 14840 FacilityCache$HandlerReference instances, so there
are about that much live $Impl objects. I have not tracked the $Impl
objects thoroughly, so I'm attaching only one quite suspicious trace:

org.netbeans.modules.javacore.internalapi.JavaMetamodel.defaultRepository->
org.netbeans.mdr.NBMDRepositoryImpl@b50->
org.netbeans.mdr.storagemodel.MdrStorage@2303->
org.netbeans.mdr.util.EventNotifier@5063->
org.netbeans.mdr.util.EventNotifier$Instance@9113->
java.util.HashMap@14fe0->
[Ljava.util.HashMap$Entry;@284d2->
java.util.HashMap$Entry@3f615->
org.netbeans.jmi.javamodel.MultipartId$Impl@2befa

One source of the highly increased number of strings is the MOFID
string-form caching. There are ~43.000 MOFID instances (1MB) on the
heap, roughy half of them keeping also the string form - ~19.800
Strings consuming about 2.5MB.

The MOFIDs are related to the facility cache ($Impl instances keep
MOFID in them) so reducing number of live $Impls should help here,
but I thought we've reduced string-form usage so we can:
A) avoid the string form usage altogether
B) avoid caching the value (as long as we keep all remaining usages
out of critical paths)

One place that uses the string form explicitely is the
FacilityCache/HandlerReference.

re Martin on Aug 27: ASParser is still referenced by
ASParser.BridgeContext

Yesterday I have commited several changes to MDR and javacore that
should significanlty reduce the heap space needed per classpath
element. I believe these improvements should be good enough to lower
the priority off this issue. Anyway, I haven't tried it with 900 jars.
Tondo, could you please try it? (it should be in today's build, I
commited the changes around 19:00)

These changes are really in build from 2004-08-30 -

2004-08-30 16:58  mmatula
 
        * java/javacore/src/org/netbeans/modules/javacore/:
JMManager.java,
        jmiimpl/javamodel/UsageFinder.java:
         Heap usage improved. Special cache provided for storage
        partitions.
 
2004-08-30 16:57  mmatula
 
        * mdr/src/org/netbeans/mdr/persistence/btreeimpl/:
        btreeindex/IntInfo.java, btreeindex/MOFIDInfo.java,
        btreeindex/StringInfo.java, btreestorage/BtreeDatabase.java,
        btreestorage/BtreeFactory.java, btreestorage/BtreeStorage.java,
        btreestorage/CachedPage.java, btreestorage/FileCache.java,
        btreestorage/MDRCache.java, btreestorage/TransactionCache.java:
         Heap usage significantly improved. Caches are now static rather
        than per storage file.

Let's talk about this issue on tomorrow's performance meeting.

As discussed on the meeting. Let's keep the bug open and P1 until the
three outstanding issues are fixed:

1) ASParser instances unnecessarily hanging around,

2) facility cache should not be using MOFIDs as strings and
MOFID.toString() caching should be turned off,

3) the likely leak with impl instances referenced by EventNotifier

The first problem seems to be fixed:

User: tball   
  Date: 04/08/31 13:51:07

  Modified:    external gjast.jar.scrambled
  Log:
  Made gjast's BridgeContext static, so that its associate parser can be
  released.  Verified with JFluid.
  
  Revision  Changes    Path
  1.68      +297 -299  java/external/gjast.jar.scrambled
  
 
http://java.netbeans.org/source/browse/java/external/gjast.jar.scrambled?rev=1.68&content-type=text/x-cvsweb-markup
  
  	<<Binary file>>

Fixed 2:

Checking in org/netbeans/mdr/NBMDRepositoryImpl.java;
/cvs/mdr/src/org/netbeans/mdr/NBMDRepositoryImpl.java,v  <-- 
NBMDRepositoryImpl.java
new revision: 1.81; previous revision: 1.80
done
Processing log script arguments...
More commits to come...
Checking in org/netbeans/mdr/handlers/BaseObjectHandler.java;
/cvs/mdr/src/org/netbeans/mdr/handlers/BaseObjectHandler.java,v  <-- 
BaseObjectHandler.java
new revision: 1.46; previous revision: 1.45
done
Checking in org/netbeans/mdr/handlers/InstanceHandler.java;
/cvs/mdr/src/org/netbeans/mdr/handlers/InstanceHandler.java,v  <-- 
InstanceHandler.java
new revision: 1.34; previous revision: 1.33
done
Checking in org/netbeans/mdr/handlers/PackageProxyHandler.java;
/cvs/mdr/src/org/netbeans/mdr/handlers/PackageProxyHandler.java,v  <--
 PackageProxyHandler.java
new revision: 1.23; previous revision: 1.22
done
Processing log script arguments...
More commits to come...
Checking in org/netbeans/mdr/persistence/MOFID.java;
/cvs/mdr/src/org/netbeans/mdr/persistence/MOFID.java,v  <--  MOFID.java
new revision: 1.6; previous revision: 1.5
done

Fixed 3:

Checking in org/netbeans/mdr/util/EventNotifier.java;
/cvs/mdr/src/org/netbeans/mdr/util/EventNotifier.java,v  <-- 
EventNotifier.java
new revision: 1.14; previous revision: 1.13
done

Keeping String version of MOFID through WeakReference is not good
solution IMO. It introduces big overhead for garbage collector. OK,
the MOFID string is shared for all calls but we would like to avoid
producing these string. Currently I see >2500 MOFIDs created from
TagSupport.getTagValue just during startup. Bunch of other calls can
be saved in MDRClassLoader.resolveInterface if we use different key
(directly MOFID instance? or even long value seems to be working).
Another place where we can avoid it is ResourceImpl (used during
editing).  

Created attachment 17335 [details]
patch to search for MOFID.toString callers

Lowering priority (maybe you should file a separate issue and leave
this one closed). I really do not think this is a P1. Even if we catch
every single usage of the string representation of MOFID, we just
cannot catch them all, since MOFID is not an API. In javacore module
we are able to use it right know just thanks to the implementation
dependency on mdr. Given this, I don't think we should remove weak
referencing of the string MOFID representation. It is quite common for
clients of MDR to remember just mof ids and fetch objects on demand.
Without weak referencing we may end up with several copies of the
string representing the same MOFID.

Using WR for Strings is good intermediate solution.
Anyway, I can verify #2 fixed, for my last dump, for >40.000 MOFIDs,
only ~900 of them had string representation which means less than
150kB overhead.

I can't verify #1 though!
Although all ASParser instances are really gone and no Name$Table
instances can be found, there are tons of remaining Name instances
comming from org.netbeans.lib.gjast.ASNameTable.
There seems to be remaining singke instance of the table held by one
of the names in it through following chain:
org.netbeans.modules.javacore.parser.ElementInfo$CachedReference.CACHE->
org.netbeans.modules.javacore.parser.ElementInfo$1@6fb->
[Ljava.util.HashMap$Entry;@1d47->
java.util.LinkedHashMap$Entry@3b2d->
org.netbeans.modules.javacore.parser.MDRParser@a089->
org.netbeans.lib.gjast.ASTopLevel@19e8a->
com.sun.tools.javac.util.Name@3317d->
org.netbeans.lib.gjast.ASNameTable@4aa49

In my case it still consumes about 1MB (20.000 Names), but not that
big problem anymore.


But we can't lower the priority it seems, as there seems to be
another big (and easily reproducible) problem with leaking
$Impl instances. Try following scenario:

Start IDE against clean userdir
Open java/j2seproject with all subprojects
Let it parse
Restart the IDE (to benefit from String interning in MDR)
Now the IDE runs in ~30MB of heap (including loaded all class indexes)
Find all subclasses of HelpCtx.Provider
The scan will eventually finish but will fight with memory at the end.
Close the results
GC as much as you can.
Now the IDE runs in ~80MB of heap.

This means 50MB is wasted somewhere, be it untuned caches or whatever.
This also means that the IDE has very limited remaining memory
for real usage (e.g. opening some file and editing it!)

Investigating more.

Example trace to Method$Impl instance:
org.netbeans.modules.javacore.jmiimpl.javamodel.JavaPackageClassImpl.mainProxy->
org.netbeans.jmi.javamodel.JavaPackageClass$Impl@cb6->
org.netbeans.mdr.storagemodel.MdrStorage@1990->
org.netbeans.mdr.util.EventNotifier@33e8->
org.netbeans.mdr.util.EventNotifier$Instance@62da->
org.netbeans.mdr.util.EventNotifier$1@1001f->
java.util.HashMap@22c8c->
[Ljava.util.HashMap$Entry;@3e5b0->
java.util.HashMap$Entry@6056e->
org.netbeans.mdr.util.EventNotifier$RegisteredListenerSet@93ebd->
java.util.HashMap@cea4d->
[Ljava.util.HashMap$Entry;@fab19->
java.util.HashMap$Entry@11a013->
org.netbeans.modules.java.bridge.TopClassesCollection$TopClassesListener@12fb3f->
java.util.ArrayList@146c91->
[Ljava.lang.Object;@15a9ec->
org.netbeans.jmi.javamodel.JavaClass$Impl@3e5e2->
org.netbeans.modules.javacore.jmiimpl.javamodel.LightAttrList@60795->
java.util.ArrayList@9426f->
[Ljava.lang.Object;@ced57->
org.netbeans.jmi.javamodel.Method$Impl@22cb0

Notes: All references to other instances are going from mainProxy.
All of them goes through TopClassesListener.
Interresting instance counts after the mentioned scenario:
Method$Impl: 10747
Parameter$Impl: 9673
Field$Impl: 5054
JavaPackage$Impl: 2598
JavaClass$Impl: 1461
Constructor$Impl: 1178
Resource$Impl: 693
TopClassesCollection: 678
TopClassesListener: 678
...

Tom, could you please investigate if something could be done about the
Name instances held by ASTopLevel (mentioned by the comment before the
last one)?

I guess the trace containing the mainProxy is misleading. As you can
see, mainProxy transitively holds MdrStorage which holds
EventNotifier. There is just one instance of these two objects in the
VM, so I guess anything that references MdrStorage transitively
references also the objects you mention (in case they are all
referenced by MdrStorage). Is this the case?

Reassigning to Pavel, since I am on vacation.

This seems to be a problem in src bridge given that all references go
through TopClassesListener.

The name table entries are not a leak:  they are now in a single,
shared name table.  I did this because scanning the src.zip file
created thousands of name tables which contained lots of identical
entries.  

"In my case it still consumes about 1MB (20.000 Names), but not that
big problem anymore."

So, I can remove the table sharing, but you'll get a performance hit
-- you cannot have it both ways.  It's your call, gentlemen.

Tom,
As long as the table is usefull there (i.e. is reused during the next
parsing 20s later), it should stay. If it is used only during one-shot
parsing of several files and have to be discarded for next batch,
it should go away after the batch. When I was discussing those
original 16 tables, I was told they are not needed and shouldn't be there.

The shared name table is used for every parse request from MDR and
every error check request from the Editor.  That said, it should be
soft-referenced so that the garbage collector can discard it if
necessary in between parses.  That way it doesn't have to stay 1 MB,
since it only grew that big from the union of all sources during the
initial scan.  I'll take care of that today.

One more problem in src bridge it seems:
I found ~15MB in Strings which represented ~700 whole source files.
The traces were:
org.netbeans.modules.javacore.jmiimpl.javamodel.JavaPackageClassImpl.mainProxy->
org.netbeans.jmi.javamodel.JavaPackageClass$Impl@cb6->
org.netbeans.mdr.storagemodel.MdrStorage@1990->
org.netbeans.mdr.util.EventNotifier@33e8->
org.netbeans.mdr.util.EventNotifier$Instance@62da->
org.netbeans.mdr.util.EventNotifier$1@1001f->
java.util.HashMap@22c8c->
[Ljava.util.HashMap$Entry;@3e5b0->
java.util.HashMap$Entry@60712->
org.netbeans.mdr.util.EventNotifier$RegisteredListenerSet@941a9->
java.util.HashMap@cec8c->
[Ljava.util.HashMap$Entry;@fad25->
java.util.HashMap$Entry@11a3a1->
org.netbeans.modules.java.bridge.SourceElementImpl$SourceListener@fd3b5->
org.netbeans.jmi.javamodel.Resource$Impl@23b10->
org.netbeans.modules.javacore.parser.ResourceInfo@3f6c6->
java.lang.String@62565

The source files held this way are probably the sources that
matched the hash for HelpCtx.Provider and had to be parsed
to resolve their subclass status for my Where Used query.

Martin, to answer your question re. mainProxy:
Sure, path through mainProxy is by chance just a shortest
static referenced path to the objects in question.
In fact, everything up to [Ljava.util.HashMap$Entry;@3e5b0
is common to all the traces, as the objects on the path before
are singletons.

Dane, can you take a look at the src bridge?

I tested scenario as Petr Nejedly described on 2004-09-02. It seems it
is not reproducible in current trunk build. After closing results, the
memory meter was under 50MB. Dan Prusa tried it also and had similar
results. Also I didn't find thousands of JMI objects in memory when
results were closed. Lowering priority because I am not able to
reproduce it.

I tried it also and get memory usage under 50MB. The count of
instances is lower than Petr's except JavaPackage$Impl:
    135 org.netbeans.jmi.javamodel.AnnotationTypeClass$Impl
     39 org.netbeans.jmi.javamodel.ArrayClass$Impl
      1 org.netbeans.jmi.javamodel.ArrayReferenceClass$Impl
    135 org.netbeans.jmi.javamodel.CodebaseClass$Impl
     43 org.netbeans.jmi.javamodel.ConstructorClass$Impl
     53 org.netbeans.jmi.javamodel.Constructor$Impl
     81 org.netbeans.jmi.javamodel.ContainsResources$Impl
      2 org.netbeans.jmi.javamodel.ElementPartKindEnum
      1 org.netbeans.jmi.javamodel.Extends$Impl
     47 org.netbeans.jmi.javamodel.FieldClass$Impl
      8 org.netbeans.jmi.javamodel.FieldGroupClass$Impl
      1 org.netbeans.jmi.javamodel.FieldGroup$Impl
    172 org.netbeans.jmi.javamodel.Field$Impl
     42 org.netbeans.jmi.javamodel.Implements$Impl
     17 org.netbeans.jmi.javamodel.ImportClass$Impl
    189 org.netbeans.jmi.javamodel.Import$Impl
     28 org.netbeans.jmi.javamodel.InitializerClass$Impl
      3 org.netbeans.jmi.javamodel.Initializer$Impl
    136 org.netbeans.jmi.javamodel.JavaClassClass$Impl
     65 org.netbeans.jmi.javamodel.JavaClass$Impl
      2 org.netbeans.jmi.javamodel.JavaDocClass$Impl
     20 org.netbeans.jmi.javamodel.JavaDoc$Impl
    135 org.netbeans.jmi.javamodel.JavaEnumClass$Impl
    136 org.netbeans.jmi.javamodel.JavaModelPackage$Impl
    136 org.netbeans.jmi.javamodel.JavaPackageClass$Impl
   3878 org.netbeans.jmi.javamodel.JavaPackage$Impl
     52 org.netbeans.jmi.javamodel.MethodClass$Impl
    368 org.netbeans.jmi.javamodel.Method$Impl
      2 org.netbeans.jmi.javamodel.MultipartIdClass$Impl
      6 org.netbeans.jmi.javamodel.MultipartId$Impl
     51 org.netbeans.jmi.javamodel.ParameterClass$Impl
     65 org.netbeans.jmi.javamodel.Parameter$Impl
     52 org.netbeans.jmi.javamodel.ParameterizedTypeClass$Impl
      1 org.netbeans.jmi.javamodel.ParameterizedType$Impl
      5 org.netbeans.jmi.javamodel.PrimitiveTypeClass$Impl
      9 org.netbeans.jmi.javamodel.PrimitiveType$Impl
      9 org.netbeans.jmi.javamodel.PrimitiveTypeKindEnum
    135 org.netbeans.jmi.javamodel.ResourceClass$Impl
     16 org.netbeans.jmi.javamodel.Resource$Impl
      2 org.netbeans.jmi.javamodel.TagDefinitionClass$Impl
     15 org.netbeans.jmi.javamodel.TagDefinition$Impl
      2 org.netbeans.jmi.javamodel.TagValueClass$Impl
     19 org.netbeans.jmi.javamodel.TagValue$Impl
      1 org.netbeans.jmi.javamodel.Throws$Impl
     39 org.netbeans.jmi.javamodel.TypeClass$Impl
      1 org.netbeans.jmi.javamodel.UnresolvedClassClass$Impl
      1 org.netbeans.jmi.javamodel.UnresolvedClass$Impl


and I have only two instances of TopClassesCollection and
TopClassesListener.

I can hapily confirm that on 040907, my scenario works OK.
(Although I can reproduce the problem with 040902, it was my custom
build, so maybe I was missing some fix).

For comparison, my find usages now finishes in 3:54 (vs. 8:15) and
needs 87MB* (vs. 110MB) of heap to finish. Memory after found usages
is 51.2MB (vs. 96.9MB) and after closing results is 37.6MB (vs.
77.0MB), up by 10MB from before find usages, but that's probably
acceptable as the IDE was cold started with all caches empty initially.

*) needed memory measured as the memory usage after last few GC cycles
during scan. It is bottom guess (modulo softrefs) of memory needed to
perform the operation. 

Initial name tables are now much smaller than javac's default size. 
This is possible because javacore only feeds gjast one source file at
a time, rather than javac's normal (and more efficient) batch model.

Is this bug still valid?
I opened core project, and the IDE consumes 23 MB.
Can we close this issue? 

I suggest that we discuss this issue on Wednesday's perf meeting.

After discussion with the performance team, this issue can be
considered to be fixed.

Yes, we are done with this umbrella issue for 4.0.
We'll file new issues for memory related problems for 4.1 as appropriate.

Reorganization of java component