This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
I made comparison b/w trunk build May/13 and refactoring build May/12. jdk 1.4.2_04 on Linux The same steps for both versions of the IDE - fresh userdir - open nb_all/core projects and its required projects - close all projects - reopen nb_all/core projects and its required projects - close all projects - reopen nb_all/core projects and its required projects After each step I wait for the IDE to get the the idle state and press GC 5 times. Then I recorded the heap usage Both IDEs start with an approximately same heap usages (around 9MB) but there is a huge difference after I open/close the above mentioned projects. Closing all of them doesn't make the heap usage to go down. Trunk: the heap usage went from the 9 MB after the fresh start to 12 MB Refactoring: the increase is from 9MB to 53MB All the above was without doing anything else in the IDE, no code completion, no find usages, absolute nothing.
Refactoring also requires more memory in permanent area. Some results from May21 build on JDK1.4.2, Linux obtained by visualgc (part of jvmstat tools) Empty IDE start with 20-21MB of permarea both in trunk and mdr (obviously MDR is not initialized) With one simple project it is 21.250 vs 22. The same project and opened editor shows eats again more with refactoring - 23.5 vs 25.5. With 20 projects (jEdit, new java app, rest from netbeans - mdr+java+deps) it needs 24.750MB in trunk and 27MB with refactoring. Anyway I do not know if we can save something here quickly.
Moved to new subcomponent java/javacore.
*** Issue 46164 has been marked as a duplicate of this issue. ***
In refactoring builds, there are nearly 100.000 more String instances (and also 100.000 char[] instances) than before refactoring merge. Only the char[] instances themself account for 12MB of heap usage icrease (char[] from ~3MB to ~15MB) All of these numbers are just after start with quite trivial project opened. I can provide you more info about the increase, but few initial observations: There are tons of String representations of MOFIDs (where String if much more memory expensive than the MOFID it represents), most of them are held by TreeMaps referenced by instances of javacore.ClassIndex. Here is the distribution of memory consumed by all the treemaps used from ClassIndex-es: java.util.TreeMap: 100/4000B [Ljava.util.HashMap$Entry;: 4088/142960B java.util.HashMap: 4088/163520B java.util.HashSet: 4088/65408B java.util.HashMap$Entry: 10090/242160B java.util.TreeMap$Entry: 35004/1120128B java.lang.String: 76009/1824216B [C: 76009/7403608B Total: 209477/10966008B
Correction: The project wasn't *that* simple, it contained ant as a library (ant means ~20 jars). Anyway, most of the memory was apparently consumed by the data from rt.jar and src.zip. If I also tracked HashMaps (identifierMap), I got >14MB of my 40MB heap and 90.000 Strings covered just by ClassIndex. Suggestions: 1. All MofIDs in gived ClassIndex instance have the same storageID. Don't waste precious heap space (5MB) on it. No need to use String at all, Long should be enough 2. why fqnMap and simpleMap? Wouldn't it be possible to cover it with single map? (Hint: keep the package prefix separate and only once, use smarter search) 3. Strings are expensive, especially for such a big database. Try to avoid using them. Use e.g. byte arrays instead. 4. For read-only resources, you can use other means of reducing memory usage at the cost of runtime modification. Using e.g. single large byte array instead of a lot of small ones can save 9B on each in average.
It also seems that each identifier is twice on the heap (two String instances that are != but are eqauls()) This is probably because of src.zip / rt.jar It is surely worth finding a mechanism for sharing the String instances (better NOT using intern()) Also note that those String MOFID representations are also duplicated, probably because of two maps in the ClassIndex.
Thanks for your measurements. Re 1 - yes, we are planning to implement this (i.e. store just longs). Hopefuly the performance will not be hurt too much (especially the "new |" case in code completion could be affected - we will see) Re 2 - I am affraid this would have huge performance implications since fqn index is used very heavily and simple name index has to be case insensitive. Moreover implementing this would be quite risky at this point of time. And the gain would be maybe not even 1MB for 20 classpath elements after 1) is implemented. Re 3,4 - I am not sure if I understad. Could you give some examples?
Remeasured again for really simple project (start with clean userdir, create project (let it parse JDK), restart, measure), the results are: JDK1.4.2_04: ~11.5MB, 7+1.7MB in char[]+String JDK1.5.0: ~17MB, 10.5+2.5 in char[]+String re 3: String "TreeTableModel" consumes 24+40=64B, while its byte array representation consumes just 28B, including array object overhead. re 4: If you merge more byte arrays together \0 separated and keep just indexed, you have 5 bytes overhead instead of average 14. That makes additional >250KB difference on 30000 strings You should be familiar with this technique, it is used e.g. by javac for keeping symbol tables. What we're talking here about is the reduction of heap space used solely by ClassIndex from 15MB to less than 5MB while keeping the same or better performance.
The classindex now stores longs instead of Strings for MOFIDs -> lowering priority back to P2. Surprisingly the memory consumption after initial scanning did not get any better (or at least not better than about 1MB), which is definitely strange. Petr N., could you please try to measure it again if there is something else we need to fix? ------- Checking in javacore/src/org/netbeans/modules/javacore/ClassIndex.java; /cvs/java/javacore/src/org/netbeans/modules/javacore/ClassIndex.java,v <-- ClassIndex.java new revision: 1.10; previous revision: 1.9 done
I really see less cahr arrays containing MOFIDs. There are ~1500 of them in my IDE with two simple projects and 3 short files opened in editor. There were tens thousands of them before. I don't have numbers from GC toolbar (now it is 27M of 45M). Tonda will add them.
I can confirm saving of over 8MB just by using this fix. Mesaured after restart, not after the initial scan. I have not yet traced memory after initial scan, as it is not _that_ important. Distribution of ClassIndex objects with this fix: [Ljava.util.HashMap$Entry;: 12/133952B java.util.HashMap: 12/480B java.util.TreeMap: 24/960B [I: 14289/1673840B java.lang.Long: 14289/228624B java.util.HashMap$Entry: 14289/342936B java.lang.String: 35480/851520B [J: 35480/865584B [C: 35480/2685392B java.util.TreeMap$Entry: 35480/1135360B Total: 184835/7918648B That means the rest of Strings consume 3.55MB, which can probably still be halved by using temporal String pool to eliminate duplicates.
I can also confirm a significant saving, measuring using the memory toolbar. Comparing a build with the fix and before the fix running on jdk 1.5.0-b58 on W2K I see a gain of 4.4-7.4MB. Empty start: 7.7/23.9MB (used to be 8.6/23.9) Open jEdit, initial scan: 24.8/54.2MB (used to be 29.2/73.5) Open 3 files and restart: 29.2/63.3MB (used to be 36.6/70.5MB)
Thanks for verifying this. I know there are definitely less char arrays. There is no doubt. According to our computations from the number of classes in index and length of the MOFID string we should really save about 8MB. We did see this difference after IDE restart. What surprised me was that the heap size after the initial scan was not affected much so I was wondering what might be wrong. Btw. the numbers that Tonda attached may also be affected by the fact that we are no longer scanning sun.* and sunw.* packages - that should also save us some memory. Decrease in memory consumption after the very first start should have nothing to do with the changes in javacore.
> Btw. the numbers that Tonda attached may also be affected by the fact > that we are no longer scanning sun.* and sunw.* packages - that should > also save us some memory No, my numbers are not affected by that fix. I've used the build 20040722-0713 for measuring.
FYI: The numbers with wholesale interning of all Strings comming through IOUtils.readString: java.lang.String: 19500/468000B [C: 19500/1471896B Total: 152875/6322528B Another 1.6MB spared on this structure - nearly half of the Strings. The general savig over the IDE was bigger, as other duplicate Strings got merged by this change (issue 46347) (For anybody doing measurements: Watching memory meter for this change is not enough anymore, part of the strings were shifted to perm. generation, which is not covered by memory meter)
So, should I intern those strings?
Talked about this with Petr today. Yes, please, do intern the strings.
All strings deserialized using IOUtils.readString are now interned. Checking in org/netbeans/mdr/util/IOUtils.java; /cvs/mdr/src/org/netbeans/mdr/util/IOUtils.java,v <-- IOUtils.java new revision: 1.29; previous revision: 1.28 done
Petre, please, add a note here if you find anything interesting in the insane dump you talked about on today's meeting. Otherwise, are we out of ideas?
Hard to process such a big dataset... Preliminary findings: My 92MB heap (after closing all editors and collapsing all trees) consumed about 15MB just in byte arrays, several MB each in char arrays (163.000 instances), object arrays(77.000 instances), String instances (170.000), HashMap$Entry instances (180.000), ArrayList instances (56.000), MOFIDs (42.000). I've (a bit) analyzed only byte arrays so far, here over 8MB is consumed by 4273 instances of byte[2048], mostly referenced by CachedPage (3712) instances. The rest of byte arrays are mostly small (byte[] representation of MOFIDs, not consuming significat amount of memory. The rest is few large arrays of (generally) different sizes used by name tables (com.sun.tools.javac.util.Name$Table) I haven't yet followed the incomming references to individual kinds of arrays...
Re. name tables: There are 16 of them in my dump and they cover following memory: [B: 16/2097408B [Lcom.sun.tools.javac.util.Name;: 16/2097408B com.sun.tools.javac.util.Name$Table: 16/6528B com.sun.tools.javac.util.Name: 5753/138072B Total: 5801/4339416B I don't know why they are held so much and what they are used for, but from the numbers, it is obvious that nearly 2MB of all 4MB is wasted on oversized Name[] tables (2MB for 16 tables means 32k slots for each table - 512k total slots - while there are only 5753 slots really used!) They are referenced through a (strong) cache of MDR parsers: org.netbeans.modules.javacore.parser.ElementInfo$CachedReference.CACHE-> org.netbeans.modules.javacore.parser.ElementInfo$1@a26-> [Ljava.util.HashMap$Entry;@2e0f-> java.util.LinkedHashMap$Entry@630b-> org.netbeans.modules.javacore.parser.MDRParser@aefe-> org.netbeans.lib.gjast.ASParser@173c9-> com.sun.tools.javac.util.Name$Table@2b0a1
CachedPages are referenced from FileCaches (org.netbeans.mdr.persistence) There are 58 FileCaches in my dump, each holding 64 pages, 2KB a piece. They are all strongly referenced and probably never freed from memory, which is very poor caching strategy (for my few opened projects, this means 7.5MB just in byte arrays)
Hm. This may be it. I haven't realized there is one more cache in the b-tree (besides MDRCache), so the optimizations I tried were done in MDRCache only. (I cannot get OptimizeIt to work on my Linux :( ) I think we also can do something about the name tables Petr mentioned in the previous comment.
Note: I've (for verification) remeasured memory consumed by class indexes, and only ~6MB (e.g. 20.000 ot of 180.000 Strings) was consumed by them, so we'll need to look for other String consumers as well. More to come...
Some numbers taken on build from Aug 23 running for two days. I will try to analyze the dump. It is IDE with 38 projects (mostly NB modules). [Lorg.netbeans.lib.java.parser.Token;: 27914, 460888B org.netbeans.mdr.persistence.btreeimpl.btreestorage.CachedPage: 10752, 516096B org.netbeans.mdr.NBMDRepositoryImpl$FacilityCache$HandlerReference: 13142, 525680B java.lang.Integer: 43657, 698512B java.util.ArrayList: 33887, 813288B java.util.HashMap: 24926, 997040B org.netbeans.lib.java.parser.ScannerToken: 34416, 1376640B org.netbeans.mdr.persistence.MOFID: 70550, 1693200B [J: 66605, 1697600B java.util.TreeMap$Entry: 67233, 2151456B [Ljava.lang.Object;: 46591, 2865336B [Ljava.util.HashMap$Entry;: 25183, 3657704B java.lang.String: 179297, 4303128B java.util.HashMap$Entry: 221695, 5320680B [I: 33093, 7943104B [C: 173818, 16640344B [B: 33228, 31869408B Total size of all found objects: 101322152
Hm, my instances of CachedPage are held by FileCache or IntrusiveList$Member (cool name) and I have 168 instances of FileCache objects. It means each FileCache has 64 CachedPages -> 0.5MB of CachedPages + keeping 2kB per object gives more than 20MB. So we have >128kB per FileCache and several of them per project (/platform/classpath item?).
Guys, I think I finally found the main cause of the memory problems. It is the partitioning of TransactionCache and FileCache. I will try to fix this tomorrow. I've already spent about 3 hours experimenting with the TransactionCache and it is looking very promissing - I fixed growing of the TransactionCache with the number of classpath elements. I still get OOME if I do Find Usages on FileObject when web/project and required projects open. I hope this will be fixed by sharing the FileCache between storage partitions which I will focus on tomorrow. Thanks to all for your continuous help with memory profiling.
Per discussion, -> P1
re name tables again: Petr wrote that we have 2MB in Name[] kept by cache of 16 parsed trees. There are also 16 byte buffers each of them keeping 64kB kept by this chain - ASParser -> ASEnter -> ASClassReader -> byte array 64kB (more precisely kept by com.sun.tools.javac.jvm.ClassReader.buff field). This is another 1MB of memory. Perhaps we can split this issue into smaller pieces - name tables, Btree storage (file caches, transaction caches).
The ASParser is not held by the cache anymore - so the name tables and ASEnter stuff should disappear. This was fixed by Tom Ball yesterday.
Those tons of ArrayLists are comming from: org.netbeans.modules.javacore.jmiimpl.javamodel.DeferredAttrList: 13457 org.netbeans.mdr.storagemodel.AttrList: 11905 org.netbeans.modules.java.bridge.CallableImpl$ExceptionsListener: 7710 org.netbeans.modules.java.bridge.CallableImpl$ParametersListener: 7710 AttrLists are held by different paths from different $Impl classes, but there are 14840 FacilityCache$HandlerReference instances, so there are about that much live $Impl objects. I have not tracked the $Impl objects thoroughly, so I'm attaching only one quite suspicious trace: org.netbeans.modules.javacore.internalapi.JavaMetamodel.defaultRepository-> org.netbeans.mdr.NBMDRepositoryImpl@b50-> org.netbeans.mdr.storagemodel.MdrStorage@2303-> org.netbeans.mdr.util.EventNotifier@5063-> org.netbeans.mdr.util.EventNotifier$Instance@9113-> java.util.HashMap@14fe0-> [Ljava.util.HashMap$Entry;@284d2-> java.util.HashMap$Entry@3f615-> org.netbeans.jmi.javamodel.MultipartId$Impl@2befa
One source of the highly increased number of strings is the MOFID string-form caching. There are ~43.000 MOFID instances (1MB) on the heap, roughy half of them keeping also the string form - ~19.800 Strings consuming about 2.5MB. The MOFIDs are related to the facility cache ($Impl instances keep MOFID in them) so reducing number of live $Impls should help here, but I thought we've reduced string-form usage so we can: A) avoid the string form usage altogether B) avoid caching the value (as long as we keep all remaining usages out of critical paths) One place that uses the string form explicitely is the FacilityCache/HandlerReference.
re Martin on Aug 27: ASParser is still referenced by ASParser.BridgeContext
Yesterday I have commited several changes to MDR and javacore that should significanlty reduce the heap space needed per classpath element. I believe these improvements should be good enough to lower the priority off this issue. Anyway, I haven't tried it with 900 jars. Tondo, could you please try it? (it should be in today's build, I commited the changes around 19:00)
These changes are really in build from 2004-08-30 - 2004-08-30 16:58 mmatula * java/javacore/src/org/netbeans/modules/javacore/: JMManager.java, jmiimpl/javamodel/UsageFinder.java: Heap usage improved. Special cache provided for storage partitions. 2004-08-30 16:57 mmatula * mdr/src/org/netbeans/mdr/persistence/btreeimpl/: btreeindex/IntInfo.java, btreeindex/MOFIDInfo.java, btreeindex/StringInfo.java, btreestorage/BtreeDatabase.java, btreestorage/BtreeFactory.java, btreestorage/BtreeStorage.java, btreestorage/CachedPage.java, btreestorage/FileCache.java, btreestorage/MDRCache.java, btreestorage/TransactionCache.java: Heap usage significantly improved. Caches are now static rather than per storage file.
Let's talk about this issue on tomorrow's performance meeting.
As discussed on the meeting. Let's keep the bug open and P1 until the three outstanding issues are fixed: 1) ASParser instances unnecessarily hanging around, 2) facility cache should not be using MOFIDs as strings and MOFID.toString() caching should be turned off, 3) the likely leak with impl instances referenced by EventNotifier
The first problem seems to be fixed: User: tball Date: 04/08/31 13:51:07 Modified: external gjast.jar.scrambled Log: Made gjast's BridgeContext static, so that its associate parser can be released. Verified with JFluid. Revision Changes Path 1.68 +297 -299 java/external/gjast.jar.scrambled http://java.netbeans.org/source/browse/java/external/gjast.jar.scrambled?rev=1.68&content-type=text/x-cvsweb-markup <<Binary file>>
Fixed 2: Checking in org/netbeans/mdr/NBMDRepositoryImpl.java; /cvs/mdr/src/org/netbeans/mdr/NBMDRepositoryImpl.java,v <-- NBMDRepositoryImpl.java new revision: 1.81; previous revision: 1.80 done Processing log script arguments... More commits to come... Checking in org/netbeans/mdr/handlers/BaseObjectHandler.java; /cvs/mdr/src/org/netbeans/mdr/handlers/BaseObjectHandler.java,v <-- BaseObjectHandler.java new revision: 1.46; previous revision: 1.45 done Checking in org/netbeans/mdr/handlers/InstanceHandler.java; /cvs/mdr/src/org/netbeans/mdr/handlers/InstanceHandler.java,v <-- InstanceHandler.java new revision: 1.34; previous revision: 1.33 done Checking in org/netbeans/mdr/handlers/PackageProxyHandler.java; /cvs/mdr/src/org/netbeans/mdr/handlers/PackageProxyHandler.java,v <-- PackageProxyHandler.java new revision: 1.23; previous revision: 1.22 done Processing log script arguments... More commits to come... Checking in org/netbeans/mdr/persistence/MOFID.java; /cvs/mdr/src/org/netbeans/mdr/persistence/MOFID.java,v <-- MOFID.java new revision: 1.6; previous revision: 1.5 done Fixed 3: Checking in org/netbeans/mdr/util/EventNotifier.java; /cvs/mdr/src/org/netbeans/mdr/util/EventNotifier.java,v <-- EventNotifier.java new revision: 1.14; previous revision: 1.13 done
Keeping String version of MOFID through WeakReference is not good solution IMO. It introduces big overhead for garbage collector. OK, the MOFID string is shared for all calls but we would like to avoid producing these string. Currently I see >2500 MOFIDs created from TagSupport.getTagValue just during startup. Bunch of other calls can be saved in MDRClassLoader.resolveInterface if we use different key (directly MOFID instance? or even long value seems to be working). Another place where we can avoid it is ResourceImpl (used during editing).
Created attachment 17335 [details] patch to search for MOFID.toString callers
Lowering priority (maybe you should file a separate issue and leave this one closed). I really do not think this is a P1. Even if we catch every single usage of the string representation of MOFID, we just cannot catch them all, since MOFID is not an API. In javacore module we are able to use it right know just thanks to the implementation dependency on mdr. Given this, I don't think we should remove weak referencing of the string MOFID representation. It is quite common for clients of MDR to remember just mof ids and fetch objects on demand. Without weak referencing we may end up with several copies of the string representing the same MOFID.
Using WR for Strings is good intermediate solution. Anyway, I can verify #2 fixed, for my last dump, for >40.000 MOFIDs, only ~900 of them had string representation which means less than 150kB overhead. I can't verify #1 though! Although all ASParser instances are really gone and no Name$Table instances can be found, there are tons of remaining Name instances comming from org.netbeans.lib.gjast.ASNameTable. There seems to be remaining singke instance of the table held by one of the names in it through following chain: org.netbeans.modules.javacore.parser.ElementInfo$CachedReference.CACHE-> org.netbeans.modules.javacore.parser.ElementInfo$1@6fb-> [Ljava.util.HashMap$Entry;@1d47-> java.util.LinkedHashMap$Entry@3b2d-> org.netbeans.modules.javacore.parser.MDRParser@a089-> org.netbeans.lib.gjast.ASTopLevel@19e8a-> com.sun.tools.javac.util.Name@3317d-> org.netbeans.lib.gjast.ASNameTable@4aa49 In my case it still consumes about 1MB (20.000 Names), but not that big problem anymore. But we can't lower the priority it seems, as there seems to be another big (and easily reproducible) problem with leaking $Impl instances. Try following scenario: Start IDE against clean userdir Open java/j2seproject with all subprojects Let it parse Restart the IDE (to benefit from String interning in MDR) Now the IDE runs in ~30MB of heap (including loaded all class indexes) Find all subclasses of HelpCtx.Provider The scan will eventually finish but will fight with memory at the end. Close the results GC as much as you can. Now the IDE runs in ~80MB of heap. This means 50MB is wasted somewhere, be it untuned caches or whatever. This also means that the IDE has very limited remaining memory for real usage (e.g. opening some file and editing it!) Investigating more.
Example trace to Method$Impl instance: org.netbeans.modules.javacore.jmiimpl.javamodel.JavaPackageClassImpl.mainProxy-> org.netbeans.jmi.javamodel.JavaPackageClass$Impl@cb6-> org.netbeans.mdr.storagemodel.MdrStorage@1990-> org.netbeans.mdr.util.EventNotifier@33e8-> org.netbeans.mdr.util.EventNotifier$Instance@62da-> org.netbeans.mdr.util.EventNotifier$1@1001f-> java.util.HashMap@22c8c-> [Ljava.util.HashMap$Entry;@3e5b0-> java.util.HashMap$Entry@6056e-> org.netbeans.mdr.util.EventNotifier$RegisteredListenerSet@93ebd-> java.util.HashMap@cea4d-> [Ljava.util.HashMap$Entry;@fab19-> java.util.HashMap$Entry@11a013-> org.netbeans.modules.java.bridge.TopClassesCollection$TopClassesListener@12fb3f-> java.util.ArrayList@146c91-> [Ljava.lang.Object;@15a9ec-> org.netbeans.jmi.javamodel.JavaClass$Impl@3e5e2-> org.netbeans.modules.javacore.jmiimpl.javamodel.LightAttrList@60795-> java.util.ArrayList@9426f-> [Ljava.lang.Object;@ced57-> org.netbeans.jmi.javamodel.Method$Impl@22cb0 Notes: All references to other instances are going from mainProxy. All of them goes through TopClassesListener. Interresting instance counts after the mentioned scenario: Method$Impl: 10747 Parameter$Impl: 9673 Field$Impl: 5054 JavaPackage$Impl: 2598 JavaClass$Impl: 1461 Constructor$Impl: 1178 Resource$Impl: 693 TopClassesCollection: 678 TopClassesListener: 678 ...
Tom, could you please investigate if something could be done about the Name instances held by ASTopLevel (mentioned by the comment before the last one)?
I guess the trace containing the mainProxy is misleading. As you can see, mainProxy transitively holds MdrStorage which holds EventNotifier. There is just one instance of these two objects in the VM, so I guess anything that references MdrStorage transitively references also the objects you mention (in case they are all referenced by MdrStorage). Is this the case?
Reassigning to Pavel, since I am on vacation.
This seems to be a problem in src bridge given that all references go through TopClassesListener.
The name table entries are not a leak: they are now in a single, shared name table. I did this because scanning the src.zip file created thousands of name tables which contained lots of identical entries. "In my case it still consumes about 1MB (20.000 Names), but not that big problem anymore." So, I can remove the table sharing, but you'll get a performance hit -- you cannot have it both ways. It's your call, gentlemen.
Tom, As long as the table is usefull there (i.e. is reused during the next parsing 20s later), it should stay. If it is used only during one-shot parsing of several files and have to be discarded for next batch, it should go away after the batch. When I was discussing those original 16 tables, I was told they are not needed and shouldn't be there.
The shared name table is used for every parse request from MDR and every error check request from the Editor. That said, it should be soft-referenced so that the garbage collector can discard it if necessary in between parses. That way it doesn't have to stay 1 MB, since it only grew that big from the union of all sources during the initial scan. I'll take care of that today.
One more problem in src bridge it seems: I found ~15MB in Strings which represented ~700 whole source files. The traces were: org.netbeans.modules.javacore.jmiimpl.javamodel.JavaPackageClassImpl.mainProxy-> org.netbeans.jmi.javamodel.JavaPackageClass$Impl@cb6-> org.netbeans.mdr.storagemodel.MdrStorage@1990-> org.netbeans.mdr.util.EventNotifier@33e8-> org.netbeans.mdr.util.EventNotifier$Instance@62da-> org.netbeans.mdr.util.EventNotifier$1@1001f-> java.util.HashMap@22c8c-> [Ljava.util.HashMap$Entry;@3e5b0-> java.util.HashMap$Entry@60712-> org.netbeans.mdr.util.EventNotifier$RegisteredListenerSet@941a9-> java.util.HashMap@cec8c-> [Ljava.util.HashMap$Entry;@fad25-> java.util.HashMap$Entry@11a3a1-> org.netbeans.modules.java.bridge.SourceElementImpl$SourceListener@fd3b5-> org.netbeans.jmi.javamodel.Resource$Impl@23b10-> org.netbeans.modules.javacore.parser.ResourceInfo@3f6c6-> java.lang.String@62565 The source files held this way are probably the sources that matched the hash for HelpCtx.Provider and had to be parsed to resolve their subclass status for my Where Used query. Martin, to answer your question re. mainProxy: Sure, path through mainProxy is by chance just a shortest static referenced path to the objects in question. In fact, everything up to [Ljava.util.HashMap$Entry;@3e5b0 is common to all the traces, as the objects on the path before are singletons.
Dane, can you take a look at the src bridge?
I tested scenario as Petr Nejedly described on 2004-09-02. It seems it is not reproducible in current trunk build. After closing results, the memory meter was under 50MB. Dan Prusa tried it also and had similar results. Also I didn't find thousands of JMI objects in memory when results were closed. Lowering priority because I am not able to reproduce it.
I tried it also and get memory usage under 50MB. The count of instances is lower than Petr's except JavaPackage$Impl: 135 org.netbeans.jmi.javamodel.AnnotationTypeClass$Impl 39 org.netbeans.jmi.javamodel.ArrayClass$Impl 1 org.netbeans.jmi.javamodel.ArrayReferenceClass$Impl 135 org.netbeans.jmi.javamodel.CodebaseClass$Impl 43 org.netbeans.jmi.javamodel.ConstructorClass$Impl 53 org.netbeans.jmi.javamodel.Constructor$Impl 81 org.netbeans.jmi.javamodel.ContainsResources$Impl 2 org.netbeans.jmi.javamodel.ElementPartKindEnum 1 org.netbeans.jmi.javamodel.Extends$Impl 47 org.netbeans.jmi.javamodel.FieldClass$Impl 8 org.netbeans.jmi.javamodel.FieldGroupClass$Impl 1 org.netbeans.jmi.javamodel.FieldGroup$Impl 172 org.netbeans.jmi.javamodel.Field$Impl 42 org.netbeans.jmi.javamodel.Implements$Impl 17 org.netbeans.jmi.javamodel.ImportClass$Impl 189 org.netbeans.jmi.javamodel.Import$Impl 28 org.netbeans.jmi.javamodel.InitializerClass$Impl 3 org.netbeans.jmi.javamodel.Initializer$Impl 136 org.netbeans.jmi.javamodel.JavaClassClass$Impl 65 org.netbeans.jmi.javamodel.JavaClass$Impl 2 org.netbeans.jmi.javamodel.JavaDocClass$Impl 20 org.netbeans.jmi.javamodel.JavaDoc$Impl 135 org.netbeans.jmi.javamodel.JavaEnumClass$Impl 136 org.netbeans.jmi.javamodel.JavaModelPackage$Impl 136 org.netbeans.jmi.javamodel.JavaPackageClass$Impl 3878 org.netbeans.jmi.javamodel.JavaPackage$Impl 52 org.netbeans.jmi.javamodel.MethodClass$Impl 368 org.netbeans.jmi.javamodel.Method$Impl 2 org.netbeans.jmi.javamodel.MultipartIdClass$Impl 6 org.netbeans.jmi.javamodel.MultipartId$Impl 51 org.netbeans.jmi.javamodel.ParameterClass$Impl 65 org.netbeans.jmi.javamodel.Parameter$Impl 52 org.netbeans.jmi.javamodel.ParameterizedTypeClass$Impl 1 org.netbeans.jmi.javamodel.ParameterizedType$Impl 5 org.netbeans.jmi.javamodel.PrimitiveTypeClass$Impl 9 org.netbeans.jmi.javamodel.PrimitiveType$Impl 9 org.netbeans.jmi.javamodel.PrimitiveTypeKindEnum 135 org.netbeans.jmi.javamodel.ResourceClass$Impl 16 org.netbeans.jmi.javamodel.Resource$Impl 2 org.netbeans.jmi.javamodel.TagDefinitionClass$Impl 15 org.netbeans.jmi.javamodel.TagDefinition$Impl 2 org.netbeans.jmi.javamodel.TagValueClass$Impl 19 org.netbeans.jmi.javamodel.TagValue$Impl 1 org.netbeans.jmi.javamodel.Throws$Impl 39 org.netbeans.jmi.javamodel.TypeClass$Impl 1 org.netbeans.jmi.javamodel.UnresolvedClassClass$Impl 1 org.netbeans.jmi.javamodel.UnresolvedClass$Impl and I have only two instances of TopClassesCollection and TopClassesListener.
I can hapily confirm that on 040907, my scenario works OK. (Although I can reproduce the problem with 040902, it was my custom build, so maybe I was missing some fix). For comparison, my find usages now finishes in 3:54 (vs. 8:15) and needs 87MB* (vs. 110MB) of heap to finish. Memory after found usages is 51.2MB (vs. 96.9MB) and after closing results is 37.6MB (vs. 77.0MB), up by 10MB from before find usages, but that's probably acceptable as the IDE was cold started with all caches empty initially. *) needed memory measured as the memory usage after last few GC cycles during scan. It is bottom guess (modulo softrefs) of memory needed to perform the operation.
Initial name tables are now much smaller than javac's default size. This is possible because javacore only feeds gjast one source file at a time, rather than javac's normal (and more efficient) batch model.
Is this bug still valid? I opened core project, and the IDE consumes 23 MB. Can we close this issue?
I suggest that we discuss this issue on Wednesday's perf meeting.
After discussion with the performance team, this issue can be considered to be fixed.
Yes, we are done with this umbrella issue for 4.0. We'll file new issues for memory related problems for 4.1 as appropriate.
Reorganization of java component