This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
Created attachment 129482 [details] Snapshot of XML editor memory usage after opening 15 MB XML file. The XML editor's memory usage uses too much memory. Upon opening a large file (15 MB) in size causes an explosion in memory consumption. I explored this by taking a baseline snapshot of the heap ("baseline"), and comparing it to a snapshot after the file was opened ("after"). I attached the results. Here is a list of the largest offenders: char[] 150,913,440 B (47.2%) org.netbeans.modules.editor.url.HighlightURLs$3$1 41,700,384 B (13%) java.lang.String 21,608,496 B (6.8%) org.netbeans.editor.ext.ExtSyntaxSupport$TokenItemTP$Item 19,948,720 B (6.2%)
Created attachment 129483 [details] Baseline memory usage prior prior to opening an XML file in the editor.
Are you sure that you invoked GC prior to making the snapshots ? Especially the class org.netbeans.modules.editor.url.HighlightURLs$3$1 corresponds to a very short-lived anonymous class, whose instances are created rapidly, but live only for such a short time, that they should never escape the Eden space. This does not mean the Editor does not consume unreasonably large memory, only the real cause may be hidden by those temporaries. Could you please attach a heap dump, so I inspect see the reference chains ? Thanks.
I reprofiled the app, capturing the same set of data under the same conditions. Identical results were observed. Next, a dump of the heap was performed and the garbage collector was executed via the "Run Garbage Collection in Profile Application" action. No appreciable change was observed in the "Live Results" window. The attached Zip contains a snapshot of the baseline memory utilization, a snapshot after the 15 meg file was opened, and the file resulting from the heap dump.
The resulting file is HUGE. I instead took a screenshot of the Heap results. Let me know what other information I cna provide.
Created attachment 129557 [details] Screen capture of heap dump.
Created attachment 129558 [details] Image depicting stack trace of TokenItem
Created attachment 129559 [details] Image depicting stack trace of HighlightURLs
OK. First, try to GC before creating the heapdump. The HighlightURL$3$1 is created for each char in the document (ouch!), but since it should be trashed almost immediately, it should not eat your memory. Look on an instance of the HighlightURLs$3$1, pick one of them and explore the path to the GC root; it would be interesting to see what object references the throwaway instance. Still it would be great if you could use http://deadlock.netbeans.org/hudson/job/upload/build to post the compressed heap dump.
Created attachment 130553 [details] GC Root Screenshot
Aplogize for the late response. I was on leave and Mike Cress who opened this request is no longer working on this project, hence I am following up now. Tried to locate GC root for an instance of the HighlightURLs, but got meesage that "No GC root found". Please find attached screenshot for the same. Also uploaded the heap dump using the link below against this bug. http://deadlock.netbeans.org/hudson/job/upload/build Appreciate your help if you can work on this again and do the needful ASAP. Please let me know if you need any more details.
Can someone look into this ASAP? We have provided the requested info.
I cannot see any problem in the heapdump (223953_2013-01-23_20-48-30_7d3f1977f5e7f447fedcb6769206c9e8.zip); 35megs of Strings allocated, no instances of HighlightURLs* or TokenItemTP* at all. Suprisingly, I cannot see any HighlightURLs$FactoryImpl instance, which strange, given your previous screenshots. Are you sure the file was shot at the time the IDE has a memory problem ? The dump shows rather small number of objects to make up a memory perf issue. Just to be sure, I've generated ~19meg XML file with URLs in element attributes. I can see a huge number of lexer tokens in my dump - which is unfortunate, and should be fixed (I already plan to migrate from old lexer infrastructure that causes this), but the report is rather different from what was posted here as screenshots during 12/2012
I am not sure why the report looks different, any ways I hope you have enough info to work on this and assume that you are going to work on fixing the performance enhancement issue. Please let me know if you need any further info. Can you please let me know, how long it might take to have the fix available for this performance enhancement?
(In reply to comment #13) > I am not sure why the report looks different, any ways I hope you have enough > info to work on this and assume that you are going to work on fixing the > performance enhancement issue. Please let me know if you need any further info. > OK, to be more explicit, the report attached was completely useless. It didn't match the issues exhibited on the previous screenshots, and did in fact show just usual footprint and object counts (the dump attached on 23rd Jan showed approx 35k Strings, which is quite normal, but the original snapshot created on 17th Dec reported about 16M String instances). It did not contain but one instance of the classes previously reported to have high counts and large allocations, so I could not obviously trace the reference chains. Please - start up NetBeans, attach VisualVM to it, open the huge xml file or otherwise try to get NB to the state when it eats a lot of memory for no reason. Then use Visual VM (Monitor tab) to first perform GC. If the used memory in the VisualVM monitor decreases after GC, then it's OK. I don't know exactly how did you detect that the IDE takes "too much" of memory which made you/your colleague to inspect the details with the profiler or self-sampler, so I cannot give explicit instructions how to reach the state. You can however monitor - e.g. using Sampler in VisualVM number of object instances; see the original screenshots, if you can make the number of HighlightURL$* classes grow, and they are not freed after GC cycle, then it's the right time to take heapdump.
When I run profiler and try to run the app and open the huge file to simulate the memory issue, First I get a warning that file is too large, I say ok then I get a message saying that Profiler has crashed and heap dump is created, and heap dump opens up. That is the one I had uploaded. Since the profiler crashes that may be the reason why those objects are not showing up.. With the screenshots provided earler, can you further investigate the issue as we are unable to get similar dump now? The profiles does not even run and crashes when we try to open up huge file in the editor. If you would like us to try any other approach, please let us know.
Please realize, that the screenshot is just an image. So how could I get a chain of references out of the screenshot, if it only contains object counts ? From code inspection, it does not seem that the reported objects could be held, the code uses the instance just to execute a code under lock, then throws the reference away - and it should be GCed. According to the process, I will have to close the issue as INCOMPLETE if no relevant information can be provided by the reporter (or by me, during testing on huge file). Just to be sure: you don't use internal sampler in NetBeans, do you ? If so, please use visualvm available in the JDK distribution. If the file does not contain proprietary data, could you upload it somewhere ? Maybe the defect can be reproduced on some specific file strucutre (?). Please also attach IDE log, so the environment with +- the same set of modules can be created.
I was using internal sampler. Now I have used visualvm to take the head dumps. I have taken head dump of netbeans IDE - before opening huge file in IDE and after opening IDE and after performing GC. I do see the heap increasing and even after GC it is not reducing. Please refer to the following files uploaded : heapdump-IDE-before opening the huge file.hprof heapdump-IDE-after opening the huge file.hprof heapdump-IDE-after opening the huge file - after GC.hprof IDE - Memory Usage - screenshot.docx Hope these help. Please let me know if you need more info.
To answer to your question about uploading the file, yes this file contains proprietary data, hence I can not upload. Hope the info I provided helps. Pls let me know if you need any more info. Appreciate your help with this.
I downloaded 9 zipped heapdumps from deadlock.netbeans.org + 2 docx files (1 screenshot of memory usage, 1 of OOME message from your application). Sadly the filenames you wrote were not preserved for some reason. 3 of those heapdumps are ~6MB, 3 are ~20-25MB and 2 of them are ~90MByte large. One zip contains only 23kByte dump (?) Most allocated objects: 29k Fold objects = 1MB 263k LineElement = 6MB 263k ParagraphView = 12M 381k EditorPosition = 4M 381k Mark = 12M 263k XMLDefaultSyntax$XMLStateInfo = 5M 1,9M of DefaultToken = 46M 1,9M of Integer = 23M 2M of char[] = 108M 2M of String = 49M The number of DefaultTokens will probably remain unchanged in NB-7.2, although the number of Strings and char[] (and their held memory) should decrease significantly (by # of DefaultToken instances) as DefaultToken was changed not to cache content in Oct 2012. The number of EditorPositions should be O(# line elements + folds), which seems to be OK, and will hopefully decrease in 8.0 (there's an enhancement filed to remove 2 positions per fold). The number of Integers could be probably optimized somehow. See defect #225628. Milo, is there any cure for holding all the DefaultTokens ? Currently they are held by TokenHierarchy referenced from SyntaxHighlighting in editor.lib2.
radhikaboddu: for immediate help, you may try to increase the amount of memory available for the app. Sure it is not a fix, but can help you to work with a small number of documents of such size. look for {netbeans-install-dir}/etc/netbeans.conf, locate the line netbeans_default_options= and insert -J-Xmx800m among the options. Or perhaps increase the 800m limit (800 megabytes) even to a larger number, if possible for your hw. See java documentation for -Xmx parameter (the -J additional prefix is just for NB launcher).
(In reply to comment #19) > The number of DefaultTokens will probably remain unchanged in NB-7.2, although > the number of Strings and char[] (and their held memory) should decrease > significantly (by # of DefaultToken instances) as DefaultToken was changed not > to cache content in Oct 2012. > > The number of EditorPositions should be O(# line elements + folds), which seems > to be OK, and will hopefully decrease in 8.0 (there's an enhancement filed to > remove 2 positions per fold). > > The number of Integers could be probably optimized somehow. See defect #225628. > > Milo, is there any cure for holding all the DefaultTokens ? Currently they are > held by TokenHierarchy referenced from SyntaxHighlighting in editor.lib2. Currently we don't drop tokens from token hierarchy although I've considered that for documents not used for a long time. I've also considered to hold lexer state objects not after each token but just e.g. after each 5th token to decrease usage for states (so the restart would be from the last preceding token with the state) but there was not yet a big demand for that. Not sure whether xml uses flyweight tokens but they could help somewhat although not so much like e.g. in html where the tag and attr names are predefined.
I am glad that you were able to get the dumps and review them. Regarding JVM options to increase memory settings, we have already tried with even 1024M, it is really not helping. We distribute our netbeans app as desktop client app. Unfortunately some of the end user machines are windows 32 bit and they have hw limitation, 512MB is the max that can be allocated. Hence this is not a feasible option for us. So at this point, after your observation, what are your thoughts? Are you going to work on enhancing platform editor performance? If so any tentative timeline? Please let me know if you need more info from me. Appreciate your help with this.
(In reply to comment #22) > So at this point, after your observation, what are your thoughts? Are you going > to work on enhancing platform editor performance? If so any tentative timeline? > Well, one of the issues is already solved; I tried to open a large file with NB 7.2 and 7.3RC1, and the profiler indicated that the char[] and String instances decreased significantly. It may or may not help you, depending on the exact structure of your document. My testing document seems to have a different characteristics. Apart of Strings/char[], which should be fixed in 7.3 codebase, the major consumer seems to be DefaultTokens and lexer states - see comment #21; demand for improvements were not so high, so I guess no time is planned to this area. Fixing the issue #225628 may improve things, say by 10-15% (considering object counts/sizes in your dump). That's from the technical standpoint. I believe I am not entitled to give any schedule promises; except for #225628 (which is a bugfix), improvements in related area seem as a nontrivial (= resource consuming) work which must be planned with respect to the next release primary goals.
(In reply to comment #23) > Apart of Strings/char[], which should be fixed in 7.3 codebase, the major > consumer seems to be DefaultTokens and lexer states - see comment #21; demand > for improvements were not so high, so I guess no time is planned to this area. > Fixing the issue #225628 may improve things, say by 10-15% (considering object > counts/sizes in your dump). > That's from the technical standpoint. > I believe I am not entitled to give any schedule promises; except for #225628 > (which is a bugfix), improvements in related area seem as a nontrivial (= > resource consuming) work which must be planned with respect to the next release primary goals. Thanks for your analysis & detailed response. I totally understand that the demand for these improvement requests are not high, but for us this has become a show stopper as we were unable to deliver the product for this reason. We were hoping that the platform enhancemnent request will be taken care. Can you do anything like talk to your other teams who is responsible for prioritizing the requests to help us move this request forward and do the needful? Appreciate your help.
Per Geertjan's advise, I have tried opening the huge file with Netbeans 7.3RC1, no luck. I still see the Integer & DefaultTokens being major consumer. I have uploaded the head dumps taken before opening the huge file, after, and after performing GC against this ticket. Please take a look at them and let me know.