Bug 221386 - memory leak (java parsing)
memory leak (java parsing)
Status: RESOLVED FIXED
Product: java
Classification: Unclassified
Component: Source
7.1.2
PC Linux
: P2 (vote)
: 7.3
Assigned To: Jan Lahoda
issues@java
http://netbeans.org/projects/profiler...
: PERFORMANCE
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2012-11-02 19:25 UTC by kvaso
Modified: 2013-01-03 09:51 UTC (History)
3 users (show)

See Also:
Issue Type: DEFECT
:


Attachments
jmap output - almost after NB start (654.31 KB, text/plain)
2012-11-05 17:42 UTC, kvaso
Details
jmap - first GC world stop (704.93 KB, text/plain)
2012-11-05 17:42 UTC, kvaso
Details
jmap - not enough memory (712.82 KB, text/plain)
2012-11-05 17:43 UTC, kvaso
Details
messages.log after out of memory (199.24 KB, text/x-log)
2012-11-05 17:46 UTC, kvaso
Details
netb7.3 - messages.log (138.43 KB, text/x-log)
2012-11-06 16:52 UTC, kvaso
Details

Note You need to log in before you can comment on or make changes to this bug.
Description kvaso 2012-11-02 19:25:40 UTC
This happens to me everytime - after about 2-3 hours of using NB (sometimes it is enough, that netbeans is opened, but I am not using it), JVM memory usage reaches maximum

I've tried to increase heap to 3GB and use concurrent mark&sweep GC, but that obviously did not solve the problem

It wouldn't be so strange, that some memory leak occurs, when I am using NB, but as I wrote before - also, when NB is running in background, memory slowly increases to maximum.

I remember this happend to me also with NB 7.1.

I know, that this bug report will end up as "incomplete" or "wont fix", but I would really like to help you guys to fix this issue - so let me know, if you want some more info about this, because I don't know what data you need (I thought about heap snapshot, but to profile NB for a half hour and send you collected data... well, the file would be pretty big :)

I am running linux, JDK 1.6.0_37 and I am developing NB platform application (but I guess that does not matter).
Comment 1 Antonin Nebuzelsky 2012-11-05 14:15:12 UTC
You are right. Some additional information will be needed.

Please, provide at least your full log file:
http://wiki.netbeans.org/FaqLogMessagesFile

and a memory histogram (see the bottom part of the following page) for a start:
http://wiki.netbeans.org/FaqMemoryDump

Better to do that with the standard heap size setting where you will run into OOME sooner and there will be less data to analyze.

After attaching those here, reopen. Thanks!
Comment 2 kvaso 2012-11-05 17:42:14 UTC
Created attachment 127170 [details]
jmap output - almost after NB start
Comment 3 kvaso 2012-11-05 17:42:53 UTC
Created attachment 127171 [details]
jmap - first GC world stop
Comment 4 kvaso 2012-11-05 17:43:51 UTC
Created attachment 127172 [details]
jmap - not enough memory
Comment 5 kvaso 2012-11-05 17:46:34 UTC
Created attachment 127173 [details]
messages.log after out of memory
Comment 6 kvaso 2012-11-05 17:54:39 UTC
I've added some logs as you asked:
3 jmap outputs: one after NB start, then when first "world stop" occured and the last output file is when NB showed a popup that there is not enough memory to compile

there is also messages.log file after NB showed "not enough memory" popup

btw. I breafly looked at messages.log, so here is some explanation: there are some ClassNotFound exceptions about org.japura.gui.CheckComboBox or org.jdesktop.swingx.<anything> - those are third party GUI components that I added to NB palette (they are working just fine, so I do not understand those exceptions...)

also sometimes there are some lines containing this string: "/proc/25797/" -> NB process was running with PID 25797



hope it helps
Comment 7 Antonin Nebuzelsky 2012-11-05 22:02:25 UTC
Thanks.

Reassigning to java parsing for evaluation. The heap histograms are showing it is java.source.parsing and javac.code objects filling the heap.

/proc/25797/fd/x numbers are file descriptors used for the particular files. No clue why they are logged.
Comment 8 Petr Cyhelsky 2012-11-06 07:46:23 UTC
there are 82 instances of com.sun.tools.javac.code.Symtab in your jmap output, some bugs concerning this were fixed recently, can you please try using latest dev build, it should be better. Also please attach zipped heap dump, so that potential leaks can be investigated. Use  http://deadlock.netbeans.org/hudson/job/upload/build to upload it.
Comment 9 kvaso 2012-11-06 16:52:44 UTC
Created attachment 127240 [details]
netb7.3 - messages.log
Comment 10 kvaso 2012-11-06 17:04:11 UTC
I downloaded newest NB 7.3 dev (Build 201211060001)
the memory problem still persists

I created memory dump, but I wasn't able to upload there anything - I got HTTP error 400 (when I cilcked 'Build' button or when I pressed 'Build Now' link, the page just refreshed - I don't think that my internet connection is that fast, it can upload 1MB file in less then sec...)

I uploaded it to my ftp server (well, it is my university's ftp, so I am not fully in control of it, but the file will remain there for at least half a year :)
http://student.fiit.stuba.sk/~kvasnick08/netb-profile.zip

sorry about the inconvenience

I also uploaded new messages.log file (here on bugzilla)
Comment 11 kvaso 2012-11-06 17:11:55 UTC
I forgot to add: just before heap dump ends, I recompiled the whole project and the heap rises rapidly

also I've got heap dump in nss format - when I wanted to open it in visualvm, I got Out of memory error (yes, visualvm threw this exception :) 
you can download it here
http://student.fiit.stuba.sk/~kvasnick08/selfsampler.zip
Comment 12 Tomas Hurka 2012-11-06 17:26:09 UTC
(In reply to comment #10)
> I created memory dump, but I wasn't able to upload there anything - I got HTTP
> error 400 (when I cilcked 'Build' button or when I pressed 'Build Now' link,
> the page just refreshed - I don't think that my internet connection is that
> fast, it can upload 1MB file in less then sec...)
> 
> I uploaded it to my ftp server (well, it is my university's ftp, so I am not
> fully in control of it, but the file will remain there for at least half a year
> :)
> http://student.fiit.stuba.sk/~kvasnick08/netb-profile.zip
I am sorry, but we need heap dump - not a self-samler file.
 
> I also uploaded new messages.log file (here on bugzilla)
Attached messages.log file does not show any memory problem.
Comment 13 kvaso 2012-11-06 18:00:38 UTC
(In reply to comment #12)
> I am sorry, but we need heap dump - not a self-samler file.

ok, so I've created heap dump - its 99MB (already zipped) and http://deadlock.netbeans.org/hudson/job/upload/build does not work for me
any other way, how to send you this? :)
Comment 14 Tomas Hurka 2012-11-07 07:34:48 UTC
(In reply to comment #13)
> (In reply to comment #12)
> > I am sorry, but we need heap dump - not a self-sampler file.
> 
> ok, so I've created heap dump - its 99MB (already zipped) and
> http://deadlock.netbeans.org/hudson/job/upload/build does not work for me
What does it mean that it does not work? Can you be more specific? 
 
>any other way, how to send you this? :)
I think you can use <http://student.fiit.stuba.sk/~kvasnick08>.
Comment 15 kvaso 2012-11-07 15:25:45 UTC
so, I've managed to upload heap dump to hudson (using link you provided)

(In reply to comment #14)
> What does it mean that it does not work? Can you be more specific? 
I was getting error 400 with message something like: form needs parameters to submit - I've tried it several times with this result - today I tried it again and it worked...

> I think you can use <http://student.fiit.stuba.sk/~kvasnick08>.
I've got only 3MB quota there and the file to upload is 99MB
Comment 16 Tomas Hurka 2012-11-07 17:08:46 UTC
Heap dump is available here: <http://netbeans.org/projects/profiler/downloads/download/Heapdumps/heapdump-221386.zip>
Comment 17 Jan Lahoda 2012-11-08 20:51:40 UTC
Thanks for the heap dump. There are 30 instances of javac in the heap dump:
-about 25 held through the Lombok's annotation processor. This appears to be Lombok bug #242:
http://code.google.com/p/projectlombok/issues/detail?id=242
I would suggest upgrading Lombok to a version that has this bug fixed, and most of the problems should go away.
-2 (or so) are held through LayerGeneratingProcessor's originatingElementsByProcessor map. This map is weak, but that is not pointless in this case, as the values (Element) do transitively refer to the key (Filer). This map is normally cleared in the last round of the annotation processing. The only way that comes to my mind that could cause retaining unneeded mapping is that the processing would be cancelled (while opened in the editor, presumably), which would prevent the last AP round. Not quite sure how to solve that (without slowing down editing significantly) - would help to make the in the map Elements weak only, but I cannot say offhand what other effects this could have.
-1 held through a strange chain including org.netbeans.modules.masterfs.filebasedfs.naming.NameRef.next(the one from Reference). No idea whatsoever what that means. Will probably ignore that unless seen in more dumps.
-1 held as a cache for the currently opened editor.
Comment 18 kvaso 2012-11-10 10:07:39 UTC
I've upgraded to newest Lombok and it seems that problem is gone

thanks a lot for your effort
Comment 19 kvaso 2012-11-10 10:42:16 UTC
correction: the problem is not solved completely - heap size is still rising, but not so rapidly
Comment 20 Jan Lahoda 2012-11-14 13:04:30 UTC
(In reply to comment #19)
> correction: the problem is not solved completely - heap size is still rising,
> but not so rapidly

It would be awesome if you could upload a heap dump after running for some time (not necessarily after some actual problems show up). Would help us/me determine if that is something known or something new.

Thanks.
Comment 21 kvaso 2012-11-15 20:19:54 UTC
I've created and uploaded new heap dump

I was running NB for almost 2 hours - most of the time it was just opened and I did not use it

but even when I did not work with NB, I noticed, that (max) heap size increased - it might be some unignificant random event, but I don't think that (any) application should consume more and more RAM even when user is not using it...

tomorrow I will have got more time to run NB for a longer time, so I will be able to give you better heap dump if you want
Comment 22 Jan Lahoda 2012-11-15 21:12:28 UTC
(In reply to comment #21)
> I've created and uploaded new heap dump

Thanks for the dump. I took a look, and did not spot anything particularly wrong - one javac, one NbEditorDocument, etc.

> 
> I was running NB for almost 2 hours - most of the time it was just opened and I
> did not use it
> 
> but even when I did not work with NB, I noticed, that (max) heap size increased
> - it might be some unignificant random event, but I don't think that (any)
> application should consume more and more RAM even when user is not using it...

The sad truth is that even when not working with the IDE, there are some allocations running (during repaints for blinking with the cursor, for example, and there are surely others). And the JVM may decide to increase (or shrink for completeness) the allocated heap size during GC. But I cannot say if that is what was happening.

> 
> tomorrow I will have got more time to run NB for a longer time, so I will be
> able to give you better heap dump if you want
Comment 23 kvaso 2012-11-16 17:40:49 UTC
today I gave it a try - NB had a hard time, but heap size is about 250 MB (with some peaks at 400, but after a while it is back to normal)

so I guess it is OK, now

thanks for your effort and sorry to bother you with bug, that wasn't in Netbeans

I think you can close the bug now
Comment 24 Jan Lahoda 2013-01-03 09:51:15 UTC
(In reply to comment #23)
> today I gave it a try - NB had a hard time, but heap size is about 250 MB (with
> some peaks at 400, but after a while it is back to normal)
> 
> so I guess it is OK, now
> 
> thanks for your effort and sorry to bother you with bug, that wasn't in
> Netbeans

No problem.

> 
> I think you can close the bug now

I was thinking about possibilities to handle the LayerGenerating part better, but I don't have a reasonable solution, so we will probably need to rely on the standard SoftReference that holds the ClassLoader that loads the annotation processors.

Thanks for the report.


By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2014, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo