This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
Memory consumption in Netbeans causes long Netbeans and system locking pauses. The maximum limit of the heap creeps up infinitely causing large Netbean locking pauses, and the same set of virtual pages is not reused causing a cyclic flushing of all of the pages of other concurrent applications eventually resulting in virtual paging thrash. I have seen this problem follow the same pattern, but with varying degrees of impact on Windows, Linux, and Mac OSX systems. The current state of Netbeans 4.0rc2 memory consumption follows these two crippling scenarios: 1 Heap Creep 1 Netbeans opens with an initial heap limit. 2 As Netbeans is used, incremental GCs recover some, but never as much memory as a full GC, even when it sits idle. 3 Heap consumption gradually builds despite incremental GCs until the heap limit is reached. 4 A full GC is done that recovers memory the incremental GCs could not, but the heap limit is now increased. 5 Steps 2 - 4 repeat and the heap limit continues to grow, even when no operation requires that much heap. 6 As the heap limit grows, the time Netbeans spends locked up in a full GCs grows longer and longer until you're sitting there for over 10 minutes. 7 There is no way to decrease or reset the heap limit without restarting Netbeans. It will grow until Netbeans or your system crashes. 2 Virtual Page Thrash 1 Netbeans opens with an initial set of allocated virtual memory pages from the heap, taken from virtual pages possibly used by other concurrently running applications. 2 As objects are created and released, memory pages are cluttered with allocated but unused space which cannot be deallocated until the GC gets around to finalizing those objects. 3 As more objects are allocated, more virtual pages are taken from other applications. 4 Since incremental GC does not reclaim all that a full GC does, virtual pages continue to be allocated away from other applications. 5 When pages are finally cleared by the GC, they are not reused. New pages are taken for new objects. This continually cycles through the available virtual pages and constantly flushes the data of other applications. 6 Whenever focus is changed to another concurrent application, or a background task of another concurrent application is started, it must first reallocate the virtual pages it lost to the Netbeans creeping heap. 7 For Netbeans to continue it must, again, reclaim those virtual pages lost to other concurrent applications. As the heap creeps up in size, the cyclic flushing of virtual pages occurs more often, creating more lag time spent in virtual page reallocation thrash between Netbeans and the other concurrent applications. This can lock up your entire system for over 30 minutes during a full GC. The primary reason for Heap Creep is the fact that incremental GC does not eventually perform the same job as a single full GC. When Netbeans is left idle, incremental GCs should eventually bring the head down to what a single full GC does. In this way, as long as you don't actually need more memory than your current heap limit, you will not exceed that limit due to sluggish or incomplete GCs. The primary reason for Virtual Page Thrash is that Netbeans does not reuse the same set of virtual pages as do other well-behaved applications, resulting in the constant flushing of virtual page data of the other concurrently running applications. After the Netbeans heap creeps up enough, most of your processing time is spent reallocating and refreshing virtual pages between Netbeans and other applications. The same set of pages in a heap should be constantly reused as much as possible before grabbing any more from other concurrent applications. Together, these problems can make a simple 30 minute coding job in Netbeans, with concurrent File Explorer, Browser with Java documentation windows, and a Mail application, take several hours requiring several Netbeans restarts. This is an extremely serious problem that has existed in list discussion and bug report (#40231) since before 3.6. It was given a P1 priority since it does create temporary but growing application and system lockup. Personally, I have always considered these memory problems to the #2 priority problem with Netbeans (#1 being documentation).
First, it would be good if you could provide parameters of the HW configurations you tested on. Also, please provide any command line and/or netbeans.conf switches you use when running NetBeans (-Xmx, -Xms, etc.) The approximate sizes of your projects (number of classes, number of jars, number of dependent projects) would also be helpful. Secondly, what you describe in 1-4 of the "Heap Creep" paragraph is basically a description of the standard behavior of the JVM with the default GC strategy. Simply put, if the VM determines the application needs more memory, it keeps resizing the heap upto the -Xmx limit. The full GC use the "stop the world" strategy and once allocated heap is never reclaimed. The same applies to the "Virtual Page Thrash" section. NetBeans does not deal with virtual pages. Instead, as a pure Java app, it fully depends on the JVM to handle all the memory management. The only things NetBeans can do is to be conservative in memory requirements and not to generate to much garbage. Reuse and reclaiming of virtual memory pages is out of NetBeans' control. 10 to 30 minutes long GC cycles indicate that there must be something seriously wrong with your system setup. How much physical RAM do you have on your system(s)? How big maximum heap size do you set for NetBeans? I've seen GC cycles taking up to 30 seconds after a heavy use of NetBeans on a minimum HW configuration. I admit even that can be very annoying, but I've never seen anything close to 10 or 30 minutes. Please provide more details. Thanks.
I'd like to add my 2 cents here. I've been dealing with this exact problem ever since my project grew to about 250 classes (it's at 380 now). I was part of a group that brought this issue up during 36cat, a bug was filed, and we all added our info to it, but there was never any resoultion. Here are my comments from Bug 40231: <paste> I'm adding my comments from the NetCAT forum: This isn't an issue related solely to 3.6 but it does continue the behavior I've seen in previous versions. When I start the IDE and open my project the IDE takes about 83MB of memory and 110MB of Virtual Memory, as viewed in the windows task manager. By the end of the day it is at 180MB mem and 210MB VM, the only way to free the memory is to quit the IDE, changing projects doesn't help, manual garbage collection will reduce the usage a little bit, but only for a little while. I have been able to slow the process by adding -Xmaxf .15 -Xminf .10 to ide.cfg but it always grows....if I leave the it running long enough (only happened once or twice) I will start getting out of memory exceptions. I've reported these problems over the years and I usually get a "can't reproduce", or "it's a JVM issue" response. Anyways I was hoping in this forum my concerns might fall on more receptive ears. </paste> The last comment said: "The full GC use the "stop the world" strategy and once allocated heap is never reclaimed." And I don't think that is true, if you tweak the GC with options like -Xmaxf .15 -Xminf .10 java WILL return the memory. I've been toying with the idea of video taping my display to show how long it can take to restore the app (4-5 minutes) when it's memory use is really high, or when it's reached the -Xmx limit, and starts doing what feels like a full GC every time I type a character. One other serious problem is the parser. When memory is tight, and I have syntax errors in the code, just moving the cusor near a line with errors causes a pause of 2-3 seconds, which is agonizing. My project has 380 classes, and it doesn't depend on any other projects. I've got a couple jar's on the class path, nothing too big, one is 3k and the other is 384k. When I was working in 3.6 I had the JDK's src.jar mounted (and rt.jar compiled with symbols), because I needed to step into the JRE. 4.0 seems to work differently, since you have to click the checkbox in the Sources window, I don't know if that counts as a JAR the project is "using". BTW I'm using a rt.jar with symbols in 4.0 as well. I'm currently using -Xms 128M -Xmx 384M. My machine is a 2.2GHz P4 with 768MB of RAM.
Re "heap never reclaimed. - And I don't think that is true, if you tweak the GC with options like -Xmaxf .15 -Xminf .10 java WILL return the memory.": Note that I was referring to the default GC policy. No parameter tweaking. 380 classes is by no means a big project. I wouldn't even call it mid-size. Developers use NetBeans to develop NetBeans which is thousands and thousands of classes and they don't seem to run into such serious problems. I would question whether you really need -Xms128m. It basically tells the VM "don't even expect NetBeans to need less than 128 MB of heap." In the default NB configuration, 128 MB is the upper heap limit whereas the lower limit is set to -Xms32m. This should be completely adequate for a project consisting of 380 classes and 2 jars, unless you do something special (i.e. heavily use some memory intensive operations (refactoring, vcs), have some unstable experimental modules installed). Re "I've been toying with the idea of video taping my display to show how long it can take to restore the app (4-5 minutes)": You might want to look at issue #40177 and try JDK 1.5.0 where this problem is supposedly addresed.
I do use CVS integration. I refactor infrequently. I changed the ms and mx settings specifically because the IDE would bog down very quickly with the default settings. I tend to leave the IDE running for 3 or 4 days at a time, could it be just that I'm not using it the way others do? I am running against the 1.5 JDK, although I'm building against a 1.4 JDK.
Ok, thanks for the info. I still am not convinced you really need -Xms128m. -Xmx setting might be necessary, depending on the size of your project and usage scenarios.
Petr, you wanted to respond to this report. My comments: re 1.1-1.5 - this is esentialy default bahaviour of Java garbage collector. You can change it in etc/netbeans.conf file using parameter described on java.sun.com site. 1.6 - If you have a reproducible test case showing that full GC run takes minutes tell us how we can reproduce it and can address this 1.7 - first sentence is usualy right though JVM can decrease size of heap during runtime. yet we are not going to change this. 2nd sentence is wrong. It can only lead to OutOfMemoryError although this has similar impact (dangerous inconsistency of application data) re 2.x there are many mistakes
BTW: GC of Java heap should be improved with ConcMarkSweepGC used in 4.1 builds.
We are not going to fix this issue. Anyway, it is too general. We can do something about specific reproducible and concrete issues, not fix an issue like this. It is not necessary to keep this issue open. We know we have to focus on memory issues.
closing