This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
After you close an open project, there are lots of objects belonging to MDR and Java Core (e.g. org.netbeans.mdr.persistence.btreeimpl.*, org.netbeans.mdr.storagemodel.*, org.netbeans.modules.javacore.jmiimpl.javamodel.*,...) left on heap forever. IMHO all these objects should be cleaned at project closing. They are unnecessarily occupying memory if I want to open a different project which does not have anything in common with the closed project.
Created attachment 16438 [details] Objects left in memory after project core (and all projects it depends on) are closed
Created attachment 16439 [details] Object left in memory after jEdit project is closed; these objects belong just to jEdit because before marking current status of the heap I had core project open and closed
Created attachment 16440 [details] JProfiler snapshot
We are weak-referencing classpaths and remove the objects associated with a given classpath element from memory only when the weak reference to a given classpath element is collected. The fact that there are still objects corresponding to the classpath elements means that the classpaths for the closed projects are still being hard-referenced from somewhere. Some of them surely are (from our classpath cache, which is quite small). But there may possibly be (and it seems there are) other leaks not related to javacore. Do you consider the problem with the cache itself being severe? It is a LRU cache, so as soon as you open other project and start to work with its classpaths the old classpath should get collected. I can still try to change the cache to weak-reference the classpaths - that should work too.
Martine, could you please change the cache to weak-reference the classpaths as you suggest? Even if its impact on the leak is limited and disappears after you start using next open project, eliminating these hard references would help us see and eliminate the other leaks not related to javacore... THX.
I realized that changing the caches would not be that simple since the cache references our "filtered classpath" rather than the real classpaths directly. The real classpaths are referenced from the filtered one, which registers some listeners on them and also it is lazily initialized. It would be more complicated to reimplement it to be initialized eagerly and listen to the classpaths using weak listeners rather than regular ones. Anyway, I found one leak in the cache which caused that one of the classpath caches had an infinite size. That's now fix and should help.
*** Issue 46797 has been marked as a duplicate of this issue. ***
see issue 47589
*** Issue 47589 has been marked as a duplicate of this issue. ***
Pavel Flaska tried to measure this in JProfiler. It seems that all classpaths are held in memory (even after all projects are closed) by o.n.m.apisupport.project.ClassPathProviderImpl, which is held by org.openide.util.lookup.InstanceContent$SimpleItem. This can be reproduced by simply starting up the IDE, opening refactoring project (and all required) and closing all the projects. BTW, I have changed implementation of our caches so that they should no longer confuse profiler - all our caches now weak-reference classpaths. I will commit this change as soon as it will pass commit validation.
Not specific to apisupport projects; a similar situation exists for j2seproject's and freeform projects, and I presume web/project's. A project cannot simply discard all its ClassPath objects when it is closed, since it may need to continue to fire changes in them. It could hold them weakly or softly. (j2seproject, for example, holds them softly for whatever reason; NBM and freeform projects hold them strongly.) But it doesn't really matter since the project itself should be held only softly (by the project manager) after it is closed. In other words, if you change your mind and quickly reopen the project e.g. from File -> Recent Projects, the same ClassPath objects will still be in use. If a lot of standing memory is consumed for other reasons - e.g. other newly opened projects need some refactoring, etc. - the soft references to the projects should be cleared by the garbage collector, taking the classpaths with them. So I do not see any actual problem, unless someone knows differently. An increase in standing heap caused by soft refs should be harmless, so long as the added objects are in fact at most softly reachable.
Note: it would be straightforward (I think) to hold the ClassPath's from the ClassPathProvider impl using weak references, which would have the effect of making them disappear quickly after the project is closed. (When open, at least GlobalPathRegistry should keep strong refs to them.) I am not sure if this would improve actual memory performance, but it would probably not be hard to do if so. First I would like to hear from Tomas why he decided to use soft references in j2seproject.
> hold the ClassPath's from the ClassPathProvider impl using weak > references, which would have the effect of making them disappear > quickly after the project is closed > ... > first I would like to hear from Tomas why he decided to use soft > references in j2seproject Any reason why not modify this to behave more properly and allow the closed projects to be cleared from memory?
As I said, the closed projects *can* be collected (barring any unrelated memory leaks, of course); they are held softly. Still awaiting comment from Tomas re. intention of using soft refs to hold ClassPath objects in j2seproject.
> As I said, the closed projects *can* be collected (barring any > unrelated memory leaks, of course); they are held softly. Please, don't use soft references to "cache" things. If everyone used soft references we would be in the near-OOME state all the time...
I've used the SoftReference since there was no reason why to hold the ClassPath which is not used by anyone and can be freed and recreated later. This has changed when the GlobalPathRegistry was introduced and all these paths are hold by it. Now there is no reason why to use (Soft/Weak)References and hard references should work as well.
This is not classpath issue, the issue must be fixed by the code which holds the classpath. ClassPath API can not solve this problem at all. Reassigning back to apisupport/project. But I still think that the MDR storage clean up should not be based on WeakListeners.
What do you think the MDR cleanup *should* be based on?
"Please, don't use soft references to "cache" things." - what else are soft references for?? IMHO if the VM does not clear soft refs soon enough, that is the VM's problem, not ours. (Or perhaps it needs to tune itself better.) "If everyone used soft references we would be in the near-OOME state all the time..." - so what? Another name for that is "full memory utilization". The only user problems are (1) using too many active memory pages and so swapping, (2) getting an OOME which breaks functionality, (3) using too little heap and needing constant GCs to allocate anything.
OK, I will remove the SoftReferences, but I'm afraid it will not help, it seems that projects are hold in memory after they are closed.
> Another name for that is "full memory utilization". :-) I guess the rest of performance team would agree with me that we don't want full memory utilization. Rather the opposite. Too much time spent in GCs and running into problems in the moment you suddenly need a few megs of heap for something is what we want to avoid.
Would prefer to hold ClassPath objects strongly from projects in all cases. It is simply part of the configuration of the project, and in some cases may involve some nontrivial computation to create (e.g. loading properties files). I agree with Martin that the merged classpath cache used for JMI needs to be keyed off of ClassPath objects. Projects are currently held softly when closed. Using weak refs here (or no caching at all) would not be a good idea, I think, as you can certainly access a project when it is closed (for various queries, from the Open Project dialog, etc. etc.). In some cases this can happen repeatedly and rapidly; recreating the project too often would be a big performance hit. Could perhaps use some kind of soft reference with timeout, as is used with apparent success by NbBundle. (Unfortunately there is no API for this currently so I would have to copy the code.) Ultimately I would prefer to remove such hacks and have the VM better manage its own heap, clearing soft references more aggressively when memory is tight (but not yet exhausted). Trying to bypass the VM's policies at the application level is not very wise, I think. In fact I think the server VM already behaves quite differently in this respect from the client VM.
Well I will change two things: - projects will hold their ClassPath's strongly, or not at all, after closing (acc. to project type; arbitrary) - the project manager will hold timed weak refs to projects, so that projects are held strongly while open, or for 15 seconds after closing, and thereafter weakly, except strongly for 15 seconds after last access
Checking in j2seproject/src/org/netbeans/modules/java/j2seproject/classpath/ClassPathProviderImpl.java; /cvs/java/j2seproject/src/org/netbeans/modules/java/j2seproject/classpath/ClassPathProviderImpl.java,v <-- ClassPathProviderImpl.java new revision: 1.11; previous revision: 1.10 done
Hmm, I actually had a patch ready for j2seproject, but you beat me to it... committed * Up-To-Date 1.9 projects/projectapi/src/org/netbeans/api/project/ProjectManager.java added * Up-To-Date 1.1 projects/projectapi/src/org/netbeans/modules/projectapi/TimedWeakReference.java committed * Up-To-Date 1.6 projects/projectapi/test/unit/src/org/netbeans/api/project/ProjectManagerTest.java
*** Issue 48194 has been marked as a duplicate of this issue. ***