Bug 204645 - NetBeans runs out of memory selecting files in project.
NetBeans runs out of memory selecting files in project.
Status: RESOLVED DUPLICATE of bug 235748
Product: platform
Classification: Unclassified
Component: Lookup
7.1
PC Linux
: P3 (vote)
: TBD
Assigned To: Jaroslav Tulach
issues@platform
: PERFORMANCE
: 204476 221412 (view as bug list)
Depends on: 212619
Blocks:
  Show dependency treegraph
 
Reported: 2011-11-03 18:09 UTC by paolosca
Modified: 2014-04-04 08:50 UTC (History)
18 users (show)

See Also:
Issue Type: DEFECT
:


Attachments
Thread dump taken while the ide was unresponsive. (21.34 KB, text/plain)
2011-11-03 18:09 UTC, paolosca
Details
Netbeans messages log (38.88 KB, application/octet-stream)
2011-11-03 18:10 UTC, paolosca
Details
netbeans metrics file (62.02 KB, application/octet-stream)
2011-11-03 18:10 UTC, paolosca
Details
netbeans uigestures file (86.85 KB, application/octet-stream)
2011-11-03 18:11 UTC, paolosca
Details
messages.log I got after starting netbeans and replicating the issue (37.07 KB, text/x-log)
2011-11-04 08:07 UTC, paolosca
Details
profiler snapshot (1.75 MB, application/octet-stream)
2011-11-17 17:35 UTC, Tomas Hurka
Details
An attempt to prevent changes to be fired from ProxyLookup after each beforeLookup (12.74 KB, patch)
2012-05-17 09:27 UTC, Jaroslav Tulach
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description paolosca 2011-11-03 18:09:26 UTC
Created attachment 112791 [details]
Thread dump taken while the ide was unresponsive.

Product Version: NetBeans IDE Dev (Build 201111020600)
Java: 1.6.0_26; Java HotSpot(TM) Client VM 20.1-b02
System: Linux version 3.0.0-2-686-pae running on i386; UTF-8; en_US (nb)
User directory: /home/paolo/.netbeans/dev
Cache directory: /run/netbeans

I opened the project panel, opened a directory in the project tree containing about 230 files, mostly images. I selected all the files clicking on the last and while holding the shift key I clicked on the first with the intention of deleting them.

After doing the selection I had no visual feedback of the operation (the selected files didn't change the background color and the processor fan began running faster. After several seconds the files background turned blue and I right clicked on the files to delete them using the context menu but the delete item was disabled.

At this point I noticed a progress bar at the bottom of the window saying "saving snapshot" and NetBeans became progressively unresponsive.

I let NetBeans run for several minutes during which time the CPU was running at 100%, when the progressbar reached 5% a Out of memory error popped up.

The files I was selecting were marked as svn ignore, but I'm not sure it has anything to do with this problem.

Also, I tried to upload the error from netbeans itself using the exception reporter but it ended with another Out of memory.

I attach a thread dump and logs. To take the thread dump I had to use the -F switch after getting the following error from jstack:

941: Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding

Regards,

Paolo
Comment 1 paolosca 2011-11-03 18:10:21 UTC
Created attachment 112792 [details]
Netbeans messages log
Comment 2 paolosca 2011-11-03 18:10:48 UTC
Created attachment 112793 [details]
netbeans metrics file
Comment 3 paolosca 2011-11-03 18:11:22 UTC
Created attachment 112794 [details]
netbeans uigestures file
Comment 4 paolosca 2011-11-03 18:14:04 UTC
I just noticed that in var/log I have a file called heapdump.hprof.old which is almost 1GB in size:

928270209 Nov  4 01:40 heapdump.hprof.old

If you need it I can try to upload it somewhere. 

Please, let me know if I can delete it, thank you.
Comment 5 paolosca 2011-11-04 08:06:17 UTC
I can repeat the bug every time I try to select all the files in the same directory. The order in which I select them starting from the first or from the last doesn't change the result. I always get a java.lang.OutOfMemoryError exception.

Maybe because I just started netbeans and the memory used was lower this time the saving snapshot reached 16%. I attach the latest messages.log, since I just started netbeans it should be cleaner.
Comment 6 paolosca 2011-11-04 08:07:41 UTC
Created attachment 112811 [details]
messages.log I got after starting netbeans and replicating the issue
Comment 7 Jesse Glick 2011-11-10 23:32:54 UTC
Reproducible in a dev build:

for i in 0 1 2 3 4 5 6 7 8 9; do for j in 0 1 2 3 4 5 6 7 8 9; do for k in 0 1 2 3 4 5 6 7 8 9; do touch /tmp/test204645/y$i$j$k.txt; done; done; done

then expand that folder in Favorites and multiselect it.

Seems to be a stack overflow, or at least very deep recursion:

	at org.openide.nodes.NodeLookup.updateLookupAsCookiesAreChanged(NodeLookup.java:192)
	at org.openide.nodes.Node.fireCookieChange(Node.java:1247)
	at org.openide.loaders.DataNode.access$700(DataNode.java:68)
	at org.openide.loaders.DataNode$4.run(DataNode.java:774)
	at org.openide.util.Mutex.doEvent(Mutex.java:1341)
	at org.openide.util.Mutex.writeAccess(Mutex.java:455)
	at org.openide.loaders.DataNode.fireChange(DataNode.java:750)
	at org.openide.loaders.DataNode$PropL.propertyChange(DataNode.java:894)
	at org.openide.util.WeakListenerImpl$PropertyChange.propertyChange(WeakListenerImpl.java:196)
	at java.beans.PropertyChangeSupport.firePropertyChange(PropertyChangeSupport.java:339)
	at java.beans.PropertyChangeSupport.firePropertyChange(PropertyChangeSupport.java:276)
	at org.openide.loaders.DataObject.firePropertyChange(DataObject.java:943)
	at org.openide.loaders.MultiDataObject.fireCookieChange(MultiDataObject.java:1007)
	at org.openide.loaders.MultiDataObject$ChangeAndBefore.stateChanged(MultiDataObject.java:1438)
	at org.openide.util.ChangeSupport.fireChange(ChangeSupport.java:133)
	at org.openide.util.ChangeSupport.fireChange(ChangeSupport.java:119)
	at org.openide.nodes.CookieSet.fireChangeEvent(CookieSet.java:303)
	at org.openide.nodes.CookieSetLkp.replaceInstances(CookieSetLkp.java:132)
	at org.openide.nodes.CookieSet.assign(CookieSet.java:520)
	at org.openide.loaders.MultiDataObject.updateFilesInCookieSet(MultiDataObject.java:1417)
	at org.openide.loaders.MultiDataObject$ChangeAndBefore.beforeLookup(MultiDataObject.java:1444)
	at org.openide.nodes.CookieSetLkp.beforeLookupImpl(CookieSetLkp.java:146)
	at org.openide.nodes.CookieSet.getCookie(CookieSet.java:173)
	at org.openide.loaders.MultiDataObject.getCookie(MultiDataObject.java:947)
	at org.openide.loaders.DefaultDataObject.getCookie(DefaultDataObject.java:233)
	at org.openide.loaders.DefaultDataObject.getCookie(DefaultDataObject.java:214)
	at org.openide.loaders.DataNode.getCookie(DataNode.java:435)
	at org.openide.nodes.NodeLookup.addCookie(NodeLookup.java:100)
	at org.openide.nodes.NodeLookup.updateLookupAsCookiesAreChanged(NodeLookup.java:192)

On one attempt to reproduce, I got some stack traces from org.netbeans.modules.project.ui.actions.ActionsUtil.getProjectsFromLookup that I should look up FileObject, not DataObject (bug #199391). This would be fine except this method must be usable for all kinds of selections, and not all of these have been updated to offer FileObject yet (e.g. the form editor). Even when changing that, just selecting the first item and then holding down Shift while extending the selection for a few dozen files results in many stack traces from the explorer which I do not really follow:

java.lang.Exception: Find for /tmp/test204645/x044.gif@5fb73852:e8cc00
	at org.openide.loaders.FolderChildren$DelayedNode.convert(FolderChildren.java:451)
	at org.openide.loaders.FolderChildren$DelayedNode.convert(FolderChildren.java:399)
	at org.openide.util.lookup.InstanceContent$ConvertingItem.getInstance(InstanceContent.java:316)
	at org.openide.explorer.DefaultEMLookup$NoNodeLookup$ExclusionResult.allItems(DefaultEMLookup.java:258)
	at org.openide.util.lookup.ProxyLookup$R.computeResult(ProxyLookup.java:563)
	at org.openide.util.lookup.ProxyLookup$R.allItems(ProxyLookup.java:517)
	at org.openide.util.lookup.ProxyLookup$R.collectFires(ProxyLookup.java:628)
	at org.openide.util.lookup.ProxyLookup.setLookups(ProxyLookup.java:168)
	at org.openide.util.lookup.ProxyLookup.setLookups(ProxyLookup.java:124)
	at org.openide.explorer.DefaultEMLookup.updateLookups(DefaultEMLookup.java:134)
	at org.openide.explorer.DefaultEMLookup.propertyChange(DefaultEMLookup.java:166)
	at org.openide.util.WeakListenerImpl$PropertyChange.propertyChange(WeakListenerImpl.java:196)
	at java.beans.PropertyChangeSupport.firePropertyChange(PropertyChangeSupport.java:339)
	at java.beans.PropertyChangeSupport.firePropertyChange(PropertyChangeSupport.java:276)
	at org.openide.explorer.ExplorerManager$2.run(ExplorerManager.java:868)
	at org.openide.util.Mutex.doEvent(Mutex.java:1341)
	at org.openide.util.Mutex.readAccess(Mutex.java:348)
	at org.openide.explorer.ExplorerManager.fireInAWT(ExplorerManager.java:865)
	at org.openide.explorer.ExplorerManager$1AtomicSetSelectedNodes.fire(ExplorerManager.java:285)
	at org.openide.explorer.ExplorerManager.setSelectedNodes(ExplorerManager.java:296)

and just printing all these is enough to make the IDE slow down to a crawl.
Comment 8 Jesse Glick 2011-11-10 23:37:18 UTC
BTW I did not specifically notice OOMEs, just CPU usage. Reporter, if you encounter temporary (recoverable) freezes, the best thing is to reproduce them on a small enough sample that there is a freeze for several seconds but it recovers; then a self-profiler snapshot should automatically get sent to the exception reporter. Or, you can manually Alt-Shift-Y to start and stop the self-sampler, and attach the result to an existing bug report. If you reproduce OOMEs not necessarily connected to freezes, you can use jmap -histo to get a summary of what is wrong.
Comment 9 paolosca 2011-11-13 17:48:57 UTC
(In reply to comment #8)
> then a self-profiler snapshot should automatically get sent to the
> exception reporter.

If I do the same operation (select files in the panel) after starting NetBeans I don't get an out of memory error. After selecting the IDE freezes for a few (9-10 seconds) then the "saving snapshot" progress bar appears. The snapshot saving takes a few minutes, I tried to report the "slowness detected" but I can't find the report anywhere on statistics.netbeans.org. 

I'm also unable to find the saved snapshot anywhere in the NB home and cache directories, all I can find are the usual log files.

> Or, you can manually Alt-Shift-Y to start and stop the
> self-sampler, and attach the result to an existing bug report.

If I do that it will start saving a snapshot while the slowness detector is already saving one. The saving snapshot stopped at 50% and some partes of the IDE stopped working (i.e. The plugins list in the plugin manager is empty). I didn't get any Out of Memory error, is there any chance the error has been suppressed or the operation halted before the memory limit reached?

I believe the out of memory is due to the IDE using all the memory to save the snapshot rather then the files selection operation.
Comment 10 Jaroslav Tulach 2011-11-14 12:01:51 UTC
*** Bug 204476 has been marked as a duplicate of this bug. ***
Comment 11 Jesse Glick 2011-11-16 01:18:00 UTC
(In reply to comment #9)
> (In reply to comment #8)
> I tried to report the "slowness detected" but I
> can't find the report anywhere on statistics.netbeans.org.

The server is broken at the moment if I recall correctly - successfully uploading but not showing the result in the IDE.

> I didn't get any Out of Memory error, is there any chance the error has been
> suppressed

Possible.
Comment 12 Tomas Hurka 2011-11-17 17:35:16 UTC
Created attachment 113292 [details]
profiler snapshot

I was able to reproduce the problem by selecting a lot of jpegs in one package (under the source root). The problem is caused by extraordinary deep stack trace in AWT. See attached npss file.
Comment 13 Jaroslav Tulach 2012-05-17 09:26:05 UTC
It is hard to debug the problem, so I reported bug 212619 and I am making this one depend on it (in spite the technical solution will not depend on bug 212619).
Comment 14 Jaroslav Tulach 2012-05-17 09:27:18 UTC
Created attachment 119559 [details]
An attempt to prevent changes to be fired from ProxyLookup after each beforeLookup
Comment 15 Jaroslav Tulach 2012-11-05 13:38:45 UTC
*** Bug 221412 has been marked as a duplicate of this bug. ***
Comment 16 Ondrej Vrabec 2014-02-10 08:40:58 UTC
lookup issue, firing too many events. Prob. a dup of #235748
Comment 17 Jaroslav Tulach 2014-04-04 08:50:51 UTC

*** This bug has been marked as a duplicate of bug 235748 ***


By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo