221885 – "background scanning" running out of memory - adding ExtJs javascript libraries to the project's ignore list sorts the issue while its scanning

This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 221885 - "background scanning" running out of memory - adding ExtJs javascript libraries to the project's ignore list sorts the issue while its scanning

Summary: "background scanning" running out of memory - adding ExtJs javascript librari...

Status:	RESOLVED DUPLICATE of bug 222823

Alias:	None

Product:	javascript
Classification:	Unclassified
Component:	Editor (show other bugs)
Version:	7.3
Hardware:	PC Windows 7

Importance:	P2 normal with 1 vote (vote)
Assignee:	Petr Pisl

URL:
Keywords:	PERFORMANCE

Depends on:
Blocks:

Reported:	2012-11-11 00:30 UTC by padraigdoran
Modified:	2013-01-18 07:18 UTC (History)
CC List:	4 users (show)

See Also:
Issue Type:	DEFECT
Exception Reporter:

Attachments
export of MemoryAnalyzer run of heapdump file (820.47 KB, application/zip) 2012-11-29 19:36 UTC, padraigdoran	Details
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.

Description padraigdoran 2012-11-11 00:30:57 UTC

Product Version = NetBeans IDE Dev (Build 201211100001)
Operating System = Windows 7 version 6.1 running on x86
Java; VM; Vendor = 1.7.0_09
Runtime = Java HotSpot(TM) Client VM 23.5-b02


Running NetBeans PHP Nightly 201211100001

Fresh install:
delete Netbeans folder,
deleted AppData\Local\NetBeans folder
deleted AppData\Roaming\NetBeans folder

Delete nbproject folder from my project

So, opened up NetBeans, created a new PHP project "with existing sources", then the "background scanning" kicked in and same problems as before.

The project contains a javascript library folder in it. It's main component is ExtJs 3 and 4 which are fairly large includes.
The "background scanning" issue goes away if I put the ext folders into the "ignore" folder list on the project properties list.

Hope this helps, the error is fairly annoying for myself and my collegues.

If you need me to anything specific I'll do my best to help.

The following are multiple exceptions that were reported, here are the messages:

=============================================================================================================
#1

java.lang.OutOfMemoryError: Java heap space
	at java.io.ByteArrayOutputStream.<init>(Unknown Source)
	at org.netbeans.modules.sampler.Sampler.start(Sampler.java:171)
	at org.netbeans.core.TimableEventQueue.run(TimableEventQueue.java:226)
	at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1454)
	at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:2036)

=============================================================================================================
	
#2

java.lang.OutOfMemoryError: Java heap space
	at com.sun.jna.Native.updateLastError(Native.java:818)
	at com.sun.jna.Native.invokeInt(Native Method)
	at com.sun.jna.Function.invoke(Function.java:344)
	at com.sun.jna.Function.invoke(Function.java:276)
	at com.sun.jna.Library$Handler.invoke(Library.java:216)
	at org.netbeans.modules.masterfs.watcher.windows.$Proxy3.GetQueuedCompletionStatus(Unknown Source)
	at org.netbeans.modules.masterfs.watcher.windows.WindowsNotifier.waitForChange(WindowsNotifier.java:430)
	at org.netbeans.modules.masterfs.watcher.windows.WindowsNotifier.access$100(WindowsNotifier.java:75)
	at org.netbeans.modules.masterfs.watcher.windows.WindowsNotifier$2.run(WindowsNotifier.java:361)

	
=============================================================================================================

#3

java.lang.OutOfMemoryError: Java heap space
	at java.util.HashMap.<init>(Unknown Source)
	at java.util.HashMap.<init>(Unknown Source)
	at java.util.HashSet.<init>(Unknown Source)
	at org.netbeans.modules.javascript2.editor.model.impl.ModelUtils$SemiTypeResolverVisitor.<init>(ModelUtils.java:674)
	at org.netbeans.modules.javascript2.editor.model.impl.ModelUtils.resolveSemiTypeOfExpression(ModelUtils.java:281)
	at org.netbeans.modules.javascript2.editor.model.impl.ModelVisitor.enter(ModelVisitor.java:665)
	at com.oracle.nashorn.ir.PropertyNode.accept(PropertyNode.java:88)
	at com.oracle.nashorn.ir.ObjectNode.accept(ObjectNode.java:88)
	at com.oracle.nashorn.ir.CallNode.accept(CallNode.java:212)
	at com.oracle.nashorn.ir.CallNode.accept(CallNode.java:212)
	at com.oracle.nashorn.ir.ReturnNode.accept(ReturnNode.java:99)
	at org.netbeans.modules.javascript2.editor.model.impl.ModelVisitor.enter(ModelVisitor.java:486)
	at com.oracle.nashorn.ir.FunctionNode.accept(FunctionNode.java:308)
	at org.netbeans.modules.javascript2.editor.model.impl.ModelVisitor.enter(ModelVisitor.java:688)
	at com.oracle.nashorn.ir.ReferenceNode.accept(ReferenceNode.java:69)
	at com.oracle.nashorn.ir.PropertyNode.accept(PropertyNode.java:92)
	at com.oracle.nashorn.ir.ObjectNode.accept(ObjectNode.java:88)
	at com.oracle.nashorn.ir.CallNode.accept(CallNode.java:212)
	at com.oracle.nashorn.ir.ExecuteNode.accept(ExecuteNode.java:79)
	at org.netbeans.modules.javascript2.editor.model.impl.ModelVisitor.enter(ModelVisitor.java:486)
	at com.oracle.nashorn.ir.FunctionNode.accept(FunctionNode.java:308)
	at org.netbeans.modules.javascript2.editor.model.impl.ModelVisitor.enter(ModelVisitor.java:688)
	at com.oracle.nashorn.ir.ReferenceNode.accept(ReferenceNode.java:69)
	at com.oracle.nashorn.ir.CallNode.accept(CallNode.java:209)
	at com.oracle.nashorn.ir.ExecuteNode.accept(ExecuteNode.java:79)
	at org.netbeans.modules.javascript2.editor.model.impl.ModelVisitor.enter(ModelVisitor.java:486)
	at com.oracle.nashorn.ir.FunctionNode.accept(FunctionNode.java:308)
	at org.netbeans.modules.javascript2.editor.model.Model.getModelVisitor(Model.java:86)
	at org.netbeans.modules.javascript2.editor.model.Model.getGlobalObject(Model.java:96)
	at org.netbeans.modules.javascript2.editor.index.JsIndexer.index(JsIndexer.java:92)
	at org.netbeans.modules.parsing.spi.indexing.Indexable$MyAccessor$3.run(Indexable.java:216)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater.runIndexer(RepositoryUpdater.java:267)

=============================================================================================================

	
#4

java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Unknown Source)
	at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source)
	at java.lang.AbstractStringBuilder.ensureCapacityInternal(Unknown Source)
	at java.lang.AbstractStringBuilder.append(Unknown Source)
	at java.lang.StringBuilder.append(Unknown Source)
	at org.netbeans.core.startup.logging.NbFormatter.print(NbFormatter.java:110)
	at org.netbeans.core.startup.logging.NbFormatter.format(NbFormatter.java:68)
	at java.util.logging.StreamHandler.publish(Unknown Source)
	at org.netbeans.core.startup.logging.DispatchingHandler.run(DispatchingHandler.java:175)
	at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1454)
	at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:2036)


=============================================================================================================

#5


java.lang.OutOfMemoryError: Java heap space

=============================================================================================================

#6

java.lang.OutOfMemoryError: Java heap space

=============================================================================================================

Comment 1 Petr Hejl 2012-11-28 10:44:10 UTC

Can you share a heapdump (http://wiki.netbeans.org/FaqMemoryDump)? Can you also share a project or describe its setup in more detail? Thanks.

Comment 2 Petr Hejl 2012-11-29 13:36:50 UTC

Can you also test with the latest daily?

Comment 3 padraigdoran 2012-11-29 15:43:23 UTC

Hi Petr, was looking at the headdump file and it contains file names of PHP files of our software. As such I won't be able to post that file without removing them. Is there an easy way to remove private data from the dump?

Comment 4 padraigdoran 2012-11-29 17:24:40 UTC

Hi,

I've tested it on 2012 11 28 and it's not working. Will try on 2012 11 29 shortly.

I made an example test folder by stripping out all the file except the EXT JS libraries but that worked okay. Background scanning took a fraction of the time.

The system is mainly PHP based, has EXT JS 3.4 and 4 as the external JS libraries, uses the Smarty Templating Engine 2. Another point is that it uses Mercurial for version control so the .hg folder is there too.

My main systems has:
PHP in multiple folders (mainly root),
/include/external for external PHP libraries (PEAR, Smarty, tcpdf, etc.)
/js/external for the external libraries,
/js/internal for our own JS files,
/css folder,
/templates & /templates_c folders for the Smarty tpl files and compiled versions of same,
/images for the images :)

The folder size is about 265MB, has 17,882 Files and 1,831 Folders


Anyway, it seems like it's a combination of the above that's causing the out of mem exception, not just the EXT JS library as I was thinking.

Comment 5 padraigdoran 2012-11-29 19:36:09 UTC

Created attachment 128604 [details]
export of MemoryAnalyzer run of heapdump file

I've analysed the heapdump file with Eclipse MemoryAnalyzer and have zipped up the reports into this attached file. Hopefully this will help.

Comment 6 padraigdoran 2012-11-29 21:29:04 UTC

By the way, I've tested with latest nightly (2012 11 29) my work PC and home PC and both get the error

Comment 7 Petr Hejl 2012-11-30 13:42:45 UTC

Thanks for your help padraigdoran. BTW does this happen even without any file opened in the editor on start?

Petre, this looks suspicious:
org.netbeans.modules.javascript2.editor.model.impl.JsFunctionImpl one instance consuming 78MB, containing 1,816 declared scopes.

Comment 8 padraigdoran 2012-11-30 13:56:24 UTC

Hi Petr, yes it still happens without any file open in the editor.

I've tested it with other JRE and JDK versions but it's still the same thing. Here are the version I tested, all with NB Nightly 20121130

JRE 6u37
JDK 6u37
JDK 6u38 (dev edition)


JRE 7u9
JDK 7u9
JDK 7u10 (dev edition)

Comment 9 Petr Hejl 2012-11-30 14:01:37 UTC

Investigating a bit more there is also org.netbeans.modules.parsing.impl.indexing.ClusteredIndexables$DocumentIndexCacheImpl occupying 193MB. This might be caused by our use of Document I guess - so it might not be a real leak but it just runs out of memory during indexing.

Comment 10 Petr Hejl 2012-11-30 14:06:46 UTC

padraigdoran can you also try to increase -Xmx being used by netbeans value to some really large value. You can done that by -J-Xmx1024m (as an example) in etc/netbeans.conf in nb installation dir.

Once done please test if the scan passes and if so please check by clicking on memory toolbar in daily build whether the memory is reclaimed (only some reasonable amount is occupied when the scan finishes). Thanks a lot.

Comment 11 padraigdoran 2012-11-30 15:32:25 UTC

Hi again,

I changed config to -J-Xmx1024m and ran it again. It took a while but it completed without any issues. The memory toolbar was bouncing around 650-780MB out of 980MB.

Once the scanning had been completed, I clicked on the memory toolbar and usage went down to about 130MB.

After opening a few files and editing them, mem usage is 320MB/890MB.

Anything else I can do to help?

Comment 12 Petr Hejl 2012-11-30 16:45:31 UTC

Thanks padraigdoran. I think we have all we need. At least for now.

Petre it looks like the indexing infrastructure is not optimised for the way we use in javascript2 and thus a big amount of memory is needed to finish the indexing (at least for big projects).

I don't think this is particular issue is memory leak.

Comment 13 padraigdoran 2012-11-30 17:17:24 UTC

Ok, that's fine. At least ye have a starting point. If ye need me to test anymore things or even test custom builds or whatever let me know here. Thanks for the help guys.

Comment 14 Petr Pisl 2012-12-04 09:43:26 UTC

On Friday there was done a fix that improve the behavior of indexing, when there are many embedings. Probably not related to this issue, but could you please try the latest build? Just to be sure, that the fix doesn't improve the situation.

Thanks

Comment 15 padraigdoran 2012-12-04 12:39:11 UTC

(In reply to comment #14)
> On Friday there was done a fix that improve the behavior of indexing, when
> there are many embedings. Probably not related to this issue, but could you
> please try the latest build? Just to be sure, that the fix doesn't improve the
> situation.
> 
> Thanks

Hi,

I've tested it again with a clean install/extract of NB 20121204 and it seems to work okay. The program did not run out of memory. The standard Xmx was left as is, i.e. I didn't change any setting. According to messages.log the setting was -Xmx512m

While background scanning the memory usage was around 480MB/492MB, then it would drop down to around 100MB then back up again. I didn't get any exception and the program remained responsive. So, whatever that last fix was last Friday it did a good job in helping this issue. The scan itself is still slow though.

Comment 16 Petr Pisl 2012-12-04 13:11:28 UTC

Thanks a lot. I close it as duplicate of issue #222823

*** This bug has been marked as a duplicate of bug 222823 ***

Comment 17 padraigdoran 2012-12-04 14:58:58 UTC

Excellent, thanks for the help.

Comment 18 padraigdoran 2013-01-03 23:15:38 UTC

With the latest nightly, 20130103 the background scanning problem has come back. As far as I know, the bug was reintroduced around 2012-12-24.

Comment 19 padraigdoran 2013-01-17 17:18:34 UTC

(In reply to comment #18)
> With the latest nightly, 20130103 the background scanning problem has come
> back. As far as I know, the bug was reintroduced around 2012-12-24.

Ignore that comment, I think I did something wrong on my side. It's been working perfectly since. Closing issue now. Thanks again.

Comment 20 Vladimir Riha 2013-01-18 07:14:58 UTC

Thanks padraigdoran.

So I guess this is no longer HR_FIX_CANDIDATE since there are no changes made after code freeze and it works fine for reporter => removing keyword

Comment 21 Marian Mirilovic 2013-01-18 07:18:34 UTC


*** This bug has been marked as a duplicate of bug 222823 ***