Bug 207991 - Large project hangs Netbeans 7.1 indefinitely.
Large project hangs Netbeans 7.1 indefinitely.
Status: RESOLVED FIXED
Product: web
Classification: Unclassified
Component: HTML Editor
7.1
All All
: P1 (vote)
: 7.2
Assigned To: Marek Fukala
issues@javaee
http://netbeans.org/projects/profiler...
:
: 200246 204525 205603 207222 207535 207837 209270 209539 209657 209704 209749 211770 212207 212740 214473 215385 220253 (view as bug list)
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2012-02-01 15:37 UTC by robshelby
Modified: 2012-11-22 14:42 UTC (History)
38 users (show)

See Also:
Issue Type: DEFECT
:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description robshelby 2012-02-01 15:37:27 UTC
This problem is very similar to http://netbeans.org/bugzilla/show_bug.cgi?id=201531

When a large PHP project is opened, it hangs (spins) during "scanning projects".

This works perfect for the same projects in 7.0.1.

I've tried the new dev preview of Java today, but it didn't fix it.

Here is a link to a heapdump.
http://ubuntuone.com/76IsXeAWsS5GlTs9RvuG8t

Please let me know when you download this so it can be deleted.

Thanks!
Comment 1 Tomas Hurka 2012-02-02 16:51:56 UTC
Your project has at least one source file which is 9,371,506 bytes long. What kind of source file it is? It does not make much sense to me, unless it is machine generated.
Comment 2 robshelby 2012-02-02 17:00:32 UTC
Yes, they are machine generated csv files. They've never been opened by Netbeans, they just happen to be within the project folder.

Thanks!
Comment 3 Tomas Mysik 2012-02-03 05:17:46 UTC
Perhaps it could help also if you could attach a profiler snapshot [1] taken when the IDE hangs (snapshot for a minute or so should be fine).

Thanks.
[1] http://wiki.netbeans.org/FitnessViaPartnership
Comment 4 Tomas Hurka 2012-02-03 12:22:33 UTC
(In reply to comment #0)
> Here is a link to a heapdump.
> http://ubuntuone.com/76IsXeAWsS5GlTs9RvuG8t
> 
> Please let me know when you download this so it can be deleted.
You can delete the heap dump. It is not available here: http://netbeans.org/projects/profiler/downloads/download/Heapdumps/heapdump-207991.zip
Comment 5 Tomas Hurka 2012-02-03 12:59:02 UTC
(In reply to comment #3)
> Perhaps it could help also if you could attach a profiler snapshot [1] taken
> when the IDE hangs (snapshot for a minute or so should be fine).
> 
> Thanks.
> [1] http://wiki.netbeans.org/FitnessViaPartnership
Tomas, I think that the hang is caused by OOME. In this case profiler snapshot does not help you much.
Comment 6 Tomas Mysik 2012-02-03 13:22:40 UTC
(In reply to comment #5)
> Tomas, I think that the hang is caused by OOME. In this case profiler snapshot
> does not help you much.

Aha, thanks. I will evaluate it soon.
Comment 7 robshelby 2012-02-03 14:47:32 UTC
I ran that fitness debugger, but I can't seem to find where it saved the file. Do you still need that?

Thanks!
Comment 8 Tomas Mysik 2012-02-06 08:37:27 UTC
I will let you know soon.

Thanks.
Comment 9 Tomas Mysik 2012-02-06 14:34:16 UTC
It seems that the problem is in the CSS3 and HTML5 parsing (introduced in NB 7.1). There is one huge HTML file [1] (around 9 MB) and during its parsing, OOME occurs. The file is from the documentation of PHP Excel library. If we are correct then removing (deleting) this documentation (the whole directory phpexcel/Documentation) should help.

Anyway, I will try to reproduce this issue and reassign to HTML if the OOME occurs.

Thanks.
[1] system/libraries/phpexcel/Documentation/API/__filesource/fsource_com-tecnick-tcpdf__PHPExcelSharedPDFtcpdf.php.html
Comment 10 Tomas Mysik 2012-02-06 14:38:25 UTC
Yes, reproducible in the current dev build. Marku, here are the exact steps:

- download PHP Excel library [1],
- unzip it,
- create a php project with existing sources from its directory,

-> after a while, OOME occurs.

TomasZ could have some ideas what could be done/improved in CSS/HTML parsing.

Product Version: NetBeans IDE Dev (Build 20120206-429eb201b368)
Java: 1.6.0_30; Java HotSpot(TM) 64-Bit Server VM 20.5-b03
System: Linux version 3.2.2 running on amd64; UTF-8; cs_CZ (nb)

[1] http://phpexcel.codeplex.com/
Comment 11 Tomas Zezula 2012-02-17 10:41:10 UTC
*** Bug 207222 has been marked as a duplicate of this bug. ***
Comment 12 Tomas Zezula 2012-02-20 14:25:56 UTC
*** Bug 207222 has been marked as a duplicate of this bug. ***
Comment 13 Tomas Zezula 2012-02-20 17:33:18 UTC
*** Bug 207222 has been marked as a duplicate of this bug. ***
Comment 14 Marek Fukala 2012-02-21 12:03:08 UTC
To me it seems simply too big file to work with. What I got is:

Heap dump file created [915015990 bytes in 31.745 secs]
SEVERE [org.openide.util.Exceptions]
java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:2760)
	at java.util.Arrays.copyOf(Arrays.java:2734)
	at java.util.ArrayList.toArray(ArrayList.java:275)
	at java.util.Collections.sort(Collections.java:158)
	at org.netbeans.modules.parsing.api.Embedding.create(Embedding.java:149)
	at org.netbeans.modules.html.editor.gsf.embedding.CssEmbeddingProvider.getEmbeddings(CssEmbeddingProvider.java:116)
	at org.netbeans.modules.parsing.impl.TaskProcessor.callEmbeddingProvider(TaskProcessor.java:573)
	at org.netbeans.modules.parsing.impl.SourceCache.refresh(SourceCache.java:328)
[catch] at org.netbeans.modules.parsing.impl.TaskProcessor$CompilationJob.run(TaskProcessor.java:714)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:680)

which is actually just a creation of css virtual source (parser embedding in the html source).
Comment 15 Marek Fukala 2012-02-21 12:40:37 UTC
just some examples:

~2.4M of Css parser tree nodes in 84MB
~2M of Html parser nodes in 65MB
~2.3M of lexer tokens in 55MB
~3M of int[] objects held by Snapshots in 50MB
...

PHP -> HTML virtual source (snapshot, tokens, parser) -> css virtual source (snapshot, tokens, parser)

there's definitively a room for improvements on both sides (parsing api & css/html)

I can do some optimization on the css/html side but still it'd be handy if a file size limit is introduced for the indexing since even with the optimization the OOM will happen on bigger files.
Comment 16 Tomas Zezula 2012-02-21 13:51:06 UTC
>limit is introduced for the indexing
Yes, such a limit is good idea. Unfortunately such a limit is not generic it has to be defined per language.
Comment 17 robshelby 2012-02-21 14:32:25 UTC
I doubt this helps, but it still happens in Mountain Lion.
Comment 18 Exceptions Reporter 2012-03-15 13:20:15 UTC
This bug already has 100 duplicates 
see http://statistics.netbeans.org/exceptions/detail.do?id=184356
Comment 19 Marek Fukala 2012-03-20 16:28:20 UTC
changeset:   216691:db08b987a0a2
summary:     #207991 - decreasing the SyntaxElement$Tag instance retained size
changeset:   216692:e29fbf03c08b
summary:     #207991 - using CharSequence instead of String where appropriate
changeset:   216694:e07f48bf772b
summary:     providing no folds and structure items for snapshots > 4MB. User should be warned of opening such big files, but it apparently doesn't happen for some php files. This change is considered to be rather temporary to allow to reveal other problems.
changeset:   216695:eb94a785996d
summary:     #207991 - The css parsing.api embedding won't be created for html snapshots bigger than 4MB
Comment 20 Quality Engineering 2012-03-21 12:05:22 UTC
Integrated into 'main-golden', will be available in build *201203210400* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main-golden/rev/db08b987a0a2
User: Marek Fukala <mfukala@netbeans.org>
Log: #207991 - decreasing the SyntaxElement$Tag instance retained size
Comment 21 athompson 2012-03-21 15:25:52 UTC
Just to add that this happens to me with a grails plugin project.  It contains no HTML or PHP.
Comment 22 athompson 2012-03-21 15:26:28 UTC
...and no large files.  In fact, a very small project.
Comment 23 Tomas Zezula 2012-03-21 18:20:26 UTC
I've looked at duplicates and it seems that there are lots of completely different problems automatically marked as duplicate reports. This issue and original reports were about memory consumption of the CSS and HTML. The Groovy reports belong to groovy, etc. Unfortunately all these reports were added by exception reporter into this one. The reports need to be split case by case to individual modules, lots of them are already fixed in dev especially the java related.
Comment 24 Quality Engineering 2012-03-22 11:14:16 UTC
Integrated into 'main-golden', will be available in build *201203220400* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main-golden/rev/e5e104c201af
User: Marek Fukala <mfukala@netbeans.org>
Log: #207991 - 9MB saving on AstNode class optimalization (for the big testing html file)
Comment 25 athompson 2012-03-26 16:33:50 UTC
added issue #210133 for the grails project freezes
Comment 26 Marek Fukala 2012-04-02 12:24:00 UTC
changeset:   218458:4733d29d47b4
summary:     #207991 - lowering the retained size of the plain syntax elements by tens megabytes

changeset:   218468:30397e1e1201
summary:     #207991 - great memory save - using html element's iteration instead of passing elements collections.

Both changes improved the memory usage for the sample 9MB file by I dare to say several hundreds of megabytes. Now even the editing is possible including indentation and code completion.
Comment 27 Marek Fukala 2012-04-02 12:26:59 UTC
I forgot to list an earlier change - some redesign in the html.editor.lib module allowed to save quite a lot of memory as well:

http://hg.netbeans.org/web-main/rev/1af99f09e439
Comment 28 Quality Engineering 2012-04-03 11:11:33 UTC
Integrated into 'main-golden', will be available in build *201204030400* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main-golden/rev/4733d29d47b4
User: Marek Fukala <mfukala@netbeans.org>
Log: #207991 - lowering the retained size of the plain syntax elements by tens megabytes
Comment 29 robshelby 2012-04-03 12:44:26 UTC
Just FYI, I've been using dev 7.2 for a week now with the same project that was crashing 7.1...

It's been working great!
Comment 30 Tomas Zezula 2012-04-03 13:00:34 UTC
The infrastructure as well as several indexers affecting indexing performance were rewritten in NB 7.2, see http://wiki.netbeans.org/IndexingPerformance72.
Comment 31 Marek Fukala 2012-04-03 15:24:08 UTC
changeset:   218478:2d38129a461f
summary:     #207991 - do not hold html.editor.lib elements

changeset:   218477:59d24350ccdf
summary:     #207991 - enable/disable html error check don't hold snapshots, but file and document instead

changeset:   218610:74a5b9a2b17e
summary:     #207991 - using a java.io.Reader for CharSequence instead of cloning the source into a new String object in html5 parser. Also masking of the content hidden to the parser is done via a special masking CharSequence instead of wiping the ares in a new StringBuilder

now the file keeps at ~280MB of heap during editing. Still there's a memory leak when the file is close and reopened - some of the parser results are held twice.
Comment 32 Tomas Zezula 2012-04-03 17:13:50 UTC
As the issue had lots of wrong duplicates not related to the html/css.
Here is a list of fixes in the generic and non html/css indexers:

#206062:Navigator should provide data for file during scan 41985eb45913

#182653: Transactional Index 41985eb45913

#208097: Update CSL languages to support transactional index 8c30d396959b

#206059: Filter unrelated BinaryIndexers  d00d64d8b0ad

#206057, #206054: minimize IO operations in Java during up to date check 2f9cd7092a7e, 9a9337ad3afc

#206024: Fix slow project queries 263679ed6809, ee4e39fada10, 38c1bcc2f56b

#206021: Speed up file content loading for embed able languages e0ee7f0f0f1d

#206020: Run BinaryIndexers in parallel c0b7423861a8

#206017: Prefetch java source files during indexing 1687f3d12679
Comment 33 robyt 2012-04-04 07:33:20 UTC
(In reply to comment #29)
> Just FYI, I've been using dev 7.2 for a week now with the same project that was
> crashing 7.1...
> 
> It's been working great!

Work for me too without problem
Comment 34 Tomas Zezula 2012-04-04 07:58:34 UTC
One more mostly related to PHP projects with symbolic links in projects:
#208392: Scanning project with multiple symlinks does not end if placed in symlinked folder eab0e72268c9
Comment 35 snez 2012-04-04 08:05:22 UTC
The fixes sound promising, waiting for an os x binary release.
Comment 36 Marek Fukala 2012-04-04 08:10:26 UTC
changeset:   218769:f9adfc591c0a
summary:     #207991 - HtmlNavigationSideBar element's don't hold parser result via MouseListener. That was a problem since when an opened html file is closed an instance of NavigationSideBar still exists and cannot be GCed (looks like another bug in editor)

filed a new editor/spellchecker bug 210669 - memory leak: QuietEditorPane -> TokenList
Comment 37 Marek Fukala 2012-04-04 08:37:04 UTC
Most of the html/css problems has been fixed. According to tzezula's statement the other problematic areas have been addressed as well so lets consider the issue as fixed.
Comment 38 Quality Engineering 2012-04-04 10:09:01 UTC
Integrated into 'main-golden', will be available in build *201204040400* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main-golden/rev/74a5b9a2b17e
User: Marek Fukala <mfukala@netbeans.org>
Log: #207991 - using a java.io.Reader for CharSequence instead of cloning the source into a new String object in html5 parser. Also masking of the content hidden to the parser is done via a special masking CharSequence instead of wiping the ares in a new StringBuilder
Comment 39 Quality Engineering 2012-04-05 09:39:25 UTC
Integrated into 'main-golden', will be available in build *201204050400* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main-golden/rev/f9adfc591c0a
User: Marek Fukala <mfukala@netbeans.org>
Log: #207991 - HtmlNavigationSideBar element's don't hold parser result via MouseListener. That was a problem since when an opened html file is closed an instance of NavigationSideBar still exists and cannot be GCed (looks like another bug in editor)
Comment 40 Marek Fukala 2012-04-11 13:32:41 UTC
*** Bug 209749 has been marked as a duplicate of this bug. ***
Comment 41 Marek Fukala 2012-04-11 13:33:12 UTC
*** Bug 209704 has been marked as a duplicate of this bug. ***
Comment 42 Marek Fukala 2012-04-11 13:33:48 UTC
*** Bug 209657 has been marked as a duplicate of this bug. ***
Comment 43 Marek Fukala 2012-04-11 13:34:14 UTC
*** Bug 209539 has been marked as a duplicate of this bug. ***
Comment 44 Marek Fukala 2012-04-19 21:15:19 UTC
*** Bug 207535 has been marked as a duplicate of this bug. ***
Comment 45 Marek Fukala 2012-04-20 10:24:14 UTC
*** Bug 200246 has been marked as a duplicate of this bug. ***
Comment 46 Marek Fukala 2012-04-20 11:27:54 UTC
*** Bug 204525 has been marked as a duplicate of this bug. ***
Comment 47 Marek Fukala 2012-04-20 11:51:10 UTC
*** Bug 205603 has been marked as a duplicate of this bug. ***
Comment 48 Tomas Zezula 2012-04-26 12:46:12 UTC
*** Bug 211770 has been marked as a duplicate of this bug. ***
Comment 49 Tomas Zezula 2012-05-02 16:28:40 UTC
*** Bug 207837 has been marked as a duplicate of this bug. ***
Comment 50 Tomas Zezula 2012-05-10 08:05:49 UTC
*** Bug 212207 has been marked as a duplicate of this bug. ***
Comment 51 Marek Fukala 2012-05-25 11:23:50 UTC
*** Bug 212740 has been marked as a duplicate of this bug. ***
Comment 52 Tomas Zezula 2012-06-11 18:26:19 UTC
*** Bug 209270 has been marked as a duplicate of this bug. ***
Comment 53 Milutin Kristofic 2012-07-11 11:54:30 UTC
*** Bug 215385 has been marked as a duplicate of this bug. ***
Comment 54 Petr Hejl 2012-09-07 22:24:38 UTC
*** Bug 214473 has been marked as a duplicate of this bug. ***
Comment 55 Marek Fukala 2012-11-22 14:42:38 UTC
*** Bug 220253 has been marked as a duplicate of this bug. ***


By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2014, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo