This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
This problem is very similar to http://netbeans.org/bugzilla/show_bug.cgi?id=201531 When a large PHP project is opened, it hangs (spins) during "scanning projects". This works perfect for the same projects in 7.0.1. I've tried the new dev preview of Java today, but it didn't fix it. Here is a link to a heapdump. http://ubuntuone.com/76IsXeAWsS5GlTs9RvuG8t Please let me know when you download this so it can be deleted. Thanks!
Your project has at least one source file which is 9,371,506 bytes long. What kind of source file it is? It does not make much sense to me, unless it is machine generated.
Yes, they are machine generated csv files. They've never been opened by Netbeans, they just happen to be within the project folder. Thanks!
Perhaps it could help also if you could attach a profiler snapshot [1] taken when the IDE hangs (snapshot for a minute or so should be fine). Thanks. [1] http://wiki.netbeans.org/FitnessViaPartnership
(In reply to comment #0) > Here is a link to a heapdump. > http://ubuntuone.com/76IsXeAWsS5GlTs9RvuG8t > > Please let me know when you download this so it can be deleted. You can delete the heap dump. It is not available here: http://netbeans.org/projects/profiler/downloads/download/Heapdumps/heapdump-207991.zip
(In reply to comment #3) > Perhaps it could help also if you could attach a profiler snapshot [1] taken > when the IDE hangs (snapshot for a minute or so should be fine). > > Thanks. > [1] http://wiki.netbeans.org/FitnessViaPartnership Tomas, I think that the hang is caused by OOME. In this case profiler snapshot does not help you much.
(In reply to comment #5) > Tomas, I think that the hang is caused by OOME. In this case profiler snapshot > does not help you much. Aha, thanks. I will evaluate it soon.
I ran that fitness debugger, but I can't seem to find where it saved the file. Do you still need that? Thanks!
I will let you know soon. Thanks.
It seems that the problem is in the CSS3 and HTML5 parsing (introduced in NB 7.1). There is one huge HTML file [1] (around 9 MB) and during its parsing, OOME occurs. The file is from the documentation of PHP Excel library. If we are correct then removing (deleting) this documentation (the whole directory phpexcel/Documentation) should help. Anyway, I will try to reproduce this issue and reassign to HTML if the OOME occurs. Thanks. [1] system/libraries/phpexcel/Documentation/API/__filesource/fsource_com-tecnick-tcpdf__PHPExcelSharedPDFtcpdf.php.html
Yes, reproducible in the current dev build. Marku, here are the exact steps: - download PHP Excel library [1], - unzip it, - create a php project with existing sources from its directory, -> after a while, OOME occurs. TomasZ could have some ideas what could be done/improved in CSS/HTML parsing. Product Version: NetBeans IDE Dev (Build 20120206-429eb201b368) Java: 1.6.0_30; Java HotSpot(TM) 64-Bit Server VM 20.5-b03 System: Linux version 3.2.2 running on amd64; UTF-8; cs_CZ (nb) [1] http://phpexcel.codeplex.com/
*** Bug 207222 has been marked as a duplicate of this bug. ***
To me it seems simply too big file to work with. What I got is: Heap dump file created [915015990 bytes in 31.745 secs] SEVERE [org.openide.util.Exceptions] java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2760) at java.util.Arrays.copyOf(Arrays.java:2734) at java.util.ArrayList.toArray(ArrayList.java:275) at java.util.Collections.sort(Collections.java:158) at org.netbeans.modules.parsing.api.Embedding.create(Embedding.java:149) at org.netbeans.modules.html.editor.gsf.embedding.CssEmbeddingProvider.getEmbeddings(CssEmbeddingProvider.java:116) at org.netbeans.modules.parsing.impl.TaskProcessor.callEmbeddingProvider(TaskProcessor.java:573) at org.netbeans.modules.parsing.impl.SourceCache.refresh(SourceCache.java:328) [catch] at org.netbeans.modules.parsing.impl.TaskProcessor$CompilationJob.run(TaskProcessor.java:714) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:680) which is actually just a creation of css virtual source (parser embedding in the html source).
just some examples: ~2.4M of Css parser tree nodes in 84MB ~2M of Html parser nodes in 65MB ~2.3M of lexer tokens in 55MB ~3M of int[] objects held by Snapshots in 50MB ... PHP -> HTML virtual source (snapshot, tokens, parser) -> css virtual source (snapshot, tokens, parser) there's definitively a room for improvements on both sides (parsing api & css/html) I can do some optimization on the css/html side but still it'd be handy if a file size limit is introduced for the indexing since even with the optimization the OOM will happen on bigger files.
>limit is introduced for the indexing Yes, such a limit is good idea. Unfortunately such a limit is not generic it has to be defined per language.
I doubt this helps, but it still happens in Mountain Lion.
This bug already has 100 duplicates see http://statistics.netbeans.org/exceptions/detail.do?id=184356
changeset: 216691:db08b987a0a2 summary: #207991 - decreasing the SyntaxElement$Tag instance retained size changeset: 216692:e29fbf03c08b summary: #207991 - using CharSequence instead of String where appropriate changeset: 216694:e07f48bf772b summary: providing no folds and structure items for snapshots > 4MB. User should be warned of opening such big files, but it apparently doesn't happen for some php files. This change is considered to be rather temporary to allow to reveal other problems. changeset: 216695:eb94a785996d summary: #207991 - The css parsing.api embedding won't be created for html snapshots bigger than 4MB
Integrated into 'main-golden', will be available in build *201203210400* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main-golden/rev/db08b987a0a2 User: Marek Fukala <mfukala@netbeans.org> Log: #207991 - decreasing the SyntaxElement$Tag instance retained size
Just to add that this happens to me with a grails plugin project. It contains no HTML or PHP.
...and no large files. In fact, a very small project.
I've looked at duplicates and it seems that there are lots of completely different problems automatically marked as duplicate reports. This issue and original reports were about memory consumption of the CSS and HTML. The Groovy reports belong to groovy, etc. Unfortunately all these reports were added by exception reporter into this one. The reports need to be split case by case to individual modules, lots of them are already fixed in dev especially the java related.
Integrated into 'main-golden', will be available in build *201203220400* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main-golden/rev/e5e104c201af User: Marek Fukala <mfukala@netbeans.org> Log: #207991 - 9MB saving on AstNode class optimalization (for the big testing html file)
added issue #210133 for the grails project freezes
changeset: 218458:4733d29d47b4 summary: #207991 - lowering the retained size of the plain syntax elements by tens megabytes changeset: 218468:30397e1e1201 summary: #207991 - great memory save - using html element's iteration instead of passing elements collections. Both changes improved the memory usage for the sample 9MB file by I dare to say several hundreds of megabytes. Now even the editing is possible including indentation and code completion.
I forgot to list an earlier change - some redesign in the html.editor.lib module allowed to save quite a lot of memory as well: http://hg.netbeans.org/web-main/rev/1af99f09e439
Integrated into 'main-golden', will be available in build *201204030400* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main-golden/rev/4733d29d47b4 User: Marek Fukala <mfukala@netbeans.org> Log: #207991 - lowering the retained size of the plain syntax elements by tens megabytes
Just FYI, I've been using dev 7.2 for a week now with the same project that was crashing 7.1... It's been working great!
The infrastructure as well as several indexers affecting indexing performance were rewritten in NB 7.2, see http://wiki.netbeans.org/IndexingPerformance72.
changeset: 218478:2d38129a461f summary: #207991 - do not hold html.editor.lib elements changeset: 218477:59d24350ccdf summary: #207991 - enable/disable html error check don't hold snapshots, but file and document instead changeset: 218610:74a5b9a2b17e summary: #207991 - using a java.io.Reader for CharSequence instead of cloning the source into a new String object in html5 parser. Also masking of the content hidden to the parser is done via a special masking CharSequence instead of wiping the ares in a new StringBuilder now the file keeps at ~280MB of heap during editing. Still there's a memory leak when the file is close and reopened - some of the parser results are held twice.
As the issue had lots of wrong duplicates not related to the html/css. Here is a list of fixes in the generic and non html/css indexers: #206062:Navigator should provide data for file during scan 41985eb45913 #182653: Transactional Index 41985eb45913 #208097: Update CSL languages to support transactional index 8c30d396959b #206059: Filter unrelated BinaryIndexers d00d64d8b0ad #206057, #206054: minimize IO operations in Java during up to date check 2f9cd7092a7e, 9a9337ad3afc #206024: Fix slow project queries 263679ed6809, ee4e39fada10, 38c1bcc2f56b #206021: Speed up file content loading for embed able languages e0ee7f0f0f1d #206020: Run BinaryIndexers in parallel c0b7423861a8 #206017: Prefetch java source files during indexing 1687f3d12679
(In reply to comment #29) > Just FYI, I've been using dev 7.2 for a week now with the same project that was > crashing 7.1... > > It's been working great! Work for me too without problem
One more mostly related to PHP projects with symbolic links in projects: #208392: Scanning project with multiple symlinks does not end if placed in symlinked folder eab0e72268c9
The fixes sound promising, waiting for an os x binary release.
changeset: 218769:f9adfc591c0a summary: #207991 - HtmlNavigationSideBar element's don't hold parser result via MouseListener. That was a problem since when an opened html file is closed an instance of NavigationSideBar still exists and cannot be GCed (looks like another bug in editor) filed a new editor/spellchecker bug 210669 - memory leak: QuietEditorPane -> TokenList
Most of the html/css problems has been fixed. According to tzezula's statement the other problematic areas have been addressed as well so lets consider the issue as fixed.
Integrated into 'main-golden', will be available in build *201204040400* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main-golden/rev/74a5b9a2b17e User: Marek Fukala <mfukala@netbeans.org> Log: #207991 - using a java.io.Reader for CharSequence instead of cloning the source into a new String object in html5 parser. Also masking of the content hidden to the parser is done via a special masking CharSequence instead of wiping the ares in a new StringBuilder
Integrated into 'main-golden', will be available in build *201204050400* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main-golden/rev/f9adfc591c0a User: Marek Fukala <mfukala@netbeans.org> Log: #207991 - HtmlNavigationSideBar element's don't hold parser result via MouseListener. That was a problem since when an opened html file is closed an instance of NavigationSideBar still exists and cannot be GCed (looks like another bug in editor)
*** Bug 209749 has been marked as a duplicate of this bug. ***
*** Bug 209704 has been marked as a duplicate of this bug. ***
*** Bug 209657 has been marked as a duplicate of this bug. ***
*** Bug 209539 has been marked as a duplicate of this bug. ***
*** Bug 207535 has been marked as a duplicate of this bug. ***
*** Bug 200246 has been marked as a duplicate of this bug. ***
*** Bug 204525 has been marked as a duplicate of this bug. ***
*** Bug 205603 has been marked as a duplicate of this bug. ***
*** Bug 211770 has been marked as a duplicate of this bug. ***
*** Bug 207837 has been marked as a duplicate of this bug. ***
*** Bug 212207 has been marked as a duplicate of this bug. ***
*** Bug 212740 has been marked as a duplicate of this bug. ***
*** Bug 209270 has been marked as a duplicate of this bug. ***
*** Bug 215385 has been marked as a duplicate of this bug. ***
*** Bug 214473 has been marked as a duplicate of this bug. ***
*** Bug 220253 has been marked as a duplicate of this bug. ***