This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
Build: NetBeans IDE 7.4 (Build 201310111528) VM: Java HotSpot(TM) 64-Bit Server VM, 23.25-b01, Java(TM) SE Runtime Environment, 1.7.0_25-b15 OS: Mac OS X User Comments: GUEST: Background scanning of projects caused NetBeans to become slow and unresponsive. ain: Versioning labels were lost, no "[master]" showing up next to HTML5 project despite the respective ticker in the View menu. Hung, had to restart. Seems to be after restart. Stacktrace: java.lang.OutOfMemoryError: Java heap space at java.util.HashMap.newValueIterator(HashMap.java:943) at java.util.HashMap$Values.iterator(HashMap.java:1010) at java.util.AbstractCollection.toArray(AbstractCollection.java:137) at java.util.ArrayList.<init>(ArrayList.java:151) at org.tmatesoft.svn.core.internal.io.svn.ssh.SshSessionPool$1.run(SshSessionPool.java:31) at java.util.TimerThread.mainLoop(TimerThread.java:555)
Created attachment 142633 [details] stacktrace
there are ~3.5M instances of org.netbeans.modules.css.lib.TokenNode. oome happens as a consequence of scanning of a big .scss file. The OOME is easily reproducible: just include the unpacked attached file in the sources of your project and the oome will occur during the scanning.
Created attachment 142639 [details] packed .scss file
*** Bug 239401 has been marked as a duplicate of this bug. ***
This bug already has 5 duplicates see http://statistics.netbeans.org/exceptions/detail.do?id=198069
*** Bug 241817 has been marked as a duplicate of this bug. ***
*** Bug 239822 has been marked as a duplicate of this bug. ***
*** Bug 241259 has been marked as a duplicate of this bug. ***
bit artificial use case, right? attached .scss file has 10 000 000 empty lines
well it is not artificial, because this file is a part of some test framework which seems to be used by all the affected users...
This is link to the large_empty.scss file: https://github.com/hcatlin/sass-spec/blob/master/spec/benchmarks/large_empty.scss
It looks like CSS parser creates token for each empty line (10M tokens). Just wandering what happens if there is for instance PHP file with 10M new lines. As a hot fix for this case, we can just skip scanning of files which are bigger than something...
*** Bug 244669 has been marked as a duplicate of this bug. ***
Created attachment 147915 [details] patch, which ignores files bigger than 1M
I don't know how to fix it properly. Looks like CSS Parser creates token for each new line. In this case it tries to create 10M of tokens and runs out of memory. The only fix I have is to ignore files bigger than some constant (1M?)
I just wanted to chime in a sec and ask, since you marked my bug as a duplicate, even though it isn't with the CSS processors, but just the fact is scanning so many files. This isn't directly related to the CSS processors, but just the background scanning, which I assume uses the tokenNode class? If there are too many files, it just uses up all of its allotted memory, correct?
> I just wanted to chime in a sec and ask, since you marked my bug as a duplicate, even though it isn't with the CSS processors, but just the fact is scanning so many files. > This isn't directly related to the CSS processors, but just the background scanning, which I assume uses the tokenNode class? If there are too many files, it just uses up all of its allotted memory, correct? Please reopen your bug, if you don't think, that it is duplicate.
Jan, I'm simply asking you if you think it is. I'm not familiar with the inner workings on netbeans, so for all I know, it could be if they both are using the tokenNode class.
I've changes the css grammar so it does merge the NL tokens into one. changeset: 275198:b45179127f88 user: Marek Fukala <mfukala@netbeans.org> date: Wed Jul 09 12:03:15 2014 +0200 summary: #238864 - css lexer joins new lines chars into one token Tests proves we can create a TokenHierarchy for 10k of \n css code and iterate via the single fat token. However when I open the file in editor now, the VM breaks into gallop and all it seems to be doing is this: [1]. I've no idea which is going on there, but it doesn't seem to be related to css editor as the same problem happens if you rename it to large_empty.txt, run NB again and open the file. I'm passing the issue to editor for further evaluation related to [1]. [1] --- snippet of the thread dump taken after opening the large_empty.scss: ------------------ "org.openide.text Document Processing" #52 daemon prio=1 os_prio=31 tid=0x00007faa54949000 nid=0xc67b waiting on condition [0x0000000131837000] java.lang.Thread.State: RUNNABLE at org.netbeans.modules.editor.lib2.document.MarkVector.newPosition(MarkVector.java:198) at org.netbeans.modules.editor.lib2.document.MarkVector.createPosition(MarkVector.java:184) at org.netbeans.modules.editor.lib2.document.MarkVector.position(MarkVector.java:171) at org.netbeans.modules.editor.lib2.document.EditorDocumentContent.createPosition(EditorDocumentContent.java:199) - locked <0x0000000785c8d538> (a org.netbeans.modules.editor.lib2.document.EditorDocumentContent) at javax.swing.text.AbstractDocument.createPosition(AbstractDocument.java:829) - locked <0x0000000785c8c968> (a org.netbeans.modules.csl.core.GsfDocument) at org.netbeans.modules.editor.lib2.document.LineRootElement.insertUpdate(LineRootElement.java:121) at org.netbeans.editor.BaseDocument.insertUpdate(BaseDocument.java:1094) at org.netbeans.editor.BaseDocument.handleInsertString(BaseDocument.java:825) at org.netbeans.editor.BaseDocument.insertString(BaseDocument.java:757) at org.netbeans.editor.BaseDocument.read(BaseDocument.java:1452) at org.netbeans.editor.BaseKit.read(BaseKit.java:663) at org.openide.text.DataEditorSupport.loadFromStreamToKit(DataEditorSupport.java:490) at org.openide.text.DocumentOpenClose$DocumentLoad.atomicLockedRun(DocumentOpenClose.java:762) at org.openide.text.DocumentOpenClose$DocumentLoad.run(DocumentOpenClose.java:613) at org.netbeans.editor.GuardedDocument.runAtomic(GuardedDocument.java:321) at org.openide.text.NbDocument.runAtomic(NbDocument.java:436) at org.openide.text.DocumentOpenClose$DocumentLoad.run(DocumentOpenClose.java:653) at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1423) at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:2033)
forgot to mention it's been pushed to web-main
Integrated into 'main-silver', will be available in build *201407100001* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main-silver/rev/b45179127f88 User: Marek Fukala <mfukala@netbeans.org> Log: #238864 - css lexer joins new lines chars into one token
The 10MB of newlines causes NB to throw OOME. Only the char content insertion passes but fixing of LineRootElement (creation of new positions for line elements) fails with OOME which is a bit strange since there should only be about 32 bytes per Mark and 16 bytes per position i.e. 48M taken. I continue investigation (whether e.g. there would not be any parallel thread causing an increased memory consumption).
I can safely open a file with 2MB of newlines on my machine without setting an explicit -Xmx. For each line in the editor there's certain overhead: LineElement 24 bytes (48 bytes on 64-bit JVM when not using compressed pointers) it holds position of line's start so it's EditorPosition 16 bytes (24 bytes on 64-bit) which holds Mark 32 bytes (64 bytes on 64-bit) For each line there's ParagraphView 48 bytes (80 bytes on 64-bit) In addition for plain text there's Syntax$BaseStateInfo 16 bytes (24 on 64-bit) It's roughly 136 bytes per line (240 bytes on 64-bit). (In reply to Marek Fukala from comment #19) > I've changes the css grammar so it does merge the NL tokens into one. > > However when I open the file in editor now, the VM breaks into gallop and > all it seems to be doing is this: [1]. I've found nothing special when opening .scss or .txt file for both the 2MB file passed and 3M failed with OOME. Since we cannot lower the per-line overhead easily without making significant architectural changes I mark this as wontfix.
*** Bug 247720 has been marked as a duplicate of this bug. ***
*** Bug 255408 has been marked as a duplicate of this bug. ***
*** Bug 256232 has been marked as a duplicate of this bug. ***
I just had this error without opening a large file, see http://statistics.netbeans.org/analytics/exception.do?id=858545