This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
There are files with such extensive macro usage (in Boost for example) that parsing these files consume too much memory. It happens because we tokenize, expand and keep the full file instead of a small portion that we need at the moment.
The fix is to get back optimization we threw away long ago.
fixed in: http://hg.netbeans.org/cnd-main/rev/559d4c029670
Integrated into 'main-golden', will be available in build *201211211016* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main-golden/rev/559d4c029670 User: Egor Ushakov <gorrus@netbeans.org> Log: fixed #222460 - Too much memory used to parse some files
the fix does not work with the current infrastructure that keeps all tokens in AST
Created attachment 128251 [details] proposed fix
Integrated into 'main-golden', will be available in build *201211300001* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main-golden/rev/ee02326e791b User: Egor Ushakov <gorrus@netbeans.org> Log: reverted fix for #222460 - Too much memory used to parse some files: it does not help existing infrastructure
It is a stopper for a studio target. Biggest objects on OOM: +-----------------------------------------------------------------------------------+---------------------+------------------------+------------------------+ | Class Name | Objects | Shallow Size | Retained Size | +-----------------------------------------------------------------------------------+---------------------+------------------------+------------------------+ | +---java.lang.Object[] | 72,105 0 % | 189,871,776 5 % | 3,850,116,392 91 % | | | | | | | | +---java.lang.Thread | 14 0 % | 2,240 0 % | 3,846,986,848 91 % | | | | | | | | +---org.netbeans.modules.cnd.modelimpl.parser.ParserProviderImpl$Antlr2CppParser | 1 0 % | 128 0 % | 3,826,562,744 91 % | | | | | | | | +---org.netbeans.modules.cnd.modelimpl.parser.CPPParserEx | 1 0 % | 232 0 % | 3,826,562,576 91 % | | | | | | | | +---org.netbeans.modules.cnd.antlr.TokenBuffer | 1 0 % | 48 0 % | 3,826,561,784 91 % | | | | | | | | +---java.util.ArrayList | 56,016 0 % | 2,240,640 0 % | 3,774,507,624 90 % | | | | | | | | +---org.netbeans.modules.cnd.apt.impl.support.MacroExpandedToken | 30,637,800 37 % | 1,470,614,400 35 % | 3,567,130,160 85 % | | | | | | | | +---org.netbeans.modules.cnd.apt.impl.support.APTMacroParamExpansion | 43,540,324 53 % | 2,089,935,552 50 % | 2,658,139,080 63 % | | | | | | | +-----------------------------------------------------------------------------------+---------------------+------------------------+------------------------+
limit number of generated tokens for one macro expanded stream as 8M: http://hg.netbeans.org/cnd-main/rev/8968ad1ff1fd http://hg.netbeans.org/cnd-main/rev/1d2257bfe461
Integrated into 'main-silver', will be available in build *201402190001* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main-silver/rev/8968ad1ff1fd User: Vladimir Voskresensky <vv159170@netbeans.org> Log: fixed #222460 - Too much memory used to parse some files - limit generated tokens as 8 millions
extra fix under flag to check memory optimization http://hg.netbeans.org/cnd-main/rev/b55755974792
better place for tokens limitation: http://hg.netbeans.org/cnd-main/rev/0a93d7aafaaa
Integrated into 'main-silver', will be available in build *201402200001* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main-silver/rev/b55755974792 User: Vladimir Voskresensky <vv159170@netbeans.org> Log: extra fix #222460: Too much memory used to parse some files
Integrated into 'releases/release80', will be available in build *201404240045* or newer. Wait for official and publicly available build. Changeset: http://hg.netbeans.org/releases/rev/afd87eaacf38 User: Vladimir Voskresensky <vv159170@netbeans.org> Log: extra fix #222460: Too much memory used to parse some files - move logic into APTMacroExpandedStream (transplanted from 0a93d7aafaaaf91a42aa33c1587b4f8d743fcf56)