This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
Created attachment 148579 [details] Small project to create large memory consumption resulting from macro expansion. Attached is a small project which has only two source files. The header file defines a macro that is 6 levels deep. When attached project is opened and parsed, APT gets into very deep recursion which eventually leads to exhaustion of the heap memory and IDE failure for average PC configurations. Keep in mind that this project is small and consumes more than 2G of memory apparently all related to expand macro capability. Nominal project size for advanced customers using stack libraries etc is often greater than 200 source files. Even with 4G of memory allocated for JVM, the IDE performance drops to its knees and project becomes very cumbersome or impossible to work with. Heap dump for this small project indicates seemingly excessive memory consumption by APTMacroParamExpansion and MacroExpandedToken objects. Bottom line is that the IDE gets a bad reputation for requiring excessive amounts of memory in general. It may occur often in embedded C development related to unique characteristics of some macro declarations. Having macro expansion capability at the expense of IDE performance stability is not a good default I think. If memory consumption for macro processing cannot be reduced to a negligible impact on IDE performance, then my request would be to instrument a means of opting out of this portion of parsing functionality. That is needed in cases where problem manifests on systems where resources are constrained. We need a way to allow customers to continue work and make progress on critical project development. At present, the general experience is to end up dead in the water.
We know this problem. To solve problem IDE has limit of expanded tokens. By default it equals 8,000,000 tokens. It fits about 2G of memory at parsing time or editing file. You can change limit by option. For example: -J-Dapt.limit.tokens=50000 will restrict maximal token stream by 50,000 tokens. Does it fix you problem?
work around: http://hg.netbeans.org/cnd-main/rev/d59f2af74f57 - restrict size of macro parameters size
(In reply to Alexander Simon from comment #1) > We know this problem. > To solve problem IDE has limit of expanded tokens. > By default it equals 8,000,000 tokens. > It fits about 2G of memory at parsing time or editing file. > You can change limit by option. For example: > -J-Dapt.limit.tokens=50000 > will restrict maximal token stream by 50,000 tokens. > > Does it fix you problem? I was curious about that limit and tried it already. It did not deliver result I was hoping for. Problem at a different level within that limit I think.
(In reply to mrellis63 from comment #3) > (In reply to Alexander Simon from comment #1) > > We know this problem. > > To solve problem IDE has limit of expanded tokens. > > By default it equals 8,000,000 tokens. > > It fits about 2G of memory at parsing time or editing file. > > You can change limit by option. For example: > > -J-Dapt.limit.tokens=50000 > > will restrict maximal token stream by 50,000 tokens. > > > > Does it fix you problem? > > I was curious about that limit and tried it already. It did not deliver > result I was hoping for. Problem at a different level within that limit I > think.
(In reply to Alexander Simon from comment #2) > work around: > http://hg.netbeans.org/cnd-main/rev/d59f2af74f57 > - restrict size of macro parameters size Sorry previous response was before I updated web browser to see that a patch was offered. This patch does appear to offer the behavior and memory consumption control I was hoping for. With patch applied, memory consumption on fairly large (200 files) customer project with same macros as attached example used in their code is much lower. Heap memory used threshold now stays below reasonable -Xmx limits for average PC system. Macro expansion view still works too. System responsiveness is much better now since memory does not hit heap limit causing GC to run constantly. Thanks very much for paying attention to the specifics of what I had captured in the bug report and demonstrated with example project. I think this will be a real break through for user experience and perception of IDE performance for C/C++ project development.
Integrated into 'main-silver', will be available in build *201408080001* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main-silver/rev/d59f2af74f57 User: Alexander Simon <alexvsimon@netbeans.org> Log: fixing Bug #246314 APT consumes excessive amount of heap memory for macro expand capability. - work around: restrict size of macro parameters size
(In reply to mrellis63 from comment #5) > With patch applied, memory consumption on fairly large (200 files) customer > project with same macros as attached example used in their code is much > lower. Heap memory used threshold now stays below reasonable -Xmx limits for > average PC system. Macro expansion view still works too. System > responsiveness is much better now since memory does not hit heap limit > causing GC to run constantly. Hi mrellis63, Thank you very much for trying the patch. Alexander