Bug 78222

Summary: *Code Model* Decrease memory usage
Product: cnd Reporter: Vladimir Kvashin <vkvashin>
Component: Code ModelAssignee: Egor Ushakov <gorrus>
Status: RESOLVED FIXED QA Contact: issues <issues.netbeans.org>
Priority: P1 CC: issues
Version: 5.xKeywords: PERFORMANCE
Target Milestone: 6.x   
Hardware: All   
OS: All   
Whiteboard: CND1-FCS parser
Issue Type: DEFECT Exception Report:

Description Vladimir Kvashin 2006-06-19 07:41:05 UTC
Now, code model implementation uses a lot of memory. For example, mysql project
needs about 550 Mb to be completely parsed. This is too much. The amount of
memory used be code model should be decreased.
Comment 1 Vladimir Voskresensky 2006-09-05 15:52:38 UTC
Some work was done trying to fix
http://www.netbeans.org/issues/show_bug.cgi?id=83993:

several prototypes were added to reduce the amount of used memory:
1) APT related:
- after using token stream from APT we dispose unnecessary tokens and leave only
APT structure (this gives us reducing from 51 to 46Mb - 10%)
- responsible flag is "apt.dispose.token" 
- this approach is not safe, because by design APT supposed to be immutable, but
now we parse only once => it is OK
2) String objects related:
- use shared string Map for all APT tokens and CsmAST objects
- this costs reducing in memory from 51 to 36Mb - 30%
- responsible flag is "apt.share.text"

So, as result we reduced by about 40% and I turned on "apt.share.text" (30%
reducing) flag.

What can be done additionally:
 - we can merge or even remove all non-structure responsible APT nodes from APT
tree, the same note as above - APT is immutable
 - we can remove fake AST part, because now AST structure takes huge memory and
it hanges in lazy parts of model
Comment 2 Vladimir Voskresensky 2006-09-06 15:27:53 UTC
*** Issue 83993 has been marked as a duplicate of this issue. ***
Comment 3 Egor Ushakov 2006-10-12 13:09:11 UTC
*** Issue 84512 has been marked as a duplicate of this issue. ***
Comment 4 Egor Ushakov 2006-10-12 14:00:35 UTC
Sevral things done to achieve current memory usage of 490Mb for MySQL:
1. Usage of APTLight (Made by VladimirV)
2. Removal of unneded endOffset field from APTConstTextToken (gives ~ 5%)
3. Usage of empty Tokens for comments
4. No copy of macro maps, just creating a tree of snapshots (gives ~30%)
5. Store compound statement bodies as token list, not AST (gives ~15%)
Comment 5 _ rkubacki 2006-10-12 16:34:53 UTC
Since this is dup of P1 bug this should be P1 too.

May I ask how did you verify your improvements?

Do you really think that 550MB -> 490MB improvement mean that this is resolved
problem?

Comment about APTStringManager: this class is broken. You use WeakHashMap where
only key is weakly referenced
(file:///usr/local/jdk1.5.0_docs/docs/api/java/util/WeakHashMap.html) and
because you use the same object as a key and and value your key is always
strongly referenced. So I guess it does not work as you expect. The
synchronization also seems to be suspicious as the map content can be mutated in
get() too (GC thread can work at any time though it you couldn't notice so far).
Comment 6 Egor Ushakov 2006-10-13 09:05:00 UTC
Sorry, I have forgotten to write initial metrics:
when I started this issue, MySQL were eating 1 Gb of memory.
So we have improved this number to 490Mb.

Vladimir Kvashin - the reporter of the issue, said that number of 550 Mb was
measured on the MySQL which was not fully configured and so not all files were
parsed. He agreed with the current initial number of 1Gm.
So I'm closing this issue.

Also thank you for comments on the APTStringManager implementation, I've
forwarded them to the developer responsible for this class.
By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo