This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
Summary: | [code model] Low performance in the creation of huge projects | ||
---|---|---|---|
Product: | cnd | Reporter: | rmartins <rmartins> |
Component: | Code Model | Assignee: | Alexander Simon <alexvsimon> |
Status: | RESOLVED FIXED | ||
Severity: | blocker | Keywords: | PERFORMANCE |
Priority: | P2 | ||
Version: | 6.x | ||
Hardware: | All | ||
OS: | All | ||
Issue Type: | DEFECT | Exception Reporter: |
Description
rmartins
2009-09-10 14:53:32 UTC
Hi, I tested build artifact #1625 and the parsing seems better, but the problem still lays in the analyzing part. It's still terrible slow if compared to the actual parsing process. Is there any work planned for optimizing the analyzing part? Thanks, Rolando Hi Rolando I want to clarify current state of performance for 4-core computer on IDE with 1G xms memory: - creating project and analyzing: 15 minutes - parsing, fixing includes and second parsing: 25 minutes Close IDE, clear cache and restart IDE: - reparsing time: 7 minutes CND has target for NB6.8: Reparsing for 5 minutes in 1G xms memory. We did a lot of changes to achieve this target. The initial creating time improvement is not a target for NB6.8. But we improved it also. Unfortunately we cannot improve analyzing step because 80% of analyzing time is reading object file in native method. I see one way to improve analyzing time: use Sun Studio compilers. In real we can only improve double parsing on creating project. Alexander It is not quite clear to me how Sun Studio compilers help to speed up the initial phase. Alexander, can you please elaborate? Integrated into 'main-golden', will be available in build *200909111401* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main-golden/rev/0e88db90c4e7 User: Alexander Simon <alexvsimon@netbeans.org> Log: fixing: IZ#171937 [code model] Low performance in the creation of huge projects - fixed regression in "file exists" Hi Alexander, thanks for the effort and clarification. "- creating project and analyzing: 15 minutes - parsing, fixing includes and second parsing: 25 minutes " So the total time for a fresh project is 40 minutes. If so, that's my main concern (and Xref performance), because when I used Eclipse it would take, if I am not mistaken, about 8 minutes for a fresh project creation. But there is a major difference, eclipse's used compilation output (but I think they also have support for dwarf discovery) and I am not counting the compilation time. I am mention this for a fair comparison; if we CND used output parsing would the results be better? Thanks again and keep the good work, Rolando I also intrigued about how Sun's compilers would improve the analyzing... >how Sun Studio compilers help to speed up the initial phase.
Sun Studio compiler adds command line in the debug info of compilation unit.
So IDE does not need to read other debug information.
Hi Alexander, what about gcc dwarf-3, does it do the same as Sun's compiler? If not, what about output parsing? Would that be better in terms of performance? Or is there any way to improve this on Linux using GCC? Thanks, Rolando Hi Rolando, >what about gcc dwarf-3, does it do the same as Sun's compiler? No, Sun Compiler produces dwarf-2 (that does not allow to find macros). Compiler line is an Sun-specific attribute of compilation unit. I did not investigate dwarf-3 yet. >If not, what about output parsing? Would that be better in terms of performance? Yes it is better, but unfortunately, log of most open source projects have an overcomplicated output. For example: /bin/bash ../libtool --tag=CXX --mode=compile CC -DHAVE_CONFIG_H -I../.. -I.. -DACE_BUILD_DLL -D_REENTRANT -D_THREAD_SAFE -I/usr/openwin/include -g -O -c -o libACE_la-UPIPE_Stream.lo `test -f 'UPIPE_Stream.cpp' || echo '../../ace/'`UPIPE_Stream.cpp IDE can only guess that it is an invoking CC compiler for "./../ace/UPIPE_Stream.cpp" source file. Thanks, Alexander Hi Alexander, thanks for the excellent explanation! I had put an issue regarding the discovery on the GNU makefiles, but the problem really lays in libtool complex output. I will post an request for enhancement. Do you agree? (Despite the work, I think is feasible to "decode" the libtool's output) Thanks, Rolando Hi, I tried playing with libtool and found out that if we use --debug we get: ... + command=' g++ -DHAVE_CONFIG_H -I../.. -I.. -DACE_BUILD_DLL -W -Wall -Wpointer-arith -g3 -gdwarf-2 -pthread -pipe -O3 -MT libACE_la-Based_Pointer_Repository.lo -MD -MP -MF .deps/libACE_la-Based_Pointer_Repository.Tpo -c ../../ace/Based_Pointer_Repository.cpp -fPIC -DPIC -o .libs/libACE_la-Based_Pointer_Repository.o' + rm -f .libs/libACE_la-Based_Pointer_Repository.o '' ... So it's possible to get the "final" shell output. Rolando Integrated into 'main-golden', will be available in build *200909181401* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main-golden/rev/b50930ba4039 User: Alexander Simon <alexvsimon@netbeans.org> Log: fixing: IZ#171937 [code model] Low performance in the creation of huge projects - fixed memory leak Please evaluate this issue Integrated into 'main-golden', will be available in build *200910100201* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main-golden/rev/9c25fe445851 User: Alexander Simon <alexvsimon@netbeans.org> Log: fixing IZ#171937 [code model] Low performance in the creation of huge projects - fix performance problem of discovery by model Hi Alexander, is there anyway to improve parsing speed by using "native methods" on your ANTLR implementation and others subsystems that the parser uses? Even with small projects the re-parsing of a file and its dependencies is slow on a dual-core... Thanks, Rolando Improved analyzing stage, change set: http://hg.netbeans.org/cnd-main/rev/14bfac6d7b50 - add buffering in the random access file - release file handlers if exception in the constructor - improve grep file Cumulative improvements of creating project on 4-core computer: Was: CPU time 32:57, real time 30 minutes Now: CPU time 27:53, real time 20 minutes So it is 33% of improvements. Hi Alexander, excellent work! BTW, I saw that your RandomAcessFile wrapper uses 8k buffers. Could we have a switch (a property, -J-D) for configuring this? The balance between buffer size and syscall count is always a very os (FS) dependent thing... What do you think about this? Thanks, Rolando option in change set: http://hg.netbeans.org/cnd-main/rev/16df1d44baf3 Integrated into 'main-golden', will be available in build *200910290252* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main-golden/rev/14bfac6d7b50 User: Alexander Simon <alexvsimon@netbeans.org> Log: fixing IZ#171937 [code model] Low performance in the creation of huge projects Integrated into 'main-golden', will be available in build *200910300201* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main-golden/rev/16df1d44baf3 User: Alexander Simon <alexvsimon@netbeans.org> Log: fixing IZ#171937 [code model] Low performance in the creation of huge projects - user request for buffer size no objections, mark as 6.8_WAIVER_APPROVED Added double buffering for random access file. It should improve performance of analyzing phase on 25%. Change set: http://hg.netbeans.org/cnd-main?cmd=changeset;node=24cc02b76293 Integrated into 'main-golden', will be available in build *200912230201* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main/rev/693f0e1c1e42 User: Alexander Simon <alexvsimon@netbeans.org> Log: fixing BZ#171937 [code model] Low performance in the creation of huge projects - improve performance of grep source Original issue was fixed. Other improvements are tracked in the issue #178655 [code model] Low performance in the creation of huge projects |