This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
Hi, Despite the huge performance boost present in 6.8, it's not sufficient for huge projects (specially on non high-end machines). The reference project is ACE+TAO: (http://download.dre.vanderbilt.edu/previous_versions/ACE+TAO-5.6.8.tar.bz2). Here are the steps that I took: 1. mkdir build (in ACE_TAO main directory) 2. cd build 3. ../configure CFLAGS="-g3 -gdwarf-2" CXXFLAGS="-g3 -gdwarf-2" --enable-ace-reactor-notification-queue After creating the project in Netbeans (creating using existing source files), it still takes a very long time to finished and the memory consumption is too high. Citation from Alex (6.8): "Best results was 3 weeks ago: - parsing in 1G on 32 bit JVM on 4 core computer - astronomical time: 7 min - CPU time: 19 min Possible times were changed due to code model fixes ;-) " My hopes are that the target time for the next milestone would be : - 1Gb on 32bit JVM on a dual core computer - CPU time: 5-7 minutes (to match the competition) Thanks, Rolando
Hi, I tested build artifact #1625 and the parsing seems better, but the problem still lays in the analyzing part. It's still terrible slow if compared to the actual parsing process. Is there any work planned for optimizing the analyzing part? Thanks, Rolando
Hi Rolando I want to clarify current state of performance for 4-core computer on IDE with 1G xms memory: - creating project and analyzing: 15 minutes - parsing, fixing includes and second parsing: 25 minutes Close IDE, clear cache and restart IDE: - reparsing time: 7 minutes CND has target for NB6.8: Reparsing for 5 minutes in 1G xms memory. We did a lot of changes to achieve this target. The initial creating time improvement is not a target for NB6.8. But we improved it also. Unfortunately we cannot improve analyzing step because 80% of analyzing time is reading object file in native method. I see one way to improve analyzing time: use Sun Studio compilers. In real we can only improve double parsing on creating project. Alexander
It is not quite clear to me how Sun Studio compilers help to speed up the initial phase. Alexander, can you please elaborate?
Integrated into 'main-golden', will be available in build *200909111401* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main-golden/rev/0e88db90c4e7 User: Alexander Simon <alexvsimon@netbeans.org> Log: fixing: IZ#171937 [code model] Low performance in the creation of huge projects - fixed regression in "file exists"
Hi Alexander, thanks for the effort and clarification. "- creating project and analyzing: 15 minutes - parsing, fixing includes and second parsing: 25 minutes " So the total time for a fresh project is 40 minutes. If so, that's my main concern (and Xref performance), because when I used Eclipse it would take, if I am not mistaken, about 8 minutes for a fresh project creation. But there is a major difference, eclipse's used compilation output (but I think they also have support for dwarf discovery) and I am not counting the compilation time. I am mention this for a fair comparison; if we CND used output parsing would the results be better? Thanks again and keep the good work, Rolando I also intrigued about how Sun's compilers would improve the analyzing...
>how Sun Studio compilers help to speed up the initial phase. Sun Studio compiler adds command line in the debug info of compilation unit. So IDE does not need to read other debug information.
Hi Alexander, what about gcc dwarf-3, does it do the same as Sun's compiler? If not, what about output parsing? Would that be better in terms of performance? Or is there any way to improve this on Linux using GCC? Thanks, Rolando
Hi Rolando, >what about gcc dwarf-3, does it do the same as Sun's compiler? No, Sun Compiler produces dwarf-2 (that does not allow to find macros). Compiler line is an Sun-specific attribute of compilation unit. I did not investigate dwarf-3 yet. >If not, what about output parsing? Would that be better in terms of performance? Yes it is better, but unfortunately, log of most open source projects have an overcomplicated output. For example: /bin/bash ../libtool --tag=CXX --mode=compile CC -DHAVE_CONFIG_H -I../.. -I.. -DACE_BUILD_DLL -D_REENTRANT -D_THREAD_SAFE -I/usr/openwin/include -g -O -c -o libACE_la-UPIPE_Stream.lo `test -f 'UPIPE_Stream.cpp' || echo '../../ace/'`UPIPE_Stream.cpp IDE can only guess that it is an invoking CC compiler for "./../ace/UPIPE_Stream.cpp" source file. Thanks, Alexander
Hi Alexander, thanks for the excellent explanation! I had put an issue regarding the discovery on the GNU makefiles, but the problem really lays in libtool complex output. I will post an request for enhancement. Do you agree? (Despite the work, I think is feasible to "decode" the libtool's output) Thanks, Rolando
Hi, I tried playing with libtool and found out that if we use --debug we get: ... + command=' g++ -DHAVE_CONFIG_H -I../.. -I.. -DACE_BUILD_DLL -W -Wall -Wpointer-arith -g3 -gdwarf-2 -pthread -pipe -O3 -MT libACE_la-Based_Pointer_Repository.lo -MD -MP -MF .deps/libACE_la-Based_Pointer_Repository.Tpo -c ../../ace/Based_Pointer_Repository.cpp -fPIC -DPIC -o .libs/libACE_la-Based_Pointer_Repository.o' + rm -f .libs/libACE_la-Based_Pointer_Repository.o '' ... So it's possible to get the "final" shell output. Rolando
Integrated into 'main-golden', will be available in build *200909181401* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main-golden/rev/b50930ba4039 User: Alexander Simon <alexvsimon@netbeans.org> Log: fixing: IZ#171937 [code model] Low performance in the creation of huge projects - fixed memory leak
Please evaluate this issue
Integrated into 'main-golden', will be available in build *200910100201* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main-golden/rev/9c25fe445851 User: Alexander Simon <alexvsimon@netbeans.org> Log: fixing IZ#171937 [code model] Low performance in the creation of huge projects - fix performance problem of discovery by model
Hi Alexander, is there anyway to improve parsing speed by using "native methods" on your ANTLR implementation and others subsystems that the parser uses? Even with small projects the re-parsing of a file and its dependencies is slow on a dual-core... Thanks, Rolando
Improved analyzing stage, change set: http://hg.netbeans.org/cnd-main/rev/14bfac6d7b50 - add buffering in the random access file - release file handlers if exception in the constructor - improve grep file Cumulative improvements of creating project on 4-core computer: Was: CPU time 32:57, real time 30 minutes Now: CPU time 27:53, real time 20 minutes So it is 33% of improvements.
Hi Alexander, excellent work! BTW, I saw that your RandomAcessFile wrapper uses 8k buffers. Could we have a switch (a property, -J-D) for configuring this? The balance between buffer size and syscall count is always a very os (FS) dependent thing... What do you think about this? Thanks, Rolando
option in change set: http://hg.netbeans.org/cnd-main/rev/16df1d44baf3
Integrated into 'main-golden', will be available in build *200910290252* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main-golden/rev/14bfac6d7b50 User: Alexander Simon <alexvsimon@netbeans.org> Log: fixing IZ#171937 [code model] Low performance in the creation of huge projects
Integrated into 'main-golden', will be available in build *200910300201* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main-golden/rev/16df1d44baf3 User: Alexander Simon <alexvsimon@netbeans.org> Log: fixing IZ#171937 [code model] Low performance in the creation of huge projects - user request for buffer size
no objections, mark as 6.8_WAIVER_APPROVED
Added double buffering for random access file. It should improve performance of analyzing phase on 25%. Change set: http://hg.netbeans.org/cnd-main?cmd=changeset;node=24cc02b76293
Integrated into 'main-golden', will be available in build *200912230201* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main/rev/693f0e1c1e42 User: Alexander Simon <alexvsimon@netbeans.org> Log: fixing BZ#171937 [code model] Low performance in the creation of huge projects - improve performance of grep source
Original issue was fixed. Other improvements are tracked in the issue #178655 [code model] Low performance in the creation of huge projects