Bug 152213 - Large header files cause massive performance problem
Large header files cause massive performance problem
Status: RESOLVED FIXED
Product: cnd
Classification: Unclassified
Component: Code Completion
6.x
All Windows XP
: P3 (vote)
: 6.x
Assigned To: Alexander Simon
issues@cnd
: PERFORMANCE
Depends on: 152491 152494 156583 157111 157229 168651
Blocks:
  Show dependency treegraph
 
Reported: 2008-11-03 17:20 UTC by fse
Modified: 2009-10-22 23:44 UTC (History)
1 user (show)

See Also:
Issue Type: DEFECT
:


Attachments
Header file for SolidWorks COM-Client (785.12 KB, application/x-compressed)
2008-11-05 17:42 UTC, fse
Details
biggest objects when 300K line file is opened in editor (1.94 MB, text/html)
2009-01-23 09:26 UTC, Alexander Simon
Details

Note You need to log in before you can comment on or make changes to this bug.
Description fse 2008-11-03 17:20:33 UTC
I have a very large, machine generated header file (generated by Microsoft's MIDL compiler out of a COM IDL file); size
ist about 12 MB, 300000 lines.
Menue item "Navigate/Go to Declaration" into this file produces a warning "File seems to be too large ...", and this is
true: if you try to open it anyway netbeans crashes. Even if you don't open the file the performace is very bad,
Netbeans needs 100% load very often, and the editor does not response then. Parsing the project needs very long time.

Microsoft Visual Studio has no problems ...
Comment 1 Leonid Lenyashin 2008-11-03 19:52:30 UTC
Ideally we should have this fixed by patch_1.
Comment 2 Alexander Simon 2008-11-05 15:54:41 UTC
please, attach zipped the header file.
Or steps how produce the header file.
Comment 3 fse 2008-11-05 17:42:59 UTC
Created attachment 73316 [details]
Header file for SolidWorks COM-Client
Comment 4 fse 2008-11-05 17:58:26 UTC
May be also responsible for the problem is my crude folder structure:

home
 |-NetbeansProject1
 |-NetbeansProject2
 `-src
    |-main.cc
    |-sldworks.h
    |- ...

I. e. I have two Netbeans projects using some same code located in folder src "besides" the project folders; this is
where my big header file is in.
I just figured out that the debugger is not stopping in my main.cc, it seems that Netbeans don't like such a structure?
I will file another issue for that soon.
Comment 5 Alexander Simon 2008-11-06 16:28:23 UTC
It seems
Please,
- provide NB additional memory in file ${NB}/etc/netbeans.conf: "-J-Xmx800M".
- do not open the header in editor.
Is problem gone away?
Also, could you try NB 6.5?
Comment 6 fse 2008-11-11 09:18:35 UTC
I just tried the setting -J-Xmx800M, it seems to be much better: I have only short peaks with 100% CPU load now. My old
setting has been -J-Xmx300M; since the memory toolbar displays about 250/280 MB I thought it to be enough. (But: My
development Notebook has only 1GB RAM, and I have to run SolidWorks (a resource-hungry CAD application) together with
Netbeans ...)
The problem with the crash when opening the file still remains.

I don't want to install Netbeans 6.5 now because I fear it could interfere with my current installation, sorry. I will
install the new version on my computer at home during the next days and try again there.
Comment 7 Alexander Simon 2008-12-08 08:21:56 UTC
Request of another user:

Unfortunately, the project I work on is proprietary and I cannot share any of the source code. I have no performance
issues working on a normal project, but source files that I deal with at work can exceed 50K lines in size.

To use netbeans at my work, I manually increase the maximum memory of the process to 1 GB so that I have no problems
viewing any files or navigating through the code. However, whenever I need to edit a large file I usually revert to
using gvim because the reparse cycles take too long while I'm editing. Also, one notices a similar degradation editing
header files, regardless of their size, that are included in every single source file in the project, because netbeans
appears to be reparsing every file that includes the header file while I edit the header file.  Finally, I periodically,
usually once every other day, need to shut down and restart netbeans because its minimum memory usage after a garbage
collection grows too high. I think this is related to large files as well because memory usage usually only spikes while
editing large files.

Looking at the ticket you referenced in your email, I would say that my issue is either similar or exactly the same as
the issue described in the ticket, but at a smaller scale because he is working with a 300k line header file that is
included in other files, as opposed to the 50K+ line source files that I see in my project.

This scalability issue with large files is the only one I deal with when using cnd. A similar open source project I
haved worked on that is over half the size of the proprietary project I deal with at work but has more reasonably sized
source files, usually never exceeding 1k lines per file, causes no trouble for netbeans at all for either memory usage
or performance.

Let me know if there is  any other useful information I can provide.

Sean Reque
Comment 8 Alexander Simon 2009-01-16 15:56:51 UTC
- use file reference content in mark occurrences reference visitor.
Change set: http://hg.netbeans.org/main?cmd=changeset;node=c1060ab7617c
Comment 9 Quality Engineering 2009-01-23 07:30:11 UTC
Integrated into 'main-golden', will be available in build *200901230201* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/a72b99cdd834
User: Alexander Simon <alexvsimon@netbeans.org>
Log: fixing IZ#152213:Large header files cause massive performance problem
- provide direct access to file declarations by offset and kind
Comment 10 Alexander Simon 2009-01-23 09:26:02 UTC
Created attachment 76175 [details]
biggest objects when 300K line file is opened in editor
Comment 11 Quality Engineering 2009-02-18 10:35:56 UTC
Integrated into 'main-golden', will be available in build *200902180201* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/a1c192fbfe59
User: Alexander Simon <alexvsimon@netbeans.org>
Log: fixing: IZ#152213:Large header files cause massive performance problem
- fix low performance of c/c++ code folding
Comment 12 sreque 2009-03-16 16:57:29 UTC
I checked out the revision a1c192fbfe59 and netbeans cnd had the best performance I have seen to date. I think the
biggest improvement came from the fact that the IDE was no longer re-parsing files automatically, but only when I saved
a file. I was able to edit both large source files (<= 70K lines) and large ubiquitously-included projected header files
with only moderate slowdown. However, after encountering an unrelated bug I upgraded to a later revision, e27585b9d94d,
and performance has again degraded to unusable levels. It seems that cnd is also again re-parsing files automatically,
not waiting until I save them.
Comment 13 Quality Engineering 2009-03-19 20:32:42 UTC
Integrated into 'main-golden', will be available in build *200903191401* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/8a872e037ac9
User: Alexander Simon <alexvsimon@netbeans.org>
Log: fixing IZ#152213:Large header files cause massive performance problem
- increase repase delay on file edit for large files
Comment 14 Quality Engineering 2009-03-21 08:32:26 UTC
Integrated into 'main-golden', will be available in build *200903210201* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/930382d09944
User: Alexander Simon <alexvsimon@netbeans.org>
Log: fixing IZ#152213:Large header files cause massive performance problem
- change algorithm of counting of repase delay
Comment 15 Quality Engineering 2009-03-28 20:37:36 UTC
Integrated into 'main-golden', will be available in build *200903281400* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main-golden/rev/496658d9f689
User: Alexander Simon <alexvsimon@netbeans.org>
Log: fixing IZ#152213:Large header files cause massive performance problem
- remove expensive macro iterator by file
Comment 16 Leonid Lenyashin 2009-04-10 09:36:30 UTC
Looks like the most critical part of the issue has been fixed. Downgrading to P3. Alexander, please consider if we can
close this IZ and open some more specific IZ (may be a few of them).
Comment 17 Quality Engineering 2009-04-22 08:38:30 UTC
Integrated into 'main-golden', will be available in build *200904220201* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main-golden/rev/5b4bfb0ef497
User: Alexander Simon <alexvsimon@netbeans.org>
Log: fixing IZ#152213:Large header files cause massive performance problem
- fix rendering of typedefs (wrong algorithm consumes 15% of parsing time)
Comment 18 Alexander Simon 2009-04-24 13:31:13 UTC
first implementation of "precompiled" header in the change set:
http://hg.netbeans.org/cnd-main/rev/d96cb7fcc78a
- fixed responsiveness problem in following case:
Source file includes "large" header as first included file.
In this case reparsing time of source file do not depend on size of first header file.
Comment 19 Alexander Simon 2009-10-21 11:44:58 UTC
sreque,
now you can forbids reparsing on document changed (see IZ#168651).

IMHO performance is good enough, dynamic reparsing can be forbidding, so I close issue as fixed.
Comment 20 Alexander Simon 2009-10-21 11:45:17 UTC
fixed
Comment 21 Quality Engineering 2009-10-22 23:44:09 UTC
Integrated into 'main-golden', will be available in build *200910221401* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main-golden/rev/58f474b9552a
User: Alexander Simon <alexvsimon@netbeans.org>
Log: fixing IZ#168651 Add an option to stop the dynamic code indexing
fixing IZ#152213 Large header files cause massive performance problem


By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2014, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo