This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 226849 - Add feature to disable auto project parsing, and add a toolbar + menu item for manual initiating project parsing
Summary: Add feature to disable auto project parsing, and add a toolbar + menu item fo...
Status: NEW
Alias: None
Product: cnd
Classification: Unclassified
Component: -- Other -- (show other bugs)
Version: 7.3
Hardware: PC Linux
: P2 normal with 1 vote (vote)
Assignee: Vladimir Voskresensky
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-02-28 15:50 UTC by david_allen
Modified: 2013-03-04 08:46 UTC (History)
0 users

See Also:
Issue Type: ENHANCEMENT
Exception Reporter:

Attachments
messages.log file of a clean parsing run (144.80 KB, text/plain)
2013-03-01 13:35 UTC, david_allen 		Details
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description david_allen 2013-02-28 15:50:47 UTC 
I have a fairly large C++ project group, and the project parsing consumes huge amounts of CPU. (machine is an i7 quad-core, 8gb ram).

When modifying code the project parsing starts in response to the saved file, and can result in Netbeans "locking-up" so you can no longer type your code.  This is a major productivity drawback and reminds me of MS Visual Studio Intellisense woes :(

Also when compiling my code (via makefiles) it will generate more C++ files, which can result in the Netbeans project parsing to start, and slow everything down. To avoid this I have to close Netbeans before a major compile, which is not convenient.

Simple solution : Add ability to disable auto project parsing. Add a menuitem/toolbar for manual initiation of project parsing.

This will allow you to code without interruption, and then rescan when you go get a cup of tea etc..

Note: Their is an option under "Tools>Options>Miscellaneous>Files" called "Enable auto-scanning of sources". I have unticked this but it doesnt make any difference. 

Note: I hoped the parsing performance would improve in 7.3, but I have not noticed an improvement - almost seems worse !

...If we cant solve the performance at least, allow us to choose when we take the CPU hit.
Comment 1 Vladimir Voskresensky 2013-02-28 17:04:56 UTC 
to already disabled
"Tools>Options>Miscellaneous>Files"->"Enable auto-scanning of sources"
Try additionally:
Tools->C/C++->Other and uncheck "Reparse on file change"
Later on use in project context menu: Code Assistance->Reparse

Does it solve your use case?
Comment 2 Leonid Lenyashin 2013-02-28 17:08:45 UTC 
David, thank you very much for reporting this issue. The 7.3 is supposed to handle well the build-generated sources and consume less memory which in turn gives a performance boost if memory is the bottle neck (frequent case for large projects).
It sounds like the improvements did not help you. May I suggest we reclassify your request as a defect - performance issue/high cpu consumtion - and follow up with you asking for more details and maybe profiles and snapshots? I really want to solve the real issue rather than provide a workaround.
Going back to your request: there is something not exactly what you asking for but close. If you right click on project in the context menu you will find "Code Assistance" item and under it you can Enable/Disable Code Assistance as well as trigger Reparse explicitly.
I'd also suggest you make sure you gave JVM enough heap space. If it is not the case then most likely you should see a warning in the bottom-right corner. If it is there it will explain how to change the settings. 
Regards,
Leonid.
Comment 3 Vladimir Voskresensky 2013-03-01 11:36:26 UTC 
David, could you run netbeans with extra params:
#netbeans -J-Dcnd.modelimpl.timing=true -J-Xmx4G

Open your project, wait till parsed and attach your message.log file here.

Thanks!
Vladimir.
Comment 4 Vladimir Voskresensky 2013-03-01 11:40:54 UTC 
I'd like to ask you start NB with clean cache dir (can be found in Help->About) to have full parse.
message.log files (multiple) can be found in user dir (again Help->About can show folder)
${user_dir}/var/log/
Comment 5 david_allen 2013-03-01 11:47:19 UTC 
(In reply to comment #1)
> to already disabled
> "Tools>Options>Miscellaneous>Files"->"Enable auto-scanning of sources"
> Try additionally:
> Tools->C/C++->Other and uncheck "Reparse on file change"
> Later on use in project context menu: Code Assistance->Reparse
> 
> Does it solve your use case?

I hadn't spotted this before. Had a quick try of it but it does not seem practical for me, because there are probably 40 projects in my project group, and to click Reparse on them individually would be painful - ideally need a "Reparse Project Group" button.  In addition that Reparse seemed to do a full reparse judging by the time it took. Ideally I wanted a "Reparse Project Group" but only for files that have changed since last reparse i.e. incremental.
Comment 6 soldatov 2013-03-01 11:52:06 UTC 
Raparse menu item reparse all selected projects. Am I wrong?
Comment 7 david_allen 2013-03-01 12:00:26 UTC 
(In reply to comment #2)
> David, thank you very much for reporting this issue. The 7.3 is supposed to
> handle well the build-generated sources and consume less memory which in turn
> gives a performance boost if memory is the bottle neck (frequent case for large
> projects).
> It sounds like the improvements did not help you. May I suggest we reclassify
> your request as a defect - performance issue/high cpu consumtion - and follow
> up with you asking for more details and maybe profiles and snapshots? I really
> want to solve the real issue rather than provide a workaround.
> Going back to your request: there is something not exactly what you asking for
> but close. If you right click on project in the context menu you will find
> "Code Assistance" item and under it you can Enable/Disable Code Assistance as
> well as trigger Reparse explicitly.
> I'd also suggest you make sure you gave JVM enough heap space. If it is not the
> case then most likely you should see a warning in the bottom-right corner. If
> it is there it will explain how to change the settings. 
> Regards,
> Leonid.

My netbeans.conf modification is as below :

# The original options
#netbeans_default_options="-J-client -J-Xss2m -J-Xms32m -J-XX:PermSize=32m
-J-Dapple.laf.useScreenMenuBar=true -J-Dapple.awt.graphics.UseQuartz=true
-J-Dsun.java2d.noddraw=true -J-Dsun.java2d.dpiaware=true
-J-Dsun.zip.disableMemoryMapping=true -J-Dsun.awt.disableMixing=true"

# My new options
netbeans_default_options="-J-client -J-Xss2m -J-Xms256m -J-Xmx512m
-J-XX:PermSize=256m -J-Xverify:none -J-Dapple.laf.useScreenMenuBar=true
-J-Dapple.awt.graphics.UseQuartz=true -J-Dsun.java2d.noddraw=true
-J-Dsun.zip.disableMemoryMapping=true -J-Dsun.awt.disableMixing=true"

Thanks for tip on Enable/Disable Code Assistance.  In my prior comment I have
detailed why this doesn't seem practical with 40 projects, and really it needs
to be a Disable *Automatic* Code Assistance reparsing.

I agree on trying to fix root cause i.e. high demands of parser, however this
assumes everyones PC can handle what is deemed "acceptable" demands of parser,
and also assumes an acceptable amount of source code. Ideally we want Netbeans
to be able to handle large enterprise scale.  Its seems pragmatic to give
control of the parser to the user so they can find a suitable time to reparse
(just their recent code changes).
Comment 8 Vladimir Voskresensky 2013-03-01 12:20:01 UTC 
-J-Xmx512m is limiting memory up-to 0.5 Gb. For 40 projects it can be low. Btw, you should see yellow warning in status bar saying about not enough memory.
Try what was suggested before: 
instead of -J-Xmx512m use -J-Xmx4Gb or -J-Xmx2Gb
Comment 9 soldatov 2013-03-01 12:20:37 UTC 
Do you speak about 64-bit java and -J-Xmx512m?
Comment 10 david_allen 2013-03-01 13:18:15 UTC 
(In reply to comment #9)
> Do you speak about 64-bit java and -J-Xmx512m?

It is a linux fedora 17 64-bit OS with java details :

$ java -version
java version "1.7.0_09-icedtea"
OpenJDK Runtime Environment (fedora-2.3.4.fc17-x86_64)
OpenJDK 64-Bit Server VM (build 23.2-b09, mixed mode)

As you mention using Xmx of 1Gb and upwards results in error :

"Invalid maximum heap size: -Xmx1Gb
Error: Could not create the Java Virtual Machine."
Comment 11 david_allen 2013-03-01 13:35:06 UTC 
Created attachment 132066 [details]
messages.log file of a clean parsing run

As advised, I cleaned the cache and started netbeans with "-J-Dcnd.modelimpl.timing=true"
Comment 12 david_allen 2013-03-01 13:37:25 UTC 
(In reply to comment #4)
> I'd like to ask you start NB with clean cache dir (can be found in Help->About)
> to have full parse.
> message.log files (multiple) can be found in user dir (again Help->About can
> show folder)
> ${user_dir}/var/log/

Thanks for the help/advice Vladimir. I have attached a messages.log file for the full parsing run (cache cleaned prior to run).  Note: I removed the prior messages.log so this log file should be just for the single parsing run.

I noticed there were some exception messages in the log - not sure if important, though.
Comment 13 Vladimir Voskresensky 2013-03-01 15:46:30 UTC 
Let's try again (sorry, I incorrectly told you about "Gb")
clean cache dir, use -J-Xmx4G => you should see much faster parsing.

then clean cache dir again and try reduce to -J-Xmx2G

Looking for the result
Comment 14 david_allen 2013-03-01 17:23:45 UTC 
(In reply to comment #13)
> Let's try again (sorry, I incorrectly told you about "Gb")
> clean cache dir, use -J-Xmx4G => you should see much faster parsing.
> 
> then clean cache dir again and try reduce to -J-Xmx2G
> 
> Looking for the result

Thanks Vladimir - tried that and I think it is running a bit smoother.
Comment 15 Leonid Lenyashin 2013-03-01 19:40:45 UTC 
Vladimir, I think it is worth trying to limit number of threads to 3, and if it still produce 100% cpu load and block any productive work, then it has to be filed as a separate bug.
I'm also curious why builds are causing reparse, while they should not. Another bug?
Comment 16 Leonid Lenyashin 2013-03-01 19:45:48 UTC 
(In reply to comment #7)

> 
> Thanks for tip on Enable/Disable Code Assistance.  In my prior comment I have
> detailed why this doesn't seem practical with 40 projects, and really it needs
> to be a Disable *Automatic* Code Assistance reparsing.
Yes, I understand it is not what you are looking for, but still...
> 
> I agree on trying to fix root cause i.e. high demands of parser, however this
> assumes everyones PC can handle what is deemed "acceptable" demands of parser,
> and also assumes an acceptable amount of source code. Ideally we want Netbeans
> to be able to handle large enterprise scale.  Its seems pragmatic to give
> control of the parser to the user so they can find a suitable time to reparse
> (just their recent code changes).
Actually smooth handling of large enterprise apps is our main goal at the moment. And we do a good job with our test apps and some customers apps which are enormous in size. But in your case something does not work, so I do want to know why and fix it.
Honestly I do not like to invest into the "pragmatic" approach. I do want to stay "romantic" and make IDE working smoothly over your code base w/o been manually managed by you and your colleagues (we do productivity tools, and your productivity is not measured by how efficiently you manage Code Assistance, right?)
Comment 17 Vladimir Voskresensky 2013-03-04 08:46:32 UTC 
(In reply to comment #14)
> (In reply to comment #13)
> > Let's try again (sorry, I incorrectly told you about "Gb")
> > clean cache dir, use -J-Xmx4G => you should see much faster parsing.
> > 
> > then clean cache dir again and try reduce to -J-Xmx2G
> > 
> > Looking for the result
> 
> Thanks Vladimir - tried that and I think it is running a bit smoother.
Could you share more details, please :-)
Could you attach message.log (clean userdir, full parse) after giving IDE -J-Xmx2G 

From original message.log I see that "Lack of memory" was detected after 30 sec. I assume notification was shown, right?

Thanks!
Vladimir.