This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 134990 - Source path modification initiated from build script starts scanning.
Summary: Source path modification initiated from build script starts scanning.
Status: RESOLVED DUPLICATE of bug 182653
Alias: None
Product: java
Classification: Unclassified
Component: Source (show other bugs)
Version: 6.x
Hardware: All All
: P2 blocker with 34 votes (vote)
Assignee: Tomas Zezula
URL:
Keywords: PERFORMANCE
Depends on: 185457
Blocks:
  Show dependency tree
 
Reported: 2008-05-13 21:57 UTC by jessholle
Modified: 2014-10-31 07:43 UTC (History)
15 users (show)

See Also:
Issue Type: DEFECT
Exception Reporter:


Attachments
Thread dump when 'Scanning Projects' happens every 10 seconds (24.83 KB, text/plain)
2009-04-05 16:20 UTC, theshadow27
Details
See BreakNBManyFiles.java in the Default Package (621.62 KB, text/plain)
2009-04-28 19:38 UTC, theshadow27
Details

Note You need to log in before you can comment on or make changes to this bug.
Description jessholle 2008-05-13 21:57:10 UTC
We have numerous projects, some of which are very large.  Some of these involve not only large jars, but also large
loose codebase directories in their classpaths.

With this reality (this is not going to change), NetBeans has in one respect continuously gotten worse despite its
numerous improvements: the IDE is ever more insistent on scanning all of these directories before letting you do real
work -- even in 6.1 where this is purported not to be the case.  Also in cases where one rebuilding the output jar in a
project upon which many, many other open projects depend the IDE will decide that their sources needs to be rescanned in
their entirety in cases.  This is *not* a recipe for large code set scalability.

Perhaps we're alone in this boat -- or perhaps others in it have already jumped ship to Eclipse with its incremental
compilation logic, but I wanted to raise the issue for community discussion.  I did so on nbuser and did not seem to get
any response, so I'm recording this as an issue.  This really should be noted as a defect in many respects due to its
severity, but I'm calling it an enhancement to avoid any arguments over issue type.

In my book I think the IDE needs to provide more control over when code scanning is done at a bare minimum -- especially
for those unfortunate enough to have loose codebase directories.  It should also likely provide some control over the
priority the background code scanning thread is given -- as this thread has been aggressive enough to render the
remainder of the IDE barely usable.  There's also got to be a better way than rescanning all downstream projects in
their entirety when you rebuild a project upon which many others depend.  Finally, I'd like to see the IDE live up to
the promises in the 6.1 release notes and allow me to do "Go to Type", etc, with existing code completion information
without waiting for all scanning to complete.  When I start the IDE I know I have full (and up-to-date) code completion,
etc, data in many cases yet the IDE refuses to "Go To Type", etc, until it has rescanned the world.

On top of all this, I really think "Go to Type", etc, should still be faster, though I certainly recognize progress in
this regard.  That said, as a point of comparison I've used NetBeans' classfile module to analyze all our class files,
store cross-references in a relational database (with appropriate indexing, of course), and then fronted this with a web
app.  The results of "Go to Type" type lookups (and many others) are *very* fast in all cases with this web app (even
when running everything involved on the same laptop that I'm running NetBeans on).  I guess I should integrate that with
the IDE, but I'd think the IDE should do this sort of thing better than my somewhat ad hoc web app!  [Also I limit the
fidelity of my Find Usages to which class is using another and don't record/divulge which lines the usages occur upon.]
Comment 1 Tomas Zezula 2008-05-14 16:32:26 UTC
Too many things here, if I forgot to answer something please lat me know.
1) Probably one of the problem is that you use the option which prefers the output.jar to sources, which causes all the project's depending on the output.jar 
being rescanned when you rebuild the output.jar.
2) >I think the IDE needs to provide more control over when code scanning - can you provide more details? Thread priority is clear, but not sure if it will 
help, most OSes increase dynamic priority to threads doing syscalls (IO) which is case of this thread. But it can be tried.
3)> There's also got to be a better way than rescanning all downstream projects in their entirety when you rebuild a project upon which many others 
depend. This is caused by preferring output.jar.
4)>data in many cases yet the IDE refuses to "Go To Type", etc, until it has rescanned the world - sounds as a bug. Adding pflaska to cc, who is working on 
this feature.
5)> The Goto type in NB 6.1 needs to wait until the source root is indexed, there is quite big delay, which should be fixed in 6.5, again pflaska should 
know more.
6) The classfiles indexing is very fast. The rt.jar (~18000 classes) is scanned and stored in less than 10s, the problem is class files doesn't contain all 
information, for project you need to prefer sources which is much more time expensive, but I believe it can be faster than now.

The most important, at least from my point of view, is to prevent non needed indexing.
Comment 2 Tomas Zezula 2008-05-14 16:33:32 UTC
I believe pflaska will work on this in NB 6.5.
Comment 3 jessholle 2008-05-14 16:48:45 UTC
Thanks for all the information.

I don't know that we've used any "prefer jars to sources" option per se.  We have used -J-DCacheClassPath.keepJars=true,
which we absolutely must for now as per issue #134992 and issue #121853.  See these issues as to why we're using this. 
We don't want to "prefer jars to source" overall -- we just want a means of dealing with classes resulting from non-Java
sources (without having to split them into separate projects, which is a non-starter since they're interspersed in the
same source trees as the Java sources).

Overall indexing/scanning appears to be pretty fast except when:
  1) Indexing a large loose codebase directory containing classes and other files.
       * There really ought to be a way to set a maximum reindexing frequency
         for such directories (e.g. once a day) and then to force reindexing on demand.
  2) NetBeans decides a large source directory needs to be recompiled/scanned.
       * This can grind NetBeans to a halt and make even other apps hard to use during
         this interval.

On what is really another front entirely "Go To Type", etc, are still *far* slower than they should be.  I don't see any
reason that the code index/scan can't produce data that's just as efficient to search as the relational database case I
noted above.  This should be blindingly fast.
Comment 4 jessholle 2008-08-18 16:09:29 UTC
Note that the need for -J-DCacheClassPath.keepJars=true is now gone in NetBeans 6.5 (at least for us) due to issue
#134992 being resolved.

Not using this option has resolved the issue where building one project causes all downstream projects to do a code
completion/navigation database rescan/recompile.

The other issues still remain, however.  I often try to use code completion or navigation only to get "scanning in
progress" and no action.  Sometimes when this occurs there is no visible indication that such activity is in progress in
the status bar.  Moreover, scanning should never prevent code completion or navigation.  As each source tree is being
updated its previous code completion/navigation data should be used with a warning that the data may be stale. 
[Actually code completion/navigation should be prevented when there is *no* code completion/navigation data whatsoever,
but that's rarely the case.]
Comment 5 twolf2919 2008-08-18 18:24:36 UTC
I second everything Jesse mentioned with respect to code completion and go-to-type becoming useless too often while
scanning is taking place.  In addition, I want to add that "Find Usages" is also VERY slow (I'm running 08/17 daily
build of 6.5).  On our project (3825 Java source files), I just did a "Find Usages" on a static method and it took 33
seconds to find 26 usages.  Granted, my machine is no speed deamon (single core Dell Latitude D810 laptop with 2GB
memory), but it's not exactly a low-end box either.  I'm running Ubuntu 8.10 w. JDK 1.6b10 (build 25).

Comment 6 jessholle 2008-08-18 18:33:38 UTC
The various forms of "Find Usages" are also nearly instantaneous (unless the result set gets really large) with my
little relational database plus web app.
Comment 7 tkellerer 2008-08-18 18:39:14 UTC
I cannot confirm that "Find usages" is fast. Sometimes it takes up to 30 seconds until the progress window is
initialized and the progress bar "starts moving". This happens on a single core as well as on a dual core machine as
well. Most of the time, cancelling the dialog and re-issuing a "Find usage" is then faster.
Comment 8 jessholle 2008-08-18 18:46:11 UTC
In my previous comment I meant that the "find usages" query types are nearly instantaneous against our (huge) data set
when the searching is done via the static analysis web app backed by a relational database that I mentioned in my
original description.

"Find usages" is far from fast.  Part of this, getting from which classes used something to which lines in those classes
used something, is somewhat understandable to me -- as my web app only tracks usages to the class level and then uses
heuristics to highlight the source lines that appear to be involved.  That said, getting to the list of classes that
use/subclass/whatever something should be far, far faster than it is.

Still I can live with "Find Usages" being a bit slower as I use it less.  Code completion and navigation must be always
ready (no "scanning in progress" business) and always nearly instantaneous.
Comment 9 Tomas Pavek 2008-08-21 17:53:52 UTC
> I often try to use code completion or navigation only to get "scanning in progress" and no action.

So "Go to Type" or code completion *triggers* the scanning by itself?
I wonder if this is ever supposed to happen. Bug?
Comment 10 jessholle 2008-08-21 18:38:34 UTC
I don't know that "Go To Type" or code completion triggers the scanning.

Rather I get a "Scanning in Progress" tooltip instead of a code completion pop-up for instance.  All the while there is
no scanning or compiling note or progress bar in the status bar -- so the scanning could have been going on for some
time without the IDE telling me or it is possible that no scanning is actually occurring and the IDE is confused. 
Either way it is not very helpful :-)
Comment 11 neu242 2009-02-20 08:10:45 UTC
I can confirm that "Go To Type", "Go To File" and the auto-completion features in NetBeans usually are downright
unusable in large projects.  I've seen reponse times up to a minute or two, with no visual feedback.  Find and grep in a
unix shell is unfortunately much faster.  

Eclipse and IntelliJ has near-instantaneous response on similar operations on the same project base, I don't see why
NetBeans have to be that slow.

I've tested this on NetBeans 6.5 and 7.0-dev.
Comment 12 jessholle 2009-02-20 12:24:18 UTC
I am being so bold as to change this to a defect.  I don't know how I was ever talked into letting this be an
"enhancement".  Sure I argued for some things that were an enhancement, but overall this is about having usable behavior
for code scanning and scan data usage, e.g. Go To Type.

As per a recent nbuser post, I can sum up the inter-related issues as:
   1. Scanning is fairly slow, whether this can be improved is unclear.
   2. Scanning occurs when you do not want it.
        * You should be able to tell it "honestly your data for X is good enough for now, don't scan X now".
        * This is especially horrific if you have a large directory containing loose class files instead of a jar
somewhere (an unfortunate situation, but truly necessary in some cases), but also applies to large jars.
        * Addressing this is much less critical if (4) is fully addressed.
   3. Scanning sometimes occurs behind your back without clear indication in the GUI
        * Sometimes you'll try to do an operation and you'll get "scanning in progress, please wait", yet there's no
indication of scanning in progress in the status bar, much less of what's being scanned.
   4. While scanning is occurring the IDE is unnecessarily crippled
        * One should still be able to do any/all "Go To Type", etc, navigations, debugger, etc, without any ill-effects
from scanning except: lack of new data being produced by the scan and a warning of this lack.  If jar X is being scanned
but you already had scan data for jar X, then the previous scan data for that jar should be used while jar X is being
scanned -- with a warning, of course.  There should be no "scanning in progress, please wait" or "Go To Type" dialogs
missing results that were present in both the old and new jar.
        * Scan data for directories in the classpath should be stored as well and treated in the same fashion just
described for jars -- no difference.
   5. After scanning is actually done searches using the scanned data are often slower than expected.
        * I scan our class files with NetBeans' classfile module and pump loads of static analysis data into a
relational database running on my laptop.  The equivalent of Go To Type, Find Usages, etc, searches are generally
noticeably faster with my database searches than they are for NetBeans.
        * Overall, the speed of Go To Type and other navigation using the scan database is critical.  If this is near
instantaneous, NetBeans seems miraculous to users and allows them greater productivity on large bodies of code.  If this
is not, then NetBeans is a bloated whale.  It's that simple for users.

This really should be a P1 bug -- it is killing NB usage, and is NB complaint #1 through 5 of those I work with, both
those who use NB and those who refuse to.  I'm setting it to P1 knowing it will get bumped back down, but maybe someone
will notice it in the interim.
Comment 13 Jan Jancura 2009-02-20 12:55:00 UTC
Not a p1, I think.
I will try to investigate. Thanks for your help.
Comment 14 jessholle 2009-02-20 13:32:03 UTC
Thanks for giving this some attention.  No, it really is probably not a P1, just a high P2 that someone talked me into
letting be labeled an enhancement.

Note that this isn't just a matter of slow "find usages", "go to type", and code completion ("go to type", Ctrl-B, and
code completion being the critical use cases here for speed as they're things you do mid-thought-stream, whereas "find
usages" is an activity to itself).  The slow scanning combined with an inability to continue reliably using the old scan
data during a scan that is updating said data are just as big of issues.

There's also the matter of fragile scan data that some developers are seeing around here and on the list.  This should
be a separate bug but I don't have enough information to file a meaningful one as I've not reproduced this personally. 
I have watched it be reproduced in front of me, though.  Essentially this involves having a set of projects that build
fine and scan fine, showing no inappropriate error markers.  The developer proceeds to compile certain Java sources (not
just any Java source will cause the issue) and then the NB editor starts marking all sorts of things with "cannot find
symbol" sorts of errors.  This seems to occur more in 6.5 and has caused a good number of developers to revert to 6.1.

Overall there are some big problems in this area that are causing NB users to switch away or strongly consider doing so
-- and preventing Eclipse users from considering NB.  Scan reliability and speed, both on scan update and scan data
usage is right up there with basic editor speed as far as criticality to IDE usability.  This area needs attention!
Comment 15 Jan Jancura 2009-02-20 14:01:24 UTC
You wrote that you are working on some very large projects. Is it possible to download some of them, to test performance?
Can you attach information how big they are (how many source roots, jars, source files, megabytes...).
What type of NetBeans project you are using?
Do you have some numbers, like start NetBeans with foo project opened (+info about size of project) + fresh userdir =>
scanning takes 5 minutes?
Opening the same project with Eclipse => 1 minute...

We will try to do some comparison anyway, but I will appreciate any data.
Comment 16 tkellerer 2009-02-20 14:03:21 UTC
I open the source for the Liferay portal every now and then, but I make sure it's closed before I close NetBeans.

Otherwise the next startup wil increase my coffee consumption too much :)

Comment 17 jessholle 2009-02-20 14:21:34 UTC
Is there some sort of module you can give me that will just collect size metric information you're interested in from my
open projects?

We're using free-form web projects with a complex Ant framework being invoked for the builds.  The source is all quite
proprietary and thus I'd have to go through various legalities to hand out a copy.

I can try to generate some numbers for NB -- again a NB module (or log level configuration) that produces such metrics
would be nice to ensure consistency and allow many developers to easily give you numbers.  I don't actually use Eclipse
myself at all, so I can't give Eclipse numbers.  I don't know that Eclipse is faster except insofar as NB users move to
it for this reason and Eclipse users who might otherwise dislike Eclipse won't consider NB for this reason (amongst others).

Overall I'm not dead set of scanning being dramatically faster.  If NB is still perfectly usable while it is scanning
and navigation / code completion using the scan data is lightning fast, that would be a big step forward.  The fact that
scanning can sometimes hog too much of the CPU for the IDE to be usable while scanning is a big problem (and a
separately filed issue).  The fact you can't do certain operations at all while scanning but instead get a "please wait"
is unacceptable [unless scanning is all but instantaneous irrespective of project size/number].  The fact that "go to
type", etc, forget the existence of types that it new about from previous scans while rescanning a jar is also
unacceptable [even *if* scanning is all but instantaneous].  Finally, the long delay in getting results from "Go to
Type" or Ctrl-B in many cases [sometimes it is quite fast] is unacceptable.  If all of those items are fixed, then the
speed of scanning itself is much, much less critical.
Comment 18 _ gtzabari 2009-02-20 14:35:03 UTC
From memory I remember that if you have 4-5 projects open in Netbeans and you have over a total of 100 classes then
Netbeans will begin crawling. Please note I have no retested this recently but it used to be the case a few months ago.

I was surprised by how quickly things deteriorated. 100 classes is not large by any stretch of the imagination. I don't
want to point you guys in too many different directions. I think the best first step would be to measure how our
performance degrades as the number of classes increases. I suspect (but we need to prove this) that performance degrades
exponentially.
Comment 19 Jan Jancura 2009-02-20 14:44:41 UTC
Creating some module that can collect performance data is great idea! Thanks. I will ask here if we have something like
this.
Comment 20 jessholle 2009-02-20 14:56:28 UTC
I see the need for at least 2 different areas of metrics:
  1) Something to report on the number and size of projects loaded
       * For each project list how many source files, how many K of source files,
         how many jars referenced, how many MB of jars referenced, etc.
       * List totals for all open projects
       * Developers would like the ability to generate such a report anyway; this would be a nice feature overall
  2) Something to report on time consumed for various IDE tasks
       * This may just be a set of timing loggers that you can tell users to enable when they complain.  Thus this
         may not be so much one new module as enhancements to the logging in many modules.
       * There may be some general points where one could insert timing from which lots of things might benefit
         quickly, e.g. the NetBeans background task processor could time each task it processes.

Apart from these items, however, I have been pushing for the NetBeans CPU profiler (and VisualVM) to offer a simple
sampling mode which one could use to determine which areas of a large application to re-test with NetBeans CPU profiling
instrumentation.  Apart from the general utility of such a feature, it would be great if NB offered a simple "profile me
[start/stop]" action which would capture a sampling profile during some operation.  Users could then be directed to do
this for any operation that isn't performing as well as they'd like (or that seems to hang as long as it doesn't hang
the whole UI) and attach the result to an issue.  This would seem to streamline the process of addressing performance
issues in NB overall.
Comment 21 Michel Graciano 2009-02-20 15:27:43 UTC
Please, update TM.
Comment 22 jessholle 2009-02-20 15:44:18 UTC
One of our projects has over 7900 source files and has 56 libraries in the classpath.

Many of these source files are quite large often with numerous inner classes.  Many of the classes are quite
interdependent.  Many of the libraries are quite large.  A good number of the libraries are outputs of other projects. 
There are then dozens of projects depending on this project (and other projects besides).  These projects are also often
fairly large, but not as large, say 1100 source files in cases.  Projects generally have between 1 and 3 source roots.

[All project source file counts were provided by SQE, so hopefully it is reliable in that regard.]
Comment 23 jessholle 2009-02-20 16:16:58 UTC
The primary output jar from the 7900 source file project is around 27MB (compressed) by the way.  That includes a lot of
.ser files as well, but class files are definitely the bulk of the jar's content.
Comment 24 Tomas Pavek 2009-02-20 17:09:58 UTC
As for the "Profile Me" feature, it's already in current dev builds (will be in M2). See issue 153221, and
http://wiki.netbeans.org/FitnessViaPartnership.

As for the metrics, would be nice, definitely to have some plugin that would collect various useful data. It's not that
clear, though, what the useful data should be. The size of the project, number of classes, etc, definitely matter, but
is not the main problem. See gtzabari having problems with only 100 classes. OTOH we (NetBeans developers) work on quite
large project, with tons of projects, zillions classes, many dependencies, but don't see significant problems - neither
with scanning nor with code completion or Go to Type navigation. We also do some measurements on a reasonably large
project (20MB of sources) where we measure startup of the IDE, time to open the project, time to complete the scan, time
to use code completion, time to build the project, consumed memory, etc. In all these areas NetBeans is comparable or
even better than Eclipse. So all in all, it's not generally true that NetBeans is slow on big projects. There are other
(bigger) problems than size.

The various scanning and related problems seem to be always tied to the particular project, its configuration, custom
tasks running as part of build, etc. There's usually some problem causing the IDE to scan sources too often, or scan
parts of classpath that would not need to be scanned. Whether it is misconfigured project, or too sensitive/fragile
mechanism of detecting and updating changes on classpath, I don't know (I'm not expert on the java infrastructure).

Anyway, this means that we are able to do very little with general statements like that scanning is slow etc. Ideally we
need the project itself to test it and profile. So if you can, please send us your projects, with description what to
try. There are also other ways: One is the new "Profile Me" feature, another one is to use a special startup switch to
create scanning log:
-J-Dorg.netbeans.modules.java.source.usages.RepositoryUpdater.level=FINEST
The log will contain what scanning happened and why. Cane use FINE level for less details. Since the log can be pretty
huge, it's necessary to find the case where you see scanning which should not happen, and run the IDE  with the switch
only for reproducing this one problem. From the log we might be able to find out what is wrong.


BTW we also plan to create an experimental module which would turn off some of automatic classpath updates/scanning and
give the user the option to invoke the refresh/rescan explicitly. See issue 158177. There are concerns this could lead
to various errors and inconsistencies, but is probably worth trying if it can solve some problems.
Comment 25 jessholle 2009-02-20 17:24:28 UTC
Glad to hear it about the "profile me" progress.  I've been begging for almost exactly this for some time.  Will this be
made part of 6.7?  Please!!!  This is huge to both NB and VisualVM.

As for scan speed, I can try the logging, but again I'm not really too concerned about scan speed by itself.  I realize
they're going to take *some* time in my case however hard you try.  I'm concerned by loss of functionality/usability
during these scans and speed of navigation features based on the scan data.
Comment 26 Tomas Pavek 2009-02-20 17:41:05 UTC
Re "Profile Me" - it's already in, you can download a daily build and try it.

Re Logging - the log should show why scanning happens when it is not expected. It can also show that bigger amount of
files is scanned than needed. I.e. it may reveal bugs in the scanning infrastructure, or problems with project setup.

As for IDE usable even during scanning (i.e. use the previous version of data until scan is finished) - that was
discussed several times, but always found too difficult to implement. See also issue 133943. For 6.7 we hope for an
improvement of Go to type navigation during scanning, at least (new indexing API should allow that).
Comment 27 _ gtzabari 2009-02-20 18:33:41 UTC
tpavek is right. We need a *specific* testcase (complete with the source-codes if possible) to work against. Otherwise
we'll spin our wheels trying to fix a theoretical problem without being able to verify whether our work made a
difference or not.
Comment 28 twolf2919 2009-02-20 18:45:20 UTC
Most of us work for commercial companies and aren't allowed to send source code.

This might sound silly, but throughout the conversation, NB devs always indicate that they have no problem with a pretty
big project: Netbeans!  Can the NB developers put together a nice self-contained zip file with the entire NB project &
src and let people who have trouble with their project try it out to see if they also have problems with the NB project.
 If so, we learned something.

(I realize that the NB src is readily available.  If it's already *very* easy and self-contained, then maybe just point
to it) and we can try it.  I just made the above suggestion on the off-chance that building NB takes more than simply
downloading and opening/building in the IDE).

Comment 29 greggwon 2009-02-20 19:13:54 UTC
Create a library project with perhaps 100 classes in a jar file.  Change the build.xml for that library so that
"-post-jar" copies the jar file into a "public directory" that is the location that the library manager has in its
configuration for the library.

Next, create 5-10 projects with approximately 500 classes in each, have each of those projects reference this library.
Make sure all of these projects are open in the IDE along with the library.

Finally, open some .java files in one or more of those 5-10 projects.

Make a change in the library and rebuild it.

As it's building start editing and trying to do work on those java files.  Build them, try and run things and do normal
development tasks.  At some point, the IDE will start recompiling those 5-10 projects because the library timestamp changed.

During that time, you will not be able to work.
Comment 30 Tomas Pavek 2009-02-23 17:20:18 UTC
As for NB sources, can be downloaded as one zip file e.g. from here:
http://www.netbeans.org/downloads/start.html?filename=zip/netbeans-6.5-200811100001-ml-src.zip&version=6.5

To test it, you can open whatever project in the IDE - e.g. "form". Note that nobody opens all projects, but having 
just one project opened brings in all the projects it depends on (transitively), which is typically quite big number. 
You can let the IDE to open all the dependent projects explicitly. We usually don't build entire NetBeans in NetBeans, 
possibly just the module being worked on (plus dependencies).

When opening the project, you'll see quite long initial scan (much shorter next time when IDE is restarted). Then you 
can try to jump to some class to test Go to Type speed (try e.g. FileObject or RepositoryUpdater), try to modify some 
file to see code completion speed, do some refactoring, etc.

This way you can get some idea of speed at which the IDE can work on a big set of projects...
Comment 31 greggwon 2009-02-23 18:46:07 UTC
I have a particular configuration which I illustrated the details thereof.  It creates problems for the implementation
of scanning and error badging related to classpath detected changes.  Many others have lamented on the mailing list the
same types of problems which they experience, and based on what I can tell, they have similar project numbers and
layouts and are experiencing the same problem.

I don't know how else to explain what I see...  I know that the NB team says you all can develop the IDE just fine.  I
also know that there are specific optimizations in the IDE's behavior regarding netbeans platform development.

I am not developing a netbeans platform app and I am not developing the netbeans IDE (a form of the platform).  I and
many others it seems, need help with fixing the problems that occur in our environments.
Comment 32 Tomas Pavek 2009-02-24 10:26:12 UTC
> I and many others it seems, need help with fixing the problems that occur in our environments.

That's exactly why we ask for the details. We need to know how your environments look like. Thanks Gregg for describing
yours, we should take a look at it. Let me summarize your case, someone from NB java team can comment more on that.

By copying a built JAR file to a "public directory" which is used by other projects you are changing the classpath of
these projects. When this happens, the IDE can see the entire JAR file changing, but has no idea what actually has
changed inside the JAR file. It does not compare it to previous state to determine some "incremental" changes. At this
point it needs to rescan the projects that depend on the JAR file because there could have been potentially any change
affecting them (some errors in source code may appear, some existing may be resolved). In reality, there is usually no
significant change (affecting public API) and such huge scanning is not needed. It's even bigger problem if you need to
rebuild your library JAR often...

The system is designed the way it expects libraries/JARs on classpath are not changing, or only exceptionally.
Definitely not as part of compiling/building. If you rebuild your libraries often, then you should have your dependency
defined on the project level - i.e. you should depend on the library project output which is associated with sources.
Having the association to project sources for a JAR file, the IDE knows that the change of the JAR file (when built) is
not significant - because the IDE is checking against the library sources and processing incremental changes if you edit
them. This may still cause scanning, but in much less cases.

This is a simple view on the problem. Someone from java team please correct me if I misrepresented something.

Now the question is whether you can alter your project setup (or at least try - to see if it can really help), or if
NetBeans can behave differently even in the case it has not the sources associated. I only know about issue 158177,
which could help solving this, but it is only an experiment.
Comment 33 matthies 2009-02-24 13:52:59 UTC
With regard to "If you rebuild your libraries often, then you should have your dependency
defined on the project level". When you build using non-NetBeans ant scripts (i.e. freeform project) and a non-NetBeans 
dependency management, for example using Ivy, then you want to avoid project-level dependencies in NetBeans in the 
general case, as the results of depending on local project sources won't be consistent with your build infrastructure. 
It would be acceptable while you work on two dependent projects simultaneously, with dependent changes. but not 
otherwise, because the reason for the build infrastructure is for the builds to not be dependent on the IDE or local 
settings. In this scenario, which for us is the default scenario, what would at least be needed then is the ability to 
quickly switch between project-level dependencies and classpath dependencies in NetBeans, without the world being 
rescanned.

For some reason all of this doesn't seem to be a problem in Eclipse, and our impression is that this is a killer 
advantage of Eclipse over NetBeans in "enterprisy" environment, no matter how much better NetBeans maybe in other areas.
Comment 34 matthies 2009-02-24 13:53:01 UTC
With regard to "If you rebuild your libraries often, then you should have your dependency
defined on the project level". When you build using non-NetBeans ant scripts (i.e. freeform project) and a non-NetBeans 
dependency management, for example using Ivy, then you want to avoid project-level dependencies in NetBeans in the 
general case, as the results of depending on local project sources won't be consistent with your build infrastructure. 
It would be acceptable while you work on two dependent projects simultaneously, with dependent changes. but not 
otherwise, because the reason for the build infrastructure is for the builds to not be dependent on the IDE or local 
settings. In this scenario, which for us is the default scenario, what would at least be needed then is the ability to 
quickly switch between project-level dependencies and classpath dependencies in NetBeans, without the world being 
rescanned.

For some reason all of this doesn't seem to be a problem in Eclipse, and our impression is that this is a killer 
advantage of Eclipse over NetBeans in "enterprisy" environment, no matter how much better NetBeans maybe in other areas.
Comment 35 Tomas Pavek 2009-02-24 14:17:16 UTC
"Depending on local project sources" should be only for the IDE, and possibly need not affect the building
infrastructure. Only the IDE would resolve against the sources, building would work with JAR files. So in other words it
should be possible to specify the sources for any JAR/library on classpath. I'm not sure right now if that is possible
in such general way (for free form projects configuration; let's say it should be there). But I don't see how else the
IDE could determine that it should ignore a changed JAR on classpath. If Eclipse does that without rescanning, does it
catch and show errors correctly in depending projects if you e.g. delete a public method in the library and rebuild it?
Comment 36 matthies 2009-02-24 14:45:51 UTC
Eclipse does rescan, but it doesn't block anything during that scanning, and it also doesn't "forget" the individual 
results of a previous scan until they are dynamically replaced by results of the new scan. I suppose the update of 
these informations are done per-class or something.

As for "only the IDE would resolve against the sources", that's exactly what would cause inconsistencies: Unless these 
local sources are exactly the same version as the ones the jar you build against was built from, the IDE can show 
errors from its internal compilation where the actual build won't produce errors, or vice versa, and code completion 
and refactorings will not use the correct information.
Comment 37 Tomas Pavek 2009-02-24 17:27:50 UTC
OK, so to sum up the main use case here:

There is a "library" project which is built quite often, but separately from other projects that use it. The separation
is intentional and the dependency should only be on the produced JAR file (its copy in fact). Relating the JAR file to
the library project sources is not wanted, because as explained, the JAR file is held separately and may not correspond
to the sources. (Direct dependency between projects would ensure the consistency and eliminate the scanning problem, but
then the projects would not be separated as requested.)

Now the problem with this setup is that rebuilding the library and then switching to some of the projects that use it
cause "entire world" rescanning which makes the IDE unusable. The rescanning is needed, but Eclipse is supposed to
handle this situation better because it's doing it "on background" while the IDE is still responsive and usable,
navigation works, nothing is blocked etc. Seems its able to use the index and data even during the scan, updating it
continuously.

From this viewpoint, NetBeans does not scan needlessly but should try to fix the scanning itself to be more user
friendly (as the issue 133943 is about). It is not obvious the problem should be solved by making the IDE scan less (not
react on some changes, etc, as issue 158177 is basically about). So matthies, greggwon - is this basically your
standpoint? I must admit I thought the second one was the main problem and also the difference from Eclipse. Still it
can be both...
Comment 38 greggwon 2009-02-24 18:05:13 UTC
I think that there is a very simple issue here.  The NB team wants to help developers out by pointing at problems.  The
issue is that the timeliness of the reports is not happening and the NB team thinks its more important to guarantee the
integrity of the "view of the world" than to guarantee that the developer can work on the problems which they can see
and do know about.

Because of the speed of disk drives, the size of projects, the I/O issues through systems in general, there is no way,
at all, that all the work that Netbeans tries to do for developers, to update the "view of the world", can be done in a
timely fashion, and make that update appear synchronously with editing changes and other asynchronous file system updates.

We really have to decide that this is true!  Instead, we need to move to the point of agreeing that it's really okay for
netbeans to show (only) exactly what it knows at any particular moment, but that it's knowledge can change over time as
it works behind the scenes to update the "view of the world" towards the current reality (a moving target at best).

More importantly, we also need to confine the things that netbeans tries to learn about as its "view of the world" to
only be the things that are truely important for the current moment in time.

My thoughts are that for any outside (not from the editor) indication of a change, the IDE should only check the current
file in the editor, then the other files in the editor related in order by

Comment 39 greggwon 2009-02-24 18:10:24 UTC
I think that there is a very simple issue here.  The NB team wants to help developers out by pointing at problems.  The
issue is that the timeliness of the reports is not happening and the NB team thinks its more important to guarantee the
integrity of the "view of the world" than to guarantee that the developer can work on the problems which they can see
and do know about.

Because of the speed of disk drives, the size of projects, the I/O issues through systems in general, there is no way,
at all, that all the work that Netbeans tries to do for developers, to update the "view of the world", can be done in a
timely fashion, and make that update appear synchronously with editing changes and other asynchronous file system updates.

We really have to decide that this is true!  Instead, we need to move to the point of agreeing that it's really okay for
netbeans to show (only) exactly what it knows at any particular moment, but that it's knowledge can change over time as
it works behind the scenes to update the "view of the world" towards the current reality (a moving target at best).

More importantly, we also need to confine the things that netbeans tries to learn about as its "view of the world" to
only be the things that are truely important for the current moment in time.

My thoughts are that for any outside (not from the editor) indication of a change, the IDE should only check the current
file in the editor, then the other files in the editor related in order by

1. dependended on by the currently edited file
2. In the package of the currently edited file
3. In the package of any other opened file with depended on being first
4. In the current project
5. In the projects of other dependent and opened in the editor files.

If the keyboard goes idle for more than 30 seconds, the IDE could then start assessing the effects on other projects.
It could initially just mark dependent projects (for example if a library changed) as "dirty" and use the project
dependency information to note which files might be dependent on that library and mark those in particular as dirty.

But it really should not do anything behind the scenes until the user is not actively using the IDE.  As the user
changes the currently edited file, the 1-5 above should be the order of processing to work on updating the "view of the
world".

If I break something in a library API, I do it on purpose and I know what to change to fix everything and I don't need
the IDE telling me I can't work on what I need to fix until it makes sure that another, unrelated project has no problems.

That's what I currently see happening when I update a library.  I can't edit the current project because the IDE is off
compiling another unrelated project.
Comment 40 greggwon 2009-02-24 18:49:36 UTC
The system is designed the way it expects libraries/JARs on classpath are not changing, or only exceptionally.
Definitely not as part of compiling/building. If you rebuild your libraries often, then you should have your dependency
defined on the project level - i.e. you should depend on the library project output which is associated with sources.
Having the association to project sources for a JAR file, the IDE knows that the change of the JAR file (when built) is
not significant - because the IDE is checking against the library sources and processing incremental changes if you edit
them. This may still cause scanning, but in much less cases.
-------------

Are we then just missing a piece of the chain that the IDE doesn't know about?  The source files in my library project
are known.  The output file is known, but the place that I copy it to, the library manager designated path, is not part
of the puzzle.  Should it be?  Is there a different way that a freeform project can make this part of the chain visible
to the IDE?
Comment 41 matthies 2009-02-25 19:12:31 UTC
I agree with greggwon on the point that the scanning could be more demand-driven. For example, if member code 
completion is invoked on a variable whose class is known to have out-of-date scan data, then specifically this class 
could be scanned as needed to update the member list for code completion, rather than waiting that "the entire world" 
has been scanned. The code completion could be dynamically updated, so that the user isn't blocked even during this 
demand-driven scan.

Similar "targeted" scanning would be helpful for Find Usages and refactorings. For such operations the user could also 
be given the option to have the operation performed immediately based on the current, partially out-of-date scan data, 
because the user may know that the yet-to-be-scanned changes don't affect the operation, or is willing to take the risk.

Unlike greggwon, I think that "the entire world" should *eventually* be fully scanned, to correctly update the error 
badges etc. But this could be done with lower priority and shouldn't disrupt or prevent user operations.

I suspect that there's also room for improvement in scanning and indexing speed even intra-project, though this is 
probably a separate issue. Rescanning sometimes appears to be significantly slower than equivalent recompiling. Or for 
example when adding a new method to a widely used interface in a largish project, and then immediately removing the 
method again using Safely Delete, the Safely Delete and the Refreshing Workspace that ensues afterwards takes ages 
(okay, minutes). It also appears that use of generics and circular dependencies (e.g. visitor pattern) affects scanning 
times disproportionately high. Another thing is that CPU usage rarely seems to be the bottleneck for scanning, so maybe 
the performance of I/O and index storage layout could be improved. I'm just speculating here though.
Comment 42 jessholle 2009-03-02 01:02:40 UTC
"Eclipse does rescan, but it doesn't block anything during that scanning, and it also doesn't "forget" the individual 
results of a previous scan until they are dynamically replaced by results of the new scan. I suppose the update of 
these informations are done per-class or something."

That says it all in my book.  Scans on really big stuff are never going to be sufficiently fast to avoid causing the
user lost productivity unless (1) nothing is blocked during scanning and (2) previous scan results are replaced at a
fine-grained level rather than being forgotten at a macroscopic level thus leaving you with huge gaps in your data
during rescans.
Comment 43 Tomas Zezula 2009-04-03 07:57:53 UTC
The indexing api (part of the parsing api) allows access to the class index data in time of scanning, the java features may use it. The 6.7 scan is much slower 
than it was in the 6.5. One of the reason may be creation of the FileObjects where Files were used before. I will try to return the possibility to use the Files, 
this requires the java indexer not to touch the FileObjects, I will create JavaFileManager based on the Indexables and Dusan should fix the indexer.
Comment 44 theshadow27 2009-04-04 16:17:52 UTC
I suspect that most people can only do one thing at a time, with the exception of batch refactoring, so only a small percentage of files change between 
builds and it is wasteful to scan entire "finished" code bases (like libraries) on every timestamp change. Instead, we should store a priority based change-
map that starts all classes with (priority) 1 change, and every time a change is found increment the priority. With logic "scan only files that have changed 
the most number of times" we could scan 'suspect' classes in real time and leave the low change rate classes for idle/background scanning. If this 
information was stored between sessions, it could also greatly reduce startup time. 

For example, we maintain a common "toolkit" class library that is shared between three products. Most of the classes do not change, but there are a few 
enums that are regularly expanded. Instead of the whole 3mb jar, I would much rather see only these 2 files scanned in real time when the project is built. 

Not only do most files not change, but I would go so far as to say that in most cases, the majority of classes in a library are not used. By joining a map of 
changes with a map of usages, it may be possible to heuristically scan a small number, say %5, of classes initially without compromising utility or the 
"NetBeans magic" that makes it such a good IDE. 

As much of a nightmare as it would be (I surely wouldn't want to do it...) a full out relational database with the ability to hold interfaces, method 
definitions, and javadoc may be the only 'real' solution. 
Comment 45 jessholle 2009-04-04 18:50:19 UTC
Doing a relational database containing the initial pieces, e.g. answering what classes are defined and where, what
methods and fields they define, which classes use them, etc, is not that hard.  I can say this as I've done it.

Now I currently do this in bulk using batch statements for many megabytes of jars in one swoop, but it is easy enough to
see how one could do a transaction for smaller updates, e.g. of the data related to a given source file, thus allowing
continuous access to the most accurate information possible.

With appropriate indexing on the resulting tables the queries for things like "go to type" are nearly instantaneous as well.

Of course I did this on work time, so it's not open source...
Comment 46 theshadow27 2009-04-05 16:20:04 UTC
Created attachment 79469 [details]
Thread dump when 'Scanning Projects' happens every 10 seconds
Comment 47 Pavel Flaska 2009-04-06 07:56:56 UTC
I do not think that relation database will help. Queries to Lucene index is fast enough without any doubt. 
Scanning time is composed from several things and they together takes a log time:
i) opening indexes,
ii) computing classpath,
iii) going through all files and checking timestamps.
iv) updating changed files.

When using "Go To Type" feature, again,
i) classpath computation,
ii) index search.

Especially for API support type projects, classpath computation (scope for search) is not fast enough.
Comment 48 jessholle 2009-04-06 12:19:28 UTC
I have 2 big issues with the existing speed.

1) Indexing/scanning takes time for large sets of code (unsurprising) *and* one can't use large portions of the IDE
during this indexing.  If the latter were solved, then I'd be okay with 6.5's indexing speed.  One possibility here
would be a separate relational database transaction for each source file being scanned, but there are certainly other
approaches.

2) Go To Type (and other critical quick navigation actions) can be maddeningly slow at times.  I've not seen the
equivalent take so long with a relational database.  Perhaps this is all due to computing classpaths.  In any case this
needs fixing.
Comment 49 theshadow27 2009-04-08 02:47:50 UTC
It might be my imagination, but 200904070200 seems much better about the indexing thing. I'm not sure if any code has changed here or not, but it seems 
like it has. 

Now, a single class will disappear without warning from the IDE's 'world'. That is it looked OK just minutes before, but suddenly one will get errors about a 
class not existing even though it compiles and runs fine. It's not always the same class, actually I've never seen it do the same one twice. 

It is also not noticeable until the IDE has been running for a while, which makes me think of issue 162176 and the slow memory leak. Perhaps the asynchronous code is all and well when fresh, but there it gets bogged down somewhere. That's for yall to decide :)
Comment 50 theshadow27 2009-04-28 19:37:36 UTC
Attaching a test case. This makes it very easy to see what this issue is talking about.

Steps to reproduce:

1) Decompress attached file. It is in .zip format
2) Open the project in NetBeans 6.7 (It was created using build 200904270201)
3) Open the "BreakNBManyFiles.java" under the Default Package in the editor
4) Press <shift> + F6 to run the file. This will create a package under auto.gen.code in "src" and "test"

Problems you will encounter:

A) Open up one of the packages, and try to select all files. This takes several seconds on a fast computer. 
B) Deleting all files takes a minimum of 30 seconds, if it does not freeze the IDE
C) Running the test file more than once will trigger the "Scanning Files" loop-of-death (Each time generates a new set of files) 
D) The JVM maxes out on physical memory, and starts swapping (fresh user directory on Java SE install with 3.0gb virtual after running the test project)
Comment 51 theshadow27 2009-04-28 19:38:28 UTC
Created attachment 81114 [details]
See BreakNBManyFiles.java in the Default Package
Comment 52 Jiri Kovalsky 2009-05-05 12:38:29 UTC
This issue was discussed during last VOC meeting #8 [1] and it was strongly suggested to fix this after 6.7.

[1] http://qa.netbeans.org/processes/voc/chats/transcript_090424.html
Comment 53 jessholle 2009-05-05 13:20:58 UTC
So performance and scalability of the most critical core functionality is always "next release"?

Sadly this sounds a lot like other software products (that shall remain nameless), but that's no excuse.
Comment 54 Pavel Flaska 2009-05-05 13:54:10 UTC
Try the "Scan On Demand" module from Development Update Centre. That should reduce the scanning. Bear in mind that usage of module is at your own 
risk! Any issue reported when module is on will be considered as a bug against experimental module. 
Comment 55 nleck 2009-05-06 02:31:10 UTC
The project scan *MUST* be improved in 6.7 otherwise you may as well just install Eclipse for remaining the NB
developers here, they have almost given up on NB and this will just be the death nail. 

I really don't think you guys understand how badly NB is coming up in comparison to other IDEs in this area.
Comment 56 jessholle 2009-05-06 02:44:12 UTC
Along these lines, while I can't reasonably share/upload my company's projects or source code as a test case, I will be
at JavaOne and could certainly let some NetBeans folk poke around in these via NetBeans on my laptop.

I'm guessing that there may well be others who are greatly pained by this issue who are similarly equipped.  It might be
good for the NetBeans guys to organize a "show me your irreproducible NetBeans problems" meet up or some such.

[No, I won't be attending CommunityOne or at least most of it -- as I've attended it every year thus far and haven't
found it worth the extra day of lodging and losing part of my weekend.]
Comment 57 helius 2009-05-06 09:58:28 UTC
I work for a big sized company and I have used Netbeans since the first version. We are more than 30 developers working
in the same J2EE projects. We are willing to use Netbeans but this issue and issue #42683 are preventing us to migrate
from JBuilder.

Comparing with other IDEs Netbeans lacks the stability and scalability for enterprise sized projects, and in every new
version this issues are postponed. Aren't this more important than new languages or eye-candy?

Scanning is hell. If you open the IDEs with ten or more projects it becames very slowly an unstable, freezing sometimes.
Opening a single java or jsp file may take 5 seconds to appear. If the are many classes in the project badging of
compiling errors are almost always wrong... If you compile the full project with dependecies takes forever, and after
that the badges remain wrong.

IMHO Netbeans is sadly becoming a tool for educational purposes, but not for the real work.
Comment 58 Jan Jancura 2009-05-06 14:55:35 UTC
Discussion here is getting long, so I would like to summarize it.

THIS BUG IS NOT ABOUT SLOW NETBEANS, BECAUSE NETBEANS ARE FAST!!! 
We have several comparison test on different large projects, and we KNOW that NetBeans are comparable to other IDEs on
these projects. We are developing NetBeans in NetBeans, so we REALLY KNOW that ... NetBeans are fast!!! (NetBeans source
base is really big)

So, this issue is about one concrete usecase:
User has custom Ant build script (free form project) that generates something to the Sources.
(Free form project sources are defined in free form project properties - "Java Sources" and "Java Sources Classpath" lists.)
Your Ant project should not modify files / Jars that are listed there!!!
If you broke that rule - you have troubles. NetBeans listens on that folders / Jars, and starts scanning.

The fact that NetBeans are listening on source path changes is design decision (not a bug) and we are not going to
change it. It is very useful feature.

So, what can we do with it? Stop listening on source path during build + add some manual rescan? We are investigating,
but there is no simple solution.

And what can you do with it? You can try to modify your build script to separate source folders / Jars from Output
folders / Jars. And you can modify your "Java Sources" and "Java Sources Classpath" lists to not contain any files
generated / modified by your build script. It should help to fix this issue for you.

And if you see some other issue why your NetBeans are slow, FILE ANOTHER BUG, PLEASE!

This issue is about "Source path modification initiated from build script starts scanning." only.


Comment 59 akochnev 2009-05-06 15:47:25 UTC
jjancura: I don't think it is very useful to say that the problem doesn't exist after all the comments and complaints on
the issue. It's great that the NetBeans team uses NetBeans for development and doesn't see any issues w/ performance.
However,  when someone was designing / building this auto-scanning feature they envisioned a particular usage model for
NetBeans, and it appears (rightly or wrongly) that how people use the IDE falls outside that envisioned usage model,
which causes them trouble (resulting in a slow IDE). 

The suggestions that you gave are useful in analyzing the problems and hopefully folks will go back and analyze their
projects for the issues that you describe. There is a problem with pointing the finger at the users and saying "You're
wrong": if the users set up their project wrong, there is a chance that the IDE didn't necessarily support them in
creating a "correct" project setup. The next steps that you identified sound reasonable - it would be nice if there is
some resolution to this SOON, you can see a lot of loyal / long time NetBeans users upset over this. 

One potential issue in what you describe as the source of the problem is that NetBeans can relaunch the scanning of
something changes in the "Java Sources Classpath" setup : it seems more or less normal for this path to contain class
directories and jars. It seems just as normal that an Ant build will modify some of those during the build. In a
different issue, jglick pointed out that if a classes directory or a jar is associated w/ sources, the IDE wouldn't
rescan even if the artifact changed. Thus, an additional suggestion to review the project structures is : are all of
your freeform projects correctly configured to identify the "output" of their source (so that NB can associated classes
and jar w/ source and not scan when the clasess or jars change during the build). 
Comment 60 matthies 2009-05-06 16:37:00 UTC
jjancura: I don't agree with your appraisement and narrowing of the issue. The scanning/refreshing has a design and/or 
implementation flaw which is not just related to custom Ant tasks. The scanning is often a problem too in projects or 
project groups without custom Ant tasks. Excessive scanning and/or wrong error badges happens in the context of:
- in-between saving of files with compilation errors
- larger refactorings
- CVS updates
- building of dependee or sub-projects, as common in J2EE project structures
- concurrent executions/combinations of the above, which NB doesn't (and shouldn't) prevent
- projects with heavy use of generics and/or recursive dependencies (source model appears to be buggy)
- incomplete or inconsistent source model data
- ...others may add more

The problem is not necessarily the rescanning or "refreshing workspace" itself, but that it
- is sometimes modal in that it is blocking certain user actions
- consumes too much CPU
- does a lot of I/O (much more than e.g. javac for the same set of source files)
- assumes a bug-free source and dependency model
- assumes no concurrent actions (or so it seems)

I believe there's both a performance problem in that updates of the source model appear to consume more resources than 
they should (and more than they appear to in other IDEs), and, as far as I can gather, a conceptual problem in that the 
design seems to be to "stop the world" until the source model is (believed to be) complete and consistent again, as 
opposed to being able to work with a possibly incomplete and inconsistent, incrementally updated model (which appears 
to be how other IDEs work).

So, source path modifications caused by Ant execution may well be an individual issue in itself, but is not the larger 
issue this Issuezilla issue was about, IMO.
Comment 61 nleck 2009-05-06 23:05:55 UTC
NB is *NOT* fast!!. The "project scan" issues are an embarrassment. Look at the message boards. 

As most devolopers use other IDEs when working on large existing projects we can't just change the project structure for
all developers to suite NB when the other IDEs have no trouble with the existing project structure. 

I have never seen Eclispe block the developer while trying to navigate to a class. With NB it seems to block far too often. 
Comment 62 theshadow27 2009-05-07 03:44:50 UTC
@jjancura When you say "And if you see some other issue why your NetBeans are slow, FILE ANOTHER BUG, PLEASE!"

The summery of this issue has been changed. According to my records on 30 APR 2009 (as of my test case), the summery was 
[Issue 134990] Java support is slow on big projects

Then, today (5 MAY 2009), I received this by email:

User jjancura changed the following:

               What    |Old value                 |New value
================================================================================
                Summary|Java support is slow on bi|Source path modification i
                       |g projects (code completio|nitiated from build script
                       |n, find usages)           | starts scanning.
--------------------------------------------------------------------------------

Instead of changing issues so that they no longer exist, then denying that they are issues, how about putting effort into fixing them?

Perhaps on NB source, NB works fine. All of the NBM that I have seen have custom ANT scripts, not the scripts that NB generates when a user creates a 
project. Have you considered that of the 36 registered users who voted for this issue, that the majority of them do not use custom ANT scripts?

I will file another issue; But coverups should have died with the CCCP.. Grow up.
Comment 63 Jan Jancura 2009-05-07 10:02:05 UTC
Fist, I must say that I do not see any big disagreement here. All we are trying to make NetBeans better, faster. We
should only clarify how to do it effectively.

Messages like "NetBeans are slow", "NetBeans are ugly", "NetBeans are buggy" are not fixable issues. See
http://wiki.netbeans.org/IssueReportingGuidelines. There is description how to write useful issue (see mainly "One
problem - one issue", "Provide detailed description", and "Provide steps to reproduce" sections)

"NetBeans are slow" is great point to discussion. We definitely need such great source of feedback from community. But
it should be discussed on mailing lists. Visibility of one issue is definitely lower than a number of people watching
nbusers mailing list. We need more users to discuss it, more feedback! Issues like "Source path modification initiated
from build script starts scanning." can be one of results of such "NetBeans are slow" discussion. You probably
understand that useful bug report have to be concrete, reproducible, measurable...

I have identified that "Source path modification initiated from build script starts scanning." is root cause of many
problems described here. That is why this issue is now about this usecase only. But it does not mean that we will throw
rest or reproducible usecases mentioned in this issue out of window. We will check all concrete user reports from this
issue and file separate issues for them.

Please,
If you would like to discuss NetBeans performance, go to nbusers (more users = more feedback). You can tag your message
with "[PERFORMANCE]" to be more visible.

If you would like to discuss NetBeans architecture go to nbdev mailing list. Themes like "Manual or automatic scanning",
"source models", "incrementaly updating scanning" are great points for discussion there. I am not able to attach all
developers, architects and other people responsible for NetBeans architecture to this single issue, but all of them are
definitely on nbdev! We can find some solutions there, and file concrete issues / RFEs than.

And, please,
if you have some concrete, reproducible usecases that are not covered by this "Source path modification initiated from
build script starts scanning." usecase - file separate issues for them, please!

Thanks
Comment 64 auxbuss 2009-05-07 14:49:00 UTC
This is ridiculous. By changing the summary, the issue no longer applies to me. But look! The problem still exists.

You may as well have changed it to, "My car won't start because there is no fuel in it".

Madness.
Comment 65 _ gtzabari 2009-05-07 15:26:41 UTC
jjancura's initial post ruffled a few feathers, but his second post makes perfect sense. One possible solution that
might please everyone is to transform this issue into an umbrella issue, with multiple sub-issues (one per problem).
I've seen the Netbeans team do this before...
Comment 66 greggwon 2009-05-12 15:11:11 UTC
The predominate issue is that scanning, no matter what initiates it, requires in its current design, too much CPU and
disk I/O bandwidth.  Part of that is based on the limited ability of or incorrectly established content of a cached view
of the source and class file content.

There really are bugs involved.  As was recently reported, one such logic bug was found.  I still contend from a point
of ignorance that there are architectural bugs which need to be dealt with to remedy the problem overall.

Clearly the dev team are defensive about where they believe the software is at. The users who are having this issue
impact their productivity, some in quite dramatic ways based on what we read in this issue, are not content with that
positioning.

Progress will only happen if we can all get on the same page and agree that there are issues that need to be addressed.  
Comment 67 adlepoo 2009-05-12 16:46:06 UTC
The main reason I noticed this is because in 6.1 IT DOESN'T HAPPEN.  When I "upgraded" to 6.5 is when the problem
appeared.  Maybe it doesn't matter at this point but it may be a clue as to what the problem is.

Also a possibly important note is that it isn't bad from the beginning.  It seems to be manageable until some threshold
is crossed (time, number of clean rebuilds?) and then the project scanning goes into its never ending mode.
Comment 68 kidvid 2009-05-26 22:51:14 UTC
I'd like to add my two cents to this problem.

First I'd like to add that many of the developers here have a lot of very positive things to say about NetBeans.  It's
amazing how much the product has improved over the past couple of years.  We like the way the GUI is laid out.  We love
the EJB generation wizards.  Some of us have used the Swing drag-and-drop editor, and I've heard good things about that.
 We also like the way plugins are handled.

Having said that, the "scanning..." problem described in this ticket is really killing us.  No joke.  We consider this
to be a high priority defect, if there ever was one.  Although it's not our primary build mechanism for making installs,
we do compilations within Netbeans just to make sure everything is Kosher, and we stick with the default ANT script
without any custom ANT tasks.

As mentioned previously, we wish we had more control over the scanning process, and we wish that it didn't consume so
many resources.  

Please fix this ASAP.

Comment 69 nleck 2009-05-27 00:07:07 UTC
I'd like to second kidvid comments. The reason I care so much about the "scanning" issues is that I love all the other
parts of the IDE, so much better than Eclipse in so many ways but one. The "scanning" issues are what have forced the
bulk of the developers here off NB. 
Comment 70 Tomas Pavek 2009-05-27 17:55:09 UTC
As was already mentioned, please try the Scan on Demand module, especially if you experience frequent or long re-scans:
http://wiki.netbeans.org/ScanOnDemand

Note the module is experimental, it's not meant to be the fix for the scanning problems, but it might give us valuable 
idea how the performance can really improve by reducing scanning and what other problems may arise. And it may 
actually help in many cases. We continue working on understanding and fixing the scanning issues on other fronts too.
Comment 71 Petr Jiricka 2009-09-14 16:35:32 UTC
I tried again the scenario described by theshadow27 in:
http://www.netbeans.org/issues/show_bug.cgi?id=134990#desc51
in the latest build of NetBeans 6.8. Here are my findings:

A) Select all files: yes it is slow, IDE autodetected frozen AWT queue. Reported as a separate issue 172191.
B) Deleting all files: also slow, reported as a separate issue 172199.
C) Running the test file more than once: there was a brief period of scanning, but nothing unreasonable or unexpected.
Seemed fine to me.
D) Also did not experience this (though my machine has 4GB of RAM).
Comment 72 Tomas Pavek 2009-09-14 17:29:03 UTC
> B) Deleting all files: also slow, reported as a separate issue 172199.

That's about performing the delete operation itself. There's also issue 168267 about optimizing the preparatory phase of
the delete (i.e. not to initialize refactoring when Safe Delete may not be chosen at all, and not to block invocation of
the action if scanning is running). Planned for 6.8.
Comment 73 Petr Jiricka 2009-09-29 12:42:04 UTC
There is one other sub-scenario that leads to undesirable scanning:

1. Create a Java SE application
2. Create some valid XML schema in this application
3. Create JAXB binding against this schema. This will generate the sources and build them.
4. Use these generated sources in some other classes of the project (e.g. the Main class)
5. Clean the project (which deletes the generated sources) -> sources are scanned
6. Build the project (which re-generates sources) -> sources are also scanned

BTW, let's really, really use this bug only for the "sources generated by build script" use case - for any other use
cases where the build script does not generate Java sources, separate bug reports need to be filed. Thanks.
Comment 74 jessholle 2009-09-29 13:04:17 UTC
What about the annotation processor case?

For instance, in Java 6 one can even create an annotation processor to be invoked as part of javac that takes

  @SpecialClass
  public class  MyClass
    extends MyBaseClass
  {
    // ...
  }

and auto generates MyBaseClass.java.  Even in less "extreme" use cases, it is common for an annotation processor to
create (and update?) source files as part of javac invocation.
Comment 75 Jan Lahoda 2009-09-29 14:00:18 UTC
Re pjiricka's last comment (desc 74): could you please be more specific about what should happen, in your opinion?
Specifically, should there be error badges for missing classes after step 5 (the classes are missing in reality, are
they not)? If not, why (usually, such situations are reported as P2s, why it is not a P2 in this particular case)?
Should the code completion show the classes that are missing? What should happen after step 6? Should the error badges
disappear? Should the newly appeared classes be visible in the code completion? Thanks.
Comment 76 Petr Jiricka 2009-09-29 15:14:40 UTC
I don't have a strong opinion about what should happen after step 5 - anything we do is somehow "incorrect".
After step 6, the generated classes should of course be visible in code completion, there should be no error badges etc.
Regarding the performance aspect, it would be nice if after steps 5 and 6, sources were not rescanned (after step 6,
everything is the same as was before step 5, so the IDE could reuse what it already knew at that point).
Comment 77 matthies 2009-09-29 15:34:56 UTC
It's not really the scanning itself that is undesirable, but the fact that it invalidates code completion data until 
the scanning is completed, and prevents all kinds of actions (source navigation, hints, refactorings) that rely on that 
data. Of course the scanning also shouldn't consume CPU and I/O resources to the point of significantly slowing down 
other operations.

It might be worth looking at how Eclipse handles such use cases, because it doesn't seem to have these scanning issues 
and users appear to be happy with how it does things.
Comment 78 jessholle 2009-09-29 16:05:36 UTC
The fact that scanning disables code completion and source navigation and louses up error badging, etc, is an underlying
issue behind most current scanning issues.  This behavior is simply untenable/unacceptable.  Instead, scanning should
behave like a transaction -- other threads should be able to continue seamlessly using old data until new data is
provided by the scanning thread.  Sure there should be a means for the other threads to check for a scan in progress,
e.g. so refactoring could *warn* that the results may be inaccurate due to a scan in progress.  Sure there's a question
as to appropriate transaction isolation granularity, e.g. should each source file's scan data be updated it is scanned
or should this be delayed until a package or source root level.  These details can clearly be worked out.  The critical
thing is to have scanning block nothing and never remove old scan data until the new replacement scan data is available
(which should be an atomic replacement).  Anything less than this and NetBeans will forever remain in crippled scan hell.

Once this is resolved, then avoiding unnecessary scans becomes purely a CPU/disk grinding issue.  Assuming one is not
too strapped for CPU/disk resources and the scan thread is not too aggressive, this then becomes a *much* lower priority
issue.
Comment 79 Milan Kuchtiak 2009-10-19 14:28:51 UTC
See also some ideas given by Peter Nabbefeld: issue 174511.
Comment 80 Tomas Zezula 2010-05-12 12:42:33 UTC
For sources generated by the annotation processor the scan is not triggered in NB 6.9 (the annotation processing has to be correctly configured). For freeform see issue #185457.
Comment 81 David Strupl 2011-02-24 08:37:27 UTC
Based on the Comment #78 of the original reporter and also the previous Comment #77 I am marking this as duplicate of #182653. The overall sentiment here is (if I understand correctly) that majority of the users want transactional index and the symptoms described by this bug would not solve the general dissatisfaction with the current state. Only solving #182653 will improve the situation so these reports are in fact the same.

*** This bug has been marked as a duplicate of bug 182653 ***
Comment 82 gfl 2014-10-30 22:55:46 UTC
Just my small input to this issue...I've tried netbeans v8, v8.0.1, v7.3.1, and v6.9.1 and all of the versions have a major performance bug with the project scanning.  The problem is major in that netbeans in unusable during any scan, which appears to get triggered for no apparent reason.  

Using sysinternal process monitor, I can see that netbeans is doing a full rescan of my entire project every few minutes.  

I'm going to try v6.1 of netbeans and if that still causes this issue, I'm done with netbeans.  

When I first reverted to v6.9.1, the problem didn't happen for several weeks, but now the program is worthless due to the performance.  

I've tried turning off all of the code completion settings, but the scan is still a full scan of the project.  I've tried the "scan on demand"...which doesn't work.

Obviously someone at Oracle isn't getting the picture that this is a show-stopper for many developers, and will likely lead me to jump to another IDE.  It can't be that difficult to just only update the changed files, a simple check of the file data or which file was saved would be an easy fix and only scan those files, instead of the ENTIRE PROJECT!

P.S. I'm a very long time MS VS6 user, due primarily to it's performance.  I still use VS6 to open 100+ MiB files as it has no issue with very large ASCII files.
Comment 83 Tomas Zezula 2014-10-31 07:43:55 UTC
In fact the scan is done for minimal set of files.
There are 2 types of update scans in case of the java project:
1st) The project metadata change like classpath change.
Such a change requires a full rescan of all sources in affected project as
the metadata change affects all sources. The sources which are now compilable
may become invalid and vice versa.

2nd) Single file(s) change like save of file (external file modification).
Such a change has 2 phases. In the first phase just the changed file(s) is scanned.
In the second phase the hash of class API signatures from the changed files are computed and compared to the last know API signatures. If changed (the class API has changed and it can affect other files) the usages of the class is found and the classes which are using the changed class are rescanned.
The scope of the second phase can be set in the Tools/Options/Editor/Editor/Hints/Java/Dependency Scanning.

If the full scan happens often it can be cause either by frequent metadata changes (custom build steps), changes in signatures of file with has lots of usages (in this case you can limit the Dependency Scanning) or it can be caused by Annotation Processor.

If you can share some test project or attach the nps file of such a scan I can help.