Bug 133943

Honza will try to cancel up-to-date checks.

Another very annoying thing (which I thought I already filed but cannot find now) is that in a j2seproject if you Run a
class (or Run Project), the IDE will wait until scanning has finished before it actually runs it, just to check that it
actually has a main method! It would be better to just go ahead and run it immediately; at worst you will see a
NoSuchMethodError in output if the class in fact lacked a main method. It is senseless to force everyone to wait just to
better report an unusual error condition.

*** Issue 135301 has been marked as a duplicate of this issue. ***

Pavel had some improvements.

*** Issue 103371 has been marked as a duplicate of this issue. ***

*** Issue 138968 has been marked as a duplicate of this issue. ***

i refuse, that my issue is a duplication of this one, because i say (more radically): (nearly) no background-tasks at
startup... (please see issue 138968).

This has been happening for a while, so I'll throw my input out here based on my experiences and frustrations thus far.

Scanning affects all projects whether it is the one being used currently or the one being scanned. I had the form
modules sources opened in a new installation of 6.5. I noticed it was taking an extremely long time to scan the NB
source structure; varies, so sometimes it is kind of fast and others it takes fffffoorrreeever. I was working on another
pure JSE Java Application project which only has a single class with a main method. I could not use code completion in
the JSE project because of the scanning. Seems if anything scanning should also be on a project by project basis instead
of being queued together if that is the case, or at least put on pause in one project, and moved to the one being used
at a given time using better state management.

-JDK index
--NB suite project index
--A separate JSE project index

So that each one is independent using stacked or hierarchical indexes. An exammple, the JDK had already been scanned,
and the NB sources were being scanned, but I couldn't use this tiny project with CC because of the bigger project being
opened and it was being scanned instead of the one I decided to use.

Also, this should not block all code completion just because of scanning and indexing. Does the project not already have
the files on the classpath at hand? Seems like having indexing pre index things on the classpath in the background is
good as long as it doesn't use as many resources (disk and CPU) as it does now and block as little features as possible
if not skip blocking any of them all together, but then to also have the process grab things on demand as different
features need them. For instance, lets say scanning is taking place over a set of JAR files:
JDK 1.6
commons logging
jasper reports

what ever basically, and it will do them sequentially. 

So, we know many classes are available on the classpath, and many of them will already have been scanned and indexed, so
many can be shown directly in code completion, things that have already been scanned, and things can be shown as they
are added to a model from background scanning which is taking place as the CC is shown...even if a message is embedded
about scanning, though any message about scanning maybe could just be avoided with these mechanisms or better. 

Then, we know that if a user types org. they are trying to use things in the org package, and logic should be able to
locate that rather quickly in the classpaths listed for the project...faster than scanning everything and only showing
when finished and show the packages. If they narrow that down by typing another character that can be located fairly
quickly on the classpath. Too, these things can be indexed as they are accessed, and can be skipped in indexing if a
package or file winds up indexed on demand.

Also, if I open a file, based on the way code completion works now that is, the process could go ahead and index things
within that file, imports, and their packages etc, if they haven't already been done. That keeps the list smaller for
what the, lets call it local indexer (which is still a background task), is having to index on the fly or based on the
recently opened file. Go bottom up to, so if I have
import java.util.GregorianCalendar;
import java.swing.JFrame

then make sure those are indexed first (GregorianCalendar and JFrame), next check util and swing, and if needed, let the
local indexer grab these and handle them *now*, and then let the background processor keep doing its thing if it is
running, yet should skip anything added to the index by on demand features. Of course, the currently opened file should
be scanned before any of the other classpath.

Also, it seems scanning and indexing those files on disk is sucking up a lot of CPU resources, either the thread isn't
yielding enough or sleeping long enough and is chomping up the disk while this takes place not allowing other processes
to run great nor the IDE to be as responsive as it could be. This hurts laptops the most.
 
Too, maybe the first thing indexing should do is to take the file timestamps, build a list of all paths and file names
which is much faster than all indexing will have to do, and then start scanning and indexing, and these values are what
can be used to make sure they don't need to be done again (I haven't seen all the current indexing is doing at the
source level). 

Basically, all the timestamps, packages, resources (.java, .gif, etc) can all be setup when a JAR or directory or
something is added to a project or when a project is first opened, in the background of course, and then indexing can
take place on all the details in the files after that. Then this other keyed information can be used by the on demand
features to update indexed material to never be updated again unless the timestamps or file sizes are changed.

That then brings us to the point of making sure indexing doesn't have to take place again unless a file or folder
timestamp changes, file size changes, etc.

Also, I see it says the target is 6.5, but I don't think anything has been done in 6.5 for this issue, not large changes
it seems, as I'm using 6.5 final release while commenting. Are there any plans for this for 7.0?

I'll throw my two cents into this issue as well. 

Making the IDE usable during scanning is a noble goal. However, when the IDE tries to be smarter than the user and does
not allow the user to direct the IDE what to do, then it's a big problem. 

My problem is this : every once in a while, when I have a big build to run on my project, I like to kick off an external
Ant build and (hopefully) continue working inside the IDE. With the automatic scanning feature always on, and when it's
not possible to disable it, the scanning kicks in and for the next 20 minutes the IDE is completely unusable, not only
that code completion is not working but the whole UI freezes up, as there don't seem to be too many CPU cycles left over. 

It seems to me that the simplest approach to addressing my problem in particular would be to have an option to
temporarily disable the project scanning. Then, I can kick off the external build and continue working. Ideally, when
the IDE doesn't launch a rescan, I might even have access to all the metadata that the IDE might have generated in the
previous scan. Even if it's not possible to keep the old metadata around when automatic scanning is disabled, being able
to do something while the build is running would be a huge improvement. To make matters worse, the intensive IO that the
scan is producing is slowing down my build and I often just have to kill the IDE, as it not only help me along but slows
down even the external build. 

A secondary positive aspect of allowing the user to select when to kick off the scanning is that if I open a big
project, I might want to do a thing or two, and then kick off the scan manually and then go get a cup of coffee. In the
current situation, when I fire up the IDE, I cannot do absolutely anything until the IDE decides that it's time for me
to get some work done. Drives me up the wall !!!!! 

As much as I dislike Eclipse, it seems that it has this part right. It has the ability to build the workspace
automatically (which seems to be the equivalent of the automatic scanning). 

Mostly independent of j2seproject, excepting my one comment about Run Project and the main class.

akochnev: if running a build triggers classpath scanning there may be something misconfigured with your projects, though
I cannot say what without a reproducible test case. The IDE scans sources, not build products. A rescan would be forced
by a build only if the build changed source files, or changed classpath JARs not associated with source projects - in
both of which cases the IDE really needs the rescan. For example, a Maven build can download artifacts not previously
available in the local repository, in which case the IDE rescans your project's sources - correctly, because it can then
verify that a project previously marked with many errors is in fact OK (or show genuine errors in the source which were
previously obscured).

I still find it interesting that we are still considering it possible to know everything that is wrong, instantly and
that scanning the world is a viable way to manage asynchronous updates and changes to the IDEs view.

I really feel that this type of feature works fine for small projects with limited outside, asynchronous changes happening.

I really know well, that what is in place now, does not work for people with many projects sharing many libraries which
might get built or updated asynchronously.  Yes I want the IDE to reflect what it knows about the compatibility of what
is on the classpath vs the current source file.  I DO NOT need the IDE, nor do I have time for it to go poking around
through 1500 source files in 40 projects to tell me that everything is okay after I updated a library jar.

I will build the projects that I need to know about compatibility with in a different place and time.  The IDE should
always allow editing, building and running of the current project, no matter what!

I am way smarter than the IDE about what incompatibilities I introduce to signatures.  I can much more quickly address
those than scanning could every hope to.

Please, please, please turn scanning into a non-locking background task if you really think it's valuable to me.  I'd
prefer to be able to turn it completely off except for the current project that I am editing a file in (as opposed to
the active project).

(based on Greg's feedback): Unless the IDE only targets and audience with only small projects with only a few projects
open, should this be a P1 ? 

Jesse : indeed, the rescan kicks in if I change source files; however, it also kicks in if I change the classpath of the
projects that my source files depend on. This is the scenario : I have project A depending on the products (jars) of
project B. I run a build that cleans the jars for A & B. The second B.jar is cleaned, the IDE starts scanning the source
for A trying to mark all classes that depend on B.jar as errors. In the meantime, B finishes building, B.jar is
produced, and the IDE then tries to rescan A once again, this time to update all source files that depend on B.jar that
they are indeed correct and don't have any errors. 

You can see how this situation gets out of hand very quickly, when I have 20 projects with dependencies on one another,
as the IDE will try to rescan the source for all projects at least TWICE (that is, assuming that there are no shared
dependencies), in the process completely discarding the first scan's information after the build products start showing
up. All in all, a very bad place to be in. 

To your point that after the jars change, the IDE really needs a rescan, that is indeed the case. However, the
intermediate rescan that gets triggered when the jars are removed is completely useless in this scenario. Additionally,
until the final scan is done, it seems that the metadata from the previous scans should be valid and the IDE should be
usable (and not locked up or giving a "scanning in progress" message) in the meantime. 

I would like to know it is scanning. I just want it to be usable. If I know it is scanning and maybe some changes
haven't been picked up yet then I can at least have a cue and a clue as to why that may be. Regardless though, things
need to be usable while all that takes place. Is it a javac thing? Is that library not thread friendly? Are yields not
used when they need to be used in such an environment? Too, there need to be some strategically placed sleeps in there
to make it not use up too many disk resources.

akochnev: the behavior you describe definitely sounds like a project misconfiguration. If b.jar is associated with a
project, the IDE's scanner should not be paying any attention to what b.jar actually contains or whether it currently
exists. I would recommend you use the NetBeans Project Metadata Inspector module to help determine why b.jar is not
being correctly associated with B/src. Again, if you can isolate this to a reproducible test case, please file it
separately. That would be a project issue, not a Java infrastructure issue.

greggwon: I don't think anyone would disagree that it would be desirable for the IDE to use "last-known-good" parse
snapshots, at least for read-only operations such as code completion and navigation. (Using stale information for e.g.
refactorings is probably unacceptable.) And that is just what tpavek originally requested in this issue.

Currently the IDE tries to maintain an up-to-date scan cache for open projects. Among other things, that means that
(modulo bugs) error badges in the Projects tab should appear if and only if you have an actual problem. This seems a
reasonable behavior; if you are not interested in the status of other projects, close them. Presumably it would be
possible to suppress scans for some but not all of the open projects, but that would seem to be a rather confusing UI.

Yesterday I was doing a lot of Maven2 POM file editing in NetBeans 6.5.  I was tweaking plugin configuration and adding
dependencies on Axis for JAX-RPC.  Every time I made a change and saved it triggered "Scanning projects...".  It was
usually quick so it didn't bother me too much.  This might not be a problem area, but I thought you might be interested
in knowing about ways to trigger scanning. 

There's a test case for this in http://www.netbeans.org/issues/show_bug.cgi?id=134990 (Apr 28)

Vita did necessary changes in infrastructure:
http://hg.netbeans.org/jet-main/rev/a440c5cbbd19

I had an additional negative experience with respect to scanning and wanted to share it in the appropriate issue.  I
searched and found issue 144109 - and added it there.  But I'm wondering if that issue isn't really a duplicate of this
one?  I'll leave it for developers to decide that.  But, in the meantime, I want to recount my observation:

Observation (valid on all recent daily builds up to and including the 5/27 build): on my freeform project with
3800 java source files and about 70+ (maybe as many as 100) 3rd-party libraries, *whenever* I do a "Clean and Build Main
Project", I patiently wait 40-50 seconds for the whole project to rebuild.  No problem - that is expected.
ut if I then hit "Run Project", I invariably get "Please wait, classpath scanning in progress..." for the next minute or
so.  WTF?  I just built the project from scratch - why can't I run it???  The friggin' "classpath scanning in progress"
is taking longer than the entire build!  I know, I know - why do I have to do a complete clean/build so often?  Probably
I don't, but that isn't the point.  I should be able to run my project (a straight Java app) as soon as I've finished
building!  I shouldn't have to wait 1-2 minutes.  It's annoying as heck.

small correction to my last comment: my project is *not* a freeform project - it's a standard java project with existing
sources.  sorry about any confusion.

There are several things in this issue.
1) The project cannot be started during scan which is obvious bug, which I will fix it as Jesse suggested (do not check main method during scan).
2) The better usability during scan (improved response time). Currently there is a "BIG" lock holding by indexing infrastructure even when its' not needed. 
While writing the indexing API I've change it to minimize this lock for languages using IndexingSupport. But  java does not uses the IndexingSupport as it 
maintains it's own performance tuned index. Fixing this problem may improve the response time of the java module during the scan specially on multi core 
cpus.

I think there are generally two areas for improvements:

1. optimizing scanning tasks for better performance, which would decrease the overall time and resources consumed by
scanning/indexing

2. identify tasks/actions that do not necessarily need index data, but are currently blocked by indexing and rewrite
them so that they can run independently on scanning tasks

I'd like to dedicate this issue as an umbrella for issues that fall into #2 category. The #1 category issues are filed
separately under editor/parsing & indexing.

When dealing with truly large bodies of code, scanning will *never* be fast enough for it to be acceptable to block
navigation (and debugger attachment!!!) while scanning.  These activities must be allowed at all times and must simply
use the previous scan data plus data from the current scan where possible.

Showing off the "great NB IDE" to others and then being unable to do anything useful for many minutes while code
scanning is occuring (and reoccuring in cases, sometimes without any indication in the progress bar!) does not cast NB
in a good light.  Worse, it is the source of a lot of forced coffee breaks...

Created attachment 83267 [details]
Go to type - blocked while scanning

Created attachment 83269 [details]
nb scan in 6.5

6.7 performance regression see attached movie

One of the main problems isn't just that it is scanning and indexing and using up the system resources, or that there
are things which the IDE invariably needs to be able to perform a certain tasks correctly, but in many cases that the
IDE could pause a large scanning task, scan what is required for a given request as it comes in from different
features/modules, mark in a queue that something has already been handled and thus doesn't need to be scanned and
indexed again (could be removed from such a queue versus marking finished as that is similar in nature, but maybe there
will be some meta information which is needed for locking, checking, etc). 

Maybe a better way to explain it: Technically a lock could be placed on items in such a queue as one which could be used
to store paths to those things needing to be scanned and indexed, the other non-locked items can continue to be scanned
and indexed by the background thread/scanner/indexer, and the immediately needed resources can be handed off to another
thread/scanner/indexer set aside to handle such immediate needs to be scanned and indexed right away instead of waiting
for non-immediate needs to finish; this the case when code-completion or refactoring needs to scan a particular item
which may branch out and require some other resources, but which are typically limited in scope.

Once those are finished, they are marked as completed or removed from the queue so the background process doesn't
reprocess those items which have already been taken care of by the immediate scanner.

Of course any optimization tricks would play into this type scanning and indexing as well, so those type things should
be able to be worked on and applied to both the background and immediate scanners/indexers rather seamlessly.

I've removed the "BIG" lock as I had described in comment 2 11:47:53 +0000 2009.

I've removed the "BIG" lock as I had described in comment 2 11:47:53 +0000 2009.

Will this:
http://forums.netbeans.org/topic14193.html

be solved with the fix perhaps?

I cannot comment on the issue any longer -- when last I left it, I was not seeing any scanning issue except if my ide
had crashed. Then sometimes I could get a  long scan. Probably something gets corrupted?

Generally speaking, and maybe this pertains to my platform, Windows XP, I am not seeing a scanning issue.  Are you
getting all the info needed to debug this -- what was the developer doing before the scanning issue started? Had there
been a crash of either the glassfish server (assumed) or the ide?

Regarding locking...I have seen issues with locks applied to the glassfish server after a Netbeans crash. I had to go to
the file location to clear the lock. I cannot comment on the exact path at this point in time. Maybe something like that
is happening where the scan is not completed because the file(s) are locked?

I think it's a good idea to queue tasks that need to be done, rather than default scanning everything.

..\Wendy

*** Issue 167674 has been marked as a duplicate of this issue. ***

I filed a separate issue, issue 181652, to track the issue that Run Project (and Run File) should not block and wait for scanning to finish.

I mostly experience this problem when working with a grails application and other projects at the same time.  If i open Netbeans and it has a grails project open by default i cannot really do anything until it has scanned the grails project.  I used the ScanOnDemand feature and REALLY liked it but my team had some major issues with the plugin and source control and we had to remove the plugin.

Hope this helps

*** Bug 182280 has been marked as a duplicate of this bug. ***

*** Bug 193391 has been marked as a duplicate of this bug. ***

This is biting me too.  NetBeans 9.2, Ubuntu 12.04

Under normal conditions, background scanning makes Netbeans very sluggish on a netbook (this one with a single-core cpu).

This is bad.

But at some point, my session will grind almost to a complete halt.  I have in the past killed it externally.  This time I just waited (and waited).  After half an hour watching it closely, I was able to properly quit such a grinding session.

Top shows java taking 90+% of the CPU.

This is very bad, guys.

From my point of view, I don't see what benefits this background scanning give me, but I do see how it shuts me down and wastes my time.

My guess is this software is built to be sluggish on big fast multi-core machines.

Listen up!  If you make it run reasonably well on a small, slower machine, it will be blazing fast on a big one.  WOW WHAT AN IDEA!!

Sounds a problem with file system. The background scan works fine on 3 years old Mac Book Pro (i5, 4GB). Also the background scan is suspended during editor operations like navigation, code completion, etc.

Tomas Zezula: "background scan works fine on 3 years" no, it has been horrible for years. The background scan is and has always been slower than Eclipse on a large project.

(In reply to comment #39)
> Tomas Zezula: "background scan works fine on 3 years" no, it has been horrible
> for years. The background scan is and has always been slower than Eclipse on a
> large project.

You got it wrong, he means the hardware is 3 years old...

On single core CPU the background scan which runs in a background thread blocks other processes as there is a single executor. The IDE suspends the scan during editor operations but it blocks other programs and builds. The indexing of java is in fact compilation and analyzation which is time consuming. On single core CPU the situation is the same like running the compilation in the same time. Even the thread has lower priority the linux kernels increases its dynamic priority as it's highly communicative thread.
Lot's of performance improvements in NB 7.2 are for 4+ cores as it's now most used configuration, the IDE scales with cores and heap memory.

>You got it wrong, he means the hardware is 3 years old...
Yes.
I am taking about NB 7.2.

(In reply to comment #37)
> This is biting me too.  NetBeans 9.2, Ubuntu 12.04
> 
> Under normal conditions, background scanning makes Netbeans very sluggish on a
> netbook (this one with a single-core cpu).
> 
> My guess is this software is built to be sluggish on big fast multi-core
> machines.
> 
> Listen up!  If you make it run reasonably well on a small, slower machine, it
> will be blazing fast on a big one.  WOW WHAT AN IDEA!!


Netbeans is not made to be used on netbooks, and netbooks are not made to run this kind of application. 



P.S: NetBeans 9.2? You mean 7.2?

@ceclock,
I don't necessarily agree with your statement that NB isn't made for netbooks.  The big/bad desktops we all used NB with 12+ years ago were inferior, in terms of performance, to today's lowly netbooks.  I'm pretty sure that if Steve ran NB 3.5 (or whatever was the current version in 2000) on his netbook, he'd be pretty happy with the performance.

So what's changed?  Well, features, bloat, and sloppy coding (and design), of course.  Just as with Microsoft Word, every version of NB over the years added more features - less and less of which are actually used by the majority of user.  Additionally, as the users of NB became "dumber" (it used to be that almost every user of NB was a hard-core developer - nowadays, judging by the questions in the forums, most don't even know the difference between the language and the IDE), NB developers added this and that "assist", so that their new target audience could be effective in the environment.  Finally, there were probably features added over the years that were just "pet projects" for one of the NB developers - something that was important to them - but unimportant to 99% of the rest of the user base.  The plugin architecture of NB helped keep the bloat somewhat at bay, but really not enough.

Over the years, I've always just downloaded the minimum IDE (well, there was a brief period at the beginning of this century when I had to do J2EE stuff - I still have nightmares about those days).  Today that means downloading a 76MB compressed file.  I just went into the NB archives and took a look at the earliest version available - 3.5.1 - take a guess at its size: about 28MB!  About a third the size of today's "basic" NB, yet I'm doing pretty much the same work as I used to do in 3.5.1 days.  I'm half inclined to download that version...then again, it doesn't support the current version of the Java language, so I won't.

I guess for some, NB - even in its most basic form - has gotten too slow and cumbersome.  I'm running the latest 7.2 "Java SE" release on a pretty powerful 8GB, i7-based MBP with an SSD and NB's scanning - although it rarely gets in the way the way it used to in 6.x days - is still an annoyance (the CPU gets hot, my CPU fans go crazy, the machine just feels like it's laboring) - it annoys me because I suspect that it's not really doing anything that I care about.  Me posting this is not meant as an attack on NB.  I'm obviously still using the IDE.  I guess @ceclock triggered my realization that - for me at least - NB isn't doing much more than it did 12 years ago - yet it requires ever greater hardware specs to do it.  Something I always advocate in GUI software development: make sure the GUI looks reasonable on the minimum supported screen resolution.  Maybe NB developers should do the equivalent for a week: be forced to develop s/w on a minimum hardware configuration - such as a netbook (but connected to a big monitor) - just to find the pain points and make it a priority to get rid of them.

Just my $.02.

I am curious now: is there a version of NB in the 3.x range (or a fork someone did) that could handle Java 7 syntax?  If I recall correctly, versions prior to 4.x didn't even use ant for building (that in itself would have made the IDE faster/lighter, no?)  If so, I could compare/contrast using our real-life application (4,200 java files J2SE app).

two points :-
1) my main machine has 12 gigs of ram and 8 cores and still I wait for NB scanning. 
2) Eclipse works fine on my laptop and I don't see these scanning issues in Eclipse on my desktop either. 

Given these points plus the fact that NB wants to pretend Android/GWT doesn't exist as a platform. Eclipse has become the major IDE used here.

I'd concur that scanning is still too slow.  It does seem somewhat improved in 7.2 over recent releases before it, which were really bad.

It has improved somewhat in terms of which UI features it locks you out of while scanning and how well "go to type" and related features work while scanning, but there still appears to be room for improvement (e.g. "go to type" should never fail for a class that was previously scanned, even while rescanning/updating the data for that particular class -- until the rescan/update of that class' data is complete the previous scan's data should be used for that class in a very atomic, db-commit-like fashion).

What *really* gets me going lately, however, is that we have our own annotation processor which generates base classes for the hand-written classes which are in the project source tree.  This processor works great from javac and Ant.  NetBeans' scanner throws odd-ball exceptions and assertions for some classes where such annotations are used, though.  Given that our annotation processor has no such issues elsewhere and given the lack of useful feedback provided by the resulting error messages we have no real option other than to throw up our hands and give up on proper scanning.  If any NetBeans folk are at JavaOne this year I'd happily demonstrate this mess -- but the code is both huge and proprietary, so I can't pass it around.

I think there are too many things going on behind the scene when you are working in a project with Netbeans. I agree that it is heavy, but it depends on the project(s) you have currently opened on the IDE. I think one of the worst cases is when there are many projects depending on each other. 

Think with me... Netbeans as the application is not so heavy if you open it "clean", with no projects loaded. Try not to have many inter-project dependencies.  Generate libraries if needed instead of setting a project as a dependency. Does it improve the speed or the same issues continue to happen?

One huge project depending on many jars (even if that project is the only one open) makes NetBeans squirm pretty good -- especially when you mix in annotation-processor-based source generation.

@ceclock,
As jessholle just noted, a single large project (e.g. mine: 4200 java files) with lots of jars (about 100) does a pretty good job of keeping NB busy.  No need to open multiple projects, have dependencies, etc.

That's why I'm now so curious as to whether a modified NB 3.5x would be faster, given today's fast machines.

I guess my project "wins".  It's got almost 8300 Java files, many of them fairly large and a substantial number using annotation processing to generate still more Java files.  The project depends on over 50 jars.  The output is over 55MB of class files (prior to compression).

This is a free-form project so as to integrate in well with our Ant-based build framework.  [The freedom to integrate into which is probably the single biggest thing that keeps me using NetBeans.]

Is this an ideal arrangement for a project?  No.  That said, it's *highly* non-trivial to pull apart at this point.  [I neither managed to pull it apart nor produced the situation.]

This project does manage to highlight all that falls apart in NetBeans scanning, though.  It is part of a much larger network of projects -- and having those all open does make things work (e.g. updating a base project can causes a cascade of rescanning), but things are plenty bad with just the one project.

Our project consists of 7508 Java files. It is a free-form project with a large number of custom Ant tasks. These ant tasks are shared between the IDE, nightly builds and production, this flexibility is a big plus for NB but the project scanning issues are a big negative.

The lack of support for Android/GWT has forced us to move much of our development to Eclipse while the Ant integration & flexibility is lower, I just don't see the same project scanning issues at all in Eclipse working on the same source code.

It's clear many of us are in the same boat, more or less.

This should be ample evidence to the NetBeans team that they still have a long way to go when it comes to code scanning improvements!

I just started to post about 10 paragraphs about why I agree that this is still not fixed. I'm just not sure, given the state of the "Netbeans World", the state of the "Oracle World", and the "Java community", whether it would make any difference, at all to provide feedback here.

At this point, I think the only thing that will make sense for many people is to switch to Eclipse (GAWD that's just frightful to think about), or for a group of people to finally "fork" Netbeans and just purge the time wasting scanning code.

Clearly the larger Netbeans community wants this issue fixed.  Obviously the Netbeans team keeps making attempts to fix it.

Practically the pace of change is abysmal.  Continuing to challenge the team with comments just continues to deteriorate into bashing and blaming, because there just seems to be no practical reason to many of us, as to why this issue could not be just fixed.

Now that I'm back to five paragraphs on this attempt, I'm wondering why I keep going.  Maybe it's because I actually do care about what happens here.  But practically my ability to be civil is waning. I no longer how to be constructive about this when I see so many good ideas and comments being ignored by the Netbeans team.

I think part of the issue here is that the NetBeans team sees their codebase as representative of Java development codebases and even as a stress-test case.

From my glance through NetBeans code, I'd beg to differ.  Their code is mostly broken down into relatively small, tight modules.

Sure most *newer* projects I deal with are nice and small modules, but we have code built up over the last ~15 years -- most of this is still in use in the latest product.  This even includes code from Java 1.x days (with some changes, of course).  Large projects happen and remain as in cases the cost to break old codebases down into small modules is simply prohibitive.  Moreover, large projects are clearly far less of a problem with other IDEs and tools!

NetBeans also does not appear to use annotation processors to generate additional sources -- so that's another area where their own codebase can't represent what we're seeing.  [Yes, I know we're somewhat unusual in this, but there are several JavaOne talks advocating this very approach this year, so if we're nuts we're not the only ones...]

NetBeans also doesn't use free-form projects whereas we do.

And so on...

In short, just because NetBeans performs well for developing NetBeans itself does not mean it works well for the rest of us!

Where it is possible to share projects where these problems occur, that would be very useful -- or subsets of those projects, i.e., including the parts causing those problems. Then we can attempt to identify how to solve these specific problems.

Unfortunately, I can't share my projects.  The closest I can do is offer for someone to crank up some verbosity and otherwise lightly poke around on my laptop (sorry, only dual not quad core) while I'm at JavaOne.

I'd also certainly be willing to crank up verbosities on my own or test with special versions of NetBeans modules with extra logging code added.

This is especially the case of the annotation processor scanning issues.  The existing error messaging in this area is sparse and cryptic -- giving me no real information other than "its broken".  When testing out betas or nightlies of new NetBeans versions there appears to be a lot bigger reaction to these issues whereas in release versions they're quietly swept under the rug in the IDE log.  For betas and nightlies I've gotten many exception reports -- and passed them along via the Exception Reporter (often anonymously in my hurry).  I can only assume that the exceptions are meaningless to the NetBeans team as well -- as nothing ever came of all these reports.

Let me make just one point please:

> I think part of the issue here is that the NetBeans team sees their codebase as
> representative of Java development codebases and even as a stress-test case.

That's not true. While NetBeans codebase itself is of course one of the test cases we use, it's not the only one and we test with a wide variety of project setups. And there is evidence - look at the NetCat community acceptance survey for the past several releases:

http://qa.netbeans.org/processes/cat/72/ca_results.html
http://qa.netbeans.org/processes/cat/71/ca_results.html
http://qa.netbeans.org/processes/cat/70/ca_results.html
http://qa.netbeans.org/processes/cat/69/ca_results.html

You see that 70% of 7.2 NetCat participants rate performance as excellent, which is by far the best result among the past several releases. Also the number of "excellent or usable" answers is the highest. So clearly NetBeans team is making progress that is visible and apparent to the majority of users, not just to the NetBeans team itself.

This is of course little consolation to jesshole and nleck who have freeform projects and who don't see the improvements themselves, and is not an excuse or reason to stop improving performance. It sounds like the freeform project is the next important case to focus on, and we need to acknowledge the performance problem in this particular case, and in other situations like that. 

But please let's be fair in assessing the NB 7.2 performance improvements - it is NOT true that NetBeans team only focuses on its own codebase. Clearly the vast majority of users benefit, though I can understand that is must be frustrating if you fall into the minority who does not.

(In reply to comment #37)
> But at some point, my session will grind almost to a complete halt.  I have in
> the past killed it externally.  This time I just waited (and waited).  After
> half an hour watching it closely, I was able to properly quit such a grinding
> session.
> 
> Top shows java taking 90+% of the CPU.

Which sounds like a possible memory leak (which may or may not be related to background scanning). A heap dump is necessary to evaluate such situations (otherwise we cannot even guess whether that is something known/already fixed or something unknown so far).

> 
> From my point of view, I don't see what benefits this background scanning give
> me, but I do see how it shuts me down and wastes my time.

Let me give an overview of what the Java part of the background scan produces and how is that used (at least as far as I know). There are two main artifacts:
1. an index containing information about classes in the sources (for each class, which other classes are used, superclasses, methods, etc.)
2. compiled classfiles for the sources (and dumps of the errors found in the files).

The first part is used (among others) by the refactoring (e.g. when renaming an element, to find out which classes need to be refactored), code completion (e.g. to show subclasses Throwable for "throw new"), the "is overridden" annotations, Go to Type/Symbol, and other features.

The usage of the second part is often a bit more hidden, but not less important. When a Java source code is opened in the editor, it typically refers to other classes - and the information about these classes is read directly from the compiled classfiles. The parser (javac in NetBeans' case) must read this information from somewhere, otherwise it could not mark errors in the editor (and do many more things). The other ways to get this information about the related classfiles without producing the cache known to me:
-use build artifacts - has a disadvantage that if there are no build artifacts, there is nothing to parse against. Also, a newly added method/class/field would not be visible until the project is rebuilt.
-parse the corresponding source - is very slow (while parsing the one file in the editor, many more would need to be parsed) and consumes a lot of memory (could easily lead to OutOfMemoryError for non-trivial projects).
-combine both of the above - use source when it is newer than the corresponding classfile - still could lead to OOME for cleaned project

The second part is also used for other features, like Compile/Deploy on Save and error badges in the projects tab.

(In reply to comment #44)
> So what's changed?  Well, features, bloat, and sloppy coding (and design), of
> course.  Just as with Microsoft Word, every version of NB over the years added
> more features - less and less of which are actually used by the majority of
> user.  Additionally, as the users of NB became "dumber" (it used to be that
> almost every user of NB was a hard-core developer - nowadays, judging by the
> questions in the forums, most don't even know the difference between the
> language and the IDE), NB developers added this and that "assist", so that
> their new target audience could be effective in the environment.  Finally,
> there were probably features added over the years that were just "pet projects"
> for one of the NB developers - something that was important to them - but
> unimportant to 99% of the rest of the user base.  The plugin architecture of NB
> helped keep the bloat somewhat at bay, but really not enough.

I would also say that the projects the IDE is expected to handle grew bigger, but I may be wrong. As far as the features go, if some of my "pet projects" is used by me and noone else, I am ready to move it to a private module or make it only avaible on option.

> I am curious now: is there a version of NB in the 3.x range (or a fork someone
> did) that could handle Java 7 syntax?  If I recall correctly, versions prior to

None that I would be aware of. I could try to dig out the 3.6 sources, if you would like to try. Just as a warning, when we wrote the current java infrastructure, it took several (quite a few) developer-years to evolve it from a prototype into something that was partially usable.

But, you might try this:
-go to Tools/Options/Editor/Hints/Language=Java and unselect Dependency Scanning
-install Scan on Demand:
http://wiki.netbeans.org/ScanOnDemand

That should make scanning much less intensive, but should (I hope at least) not make any feature "worse" than in 3.6 (there were no project error badges, no compile on save, no refactoring in 3.6).

> 4.x didn't even use ant for building (that in itself would have made the IDE
> faster/lighter, no?)  If so, I could compare/contrast using our real-life

IIRC, 4.x used ant for building. I would be suprised if just removal of ant would make background scanning significantly faster (but could make the projects much less flexible, which could possibly prevent some situations that are difficult to handle for the background scan).

> application (4,200 java files J2SE app).

(In reply to comment #54)
> I think part of the issue here is that the NetBeans team sees their codebase as
> representative of Java development codebases and even as a stress-test case.

I don't think we see NB as the only or even the most typical example of a large project (even though there are big projects other than NetBeans consisting of relatively small modules). Admittedly, it is used a lot (because NetBeans developers often develop in NetBeans), and has a somewhat privileged position, because we are really developing the code (which is somewhat different from just using a project for testing). I don't think that is a big problem: there are countless problems that would show up in any big codebase, that were first found while developing NetBeans and fixed. Surely that does not mean there are no other problems - but we use what we have and get.

It is also used to certain level as a stress-test: currently, the NetBeans "main" repository contains 47146 .java files, and uses a lot of edge-case Java features. So opening a significant portion of NetBeans puts some stress on various parts of the IDE.

Also, I wouldn't say that having (say) 800 small modules is necessarily easier for the IDE than having one or a few big ones - there are a lot of aspects in which having many more modules is making things worse. The main advantage of having small modules is that one can open only the "interesting" ones. But I would be surprised if the closure of the projects I have opened each day would consist of less than 8000 files.

> 
> From my glance through NetBeans code, I'd beg to differ.  Their code is mostly
> broken down into relatively small, tight modules.
> 
> Sure most *newer* projects I deal with are nice and small modules, but we have
> code built up over the last ~15 years -- most of this is still in use in the
> latest product.  This even includes code from Java 1.x days (with some changes,
> of course).  Large projects happen and remain as in cases the cost to break old
> codebases down into small modules is simply prohibitive.  Moreover, large
> projects are clearly far less of a problem with other IDEs and tools!
> 
> NetBeans also does not appear to use annotation processors to generate
> additional sources -- so that's another area where their own codebase can't
> represent what we're seeing.  [Yes, I know we're somewhat unusual in this, but

Actually, we do have such an annotation processor - the NbBundleProcessor. Which does not necessarily mean we see the same problems as you do. But there are/were problems related to this AP.

> there are several JavaOne talks advocating this very approach this year, so if
> we're nuts we're not the only ones...]

(In reply to comment #53)
> At this point, I think the only thing that will make sense for many people is
> to switch to Eclipse (GAWD that's just frightful to think about), or for a
> group of people to finally "fork" Netbeans and just purge the time wasting
> scanning code.

I am not convinced "forking" would help anything (is there a reasonable patch fixing scanning issues that NB developers are refusing to accept?). Also, it is somewhat unclear what exactly do you mean by "purge [snip] scanning code". Probably not just disabling the scanning, as that would disable most of the Java editor features. So, if you could clarify (ideally, using a patch), that would be awesome.

> 
> Clearly the larger Netbeans community wants this issue fixed.  Obviously the
> Netbeans team keeps making attempts to fix it.

Thanks for recognizing that.

> 
> Practically the pace of change is abysmal.  Continuing to challenge the team
> with comments just continues to deteriorate into bashing and blaming, because
> there just seems to be no practical reason to many of us, as to why this issue
> could not be just fixed.

I would say the reason is that making the scanning blazingly fast without breaking other features and while keeping the overall quality is not easy. At least that is what I think. I patch disproving me would be absolutely great.

>From my point of view, I don't see what benefits this background scanning give
> me, but I do see how it shuts me down and wastes my time.
In fact all the editor features depend on scanning, without it there will be no find usages, refactoring, code completion, navigation, etc.

As described deeply in comment #59 the java indexer does compilation of java sources + dependencies. Most of the time is spent in the javac attribution which cannot be made faster. There are some ways how to speed it up by parallel IO prefetch (NB 7.2 for multicore CPUs), caching in memory, etc. The reason why big single root project is worse then several small projects is that for big source root the caches need to be cleared.

The annotation processors generating sources (comment #46): Even the NB runs internally the APs by javac it runs the AP in incremental way and the AP has to be able to handle incremental compilation, this is the biggest difference from command line build. Also be careful to correctly pass the originatingElement. NetBeans supports EclipseLink AP which does similar thing.

The freeform (Ant from existing sources) project is unfortunately not actively developed. The last feature integrated into it was partial support for AnnotationProcessor. I don't expect big changes in it, at least now.


There were big improvements in the scan area in NB 7.2:
1st) The biggest change is that indexing is transactional, the editor features work even during scan.
2nd) Improved speed of indexing, as far as I remember ~ 20%, depends on project type, structure and even hardware (several improvements require multi core CPU).

There are still things to improve mostly in situations when some component requests scan or invalidates an index causing useless scan. Regarding the java indexer time. Most of the time is javac attribution (more than 1/2 of java indexer time) as explained in comment #59

There are several pieces of IDE behavior which require meta-data.  What the scanning is about, is how to first build, and then maintain that meta-data.

Code completion needs "names" and "types".

Searching for "used by" or "defines", requires "names" and "types".

Annotating the navigator with error badges requires "names" and "types", but, and this is the important part, it pushes out scanning to "all" dependent modules/projects, to recompile them, making sure that they don't have any problems.  This is where the larger issue is, I believe.

This feature, for me, is worthless.  I build my products myself, to deploy them.  I know, everytime, whether or not I have problems to fix, because our distribution process validates this.  I will never "stop" building my software before I distribute it, because I have to have the "jar" files to do that.

Why does netbeans insist on rebuilding dependent modules/projects?  I only need it to keep meta-data, active, for depended-on modules/projects for the current project.

If we start at this view of the "world", then we should be able to "fix" things that keep this from being "blazing fast".

Let's say I am working on a project's source file, and I find, "I need to use the functionality in another project X, but I need a slight variation on a method signature".

So, I might navigate to that class by typing the Class name, a space and a variable name and a semi colon, to create a non-error line (Netbeans doesn't seem to have short circuit evaluation that says, there were no syntax errors, the user started typing, so the current line is a new statement, I should always assume that any action that they want to take on that line ignores syntax errors), and then Shift-clicking the type name, or something.

Once I have edited that class to fix the problem, I'll want to "build" that changed class.  Netbeans should be able to do this automatically when I "leave" that editor window/tab, since it's already doing in place compilation to provide its features.

When I switch back to the window/tab that I was working on, the meta-data should have the change in the class visible, and ABSOLUTELY NO OTHER BACKGROUND WORK SHOULD HAPPEN!

(In reply to comment #58)
> Let me make just one point please:
> 
> > I think part of the issue here is that the NetBeans team sees their codebase as
> > representative of Java development codebases and even as a stress-test case.
> 
> That's not true. While NetBeans codebase itself is of course one of the test
> cases we use, it's not the only one and we test with a wide variety of project
> setups. And there is evidence - look at the NetCat community acceptance survey
> for the past several releases:

I personally stopped providing input to the quality of the "newest" netbeans release after 6.5.  Up to that moment, I had tried to engage developers about things that were bothering me, to no avail.  At that point, it was pretty much useless for me to argue and complain, as I could see that developers had already made up their mind about the credibility of my complaints about free-form projects along with others.

I think that you are erroneously believing data which has no real reflection on the reality that your user-base is experiencing.

Gregg, you write: "I could see that developers had already made up their mind about the credibility of my complaints about free-form projects along with others."

Did you see the comments above where it was pointed out that, yes, we do need to work on free-form projects and that, yes, there are problems with free-form projects and that therefore, yes, your complaints about free-form projects are valid?

>Let's say I am working on a project's source file, and I find, "I need to use
>the functionality in another project X, but I need a slight variation on a
>method signature".
>....
>Once I have edited that class to fix the problem, I'll want to "build" that
>changed class.  Netbeans should be able to do this automatically when I "leave"
>that editor window/tab, since it's already doing in place compilation to
>provide its features.
This is exactly what happens when you leave tab or save the editor content.

>When I switch back to the window/tab that I was working on, the meta-data
>should have the change in the class visible, and ABSOLUTELY NO OTHER BACKGROUND
>WORK SHOULD HAPPEN!
This is not true.
1st) The update of cache for the changed file has to happen in background thread not to block
the UI. It may still be running.
2nd) More interesting one. You changed the signature of a public method. Not only the file you have changed but also all the users of the method need to be recompiled as the change may resolve or introduce new errors (error badges in the project tree, CoS status, etc.) If you change a method which has 100 usages in different files then 100 files need to be recompiled, sometimes even with different options.
This has to be done in background as it's long running operation. All operation doing IO, compilation and similar things HAS TO BE DONE IN BACKGROUND otherwise the IDE will be frozen for the time needed to finish such an operation. The update of the java metadata was always background operation but in older releases there was a read/write lock on the cache causing that no one was able to read the data while the data was written. So you was not able to call code completion, do navigation in the time of update. Also the navigator, syntax highlighting and folds were not showing. This is not true anymore as the cache is now similar to transactional database, all the features are working on an committed state. This causes that when you switch from the tab to second one you don't notice the scan as it runs on background on one core and the rest of the IDE on others. The java indexer first fixes the cache for changed file and commits it then it fixes the dependencies (the 1000 files in the example) so in editor you already see the changed file and the IDE in background fixes the errors. On one core CPU the user and background thread share single CPU, this slows down the user thread. Even for multi core CPU there may be IO interference. For this case the background thread can be delayed to prefer the user thread. Currently the editor features like code completion, navigation, index access pauses the background thread. This is probably the best, don't block EDT do it in background but prefer user tasks. You can track the pausing of background thread by the following cmd line property:

-J-Dorg.netbeans.modules.parsing.impl.indexing.SuspendSupport.level=FINE
The IDE will show in the console something like:

SUSPEND: [java.lang.Thread.getStackTrace(Thread.java:1503), org.netbeans.modules.parsing.impl.indexing.SuspendSupport.suspend(SuspendSupport.java:101), org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater.suspend(RepositoryUpdater.java:477), org.netbeans.modules.parsing.impl.TaskProcessor$CompilationJob.run(TaskProcessor.java:662), 
FINE [org.netbeans.modules.parsing.impl.indexing.SuspendSupport]: PARK
FINE [org.netbeans.modules.parsing.impl.indexing.SuspendSupport]: RESUME
FINE [org.netbeans.modules.parsing.impl.indexing.SuspendSupport]: UNPARK

The meaning is:
SUSPEND - user thread requested the background thread to be paused + stack traces showing the user feature. In this case it's an editor task like navigator, highlighting, etc.
PARK - the background thread was parked
RESUME - user thread is done with a critical task and frees the background thread
UNPARK - the background thread runs again

If you find other features which should pause the background thread we can add them.
Currently there are all language and index actions, project build + run. For project build + run it's a bit
more complex when CoS is enabled, the background thread needs to complete all operations done before the run was executed otherwise you would run the old version of the application.

(In reply to comment #61)
Gregg, do I understand it correctly that you are not interested in the error badges on the files/projects (e.g. in the projects tab) and in the compile on save functionality? If so, disabling the Dependency Scanning in:
Tools/Options/Editor/Hints/Language=Java might give you what you want.

(In reply to comment #64)
> >When I switch back to the window/tab that I was working on, the meta-data
> >should have the change in the class visible, and ABSOLUTELY NO OTHER BACKGROUND
> >WORK SHOULD HAPPEN!
> This is not true.
> 1st) The update of cache for the changed file has to happen in background
> thread not to block
> the UI. It may still be running.
> 2nd) More interesting one. You changed the signature of a public method. Not
> only the file you have changed but also all the users of the method need to be
> recompiled as the change may resolve or introduce new errors (error badges in
> the project tree, CoS status, etc.) If you change a method which has 100 usages
> in different files then 100 files need to be recompiled, sometimes even with
> different options.
> This has to be done in background as it's long running operation.

What I'm saying is that it shouldn't do all that work.  It makes no sense, practically, to spend so much effort to discover the other possible errors!

The reason is, that for me, and my coding style and software maintenance practices, either I cloned that method and changed the signature and behavior to work as I needed, because I know there are others that depend on the behavior, or I am working on a method only visible to this project, and I will recompile the appropriate modules myself.

Even if I make a mistake and miss a dependency, I'd a lot rather find that mistake manually, then be slowed by all the background processing locking up the UI and it's other features, just to do something, 100 times more often then it needs to be done.

Plain and simple!  Turn off error badges and all background scanning that looks for dependencies on my editing, in other projects.  I'm not interested in that, and it should NOT be a default feature, for anyone, out of the box.

I think Gregg makes a great point here, and I agree with him: advanced users get usually blocked by all these features. Instead of helping you having the job done the features get in your way and slow you down. And, what's worse: the bigger the project (this is, the more advanced the user) the worse it gets.

This is a clear case of featuritis, I'd say.

(In reply to comment #66)
> 
> What I'm saying is that it shouldn't do all that work.  It makes no sense,
> practically, to spend so much effort to discover the other possible errors!
> 
> The reason is, that for me, and my coding style and software maintenance
> practices, either I cloned that method and changed the signature and behavior
> to work as I needed, because I know there are others that depend on the
> behavior, or I am working on a method only visible to this project, and I will
> recompile the appropriate modules myself.
>

Gregg, could you follow Jan Lahoda's suggestion in
http://netbeans.org/bugzilla/show_bug.cgi?id=133943#c65 to disable dependency
scanning? In other words, I'm trying to establish whether doing that actually
solves the problems you're encountering or whether you're encountering some
other problem that is unrelated to dependency scanning.

Gregg, I am pretty sure that checking errors in dependencies is what most users want, it was one of most voted features. I am also heavily using it when doing changes in big projects. This is the reason why the feature is turned on be default. But you can switch it off in the Tools/Options/Editor/Hints/Java.

I am quite interested why you the refresh of errors blocks other features. I am running the IDE on quite old Intel Core 5 notebook, default heap size (-Xmx768m), having 732 source roots opened with 35202 classes and I don't have this problems. Can you attach or send me the NetBeans self profile snapshot after you did a change and switched the tabs? I changed an API signature in heavily used ClassPath, switched to next tab, do coding, whenever I invoked the code completion the error update was paused. The same holds for semantic highlighting, etc. Thanks.

(In reply to comment #69)
> Gregg, I am pretty sure that checking errors in dependencies is what most users
> want, it was one of most voted features.

I do not recall being presented a chance to vote on that feature, and I know, that for sure, I would have looked at the complexity of such processing from a 10,000 ft view and said, wow, there is no way that could be performant in an IDE environment without consuming excessive CPU and memory resources.  Without a lot of memory, this is going to result in "Swapping" of data between disk and memory to traverse the "graph".  That "Swapping" of data is what results in Java code, calling to the OS, and "hanging" that thread.  We've seen it in stack traces of "lockups" from all sources of GUI code that is not correctly using the EDT.

Any action that the user requests can take as long to complete as their resources require.  Any action that the user doesn't initiate, should not consume more than 1-2 seconds of CPU time, period.  If you hide complexity from developers, you make it possible for them to not understand their software systems, at all, and on deployment, you've only made sure that we will be plagued by all kinds of non-performant software systems where developers don't feel like they need to be responsible for the performance of their software when used by others.

CPU time is not free.  It costs money to have a computer to present those resources and it cost electricity to "run" the computer to provide those resources to the software. Throwing away "money" at something that is "not a problem", is just insane.

Background compilation (It seems like you are compiling with javac provided tools, not just doing a lexer scan), is just not the answer.  Especially as it is implemented now, where file timestamp changes end up driving behavior, instead of the user being able to request and see what happens.

Think about what it would take to provide a navigation pane-like UI component that showed every project and file be processed in the background, including progress and results.  I think if you made the display that complex, and that informative that most people would tell you that they didn't want the IDE doing that much work for them.

As another example, I have a single project that I use to build multiple versions of a library. I "checkout" the version I'm working on, into a common tree.  When that happens, I don't need every single freaking project that I have opened, and dependent on that "project" to be scanned.  I'm "WORKING ON THAT PROJECT" not all the others.

Hello again Gregg, could you follow Jan Lahoda's suggestion in
http://netbeans.org/bugzilla/show_bug.cgi?id=133943#c65 to disable dependency
scanning? In other words, I'm trying to establish whether doing that actually
solves the problems you're encountering or whether you're encountering some
other problem that is unrelated to dependency scanning.

It's now been two weeks and there has been no response to the repeated requests above to Gregg Wonderly (greggwon). 

At this stage, it appears that there are problems with free-form projects. 

At the same time, NetBeans IDE 7.2 has, indeed, made it possible for the IDE to be usable during scanning. That's been one of the main focus points of NetBeans IDE 7.2. (Watch the marketing screencast, see Twitter where many responses about performance enhancements are found, see the NetCAT program for NetBeans IDE 7.2, see the NetBeans IDE 7.2 New & Noteworthy document, see the many performance related bugs fixed in 7.2.) 

I propose that we close this issue as fixed and open a new issue specifically for free-form projects. Jess Holle (jessholle), one of the people above who has encountered problems with free-form projects, will be meeting with NetBeans engineers during JavaOne to discuss those issues.

greggwon was not at the CC List. Added him again, just to make sure he read this. Sorry for any inconvenience.

There are still huge delays in 7.2 when netbeans first starts up.  It says "opening projects" and the GUI is completely useless.  Then, it pops up multiple dialogs which say things about libraries missing, read only files (duh, I use perforce, so yes things in perforce are readonly if I don't have them checked out).  So, there is still too much stuff in the users face.  When I open the IDE, I'm ready to work.  I've allocated time in my schedule and I need to get stuff done.  I don't open the IDE up and leave it running for weeks, or days, because I do more than just write Java on my work computer.

When it first launches, the IDE shouldn't be running around validating the world.  It should let me get to work.  If I go work on a project's file(s), then it can start doing it's scan on that project and the dependencies, because that's all that I care about, at that moment.  That's all I need it to help me with, plain and simple.

greggwon, please tell us your hardware configuration (processor and memory at least).

Created attachment 149772 [details]
.npss file attached from NetBeans

.npss file

I just uploaded a snapshot of a 2-minute session during which the IDE was doing "background scanning of projects". During this time, the IDE would remain mostly frozen, but unfreeze for about half a second every four seconds. This can be seen very clearly in the "Stack depth" plot, which shows long periods of blocking between small bursts of regular activity. (Is the selfsampler itself being blocked?)

When I observed the problem, I had 21 maven module projects open and was editing the pom.xml file (adding a dependency) of one of them. This behavior occurs frequently whenever I need to edit a pom.xml file--presumably adding a POM dependency is one way to trigger a background scan. The problem is not that the background scan has to happen, but rather that the IDE is completely unusable during this time. It took about 10 minutes for the background scanning to complete and the IDE to become usable again.

This is also related to the bug report "How do I DISABLE scanning".

Some people just want a code highlighter, not a tool that tries to compile your code every time you write one single character.

Also related to bug report 

https://netbeans.org/bugzilla/show_bug.cgi?id=269867

"Option to disable error badges COMPLETELY, for EVERY LANGUAGE"