I tried to import a large source base into a 4.0
project, and it was basically not practical.
The problem has to do with how NB 4.0 allows the
specification of src packages and how our project
src base is laid out. The top level package dirs
are parallel to non-java and lib dirs, and so
there's no clean "src" vs. "lib" separation.
Since the tree is large, I end up with a huge
number of elements in the package tree, and the
IDE quickly bogs down.
Ideally, it would be more usable if src directory
selection followed an include/exclude semantics,
similar to an Ant fileset. Then we can limit the
# of src packages that we deal with, and make the
project tree more usable.
I've changed your report from enhancement to lower priority defect and
I'm passing it to project guys.
However, please add more specific numbers about how your source base
looks. How many top packages? How many subdirs? How big is your source
base? overall? etc. so that we can at least simulate your working
context and see what we could do.
Also I don't quite understand your include/exclude suggestions, but I
hope project guys will understand.
Re. "the top level package dirs are parallel to non-java and lib dirs,
and so there's no clean "src" vs. "lib" separation." - don't know what
this means exactly, no example given.
Re. OOME - should be filed separately with precise steps to reproduce
from scratch. Or read release notes re. increasing your memory - some
configurations involving a lot of files simply require a larger max heap.
Re. include/exclude semantics - no plans to implement. Doesn't match
javac's semantics well (javac would not honor your excludes anyway).
Prefer to address concrete problems by other means. You may however
look at the Maven project type (mevenide.codehaus.org) which may
support such things, I am not sure.
Re. package view being unwieldy - try unsupported switch in issue #42151.
By large project, I meant a directory structure with 12K+ directories
and over 73K+ files.
The directory structure is like
|- weblogic (actual product src code)
(119 top-level directories)
"weblogic" is the only dir here that contains product src code. If I
point to "src/wls" as the root to maintain the package structure, I
end up with the conditions I mention, where I end up with hundreds and
hundreds of "packages" in the project view, and the system bogs down, etc.
If I point to "src/wls/weblogic" as the root, to minimize the number
of files scanned, the packages are incorrectly labeled in the tree.
So, either the former or the latter needs to be addressed. The latter
issue is easier to resolve, as the package structures in the IDE tree
view should be derived from the src code, and not the path to the files.
By "include/exclude" semantics, I'm not referring to how the code gets
compiled. I'm strictly referring to how the project setup is used to
scan a directory structure and discover the source files and resources
in the project. That is how I would see this being used to implement
a solution to the first problem.
Would require extended API to inform Java parser to ignore certain
directories - currently project system just tells it which folders are
source roots and the Java scanning infrastructure creates databases
based on that. Potentially complicated, would require API changes; no
plans for 4.0.
If at all possible, you are encouraged to break apart the monolithic
source tree into logical units (projects or binary JARs) that can be
considered separately. Of course, projects with an existing fixed
source structure cannot generally do this.
Reassigning to Tomas since this would require (as yet unstudied)
changes to the Classpath API.
This seems rather as an enhancement. The issue requires extension of
ClassPath API/SPI as Jesse mentioned above. This will not be done in
NB 4.0 (too late for such a change),
Related to issue #49026.
*** Issue 52112 has been marked as a duplicate of this issue. ***
Martin your opinion on this issue would be appreciated. We have
repeatedly heard that this issue is a blocker for some people
migrating to 4.x (usually people working on huge projects that for
whatever reason cannot be reorganized into a more digestible directory
structure): unlike in 3.x, it is impossible to simply ignore unwanted
portions of a huge source tree, because the classpath scanner only
deals with complete source roots. There are other related issues (in
building and in display) which the projects team would be able to
resolve with a modest amount of work, I think, but without fixing the
scanning it would be pointless.
Is there any possibility to address this in javacore, say in F?
Probably this would involve adding some kind of refinement to the
ClassPath API which would allow GlobalPathRegistry to include only
portions of a source root (e.g. a list of named package/wildcard
includes or excludes). Javacore would then need to create an
independent scanner database for each source root fragment, I guess,
or leave portions of a single scanner database intentionally empty (to
be filled on demand). If you did e.g. a refactoring operation, only
classes within included packages would be considered. Opening a
database should load only metadata applicable to requested packages,
even if metadata were available on disk for excluded packages, to
It should be quite easy to implement some kind of filtering per
classpath element URL in the scanner (i.e. restricting which
subdirectories under a given URL should be scanned and which should
not). If there is an API where we could send root URL and relative
name of a directory and the API would return true or false indicating
whether it should be ignored or not, it could be a matter of hours to
make our scanner use this.
Being able to have multiple CP roots with the same URL but different
filters applied on it would be more tricky.
Based on Martin's encouraging comments, I am marking this a target for
F. Have heard from more than one source that it is important for
developers working on massive legacy projects with poor layout.
More to Martin: the way I am currently thinking it would work would be
1. GlobalPathRegistry would have the ability to register (or
unregister) a ClassPath with a list of package includes (i.e. resource
prefixes such as "org/foo/whatever/"). This would be all that would be
required on the SPI side - the project would handle figuring out what
2. GPR already automatically removes duplicates at the ClassPath
level, though not at the Entry level. It would need to somehow also
keep track of common Entry URLs and merge includes.
3. From the Javacore side, all you should have to do is ask for a set
of entry URLs to scan, and in each case you should be able to confirm
for each package in turn that it should be included (using some method
Still need to work this out more precisely - have to study the usage
of the GPR API from the Javacore side, in particular in
MergedClassPathImplementation, to see what makes sense. It's not just
about scanning - any operation which goes through all classes in a
root (e.g. Find Usages) would need to check the includes list, to
avoid having any slowdown in operations for unused packages.
Again a caveat: the IDE will not *enforce* that the packages you as a
user decide to include are really self-contained: if you accidentally
refer to classes in an excluded package from an included package,
javac due to its nature will quietly compile against them without
errors and you will have potentially messed up your project (depending
on what your policies are). It is much better to rearrange your
sources into smaller units to enforce your intended dependencies, so
you get the benefit of full error checking (and this scenario will
also work well with NetBeans).
It is technically possible to coerce <javac> into compiling only
included packages from a big source tree in isolation, while ensuring
that error messages link back to the real source files - cf. e.g.
openide/build.xml#do-lib-javac in NB sources - but it is quite ugly
and does not work at all with Jikes, so this is not very advisable for
use in build-impl.xml.
By the way, undocumented experimental startup option to exclude
certain package trees from classpath scanning (NO EFFECT on Ant builds):
Not tested, use at your own risk, may be removed/changed without notice.
Obviously not for 5.0.
*** Issue 70181 has been marked as a duplicate of this issue. ***
*** Issue 64495 has been marked as a duplicate of this issue. ***
*** Issue 89628 has been marked as a duplicate of this issue. ***
To avoid misinterpretation, treating this as purely a performance issue with
large source roots treated as single compilation units. Use issue #49026 for
issues relating to treating a source root as multiple compilation units.
Simple to characterize and reproduce. Run a NB dev build on a fresh userdir.
Make a Java project from existing sources and point to j2se/src/share/classes
from a JDK 6 source checkout. Takes several minutes just to unblock EQ (someone
is calling ProjectUtilities.openAndSelectNewObject, have not fully
investigated). Then you get an OOME.
NB 5.5 fares a little better - initial scan completes with no OOME. But opening
some files such as JTable.java will still yield OOME (if the project is not
I have corrected some miscellaneous performance bugs related to creating and
opening the project. So now the OOME from scanning the source root can be seen
In 061201, with -Xmx256m, opening nbbuild w/ all deps, scanning did not complete:
WARNING [global]: Not enough memory to compile folder:
This folder has 80 sources. What's wrong?
After the failed scanning, Find Usages would run, but give incomplete results.
This is pretty bad IMHO - the only reason I knew something went wrong with
scanning was because (1) the "Preparing usages view" task was still running (and
could not be cancelled!); (2) I knew for a fact there were more usages than were
On a custom build from recent sources, I was able to scan nbbuild (recent
checkout of stable_nowww) with all dependencies with default memory settings
(-Xmx128m) on both JDK1.5.0_09 and JDK1.6 b104. Is there anything special in
your setup? Any special additional modules?
My attempted scan of nbbuild was in my regular userdir, which has lots of
additional modules. I don't know which ones might have significantly affected
memory usage, of course. The scan of the JDK was in a fresh userdir with default
The cache update was redesigned in the way that the signature files are created
as far as possible. Now I am able to open the JDK project (j2se/share/classes +
j2se/solaris/classes) with 128MB in less than 4 min.
I run the IDE (fresh dev from sources, new userdir) under JDK 7 and make New
Project from Existing Sources. Select j2se/src/share/classes only.
0.30 wiz dialog closes, projects window appears
1.30 (?) all packages displayed in projects window, Retouche scan continues
5.00 IAEs start being thrown (see attachment)
8.45 still going, at ~300 exceptions I give up and kill IDE
Similar attempt in Eclipse 3.2.1:
0.00 start (pressed Finish in wizard)
0.20 project opened, building workspace
3.00 "analyzing sources"
3.40 "building workspace"
... here it gets confused by **/.svn/text-base files...
7.30 I lose patience and kill at 25% completion; IDE mostly unresponsive
Now in NB 5.5:
0.52 project opened, scanning classpaths
1.54 scanning complete
3.45 I have JTable.java open, Navigator populated, green box
4.47 I have found all usages of ComponentOrientation
6.00 IDE hung at 100% CPU, has to be killed
Created attachment 37155 [details]
The IAE is evaluated by Honza, adding Honza to CC.
The IAE should be fixed now. See issue #91745 for more information.
So trying again w/ today's build:
0.15 wiz dialog closes, projects window appears
1.12 all packages displayed in projects window, Retouche scan continues
2.53 "compiling classes", start to get some warnings on console (attached)
6.30 scan finishes
7.40 I have JTable.java open, Navigator populated, yellow box
8.30 I ask for all usages of ComponentOrientation
9.05 after freezing for a while, I get an OOME
So... better than it was, somewhat worse than 5.5, still not really usable.
BTW part of the frustration with the speed of scanning is due to issue #87968 -
there is no detailed information about the progress of the scan, unlike in 5.5.
Created attachment 37197 [details]
New log file showing various warnings
What is status of this issue. AFAIK are able to open src/classes and the exclude
mechanism was implemented as well. Should we close it?
I think there are still things to solve like interference among package view
children and initial scan. Maybe you can decrease priority.
Fine lowering the priority and removing the plan60 keyword as most of the things
really needed for 6.0 were done.
The issue is very broad. The additional performance optimalization I want to do in NB 6.0:
1) Prescan of archives like in NB 5.0
2) Partial parse (parse of changed method only)
3) Force writers of tasks to fix cancel ()
Issue should perhaps be closed or broken down into more specific perf RFEs.
I don't know the exact reasons for the performance issues but they need to be taken very seriously.
NetBeans 6.0 is almost unusable in its current state when using a large code base because it periodically disappears for
extended periods(at 100% CPU use). If this is not fixed in 6.0, I don't think that you have a seriously competitive
product, which is a shame since there is a lot of goodwill for this project and a lot of good work already done.
What can we beta testers do to help you track down these performance issues?
We did several perf improvements in 6.1 timeframe. Including partial reparse.
Is this issue still valid?
Probably best to close and any specific performance problems still reproducible in 6.1 builds should be filed
independently with details to reproduce.