This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 161367 - [67cat] Scanning projects task never terminates
Summary: [67cat] Scanning projects task never terminates
Status: VERIFIED FIXED
Alias: None
Product: editor
Classification: Unclassified
Component: Parsing & Indexing (show other bugs)
Version: 6.x
Hardware: All All
: P1 blocker with 7 votes (vote)
Assignee: Vitezslav Stejskal
URL:
Keywords: PERFORMANCE
: 166188 (view as bug list)
Depends on: 164745
Blocks: 156206 162706 164052
  Show dependency tree
 
Reported: 2009-03-27 16:21 UTC by denbo
Modified: 2009-09-17 14:45 UTC (History)
18 users (show)

See Also:
Issue Type: DEFECT
Exception Reporter:


Attachments
message log (43.62 KB, text/plain)
2009-04-04 11:51 UTC, nleck
Details
thread dump when IDE reached 780mb (near limit) (14.42 KB, text/plain)
2009-04-07 15:26 UTC, twolf2919
Details
second heap dump after IDE was restarted and hung again (14.84 KB, text/plain)
2009-04-07 15:27 UTC, twolf2919
Details
messages.log showing NullPointerException (28.05 KB, text/plain)
2009-04-07 15:31 UTC, twolf2919
Details
200904091401 Thread Dump #1 (5 minutes) (19.69 KB, text/plain)
2009-04-10 03:15 UTC, theshadow27
Details
200904091401 Thread Dump #2 (30 minutes) (18.07 KB, text/plain)
2009-04-10 03:19 UTC, theshadow27
Details
200904091401 Thread Dump #3 (1 hour) (17.06 KB, text/plain)
2009-04-10 03:21 UTC, theshadow27
Details
log of when project scan never finishes ( on windoze) (1.48 MB, text/plain)
2009-04-10 10:51 UTC, nleck
Details
multiple thread dumps while IDE is doing a project scan (151.80 KB, text/plain)
2009-04-15 04:35 UTC, nleck
Details
threaddumps (152.17 KB, text/plain)
2009-04-17 11:00 UTC, Tomas Stupka
Details
messages.log (63.22 KB, text/plain)
2009-04-17 11:03 UTC, Tomas Stupka
Details
stack dumps. (47.10 KB, text/plain)
2009-04-23 04:17 UTC, nleck
Details
many thread dumps while scanning (107.55 KB, text/plain)
2009-04-30 05:45 UTC, nleck
Details
process bar stops giving feed back well before we can actually scam (97.40 KB, image/png)
2009-05-11 04:00 UTC, nleck
Details
not working (94.46 KB, image/png)
2009-05-12 11:52 UTC, nleck
Details
sample Java file that was slow to show the members view (312.97 KB, text/plain)
2009-05-22 07:54 UTC, nleck
Details
message log with -J-Dorg.netbeans.modules.parsing.impl.indexing.RepositoryUpdater.level=FINE (806.49 KB, text/plain)
2009-05-22 10:12 UTC, nleck
Details
profile when opening one file (135.69 KB, image/png)
2009-05-22 10:23 UTC, nleck
Details
profile dump ( from profile me button) while opening a file (299.61 KB, application/octet-stream)
2009-05-22 10:26 UTC, nleck
Details
Here's a trace friom 6.1 where the event thread is blocked because of syncing with scanning/compiling (37.93 KB, text/plain)
2009-05-29 20:44 UTC, greggwon
Details
movie comparing Eclispe and NB opening a file showing the outline/navigator (989.45 KB, video/ogg)
2009-05-30 00:23 UTC, nleck
Details
horror movie comparing Eclispe and NB opening a file showing the outline/navigator (video/ogg) -- no red marks in EC (830.45 KB, video/ogg)
2009-05-30 01:03 UTC, nleck
Details
performance regression NB 6.5 v NB 6.7 daily (1.19 MB, video/ogg)
2009-05-30 04:32 UTC, nleck
Details
FIXED of performance regression (405.32 KB, video/ogg)
2009-06-02 11:45 UTC, nleck
Details
scan is out of date (17.56 KB, image/png)
2009-06-08 02:15 UTC, nleck
Details

Note You need to log in before you can comment on or make changes to this bug.
Description denbo 2009-03-27 16:21:37 UTC
[ BUILD # : 200903250219 ]
[ JDK VERSION : 1.6.* ]

One of my opened projects, a Maven web project, is being scanned but
the scan task does not terminate. 
However, I cannot reproduce this in a reliable fashion and my IDE log
does not indicate any problems.
I'm not sure if this is related, but my IDE uses 50% of my CPU power
at the moment and does no longer respond to UI input.
Comment 1 Milos Kleint 2009-04-01 06:46:05 UTC
scanning is done in java/source?
Comment 2 Rastislav Komara 2009-04-01 07:54:56 UTC
This is already resolved if I'm correct. 

@mkleint: I don't think so.
Comment 3 Vitezslav Stejskal 2009-04-01 10:24:42 UTC
Can you take several threaddumps when the IDE is scanning and attach them here? Also could you possibly run the IDE with
-J-Dorg.netbeans.modules.parsing.impl.indexing.RepositoryUpdater.level=FINE switch and attach here the
<ide-userdir>/var/log/message.log file>? Thanks
Comment 4 nleck 2009-04-04 11:49:27 UTC
I've got the same issue, 6.7m3 is *NOT* usable at all. No members ever appear, no naviagtion possible.

Our project consists of ~3,000 source files.
Comment 5 nleck 2009-04-04 11:51:51 UTC
Created attachment 79435 [details]
message log
Comment 6 nleck 2009-04-04 11:53:31 UTC
This also means that debugging is not enabled ( it seems scanning needs to complete first) 
Comment 7 Jiri Prox 2009-04-06 12:47:47 UTC
Can you, please,  run IDE with -J-Dorg.netbeans.modules.parsing.impl.indexing.RepositoryUpdater.level=FINE and attach
messages.log as suggested by vstejskal?
Comment 8 twolf2919 2009-04-07 15:25:06 UTC
I've been having this scanning issue for quite some time now.  This morning, I started witha completely new slate:
removed my $HOME/.netbeans.dev/ directory and downloaded/installed the newest daily (04/07/2009) build and started it. 
Soon after opening the project, the IDE began scanning....when, after a minute or two, it reached 650Megs (with CPU
pegged at near 100%), I went to a terminal and ran jstack against the process - with no results.  I repeated with
"jstack -F".  See attachment for resulting dump in trace.txt.

I restarted the IDE and, again, it went to max memory.  This time a plain "jstack" worked and produced a dump.  Attached
as "trace2.txt".

I made this bug a P1 because the IDE has basically become useless.  I am reverting to 6.5.1 so I can get some work done.
Comment 9 twolf2919 2009-04-07 15:26:34 UTC
Created attachment 79649 [details]
thread dump when IDE reached 780mb (near limit)
Comment 10 twolf2919 2009-04-07 15:27:30 UTC
Created attachment 79650 [details]
second heap dump after IDE was restarted and hung again
Comment 11 twolf2919 2009-04-07 15:31:35 UTC
Created attachment 79652 [details]
messages.log showing NullPointerException
Comment 12 twolf2919 2009-04-07 15:33:07 UTC
Also, I noticed that there was a NullPointerException in the messages.log, so I attached that as well.  There's also a
heap dump - but it's huge.  Not sure if that might be of use.
Comment 13 Jiri Prox 2009-04-07 15:44:10 UTC
the exception is already reported as issue 162016
Comment 14 Vitezslav Stejskal 2009-04-08 16:35:46 UTC
Do you have the tasklist window open or not? There is a problem with tasklist filter, which means that even task
scanners that are turned off are still used for scanning, see issue #162076. Could you please also check what tasklist
filter you use. It's in Windows -> Tasks, then click the filter icon and Edit... Normally, 'Default Filter' is selected
and as part of this filter there are two task scanners enabled - TODO and Java Errors.

Also, if this happens again please create several successive threaddumps and attach them here.

Thanks
Comment 15 twolf2919 2009-04-08 20:32:41 UTC
I don't use task lists (turn them off via Tools->Options->Editor->Tasks uncheck "Enable Java Tasklist") and close the
tasklist window that appears when opening the first project for the first time after installation.

As far as what Task filters are enabled: I just reverted from 6.5.1 and, lo and behold, the IDE came up without hanging
(yesterday, I could reproduce each time I started the IDE!).  For some reason the Task window came up - even though I
could have sworn yesterday I did not have it open (since, as mentioned above, I don't even use Task lists!)  I checked
and the defaults are what you described (Java Errors and TODO).  And my "Enable Java Tasklist" is unchecked as expected.

So I'm clueless - the only thing that differed between yesterday's multiple IDE startups which all resulted in hanging
and today is that in-between I reverted to 6.5.1, did some PMD stuff, rebuilt the project a few times, and then reverted
back to the daily!
Comment 16 Vitezslav Stejskal 2009-04-09 09:17:17 UTC
"close the tasklist window that appears when opening the first project for the first time after installation." - that's
exactly what is causing problems reported in issue #162076. Could you please try leaving the window open and see if you
still have the problem. On my laptop the problem was 100% reproducible with tasklist window turned off and worked like a
charm with the window on. If that's the case I'd say that this is a duplicate of #162076.

Before #162076 is fixed you might try uninstalling the tasklist modules through the plugin manager (Tools -> Plugins),
which should once and for all silence its scanners. Thanks a lot
Comment 17 theshadow27 2009-04-10 03:12:04 UTC
Scanning has continued for exactly 1 hour. Have 3 thread dumps, 5 minutes, 30 minutes, and 1 hour. 
Comment 18 theshadow27 2009-04-10 03:15:11 UTC
Created attachment 79861 [details]
200904091401 Thread Dump #1 (5 minutes)
Comment 19 theshadow27 2009-04-10 03:19:49 UTC
Created attachment 79862 [details]
200904091401 Thread Dump #2 (30 minutes)
Comment 20 theshadow27 2009-04-10 03:21:28 UTC
Created attachment 79863 [details]
200904091401 Thread Dump #3 (1 hour)
Comment 21 theshadow27 2009-04-10 03:25:46 UTC
And for what it's worth, here's a heap dump as well (hprof binary format, ~90mb)

http://theeshadow.com/files/200904091401/heapdump-1239329788619.hprof

Build 200904091401, 1 hour and *still* "scanning projects..."

Note, I have not opened any windows, touched anything at all in the IDE. Just launched via terminal, and waited for it to finish scanning. 
Comment 22 theshadow27 2009-04-10 03:31:06 UTC
Changing to All and All, I have OS X 10.5.6
Comment 23 Vitezslav Stejskal 2009-04-10 07:34:30 UTC
Thanks for the threaddumps. All of them show activity in JavaCustomIndexer and so this really is unrelated to tasklist
problem. How many projects do you have open and what's their size (approx) and type (java/web/...)? Where are the
projects stored (local disk)? How fast is scanning of the same projects in 6.5? Thanks
Comment 24 nleck 2009-04-10 08:42:22 UTC
I think we maybe confused on this bug. I mean the project scanning. We've turned off the "task list" completely, it
locked up our projects too for very little gain. 

It's the project scanning that's a show stopper for me anyway. No debugging or navigation. 
Comment 25 nleck 2009-04-10 10:51:44 UTC
Created attachment 79872 [details]
log of when project scan never finishes ( on windoze)
Comment 26 Vitezslav Stejskal 2009-04-14 10:14:18 UTC
nleck - Could you please describe your project setup (eg project type, files layout, etc)? Can you reproduce this state?
What do you do in order to get the IDE in this state? We might need more logging to diagnose it or ideally a small
reproducible testcase. Thanks
Comment 27 nleck 2009-04-14 10:34:13 UTC
Our project consists of some 6,086 java files. The project is quite old with many developers. We work in a mixed
environment of mostly Eclipse users so change the project structure is not really an option. 

The same project is developed on Solaris, Windows and Linux. The constant scanning seems to only effect Windows and Solaris.

On Solaris we have the sources on a mounted network drive ( again not by choice). 

We have the sources and the jars required to build it all checked into cvs. 

The constant scanning didn't happen in 6.5 but the slow scanning was the major reason why the most developers still use
Eclipse.
Comment 28 Vitezslav Stejskal 2009-04-14 13:47:44 UTC
What project type is it? Freeform? It looks like evrything you have in under the same root - sources, libraries, library
sources. I'm just trying to replicate your setup and simulate that never ending loop.
Comment 29 nleck 2009-04-14 14:26:58 UTC
yep, freeform with everything under one root.
Comment 30 Vitezslav Stejskal 2009-04-14 14:38:58 UTC
Could you please generate several successive thread dumps when the IDE is scanning and attach them here? Thanks
Comment 31 Vitezslav Stejskal 2009-04-14 14:41:58 UTC
And where are the project's java files compiled to?
Comment 32 nleck 2009-04-14 23:00:52 UTC
the class files are compiled out of the source tree into the $project director$/webapps/ROOT/WEB-INF/classes
Comment 33 theshadow27 2009-04-15 01:25:07 UTC
vestejskal -

Threaddumps from 2009 04 09 1401 were with one small project open on an empty user directory. The scanning never terminated. However after closing the 
tasks window and restarting the IDE, there were no problems.

I am unable to replicate the problem on 2009 04 14 0201. This build is very solid! If I can get someone else to confirm, I will change this to fixed. 
Comment 34 nleck 2009-04-15 04:35:41 UTC
Created attachment 80098 [details]
multiple thread dumps while IDE is doing a project scan
Comment 35 Tomas Stupka 2009-04-17 10:59:25 UTC
looks like i also was able to reproduce

will attache threaddumps & co.
Comment 36 Tomas Stupka 2009-04-17 11:00:04 UTC
Created attachment 80331 [details]
threaddumps
Comment 37 Tomas Stupka 2009-04-17 11:03:25 UTC
Created attachment 80332 [details]
messages.log
Comment 38 Dusan Balek 2009-04-17 12:24:35 UTC
tstupka -
are you able to reproduce the problem using the build containing
http://hg.netbeans.org/main-golden?cmd=changeset;node=6f3d60df0df9
Comment 39 nleck 2009-04-22 06:36:05 UTC
I'm getting NPE while scanning in the latest daily build. 

Is there a way to stop the scanning process from looking at the lib directory at all ? 



SEVERE [org.netbeans.modules.java.source.TreeLoader]: Coupling error: class file
jar:file:/home/nigel/src/DC/lib/xalan.jar!/org/apache/bcel/Constants.class, source file
file:/home/nigel/src/DC/lib/src/org/apache/bcel/Constants.java
SEVERE [org.netbeans.modules.java.source.TreeLoader]: Coupling error: class file
jar:file:/home/nigel/src/DC/lib/xalan.jar!/org/apache/bcel/classfile/AccessFlags.class, source file
file:/home/nigel/src/DC/lib/src/org/apache/bcel/classfile/AccessFlags.java
SEVERE [org.netbeans.modules.java.source.TreeLoader]: Coupling error: class file
jar:file:/home/nigel/src/DC/lib/xalan.jar!/org/apache/bcel/classfile/Constant.class, source file
file:/home/nigel/src/DC/lib/src/org/apache/bcel/classfile/Constant.java
SEVERE [org.netbeans.modules.java.source.TreeLoader]: Coupling error: class file
jar:file:/home/nigel/src/DC/lib/xalan.jar!/org/apache/bcel/classfile/Attribute.class, source file
file:/home/nigel/src/DC/lib/src/org/apache/bcel/classfile/Attribute.java
SEVERE [org.netbeans.modules.java.source.TreeLoader]: Coupling error: class file
jar:file:/home/nigel/src/DC/lib/xalan.jar!/org/apache/bcel/classfile/LocalVariableTable.class, source file
file:/home/nigel/src/DC/lib/src/org/apache/bcel/classfile/LocalVariableTable.java
SEVERE [org.netbeans.modules.java.source.TreeLoader]: Coupling error: class file
jar:file:/home/nigel/src/DC/lib/commons-logging-1.0.3.jar!/org/apache/commons/logging/impl/Log4JLogger.class, source
file file:/home/nigel/src/DC/lib/src/org/apache/commons/logging/impl/Log4JLogger.java
INFO [org.netbeans.modules.xml.schema.completion.util.CompletionContextImpl]
java.lang.NullPointerException
        at
org.netbeans.modules.xml.schema.completion.util.CompletionContextImpl.populateModelMap(CompletionContextImpl.java:612)
[catch] at
org.netbeans.modules.xml.schema.completion.util.CompletionContextImpl.specialCompletion(CompletionContextImpl.java:637)
        at org.netbeans.modules.xml.schema.completion.util.CompletionContextImpl.initModels(CompletionContextImpl.java:606)
        at org.netbeans.modules.xml.schema.completion.CompletionQuery.getCompletionItems(CompletionQuery.java:103)
        at org.netbeans.modules.xml.schema.completion.CompletionQuery.query(CompletionQuery.java:87)
        at org.netbeans.spi.editor.completion.support.AsyncCompletionTask.run(AsyncCompletionTask.java:218)
        at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:573)
        at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:1005)
INFO [org.netbeans.modules.xml.schema.completion.util.CompletionContextImpl]
java.lang.NullPointerException
        at
org.netbeans.modules.xml.schema.completion.util.CompletionContextImpl.populateModelMap(CompletionContextImpl.java:612)
[catch] at
org.netbeans.modules.xml.schema.completion.util.CompletionContextImpl.specialCompletion(CompletionContextImpl.java:637)
        at org.netbeans.modules.xml.schema.completion.util.CompletionContextImpl.initModels(CompletionContextImpl.java:606)
        at org.netbeans.modules.xml.schema.completion.CompletionQuery.getCompletionItems(CompletionQuery.java:103)
        at org.netbeans.modules.xml.schema.completion.CompletionQuery.query(CompletionQuery.java:87)
        at org.netbeans.spi.editor.completion.support.AsyncCompletionTask.run(AsyncCompletionTask.java:218)
        at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:573)
        at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:1005)
INFO [org.netbeans.modules.xml.schema.completion.util.CompletionContextImpl]
java.lang.NullPointerException
        at
org.netbeans.modules.xml.schema.completion.util.CompletionContextImpl.populateModelMap(CompletionContextImpl.java:612)
[catch] at
org.netbeans.modules.xml.schema.completion.util.CompletionContextImpl.specialCompletion(CompletionContextImpl.java:637)
        at org.netbeans.modules.xml.schema.completion.util.CompletionContextImpl.initModels(CompletionContextImpl.java:606)
        at org.netbeans.modules.xml.schema.completion.CompletionQuery.getCompletionItems(CompletionQuery.java:103)
        at org.netbeans.modules.xml.schema.completion.CompletionQuery.query(CompletionQuery.java:87)
        at org.netbeans.spi.editor.completion.support.AsyncCompletionTask.run(AsyncCompletionTask.java:218)
        at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:573)
        at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:1005)
INFO [org.netbeans.modules.xml.schema.completion.util.CompletionContextImpl]
java.lang.NullPointerException
        at
org.netbeans.modules.xml.schema.completion.util.CompletionContextImpl.populateModelMap(CompletionContextImpl.java:612)
[catch] at
org.netbeans.modules.xml.schema.completion.util.CompletionContextImpl.specialCompletion(CompletionContextImpl.java:637)
        at org.netbeans.modules.xml.schema.completion.util.CompletionContextImpl.initModels(CompletionContextImpl.java:606)
        at org.netbeans.modules.xml.schema.completion.CompletionQuery.getCompletionItems(CompletionQuery.java:103)
        at org.netbeans.modules.xml.schema.completion.CompletionQuery.query(CompletionQuery.java:87)
        at org.netbeans.spi.editor.completion.support.AsyncCompletionTask.run(AsyncCompletionTask.java:218)
        at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:573)
        at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:1005)
Comment 40 nleck 2009-04-22 06:40:43 UTC
The scanning now stops in the latest build but it's scanning 8871 source files ( because of the ones in our lib
directory). We should only have 3500 source files. 
Comment 41 Vitezslav Stejskal 2009-04-22 10:32:24 UTC
"The scanning now stops in the latest build" - Good. This was most likely fixed by
http://hg.netbeans.org/jet-main/rev/6f3d60df0df9 (available in builds after Apr-16).

"I'm getting NPE while scanning" - The NPE is IMO unrelated to scanning and comes from the xml.schema module. Please
file a separate issue for this.

"Is there a way to stop the scanning process from looking at the lib directory at all ?" - No sorry. You will have to
rearrange your project so that it does not mix libraries and sources under the same source root.
Comment 42 nleck 2009-04-23 01:04:13 UTC
while this project scan now completes it is still much slower than Eclipse on the same project. Can the performance be
improved ? 
Comment 43 nleck 2009-04-23 02:20:23 UTC
Gotta re-open now it can't find classes that are there. 
Comment 44 nleck 2009-04-23 02:24:39 UTC
When we first open the project it does a large project scan.  Use "Go To Type..." type in a class name that I know is
there and it doesn't find it. I then manually navigate to this class. The IDE does another "project scan" and then we
can find the class.

This is obviously seriously broken. 
Comment 45 nleck 2009-04-23 03:32:01 UTC
The latest build has taken over half an hour saying "Scanning projects 688 of 688" it seems that the IDE scans different
this directories depending on what we click on. 
Comment 46 nleck 2009-04-23 04:04:15 UTC
ok... it's been scanning for a very long time now. This bug has not been fixed
Comment 47 nleck 2009-04-23 04:17:37 UTC
Created attachment 80733 [details]
stack dumps.
Comment 48 Vitezslav Stejskal 2009-04-23 09:25:43 UTC
Can you also attach the IDE log file? With -J-Dorg.netbeans.modules.parsing.impl.indexing.RepositoryUpdater.level=FINE
the log file should now show stactraces of scan requests.
Comment 49 nleck 2009-04-24 05:30:42 UTC
I removed my .netbeans/dev/cache directory and restarted the IDE and left it for quite a while and the scan finished
plus I was able to find all the existing classes that I looked for. 

It seems to be using the IDE while it was scanning caused issues. 
Comment 50 denbo 2009-04-29 14:50:23 UTC
The project scan terminated in my first NB 6.7beta session, however it took about an hour.
One thing I noticed was that scanning of .properties files was especially slow. There were 668 UTF-8 properties-files in
my project and the IDE log said:

FEIN [org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater]: Indexing 668 indexables; using
org.netbeans.modules.tasklist.impl.TaskIndexer@9b5088; mimeType='text/x-properties'
FEIN [org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater]: Using EmbeddingIndexers for
[FileObjectIndexable@c41522
[file:/D:/_projects/tgo/3.9/tgo-mo/webapp/WEB-INF/classes/i18n/content/common/cities_zh_CN.properties],
FileObjectIndexable@14771ba ..... 

this goes on for the other files.
Comment 51 nleck 2009-04-29 23:40:07 UTC
I downloaded 6.7beta too (on windows only) and the project scan completed on first boot too and I could find all the
classes that I searched for :-) 

As I navigate I can see the project scan start again as I go into different files and the scan is for many files but no
files have been changed so this is an miss fire. 

Also in our case the project scan is picking up a lot of files that aren't in our project. It would be great if we could
omit directories from the scan.

I'll download the beta for Solaris and try it there. 
Comment 52 nleck 2009-04-30 05:42:08 UTC
Running the 6.7beta on Solaris it's been scanning 13,512 files for a every long time now ( a hour or more). 

We don't have anywhere near 13,512 files. It must have dug files out of the jar files or double counting. When we do a
full compile of the system it is 5 minutes for 6,173 Java files.

Our project is a freeform project it would be great if we could exclude directories from the scan. 

Now even given that there must be double the files scan than exists in the project that should only equate to 10 minutes
if the scan performance was the same as javac, currently it's at least 6-7 times slower. 

The IDE is currently blocked saying ( 10937 of 13512) with seemingly no progress for a long time ( 20-30 minutes), I've
taken a number of ctrl-\ and will attach to this bug. 

Comment 53 nleck 2009-04-30 05:45:26 UTC
Created attachment 81269 [details]
many thread dumps while scanning
Comment 54 nleck 2009-04-30 06:09:44 UTC
In the end it took 1.5 hours to scan 13,512 files ( should have been 6,173 Java files).... the real kicker is when I go
to another file it starts the project scan again and blocks the "Go to Type" until this is done. 

Once a full scan has been done the "Go to Type" should not block even if the IDE is scanning a new file. In this case
*NO* files have been changed so why it's scanning and why it takes so long is a complete mystery to me. 

The "Scanning projects... " is saying (3929 of 3929) THIS IS AFTER WAITING AN HOUR AN A HALF FOR THE INITIAL SCAN.  
Comment 55 nleck 2009-04-30 12:36:56 UTC
When I shutdown my IDE and start it up again.. The project scan restarts even thou no files have changed. Again the IDE
is not usable until this finishes. 
Comment 56 Vladimir Voskresensky 2009-04-30 13:29:53 UTC
> 13,512 files ( should have been 6,173 Java files)
Do you have symlinked dirs?
I have seen the same problem (+ all objects were presented twice in Go To Type dialog) in the following configuration:
I have:
/export/home/user/source/nb/trunk
and it is symlinked as 
/home/user/trunk -> /export/home/user/source/nb/trunk

Then trying to open trunk using symlink doubles all objects (and files counter)
Comment 57 nleck 2009-04-30 14:01:58 UTC
no symblic links in this project but the home directories are NFS mounted. 
Comment 58 fommil 2009-05-02 16:01:38 UTC
I'm also seeing very long Scanning tasks when editing the main-golden source code for the java.hints project. Feedback shows that 
dependencies/dependants are also being scanned! (that's a *lot* of source). I only have java.{source, editor, hints} open.

Things speed up considerably if I disable dependencies in Java Tasklist, although I can still see the projects being scanned (albeit quickly) during startup. 
Given that the Task list only cares about opened projects... why are unopened projects ever scanned in the first place?
Comment 59 _ ludo 2009-05-03 22:30:07 UTC
See as well http://www.netbeans.org/issues/show_bug.cgi?id=164325

Developing GlassFish using the NetBeans Maven project support is almost impossible due to very long and repetitive
project scans (without mentioning Editor red flags about missing classes when both Eclipse and IntelliJ can correctly
open the maven projects without errors.)
Attaching thread dumps is useless. Just check out the GlassFish open source repository, and open a few maven projects
from there. This is a very easy test case to reproduce.
https://glassfish.dev.java.net maven.
Comment 60 nleck 2009-05-04 08:27:09 UTC
When the project scans start after the initial large scan has completed why do these block the code completion ? it
should just use the old version. These scans should not be needed but if they are then at least they should not block
the code completion. This issue happens on Windows, Linux and Solaris. 
Comment 61 mjensen 2009-05-06 23:06:19 UTC
I too get this never ending project scanning, as if the scanning wasn't long enough as it was :D

Product Version: NetBeans IDE Dev (Build 200904011705)
Java: 1.5.0_15; Java HotSpot(TM) Client VM 1.5.0_15-b04
System: Windows 2003 version 5.2 running on x86; Cp1252; en_US (nb)
Userdir: C:\Documents and Settings\XXXXXX\.netbeans\6.7m3


Comment 62 Vitezslav Stejskal 2009-05-07 10:17:28 UTC
Please use latest dev build as there were several important fixes in this area, see for example issue #162706.
Comment 63 nleck 2009-05-08 02:57:40 UTC
Please see bug#164745 it's a deadlock while the project scan was running. This was with the daily build 200905071401
Comment 64 nleck 2009-05-08 03:09:41 UTC
The scan finished quickly *BUT* I can't find any classes. 

This maybe related to the deadlock which means I needed to kill/restart the ide twice.
Comment 65 nleck 2009-05-08 04:10:46 UTC
The scanning x of x message seems to no longer appear when opening files but the navigation is still disabled.  

The navigation is disabled for 10 or more minutes ( actually I haven't seen it enabled yet) 
Comment 66 nleck 2009-05-08 04:38:54 UTC
going back to the same file even after the navigation has appeared still takes far too long for the navigation to
re-appear. 

So it must be doing a large scan again. 

calling "go to declaration" on a method sightly fails. 
Comment 67 nleck 2009-05-11 04:00:32 UTC
Created attachment 81869 [details]
process bar stops giving feed back well before we can actually scam
Comment 68 nleck 2009-05-11 04:01:40 UTC
Please see attached the progress bar disappears well before we can actually navigate. 
Comment 69 nleck 2009-05-12 11:52:07 UTC
Created attachment 81946 [details]
not working
Comment 70 nleck 2009-05-12 11:53:56 UTC
Please see screen shot of the "go to type..." not working. 

There is no scanning happening, the classes have been there for years but yet the go to type can't find it. 
Comment 71 _ dcaoyuan 2009-05-19 10:30:11 UTC
I found a relate issue too:

Under erlang plugin, org.netbeans.modules.csl.editor.hyperlink.GoToSupport#perform always return null, the reason is:
IndexingManager.getDefault().isIndexing() always returns "true", it seems "state" in
org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater never be set back to "STOP". Actually, the "stop()" method
is only called in Intaller#closing().
Comment 72 Vitezslav Stejskal 2009-05-19 15:41:51 UTC
"IndexingManager.getDefault().isIndexing() always returns "true"" - As discussed in issue #165170 it was possible that
under certain circumstances the initial scan did not start after starting the IDE. This would also cause IM.isIndexing()
== true. This should already be fixed by http://hg.netbeans.org/jet-main/rev/fe24cebec2fe. Could you please try it?

Re. the STOP state - It's a terminal state used to stop indexing once and forever when the IDE is shutting down. After
the initial scan IM.isIndexing() should only return true when RepositoryUpdater.Task is running or RU is running in the
protected mode.
Comment 73 Vitezslav Stejskal 2009-05-20 13:45:53 UTC
This should fix the progress bar issue - http://hg.netbeans.org/jet-main/rev/b1a16d1468e8. As for the other problems
mentioned here I admit that it's becoming very confusing to see what this issue is about. Nevertheless I assume:

1. "slow scan / scan never finishes" is now fixed with issue #162706 and the scanning in recent dev builds perform
comparably to 6.5

2. "goto type can't find classes" - if caused by the same problem as discovered in issue #165170 it should be fixed by
http://hg.netbeans.org/jet-main/rev/fe24cebec2fe. Could you please verify once there is a build with this fix?

Anything else? I know this may sound irritating, but I'd really like to ask people to file each problem as a separate
issue and provide a reproducible testcase. Otherwise it's almost impossible to fix it. Thanks
Comment 74 nleck 2009-05-21 01:20:43 UTC
What about the re-scans ? They seem to be very frequent and blocks the navigation so much so that I now in the habit of
just doing a search instead as it seems to be faster than the goto type and more reliable. 
Comment 75 nleck 2009-05-21 01:27:02 UTC
from the news group. Please don't close this bug.. in no way is the project scanning acceptible yet. NB is far behind
Eclispe in this area. It is the reason for the majority of the developers here preferring Eclispe.


Jess Holle wrote:
> The 6.7 beta is a huge and horrible step backwards in this area.
>
> Search the archives and you'll see loads of discussions about this.
>
> From my standpoint things have either stayed the same or dis-improved in this area in most releases since 4.0, whereas
we need a marked improvement -- most especially in allowing navigation (go-to-type, etc) during scanning and in
navigation performance.
>
> 6.7 adds large amounts of rescanning whenever one opens a source file -- and is DOA as far as I'm concerned until
things are at least brought back to the state of 6.5 (which is still far from ideal).
>
> Thomas Wolf wrote:
>> Initially I was a really fervent supporter of those complaining about scanning being out of control in 6.7: on
startup, there were scans that not only seemed unnecessary (since nothing about the project had changed externally since
the last shutdown of the IDE; intermittently, there would be scans happening ad-hoc - and during those scans,
rudimentary IDE capabilities such as Go-To-Type would be non-functional.  Bugs were filed, comments added, and there
were some causes found for some of these symptoms.  Things seemed to be getting better as I downloaded newer dailies.
>>
>> But with recent builds I've started encountering this pesky companion again - this time in an unexpected spot: the
code completion:  I would start typing the name of an object or class followed by the '.' - normally, there's not much
of delay before NB offers me a popup with a list of methods/fields I could complete with....now, I often get a tooltip
"Scanning in progress" - that stays there for what appears to be like minutes.  When I move away from NB to get a
screenshot, as soon as I click outside the application, the tooltip disappears...so I can't even provide a screenshot of
the situation :-(  Anyway, if I erase the "." and re-type it (to get my auto-completion list), the tooltip
re-appears.....until, after whatever time it takes NB to do whatever it's doing, eventually I do get that list.
>>
>> "Find usages" also seems to have slowed down again - it just seems to occasionally stay in "Initializing Data..." for
prolonged periods of times (30 sec?  I didn't time it).
>>
>> I don't know what else to say - there are bugs with votes on this already and my experiences here aren't really
concrete enough to add as a comment to one of those.  I guess I just have to pray that this behavior gets resolved
before 6.7 ships.  Because if it doesn't, I think I'm finally going to give up on NB.  It's very depressing to be the
only NB developer in this Eclipse-using "pure Java" application shop and constantly getting snickers from people looking
over my shoulder as my IDE decides to do its scanning thing :-(  I've been using NB for about 10 years now - I would
hate to change, but eventually one has to wonder where the NB management's priorities lie - NB never had the greatest
performance, but if NB 6.7 goes out without this being fixed, it's obvious that the NB management is either clueless or
just decided that performance is "good enough" and putting its resources elsewhere.
>>
>> Hoping for the best,
>> Tom 
Comment 76 Quality Engineering 2009-05-21 08:28:10 UTC
Integrated into 'main-golden', will be available in build *200905210201* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main-golden/rev/b1a16d1468e8
User: Vita Stejskal <vstejskal@netbeans.org>
Log: #161367: once progress bar is shown for some work keep showing it even for jobs that don't require it
Comment 77 Vitezslav Stejskal 2009-05-21 20:17:47 UTC
"What about the re-scans?" - Ok, can you please use the latest dev build (usually anything older than a week is not
worth of trying), turn on logging in RepositoryUpdater, work as you usually do so that there are those frequent rescans
and then upload the IDE's log file here? Thanks
Comment 78 nleck 2009-05-22 07:52:39 UTC
I've downloaded/installed the 200905211401 build. 

I've got a new machine 12gigs of ram and 8 cpus

The members view for a class can take 10-30 seconds to appear a class ( first time) then a few seconds each time after
that. This seems very slow, the full clean compile of the whole project only takes 24 seconds. I'll attach a class that
was very slow to display the members view

see below output from the full compile:-

==========================================================================
build:
Compiling 4016 source files to /home/nigel/projects/ST/webapps/ROOT/WEB-INF/classes
Note: Some input files use or override a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
build-clean:
BUILD SUCCESSFUL (total time: 24 seconds)
Comment 79 nleck 2009-05-22 07:54:26 UTC
Created attachment 82606 [details]
sample Java file that was slow to show the members view
Comment 80 nleck 2009-05-22 07:58:34 UTC
While the "Members View" isn't showing the navigation is also disabled. So if I go to one of these classes ( remember
there are thousands of them) then I need to wait ~20 seconds to be able to navigate to the next class. In practice this
means it's faster to do a search. 
Comment 81 nleck 2009-05-22 08:00:32 UTC
ps. why is this bug marked as incomplete ? 
Comment 82 Vitezslav Stejskal 2009-05-22 09:30:38 UTC
"why is this bug marked as incomplete?" - Because that's its current state. There is a very simple rule, which says that
each problem should be filed as a separate issue. Or in other words, one issue should only deal with one problem.
Believe me there is a good reason for this rule. This particular issue seems to violate this rule, thanks to careless
people who just stash in here whatever pain they currently have. It's virtually impossible to fix issues like this. The
only think we can do with issues like this is to sit down, read through them and carefully categorize all the complaints
in a numbered list. And then go one by one and solve them.

Now, that's what I tried to do here and identified (1) "slow scan / scan never finishes" and (2) "goto type can't find
classes". I also linked these two problems to their separate issues and said that they should now be fixed. I then
nicely asked if there is something else that was reported in this issue and that I may have missed. You replied that
there are "frequent rescans" (3) that seem to be unnecessary. I said ok and asked you to turn on the logging in
RepositoryUpdater (steps were mentioned here in one of my first posts), work in the IDE as you normally do so that the
rescans would happen and send us the IDE's log file for examination.

Instead of doing what you were asked to and providing us the information that could help fixing (3) you started
complaining about slow members view and navigation. This simply leads nowhere and gives me no choice than ignoring this
last complaint and for that matter any further complaints voiced in this issue that do not concern (3). Please file
separate issues for problems that you find.

Do you now understand why this issue is marked as INCOMPLETE? If not, then I'm sorry, but we *really* can't help you.
Comment 83 nleck 2009-05-22 10:09:56 UTC
wow.... "I'm sorry, but we *really* can't help you"

I and many others are trying to say that the project scan is slow and it's worst than it was in 6.5. The project scan in
6.5 was very slow compared to Eclipse. 

This bug started with never ending project scans which was "fixed" but when we downloaded and tested, yes it was fast
but once finished we couldn't find any classes. Sure I guess the bug then morphed into I can find any classes. We are
now passed that and the scan is much faster than 6.7beta and the classes can be found but there seems to be lots of
smaller scans. These smaller scans are taking a far amount of time and makes navigation very slow. 

I believe that the navigation is the primary function of the IDE.

Surely for the navigation to be disabled for >30 seconds when opening a file is a major issue. This is after the initial
scan has complete, on a machine with local disks, 12 gigs of ram and 8 cpus. Maybe it should be a different bug but it
does seem to flow on. 

Saying all that we are just trying to make sure that the code completion and navigation gets some more attention. 

NB was the standard IDE here but most of the developers (~50) have defected over to another IDE of this reason alone.
Comment 84 nleck 2009-05-22 10:12:32 UTC
Created attachment 82614 [details]
message log with  -J-Dorg.netbeans.modules.parsing.impl.indexing.RepositoryUpdater.level=FINE
Comment 85 nleck 2009-05-22 10:16:06 UTC
I've just attached the message log with  -J-Dorg.netbeans.modules.parsing.impl.indexing.RepositoryUpdater.level=FINE 

I opened the IDE, waited for the initial scan which was very fast then opened DBObject.java ( source attached) the
navigation view came up after about 50 seconds.  
Comment 86 nleck 2009-05-22 10:23:44 UTC
Created attachment 82615 [details]
profile when opening one file
Comment 87 nleck 2009-05-22 10:26:10 UTC
Created attachment 82617 [details]
profile dump ( from profile me button) while opening a file
Comment 88 nleck 2009-05-22 10:29:07 UTC
attached is the profiler output. This profile was started after the ide had completed the initial scan. This was for
opening of one file and waiting for the navigation to appear. 
Comment 89 _ dcaoyuan 2009-05-22 18:26:41 UTC
The issue I reported has been fixed per my testing.
Comment 90 Dusan Balek 2009-05-28 13:59:15 UTC
Have you modified the DBObject.java document just after its opening somehow? What was the modification?
(based on the following line in the attached log file:
"FINE [org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater]: Document modified:
/home/nigel/src/ST/com/aspc/DBObj/DBObject.java Owner: file:/home/nigel/src/ST/")
Comment 91 Dusan Balek 2009-05-28 15:41:06 UTC
Vito, I have done some debugging and I have found the problem with opening files. Every file opened in editor is
automatically marked as modified (by ReposiotryUpdater using the AtomicLockListener instead of DocumentListener) and it
is immediately reindexed including all its dependent files. Also, IMHO, documents should not be reindexed on
FOCUS_GAINED property change (only on FOCUS_LOST).
Comment 92 Dusan Balek 2009-05-28 16:13:31 UTC
*** Issue 166188 has been marked as a duplicate of this issue. ***
Comment 93 nleck 2009-05-29 00:27:29 UTC
no DBObject.java doesn't change very often ( and not in this case). It's just a big file that when first open takes a
very long time for the navigation to appear. There are many other examples, I work for a big bank ( maybe not so big now
;-) so classes like Company.java and Submission.java are the same or worst but I can't attach them. 

The use case here was 1) start IDE 2) wait for initial scan to complete and things to settle down 3) Open DBObject.java
60+ seconds later the navigation panel appeared. This is on a 8 cpu machine with 12 of ram.

If I restart the IDE this happens again. There are lots of files this happens on. The main system that we work on has
some ~6000 source files and most of these are very stable ( not changed for years) but still the scan happens. 
Comment 94 bolsover 2009-05-29 11:19:33 UTC
This issue certainly gets my vote; performance issues associated with task/projectbscanning are crippling the IDE. 
Comment 95 greggwon 2009-05-29 20:44:40 UTC
Created attachment 83001 [details]
Here's a trace friom 6.1 where the event thread is blocked because of syncing with scanning/compiling
Comment 96 nleck 2009-05-30 00:23:11 UTC
Created attachment 83005 [details]
movie comparing Eclispe and NB opening a file showing the outline/navigator
Comment 97 nleck 2009-05-30 00:26:12 UTC
Please view the video attached I think it shows the performance difference between EC and NB fairly. 
Comment 98 _ ludo 2009-05-30 00:37:11 UTC
Hard to see in this small movie.
Also, I see many red icons on the Eclipse side, so the classpath is not correctly set for this project (hence we cannot
fairly compared the 2 IDEs because they are not looking and resolving the exact same classpath).

Comment 99 nleck 2009-05-30 00:45:37 UTC
Please view the attached horror movie, I think it shows the performance difference between EC and NB fairly.

step 1 - Open up NB 
step 2 - Open up EC.
step 3 - in NB call "Goto to type" for DBObect.java, window opens but no navigator
step 4 - in EC call "Open type" for DBObject.java , window and outline opens 
step 5 - in NB wait ~60 seconds 
step 6 - NB shows the navigator 

As there are many thousands of Java files in this project, the navigate functions are next to useless :-(
Comment 100 nleck 2009-05-30 01:03:57 UTC
Created attachment 83007 [details]
horror movie comparing Eclispe and NB opening a file showing the outline/navigator (video/ogg) -- no red marks in EC
Comment 101 nleck 2009-05-30 01:43:09 UTC
Just to add to it... The EC full scan ( when I change the classpath etc) is faster than the non scan that's happening
when we open a new file in NB. In the movie you can see it takes ~60 seconds for the navigator to appear, it only takes
~26 seconds to do a full build in NB and ~23 seconds for EC to do a full scan. In EC there isn't a time lag when opening
the outline view.

I could attach a movie of these if people need proof 
Comment 102 nleck 2009-05-30 04:32:26 UTC
Created attachment 83010 [details]
performance regression NB 6.5 v NB 6.7 daily
Comment 103 Marian Mirilovic 2009-05-30 13:16:33 UTC
nleck, thanks for video.

... adding performance team on cc

Oleg(ovk), 
please look at those use cases, would be nice to have this covered by automated tests.
Comment 104 Petr Jiricka 2009-05-30 20:22:40 UTC
I don't think it is necessary to provide more evidence - the problem has been acknowledged, and for example issue 166188
(which is marked as a duplicate of this issue) provides clear and easily reproducible steps. The problem is clear - the
question is how to fix it. And this issue HAS to be fixed before the final release.
Comment 105 nleck 2009-05-30 22:09:31 UTC
cool... that's what I wanted to hear " And this issue HAS to be fixed before the final release" 

I was getting frantic.

Thanks, Nigel.  
Comment 106 Marian Mirilovic 2009-06-01 11:25:25 UTC
Ok, never terminated scan is considered as a stopper for NB 6.7 .
Comment 107 Vitezslav Stejskal 2009-06-01 16:26:42 UTC
http://hg.netbeans.org/jet-main/rev/bbe2fbcd4d16

I fixed rescanning of modified documents. Changes are now correctly detected in AtomicLockListener and only modified
documents are rescanned. Also rescanning is not done on focus gained event. Nigel could you please try if it improved
situation with DBObject? Thanks
Comment 108 nleck 2009-06-01 21:39:07 UTC
Which build will http://hg.netbeans.org/jet-main/rev/bbe2fbcd4d16 be in ? 
Comment 109 Tomas Pavek 2009-06-02 05:06:15 UTC
A notification will arrive to this issue when the fix appears in a daily build.
Comment 110 Quality Engineering 2009-06-02 08:44:08 UTC
Integrated into 'main-golden', will be available in build *200906020201* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main-golden/rev/bbe2fbcd4d16
User: Vita Stejskal <vstejskal@netbeans.org>
Log: #161367 - better detection of document modifications before triggering document reindexing
Comment 112 nleck 2009-06-02 11:45:17 UTC
Created attachment 83094 [details]
FIXED of performance regression
Comment 113 Marian Mirilovic 2009-06-02 11:53:48 UTC
nleck, 40 vs. 3 seconds ... it looks fixed, right ? Thanks a lot!
Comment 114 nleck 2009-06-02 12:21:21 UTC
That is SO MUCH faster :-) 

It's back to 6.5 performance for the "initial open of file" task. 

Thank you.

The full project scan took (after deleting the .netbeans directory) took 5 min 10 seconds which is much more than the
full compile of 23 seconds but massively better than the 6.7 beta of ~1.5 hours.
Comment 115 nleck 2009-06-02 12:23:36 UTC
yes, the performance regression is fixed ( still would like some more but not for 6.7 )
Comment 116 Dusan Balek 2009-06-02 12:44:25 UTC
The fix looks OK.
Comment 117 Vitezslav Stejskal 2009-06-02 12:51:16 UTC
The fix was translated to release67 clone as http://hg.netbeans.org/release67/rev/602bbec4d872.
Comment 118 Jiri Prox 2009-06-05 10:56:50 UTC
verified
Comment 119 nleck 2009-06-06 06:22:33 UTC
Sorry need to re-open as the regression is there with one extra step as soon as you touch the file the "goto type" is
blocked and this DID NOT HAPPEN IN 6.5

performance regression in 6.7rc2 see:-
http://www.netbeans.org/nonav/issues/showattachment.cgi/83267/nb_scan.ogg

http://www.netbeans.org/nonav/issues/showattachment.cgi/83269/nb_scan-6.5.ogg
Comment 120 nleck 2009-06-08 02:15:47 UTC
Created attachment 83291 [details]
scan is out of date
Comment 121 nleck 2009-06-08 02:22:09 UTC
After making the performance regression movie in the previous comment, the file WASN'T saved. I actually even restarted
my IDE and noticed all these red marks... The indexer had stored the information of a Java file that wasn't ever saved.
Thus now out of date and will be until I change the Java file again. 
Comment 122 David Strupl 2009-06-08 12:39:59 UTC
Nigel, this seems to be a different problem. I have created a new issue #166714 and put you on cc there.
Comment 123 David Strupl 2009-06-08 12:40:38 UTC
This issue was in verified state before.
Comment 124 nleck 2009-06-08 12:49:02 UTC
The performance regression is the same issue as in
http://www.netbeans.org/nonav/issues/showattachment.cgi/83010/nb_performance_regression.ogg but with one more step. 

The performance in 6.7 *HAS* gone backwards and it wasn't good to begin with. By creating a new bug the next thing
you'll say is that'll be fix in another release but it is *THIS* release that is making the performance worst. What is
the feature that has been added in this release that is causing all the NEW scanning issues ? 

Look at the movie http://www.netbeans.org/nonav/issues/showattachment.cgi/83007/nb_scan_embarsement-small.ogg the
difference is so stark that most of the developers here have already move over now we are making it worst. 

The issue in http://www.netbeans.org/nonav/issues/showattachment.cgi/83291/scan_wrong.png is directly related to the fix
done. 

I really think you should reopen this as the fix has problems. Any large project will have the same issues that we do. 
Comment 126 nleck 2009-06-08 12:53:05 UTC
see how it was much faster in 6.5 :-

    http://www.netbeans.org/nonav/issues/showattachment.cgi/83269/nb_scan-6.5.ogg
Comment 127 Marian Mirilovic 2009-06-08 13:55:06 UTC
I agree with dstrupl here, this issue was already fixed/verified. We are treating newly reported problem (issue 166714)
as a stopper for NB 6.7 as well - its' a serious regression with target to be fixed in NB 6.7 FCS. 
So please keep those two problems in separate threads, thanks in advance.