This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
[ BUILD # : 200903250219 ] [ JDK VERSION : 1.6.* ] One of my opened projects, a Maven web project, is being scanned but the scan task does not terminate. However, I cannot reproduce this in a reliable fashion and my IDE log does not indicate any problems. I'm not sure if this is related, but my IDE uses 50% of my CPU power at the moment and does no longer respond to UI input.
scanning is done in java/source?
This is already resolved if I'm correct. @mkleint: I don't think so.
Can you take several threaddumps when the IDE is scanning and attach them here? Also could you possibly run the IDE with -J-Dorg.netbeans.modules.parsing.impl.indexing.RepositoryUpdater.level=FINE switch and attach here the <ide-userdir>/var/log/message.log file>? Thanks
I've got the same issue, 6.7m3 is *NOT* usable at all. No members ever appear, no naviagtion possible. Our project consists of ~3,000 source files.
Created attachment 79435 [details] message log
This also means that debugging is not enabled ( it seems scanning needs to complete first)
Can you, please, run IDE with -J-Dorg.netbeans.modules.parsing.impl.indexing.RepositoryUpdater.level=FINE and attach messages.log as suggested by vstejskal?
I've been having this scanning issue for quite some time now. This morning, I started witha completely new slate: removed my $HOME/.netbeans.dev/ directory and downloaded/installed the newest daily (04/07/2009) build and started it. Soon after opening the project, the IDE began scanning....when, after a minute or two, it reached 650Megs (with CPU pegged at near 100%), I went to a terminal and ran jstack against the process - with no results. I repeated with "jstack -F". See attachment for resulting dump in trace.txt. I restarted the IDE and, again, it went to max memory. This time a plain "jstack" worked and produced a dump. Attached as "trace2.txt". I made this bug a P1 because the IDE has basically become useless. I am reverting to 6.5.1 so I can get some work done.
Created attachment 79649 [details] thread dump when IDE reached 780mb (near limit)
Created attachment 79650 [details] second heap dump after IDE was restarted and hung again
Created attachment 79652 [details] messages.log showing NullPointerException
Also, I noticed that there was a NullPointerException in the messages.log, so I attached that as well. There's also a heap dump - but it's huge. Not sure if that might be of use.
the exception is already reported as issue 162016
Do you have the tasklist window open or not? There is a problem with tasklist filter, which means that even task scanners that are turned off are still used for scanning, see issue #162076. Could you please also check what tasklist filter you use. It's in Windows -> Tasks, then click the filter icon and Edit... Normally, 'Default Filter' is selected and as part of this filter there are two task scanners enabled - TODO and Java Errors. Also, if this happens again please create several successive threaddumps and attach them here. Thanks
I don't use task lists (turn them off via Tools->Options->Editor->Tasks uncheck "Enable Java Tasklist") and close the tasklist window that appears when opening the first project for the first time after installation. As far as what Task filters are enabled: I just reverted from 6.5.1 and, lo and behold, the IDE came up without hanging (yesterday, I could reproduce each time I started the IDE!). For some reason the Task window came up - even though I could have sworn yesterday I did not have it open (since, as mentioned above, I don't even use Task lists!) I checked and the defaults are what you described (Java Errors and TODO). And my "Enable Java Tasklist" is unchecked as expected. So I'm clueless - the only thing that differed between yesterday's multiple IDE startups which all resulted in hanging and today is that in-between I reverted to 6.5.1, did some PMD stuff, rebuilt the project a few times, and then reverted back to the daily!
"close the tasklist window that appears when opening the first project for the first time after installation." - that's exactly what is causing problems reported in issue #162076. Could you please try leaving the window open and see if you still have the problem. On my laptop the problem was 100% reproducible with tasklist window turned off and worked like a charm with the window on. If that's the case I'd say that this is a duplicate of #162076. Before #162076 is fixed you might try uninstalling the tasklist modules through the plugin manager (Tools -> Plugins), which should once and for all silence its scanners. Thanks a lot
Scanning has continued for exactly 1 hour. Have 3 thread dumps, 5 minutes, 30 minutes, and 1 hour.
Created attachment 79861 [details] 200904091401 Thread Dump #1 (5 minutes)
Created attachment 79862 [details] 200904091401 Thread Dump #2 (30 minutes)
Created attachment 79863 [details] 200904091401 Thread Dump #3 (1 hour)
And for what it's worth, here's a heap dump as well (hprof binary format, ~90mb) http://theeshadow.com/files/200904091401/heapdump-1239329788619.hprof Build 200904091401, 1 hour and *still* "scanning projects..." Note, I have not opened any windows, touched anything at all in the IDE. Just launched via terminal, and waited for it to finish scanning.
Changing to All and All, I have OS X 10.5.6
Thanks for the threaddumps. All of them show activity in JavaCustomIndexer and so this really is unrelated to tasklist problem. How many projects do you have open and what's their size (approx) and type (java/web/...)? Where are the projects stored (local disk)? How fast is scanning of the same projects in 6.5? Thanks
I think we maybe confused on this bug. I mean the project scanning. We've turned off the "task list" completely, it locked up our projects too for very little gain. It's the project scanning that's a show stopper for me anyway. No debugging or navigation.
Created attachment 79872 [details] log of when project scan never finishes ( on windoze)
nleck - Could you please describe your project setup (eg project type, files layout, etc)? Can you reproduce this state? What do you do in order to get the IDE in this state? We might need more logging to diagnose it or ideally a small reproducible testcase. Thanks
Our project consists of some 6,086 java files. The project is quite old with many developers. We work in a mixed environment of mostly Eclipse users so change the project structure is not really an option. The same project is developed on Solaris, Windows and Linux. The constant scanning seems to only effect Windows and Solaris. On Solaris we have the sources on a mounted network drive ( again not by choice). We have the sources and the jars required to build it all checked into cvs. The constant scanning didn't happen in 6.5 but the slow scanning was the major reason why the most developers still use Eclipse.
What project type is it? Freeform? It looks like evrything you have in under the same root - sources, libraries, library sources. I'm just trying to replicate your setup and simulate that never ending loop.
yep, freeform with everything under one root.
Could you please generate several successive thread dumps when the IDE is scanning and attach them here? Thanks
And where are the project's java files compiled to?
the class files are compiled out of the source tree into the $project director$/webapps/ROOT/WEB-INF/classes
vestejskal - Threaddumps from 2009 04 09 1401 were with one small project open on an empty user directory. The scanning never terminated. However after closing the tasks window and restarting the IDE, there were no problems. I am unable to replicate the problem on 2009 04 14 0201. This build is very solid! If I can get someone else to confirm, I will change this to fixed.
Created attachment 80098 [details] multiple thread dumps while IDE is doing a project scan
looks like i also was able to reproduce will attache threaddumps & co.
Created attachment 80331 [details] threaddumps
Created attachment 80332 [details] messages.log
tstupka - are you able to reproduce the problem using the build containing http://hg.netbeans.org/main-golden?cmd=changeset;node=6f3d60df0df9
I'm getting NPE while scanning in the latest daily build. Is there a way to stop the scanning process from looking at the lib directory at all ? SEVERE [org.netbeans.modules.java.source.TreeLoader]: Coupling error: class file jar:file:/home/nigel/src/DC/lib/xalan.jar!/org/apache/bcel/Constants.class, source file file:/home/nigel/src/DC/lib/src/org/apache/bcel/Constants.java SEVERE [org.netbeans.modules.java.source.TreeLoader]: Coupling error: class file jar:file:/home/nigel/src/DC/lib/xalan.jar!/org/apache/bcel/classfile/AccessFlags.class, source file file:/home/nigel/src/DC/lib/src/org/apache/bcel/classfile/AccessFlags.java SEVERE [org.netbeans.modules.java.source.TreeLoader]: Coupling error: class file jar:file:/home/nigel/src/DC/lib/xalan.jar!/org/apache/bcel/classfile/Constant.class, source file file:/home/nigel/src/DC/lib/src/org/apache/bcel/classfile/Constant.java SEVERE [org.netbeans.modules.java.source.TreeLoader]: Coupling error: class file jar:file:/home/nigel/src/DC/lib/xalan.jar!/org/apache/bcel/classfile/Attribute.class, source file file:/home/nigel/src/DC/lib/src/org/apache/bcel/classfile/Attribute.java SEVERE [org.netbeans.modules.java.source.TreeLoader]: Coupling error: class file jar:file:/home/nigel/src/DC/lib/xalan.jar!/org/apache/bcel/classfile/LocalVariableTable.class, source file file:/home/nigel/src/DC/lib/src/org/apache/bcel/classfile/LocalVariableTable.java SEVERE [org.netbeans.modules.java.source.TreeLoader]: Coupling error: class file jar:file:/home/nigel/src/DC/lib/commons-logging-1.0.3.jar!/org/apache/commons/logging/impl/Log4JLogger.class, source file file:/home/nigel/src/DC/lib/src/org/apache/commons/logging/impl/Log4JLogger.java INFO [org.netbeans.modules.xml.schema.completion.util.CompletionContextImpl] java.lang.NullPointerException at org.netbeans.modules.xml.schema.completion.util.CompletionContextImpl.populateModelMap(CompletionContextImpl.java:612) [catch] at org.netbeans.modules.xml.schema.completion.util.CompletionContextImpl.specialCompletion(CompletionContextImpl.java:637) at org.netbeans.modules.xml.schema.completion.util.CompletionContextImpl.initModels(CompletionContextImpl.java:606) at org.netbeans.modules.xml.schema.completion.CompletionQuery.getCompletionItems(CompletionQuery.java:103) at org.netbeans.modules.xml.schema.completion.CompletionQuery.query(CompletionQuery.java:87) at org.netbeans.spi.editor.completion.support.AsyncCompletionTask.run(AsyncCompletionTask.java:218) at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:573) at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:1005) INFO [org.netbeans.modules.xml.schema.completion.util.CompletionContextImpl] java.lang.NullPointerException at org.netbeans.modules.xml.schema.completion.util.CompletionContextImpl.populateModelMap(CompletionContextImpl.java:612) [catch] at org.netbeans.modules.xml.schema.completion.util.CompletionContextImpl.specialCompletion(CompletionContextImpl.java:637) at org.netbeans.modules.xml.schema.completion.util.CompletionContextImpl.initModels(CompletionContextImpl.java:606) at org.netbeans.modules.xml.schema.completion.CompletionQuery.getCompletionItems(CompletionQuery.java:103) at org.netbeans.modules.xml.schema.completion.CompletionQuery.query(CompletionQuery.java:87) at org.netbeans.spi.editor.completion.support.AsyncCompletionTask.run(AsyncCompletionTask.java:218) at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:573) at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:1005) INFO [org.netbeans.modules.xml.schema.completion.util.CompletionContextImpl] java.lang.NullPointerException at org.netbeans.modules.xml.schema.completion.util.CompletionContextImpl.populateModelMap(CompletionContextImpl.java:612) [catch] at org.netbeans.modules.xml.schema.completion.util.CompletionContextImpl.specialCompletion(CompletionContextImpl.java:637) at org.netbeans.modules.xml.schema.completion.util.CompletionContextImpl.initModels(CompletionContextImpl.java:606) at org.netbeans.modules.xml.schema.completion.CompletionQuery.getCompletionItems(CompletionQuery.java:103) at org.netbeans.modules.xml.schema.completion.CompletionQuery.query(CompletionQuery.java:87) at org.netbeans.spi.editor.completion.support.AsyncCompletionTask.run(AsyncCompletionTask.java:218) at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:573) at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:1005) INFO [org.netbeans.modules.xml.schema.completion.util.CompletionContextImpl] java.lang.NullPointerException at org.netbeans.modules.xml.schema.completion.util.CompletionContextImpl.populateModelMap(CompletionContextImpl.java:612) [catch] at org.netbeans.modules.xml.schema.completion.util.CompletionContextImpl.specialCompletion(CompletionContextImpl.java:637) at org.netbeans.modules.xml.schema.completion.util.CompletionContextImpl.initModels(CompletionContextImpl.java:606) at org.netbeans.modules.xml.schema.completion.CompletionQuery.getCompletionItems(CompletionQuery.java:103) at org.netbeans.modules.xml.schema.completion.CompletionQuery.query(CompletionQuery.java:87) at org.netbeans.spi.editor.completion.support.AsyncCompletionTask.run(AsyncCompletionTask.java:218) at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:573) at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:1005)
The scanning now stops in the latest build but it's scanning 8871 source files ( because of the ones in our lib directory). We should only have 3500 source files.
"The scanning now stops in the latest build" - Good. This was most likely fixed by http://hg.netbeans.org/jet-main/rev/6f3d60df0df9 (available in builds after Apr-16). "I'm getting NPE while scanning" - The NPE is IMO unrelated to scanning and comes from the xml.schema module. Please file a separate issue for this. "Is there a way to stop the scanning process from looking at the lib directory at all ?" - No sorry. You will have to rearrange your project so that it does not mix libraries and sources under the same source root.
while this project scan now completes it is still much slower than Eclipse on the same project. Can the performance be improved ?
Gotta re-open now it can't find classes that are there.
When we first open the project it does a large project scan. Use "Go To Type..." type in a class name that I know is there and it doesn't find it. I then manually navigate to this class. The IDE does another "project scan" and then we can find the class. This is obviously seriously broken.
The latest build has taken over half an hour saying "Scanning projects 688 of 688" it seems that the IDE scans different this directories depending on what we click on.
ok... it's been scanning for a very long time now. This bug has not been fixed
Created attachment 80733 [details] stack dumps.
Can you also attach the IDE log file? With -J-Dorg.netbeans.modules.parsing.impl.indexing.RepositoryUpdater.level=FINE the log file should now show stactraces of scan requests.
I removed my .netbeans/dev/cache directory and restarted the IDE and left it for quite a while and the scan finished plus I was able to find all the existing classes that I looked for. It seems to be using the IDE while it was scanning caused issues.
The project scan terminated in my first NB 6.7beta session, however it took about an hour. One thing I noticed was that scanning of .properties files was especially slow. There were 668 UTF-8 properties-files in my project and the IDE log said: FEIN [org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater]: Indexing 668 indexables; using org.netbeans.modules.tasklist.impl.TaskIndexer@9b5088; mimeType='text/x-properties' FEIN [org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater]: Using EmbeddingIndexers for [FileObjectIndexable@c41522 [file:/D:/_projects/tgo/3.9/tgo-mo/webapp/WEB-INF/classes/i18n/content/common/cities_zh_CN.properties], FileObjectIndexable@14771ba ..... this goes on for the other files.
I downloaded 6.7beta too (on windows only) and the project scan completed on first boot too and I could find all the classes that I searched for :-) As I navigate I can see the project scan start again as I go into different files and the scan is for many files but no files have been changed so this is an miss fire. Also in our case the project scan is picking up a lot of files that aren't in our project. It would be great if we could omit directories from the scan. I'll download the beta for Solaris and try it there.
Running the 6.7beta on Solaris it's been scanning 13,512 files for a every long time now ( a hour or more). We don't have anywhere near 13,512 files. It must have dug files out of the jar files or double counting. When we do a full compile of the system it is 5 minutes for 6,173 Java files. Our project is a freeform project it would be great if we could exclude directories from the scan. Now even given that there must be double the files scan than exists in the project that should only equate to 10 minutes if the scan performance was the same as javac, currently it's at least 6-7 times slower. The IDE is currently blocked saying ( 10937 of 13512) with seemingly no progress for a long time ( 20-30 minutes), I've taken a number of ctrl-\ and will attach to this bug.
Created attachment 81269 [details] many thread dumps while scanning
In the end it took 1.5 hours to scan 13,512 files ( should have been 6,173 Java files).... the real kicker is when I go to another file it starts the project scan again and blocks the "Go to Type" until this is done. Once a full scan has been done the "Go to Type" should not block even if the IDE is scanning a new file. In this case *NO* files have been changed so why it's scanning and why it takes so long is a complete mystery to me. The "Scanning projects... " is saying (3929 of 3929) THIS IS AFTER WAITING AN HOUR AN A HALF FOR THE INITIAL SCAN.
When I shutdown my IDE and start it up again.. The project scan restarts even thou no files have changed. Again the IDE is not usable until this finishes.
> 13,512 files ( should have been 6,173 Java files) Do you have symlinked dirs? I have seen the same problem (+ all objects were presented twice in Go To Type dialog) in the following configuration: I have: /export/home/user/source/nb/trunk and it is symlinked as /home/user/trunk -> /export/home/user/source/nb/trunk Then trying to open trunk using symlink doubles all objects (and files counter)
no symblic links in this project but the home directories are NFS mounted.
I'm also seeing very long Scanning tasks when editing the main-golden source code for the java.hints project. Feedback shows that dependencies/dependants are also being scanned! (that's a *lot* of source). I only have java.{source, editor, hints} open. Things speed up considerably if I disable dependencies in Java Tasklist, although I can still see the projects being scanned (albeit quickly) during startup. Given that the Task list only cares about opened projects... why are unopened projects ever scanned in the first place?
See as well http://www.netbeans.org/issues/show_bug.cgi?id=164325 Developing GlassFish using the NetBeans Maven project support is almost impossible due to very long and repetitive project scans (without mentioning Editor red flags about missing classes when both Eclipse and IntelliJ can correctly open the maven projects without errors.) Attaching thread dumps is useless. Just check out the GlassFish open source repository, and open a few maven projects from there. This is a very easy test case to reproduce. https://glassfish.dev.java.net maven.
When the project scans start after the initial large scan has completed why do these block the code completion ? it should just use the old version. These scans should not be needed but if they are then at least they should not block the code completion. This issue happens on Windows, Linux and Solaris.
I too get this never ending project scanning, as if the scanning wasn't long enough as it was :D Product Version: NetBeans IDE Dev (Build 200904011705) Java: 1.5.0_15; Java HotSpot(TM) Client VM 1.5.0_15-b04 System: Windows 2003 version 5.2 running on x86; Cp1252; en_US (nb) Userdir: C:\Documents and Settings\XXXXXX\.netbeans\6.7m3
Please use latest dev build as there were several important fixes in this area, see for example issue #162706.
Please see bug#164745 it's a deadlock while the project scan was running. This was with the daily build 200905071401
The scan finished quickly *BUT* I can't find any classes. This maybe related to the deadlock which means I needed to kill/restart the ide twice.
The scanning x of x message seems to no longer appear when opening files but the navigation is still disabled. The navigation is disabled for 10 or more minutes ( actually I haven't seen it enabled yet)
going back to the same file even after the navigation has appeared still takes far too long for the navigation to re-appear. So it must be doing a large scan again. calling "go to declaration" on a method sightly fails.
Created attachment 81869 [details] process bar stops giving feed back well before we can actually scam
Please see attached the progress bar disappears well before we can actually navigate.
Created attachment 81946 [details] not working
Please see screen shot of the "go to type..." not working. There is no scanning happening, the classes have been there for years but yet the go to type can't find it.
I found a relate issue too: Under erlang plugin, org.netbeans.modules.csl.editor.hyperlink.GoToSupport#perform always return null, the reason is: IndexingManager.getDefault().isIndexing() always returns "true", it seems "state" in org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater never be set back to "STOP". Actually, the "stop()" method is only called in Intaller#closing().
"IndexingManager.getDefault().isIndexing() always returns "true"" - As discussed in issue #165170 it was possible that under certain circumstances the initial scan did not start after starting the IDE. This would also cause IM.isIndexing() == true. This should already be fixed by http://hg.netbeans.org/jet-main/rev/fe24cebec2fe. Could you please try it? Re. the STOP state - It's a terminal state used to stop indexing once and forever when the IDE is shutting down. After the initial scan IM.isIndexing() should only return true when RepositoryUpdater.Task is running or RU is running in the protected mode.
This should fix the progress bar issue - http://hg.netbeans.org/jet-main/rev/b1a16d1468e8. As for the other problems mentioned here I admit that it's becoming very confusing to see what this issue is about. Nevertheless I assume: 1. "slow scan / scan never finishes" is now fixed with issue #162706 and the scanning in recent dev builds perform comparably to 6.5 2. "goto type can't find classes" - if caused by the same problem as discovered in issue #165170 it should be fixed by http://hg.netbeans.org/jet-main/rev/fe24cebec2fe. Could you please verify once there is a build with this fix? Anything else? I know this may sound irritating, but I'd really like to ask people to file each problem as a separate issue and provide a reproducible testcase. Otherwise it's almost impossible to fix it. Thanks
What about the re-scans ? They seem to be very frequent and blocks the navigation so much so that I now in the habit of just doing a search instead as it seems to be faster than the goto type and more reliable.
from the news group. Please don't close this bug.. in no way is the project scanning acceptible yet. NB is far behind Eclispe in this area. It is the reason for the majority of the developers here preferring Eclispe. Jess Holle wrote: > The 6.7 beta is a huge and horrible step backwards in this area. > > Search the archives and you'll see loads of discussions about this. > > From my standpoint things have either stayed the same or dis-improved in this area in most releases since 4.0, whereas we need a marked improvement -- most especially in allowing navigation (go-to-type, etc) during scanning and in navigation performance. > > 6.7 adds large amounts of rescanning whenever one opens a source file -- and is DOA as far as I'm concerned until things are at least brought back to the state of 6.5 (which is still far from ideal). > > Thomas Wolf wrote: >> Initially I was a really fervent supporter of those complaining about scanning being out of control in 6.7: on startup, there were scans that not only seemed unnecessary (since nothing about the project had changed externally since the last shutdown of the IDE; intermittently, there would be scans happening ad-hoc - and during those scans, rudimentary IDE capabilities such as Go-To-Type would be non-functional. Bugs were filed, comments added, and there were some causes found for some of these symptoms. Things seemed to be getting better as I downloaded newer dailies. >> >> But with recent builds I've started encountering this pesky companion again - this time in an unexpected spot: the code completion: I would start typing the name of an object or class followed by the '.' - normally, there's not much of delay before NB offers me a popup with a list of methods/fields I could complete with....now, I often get a tooltip "Scanning in progress" - that stays there for what appears to be like minutes. When I move away from NB to get a screenshot, as soon as I click outside the application, the tooltip disappears...so I can't even provide a screenshot of the situation :-( Anyway, if I erase the "." and re-type it (to get my auto-completion list), the tooltip re-appears.....until, after whatever time it takes NB to do whatever it's doing, eventually I do get that list. >> >> "Find usages" also seems to have slowed down again - it just seems to occasionally stay in "Initializing Data..." for prolonged periods of times (30 sec? I didn't time it). >> >> I don't know what else to say - there are bugs with votes on this already and my experiences here aren't really concrete enough to add as a comment to one of those. I guess I just have to pray that this behavior gets resolved before 6.7 ships. Because if it doesn't, I think I'm finally going to give up on NB. It's very depressing to be the only NB developer in this Eclipse-using "pure Java" application shop and constantly getting snickers from people looking over my shoulder as my IDE decides to do its scanning thing :-( I've been using NB for about 10 years now - I would hate to change, but eventually one has to wonder where the NB management's priorities lie - NB never had the greatest performance, but if NB 6.7 goes out without this being fixed, it's obvious that the NB management is either clueless or just decided that performance is "good enough" and putting its resources elsewhere. >> >> Hoping for the best, >> Tom
Integrated into 'main-golden', will be available in build *200905210201* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main-golden/rev/b1a16d1468e8 User: Vita Stejskal <vstejskal@netbeans.org> Log: #161367: once progress bar is shown for some work keep showing it even for jobs that don't require it
"What about the re-scans?" - Ok, can you please use the latest dev build (usually anything older than a week is not worth of trying), turn on logging in RepositoryUpdater, work as you usually do so that there are those frequent rescans and then upload the IDE's log file here? Thanks
I've downloaded/installed the 200905211401 build. I've got a new machine 12gigs of ram and 8 cpus The members view for a class can take 10-30 seconds to appear a class ( first time) then a few seconds each time after that. This seems very slow, the full clean compile of the whole project only takes 24 seconds. I'll attach a class that was very slow to display the members view see below output from the full compile:- ========================================================================== build: Compiling 4016 source files to /home/nigel/projects/ST/webapps/ROOT/WEB-INF/classes Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. Note: Some input files use unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. build-clean: BUILD SUCCESSFUL (total time: 24 seconds)
Created attachment 82606 [details] sample Java file that was slow to show the members view
While the "Members View" isn't showing the navigation is also disabled. So if I go to one of these classes ( remember there are thousands of them) then I need to wait ~20 seconds to be able to navigate to the next class. In practice this means it's faster to do a search.
ps. why is this bug marked as incomplete ?
"why is this bug marked as incomplete?" - Because that's its current state. There is a very simple rule, which says that each problem should be filed as a separate issue. Or in other words, one issue should only deal with one problem. Believe me there is a good reason for this rule. This particular issue seems to violate this rule, thanks to careless people who just stash in here whatever pain they currently have. It's virtually impossible to fix issues like this. The only think we can do with issues like this is to sit down, read through them and carefully categorize all the complaints in a numbered list. And then go one by one and solve them. Now, that's what I tried to do here and identified (1) "slow scan / scan never finishes" and (2) "goto type can't find classes". I also linked these two problems to their separate issues and said that they should now be fixed. I then nicely asked if there is something else that was reported in this issue and that I may have missed. You replied that there are "frequent rescans" (3) that seem to be unnecessary. I said ok and asked you to turn on the logging in RepositoryUpdater (steps were mentioned here in one of my first posts), work in the IDE as you normally do so that the rescans would happen and send us the IDE's log file for examination. Instead of doing what you were asked to and providing us the information that could help fixing (3) you started complaining about slow members view and navigation. This simply leads nowhere and gives me no choice than ignoring this last complaint and for that matter any further complaints voiced in this issue that do not concern (3). Please file separate issues for problems that you find. Do you now understand why this issue is marked as INCOMPLETE? If not, then I'm sorry, but we *really* can't help you.
wow.... "I'm sorry, but we *really* can't help you" I and many others are trying to say that the project scan is slow and it's worst than it was in 6.5. The project scan in 6.5 was very slow compared to Eclipse. This bug started with never ending project scans which was "fixed" but when we downloaded and tested, yes it was fast but once finished we couldn't find any classes. Sure I guess the bug then morphed into I can find any classes. We are now passed that and the scan is much faster than 6.7beta and the classes can be found but there seems to be lots of smaller scans. These smaller scans are taking a far amount of time and makes navigation very slow. I believe that the navigation is the primary function of the IDE. Surely for the navigation to be disabled for >30 seconds when opening a file is a major issue. This is after the initial scan has complete, on a machine with local disks, 12 gigs of ram and 8 cpus. Maybe it should be a different bug but it does seem to flow on. Saying all that we are just trying to make sure that the code completion and navigation gets some more attention. NB was the standard IDE here but most of the developers (~50) have defected over to another IDE of this reason alone.
Created attachment 82614 [details] message log with -J-Dorg.netbeans.modules.parsing.impl.indexing.RepositoryUpdater.level=FINE
I've just attached the message log with -J-Dorg.netbeans.modules.parsing.impl.indexing.RepositoryUpdater.level=FINE I opened the IDE, waited for the initial scan which was very fast then opened DBObject.java ( source attached) the navigation view came up after about 50 seconds.
Created attachment 82615 [details] profile when opening one file
Created attachment 82617 [details] profile dump ( from profile me button) while opening a file
attached is the profiler output. This profile was started after the ide had completed the initial scan. This was for opening of one file and waiting for the navigation to appear.
The issue I reported has been fixed per my testing.
Have you modified the DBObject.java document just after its opening somehow? What was the modification? (based on the following line in the attached log file: "FINE [org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater]: Document modified: /home/nigel/src/ST/com/aspc/DBObj/DBObject.java Owner: file:/home/nigel/src/ST/")
Vito, I have done some debugging and I have found the problem with opening files. Every file opened in editor is automatically marked as modified (by ReposiotryUpdater using the AtomicLockListener instead of DocumentListener) and it is immediately reindexed including all its dependent files. Also, IMHO, documents should not be reindexed on FOCUS_GAINED property change (only on FOCUS_LOST).
*** Issue 166188 has been marked as a duplicate of this issue. ***
no DBObject.java doesn't change very often ( and not in this case). It's just a big file that when first open takes a very long time for the navigation to appear. There are many other examples, I work for a big bank ( maybe not so big now ;-) so classes like Company.java and Submission.java are the same or worst but I can't attach them. The use case here was 1) start IDE 2) wait for initial scan to complete and things to settle down 3) Open DBObject.java 60+ seconds later the navigation panel appeared. This is on a 8 cpu machine with 12 of ram. If I restart the IDE this happens again. There are lots of files this happens on. The main system that we work on has some ~6000 source files and most of these are very stable ( not changed for years) but still the scan happens.
This issue certainly gets my vote; performance issues associated with task/projectbscanning are crippling the IDE.
Created attachment 83001 [details] Here's a trace friom 6.1 where the event thread is blocked because of syncing with scanning/compiling
Created attachment 83005 [details] movie comparing Eclispe and NB opening a file showing the outline/navigator
Please view the video attached I think it shows the performance difference between EC and NB fairly.
Hard to see in this small movie. Also, I see many red icons on the Eclipse side, so the classpath is not correctly set for this project (hence we cannot fairly compared the 2 IDEs because they are not looking and resolving the exact same classpath).
Please view the attached horror movie, I think it shows the performance difference between EC and NB fairly. step 1 - Open up NB step 2 - Open up EC. step 3 - in NB call "Goto to type" for DBObect.java, window opens but no navigator step 4 - in EC call "Open type" for DBObject.java , window and outline opens step 5 - in NB wait ~60 seconds step 6 - NB shows the navigator As there are many thousands of Java files in this project, the navigate functions are next to useless :-(
Created attachment 83007 [details] horror movie comparing Eclispe and NB opening a file showing the outline/navigator (video/ogg) -- no red marks in EC
Just to add to it... The EC full scan ( when I change the classpath etc) is faster than the non scan that's happening when we open a new file in NB. In the movie you can see it takes ~60 seconds for the navigator to appear, it only takes ~26 seconds to do a full build in NB and ~23 seconds for EC to do a full scan. In EC there isn't a time lag when opening the outline view. I could attach a movie of these if people need proof
Created attachment 83010 [details] performance regression NB 6.5 v NB 6.7 daily
nleck, thanks for video. ... adding performance team on cc Oleg(ovk), please look at those use cases, would be nice to have this covered by automated tests.
I don't think it is necessary to provide more evidence - the problem has been acknowledged, and for example issue 166188 (which is marked as a duplicate of this issue) provides clear and easily reproducible steps. The problem is clear - the question is how to fix it. And this issue HAS to be fixed before the final release.
cool... that's what I wanted to hear " And this issue HAS to be fixed before the final release" I was getting frantic. Thanks, Nigel.
Ok, never terminated scan is considered as a stopper for NB 6.7 .
http://hg.netbeans.org/jet-main/rev/bbe2fbcd4d16 I fixed rescanning of modified documents. Changes are now correctly detected in AtomicLockListener and only modified documents are rescanned. Also rescanning is not done on focus gained event. Nigel could you please try if it improved situation with DBObject? Thanks
Which build will http://hg.netbeans.org/jet-main/rev/bbe2fbcd4d16 be in ?
A notification will arrive to this issue when the fix appears in a daily build.
Integrated into 'main-golden', will be available in build *200906020201* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main-golden/rev/bbe2fbcd4d16 User: Vita Stejskal <vstejskal@netbeans.org> Log: #161367 - better detection of document modifications before triggering document reindexing
Build with the fix can be found : http://bertram.netbeans.org/hudson/job/jet-main/753/artifact/nbbuild/NetBeans-dev-jet-main-753-on-090601-full.zip or http://bertram.netbeans.org/hudson/job/jet-main/755/artifact/nbbuild/NetBeans-dev-jet-main-755-on-090602-full.zip
Created attachment 83094 [details] FIXED of performance regression
nleck, 40 vs. 3 seconds ... it looks fixed, right ? Thanks a lot!
That is SO MUCH faster :-) It's back to 6.5 performance for the "initial open of file" task. Thank you. The full project scan took (after deleting the .netbeans directory) took 5 min 10 seconds which is much more than the full compile of 23 seconds but massively better than the 6.7 beta of ~1.5 hours.
yes, the performance regression is fixed ( still would like some more but not for 6.7 )
The fix looks OK.
The fix was translated to release67 clone as http://hg.netbeans.org/release67/rev/602bbec4d872.
verified
Sorry need to re-open as the regression is there with one extra step as soon as you touch the file the "goto type" is blocked and this DID NOT HAPPEN IN 6.5 performance regression in 6.7rc2 see:- http://www.netbeans.org/nonav/issues/showattachment.cgi/83267/nb_scan.ogg http://www.netbeans.org/nonav/issues/showattachment.cgi/83269/nb_scan-6.5.ogg
Created attachment 83291 [details] scan is out of date
After making the performance regression movie in the previous comment, the file WASN'T saved. I actually even restarted my IDE and noticed all these red marks... The indexer had stored the information of a Java file that wasn't ever saved. Thus now out of date and will be until I change the Java file again.
Nigel, this seems to be a different problem. I have created a new issue #166714 and put you on cc there.
This issue was in verified state before.
The performance regression is the same issue as in http://www.netbeans.org/nonav/issues/showattachment.cgi/83010/nb_performance_regression.ogg but with one more step. The performance in 6.7 *HAS* gone backwards and it wasn't good to begin with. By creating a new bug the next thing you'll say is that'll be fix in another release but it is *THIS* release that is making the performance worst. What is the feature that has been added in this release that is causing all the NEW scanning issues ? Look at the movie http://www.netbeans.org/nonav/issues/showattachment.cgi/83007/nb_scan_embarsement-small.ogg the difference is so stark that most of the developers here have already move over now we are making it worst. The issue in http://www.netbeans.org/nonav/issues/showattachment.cgi/83291/scan_wrong.png is directly related to the fix done. I really think you should reopen this as the fix has problems. Any large project will have the same issues that we do.
THIS IS *NOT* VERIFIED. compare:- http://www.netbeans.org/nonav/issues/showattachment.cgi/83010/nb_performance_regression.ogg and :- http://www.netbeans.org/nonav/issues/showattachment.cgi/83267/nb_scan.ogg THIS IS NOT FIXED.
see how it was much faster in 6.5 :- http://www.netbeans.org/nonav/issues/showattachment.cgi/83269/nb_scan-6.5.ogg
I agree with dstrupl here, this issue was already fixed/verified. We are treating newly reported problem (issue 166714) as a stopper for NB 6.7 as well - its' a serious regression with target to be fixed in NB 6.7 FCS. So please keep those two problems in separate threads, thanks in advance.