This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
I have a fair number of large projects open in NetBeans 6 (9/20/2007 daily build). Whenever I open the IDE, it spends *many* minutes rescanning -- and even recompiling -- source directories from these projects. Note that the project sources, output jars, classes, etc, etc, have not changed in any way since the last time I used the IDE. I'm talking well in excess of 15 minutes here! Needless to say I can't wait over 15 minutes to use code navigation features every time after I start my IDE. That's not efficient use of my time and my management would be well within their rights to tell me to find another tool rather than waste this much time (especially since I have a laptop and thus can't always just leave the IDE running all day). Given that nothing has changed: (1) No "compiling" should be done whatsoever inn this case! All compilation should be accomplished and cached and no further compilation should be performed unless things actually change. I'm tired of waiting for what the IDE should be all done with over and over again! (2) Scanning should be very fast and minimal. (3) Scanning should not prevent code navigation and completion features from working -- only "compiling" should. Just because the IDE does not yet *know* that all code completion data is up-to-date shouldn't prevent me from using such data. Once it *knows* that, that's fine. (4) Go-to-Type, etc, no longer state that the code completion data is being updated and to please wait. They just say "searching..." That just gives the impression that the IDE has hung -- and is not helpful. NetBeans 5.5 also had a rescan delay on startup, but it was *much* shorter (approximately 2 minutes with even more projects open) and also got noticeably better on subsequent starts on the same day (or possibly machine boot -- I shutdown at least twice a day for commuting purposes). In both cases the code completion data is not even complete. Navigation fails due to various bugs, e.g. issue #57656, even after I work around this by creating scads of redundant libraries. From my usage thus far it appears NetBeans 6 has only gotten worse in this regard. Also see issue #116208. Finally NetBeans is rather slow about searching this information for go-to-type, etc, though this seems marginally better than NetBeans 5.5 -- *when* it works. Thus so far NetBeans 6's code completion data is produced very slowly, is incomplete / fails part of the time, and is slow (at least for collections of large inter-dependent projects). As a critical piece of the IDE, this needs fixing!
The repeated attempts by the IDE to recompile *may* be related to the fact that the code completion compilation encounters many errors due to the fact that it does not include project output directories and/or jars in the compilation classpath -- as NetBeans 5.5 implicitly did. I filed issue #116169 on this -- which was closed as WONTFIX. I can understand the reasoning for that, but the workaround for those who need this does not work in my case -- see issue #116208. Thus I'm stuck with compilation errors marked throughout my numerous projects. That's extraordinarily annoying, but having to wait >15 minutes on every startup for code completion scanning before I can navigation (I use go-to-type/declaration for all source file opening) is unworkable. Even if this is due to the compilation failures, this behavior is unworkable. Having to wait this long is unacceptable even if one's projects are not currently compiling from the IDE's perspective. [The projects build fine via their Ant scripts.]
Also note that a long way into the scanning/compiling phase I get a NullPointerException which I auto-reported and got a message that this appears to be a duplicate with issue #3001.
Hmmm.... Going to issue #3001 gives me something different than what I saw from the auto-report link. I guess at some point I may have to wait the 15 minutes and see what it says again... Overall, I'm done with NetBeans 6 in its current state. I've given it a good go, but the plethora of code completion issues renders it unusable for me. Also the one compelling feature for me in the short term -- the profiler -- also fails to read heaps of any substantive size on a 32-bit machine and fails to dynamically attach to Tomcat (the JVM crashes) anyway.
The compilation is supposed to happen only if necessary (eg. the timestamps in the caches are older that the timestamps of the source files/jars). The scanning is supposed to be very minimal - only timestamps check. Both works for me. I am afraid we will need help to find out why it does not work in your case. As first, could you please go to Tools/Options/Java Code/Tasklist and disable the Tasklist and try restarting the IDE? Also, could you please double check that you do not have source files modified in the future (probably no, but I have to ask :-) )? Thanks. The number 3001 you got was (I guess) meant into the exception reporter database, not into the issuezilla: http://statistics.netbeans.org/exceptions/detail.do?id=3001 The issue is already reported in the IZ, I will interlink the to reports together soon.
Scanning *might* be considered fast given the number of source files involved. I don't see the IDE saying "scanning" all that long -- the issue is primarily it saying "compiling". As requested I disabled the Tasklist, restarted, and scanning/compiling still took over 10 minutes. I'm not sure if the drop from 15 to 10 was due to partially prepopulated file caches and other "warm up" effects or due to disabling the TaskList, though. It kind of seems like it is trying to recompile anything it failed to compile previously due to issue #116208. I'm not entirely certain, though. I am rather certain all file dates are in the past. The source dates are controlled by ClearCase and my ClearCase view is configured to use the timestamps from ClearCase, i.e. when the sources were checked in. Oddly on this occasion I do not have the red error marks next to projects, packages, and files containing compilation errors (as the scan/compile pass sees them). When I open one of the files impacted by issue #116208, however, the editor still marks these lines as errors. This is very strange and inconsistent. Overall I would hope that there is logging built into the IDE that could give you a sense of the scope of my projects, their classpaths, their inter-dependencies, and the progress of the scanning/compilation pass algorithm -- that I could simply enable. The scope of my project data and its ownership by my employer makes it impractical to allow a straightforward reproduction of my exact situation by simply transferring my data. Enabling some logging to report what's going on in-situ would thus seem the ideal solution -- and hopefully the NetBeans team has such logging already in place :-)
Such logging would also be *really* helpful for code navigation. I have issues in both NB 5.5 and 6 with code navigation (e.g. go-to-declaration) not working between inter-related projects though go-to-type works just fine. Issue #57656 hits hard here, but even when I add what should be sufficient redundant libraries to work around the issue *some* inter-project code navigation fails, whereas other succeeds. This is a pretty frustrating aspect of NetBeans -- and as noted just gets worse with NB 6.
A stacktrace op two from the 15 minutes scan would be useful as well.
It's strange the compiling should be done only in case when the cache is older than sources. If I provide debug messages and attach modified version of the java module can you try to run the ide with it and send us the output?
BTW what OS do you use? Windows XP? With NTFS?
I can certainly restart my IDE with any debug-message ridden variation of the module provided and provide the resulting IDE log file. I'm not using NetBeans 6 for anything real until it is improved, so pretty much anything goes :-) I'm using Windows XP with NTFS.
Very funny. What about sending the stacktrace and trying to switch off the tasklist?
I already turned off the tasklist as noted in one of the comments. I can probably get around to a few stacktraces, but doing so in the midst of a >10 minute operation seems like potshots in the dark. I'm guessing adding stacktraces to the module's own logging would be a lot more productive. If you really want some random stacktraces in the interim, however, let me know and I'll do some jstack dumps.
Created attachment 49296 [details] jstack dumps taken at various points during scan and compile
I added some stack traces taken during the scan/compile phase.
I've integrated the logging code. Checking in src/org/netbeans/modules/java/source/usages/RepositoryUpdater.java; /cvs/java/source/src/org/netbeans/modules/java/source/usages/RepositoryUpdater.java,v <-- RepositoryUpdater.java new revision: 1.89; previous revision: 1.88 done I will attach the link to hudson where you can download the build with logging when it will be available. The logging is started by running NetBeans with following command line argument: -J-Dorg.netbeans.modules.java.source.usages.level=300 It would be good if you can start the IDE with empty userdir let the initial scan to finish, turn off the ide and start it again with the same user dir and wait to the end of initial scan. It's important to have these two logs, in the first one everything should be scanned, in the second one everything should be up to date. Thanks.
The hudson build: http://deadlock.netbeans.org/hudson/job/trunk/3477/
I'm on vacation (all week including now) and thus didn't get a chance to try the link immediately. Unfortunately the Hudson link fails (with a 404) at this point.
Seems that this build was already removed from the build server, but you can use any build later than 3477. http://deadlock.netbeans.org/hudson/job/trunk/
Okay, I'm back from vacation (sorry for the delay). I looked at http://deadlock.netbeans.org/hudson/job/trunk/ and at newer build #'s, but I'm unclear on exactly what I should be grabbing here. Is there a particular module I should be grabbing? Or...? If it is easier for you to have me grab the latest NB 6 daily build, install fresh, and go from there that's fine too. [I want to get this worked out soon as this is the last NB 6 issue I've filed where the NB folk are waiting on info from me.]
Yes you can take the latest NB 6 daily build. You need to run the ide with -J-Dorg.netbeans.modules.java.source.usages.level=300 as I've described above.
I tried today's daily build. Not reading terribly carefully, I started with an empty directory without the logging enabled. When this completed I attempted to restart with logging enabled -- but this produced a deadlock during startup which I'll attach. I'm assuming this deadlock will also if I delete everything from the cache directory and restart with logging enabled -- I'll try that next, but at any rate something produces a deadlock on startup when logging is enabled and the cache directory *is* populated.
Created attachment 50034 [details] jstack dump taken of deadlock
Deadlock among j2ee and logger, I will let you know when resolved.
Just to be clear, this remains a serious issue in the 10/2 daily build. It takes 10-15 minutes for the IDE to complete scanning and compiling actions -- with no changes to any files involved (even with only a few minutes since I last had the IDE running).
I've did some changes in logging which should resolve the deadlock j2ee vs logger, so the deadlock shouldn't appear. I've successfully tried it on the full IDE. You need either todays night build or build from hudson 3703 or newer (http://deadlock.netbeans.org/hudson/job/trunk/3703/artifact/nbbuild/dist/zip/). Can you try generate the logs as I described above. First log: start netbeans with -J-Dorg.netbeans.modules.java.source.usages.level=300 and empty user dir, open your projects, wait to the end of scanning, exit. Second log: start netbeans with -J-Dorg.netbeans.modules.java.source.usages.level=300 and same user dir as in first log, no external changes among the first and this start of IDE, wait to the end of scanning, exit. Thanks, Tomas
I tried build 200710071200 just now -- the deadlock has been resolved. I set '-J-Dorg.netbeans.modules.java.source.usages.level=300' in netbeans.conf and captured logs for the initial startup of the IDE (with no user directory as of yet) and then on a subsequent restart of the IDE (changing nothing in between). These are captured in a zip I'll attach in a moment. 'messages-1.txt' is the first startup log and 'messages-2.txt' is the second. Do note that my situation is also impacted by 2 other issues, 57656 and 116208. For purposes of these logs I did not apply the patch from the latter issue. Also, as noted in the notes for 57656, my free-form projects currently use <build-folder> in a manner that does not actually work (it was the closest to my situation that could be expressed in NB project.xml). If it would seem to make a difference, I could re-run this scenario when the fix for 57656 is available and with projects using <build-file> rather than <build-folder>.
Created attachment 50412 [details] zip archive of logs produced with special -J option
Also note that until I've updated to a build containing the fix for issue #57656 and updated my projects to take advantage of this fix I'll also have the redundant libraries around as already noted.
Thanks, I will look what happened there.
From the attached logs it seems that the caches are cleaned on every start of the IDE because the IDE thinks that the classpaths of your project has changed (equals on the classpath from last IDE start does not equal to the classpath from the second start) which is strange, I've expect the project's weren't modified during the restart. I've added more logging. Can you repeat the test as last time. I will attach the link to the first build containing the logging. Checking in org/netbeans/modules/java/source/usages/RepositoryUpdater.java; /cvs/java/source/src/org/netbeans/modules/java/source/usages/RepositoryUpdater.java,v <-- RepositoryUpdater.java new revision: 1.94; previous revision: 1.93 done
Is your latest additional logging in today's daily build? If so, I can re-test with it later today. I'm also intending to incorporate changes in my projects to account for the fix to issue #57656.
You have to wait to todays build (20071011????) which will have the changes from 10th Oct to try it, the night build wasn't created because someone had broken the code base, but hudson build http://deadlock.netbeans.org/hudson/job/trunk/3866/artifact/nbbuild/dist/zip/ should be fine.
I tried the 200710120000 daily build. The build loaded up my NB 5.5 projects fine, but when I updated these to use <build-file> to address issue #57656 and -J-DCacheClassPath.keepJars=true to address issue #116208 I got a null pointer exception which I auto-reported. I'll attach the first log here as well, but there's no point to a second log until the null pointer exception is fixed.
Created attachment 50881 [details] Null pointer exception log
I reverted my projects to use <build-folder> rather than <build-file> and thus not to take advantage of the fix to issue #57656 but removed the redundant libraries I'd been using as part of the workaround to this issue. I did, however, apply the -J switch from issue #116208. The NullPointerException seems directly related to my use of <build-file> as that went away. I am thus attaching log files. This archive of log files contains a numbered set of logs from subsequent restarts of NetBeans. The first from after the cache directory was cleared -- the others with no changes other than moving the old log file aside.
Created attachment 50914 [details] sequence of logs for 200710120000 build
I figured out that I was missing a piece from my project.xml when attempting to use the fix to issue #57656. Once I went back to using <build-file> rather than <build-folder> and did so properly this entire issue went away -- I could not produce any long, unnecessary scanning much less compiling. Thus it would seem this issue was more intertwined with #57656 than I realized. I believe this issue can be marked as resolved.
Thanks for your help. I've look into the generated logs and the class path differs in one cache element, seems to be caused by the build folder (the folder wasn't recognized by the freeform and java infrastructure created a new cache folder for it).