This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
We have numerous projects, some of which are very large. Some of these involve not only large jars, but also large loose codebase directories in their classpaths. With this reality (this is not going to change), NetBeans has in one respect continuously gotten worse despite its numerous improvements: the IDE is ever more insistent on scanning all of these directories before letting you do real work -- even in 6.1 where this is purported not to be the case. Also in cases where one rebuilding the output jar in a project upon which many, many other open projects depend the IDE will decide that their sources needs to be rescanned in their entirety in cases. This is *not* a recipe for large code set scalability. Perhaps we're alone in this boat -- or perhaps others in it have already jumped ship to Eclipse with its incremental compilation logic, but I wanted to raise the issue for community discussion. I did so on nbuser and did not seem to get any response, so I'm recording this as an issue. This really should be noted as a defect in many respects due to its severity, but I'm calling it an enhancement to avoid any arguments over issue type. In my book I think the IDE needs to provide more control over when code scanning is done at a bare minimum -- especially for those unfortunate enough to have loose codebase directories. It should also likely provide some control over the priority the background code scanning thread is given -- as this thread has been aggressive enough to render the remainder of the IDE barely usable. There's also got to be a better way than rescanning all downstream projects in their entirety when you rebuild a project upon which many others depend. Finally, I'd like to see the IDE live up to the promises in the 6.1 release notes and allow me to do "Go to Type", etc, with existing code completion information without waiting for all scanning to complete. When I start the IDE I know I have full (and up-to-date) code completion, etc, data in many cases yet the IDE refuses to "Go To Type", etc, until it has rescanned the world. On top of all this, I really think "Go to Type", etc, should still be faster, though I certainly recognize progress in this regard. That said, as a point of comparison I've used NetBeans' classfile module to analyze all our class files, store cross-references in a relational database (with appropriate indexing, of course), and then fronted this with a web app. The results of "Go to Type" type lookups (and many others) are *very* fast in all cases with this web app (even when running everything involved on the same laptop that I'm running NetBeans on). I guess I should integrate that with the IDE, but I'd think the IDE should do this sort of thing better than my somewhat ad hoc web app! [Also I limit the fidelity of my Find Usages to which class is using another and don't record/divulge which lines the usages occur upon.]
Too many things here, if I forgot to answer something please lat me know. 1) Probably one of the problem is that you use the option which prefers the output.jar to sources, which causes all the project's depending on the output.jar being rescanned when you rebuild the output.jar. 2) >I think the IDE needs to provide more control over when code scanning - can you provide more details? Thread priority is clear, but not sure if it will help, most OSes increase dynamic priority to threads doing syscalls (IO) which is case of this thread. But it can be tried. 3)> There's also got to be a better way than rescanning all downstream projects in their entirety when you rebuild a project upon which many others depend. This is caused by preferring output.jar. 4)>data in many cases yet the IDE refuses to "Go To Type", etc, until it has rescanned the world - sounds as a bug. Adding pflaska to cc, who is working on this feature. 5)> The Goto type in NB 6.1 needs to wait until the source root is indexed, there is quite big delay, which should be fixed in 6.5, again pflaska should know more. 6) The classfiles indexing is very fast. The rt.jar (~18000 classes) is scanned and stored in less than 10s, the problem is class files doesn't contain all information, for project you need to prefer sources which is much more time expensive, but I believe it can be faster than now. The most important, at least from my point of view, is to prevent non needed indexing.
I believe pflaska will work on this in NB 6.5.
Thanks for all the information. I don't know that we've used any "prefer jars to sources" option per se. We have used -J-DCacheClassPath.keepJars=true, which we absolutely must for now as per issue #134992 and issue #121853. See these issues as to why we're using this. We don't want to "prefer jars to source" overall -- we just want a means of dealing with classes resulting from non-Java sources (without having to split them into separate projects, which is a non-starter since they're interspersed in the same source trees as the Java sources). Overall indexing/scanning appears to be pretty fast except when: 1) Indexing a large loose codebase directory containing classes and other files. * There really ought to be a way to set a maximum reindexing frequency for such directories (e.g. once a day) and then to force reindexing on demand. 2) NetBeans decides a large source directory needs to be recompiled/scanned. * This can grind NetBeans to a halt and make even other apps hard to use during this interval. On what is really another front entirely "Go To Type", etc, are still *far* slower than they should be. I don't see any reason that the code index/scan can't produce data that's just as efficient to search as the relational database case I noted above. This should be blindingly fast.
Note that the need for -J-DCacheClassPath.keepJars=true is now gone in NetBeans 6.5 (at least for us) due to issue #134992 being resolved. Not using this option has resolved the issue where building one project causes all downstream projects to do a code completion/navigation database rescan/recompile. The other issues still remain, however. I often try to use code completion or navigation only to get "scanning in progress" and no action. Sometimes when this occurs there is no visible indication that such activity is in progress in the status bar. Moreover, scanning should never prevent code completion or navigation. As each source tree is being updated its previous code completion/navigation data should be used with a warning that the data may be stale. [Actually code completion/navigation should be prevented when there is *no* code completion/navigation data whatsoever, but that's rarely the case.]
I second everything Jesse mentioned with respect to code completion and go-to-type becoming useless too often while scanning is taking place. In addition, I want to add that "Find Usages" is also VERY slow (I'm running 08/17 daily build of 6.5). On our project (3825 Java source files), I just did a "Find Usages" on a static method and it took 33 seconds to find 26 usages. Granted, my machine is no speed deamon (single core Dell Latitude D810 laptop with 2GB memory), but it's not exactly a low-end box either. I'm running Ubuntu 8.10 w. JDK 1.6b10 (build 25).
The various forms of "Find Usages" are also nearly instantaneous (unless the result set gets really large) with my little relational database plus web app.
I cannot confirm that "Find usages" is fast. Sometimes it takes up to 30 seconds until the progress window is initialized and the progress bar "starts moving". This happens on a single core as well as on a dual core machine as well. Most of the time, cancelling the dialog and re-issuing a "Find usage" is then faster.
In my previous comment I meant that the "find usages" query types are nearly instantaneous against our (huge) data set when the searching is done via the static analysis web app backed by a relational database that I mentioned in my original description. "Find usages" is far from fast. Part of this, getting from which classes used something to which lines in those classes used something, is somewhat understandable to me -- as my web app only tracks usages to the class level and then uses heuristics to highlight the source lines that appear to be involved. That said, getting to the list of classes that use/subclass/whatever something should be far, far faster than it is. Still I can live with "Find Usages" being a bit slower as I use it less. Code completion and navigation must be always ready (no "scanning in progress" business) and always nearly instantaneous.
> I often try to use code completion or navigation only to get "scanning in progress" and no action. So "Go to Type" or code completion *triggers* the scanning by itself? I wonder if this is ever supposed to happen. Bug?
I don't know that "Go To Type" or code completion triggers the scanning. Rather I get a "Scanning in Progress" tooltip instead of a code completion pop-up for instance. All the while there is no scanning or compiling note or progress bar in the status bar -- so the scanning could have been going on for some time without the IDE telling me or it is possible that no scanning is actually occurring and the IDE is confused. Either way it is not very helpful :-)
I can confirm that "Go To Type", "Go To File" and the auto-completion features in NetBeans usually are downright unusable in large projects. I've seen reponse times up to a minute or two, with no visual feedback. Find and grep in a unix shell is unfortunately much faster. Eclipse and IntelliJ has near-instantaneous response on similar operations on the same project base, I don't see why NetBeans have to be that slow. I've tested this on NetBeans 6.5 and 7.0-dev.
I am being so bold as to change this to a defect. I don't know how I was ever talked into letting this be an "enhancement". Sure I argued for some things that were an enhancement, but overall this is about having usable behavior for code scanning and scan data usage, e.g. Go To Type. As per a recent nbuser post, I can sum up the inter-related issues as: 1. Scanning is fairly slow, whether this can be improved is unclear. 2. Scanning occurs when you do not want it. * You should be able to tell it "honestly your data for X is good enough for now, don't scan X now". * This is especially horrific if you have a large directory containing loose class files instead of a jar somewhere (an unfortunate situation, but truly necessary in some cases), but also applies to large jars. * Addressing this is much less critical if (4) is fully addressed. 3. Scanning sometimes occurs behind your back without clear indication in the GUI * Sometimes you'll try to do an operation and you'll get "scanning in progress, please wait", yet there's no indication of scanning in progress in the status bar, much less of what's being scanned. 4. While scanning is occurring the IDE is unnecessarily crippled * One should still be able to do any/all "Go To Type", etc, navigations, debugger, etc, without any ill-effects from scanning except: lack of new data being produced by the scan and a warning of this lack. If jar X is being scanned but you already had scan data for jar X, then the previous scan data for that jar should be used while jar X is being scanned -- with a warning, of course. There should be no "scanning in progress, please wait" or "Go To Type" dialogs missing results that were present in both the old and new jar. * Scan data for directories in the classpath should be stored as well and treated in the same fashion just described for jars -- no difference. 5. After scanning is actually done searches using the scanned data are often slower than expected. * I scan our class files with NetBeans' classfile module and pump loads of static analysis data into a relational database running on my laptop. The equivalent of Go To Type, Find Usages, etc, searches are generally noticeably faster with my database searches than they are for NetBeans. * Overall, the speed of Go To Type and other navigation using the scan database is critical. If this is near instantaneous, NetBeans seems miraculous to users and allows them greater productivity on large bodies of code. If this is not, then NetBeans is a bloated whale. It's that simple for users. This really should be a P1 bug -- it is killing NB usage, and is NB complaint #1 through 5 of those I work with, both those who use NB and those who refuse to. I'm setting it to P1 knowing it will get bumped back down, but maybe someone will notice it in the interim.
Not a p1, I think. I will try to investigate. Thanks for your help.
Thanks for giving this some attention. No, it really is probably not a P1, just a high P2 that someone talked me into letting be labeled an enhancement. Note that this isn't just a matter of slow "find usages", "go to type", and code completion ("go to type", Ctrl-B, and code completion being the critical use cases here for speed as they're things you do mid-thought-stream, whereas "find usages" is an activity to itself). The slow scanning combined with an inability to continue reliably using the old scan data during a scan that is updating said data are just as big of issues. There's also the matter of fragile scan data that some developers are seeing around here and on the list. This should be a separate bug but I don't have enough information to file a meaningful one as I've not reproduced this personally. I have watched it be reproduced in front of me, though. Essentially this involves having a set of projects that build fine and scan fine, showing no inappropriate error markers. The developer proceeds to compile certain Java sources (not just any Java source will cause the issue) and then the NB editor starts marking all sorts of things with "cannot find symbol" sorts of errors. This seems to occur more in 6.5 and has caused a good number of developers to revert to 6.1. Overall there are some big problems in this area that are causing NB users to switch away or strongly consider doing so -- and preventing Eclipse users from considering NB. Scan reliability and speed, both on scan update and scan data usage is right up there with basic editor speed as far as criticality to IDE usability. This area needs attention!
You wrote that you are working on some very large projects. Is it possible to download some of them, to test performance? Can you attach information how big they are (how many source roots, jars, source files, megabytes...). What type of NetBeans project you are using? Do you have some numbers, like start NetBeans with foo project opened (+info about size of project) + fresh userdir => scanning takes 5 minutes? Opening the same project with Eclipse => 1 minute... We will try to do some comparison anyway, but I will appreciate any data.
I open the source for the Liferay portal every now and then, but I make sure it's closed before I close NetBeans. Otherwise the next startup wil increase my coffee consumption too much :)
Is there some sort of module you can give me that will just collect size metric information you're interested in from my open projects? We're using free-form web projects with a complex Ant framework being invoked for the builds. The source is all quite proprietary and thus I'd have to go through various legalities to hand out a copy. I can try to generate some numbers for NB -- again a NB module (or log level configuration) that produces such metrics would be nice to ensure consistency and allow many developers to easily give you numbers. I don't actually use Eclipse myself at all, so I can't give Eclipse numbers. I don't know that Eclipse is faster except insofar as NB users move to it for this reason and Eclipse users who might otherwise dislike Eclipse won't consider NB for this reason (amongst others). Overall I'm not dead set of scanning being dramatically faster. If NB is still perfectly usable while it is scanning and navigation / code completion using the scan data is lightning fast, that would be a big step forward. The fact that scanning can sometimes hog too much of the CPU for the IDE to be usable while scanning is a big problem (and a separately filed issue). The fact you can't do certain operations at all while scanning but instead get a "please wait" is unacceptable [unless scanning is all but instantaneous irrespective of project size/number]. The fact that "go to type", etc, forget the existence of types that it new about from previous scans while rescanning a jar is also unacceptable [even *if* scanning is all but instantaneous]. Finally, the long delay in getting results from "Go to Type" or Ctrl-B in many cases [sometimes it is quite fast] is unacceptable. If all of those items are fixed, then the speed of scanning itself is much, much less critical.
From memory I remember that if you have 4-5 projects open in Netbeans and you have over a total of 100 classes then Netbeans will begin crawling. Please note I have no retested this recently but it used to be the case a few months ago. I was surprised by how quickly things deteriorated. 100 classes is not large by any stretch of the imagination. I don't want to point you guys in too many different directions. I think the best first step would be to measure how our performance degrades as the number of classes increases. I suspect (but we need to prove this) that performance degrades exponentially.
Creating some module that can collect performance data is great idea! Thanks. I will ask here if we have something like this.
I see the need for at least 2 different areas of metrics: 1) Something to report on the number and size of projects loaded * For each project list how many source files, how many K of source files, how many jars referenced, how many MB of jars referenced, etc. * List totals for all open projects * Developers would like the ability to generate such a report anyway; this would be a nice feature overall 2) Something to report on time consumed for various IDE tasks * This may just be a set of timing loggers that you can tell users to enable when they complain. Thus this may not be so much one new module as enhancements to the logging in many modules. * There may be some general points where one could insert timing from which lots of things might benefit quickly, e.g. the NetBeans background task processor could time each task it processes. Apart from these items, however, I have been pushing for the NetBeans CPU profiler (and VisualVM) to offer a simple sampling mode which one could use to determine which areas of a large application to re-test with NetBeans CPU profiling instrumentation. Apart from the general utility of such a feature, it would be great if NB offered a simple "profile me [start/stop]" action which would capture a sampling profile during some operation. Users could then be directed to do this for any operation that isn't performing as well as they'd like (or that seems to hang as long as it doesn't hang the whole UI) and attach the result to an issue. This would seem to streamline the process of addressing performance issues in NB overall.
Please, update TM.
One of our projects has over 7900 source files and has 56 libraries in the classpath. Many of these source files are quite large often with numerous inner classes. Many of the classes are quite interdependent. Many of the libraries are quite large. A good number of the libraries are outputs of other projects. There are then dozens of projects depending on this project (and other projects besides). These projects are also often fairly large, but not as large, say 1100 source files in cases. Projects generally have between 1 and 3 source roots. [All project source file counts were provided by SQE, so hopefully it is reliable in that regard.]
The primary output jar from the 7900 source file project is around 27MB (compressed) by the way. That includes a lot of .ser files as well, but class files are definitely the bulk of the jar's content.
As for the "Profile Me" feature, it's already in current dev builds (will be in M2). See issue 153221, and http://wiki.netbeans.org/FitnessViaPartnership. As for the metrics, would be nice, definitely to have some plugin that would collect various useful data. It's not that clear, though, what the useful data should be. The size of the project, number of classes, etc, definitely matter, but is not the main problem. See gtzabari having problems with only 100 classes. OTOH we (NetBeans developers) work on quite large project, with tons of projects, zillions classes, many dependencies, but don't see significant problems - neither with scanning nor with code completion or Go to Type navigation. We also do some measurements on a reasonably large project (20MB of sources) where we measure startup of the IDE, time to open the project, time to complete the scan, time to use code completion, time to build the project, consumed memory, etc. In all these areas NetBeans is comparable or even better than Eclipse. So all in all, it's not generally true that NetBeans is slow on big projects. There are other (bigger) problems than size. The various scanning and related problems seem to be always tied to the particular project, its configuration, custom tasks running as part of build, etc. There's usually some problem causing the IDE to scan sources too often, or scan parts of classpath that would not need to be scanned. Whether it is misconfigured project, or too sensitive/fragile mechanism of detecting and updating changes on classpath, I don't know (I'm not expert on the java infrastructure). Anyway, this means that we are able to do very little with general statements like that scanning is slow etc. Ideally we need the project itself to test it and profile. So if you can, please send us your projects, with description what to try. There are also other ways: One is the new "Profile Me" feature, another one is to use a special startup switch to create scanning log: -J-Dorg.netbeans.modules.java.source.usages.RepositoryUpdater.level=FINEST The log will contain what scanning happened and why. Cane use FINE level for less details. Since the log can be pretty huge, it's necessary to find the case where you see scanning which should not happen, and run the IDE with the switch only for reproducing this one problem. From the log we might be able to find out what is wrong. BTW we also plan to create an experimental module which would turn off some of automatic classpath updates/scanning and give the user the option to invoke the refresh/rescan explicitly. See issue 158177. There are concerns this could lead to various errors and inconsistencies, but is probably worth trying if it can solve some problems.
Glad to hear it about the "profile me" progress. I've been begging for almost exactly this for some time. Will this be made part of 6.7? Please!!! This is huge to both NB and VisualVM. As for scan speed, I can try the logging, but again I'm not really too concerned about scan speed by itself. I realize they're going to take *some* time in my case however hard you try. I'm concerned by loss of functionality/usability during these scans and speed of navigation features based on the scan data.
Re "Profile Me" - it's already in, you can download a daily build and try it. Re Logging - the log should show why scanning happens when it is not expected. It can also show that bigger amount of files is scanned than needed. I.e. it may reveal bugs in the scanning infrastructure, or problems with project setup. As for IDE usable even during scanning (i.e. use the previous version of data until scan is finished) - that was discussed several times, but always found too difficult to implement. See also issue 133943. For 6.7 we hope for an improvement of Go to type navigation during scanning, at least (new indexing API should allow that).
tpavek is right. We need a *specific* testcase (complete with the source-codes if possible) to work against. Otherwise we'll spin our wheels trying to fix a theoretical problem without being able to verify whether our work made a difference or not.
Most of us work for commercial companies and aren't allowed to send source code. This might sound silly, but throughout the conversation, NB devs always indicate that they have no problem with a pretty big project: Netbeans! Can the NB developers put together a nice self-contained zip file with the entire NB project & src and let people who have trouble with their project try it out to see if they also have problems with the NB project. If so, we learned something. (I realize that the NB src is readily available. If it's already *very* easy and self-contained, then maybe just point to it) and we can try it. I just made the above suggestion on the off-chance that building NB takes more than simply downloading and opening/building in the IDE).
Create a library project with perhaps 100 classes in a jar file. Change the build.xml for that library so that "-post-jar" copies the jar file into a "public directory" that is the location that the library manager has in its configuration for the library. Next, create 5-10 projects with approximately 500 classes in each, have each of those projects reference this library. Make sure all of these projects are open in the IDE along with the library. Finally, open some .java files in one or more of those 5-10 projects. Make a change in the library and rebuild it. As it's building start editing and trying to do work on those java files. Build them, try and run things and do normal development tasks. At some point, the IDE will start recompiling those 5-10 projects because the library timestamp changed. During that time, you will not be able to work.
As for NB sources, can be downloaded as one zip file e.g. from here: http://www.netbeans.org/downloads/start.html?filename=zip/netbeans-6.5-200811100001-ml-src.zip&version=6.5 To test it, you can open whatever project in the IDE - e.g. "form". Note that nobody opens all projects, but having just one project opened brings in all the projects it depends on (transitively), which is typically quite big number. You can let the IDE to open all the dependent projects explicitly. We usually don't build entire NetBeans in NetBeans, possibly just the module being worked on (plus dependencies). When opening the project, you'll see quite long initial scan (much shorter next time when IDE is restarted). Then you can try to jump to some class to test Go to Type speed (try e.g. FileObject or RepositoryUpdater), try to modify some file to see code completion speed, do some refactoring, etc. This way you can get some idea of speed at which the IDE can work on a big set of projects...
I have a particular configuration which I illustrated the details thereof. It creates problems for the implementation of scanning and error badging related to classpath detected changes. Many others have lamented on the mailing list the same types of problems which they experience, and based on what I can tell, they have similar project numbers and layouts and are experiencing the same problem. I don't know how else to explain what I see... I know that the NB team says you all can develop the IDE just fine. I also know that there are specific optimizations in the IDE's behavior regarding netbeans platform development. I am not developing a netbeans platform app and I am not developing the netbeans IDE (a form of the platform). I and many others it seems, need help with fixing the problems that occur in our environments.
> I and many others it seems, need help with fixing the problems that occur in our environments. That's exactly why we ask for the details. We need to know how your environments look like. Thanks Gregg for describing yours, we should take a look at it. Let me summarize your case, someone from NB java team can comment more on that. By copying a built JAR file to a "public directory" which is used by other projects you are changing the classpath of these projects. When this happens, the IDE can see the entire JAR file changing, but has no idea what actually has changed inside the JAR file. It does not compare it to previous state to determine some "incremental" changes. At this point it needs to rescan the projects that depend on the JAR file because there could have been potentially any change affecting them (some errors in source code may appear, some existing may be resolved). In reality, there is usually no significant change (affecting public API) and such huge scanning is not needed. It's even bigger problem if you need to rebuild your library JAR often... The system is designed the way it expects libraries/JARs on classpath are not changing, or only exceptionally. Definitely not as part of compiling/building. If you rebuild your libraries often, then you should have your dependency defined on the project level - i.e. you should depend on the library project output which is associated with sources. Having the association to project sources for a JAR file, the IDE knows that the change of the JAR file (when built) is not significant - because the IDE is checking against the library sources and processing incremental changes if you edit them. This may still cause scanning, but in much less cases. This is a simple view on the problem. Someone from java team please correct me if I misrepresented something. Now the question is whether you can alter your project setup (or at least try - to see if it can really help), or if NetBeans can behave differently even in the case it has not the sources associated. I only know about issue 158177, which could help solving this, but it is only an experiment.
With regard to "If you rebuild your libraries often, then you should have your dependency defined on the project level". When you build using non-NetBeans ant scripts (i.e. freeform project) and a non-NetBeans dependency management, for example using Ivy, then you want to avoid project-level dependencies in NetBeans in the general case, as the results of depending on local project sources won't be consistent with your build infrastructure. It would be acceptable while you work on two dependent projects simultaneously, with dependent changes. but not otherwise, because the reason for the build infrastructure is for the builds to not be dependent on the IDE or local settings. In this scenario, which for us is the default scenario, what would at least be needed then is the ability to quickly switch between project-level dependencies and classpath dependencies in NetBeans, without the world being rescanned. For some reason all of this doesn't seem to be a problem in Eclipse, and our impression is that this is a killer advantage of Eclipse over NetBeans in "enterprisy" environment, no matter how much better NetBeans maybe in other areas.
"Depending on local project sources" should be only for the IDE, and possibly need not affect the building infrastructure. Only the IDE would resolve against the sources, building would work with JAR files. So in other words it should be possible to specify the sources for any JAR/library on classpath. I'm not sure right now if that is possible in such general way (for free form projects configuration; let's say it should be there). But I don't see how else the IDE could determine that it should ignore a changed JAR on classpath. If Eclipse does that without rescanning, does it catch and show errors correctly in depending projects if you e.g. delete a public method in the library and rebuild it?
Eclipse does rescan, but it doesn't block anything during that scanning, and it also doesn't "forget" the individual results of a previous scan until they are dynamically replaced by results of the new scan. I suppose the update of these informations are done per-class or something. As for "only the IDE would resolve against the sources", that's exactly what would cause inconsistencies: Unless these local sources are exactly the same version as the ones the jar you build against was built from, the IDE can show errors from its internal compilation where the actual build won't produce errors, or vice versa, and code completion and refactorings will not use the correct information.
OK, so to sum up the main use case here: There is a "library" project which is built quite often, but separately from other projects that use it. The separation is intentional and the dependency should only be on the produced JAR file (its copy in fact). Relating the JAR file to the library project sources is not wanted, because as explained, the JAR file is held separately and may not correspond to the sources. (Direct dependency between projects would ensure the consistency and eliminate the scanning problem, but then the projects would not be separated as requested.) Now the problem with this setup is that rebuilding the library and then switching to some of the projects that use it cause "entire world" rescanning which makes the IDE unusable. The rescanning is needed, but Eclipse is supposed to handle this situation better because it's doing it "on background" while the IDE is still responsive and usable, navigation works, nothing is blocked etc. Seems its able to use the index and data even during the scan, updating it continuously. From this viewpoint, NetBeans does not scan needlessly but should try to fix the scanning itself to be more user friendly (as the issue 133943 is about). It is not obvious the problem should be solved by making the IDE scan less (not react on some changes, etc, as issue 158177 is basically about). So matthies, greggwon - is this basically your standpoint? I must admit I thought the second one was the main problem and also the difference from Eclipse. Still it can be both...
I think that there is a very simple issue here. The NB team wants to help developers out by pointing at problems. The issue is that the timeliness of the reports is not happening and the NB team thinks its more important to guarantee the integrity of the "view of the world" than to guarantee that the developer can work on the problems which they can see and do know about. Because of the speed of disk drives, the size of projects, the I/O issues through systems in general, there is no way, at all, that all the work that Netbeans tries to do for developers, to update the "view of the world", can be done in a timely fashion, and make that update appear synchronously with editing changes and other asynchronous file system updates. We really have to decide that this is true! Instead, we need to move to the point of agreeing that it's really okay for netbeans to show (only) exactly what it knows at any particular moment, but that it's knowledge can change over time as it works behind the scenes to update the "view of the world" towards the current reality (a moving target at best). More importantly, we also need to confine the things that netbeans tries to learn about as its "view of the world" to only be the things that are truely important for the current moment in time. My thoughts are that for any outside (not from the editor) indication of a change, the IDE should only check the current file in the editor, then the other files in the editor related in order by
I think that there is a very simple issue here. The NB team wants to help developers out by pointing at problems. The issue is that the timeliness of the reports is not happening and the NB team thinks its more important to guarantee the integrity of the "view of the world" than to guarantee that the developer can work on the problems which they can see and do know about. Because of the speed of disk drives, the size of projects, the I/O issues through systems in general, there is no way, at all, that all the work that Netbeans tries to do for developers, to update the "view of the world", can be done in a timely fashion, and make that update appear synchronously with editing changes and other asynchronous file system updates. We really have to decide that this is true! Instead, we need to move to the point of agreeing that it's really okay for netbeans to show (only) exactly what it knows at any particular moment, but that it's knowledge can change over time as it works behind the scenes to update the "view of the world" towards the current reality (a moving target at best). More importantly, we also need to confine the things that netbeans tries to learn about as its "view of the world" to only be the things that are truely important for the current moment in time. My thoughts are that for any outside (not from the editor) indication of a change, the IDE should only check the current file in the editor, then the other files in the editor related in order by 1. dependended on by the currently edited file 2. In the package of the currently edited file 3. In the package of any other opened file with depended on being first 4. In the current project 5. In the projects of other dependent and opened in the editor files. If the keyboard goes idle for more than 30 seconds, the IDE could then start assessing the effects on other projects. It could initially just mark dependent projects (for example if a library changed) as "dirty" and use the project dependency information to note which files might be dependent on that library and mark those in particular as dirty. But it really should not do anything behind the scenes until the user is not actively using the IDE. As the user changes the currently edited file, the 1-5 above should be the order of processing to work on updating the "view of the world". If I break something in a library API, I do it on purpose and I know what to change to fix everything and I don't need the IDE telling me I can't work on what I need to fix until it makes sure that another, unrelated project has no problems. That's what I currently see happening when I update a library. I can't edit the current project because the IDE is off compiling another unrelated project.
The system is designed the way it expects libraries/JARs on classpath are not changing, or only exceptionally. Definitely not as part of compiling/building. If you rebuild your libraries often, then you should have your dependency defined on the project level - i.e. you should depend on the library project output which is associated with sources. Having the association to project sources for a JAR file, the IDE knows that the change of the JAR file (when built) is not significant - because the IDE is checking against the library sources and processing incremental changes if you edit them. This may still cause scanning, but in much less cases. ------------- Are we then just missing a piece of the chain that the IDE doesn't know about? The source files in my library project are known. The output file is known, but the place that I copy it to, the library manager designated path, is not part of the puzzle. Should it be? Is there a different way that a freeform project can make this part of the chain visible to the IDE?
I agree with greggwon on the point that the scanning could be more demand-driven. For example, if member code completion is invoked on a variable whose class is known to have out-of-date scan data, then specifically this class could be scanned as needed to update the member list for code completion, rather than waiting that "the entire world" has been scanned. The code completion could be dynamically updated, so that the user isn't blocked even during this demand-driven scan. Similar "targeted" scanning would be helpful for Find Usages and refactorings. For such operations the user could also be given the option to have the operation performed immediately based on the current, partially out-of-date scan data, because the user may know that the yet-to-be-scanned changes don't affect the operation, or is willing to take the risk. Unlike greggwon, I think that "the entire world" should *eventually* be fully scanned, to correctly update the error badges etc. But this could be done with lower priority and shouldn't disrupt or prevent user operations. I suspect that there's also room for improvement in scanning and indexing speed even intra-project, though this is probably a separate issue. Rescanning sometimes appears to be significantly slower than equivalent recompiling. Or for example when adding a new method to a widely used interface in a largish project, and then immediately removing the method again using Safely Delete, the Safely Delete and the Refreshing Workspace that ensues afterwards takes ages (okay, minutes). It also appears that use of generics and circular dependencies (e.g. visitor pattern) affects scanning times disproportionately high. Another thing is that CPU usage rarely seems to be the bottleneck for scanning, so maybe the performance of I/O and index storage layout could be improved. I'm just speculating here though.
"Eclipse does rescan, but it doesn't block anything during that scanning, and it also doesn't "forget" the individual results of a previous scan until they are dynamically replaced by results of the new scan. I suppose the update of these informations are done per-class or something." That says it all in my book. Scans on really big stuff are never going to be sufficiently fast to avoid causing the user lost productivity unless (1) nothing is blocked during scanning and (2) previous scan results are replaced at a fine-grained level rather than being forgotten at a macroscopic level thus leaving you with huge gaps in your data during rescans.
The indexing api (part of the parsing api) allows access to the class index data in time of scanning, the java features may use it. The 6.7 scan is much slower than it was in the 6.5. One of the reason may be creation of the FileObjects where Files were used before. I will try to return the possibility to use the Files, this requires the java indexer not to touch the FileObjects, I will create JavaFileManager based on the Indexables and Dusan should fix the indexer.
I suspect that most people can only do one thing at a time, with the exception of batch refactoring, so only a small percentage of files change between builds and it is wasteful to scan entire "finished" code bases (like libraries) on every timestamp change. Instead, we should store a priority based change- map that starts all classes with (priority) 1 change, and every time a change is found increment the priority. With logic "scan only files that have changed the most number of times" we could scan 'suspect' classes in real time and leave the low change rate classes for idle/background scanning. If this information was stored between sessions, it could also greatly reduce startup time. For example, we maintain a common "toolkit" class library that is shared between three products. Most of the classes do not change, but there are a few enums that are regularly expanded. Instead of the whole 3mb jar, I would much rather see only these 2 files scanned in real time when the project is built. Not only do most files not change, but I would go so far as to say that in most cases, the majority of classes in a library are not used. By joining a map of changes with a map of usages, it may be possible to heuristically scan a small number, say %5, of classes initially without compromising utility or the "NetBeans magic" that makes it such a good IDE. As much of a nightmare as it would be (I surely wouldn't want to do it...) a full out relational database with the ability to hold interfaces, method definitions, and javadoc may be the only 'real' solution.
Doing a relational database containing the initial pieces, e.g. answering what classes are defined and where, what methods and fields they define, which classes use them, etc, is not that hard. I can say this as I've done it. Now I currently do this in bulk using batch statements for many megabytes of jars in one swoop, but it is easy enough to see how one could do a transaction for smaller updates, e.g. of the data related to a given source file, thus allowing continuous access to the most accurate information possible. With appropriate indexing on the resulting tables the queries for things like "go to type" are nearly instantaneous as well. Of course I did this on work time, so it's not open source...
Created attachment 79469 [details] Thread dump when 'Scanning Projects' happens every 10 seconds
I do not think that relation database will help. Queries to Lucene index is fast enough without any doubt. Scanning time is composed from several things and they together takes a log time: i) opening indexes, ii) computing classpath, iii) going through all files and checking timestamps. iv) updating changed files. When using "Go To Type" feature, again, i) classpath computation, ii) index search. Especially for API support type projects, classpath computation (scope for search) is not fast enough.
I have 2 big issues with the existing speed. 1) Indexing/scanning takes time for large sets of code (unsurprising) *and* one can't use large portions of the IDE during this indexing. If the latter were solved, then I'd be okay with 6.5's indexing speed. One possibility here would be a separate relational database transaction for each source file being scanned, but there are certainly other approaches. 2) Go To Type (and other critical quick navigation actions) can be maddeningly slow at times. I've not seen the equivalent take so long with a relational database. Perhaps this is all due to computing classpaths. In any case this needs fixing.
It might be my imagination, but 200904070200 seems much better about the indexing thing. I'm not sure if any code has changed here or not, but it seems like it has. Now, a single class will disappear without warning from the IDE's 'world'. That is it looked OK just minutes before, but suddenly one will get errors about a class not existing even though it compiles and runs fine. It's not always the same class, actually I've never seen it do the same one twice. It is also not noticeable until the IDE has been running for a while, which makes me think of issue 162176 and the slow memory leak. Perhaps the asynchronous code is all and well when fresh, but there it gets bogged down somewhere. That's for yall to decide :)
Attaching a test case. This makes it very easy to see what this issue is talking about. Steps to reproduce: 1) Decompress attached file. It is in .zip format 2) Open the project in NetBeans 6.7 (It was created using build 200904270201) 3) Open the "BreakNBManyFiles.java" under the Default Package in the editor 4) Press <shift> + F6 to run the file. This will create a package under auto.gen.code in "src" and "test" Problems you will encounter: A) Open up one of the packages, and try to select all files. This takes several seconds on a fast computer. B) Deleting all files takes a minimum of 30 seconds, if it does not freeze the IDE C) Running the test file more than once will trigger the "Scanning Files" loop-of-death (Each time generates a new set of files) D) The JVM maxes out on physical memory, and starts swapping (fresh user directory on Java SE install with 3.0gb virtual after running the test project)
Created attachment 81114 [details] See BreakNBManyFiles.java in the Default Package
This issue was discussed during last VOC meeting #8 [1] and it was strongly suggested to fix this after 6.7. [1] http://qa.netbeans.org/processes/voc/chats/transcript_090424.html
So performance and scalability of the most critical core functionality is always "next release"? Sadly this sounds a lot like other software products (that shall remain nameless), but that's no excuse.
Try the "Scan On Demand" module from Development Update Centre. That should reduce the scanning. Bear in mind that usage of module is at your own risk! Any issue reported when module is on will be considered as a bug against experimental module.
The project scan *MUST* be improved in 6.7 otherwise you may as well just install Eclipse for remaining the NB developers here, they have almost given up on NB and this will just be the death nail. I really don't think you guys understand how badly NB is coming up in comparison to other IDEs in this area.
Along these lines, while I can't reasonably share/upload my company's projects or source code as a test case, I will be at JavaOne and could certainly let some NetBeans folk poke around in these via NetBeans on my laptop. I'm guessing that there may well be others who are greatly pained by this issue who are similarly equipped. It might be good for the NetBeans guys to organize a "show me your irreproducible NetBeans problems" meet up or some such. [No, I won't be attending CommunityOne or at least most of it -- as I've attended it every year thus far and haven't found it worth the extra day of lodging and losing part of my weekend.]
I work for a big sized company and I have used Netbeans since the first version. We are more than 30 developers working in the same J2EE projects. We are willing to use Netbeans but this issue and issue #42683 are preventing us to migrate from JBuilder. Comparing with other IDEs Netbeans lacks the stability and scalability for enterprise sized projects, and in every new version this issues are postponed. Aren't this more important than new languages or eye-candy? Scanning is hell. If you open the IDEs with ten or more projects it becames very slowly an unstable, freezing sometimes. Opening a single java or jsp file may take 5 seconds to appear. If the are many classes in the project badging of compiling errors are almost always wrong... If you compile the full project with dependecies takes forever, and after that the badges remain wrong. IMHO Netbeans is sadly becoming a tool for educational purposes, but not for the real work.
Discussion here is getting long, so I would like to summarize it. THIS BUG IS NOT ABOUT SLOW NETBEANS, BECAUSE NETBEANS ARE FAST!!! We have several comparison test on different large projects, and we KNOW that NetBeans are comparable to other IDEs on these projects. We are developing NetBeans in NetBeans, so we REALLY KNOW that ... NetBeans are fast!!! (NetBeans source base is really big) So, this issue is about one concrete usecase: User has custom Ant build script (free form project) that generates something to the Sources. (Free form project sources are defined in free form project properties - "Java Sources" and "Java Sources Classpath" lists.) Your Ant project should not modify files / Jars that are listed there!!! If you broke that rule - you have troubles. NetBeans listens on that folders / Jars, and starts scanning. The fact that NetBeans are listening on source path changes is design decision (not a bug) and we are not going to change it. It is very useful feature. So, what can we do with it? Stop listening on source path during build + add some manual rescan? We are investigating, but there is no simple solution. And what can you do with it? You can try to modify your build script to separate source folders / Jars from Output folders / Jars. And you can modify your "Java Sources" and "Java Sources Classpath" lists to not contain any files generated / modified by your build script. It should help to fix this issue for you. And if you see some other issue why your NetBeans are slow, FILE ANOTHER BUG, PLEASE! This issue is about "Source path modification initiated from build script starts scanning." only.
jjancura: I don't think it is very useful to say that the problem doesn't exist after all the comments and complaints on the issue. It's great that the NetBeans team uses NetBeans for development and doesn't see any issues w/ performance. However, when someone was designing / building this auto-scanning feature they envisioned a particular usage model for NetBeans, and it appears (rightly or wrongly) that how people use the IDE falls outside that envisioned usage model, which causes them trouble (resulting in a slow IDE). The suggestions that you gave are useful in analyzing the problems and hopefully folks will go back and analyze their projects for the issues that you describe. There is a problem with pointing the finger at the users and saying "You're wrong": if the users set up their project wrong, there is a chance that the IDE didn't necessarily support them in creating a "correct" project setup. The next steps that you identified sound reasonable - it would be nice if there is some resolution to this SOON, you can see a lot of loyal / long time NetBeans users upset over this. One potential issue in what you describe as the source of the problem is that NetBeans can relaunch the scanning of something changes in the "Java Sources Classpath" setup : it seems more or less normal for this path to contain class directories and jars. It seems just as normal that an Ant build will modify some of those during the build. In a different issue, jglick pointed out that if a classes directory or a jar is associated w/ sources, the IDE wouldn't rescan even if the artifact changed. Thus, an additional suggestion to review the project structures is : are all of your freeform projects correctly configured to identify the "output" of their source (so that NB can associated classes and jar w/ source and not scan when the clasess or jars change during the build).
jjancura: I don't agree with your appraisement and narrowing of the issue. The scanning/refreshing has a design and/or implementation flaw which is not just related to custom Ant tasks. The scanning is often a problem too in projects or project groups without custom Ant tasks. Excessive scanning and/or wrong error badges happens in the context of: - in-between saving of files with compilation errors - larger refactorings - CVS updates - building of dependee or sub-projects, as common in J2EE project structures - concurrent executions/combinations of the above, which NB doesn't (and shouldn't) prevent - projects with heavy use of generics and/or recursive dependencies (source model appears to be buggy) - incomplete or inconsistent source model data - ...others may add more The problem is not necessarily the rescanning or "refreshing workspace" itself, but that it - is sometimes modal in that it is blocking certain user actions - consumes too much CPU - does a lot of I/O (much more than e.g. javac for the same set of source files) - assumes a bug-free source and dependency model - assumes no concurrent actions (or so it seems) I believe there's both a performance problem in that updates of the source model appear to consume more resources than they should (and more than they appear to in other IDEs), and, as far as I can gather, a conceptual problem in that the design seems to be to "stop the world" until the source model is (believed to be) complete and consistent again, as opposed to being able to work with a possibly incomplete and inconsistent, incrementally updated model (which appears to be how other IDEs work). So, source path modifications caused by Ant execution may well be an individual issue in itself, but is not the larger issue this Issuezilla issue was about, IMO.
NB is *NOT* fast!!. The "project scan" issues are an embarrassment. Look at the message boards. As most devolopers use other IDEs when working on large existing projects we can't just change the project structure for all developers to suite NB when the other IDEs have no trouble with the existing project structure. I have never seen Eclispe block the developer while trying to navigate to a class. With NB it seems to block far too often.
@jjancura When you say "And if you see some other issue why your NetBeans are slow, FILE ANOTHER BUG, PLEASE!" The summery of this issue has been changed. According to my records on 30 APR 2009 (as of my test case), the summery was [Issue 134990] Java support is slow on big projects Then, today (5 MAY 2009), I received this by email: User jjancura changed the following: What |Old value |New value ================================================================================ Summary|Java support is slow on bi|Source path modification i |g projects (code completio|nitiated from build script |n, find usages) | starts scanning. -------------------------------------------------------------------------------- Instead of changing issues so that they no longer exist, then denying that they are issues, how about putting effort into fixing them? Perhaps on NB source, NB works fine. All of the NBM that I have seen have custom ANT scripts, not the scripts that NB generates when a user creates a project. Have you considered that of the 36 registered users who voted for this issue, that the majority of them do not use custom ANT scripts? I will file another issue; But coverups should have died with the CCCP.. Grow up.
Fist, I must say that I do not see any big disagreement here. All we are trying to make NetBeans better, faster. We should only clarify how to do it effectively. Messages like "NetBeans are slow", "NetBeans are ugly", "NetBeans are buggy" are not fixable issues. See http://wiki.netbeans.org/IssueReportingGuidelines. There is description how to write useful issue (see mainly "One problem - one issue", "Provide detailed description", and "Provide steps to reproduce" sections) "NetBeans are slow" is great point to discussion. We definitely need such great source of feedback from community. But it should be discussed on mailing lists. Visibility of one issue is definitely lower than a number of people watching nbusers mailing list. We need more users to discuss it, more feedback! Issues like "Source path modification initiated from build script starts scanning." can be one of results of such "NetBeans are slow" discussion. You probably understand that useful bug report have to be concrete, reproducible, measurable... I have identified that "Source path modification initiated from build script starts scanning." is root cause of many problems described here. That is why this issue is now about this usecase only. But it does not mean that we will throw rest or reproducible usecases mentioned in this issue out of window. We will check all concrete user reports from this issue and file separate issues for them. Please, If you would like to discuss NetBeans performance, go to nbusers (more users = more feedback). You can tag your message with "[PERFORMANCE]" to be more visible. If you would like to discuss NetBeans architecture go to nbdev mailing list. Themes like "Manual or automatic scanning", "source models", "incrementaly updating scanning" are great points for discussion there. I am not able to attach all developers, architects and other people responsible for NetBeans architecture to this single issue, but all of them are definitely on nbdev! We can find some solutions there, and file concrete issues / RFEs than. And, please, if you have some concrete, reproducible usecases that are not covered by this "Source path modification initiated from build script starts scanning." usecase - file separate issues for them, please! Thanks
This is ridiculous. By changing the summary, the issue no longer applies to me. But look! The problem still exists. You may as well have changed it to, "My car won't start because there is no fuel in it". Madness.
jjancura's initial post ruffled a few feathers, but his second post makes perfect sense. One possible solution that might please everyone is to transform this issue into an umbrella issue, with multiple sub-issues (one per problem). I've seen the Netbeans team do this before...
The predominate issue is that scanning, no matter what initiates it, requires in its current design, too much CPU and disk I/O bandwidth. Part of that is based on the limited ability of or incorrectly established content of a cached view of the source and class file content. There really are bugs involved. As was recently reported, one such logic bug was found. I still contend from a point of ignorance that there are architectural bugs which need to be dealt with to remedy the problem overall. Clearly the dev team are defensive about where they believe the software is at. The users who are having this issue impact their productivity, some in quite dramatic ways based on what we read in this issue, are not content with that positioning. Progress will only happen if we can all get on the same page and agree that there are issues that need to be addressed.
The main reason I noticed this is because in 6.1 IT DOESN'T HAPPEN. When I "upgraded" to 6.5 is when the problem appeared. Maybe it doesn't matter at this point but it may be a clue as to what the problem is. Also a possibly important note is that it isn't bad from the beginning. It seems to be manageable until some threshold is crossed (time, number of clean rebuilds?) and then the project scanning goes into its never ending mode.
I'd like to add my two cents to this problem. First I'd like to add that many of the developers here have a lot of very positive things to say about NetBeans. It's amazing how much the product has improved over the past couple of years. We like the way the GUI is laid out. We love the EJB generation wizards. Some of us have used the Swing drag-and-drop editor, and I've heard good things about that. We also like the way plugins are handled. Having said that, the "scanning..." problem described in this ticket is really killing us. No joke. We consider this to be a high priority defect, if there ever was one. Although it's not our primary build mechanism for making installs, we do compilations within Netbeans just to make sure everything is Kosher, and we stick with the default ANT script without any custom ANT tasks. As mentioned previously, we wish we had more control over the scanning process, and we wish that it didn't consume so many resources. Please fix this ASAP.
I'd like to second kidvid comments. The reason I care so much about the "scanning" issues is that I love all the other parts of the IDE, so much better than Eclipse in so many ways but one. The "scanning" issues are what have forced the bulk of the developers here off NB.
As was already mentioned, please try the Scan on Demand module, especially if you experience frequent or long re-scans: http://wiki.netbeans.org/ScanOnDemand Note the module is experimental, it's not meant to be the fix for the scanning problems, but it might give us valuable idea how the performance can really improve by reducing scanning and what other problems may arise. And it may actually help in many cases. We continue working on understanding and fixing the scanning issues on other fronts too.
I tried again the scenario described by theshadow27 in: http://www.netbeans.org/issues/show_bug.cgi?id=134990#desc51 in the latest build of NetBeans 6.8. Here are my findings: A) Select all files: yes it is slow, IDE autodetected frozen AWT queue. Reported as a separate issue 172191. B) Deleting all files: also slow, reported as a separate issue 172199. C) Running the test file more than once: there was a brief period of scanning, but nothing unreasonable or unexpected. Seemed fine to me. D) Also did not experience this (though my machine has 4GB of RAM).
> B) Deleting all files: also slow, reported as a separate issue 172199. That's about performing the delete operation itself. There's also issue 168267 about optimizing the preparatory phase of the delete (i.e. not to initialize refactoring when Safe Delete may not be chosen at all, and not to block invocation of the action if scanning is running). Planned for 6.8.
There is one other sub-scenario that leads to undesirable scanning: 1. Create a Java SE application 2. Create some valid XML schema in this application 3. Create JAXB binding against this schema. This will generate the sources and build them. 4. Use these generated sources in some other classes of the project (e.g. the Main class) 5. Clean the project (which deletes the generated sources) -> sources are scanned 6. Build the project (which re-generates sources) -> sources are also scanned BTW, let's really, really use this bug only for the "sources generated by build script" use case - for any other use cases where the build script does not generate Java sources, separate bug reports need to be filed. Thanks.
What about the annotation processor case? For instance, in Java 6 one can even create an annotation processor to be invoked as part of javac that takes @SpecialClass public class MyClass extends MyBaseClass { // ... } and auto generates MyBaseClass.java. Even in less "extreme" use cases, it is common for an annotation processor to create (and update?) source files as part of javac invocation.
Re pjiricka's last comment (desc 74): could you please be more specific about what should happen, in your opinion? Specifically, should there be error badges for missing classes after step 5 (the classes are missing in reality, are they not)? If not, why (usually, such situations are reported as P2s, why it is not a P2 in this particular case)? Should the code completion show the classes that are missing? What should happen after step 6? Should the error badges disappear? Should the newly appeared classes be visible in the code completion? Thanks.
I don't have a strong opinion about what should happen after step 5 - anything we do is somehow "incorrect". After step 6, the generated classes should of course be visible in code completion, there should be no error badges etc. Regarding the performance aspect, it would be nice if after steps 5 and 6, sources were not rescanned (after step 6, everything is the same as was before step 5, so the IDE could reuse what it already knew at that point).
It's not really the scanning itself that is undesirable, but the fact that it invalidates code completion data until the scanning is completed, and prevents all kinds of actions (source navigation, hints, refactorings) that rely on that data. Of course the scanning also shouldn't consume CPU and I/O resources to the point of significantly slowing down other operations. It might be worth looking at how Eclipse handles such use cases, because it doesn't seem to have these scanning issues and users appear to be happy with how it does things.
The fact that scanning disables code completion and source navigation and louses up error badging, etc, is an underlying issue behind most current scanning issues. This behavior is simply untenable/unacceptable. Instead, scanning should behave like a transaction -- other threads should be able to continue seamlessly using old data until new data is provided by the scanning thread. Sure there should be a means for the other threads to check for a scan in progress, e.g. so refactoring could *warn* that the results may be inaccurate due to a scan in progress. Sure there's a question as to appropriate transaction isolation granularity, e.g. should each source file's scan data be updated it is scanned or should this be delayed until a package or source root level. These details can clearly be worked out. The critical thing is to have scanning block nothing and never remove old scan data until the new replacement scan data is available (which should be an atomic replacement). Anything less than this and NetBeans will forever remain in crippled scan hell. Once this is resolved, then avoiding unnecessary scans becomes purely a CPU/disk grinding issue. Assuming one is not too strapped for CPU/disk resources and the scan thread is not too aggressive, this then becomes a *much* lower priority issue.
See also some ideas given by Peter Nabbefeld: issue 174511.
For sources generated by the annotation processor the scan is not triggered in NB 6.9 (the annotation processing has to be correctly configured). For freeform see issue #185457.
Based on the Comment #78 of the original reporter and also the previous Comment #77 I am marking this as duplicate of #182653. The overall sentiment here is (if I understand correctly) that majority of the users want transactional index and the symptoms described by this bug would not solve the general dissatisfaction with the current state. Only solving #182653 will improve the situation so these reports are in fact the same. *** This bug has been marked as a duplicate of bug 182653 ***
Just my small input to this issue...I've tried netbeans v8, v8.0.1, v7.3.1, and v6.9.1 and all of the versions have a major performance bug with the project scanning. The problem is major in that netbeans in unusable during any scan, which appears to get triggered for no apparent reason. Using sysinternal process monitor, I can see that netbeans is doing a full rescan of my entire project every few minutes. I'm going to try v6.1 of netbeans and if that still causes this issue, I'm done with netbeans. When I first reverted to v6.9.1, the problem didn't happen for several weeks, but now the program is worthless due to the performance. I've tried turning off all of the code completion settings, but the scan is still a full scan of the project. I've tried the "scan on demand"...which doesn't work. Obviously someone at Oracle isn't getting the picture that this is a show-stopper for many developers, and will likely lead me to jump to another IDE. It can't be that difficult to just only update the changed files, a simple check of the file data or which file was saved would be an easy fix and only scan those files, instead of the ENTIRE PROJECT! P.S. I'm a very long time MS VS6 user, due primarily to it's performance. I still use VS6 to open 100+ MiB files as it has no issue with very large ASCII files.
In fact the scan is done for minimal set of files. There are 2 types of update scans in case of the java project: 1st) The project metadata change like classpath change. Such a change requires a full rescan of all sources in affected project as the metadata change affects all sources. The sources which are now compilable may become invalid and vice versa. 2nd) Single file(s) change like save of file (external file modification). Such a change has 2 phases. In the first phase just the changed file(s) is scanned. In the second phase the hash of class API signatures from the changed files are computed and compared to the last know API signatures. If changed (the class API has changed and it can affect other files) the usages of the class is found and the classes which are using the changed class are rescanned. The scope of the second phase can be set in the Tools/Options/Editor/Editor/Hints/Java/Dependency Scanning. If the full scan happens often it can be cause either by frequent metadata changes (custom build steps), changes in signatures of file with has lots of usages (in this case you can limit the Dependency Scanning) or it can be caused by Annotation Processor. If you can share some test project or attach the nps file of such a scan I can help.