This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
I have several projects open, including core. I open FixedFileSystem.java, navigate to the class decl, and choose Find Usages. Pretty quickly (< 5 sec) I get the results: four classes (incl. FFS.java itself) refer to it. Fine. Then I navigate to its method public void add(String, FixedFileSystem.Instance); and choose Find Usages from there. Not so good. Takes several minutes, and my heap is nearly maxed out; stays at around 175-185/200Mb while the progress dialog is open. Of course there are only a few references to the method, all within the classes I already knew about. Seems that your choice of indices is wrong. Are you indexing on just "add"?! If so, this is a very poor choice, since there are numerous methods named "add" through the JRE and NB projects. If you were indexing on "FixedFileSystem.add", or "FixedFileSystem.add(java.lang.String,org.netbeans.core.projects.FixedFileSystem$Instance)", etc., then the results would be available almost immediately. In fact, if you were only indexing from fully qualified class name to FQ class name (omitting all mention of fields and methods from the indices), it would still be faster in many situations (as well as making the indices smaller, probably). In general, searches on common names are terribly slow, while searches on distinctive names are fast. This seems to be a basic problem with the algorithm.
Of course we know that the indexing you suggest would be better for the speed of finding usages. However there are many problems connected to that. E.g.: - we cannot afford to do a full parse and attribution of every file to be able to determine which method is called in which file during the scanning phase - this information is very fragile - it may change without changing the source code that is affected (e.g. if I remove "add" method from a class A which extends B and "add" is defined also in B, I would have to find all files that referenced A.add and rebuild their indexes to now reference B.add) - doing synchronization of the persisted data when the affected data are not associated with the file that has been modified is expensive These are all reasons why the first prototype of the javacore module did not work - we did it this in even a more efficient way (had a persistent links between the method and its callers) and it was wrong. Synchronizing the storage with the sources was a nightmare and the scanning was more than 20 times slower than now. We plan to address this performance issue differently by lazily filling some hints into the persisted objects. This will make it possible to minimize the number of files that we need to parse without complicating and slowing down the scanning. Then in some cases when running the usages for the first time it will be slow, but subsequent searches should be much faster.
I think you can at least add to the index key the number of args of the method, and information about any args that are primitive types or arrays or well-known final classes like String. This would at least increase the number of "hash buckets". In the case cited, rather than searching all methods named "add", you would be searching only those with two arguments, the first of which is of type String, and the second of which is of some object type. Difficult w/ JDK 1.5 autoboxing and varargs however; easier on 1.4 sources.
Determining number of parameters would still require full parsing. Determining the type of parameters additionaly requires attribution no matter if the types are primitive or not (if there is an expression in the parameter the parser would need to determine type of all operands, etc.). Any of these would make the initial scanning unacceptably time consuming. And as you pointed out, there are problems with JDK 1.5. Anyway, as I said, we still have other ideas for improving the performance.
*** Issue 45776 has been marked as a duplicate of this issue. ***
Somewhat related: issue 47651
I'm going to limit number of resources for parsing according to project dependencies.
Find Usages often fails with OOME on generic-sounding names.
> I'm going to limit number of resources for parsing according to > project dependencies. Implemented: Checking in org/netbeans/modules/refactoring/api/AbstractRefactoring.java; /cvs/refactoring/src/org/netbeans/modules/refactoring/api/AbstractRefactoring.java,v <-- AbstractRefactoring.java new revision: 1.5; previous revision: 1.4 done Processing log script arguments... More commits to come... Checking in org/netbeans/modules/refactoring/api/ui/ParametersPanel.java; /cvs/refactoring/src/org/netbeans/modules/refactoring/api/ui/ParametersPanel.java,v <-- ParametersPanel.java new revision: 1.19; previous revision: 1.18 done Processing log script arguments... More commits to come... Checking in org/netbeans/modules/refactoring/classpath/RefactoringClassPathImplementation.java; /cvs/refactoring/src/org/netbeans/modules/refactoring/classpath/RefactoringClassPathImplementation.java,v <-- RefactoringClassPathImplementation.java new revision: 1.6; previous revision: 1.5 done RCS file: /cvs/refactoring/src/org/netbeans/modules/refactoring/classpath/Util.java,v done Checking in org/netbeans/modules/refactoring/classpath/Util.java; /cvs/refactoring/src/org/netbeans/modules/refactoring/classpath/Util.java,v <-- Util.java initial revision: 1.1 done
Per discussion, -> P1
I fixed OOME during find usages was for particular case. See issue 47651.
I fixed the OOME part of this issue. Find Usages still fails with OOME when looking for heavily used classes in huge projects (e.g. to find 3000 usages of FileObject in web/project and all required I had to use -Xmx160m, since with 128MB it failed - almost at the end), but I think that is acceptable. Thus I am lowering the priority. Checking in src/org/netbeans/modules/javacore/jmiimpl/javamodel/UsageFinder.java; /cvs/java/javacore/src/org/netbeans/modules/javacore/jmiimpl/javamodel/UsageFinder.java,v <-- UsageFinder.java new revision: 1.19; previous revision: 1.18 done Processing log script arguments... More commits to come... Checking in src/org/netbeans/modules/javacore/parser/FeatureInfo.java; /cvs/java/javacore/src/org/netbeans/modules/javacore/parser/FeatureInfo.java,v <-- FeatureInfo.java new revision: 1.9; previous revision: 1.8 done Checking in src/org/netbeans/modules/javacore/parser/MDRParser.java; /cvs/java/javacore/src/org/netbeans/modules/javacore/parser/MDRParser.java,v <-- MDRParser.java new revision: 1.46; previous revision: 1.45 done Checking in src/org/netbeans/modules/javacore/parser/ResourceInfo.java; /cvs/java/javacore/src/org/netbeans/modules/javacore/parser/ResourceInfo.java,v <-- ResourceInfo.java new revision: 1.9; previous revision: 1.8 done
I noticed that searching for usages of OpenProjectList.add(Project) (in projects/projectui) takes prohibitively long; in fact I have to cancel it, wasn't doing much of anything after several minutes except gradually incrementing the progress bar (when it could get in a repaint request at all). I know this is a common method name, but IMHO it does not need to search any sources outside of this project - other modules do not depend directly on projectui (only on projectuiapi), so any usages would need to be directly in the project. I could probably find all the usages using a text search faster than by waiting for Find Usages! The poor behavior here seems to contradict Jan Becicka's promise "I'm going to limit number of resources for parsing according to project dependencies.", unless it is really this slow even within one modest-sized project, which I find hard to believe. I have about a dozen other projects open. Would be helpful if the Find Usages dialog told you which package root it was currently looking in (and perhaps also the package and even class, I don't know). That would make it immediately apparent to the user if it were searching in an area of sources that could not possibly contain any references to the target method or class. In such cases we would be able to file concrete, reproducible bugs quickly. Also, the appearance of real progress is psychologically nice - less painful to wait for the find operation to finish if you can see that it is doing real work, and you have something to look at besides a plain progress bar.
FYI, if you set "perf.refactoring" system property to true, you will see messages like "Attributing method <fqn of the class>.<method name>" - this means the javacore is searching for usages of a given symbol in this method.
At present all projects, that current project depends on, are also searched. In this case (projects/projectui project) it means, that also openide, core and other bunch of projects are also searched. This is not need unless "search from base class" option is on.
1. "Search from Base Class" should have no effect (in fact should not even be shown) in this case because the method does not override anything. (Reported as a separate bug in refactoring.) 2. As I understand it, S.f.B.C. just means to do a search starting at the base declaration. In that case there are just two operations: finding the base declaration, which should be quite fast (Go To Super Declaration is pretty fast, right?); then a regular usages search. So that is not much of an excuse. 3. Even when that checkbox off, the search is quite slow. Seems like it is somewhat faster, but still longer than you would expect any reasonable algorithm to take, if all it needed to do was parse 43 Java sources (which the javac compiler does in a couple of seconds). 4. I just realized that in fact the add and remove methods are not in OpenProjectList itself - they are in OPL.RecentProjects, a private nested class. Which means that all occurrences must logically be in the *same Java source file* as the definition!
So in this particular case, with the Search from Base Class checkbox off, I get two usages - both in OpenProjectList.java, of course. My log file with perf.refactoring=true shows 442 "Attributing method" lines! Not only from the projectui module, but other modules too, including ones it neither depends on nor depend on it, e.g. Attributing method: org.netbeans.modules.java.bridge.ElementImpl.removePropertyChangeListener3055 Attributing method: org.netbeans.api.java.classpath.GlobalPathRegistry.unregister638 Attributing method: org.netbeans.modules.javacore.ClassIndex.rollbackChanges553 Attributing method: org.netbeans.modules.clazz.ClassDataObject.forceReload944 Clearly whatever optimization you tried to do, didn't work, at least not here.
So, shouldn't this be a P2 again?
> So, shouldn't this be a P2 again? Why not > 2. As I understand it, S.f.B.C. just means to do a search starting at the base declaration. In that case there are just two operations: finding the base declaration, which should be quite fast (Go To Super Declaration is pretty fast, right?); then a regular usages search. So that is not much of an excuse. It is not that simple. Java have multiple inheritance (well - pseudo inheritance). So there can be several methods from base class (class can override method "foo" from super class, but this particular method can also be implementation of some method from several interfaces).
Hm - it looks like that Refactoring classpath is created correctly according to project dependencies, but this classpath is not used (MergedClassPath is used instead?). I'll try to find what's wrong.
Fixed. Checking in java/javacore/src/org/netbeans/modules/javacore/ExclusiveMutex.java; /cvs/java/javacore/src/org/netbeans/modules/javacore/ExclusiveMutex.java,v <-- ExclusiveMutex.java new revision: 1.24; previous revision: 1.23 done Processing log script arguments... More commits to come... Checking in refactoring/src/org/netbeans/modules/refactoring/NbAbstractRefactoring.java; /cvs/refactoring/src/org/netbeans/modules/refactoring/NbAbstractRefactoring.java,v <-- NbAbstractRefactoring.java new revision: 1.8; previous revision: 1.7 done Processing log script arguments... More commits to come... Checking in refactoring/src/org/netbeans/modules/refactoring/api/MoveClassRefactoring.java; /cvs/refactoring/src/org/netbeans/modules/refactoring/api/MoveClassRefactoring.java,v <-- MoveClassRefactoring.java new revision: 1.26; previous revision: 1.25 done Processing log script arguments... More commits to come... Checking in refactoring/src/org/netbeans/modules/refactoring/api/ui/AbstractRefactoringAction.java; /cvs/refactoring/src/org/netbeans/modules/refactoring/api/ui/AbstractRefactoringAction.java,v <-- AbstractRefactoringAction.java new revision: 1.9; previous revision: 1.8 done Changing back to P3.
"Java have multiple inheritance (well - pseudo inheritance). So there can be several methods from base class (class can override method "foo" from super class, but this particular method can also be implementation of some method from several interfaces)." - sure this is possible, but pretty rare in practice! And even if so, the "several interfaces" are not likely to be more than two (unless you are purposely making pathologically unreadable code). Finding the declarations of the method in those interfaces shouldn't take long, IMHO. If you have public interface X { run(); } public class MyClazz implements X, Runnable { public void run() {} // press Alt-F7 here } then you just need to check MyClazz's inheritance tree (a limited operation) to find that MyClazz.run could be called as X.run or Runnable.run. Then search those two independently.
You are right - current semantics of "Search From Base class" is misleading. It indeed covers "pathologically unreadable code", but it is hardly presentable to user and does not easily allow to do performance optimizations. I'll try to change the implementation this way: "Search From Base Class" will search usages of method from base class (only one method) - this method will be presented to user this way: [X] Search from Runnable.run() See issue 48027
Last 2 coments implemented: Checking in java/javacore/src/org/netbeans/modules/javacore/jmiimpl/javamodel/MethodImpl.java; /cvs/java/javacore/src/org/netbeans/modules/javacore/jmiimpl/javamodel/MethodImpl.java,v <-- MethodImpl.java new revision: 1.13; previous revision: 1.12 done Processing log script arguments... More commits to come... Checking in refactoring/src/org/netbeans/modules/refactoring/NbAbstractRefactoring.java; /cvs/refactoring/src/org/netbeans/modules/refactoring/NbAbstractRefactoring.java,v <-- NbAbstractRefactoring.java new revision: 1.9; previous revision: 1.8 done Processing log script arguments... More commits to come... Checking in refactoring/src/org/netbeans/modules/refactoring/api/ChangeParameters.java; /cvs/refactoring/src/org/netbeans/modules/refactoring/api/ChangeParameters.java,v <-- ChangeParameters.java new revision: 1.19; previous revision: 1.18 done Checking in refactoring/src/org/netbeans/modules/refactoring/api/MoveClassRefactoring.java; /cvs/refactoring/src/org/netbeans/modules/refactoring/api/MoveClassRefactoring.java,v <-- MoveClassRefactoring.java new revision: 1.27; previous revision: 1.26 done Checking in refactoring/src/org/netbeans/modules/refactoring/api/RenameRefactoring.java; /cvs/refactoring/src/org/netbeans/modules/refactoring/api/RenameRefactoring.java,v <-- RenameRefactoring.java new revision: 1.18; previous revision: 1.17 done Checking in refactoring/src/org/netbeans/modules/refactoring/api/WhereUsedQuery.java; /cvs/refactoring/src/org/netbeans/modules/refactoring/api/WhereUsedQuery.java,v <-- WhereUsedQuery.java new revision: 1.14; previous revision: 1.13 done Processing log script arguments... More commits to come... Checking in refactoring/src/org/netbeans/modules/refactoring/ui/Bundle.properties; /cvs/refactoring/src/org/netbeans/modules/refactoring/ui/Bundle.properties,v <-- Bundle.properties new revision: 1.27; previous revision: 1.26 done Checking in refactoring/src/org/netbeans/modules/refactoring/ui/WhereUsedPanel.form; /cvs/refactoring/src/org/netbeans/modules/refactoring/ui/WhereUsedPanel.form,v <-- WhereUsedPanel.form new revision: 1.6; previous revision: 1.5 done Checking in refactoring/src/org/netbeans/modules/refactoring/ui/WhereUsedPanel.java; /cvs/refactoring/src/org/netbeans/modules/refactoring/ui/WhereUsedPanel.java,v <-- WhereUsedPanel.java new revision: 1.14; previous revision: 1.13 done
Reassigning to javacore for further evaluation.
Javacore module was replaced by Retouche infrastructure. This bug is not valid in trunk any more.
Reorganization of java component