This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
I opened web.jsf in 090727. I had opened this same project a few days before, so most indices should have been present already, probably needing a few refreshes. Log file shows the IDE scanning 284 (!) modules... such as javawebstart which bear no apparent relationship to web.jsf and yet I just 236 msec scanning it.
Created attachment 85266 [details] Log file
Reassigning all moonko's java/source bugs to myself.
Summary of offline discussion (please correct me if I misunderstood something): -the scanned modules are transitive dependencies of the opened modules (its tests) -proposal: do not scan (check timestamps) of indirect dependencies if the caches exist -question: what would be broken if the above proposal would be implemented? One thing that would be broken is Go to Type/File/Symbol/... which would show incorrect data for the indirect dependencies. Also, opening a file from the direct dependency would require timestamp check of its direct dependencies, which could take quite some time (actually, it could take more time in total than the initial scan, IMO) - right now, everything is done at project opening and after that all should be fast. With the proposed change, almost any action in the IDE could cause long-time scanning. Anyway, I do not think this can be implemented on java.source level.
Ok, I'll try investigate this.
Do I really care if Go to Type etc. is not perfectly accurate for the javawebstart module when I just opened web.jsf? If I were so interested in the javawebstart module, I would have opened it to begin with. Accuracy is important (1) for code actually in open projects, (2) for signatures which might affect error badges on open projects (i.e. direct deps). If I do open a file from a direct dep such as api.web.webmodule, it seems fine for there to be a scan of any direct deps of api.web.webmodule which were not already direct deps of web.jsf - this should be pretty quick (in this case there are in fact no such additional deps at all). I would certainly tolerate such a scan if I opened the whole web.webmodule project. I'm not sure how such an incremental scan could ever take more time in total than the current transitive scan does.
I, for one, quite often open classes from un-opened projects via Go to Type, and I do not really care if the class is in direct dependency of in indirect dependency. (I typically try to type the class name, and in the rare case the containing source root is not accessible transitively from the opened projects, I open it.) Having inaccurate information would not really help me. Inducing a possibly long-lasting scan as a result of such an operation is also not something I want, as that would block me from working. Although the scan would probably be typically fast, it may be quite long in some cases. The eager scan runs during IDE startup/project group switch, which is often a long-lasting task anyway (opening nb.org project from the full main check-out is pretty slow for me), so that does not bother me so much. The total work needed by the incremental scan approach can be bigger than for the eager approach, because the first pass actually (in general) produces incorrect cache content for the direct dependencies, and this needs to be corrected when the next-level dependencies is scanned (using e.g. cross-source-root rebuild).
We are working on making loading a nb.org project much faster by avoiding a full scan in most cases including project opening (there is an issue filed for this). Of course if the indexer continues to request transitive dependencies then the benefit will be less because a large subset of the repository will need to be loaded anyway. "the first pass [may produce] incorrect cache content for the direct dependencies" - do you have an example of what this would mean in practical terms? How often would this be an issue? As another example of how much slower initial scan is than you might naively expect, if I open hudson* and kenai* modules my log shows the indexer spending 394 msec on javahelp/src, 437 msec on xml.catalog/src, 578 msec on mercurial/src, 638 msec on xml.xam/src, and on and on. (This is with a cold disk cache so just checking timestamps on these source files is pretty slow.) It is very unlikely I would attempt to open any classes from any of these modules while working on cdev modules, and changes in them could not possibly affect the compilability of my open projects.
Just opened openide.awt, a module whose transitive dependencies (including tests) should be limited to openide.util, openide.filesystems, libs.junit4, and nbjunit; huge amount of scanning occurred, I have no idea why: Resolving dependencies took: 7,169 ms Complete indexing of 0 binary roots took: 0 ms Resolving dependencies took: 360 ms Complete indexing of 36 binary roots took: 42139 ms Indexing of: file:/space/src/nb/core-main/openide.util/src/ took: 1284 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/openide.filesystems/src/ took: 996 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/spi.tasklist/src/ took: 2631 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/o.n.swing.plaf/src/ took: 1050 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/xml.xam/src/ took: 1464 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/openide.awt/src/ took: 897 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/o.jdesktop.layout/src/ took: 809 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/api.progress/src/ took: 638 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/openide.dialogs/src/ took: 679 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/openide.nodes/src/ took: 703 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/editor.mimelookup/src/ took: 509 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/openide.modules/src/ took: 645 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/openide.windows/src/ took: 594 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/openide.text/src/ took: 727 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/o.n.swing.outline/src/ took: 619 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/openide.explorer/src/ took: 656 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/openide.actions/src/ took: 727 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/java.preprocessorbridge/src/ took: 847 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/o.n.insane/insanelib/src/ took: 646 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/libs.junit4/src/ took: 602 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/nbjunit/src/ took: 743 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/queries/src/ took: 683 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/openide.loaders/src/ took: 696 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/api.xml/src/ took: 862 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/o.n.bootstrap/src/ took: 747 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/core.startup/src/ took: 649 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/openide.io/src/ took: 663 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/o.n.core/src/ took: 2021 ms (New or modified files: 1, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/spi.quicksearch/src/ took: 773 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/options.api/src/ took: 572 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/projectapi/src/ took: 1011 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/projectuiapi/src/ took: 1064 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/spi.navigator/src/ took: 843 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/openide.execution/src/ took: 801 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/o.apache.tools.ant.module/src/ took: 886 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/api.annotations.common/src/ took: 673 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/api.java.classpath/src/ took: 859 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/api.java/src/ took: 818 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/classfile/src/ took: 885 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/java.platform/src/ took: 799 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/project.libraries/src/ took: 862 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/o.apache.xml.resolver/src/ took: 807 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/xml.catalog/src/ took: 787 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/project.ant/src/ took: 771 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/o.openidex.util/src/ took: 819 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/java.project/src/ took: 789 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/javahelp/src/ took: 880 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/editor.util/src/ took: 876 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/libs.lucene/src/ took: 680 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/editor.fold/src/ took: 902 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/editor.settings/src/ took: 642 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/lexer/src/ took: 1015 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/editor.lib2/src/ took: 747 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/editor.lib/src/ took: 908 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/editor.settings.storage/src/ took: 844 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/parsing.api/src/ took: 3149 ms (New or modified files: 2, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/editor.indent/src/ took: 776 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/o.n.swing.tabcontrol/src/ took: 818 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/editor/src/ took: 856 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/options.keymap/src/ took: 937 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/options.editor/src/ took: 975 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/java.lexer/src/ took: 741 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/core.multiview/src/ took: 692 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/editor.guards/src/ took: 682 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/spi.viewmodel/src/ took: 3594 ms (New or modified files: 6, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/libs.javacapi/src/ took: 718 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/libs.javacimpl/src/ took: 743 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/masterfs/src/ took: 817 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/editor.errorstripe.api/src/ took: 844 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/editor.errorstripe/src/ took: 684 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/diff/src/ took: 871 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/refactoring.api/src/ took: 895 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/java.source/src/ took: 4242 ms (New or modified files: 3, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/libs.commons_logging/src/ took: 885 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/maven.embedder/src/ took: 783 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/maven.indexer/src/ took: 688 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/maven.model/src/ took: 2139 ms (New or modified files: 6, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/extexecution/src/ took: 798 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/api.debugger/src/ took: 868 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/api.debugger.jpda/src/ took: 213 ms (New or modified files: 0, Deleted files: 0) Indexing of: file:/space/src/nb/core-main/openide.awt/test/unit/src/ took: 786 ms (New or modified files: 0, Deleted files: 0) Complete indexing of 81 source roots took: 77294 ms (New or modified files: 18, Deleted files: 0)
Umm, strange, I just did the same - fresh build, empty userdir, opened openide.awt module. Here is my log: Resolving dependencies took: 1,382 ms Complete indexing of 27 binary roots took: 14038 ms Indexing of: file:/work/netbeans/jet-main/libs.junit4/src/ took: 1280 ms (New or modified files: 1, Deleted files: 0) Indexing of: file:/work/netbeans/jet-main/o.n.insane/insanelib/src/ took: 2153 ms (New or modified files: 23, Deleted files: 0) Indexing of: file:/work/netbeans/jet-main/openide.util/src/ took: 26379 ms (New or modified files: 108, Deleted files: 0) Indexing of: file:/work/netbeans/jet-main/nbjunit/src/ took: 1479 ms (New or modified files: 34, Deleted files: 0) Indexing of: file:/work/netbeans/jet-main/openide.awt/src/ took: 1292 ms (New or modified files: 81, Deleted files: 0) Indexing of: file:/work/netbeans/jet-main/openide.filesystems/src/ took: 2682 ms (New or modified files: 76, Deleted files: 0) Indexing of: file:/work/netbeans/jet-main/openide.awt/test/unit/src/ took: 700 ms (New or modified files: 28, Deleted files: 0) Indexing of: jar:file:/work/netbeans/jet-main/nbbuild/netbeans/ide12/jsstubs/allstubs.zip!/ took: 2855 ms (New or modified files: 189, Deleted files: 0) Complete indexing of 8 source roots took: 38820 ms (New or modified files: 540, Deleted files: 0)
Indeed I can't reproduce the problem with openide.awt on a fresh userdir. Maybe some other previously closed project still in memory triggered scanning for some reason.
It may be useful to have an option (turned on by system property?) to avoid scanning indirect dependencies; then I could run with it for a while and see how well it works in practice. If the drawbacks prove minor and the performance benefit is significant it could be promoted to a GUI option.
I think this needs to be prioritized. Why do I need to scan javascript.hints when I have opened java.j2seproject? It doesn't make any sense, and at least for nb.org module development is the main reason classpath scanning is so slow.
*** This bug has been marked as a duplicate of bug 177950 ***