Bug 169252 - Excess scanning after opening web.jsf
Excess scanning after opening web.jsf
Status: RESOLVED DUPLICATE of bug 177950
Product: editor
Classification: Unclassified
Component: Parsing & Indexing
6.x
All All
: P3 (vote)
: 7.0
Assigned To: Tomas Zezula
issues@editor
: PERFORMANCE
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2009-07-27 18:02 UTC by Jesse Glick
Modified: 2015-03-05 09:37 UTC (History)
3 users (show)

See Also:
Issue Type: DEFECT
:


Attachments
Log file (92.49 KB, application/octet-stream)
2009-07-27 18:04 UTC, Jesse Glick
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Jesse Glick 2009-07-27 18:02:42 UTC
I opened web.jsf in 090727. I had opened this same project a few days before, so most indices should have been present
already, probably needing a few refreshes. Log file shows the IDE scanning 284 (!) modules... such as javawebstart which
bear no apparent relationship to web.jsf and yet I just 236 msec scanning it.
Comment 1 Jesse Glick 2009-07-27 18:04:41 UTC
Created attachment 85266 [details]
Log file
Comment 2 Jan Lahoda 2009-08-20 09:58:15 UTC
Reassigning all moonko's java/source bugs to myself.
Comment 3 Jan Lahoda 2009-08-25 10:11:35 UTC
Summary of offline discussion (please correct me if I misunderstood something):
-the scanned modules are transitive dependencies of the opened modules (its tests)
-proposal: do not scan (check timestamps) of indirect dependencies if the caches exist
-question: what would be broken if the above proposal would be implemented?

One thing that would be broken is Go to Type/File/Symbol/... which would show incorrect data for the indirect
dependencies. Also, opening a file from the direct dependency would require timestamp check of its direct dependencies,
which could take quite some time (actually, it could take more time in total than the initial scan, IMO) - right now,
everything is done at project opening and after that all should be fast. With the proposed change, almost any action in
the IDE could cause long-time scanning.

Anyway, I do not think this can be implemented on java.source level.
Comment 4 Vitezslav Stejskal 2009-08-25 14:27:17 UTC
Ok, I'll try investigate this.
Comment 5 Jesse Glick 2009-08-25 16:02:53 UTC
Do I really care if Go to Type etc. is not perfectly accurate for the javawebstart module when I just opened web.jsf? If
I were so interested in the javawebstart module, I would have opened it to begin with. Accuracy is important (1) for
code actually in open projects, (2) for signatures which might affect error badges on open projects (i.e. direct deps).

If I do open a file from a direct dep such as api.web.webmodule, it seems fine for there to be a scan of any direct deps
of api.web.webmodule which were not already direct deps of web.jsf - this should be pretty quick (in this case there are
in fact no such additional deps at all). I would certainly tolerate such a scan if I opened the whole web.webmodule
project. I'm not sure how such an incremental scan could ever take more time in total than the current transitive scan does.
Comment 6 Jan Lahoda 2009-08-26 21:16:48 UTC
I, for one, quite often open classes from un-opened projects via Go to Type, and I do not really care if the class is in
direct dependency of in indirect dependency. (I typically try to type the class name, and in the rare case the
containing source root is not accessible transitively from the opened projects, I open it.) Having inaccurate
information would not really help me. Inducing a possibly long-lasting scan as a result of such an operation is also not
something I want, as that would block me from working. Although the scan would probably be typically fast, it may be
quite long in some cases. The eager scan runs during IDE startup/project group switch, which is often a long-lasting
task anyway (opening nb.org project from the full main check-out is pretty slow for me), so that does not bother me so much.

The total work needed by the incremental scan approach can be bigger than for the eager approach, because the first pass
actually (in general) produces incorrect cache content for the direct dependencies, and this needs to be corrected when
the next-level dependencies is scanned (using e.g. cross-source-root rebuild).
Comment 7 Jesse Glick 2009-08-29 17:12:44 UTC
We are working on making loading a nb.org project much faster by avoiding a full scan in most cases including project
opening (there is an issue filed for this). Of course if the indexer continues to request transitive dependencies then
the benefit will be less because a large subset of the repository will need to be loaded anyway.

"the first pass [may produce] incorrect cache content for the direct dependencies" - do you have an example of what this
would mean in practical terms? How often would this be an issue?

As another example of how much slower initial scan is than you might naively expect, if I open hudson* and kenai*
modules my log shows the indexer spending 394 msec on javahelp/src, 437 msec on xml.catalog/src, 578 msec on
mercurial/src, 638 msec on xml.xam/src, and on and on. (This is with a cold disk cache so just checking timestamps on
these source files is pretty slow.) It is very unlikely I would attempt to open any classes from any of these modules
while working on cdev modules, and changes in them could not possibly affect the compilability of my open projects.
Comment 8 Jesse Glick 2009-09-17 20:52:49 UTC
Just opened openide.awt, a module whose transitive dependencies (including tests) should be limited to openide.util,
openide.filesystems, libs.junit4, and nbjunit; huge amount of scanning occurred, I have no idea why:

Resolving dependencies took: 7,169 ms
Complete indexing of 0 binary roots took: 0 ms
Resolving dependencies took: 360 ms
Complete indexing of 36 binary roots took: 42139 ms
Indexing of: file:/space/src/nb/core-main/openide.util/src/ took: 1284 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/openide.filesystems/src/ took: 996 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/spi.tasklist/src/ took: 2631 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/o.n.swing.plaf/src/ took: 1050 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/xml.xam/src/ took: 1464 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/openide.awt/src/ took: 897 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/o.jdesktop.layout/src/ took: 809 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/api.progress/src/ took: 638 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/openide.dialogs/src/ took: 679 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/openide.nodes/src/ took: 703 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/editor.mimelookup/src/ took: 509 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/openide.modules/src/ took: 645 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/openide.windows/src/ took: 594 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/openide.text/src/ took: 727 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/o.n.swing.outline/src/ took: 619 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/openide.explorer/src/ took: 656 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/openide.actions/src/ took: 727 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/java.preprocessorbridge/src/ took: 847 ms (New or modified files: 0, Deleted
files: 0)
Indexing of: file:/space/src/nb/core-main/o.n.insane/insanelib/src/ took: 646 ms (New or modified files: 0, Deleted
files: 0)
Indexing of: file:/space/src/nb/core-main/libs.junit4/src/ took: 602 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/nbjunit/src/ took: 743 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/queries/src/ took: 683 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/openide.loaders/src/ took: 696 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/api.xml/src/ took: 862 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/o.n.bootstrap/src/ took: 747 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/core.startup/src/ took: 649 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/openide.io/src/ took: 663 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/o.n.core/src/ took: 2021 ms (New or modified files: 1, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/spi.quicksearch/src/ took: 773 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/options.api/src/ took: 572 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/projectapi/src/ took: 1011 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/projectuiapi/src/ took: 1064 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/spi.navigator/src/ took: 843 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/openide.execution/src/ took: 801 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/o.apache.tools.ant.module/src/ took: 886 ms (New or modified files: 0, Deleted
files: 0)
Indexing of: file:/space/src/nb/core-main/api.annotations.common/src/ took: 673 ms (New or modified files: 0, Deleted
files: 0)
Indexing of: file:/space/src/nb/core-main/api.java.classpath/src/ took: 859 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/api.java/src/ took: 818 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/classfile/src/ took: 885 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/java.platform/src/ took: 799 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/project.libraries/src/ took: 862 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/o.apache.xml.resolver/src/ took: 807 ms (New or modified files: 0, Deleted
files: 0)
Indexing of: file:/space/src/nb/core-main/xml.catalog/src/ took: 787 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/project.ant/src/ took: 771 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/o.openidex.util/src/ took: 819 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/java.project/src/ took: 789 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/javahelp/src/ took: 880 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/editor.util/src/ took: 876 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/libs.lucene/src/ took: 680 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/editor.fold/src/ took: 902 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/editor.settings/src/ took: 642 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/lexer/src/ took: 1015 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/editor.lib2/src/ took: 747 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/editor.lib/src/ took: 908 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/editor.settings.storage/src/ took: 844 ms (New or modified files: 0, Deleted
files: 0)
Indexing of: file:/space/src/nb/core-main/parsing.api/src/ took: 3149 ms (New or modified files: 2, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/editor.indent/src/ took: 776 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/o.n.swing.tabcontrol/src/ took: 818 ms (New or modified files: 0, Deleted
files: 0)
Indexing of: file:/space/src/nb/core-main/editor/src/ took: 856 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/options.keymap/src/ took: 937 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/options.editor/src/ took: 975 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/java.lexer/src/ took: 741 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/core.multiview/src/ took: 692 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/editor.guards/src/ took: 682 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/spi.viewmodel/src/ took: 3594 ms (New or modified files: 6, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/libs.javacapi/src/ took: 718 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/libs.javacimpl/src/ took: 743 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/masterfs/src/ took: 817 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/editor.errorstripe.api/src/ took: 844 ms (New or modified files: 0, Deleted
files: 0)
Indexing of: file:/space/src/nb/core-main/editor.errorstripe/src/ took: 684 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/diff/src/ took: 871 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/refactoring.api/src/ took: 895 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/java.source/src/ took: 4242 ms (New or modified files: 3, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/libs.commons_logging/src/ took: 885 ms (New or modified files: 0, Deleted
files: 0)
Indexing of: file:/space/src/nb/core-main/maven.embedder/src/ took: 783 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/maven.indexer/src/ took: 688 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/maven.model/src/ took: 2139 ms (New or modified files: 6, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/extexecution/src/ took: 798 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/api.debugger/src/ took: 868 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/api.debugger.jpda/src/ took: 213 ms (New or modified files: 0, Deleted files: 0)
Indexing of: file:/space/src/nb/core-main/openide.awt/test/unit/src/ took: 786 ms (New or modified files: 0, Deleted
files: 0)
Complete indexing of 81 source roots took: 77294 ms (New or modified files: 18, Deleted files: 0)
Comment 9 Vitezslav Stejskal 2009-09-23 09:31:45 UTC
Umm, strange, I just did the same - fresh build, empty userdir, opened openide.awt module. Here is my log:

Resolving dependencies took: 1,382 ms
Complete indexing of 27 binary roots took: 14038 ms
Indexing of: file:/work/netbeans/jet-main/libs.junit4/src/ took: 1280 ms (New or modified files: 1, Deleted files: 0)
Indexing of: file:/work/netbeans/jet-main/o.n.insane/insanelib/src/ took: 2153 ms (New or modified files: 23, Deleted
files: 0)
Indexing of: file:/work/netbeans/jet-main/openide.util/src/ took: 26379 ms (New or modified files: 108, Deleted files: 0)
Indexing of: file:/work/netbeans/jet-main/nbjunit/src/ took: 1479 ms (New or modified files: 34, Deleted files: 0)
Indexing of: file:/work/netbeans/jet-main/openide.awt/src/ took: 1292 ms (New or modified files: 81, Deleted files: 0)
Indexing of: file:/work/netbeans/jet-main/openide.filesystems/src/ took: 2682 ms (New or modified files: 76, Deleted
files: 0)
Indexing of: file:/work/netbeans/jet-main/openide.awt/test/unit/src/ took: 700 ms (New or modified files: 28, Deleted
files: 0)
Indexing of: jar:file:/work/netbeans/jet-main/nbbuild/netbeans/ide12/jsstubs/allstubs.zip!/ took: 2855 ms (New or
modified files: 189, Deleted files: 0)
Complete indexing of 8 source roots took: 38820 ms (New or modified files: 540, Deleted files: 0)
Comment 10 Jesse Glick 2009-09-24 15:08:07 UTC
Indeed I can't reproduce the problem with openide.awt on a fresh userdir. Maybe some other previously closed project
still in memory triggered scanning for some reason.
Comment 11 Jesse Glick 2010-02-27 08:29:51 UTC
It may be useful to have an option (turned on by system property?) to avoid scanning indirect dependencies; then I could run with it for a while and see how well it works in practice. If the drawbacks prove minor and the performance benefit is significant it could be promoted to a GUI option.
Comment 12 Jesse Glick 2010-03-24 19:58:00 UTC
I think this needs to be prioritized. Why do I need to scan javascript.hints when I have opened java.j2seproject? It doesn't make any sense, and at least for nb.org module development is the main reason classpath scanning is so slow.
Comment 13 Tomas Zezula 2015-03-05 09:37:44 UTC

*** This bug has been marked as a duplicate of bug 177950 ***


By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2014, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo