This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
I select a file in my project, time how long it takes until the file populates inside of the Members View. To get a 122 line file parsed it took 5 seconds. I tried the same thing with the same file in eclipse. The total time was MUCH less than a second. Note that this file is located on the network...but then again it is for eclipse too and that has no problem with it.
Please include the information from Help>About to see the exact version of NB, OS and JDK... Also, can you please try to generate a thread dump while Navigator (members view) is working, so that we can see some abnormal activity of thread? http://wiki.netbeans.org/GenerateThreadDump
Product Version: NetBeans IDE 6.5 (Build 200811100001) Java: 1.6.0_10; Java HotSpot(TM) Client VM 11.0-b15 System: Windows XP version 5.1 running on x86; Cp1252; en_US (nb) Userdir: C:\Documents and Settings\x3167\.netbeans\6.5
Created attachment 74358 [details] Text thread dump
*** Issue 158788 has been marked as a duplicate of this issue. ***
Scanning/Indexing is a constant problem in NetBeans. I'd think it's time for a redesign: - Cache seems not to be used, instead, it seems in most cases the signatures are parsed every time needed, e.g. when switching to another source in the editor, whether documents have changed or not (e.g. look at navigator behaviour, especially if methods in superclasses are shown). - Indexing isn't interruptable. Cache should be used like a database, writing transaction logs if needed, then it should easier to solve the related problems. These logs need only to know about changed files, so won't be large.
to epdv: feel free to attach some patch for things you are proposing.
Sorry, not able to add a patch, yet, because I've only a very basic understanding of indexing. Especially, cannot find an API for cache access, and indexing seems to be re-invented by every language api (gsf, csl, nbs and others seem to use either gsf or csl, ...). The problem of extensive re-scanning seems to be just a result of missing a good API, too. However, an analogy for better indexing might be the RepaintManager from Swing: use dirty files instead of dirty regions, the painting is not a Graphics object but the indexer database. In more detail: - When starting NB, there shouldn't start a non-interruptable scanning task, only an indexer setup task (IST). - Indexed data should be read using Cache Access API, considering DirtyFilesCache (usage of dirty files, i.e. opening in editor should cause immediate high-priority scanning of those, but not blocking code completion etc.). - Then indexing should be started for all files listed in DirtyFilesCache. - Every detected external file change should add the file's URI to the DirtyFilesCache. - Every signature change, added method or class etc. in editor should add the file's URI to the DirtyFilesCache. - If, for some circumstance, it is not always possible to properly detect file changes, add an "Index Refresh" action to the source path's node context menu. - The DirtyFilesCache should probably exist for every type of information (as javadoc changes could be written to the cache independently - and in an other format - than java class signatures. - The DirtyFilesCache should be written to disk, so it can indexing can restart from where it has been stopped by shutdown. BTW: You shouldn't insist on a shutdown hook, as NB might be stopped by "kill" etc. I'd guess, it's too late to get into 6.7, but probably a chance for e.g. 6.7.1?
Just a question which might be related to scanning: FileOwnerQuery can only return one Project per file - what does indexing do, if there are more projects using the same files, e.g. sources of a framework? Does it scan them once per project? You know, complex frameworks do have several classes, thus scanning will nearly never end ...
We are introducing a new APIs for indefing in NetBeans 6.7, see Mercurial repository - parsing.api package. You have definitely many good ideas. But sometimes its hard (or even impossible) to implement them. So, be patient, please :)
Thank You! My problem with parsing.api is: 1. Badly documented. Even has errors like this "... to implement ParserFactory, and register it in your manifest file in folder called "Editors/" + mimeType" in the API description. Folders in manifest???? 2. I've looked at javascript, ruby and html, but couldn't find a ParserFactory registration - seems they use gsf, csl and lexer api, no good examples, yet. 3. How is parsing related to lexing, code folding etc.? A document about general architecture is missing, so hard to look at it and make better proposals :-(
I've noticed, that JavacParser is created using snapshots and obviously parsing them, if there's only one (which will usually the case for a java file). This is contrary to the parsing api docs for ParserFactory.createParser(...), which say: "... It is important to remember that the snapshots are provided only for the factory and they reflect document states at the time when the factory is called. Since the parsing infrastructure may cache parser instances it will provide a new snapshot when it asks Parsers to do their job. Therefore the snapshots here may only be used for the parser creation, but must never be used for parsing!" Maybe, this is one possible source for the bugs???
Could you please be more specific on "JavacParser is [...] obviously parsing them [snapshots passed to ParserFactory.create]? Where is the snapshot that was passed to the factory parsed? Looking at JavacParserFactory (java.source module), it passes the snapshots to JavacParser. JavacParser then uses the snapshots to: get the size of the collection (# of snapshots) and get the Source corresponding to the first snapshot. The Source corresponding to the first snapshot is then used to get the mimetype and FileObject. Thanks.
Seems I've misunderstood sth., sorry. I've not yet found the "ultimate docs" for editing/parsing/lexing APIs :-( So I can often only guess ...
I've tried to switch off the Navigator panel - hurray, NB performs well! So I had a look now into java.navigator: ElementScanningTask contains these interesting imports: import com.sun.source.tree.ClassTree; import com.sun.source.tree.CompilationUnitTree; import com.sun.source.tree.MethodTree; import com.sun.source.tree.Tree; import com.sun.source.tree.VariableTree; import com.sun.source.util.SourcePositions; import com.sun.source.util.SourcePositions; import com.sun.source.util.TreePath; import com.sun.source.util.TreePathScanner; import com.sun.source.util.Trees; It seems, that navigator is rescanning the selected class (and, if selected, its superclasses) every time the class is selected - couldn't it just use var/cache instead?
In case of source files it does the reparse. But the reparse should be delayed, when you move over the files no parsing should be done.
1. Parsing costs are too high, especially when parsing of parents is activated. 2. (Navigator) Parsing blocks IDE. It seems, while a CancellableTask is being used, nothing cancels it.
Well, I will try to clarify a few things: -the cache contains information about Elements - classes, methods and fields. It does not contain structured information about method bodies, etc. (before NB6.5 it did not contain anything related to method bodies, since NB6.5 it contains the generated byte code for the method bodies, but that is not accessible by the clients). -if you switch tabs in the editor, the target file is always parsed from the source. Its dependencies (referenced classes, superclass, etc.) are filled from the cache, however. Many of the editor features need access to method bodies of the one file visible in the editor, which is not stored in the cache. Note that in this case the source code is taken directly from the editor, only its dependencies are load from the disk (cache). -ElementScanningTask (to my knowledge) touches only Trees corresponding to the elements appearing in the current file - it does not touch method bodies or Trees corresponding to inherited elements. Inherited elements are load from the cache. The navigator could be easily modified not to do this, but I doubt that this would help without other bigger changes (scanning the Tree without going into method bodies is very fast). -the only typical case I know of, when the source code is parsed only for the navigator, is when you select files in the explorer (Project tab, etc.). In this case, the source code is parsed and its elements are attributed. Only in this case it would be possible to load the information from the cache, but that could mean quite a big changes in the infrastructure. I think that before trying to implement such a change, we should be sure that this is the problem. Note that the parsing for navigator is implemented using (simplified) sliding tasks and should be cancelable, so going quickly through a lot of files in the explorer does not automatically mean all of them will be parsed. In the attached thread dump, the indexing is running. It would be nice to have more thread dumps, to see whether the problem is indexing, parsing for navigator or something else. epdv, your problem sounds like issue #160218 (which is most likely unrelated to parsing&caching).
Hm, reporter didn't mention, if navigator has been open, and issue #158788 has been as duplicate, though I mentioned explicitly that I had navigator window open. However, there must be a cause for navigator parsing taking such a long time. In my case, reparsing may be more expensive because I've to access my files via network - don't like it, too.
Forgot to metion, I've tried latest daily build (200904030200) already, still same problems.
Looking at the stacktrace of issue #160218, it seems to me there are some locks and several waiting threads - probably Navigator is just sleeping??? That takes me to another question: How narrow is the dependencies search in Navigator? Does it e.g. use the whole classpath (don't know about what the com.sun classes do, currently too much to investigate, but the issues already take too long to be resolved, so wanted to tell You about this idea).
Sorry, the reference to issue #160218 was meant mainly to lags while switching editor tabs - these were caused by slow creation of (visual) nodes, not related to parsing. To my knowledge, the internal navigator data are always constructed including the inherited members, and the switch to show/not show them affects only the visual part. So, if you see speed difference when the inherited members are visible/not visible, then these are most likely not related to parsing. Also, please note that there are quite a few editor tasks that run before the navigator is populated. Anyway, could you please attach several thread dumps at the slow moments (as already requested in issue #158788)? That could show where is the problem. Thanks.
Created attachment 80032 [details] stack trace
There where many (performance) improvements on that area (indexing, navigator, performance) in last half of year. I have tested Navigator, and it seems much better now. So, I think that this issue is fixed now. At least its good time to reevaluate it. So can you try to reproduce it on current build, and reevaluate this issue (priority, add some new how to reproduce steps, some measurement)?
Well, the most blocking seems to be because of scanning :( I've created a small project (7 classes only!), and NB scanned so long that it just seemed to hang ... I must admit, I've also recently added a jdbc driver library, so probably that code has been scanned - but, why is a whole library scanned, which I don't even use in sources? The most important problem here is, that navigator view is also blocked while scanning, so as a result the navigator looks lazy ... The other important point I should mention is, there's no "Slowness Report" for this - so You'll never get a chance to see, why scanning is so slow.
Please, do not (yet) remove.
We've done couple changes in NB 7.2 , is this issue still valid ? Thanks in advance.
Yes, lots of improvements were done in NB 7.2. For example: 1st) Navigator has higher priority 2nd) Due to background scan the navigator is not blocked by scanning. I've found one more problem caused by partial reparse optimization which causing the navigator to freeze with Please wait node. I will close this issue after it's fixed.
Last know problem fixed jet-main 0fe449daf668