This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 189852 - Minimize scanning of broken sources when opening Maven project for the first time
Summary: Minimize scanning of broken sources when opening Maven project for the first ...
Status: RESOLVED FIXED
Alias: None
Product: java
Classification: Unclassified
Component: Source (show other bugs)
Version: 6.x
Hardware: All All
: P3 normal (vote)
Assignee: Svata Dedic
URL:
Keywords: PERFORMANCE
Depends on:
Blocks:
 
Reported: 2010-08-25 14:41 UTC by Jaroslav Tulach
Modified: 2016-05-25 06:16 UTC (History)
3 users (show)

See Also:
Issue Type: ENHANCEMENT
Exception Reporter:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jaroslav Tulach 2010-08-25 14:41:31 UTC
According to our email conversation, the maven system shall already detect that libraries are missing and mark the whole project as containing "broken references".

At such situation user shall have access to context menu item to fix the problem (done), or even be faced with a dialog to just like Java SE projects do.


Moreover, until the project model reports no locally missing dependencies (other than SNAPSHOT dependencies with a known owner) the project source roots shall not be registered to the Java infrastructure. 

This will eliminate useless scan (full of errors) and subsequent rescan which throws the previous result away completely.
Comment 1 Jesse Glick 2010-08-25 15:09:22 UTC
Could probably be made to work. Would likely need to report the actual SourceGroup's from the start if asked, but delay calls to GlobalPathRegistry until all references are fixed. Not clear what the effects will be if the user, say, opens and begins editing source files without first fixing the references. Also remember that various metadata modifications after the project is open can result in classpath items disappearing as far as the Java parser is concerned, which may or may not result in compilation errors according to whether the dependency was actually needed.

Note that an analogous change could be made for non-Maven projects, which in many cases also have a notion of missing references. This begs the question of whether the project type is really the right place to make this fix. Since a ClassPath (e.g. of type COMPILE) can report URLs which do not yet exist, the Java indexer could simply decline to scan a source root whose classpath contains nonexistent elements, solving the problem more generally. (Automatic projects would likely choose in this case to specifically remove nonexistent elements from the reported classpath, since the IDE cannot be sure that the element is actually needed for anything.)
Comment 2 Tomas Zezula 2010-08-25 15:20:52 UTC
>opens and begins editing source files without first fixing the references
Until the classpaths are registered, the java.source does not handle the sources (the sources which are not yet scanned). The source path of such a sources is empty.
Comment 3 Jesse Glick 2010-08-25 15:49:19 UTC
(In reply to comment #2)
> Until the classpaths are registered, the java.source does not handle the
> sources

Well this can't be quite true, because you can open sources from a nonopen project and the Java editor will handle it correctly (so long as the owning project reports the correct ClassPath). How would this case be any different?
Comment 4 Tomas Zezula 2010-08-25 16:08:34 UTC
Sorry I've forgot about the SourcePathCheck which starts a scan of project sources from non opened projects. It starts the scan if the project is not yet scanned and the source root comes from project. We can add some additional rule to handle this case.
Comment 5 Jesse Glick 2010-08-25 16:49:16 UTC
What sort of additional rule? It it seeming cleaner to handle this scenario in java.source.

Also I am not yet convinced that the change is even desirable. Sometimes it is useful to see which classes are rendered uncompilable by a missing dependency, in case resolving the reference is not so trivial. It is up to the Java parser to rescan only the sources of this project (not other open projects) if and when its classpath is fixed.

Also see bug #189833; it may be possible to use an online embedder (or otherwise download valid but remote dependencies) after opening the project but before registering paths, which would in most cases happen fairly quickly before the user started to play around with sources.
Comment 6 Tomas Zezula 2010-08-25 17:09:10 UTC
>What sort of additional rule? It it seeming cleaner to handle this scenario in java.source.

Yes, this is what I've meant. We only need a way how to find out that the root should not be scanned.

>Also I am not yet convinced that the change is even desirable. Sometimes it is
>useful to see which classes are rendered uncompilable by a missing dependency,
>in case resolving the reference is not so trivial. It is up to the Java parser
>to rescan only the sources of this project (not other open projects) if and
>when its classpath is fixed.

This is exactly what the java parser do. But when such a project with broken dependencies
is a library project of other projects you need to rescan also other project depending on the public types which were changed by fixing the classpath (error types were resolved into the correct types). This is done by the JavaCustomIndexer.

Another problem is that you scan the project at least twice. First when you opened it, you scan it with broken references and error types. Second time after the libraries are downloaded (error types -> declared types). But sometimes spurious fs events are generated during the download of libraries and the project is scanned even more times.
Comment 7 Jesse Glick 2010-08-25 18:17:41 UTC
(In reply to comment #6)
> We only need a way how to find out that the root should not be scanned.

Existence of a missing classpath item seems the only sane way. You can then listen to this JAR becoming available later; you would anyway be listening to changes in it if it were available from the start.

> when such a project with broken dependencies
> is a library project of other projects you need to rescan also other project
> depending on the public types which were changed by fixing the classpath

True, but not much of an issue for Maven projects where dependencies are normally transitive (since those other projects would also have missing classpath entries), i.e. if the parser used the proposed rule then it would skip them all at first. If the dependencies are not transitive then generally you do not need to rescan the downstream project anyway, though the Java indexer might not be clever enough to realize this.

> sometimes spurious fs events are generated during the download of libraries and
> the project is scanned even more times.

This would not be a problem under either proposed fix - delaying of registration from the project type, or delaying scanning from the parser due to broken entries. In either case, things would wait until all dependencies were ready.
Comment 8 Tomas Zezula 2010-08-26 13:12:11 UTC
>Existence of a missing classpath item seems the only sane way. You can then listen to this JAR becoming >available later; you would anyway be listening to changes in it if it were available from the start.
This will probably break lots of ant based projects. It's valid to have non existing resources on the classpath, if it's unused, and lots of ant based project do it. We will not scan such projects, this should be maven specific.
Comment 9 Jesse Glick 2010-08-26 15:00:59 UTC
(In reply to comment #8)
> It's valid to have non-existing resources on the classpath, if it's unused

Well the same is true for Maven.

> lots of Ant-based projects do it

Like which? IDE-managed Ant projects will show a warning badge and prompt you to resolve broken references in this case. I don't see how that's any different from Maven projects.
Comment 10 Tomas Zezula 2010-08-26 18:48:49 UTC
>Well the same is true for Maven.
Technically yes, but not from the user point of view. In ant project this is some of projects and the user somehow accepts it in case of maven it's nearly every time you open a new project.


>Like which? IDE-managed Ant projects will show a warning badge and prompt you
>to resolve broken references in this case. I don't see how that's any different
>from Maven projects.
Yes, the project is missing some useless jar and user accepts it (does not care about the badge and disables the dialog). Or the project is not complete (eg. unshared project) and user has to fix it - this is the same behavior as in maven case. But both of these cases are different for user point of view and cannot be distinguished from each other.
Comment 11 Jesse Glick 2010-09-20 19:34:47 UTC
(In reply to comment #10)
> In ant project this is
> some of projects and the user somehow accepts it in case of maven it's nearly
> every time you open a new project.

I don't think there's any real difference when you compare projects of similar complexity and maturity. Practically every IDE-managed Ant project "in the wild" I have opened that had more than one or two deps, i.e. a realistic serious project, had to be fixed up when opened because it had a bunch of missing dependencies.

The apparent difference is just that it is much harder to set up & distribute a large Ant-based project with a lot of dependencies, so there aren't so many of them to look at. You can opt to let NB check in libraries under ./lib/*.jar but I don't think this is the common case.

> the project is missing some useless jar and user accepts it (does not care
> about the badge and disables the dialog). Or the project is not complete (eg.
> unshared project) and user has to fix it - this is the same behavior as in
> maven case. But both of these cases are different for user point of view and
> cannot be distinguished from each other.

Both of these cases can apply equally to Ant- or Maven-based projects. It simply depends whether or not the dependency was actually required by the project. In a well-maintained project I would expect that all the dependencies are actively in use, but sometimes they are not. (mvn dependency:analyze
does make it easier to perform this maintenance on a Maven project, if you bother to run it.)

If there are some deps which are missing locally, but not needed anyway, you can fix the project metadata to not refer to them and then the IDE will scan and you can confirm that your change is correct. I am not sure whether or not it is acceptable to make the user wait until all declared deps are present before displaying compiler errors, but I think the question applies equally to any Java project type.

One algorithm which might be a reasonable compromise between performance and usability: if some classpath entries are missing, do not perform a scan just because the project was opened; but if a source file is opened, perform the scan at that time, since the user is apparently really starting to use the project and needs to see parse information, errors and all.
Comment 12 Jaroslav Tulach 2016-05-25 06:16:27 UTC
It's OK in 8.1, I'd say.