This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 203222

Summary: global maven SourceForBinaryQuery impl needed - after copying projects, old locations considered dependency sources
Product: projects Reporter: monezz <monezz>
Component: MavenAssignee: Milos Kleint <mkleint>
Status: RESOLVED FIXED    
Severity: normal CC: anebuzelsky, borre, gtzabari, jbentonor, kalle1, markiewb, mkleint, mmirilovic, wadechandler
Priority: P3    
Version: 7.1   
Hardware: All   
OS: All   
Issue Type: DEFECT Exception Reporter:
Bug Depends on: 210465    
Bug Blocks:    
Attachments: experimental change that issues one SFBQ.Result per entry in local repository
Testcase in zip file
obsoleted testcase is not for this bug

Description monezz 2011-10-06 11:55:45 UTC
[ JDK VERSION : 1.6.27 ]

After closing, moving and opening a netbeans rcp project, double modules appear
in Required Projects.

Steps to reproduce
- create a new maven netbeans RCP application (RELEASE71-BETA)
- close the 3 projects (mavenproject1-app, -branding and -parent)
- copy the project directory to another parent director outside the IDE
( $cp -r dir1/mavenproject1 dir2/ )
- open Project and navigate to dir2
- select mavenproject1
- check Open Required Projects

Expected result
- required projects are: 
mavenproject1-app (application)
mavenproject1-branding (branding)

Actual result:
- required projects are: 
mavenproject1-app (application)
mavenproject1-branding (/full/path/to/dir1/branding)
mavenproject1-branding (branding)

On actual opening of the project, 2 branding modules are opened.

Workaround:
- manually open required project
- use Open Required Projects and close wrong ones.

Both workarounds are doable, but can be quite cumbersome on large projects.
Comment 1 Jesse Glick 2011-10-25 23:23:37 UTC
When you opened (or created) the initial projects, the IDE remembered these locations as the sources corresponding to the (presumably SNAPSHOT) artifacts in your local repository. Such an association of artifacts to nonopen source projects is used for many purposes.

Open Required Projects for Maven projects will collect transitive dependencies of two types:

1. <module>s of aggregator POMs, which in this case (starting from the new parent project) will include the new submodules.

2. <dependency>s of any projects, where known source projects exist.

For #2, the currently known source for the branding artifact (as declared as a dependency in the nbm-application project) is the old copy. This association is updated to point to the new copy as soon as the new aggregator is opened - but that happens _after_ the list of required projects is collected by the Open Project dialog.

Not sure how this can be fixed without breaking something else. The deeper issue in Maven is that just copying an aggregated tree of projects can cause problems since there is only one local repository (typically) and you have not changed the coordinates of your projects. (Consider that 'mvn -f old/branding/pom.xml install && mvn -f new/app/pom.xml install nbm:run-platform' will build and run the new copy of the application against the old copy of the branding.)

Simplest workaround would be to not use the Open Required Projects checkbox in the Open Project dialog; open the aggregator POM first, then open its submodules.
Comment 2 monezz 2011-10-26 13:12:05 UTC
Relative path dependencies following maven naming/sub mapping guidelines should take precedence over cache. At the moment path dependencies are combined with cached dependencies. Wouldn't it be possible to filter the required projects list on duplicates and if both a cached entry as well as a relative path depenendency are present, only include the latter?

Is it possible to clear the cache manually?

If I remember correctly we also had problems with references between modules.
A1 depends on B1
After copy A2 depends on B2 but control clicking a B2 class in A2 lead to the B1 module instead of the B2 module.
This scenario is a bit more complex to reproduce, I will make an attempt later.
Comment 3 Jesse Glick 2011-11-08 21:16:28 UTC
(In reply to comment #2)
> Relative path dependencies following maven naming/sub mapping guidelines should
> take precedence over cache.

Agreed, just a question of how to achieve that efficiently. Do not want to walk a possibly huge reactor tree when loading a submodule only to check that owners of dependencies are in the same reactor. Currently the scan for owners is done only when the root POM is opened.

A crude improvement would be to consider two submodules to be in the same reactor so long as there is some common parent directory with a pom.xml such that all intermediate parent directories of both modules also contain a pom.xml. Technically this would be wrong in a variety of cases (<parentPath>../parent/pom.xml</>, <pP>../../pom.xml</pP>, unrelated projects in parent directories, etc. etc.) but might work well enough for most cases.

> Wouldn't it be possible to filter the required projects
> list on duplicates and if both a cached entry as well as a relative path
> dependency are present, only include the latter?

Yes, though this would only solve this particular symptom and not others.

> Is it possible to clear the cache manually?

rm $userdir/config/Preferences/org/netbeans/modules/maven/externalOwners.properties

> A1 depends on B1
> After copy A2 depends on B2 but control clicking a B2 class in A2 lead to the
> B1 module instead of the B2 module.

Right, again because A2 only actually refers to "something:B:1.0-SNAPSHOT", not to "../B/src/main/java". And again the IDE is actually exposing a problem latent in Maven: mvn -f b1/pom.xml install && mvn -f a2/pom.xml compile will compile A2 against B1.

Even with a possible fix such as what is outlined above, you would not get the desired behavior when all four projects were opened at once, since ~/.m2/repository/something/B/1.0-SNAPSHOT/B-1.0-SNAPSHOT.jar will be in the classpath of both A1/src/main/java/ and A2/src/main/java/, and the IDE expects there to be a single source mapping for a given binary JAR. Things could probably be made to work if all the old projects were closed, though strange cache effects could remain if the old projects had been open in the same IDE session. Specifying the in-IDE classpath of A2/src/main/java/ as B2/target/classes/ might suffice to fix that.
Comment 4 Jesse Glick 2011-11-21 17:00:19 UTC
*** Bug 205234 has been marked as a duplicate of this bug. ***
Comment 5 Jesse Glick 2012-03-06 16:47:47 UTC
Discussed this with mkleint and came up with an outline of changes that in aggregate would solve this and similar problems more cleanly (would need to be filed as separate issues):

1. Distinguish dependency projects from submodules.

1a. Remove all GUI from Open Project dialog referring to "subprojects". If you want to open them, open the root first, then use the context menu later.

1b. Deprecate SubprojectProvider, whose current semantics is unusably vague, and introduce two separate SPIs (or modes of a single SPI): list of project dependencies (ReferenceHelper for Ant-based projects, <dependency>s mapped to projects for Maven); and list of submodules (<modules> for Maven, also idiosyncratic impls for Ant-based NBM suites and EARs). May be tricky since it seems there are a couple dozen places where foreign code (i.e. outside the project type module) is calling SubprojectProvider with some sort of expectation of semantics. Might need to do some kind of compatibility work.

1c. Refine existing context menu item "Open Required Projects" into two actions corresponding to the two new SPIs. Decide for each kind of project which of these actions to display (or neither, or both).

1d. Either remove or refine the "Project and All Required Projects" option in the "Create New Group" dialog.

2. Make MavenForBinaryQueryImpl.BinResult listen to relevant new registrations in MavenFileOwnerQuery, and (re)fire a change in response, so that the Java indexer understands that a repository JAR should now be associated with a different source project than before.

3. Investigate making MFOQ update upon project open even "cousins" in the same reactor tree, somehow avoiding redundant work when a bunch of modules are opened at once. Currently it only updates the opened project and submodules, but it can be useful to go up the tree via <parentPath> too, if performance can be made acceptable.

Too big a change for 7.2 at this point.
Comment 6 Jesse Glick 2012-03-06 16:58:09 UTC
A possible smaller fix for the particular symptoms of this issue would be to refine SubprojectProviderImpl.addKnownOwners: if SubprojectProviderImpl.project is itself not yet open, call ProjectOpenedHookImpl.registerWithSubmodules on in this project in case it is a reactor root.

A performance problem with that fix is that it will be called again if the OK button is clicked. Could perhaps optimize out the double call somehow, e.g. remember the last call to registerWithSubmodules and skip if it is a duplicate. (Does not suffice to just remember the last File basedir, but also the last Set<File> registered, since SubprojectProviderImpl will also be called on submodules recursively; need to "remember" that the whole tree was already checked.)
Comment 7 Jesse Glick 2012-05-24 02:56:04 UTC
*** Bug 212896 has been marked as a duplicate of this bug. ***
Comment 8 Milos Kleint 2012-05-25 14:52:35 UTC
Created attachment 119891 [details]
experimental change that issues one SFBQ.Result per entry in local repository
Comment 9 Milos Kleint 2012-05-25 15:01:28 UTC
(In reply to comment #8)
> Created attachment 119891 [details]
> experimental change that issues one SFBQ.Result per entry in local repository

this patch makes sure that a single Result instance is issued for a given local repository URL and it updates based on what project owns the URL or whether a local source jar is present.
does so by creating a new global SFBQI. The project's SFBQI now only responsible for target/classes folders handling..

It's not 100% workable, will need to show it to someone from java team, while I considered the returned roots correct, the editor didn't update properly. In the following scenario:
1. open maven-settings project and maven-settings-builder from maven3 sourcebase.
2. create a compilation problem in maven-settings, something that is used in builder as well..
3. change version of maven-settings to 3.0.11-SNAPSHOT or anything.  The maven-settings-builder source file should get "fixed" because it now relies on the repo binary only, but I've seen various errors where file badge was out of sync with editor content etc..


speed performance of the solution needs revisiting as well as memory performance (leaks)
especially deciding when to fire changes and when to update the current owner project are tricky..

with the current state I'm not convinced it's safe to integrate to 7.2 at current point in time..
Comment 10 Milos Kleint 2012-09-17 11:52:25 UTC
http://hg.netbeans.org/core-main/rev/07b6bad59092 and
http://hg.netbeans.org/core-main/rev/a390f6b64402

hangle the switching of SourceForBinary and javadocForBinary results from one project to the other and from local repository only backed -sources.jar to a project. because of the nature of current MavenFileOwnerQuery preferences persistence, switching from project to non-project -sources.jar is impossible in most scenarios.

looking at the subproject provider once again, I believe we can keep just the Subprojects == maven modules variant and the code that relies on subproject == dependency can be rewritten. So far I've found just 2 or 3 places like that in maven support alone.
Comment 11 Milos Kleint 2012-09-17 12:35:13 UTC
(In reply to comment #10)
> 
> looking at the subproject provider once again, I believe we can keep just the
> Subprojects == maven modules variant and the code that relies on subproject ==
> dependency can be rewritten. So far I've found just 2 or 3 places like that in
> maven support alone.

well, there also a few usages in profiler and elsewhere. It's unclear which usecase it depends on and what the expected semantics is. It appears to me that SubprojectProvider was used as a shortcut for getting Project instances on classpath.
Comment 12 Milos Kleint 2012-09-17 12:36:08 UTC
see issue 210465 for a complete list of subprojectProvider usages
Comment 13 Quality Engineering 2012-09-18 02:12:47 UTC
Integrated into 'main-golden', will be available in build *201209180001* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main-golden/rev/07b6bad59092
User: Milos Kleint <mkleint@netbeans.org>
Log: #203222 a global Source/javadoc for binary query implementation that either delegates to a project or just to repository content. handles the query for repository artifact only.
Comment 14 bht 2012-12-01 19:07:18 UTC
Created attachment 128695 [details]
Testcase in zip file
Comment 15 bht 2012-12-01 19:12:02 UTC
Created attachment 128697 [details]
obsoleted testcase is not for this bug
Comment 16 Milos Kleint 2013-01-14 10:17:28 UTC
*** Bug 194280 has been marked as a duplicate of this bug. ***
Comment 17 Milos Kleint 2013-10-03 20:08:00 UTC
*** Bug 236721 has been marked as a duplicate of this bug. ***
Comment 18 Milos Kleint 2013-11-12 06:35:59 UTC
*** Bug 238256 has been marked as a duplicate of this bug. ***
Comment 19 Milos Kleint 2013-11-12 06:42:47 UTC
*** Bug 238256 has been marked as a duplicate of this bug. ***
Comment 20 mjreged 2013-11-18 14:27:00 UTC
Hello Milos Kleint, I just wanted to confirm that this issue is very important. Consider this following usecase which is a real one in our company:

Company hosts a large multi-module maven Web application in Subversion.
*Trunk is the latest development for next release.
*Feature branches are long running project initiatives.
*Developers switch back between feature vs trunk line of code frequently.
*Lots of feature branches.
*Branched off maven projects do not have their project.version changed since it may be merged back to trunk and it will just cause local conflicts.
*User decides to close trunk maven projects and opens a feature branch project.
*User selects the Parent Folder and opts-in for "Open Required Project"
*User sees duplicate projects open. The branch and the trunk paths.
*User is not aware of the issue and continue to develop in the wrong branch.
*User coplains that his project no longer compiles.

Hope this usecase adds some more bite to the issue.  Thanks for all you work.
Comment 21 Milos Kleint 2013-11-20 13:44:42 UTC
http://hg.netbeans.org/core-main/rev/5c1f831c61fe