Bug 195300 - Maven repo generated without use of nexusIndexDirectory
Maven repo generated without use of nexusIndexDirectory
Status: VERIFIED FIXED
Product: www
Classification: Unclassified
Component: Builds & Repositories
7.0
All All
: P1 (vote)
: 7.0
Assigned To: pgebauer
nbre
http://mojo.codehaus.org/nbm-maven-pl...
: 70_HR_FIX
Depends on:
Blocks: 195301
  Show dependency treegraph
 
Reported: 2011-02-10 16:15 UTC by Jesse Glick
Modified: 2011-04-21 13:07 UTC (History)
5 users (show)

See Also:
Issue Type: DEFECT
:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jesse Glick 2011-02-10 16:15:41 UTC
http://bits.netbeans.org/netbeans/trunk/maven-snapshot/org/netbeans/external/ (and release repos starting with RELEASE69) fail to refer to preexisting public artifacts properly, instead creating copies. For example, org/netbeans/external/plexus-interpolation-1.14/ should not exist because org/codehaus/plexus/plexus-interpolation/1.14/ already exists in Central.
Comment 1 Jesse Glick 2011-02-10 16:16:26 UTC
Must not publish RELEASE70-BETA2 repository without fixing this.
Comment 2 pgebauer 2011-02-15 17:20:33 UTC
Any steps how to fix it?
Comment 3 pgebauer 2011-02-17 12:09:00 UTC
This issue appears for RELEASE69 as well so what does it happened more serious for RELEASE70-BETA2?
Comment 4 Jesse Glick 2011-02-17 15:31:55 UTC
(In reply to comment #2)
> Any steps how to fix it?

See URL which documents what you need to do.

(In reply to comment #3)
> This issue appears for RELEASE69 as well

And it was a major problem. Unfortunately it was only caught recently, and it is not good practice to retroactively change Maven release artifacts.

> [is it] more serious for RELEASE70-BETA2?

Yes, because now we may not legally distribute JUnit.
Comment 5 pgebauer 2011-02-19 00:24:24 UTC
I have checked both tasks trunk and 70-beta2 maven repository generation. Both tasks use the optional parameter nexusIndexDirectory with the correctly setup path to the unzipped Nexus Lucene index. The only suspicious think was exception "org.apache.lucene.index.CorruptIndexException: Unknown format version: -7" in the log for the 70-beta2 maven repository. The exception has disappeared after the Nexus Lucene index replacement by the newest version. In spite of that copies of already existing artifacts are still created. It seems to me like a problem in the nbm:populate-repository plugin itself.
Comment 6 Antonin Nebuzelsky 2011-02-22 12:50:25 UTC
Is a fix in nbm:populate-repository plugin needed from Jesse?

Or does it mean that you are ready to populate repository now?
Comment 7 pgebauer 2011-02-22 12:59:13 UTC
Yes, IHMO a fix is needed in the plugin. I think, BE doesn't own the plugin source code.
Comment 8 Antonin Nebuzelsky 2011-02-23 10:27:40 UTC
Jesse, please address the problem ASAP. Thanks.
Comment 9 Jesse Glick 2011-02-25 00:00:09 UTC
Working for me. I ran the 3.5 repo generator with the nexusIndexDirectory param set to a fresh download of the documented ZIP, on some clusters including java and did not see org/netbeans/external:plexus-interpolation-1.14; ~/.m2/repository/org/netbeans/modules/org-netbeans-modules-maven-embedder/nbdev/org-netbeans-modules-maven-embedder-nbdev.pom referred to org.codehaus.plexus:plexus-interpolation:1.14 as expected.

(There were more org.netbeans.external entries than there really should be, but these appear to be genuine problems with our modules. For example, ~/.m2/repository/org/netbeans/external/xerces-2.8.0/nbdev/xerces-2.8.0-nbdev.jar is the same length but somehow different from xerces/xercesImpl-2.8.0.jar as found in both Central and the official Apache download site, meaning jlahoda's c7ff2776771d is somehow suspect.)

For debugging purposes, PopulateRepositoryMojo.java:746-787 is the relevant code.
Comment 10 pgebauer 2011-02-28 13:09:24 UTC
I have tried to run the maven repo generation in my own directory and the result is the same as in the official build. I can still see following files after generation:

~/maven-snapshot/org/netbeans/external/plexus-interpolation-1.14maven-snapshot/org/netbeans/external/plexus-interpolation-1.14/SNAPSHOT/plexus-interpolation-1.14-20110228.125137-1.jar.md5
~/maven-snapshot/org/netbeans/external/plexus-interpolation-1.14/SNAPSHOT/plexus-interpolation-1.14-20110228.125137-1.pom
~/maven-snapshot/org/netbeans/external/plexus-interpolation-1.14/SNAPSHOT/plexus-interpolation-1.14-20110228.125137-1.jar.sha1
~/maven-snapshot/org/netbeans/external/plexus-interpolation-1.14/SNAPSHOT/plexus-interpolation-1.14-20110228.125137-1.pom.sha1
~/maven-snapshot/org/netbeans/external/plexus-interpolation-1.14/SNAPSHOT/plexus-interpolation-1.14-20110228.125137-1.jar
~/maven-snapshot/org/netbeans/external/plexus-interpolation-1.14/SNAPSHOT/plexus-interpolation-1.14-20110228.125137-1.pom.md5

The official build uses the following command to generate the repo:

+ /space/apache-maven-2.2.1/bin/mvn -DdeployUrl=file://maven-snapshot -DforcedVersion=SNAPSHOT -DnetbeansInstallDirectory=nbrepo/netbeans -DnetbeansJavadocDirectory=nbrepo/javadoc -DnetbeansNbmDirectory=nbrepo/nbms -DnetbeansSourcesDirectory=nbrepo/source-zips -DnexusIndexDirectory=nbrepo/nexus/ -DskipInstall=true org.codehaus.mojo:nbm-maven-plugin:3.5:populate-repository --settings settings.xml

Does anybody view any suspicious setting there?
Comment 11 Jesse Glick 2011-02-28 18:05:31 UTC
Use Maven 3.0.x, not 2.2.1 please. Other than that I don't see anything amiss in the command line; see what is happening in PopulateRepositoryMojo.java.
Comment 12 pgebauer 2011-03-01 16:51:38 UTC
The result with Maven 3.0.2 is the same as before. A debugging will be needed very likely.
Does anybody know where is the source code for PopulateRepositoryMojo?
Comment 13 pgebauer 2011-03-01 17:17:34 UTC
Just one thing before debugging. Jesse, could you please provide setting.xml files (both from maven dir and user dir) you use during successful maven repo generation? Thank you.
Comment 14 pgebauer 2011-03-07 12:04:15 UTC
During the repository generation by nbm-maven-plugin, an SHA1 string is generated for the external libraries and the class org.apache.lucene.search.IndexSearcher tries to find it out. No record is found for some libraries like plexus-interpolation etc. However some entries for example ext/jaxws22/gmball-api-only.jar or ext/groovy-all.jar are found. Those libraries are not part of the maven repository directory "external" as it is expected.

Any ideas where I could continue with investigation?
Comment 15 Jesse Glick 2011-03-07 15:19:13 UTC
(In reply to comment #14)
> No record is found for some libraries like plexus-interpolation

Do the JARs on local disk precisely match those found in Maven Central? Are you sure you are using a recent index of Central?
Comment 17 Jesse Glick 2011-03-07 17:19:04 UTC
(In reply to comment #16)
> https://nexus.codehaus.org/content/groups/public/.index/nexus-maven-repository-index.zip

As specified in the mojo docs (see BZ URL), use http://repo1.maven.org/maven2/.index/nexus-maven-repository-index.zip for the official Maven Central repository index.
Comment 18 pgebauer 2011-03-07 18:45:23 UTC
The file http://repo1.maven.org/maven2/.index/nexus-maven-repository-index.zip is not accessible. Does any mirror exist?
Comment 19 Jesse Glick 2011-03-07 19:14:42 UTC
(In reply to comment #18)
> http://repo1.maven.org/maven2/.index/nexus-maven-repository-index.zip
> is not accessible.

Depends on the HTTP client. They block access for web crawlers, but last I checked you can download it from Firefox. http://repo1.maven.org/maven2/.meta/repository-metadata.xml also shows some mirrors available; http://uk.maven.org/maven2 is official.
Comment 20 pgebauer 2011-03-08 12:06:50 UTC
> Depends on the HTTP client. They block access for web crawlers, but last I
> checked you can download it from Firefox.

I have tried both Firefox in Windows XP and Safari in Mac OS X but the URL http://repo1.maven.org/maven2/.index/nexus-maven-repository-index.zip is forbidden.

After playing with mirrors, I have find out the following URL http://mirrors.ibiblio.org/pub/mirrors/maven2/dot-index/ so I have used the file http://mirrors.ibiblio.org/pub/mirrors/maven2/dot-index/nexus-maven-repository-index.zip but I'm still not sure if this is a correct one.

Anyway, the  build result of the last maven NetBeans repository with the mentioned index file is available at http://bits.netbeans.org/download/trunk/maven-snapshot/ . It looks good from my point of view but somebody should check or test it.
Comment 21 Jesse Glick 2011-03-08 13:06:45 UTC
Well it's got org.codehaus.plexus:plexus-interpolation:1.14 now so that's good. Referring to org.netbeans.external:maven-repository-metadata-3.0.3:SNAPSHOT but that is likely because the 3.0.3 release was very recent and is not yet mentioned in the downloadable index (updated weekly AFAIK). So I think it's working correctly.

Bug #195301 still open.
Comment 22 pgebauer 2011-03-25 14:54:46 UTC
The issue is fixed and the maven repository with 70-beta2 artifacts has been released.
Comment 23 escay 2011-04-21 11:57:35 UTC
Verified fixed. RELEASE70 is correctly available from the maven repo. Building our project with RELEASE70 works correctly.
Thanks for the great work!
Comment 24 Jesse Glick 2011-04-21 13:07:16 UTC
I think escay was verifying something else. But anyway I can confirm that this looks fixed; there are inappropriate entries from org.netbeans.external but they are either (1) coming from earlier releases before this fix; (2) actually due to NB modules using nonstandard copies of common artifacts (such as Xerces).


By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo