Bug 195041 - Permit external references to NBM contents
Permit external references to NBM contents
Status: RESOLVED FIXED
Product: platform
Classification: Unclassified
Component: Autoupdate
7.0
All All
: P2 (vote)
: 7.0
Assigned To: Jesse Glick
au-issues
junitlib
: API, API_REVIEW_FAST
Depends on: 195368
Blocks: 195123
  Show dependency treegraph
 
Reported: 2011-02-03 16:53 UTC by Jesse Glick
Modified: 2011-03-25 18:32 UTC (History)
6 users (show)

See Also:
Issue Type: DEFECT
:


Attachments
Support for file.external with URL: http://.... (31.31 KB, patch)
2011-02-04 23:08 UTC, Jaroslav Tulach
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Jesse Glick 2011-02-03 16:53:08 UTC
For changes such as 3b24c68e8499 it would be cleanest to distribute a placeholder NBM which, when installed, downloads its external dependencies. The NBM format supports a "main" dir for an external installer, but this is very procedural and is anyway not supported by org.netbeans.nbbuild.AutoUpdate. Would prefer a simple declarative way of specifying that certain NBM fileset entries should be retrieved from external sources.

For the binary format, suggest that e.g.

 netbeans/modules/ext/junit-4.8.2.jar

be removed and in its place a text file added to the NBM

 external/modules/ext/junit-4.8.2.jar

with contents such as

---%<---
C94F54227B08100974C36170DCB53329435FE5AD
file:${user.home}/.m2/repository/junit/junit/4.8.2/junit-4.8.2.jar
http://repo1.maven.org/maven2/junit/junit/4.8.2/junit-4.8.2.jar
---%<---

The first line would be an expected SHA-1 sum (so that any signature on the NBM is really meaningful); subsequent lines would be URLs to try, in order, to retrieve the file contents from. Here I am assuming that system properties can be substituted. (The URL is not quite correct on Windows, but I hope it would work anyway.) If the file can be retrieved from one of the mentioned URLs, with the correct hash, it would be installed in the usual place. Otherwise the unpack operation would fail.

Both org.netbeans.updater.ModuleUpdater and org.netbeans.nbbuild.AutoUpdate would need to handle this.

No Pack200 support for external files is needed, I think.

There would also need to be some way to define such external file configuration declaratively in a module source project that <makenbm> would understand. The output cluster would have the real files (needed for testing of various kinds), with update_tracking/*.xml listing those files, but the external references would be packed into the final *.nbm in their place.

With such a feature, 3b24c68e8499 could be simplified considerably - libs.junit4 and junitlib could be published as NBMs in the stableuc cluster config from nb.org sources, and a LibraryDefiner could be created which uses AU to associate these modules with the junit & junit_4 libraries (possibly written as a generic feature, possibly just implemented directly in the junit module).

One drawback of a generic URL list compared to the current impl in maven.junit is that if junit.jar is not already in your local Maven repository, the raw download from Maven Central would not cache it there; nor would the URL automatically redirect to a local Nexus mirror according to your Maven settings. It may be tricky to use the Maven embedder to handle this kind of download when available; even using an optional SPI with an implementation in o.n.m.maven could be problematic since updater.jar is run outside the NB module system, so the download of external contents would have to happen before the updater is invoked. If such an SPI is a possibility, then you could register e.g.

C94F54227B08100974C36170DCB53329435FE5AD
m2:/junit/junit/4.8.2/junit-4.8.2.jar
http://repo1.maven.org/maven2/junit/junit/4.8.2/junit-4.8.2.jar

where the first URL would be handled by the Maven impl using the embedder and work nicely, but if the Maven impl were unavailable (e.g. if run from a non-Java IDE, or via the Ant task) then it would simply throw a MalformedURLException and would be skipped in favor of the regular HTTP download.
Comment 1 Jaroslav Tulach 2011-02-04 18:11:17 UTC
>  netbeans/modules/ext/junit-4.8.2.jar
> 
> be removed and in its place a text file added to the NBM

Actually the original NBM contains:
netbeans/modules/ext/junit-4.5.jar.pack.gz
e.g. the system is using additional extensions to describe the type of content. Thus...

>  external/modules/ext/junit-4.8.2.jar

...I'd rather name the file
modules/ext/junit-4.8.2.jar.external
that would be more consistent with the pack.gz naming.

> Both org.netbeans.updater.ModuleUpdater and org.netbeans.nbbuild.AutoUpdate
> would need to handle this.

OK. I'll try to implement that. 

> There would also need to be some way to define such external file configuration
> declaratively in a module source project that <makenbm> would understand. The
> output cluster would have the real files (needed for testing of various kinds),
> with update_tracking/*.xml listing those files, but the external references
> would be packed into the final *.nbm in their place.

You should probably deal with the creation of the NBM part. As that spans into apisupport where my knowledge is quite weak.

> C94F54227B08100974C36170DCB53329435FE5AD
> m2:/junit/junit/4.8.2/junit-4.8.2.jar
> http://repo1.maven.org/maven2/junit/junit/4.8.2/junit-4.8.2.jar

Having "m2" URL was on my mind too, but that would require that the download of the external JAR is handled by autoupdate.services and not updater.jar itself...
Comment 2 Jesse Glick 2011-02-04 21:34:31 UTC
(In reply to comment #1)
> I'd rather name the file
> [netbeans/]modules/ext/junit-4.8.2.jar.external
> that would be more consistent with the pack.gz naming.

Makes sense.

> You should probably deal with the creation of the NBM part.

OK. I can create a branch for this if you have not done so already.

> Having "m2" URL was on my mind too, but that would require that the download of
> the external JAR is handled by autoupdate.services and not updater.jar
> itself...

Right, as I wrote initially. Maybe it could write the external files to $userdir/update/downloads/, named acc. to hash. Then updater.jar would just look for them there.
Comment 3 Jaroslav Tulach 2011-02-04 23:08:59 UTC
Created attachment 105668 [details]
Support for file.external with URL: http://....

Rather than using SHA1, I suggest to use CRC (a line prefixed with CRC: in the .external file). It is computed anyway and thus it is easy to compare it and check whether it is correct. I tried to support the ${...} properties. Whether we will support m2:// protocol is for some future decision.
Comment 4 Jesse Glick 2011-02-07 20:21:10 UTC
I took the liberty of creating a branch: https://hg.netbeans.org/core-main/rev/nbm-external-195041
Comment 5 Jesse Glick 2011-02-07 20:23:14 UTC
(In reply to comment #3)
> Rather than using SHA1, I suggest to use CRC

Probably OK. One drawback that I alluded to earlier is that it weakens the NBM signature: it is theoretically possible to point to a URL whose content is changed without your knowledge. But I don't suppose this is a serious concern.
Comment 6 Jesse Glick 2011-02-07 22:47:06 UTC
I have done some work on the branch so that the JUnit libraries are now built normally and published to AU, but the NBMs contain just external references. There are still some things broken in ModuleUpdater which I have noted in comments; did not test <autoupdate>. Did not yet attempt to define an SPI to let Maven URLs be resolved.
Comment 7 Jaroslav Tulach 2011-02-08 13:00:53 UTC
I fixed few XXX and now the change shall be ready for integration (although it misses the specific maven protocol).
Comment 8 Jesse Glick 2011-02-08 18:02:03 UTC
I did some more work, including using the CRCs for JUnit download.

The major thing missing at this point is that *.external are resolved during the install phase. This means that if there is some routine problem with *.external - bad CRC, computer offline, etc. - some error messages are printed to console but the module can be partly installed and the wizard just says that there were problems installing it. This is not adequately robust for FCS, I think. *.external references really need to be resolved during the NBM download phase (which would also make it possible to inject a Maven protocol handler). For <autoupdate> this probably doesn't matter - if there is a problem there will be a BuildException and that is that (although cleaning up half-installed modules would be nice).
Comment 9 Jesse Glick 2011-02-08 18:12:45 UTC
(In reply to comment #8)
> *.external references need to be resolved during the NBM download phase

I guess this would mean that InstallSupportImpl.doDownload would need to scan dest for netbeans/**/*.external entries, copy(...) them using the same progress handle (need to expand total size) to some appropriate location such as $cluster/download/$crc (verifying the CRC in the process), and then ModuleUpdater.unpack would need to look for $nbm/../$crc upon encountering an external reference rather than trying to download the file itself.

Adding Maven protocol support might not even require a special SPI; might suffice to just use @URLStreamHandlerRegistration(protocol="m2") in maven.embedder. When not present, this would simply throw MalformedURLException which would be caught and that URL skipped.
Comment 10 Jesse Glick 2011-02-09 14:50:15 UTC
*** Bug 195190 has been marked as a duplicate of this bug. ***
Comment 11 Jesse Glick 2011-02-09 18:41:11 UTC
*** Bug 195122 has been marked as a duplicate of this bug. ***
Comment 12 Petr Jiricka 2011-02-09 20:48:42 UTC
Duplicate issue 195122 is a defect, so this needs to be a defect too.
Comment 13 Jaroslav Tulach 2011-02-10 16:54:11 UTC
Done (without a progress): http://hg.netbeans.org/core-main/rev/0fe49ee2823e

Now you can implement the m2 protocol, possibly update the documentation somewhere and we are ready for integration, I guess.
Comment 14 Jesse Glick 2011-02-10 20:59:50 UTC
core-main #9d76a9c03088
Comment 15 Jesse Glick 2011-02-11 17:05:30 UTC
In trunk now. Note that the new JUnitLibraryDownloader will not work until the first stable nbms-and-javadoc build containing this change, since it relies on libs.junit4 and junitlib being available on the update center.
Comment 16 Jesse Glick 2011-02-11 22:22:42 UTC
(In reply to comment #15)
> until the first stable nbms-and-javadoc build containing this change

Which was #6634, so it should now work, though if you have an existing user dir AU might not reload the update center before the "Check Interval" expires unless you force it to "Reload Catalog".
Comment 17 Jiri Rechtacek 2011-02-12 07:22:53 UTC
> (In reply to comment #15)
>> until the first stable nbms-and-javadoc build containing this change
> 
> Which was #6634, so it should now work, though if you have an existing user dir
> AU might not reload the update center before the "Check Interval" expires
> unless you force it to "Reload Catalog".
Or invoke _Help|Check for Updates_, it also reload the update center.
Comment 18 Quality Engineering 2011-02-16 11:44:23 UTC
Integrated into 'main-golden', will be available in build *201102160501* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/9d76a9c03088
User: Jesse Glick <jglick@netbeans.org>
Log: #195041: permit external references to NBM contents.
Comment 19 Quality Engineering 2011-02-17 10:59:15 UTC
Integrated into 'main-golden', will be available in build *201102170501* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/64e3953b3016
User: Jesse Glick <jglick@netbeans.org>
Log: #195041 refinement: have to force refresh of update providers for this to work on first start.
Otherwise getUpdateUnits silently returns nothing without making any attempt to download the lists.
Comment 20 Jesse Glick 2011-03-25 18:32:24 UTC
Needs support in the Maven mojo: http://jira.codehaus.org/browse/MNBMODULE-138


By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo