This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
For changes such as 3b24c68e8499 it would be cleanest to distribute a placeholder NBM which, when installed, downloads its external dependencies. The NBM format supports a "main" dir for an external installer, but this is very procedural and is anyway not supported by org.netbeans.nbbuild.AutoUpdate. Would prefer a simple declarative way of specifying that certain NBM fileset entries should be retrieved from external sources. For the binary format, suggest that e.g. netbeans/modules/ext/junit-4.8.2.jar be removed and in its place a text file added to the NBM external/modules/ext/junit-4.8.2.jar with contents such as ---%<--- C94F54227B08100974C36170DCB53329435FE5AD file:${user.home}/.m2/repository/junit/junit/4.8.2/junit-4.8.2.jar http://repo1.maven.org/maven2/junit/junit/4.8.2/junit-4.8.2.jar ---%<--- The first line would be an expected SHA-1 sum (so that any signature on the NBM is really meaningful); subsequent lines would be URLs to try, in order, to retrieve the file contents from. Here I am assuming that system properties can be substituted. (The URL is not quite correct on Windows, but I hope it would work anyway.) If the file can be retrieved from one of the mentioned URLs, with the correct hash, it would be installed in the usual place. Otherwise the unpack operation would fail. Both org.netbeans.updater.ModuleUpdater and org.netbeans.nbbuild.AutoUpdate would need to handle this. No Pack200 support for external files is needed, I think. There would also need to be some way to define such external file configuration declaratively in a module source project that <makenbm> would understand. The output cluster would have the real files (needed for testing of various kinds), with update_tracking/*.xml listing those files, but the external references would be packed into the final *.nbm in their place. With such a feature, 3b24c68e8499 could be simplified considerably - libs.junit4 and junitlib could be published as NBMs in the stableuc cluster config from nb.org sources, and a LibraryDefiner could be created which uses AU to associate these modules with the junit & junit_4 libraries (possibly written as a generic feature, possibly just implemented directly in the junit module). One drawback of a generic URL list compared to the current impl in maven.junit is that if junit.jar is not already in your local Maven repository, the raw download from Maven Central would not cache it there; nor would the URL automatically redirect to a local Nexus mirror according to your Maven settings. It may be tricky to use the Maven embedder to handle this kind of download when available; even using an optional SPI with an implementation in o.n.m.maven could be problematic since updater.jar is run outside the NB module system, so the download of external contents would have to happen before the updater is invoked. If such an SPI is a possibility, then you could register e.g. C94F54227B08100974C36170DCB53329435FE5AD m2:/junit/junit/4.8.2/junit-4.8.2.jar http://repo1.maven.org/maven2/junit/junit/4.8.2/junit-4.8.2.jar where the first URL would be handled by the Maven impl using the embedder and work nicely, but if the Maven impl were unavailable (e.g. if run from a non-Java IDE, or via the Ant task) then it would simply throw a MalformedURLException and would be skipped in favor of the regular HTTP download.
> netbeans/modules/ext/junit-4.8.2.jar > > be removed and in its place a text file added to the NBM Actually the original NBM contains: netbeans/modules/ext/junit-4.5.jar.pack.gz e.g. the system is using additional extensions to describe the type of content. Thus... > external/modules/ext/junit-4.8.2.jar ...I'd rather name the file modules/ext/junit-4.8.2.jar.external that would be more consistent with the pack.gz naming. > Both org.netbeans.updater.ModuleUpdater and org.netbeans.nbbuild.AutoUpdate > would need to handle this. OK. I'll try to implement that. > There would also need to be some way to define such external file configuration > declaratively in a module source project that <makenbm> would understand. The > output cluster would have the real files (needed for testing of various kinds), > with update_tracking/*.xml listing those files, but the external references > would be packed into the final *.nbm in their place. You should probably deal with the creation of the NBM part. As that spans into apisupport where my knowledge is quite weak. > C94F54227B08100974C36170DCB53329435FE5AD > m2:/junit/junit/4.8.2/junit-4.8.2.jar > http://repo1.maven.org/maven2/junit/junit/4.8.2/junit-4.8.2.jar Having "m2" URL was on my mind too, but that would require that the download of the external JAR is handled by autoupdate.services and not updater.jar itself...
(In reply to comment #1) > I'd rather name the file > [netbeans/]modules/ext/junit-4.8.2.jar.external > that would be more consistent with the pack.gz naming. Makes sense. > You should probably deal with the creation of the NBM part. OK. I can create a branch for this if you have not done so already. > Having "m2" URL was on my mind too, but that would require that the download of > the external JAR is handled by autoupdate.services and not updater.jar > itself... Right, as I wrote initially. Maybe it could write the external files to $userdir/update/downloads/, named acc. to hash. Then updater.jar would just look for them there.
Created attachment 105668 [details] Support for file.external with URL: http://.... Rather than using SHA1, I suggest to use CRC (a line prefixed with CRC: in the .external file). It is computed anyway and thus it is easy to compare it and check whether it is correct. I tried to support the ${...} properties. Whether we will support m2:// protocol is for some future decision.
I took the liberty of creating a branch: https://hg.netbeans.org/core-main/rev/nbm-external-195041
(In reply to comment #3) > Rather than using SHA1, I suggest to use CRC Probably OK. One drawback that I alluded to earlier is that it weakens the NBM signature: it is theoretically possible to point to a URL whose content is changed without your knowledge. But I don't suppose this is a serious concern.
I have done some work on the branch so that the JUnit libraries are now built normally and published to AU, but the NBMs contain just external references. There are still some things broken in ModuleUpdater which I have noted in comments; did not test <autoupdate>. Did not yet attempt to define an SPI to let Maven URLs be resolved.
I fixed few XXX and now the change shall be ready for integration (although it misses the specific maven protocol).
I did some more work, including using the CRCs for JUnit download. The major thing missing at this point is that *.external are resolved during the install phase. This means that if there is some routine problem with *.external - bad CRC, computer offline, etc. - some error messages are printed to console but the module can be partly installed and the wizard just says that there were problems installing it. This is not adequately robust for FCS, I think. *.external references really need to be resolved during the NBM download phase (which would also make it possible to inject a Maven protocol handler). For <autoupdate> this probably doesn't matter - if there is a problem there will be a BuildException and that is that (although cleaning up half-installed modules would be nice).
(In reply to comment #8) > *.external references need to be resolved during the NBM download phase I guess this would mean that InstallSupportImpl.doDownload would need to scan dest for netbeans/**/*.external entries, copy(...) them using the same progress handle (need to expand total size) to some appropriate location such as $cluster/download/$crc (verifying the CRC in the process), and then ModuleUpdater.unpack would need to look for $nbm/../$crc upon encountering an external reference rather than trying to download the file itself. Adding Maven protocol support might not even require a special SPI; might suffice to just use @URLStreamHandlerRegistration(protocol="m2") in maven.embedder. When not present, this would simply throw MalformedURLException which would be caught and that URL skipped.
*** Bug 195190 has been marked as a duplicate of this bug. ***
*** Bug 195122 has been marked as a duplicate of this bug. ***
Duplicate issue 195122 is a defect, so this needs to be a defect too.
Done (without a progress): http://hg.netbeans.org/core-main/rev/0fe49ee2823e Now you can implement the m2 protocol, possibly update the documentation somewhere and we are ready for integration, I guess.
core-main #9d76a9c03088
In trunk now. Note that the new JUnitLibraryDownloader will not work until the first stable nbms-and-javadoc build containing this change, since it relies on libs.junit4 and junitlib being available on the update center.
(In reply to comment #15) > until the first stable nbms-and-javadoc build containing this change Which was #6634, so it should now work, though if you have an existing user dir AU might not reload the update center before the "Check Interval" expires unless you force it to "Reload Catalog".
> (In reply to comment #15) >> until the first stable nbms-and-javadoc build containing this change > > Which was #6634, so it should now work, though if you have an existing user dir > AU might not reload the update center before the "Check Interval" expires > unless you force it to "Reload Catalog". Or invoke _Help|Check for Updates_, it also reload the update center.
Integrated into 'main-golden', will be available in build *201102160501* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main/rev/9d76a9c03088 User: Jesse Glick <jglick@netbeans.org> Log: #195041: permit external references to NBM contents.
Integrated into 'main-golden', will be available in build *201102170501* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main/rev/64e3953b3016 User: Jesse Glick <jglick@netbeans.org> Log: #195041 refinement: have to force refresh of update providers for this to work on first start. Otherwise getUpdateUnits silently returns nothing without making any attempt to download the lists.
Needs support in the Maven mojo: http://jira.codehaus.org/browse/MNBMODULE-138