Please use the Apache issue tracking system for new NetBeans issues (https://issues.apache.org/jira/projects/NETBEANS0/issues) !!
Bug 174846 - Ant tasks to download and install NBM files
Ant tasks to download and install NBM files
Status: RESOLVED FIXED
Product: platform
Classification: Unclassified
Component: Autoupdate
6.x
All All
: P3 (vote)
: 6.x
Assigned To: Jaroslav Tulach
issues@platform
http://hg.netbeans.org/prototypes/rev...
: API_REVIEW_FAST
: 125522 (view as bug list)
Depends on: 169991 178426
Blocks: 196428
  Show dependency treegraph
 
Reported: 2009-10-19 08:06 UTC by Jaroslav Tulach
Modified: 2012-01-10 17:51 UTC (History)
6 users (show)

See Also:
Issue Type: ENHANCEMENT
:


Attachments
hg diff -r 3f43f5c29bc3:eb41855a794b (50.29 KB, patch)
2009-11-20 13:28 UTC, Jaroslav Tulach
Details | Diff
hg diff -r 73039bf235f9:c14eff0412a5 (54.89 KB, patch)
2009-11-22 23:14 UTC, Jaroslav Tulach
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Jaroslav Tulach 2009-10-19 08:06:37 UTC
I'd like to download an NBM file as part of my build script and it would be nice if there was a way to wrap the 
autoupdate.services API with Ant tasks.

The configuration of such task would need URL to catalog, location of the NetBeans installation directory and list of 
modules to install/update. After successful execution the NBMs would be correctly installed in the installed in the 
platform.
Comment 1 Jaroslav Tulach 2009-10-19 17:08:49 UTC
I have created initial prototype and published it as a branch autoupdate-ant-174846 in the 
http://hg.netbeans.org/prototypes/. Right now I can write tests and can connect to AU center and get list of its NBM 
files. The next step is to really install something.
Comment 2 Jaroslav Tulach 2009-11-20 13:27:14 UTC
I guess it is a time for review. I need Jesse and Richard to look at the patch anyway. Whether we shall make this a public API and expose it in harness is slightly separate question, but I am inclined to say yes. At least I know that I could quite readily use the autoupdate task in NetBinox project (http://wiki.apidesign.org/wiki/Netbinox).

However the primary goal is to allow "binary builds" of NetBeans. Just do:

$ cd nb_all
$ ant clean
$ ant update
$ ant -f your.own.module/build.xml clean netbeans

Presummably this shall be faster than full clean build. Moreover one does not need to do clean. It is possible to just use:

$ cd nb_all
$ ant update

and the system downloads just available modules with newer spec version.
Comment 3 Jaroslav Tulach 2009-11-20 13:28:56 UTC
Created attachment 91449 [details]
hg diff -r 3f43f5c29bc3:eb41855a794b
Comment 4 Milos Kleint 2009-11-20 13:53:43 UTC
if the updating is based on spec versions, it will not work reliably and a clean download will be often necessary.
unless we change the current build login and instead of an arbitrary spec version in dependency declaration we always use the one from the dependency project (so everything depends on the latest spec of everything)
Comment 5 Jesse Glick 2009-11-20 14:14:31 UTC
[JG01] The branch shows
 autoupdate.services/build.xml                                                      |   27 
 autoupdate.services/nbproject/project.properties                                   |    6 
 autoupdate.services/nbproject/project.xml                                          |    6 

which I guess is bogus since the Ant task is in fact in nbbuild. So revert all modifications to autoupdate.services.


[JG02] "netbeans.dest.dir must existing be directory" is mangled.


[JG03] Do not rely on config/Modules/*.xml for findExistingModules, since these entries will not exist for {lib,core}/*.jar pseudomodules; use update_tracking/*.xml, which should exist for any NBM project and anyway is what AU is supposed to deal with. Use /module/@codename (remove /\d+$) and /module/module_version/@specification_version, or just try org.netbeans.nbbuild.UpdateTracking which probably does most of what you need already.


[JG04] <modules pattern="..."> should not expect dashes; CNBs are separated by dots (dashes are used only in filenames).


[JG05] Delete f (the temp file) in a finally block when done; deleteOnExit should only be a fallback for VM crashes. Also close the ZipFile in a finally block, as well as the InputStream and OutputStream.


[JG06] Need to deal gracefully with NBMs which fail to specify a target cluster (probably a BuildException is appropriate here).


[JG07] This code fails to delete files from the old NBM which are no longer present in the new NBM. (If you followed JG03 then you would already have a parse of update_tracking/*.xml, which is what you need to determine this information.)


[JG08] You cannot use <autoupdate> in common.xml unless it is available in the harness, which means in apisupport.harness you would need to include AutoUpdate.class in ${bundled.tasks} and define it in taskdefs.properties.


[JG09] Defining autoupdate.url to deadlock.netbeans.org is inappropriate in common.xml. Suggest simply moving the update target to projectized.xml; it does not really make sense in common.xml.
Comment 6 Jesse Glick 2009-11-20 14:18:56 UTC
MK01 - it is currently up to module developers to increment spec versions when they introduce new APIs, or for that matter perform any changes which they would want to see pushed (such as P1 bug fixes). Bug #70917 suggests more or less what you said, but I think it is largely orthogonal to this issue.
Comment 7 Milos Kleint 2009-11-21 13:52:11 UTC
MK1: I believe the feature proposed here will make the problem more visible. that's why I mentioned it.

MK2: was use of maven ant tasks considered? one reason for doing so would be possible use of repository managers, like Nexus that provide repository proxying. Use of repository managers at each remote office would significantly reduce the download times and also conserve bandwidth on our public servers. Additionally it could help with new binary staging etc.
Comment 8 Jaroslav Tulach 2009-11-22 23:14:39 UTC
Created attachment 91499 [details]
hg diff -r 73039bf235f9:c14eff0412a5
Comment 9 Jaroslav Tulach 2009-11-22 23:28:58 UTC
New patch that fixes JG01, JG02, JG03, JG05, JG07.

JG04 - tell we what is the name of a property with cnb during build. I cannot find any. So I could do pathconvert equilibristic, but accepting "-" is simpler.

JG08 & JG09 - The target and task is included in harness now. One needs to specify -Dautoupdate.catalog.url= to make it work however. I am sure I will use the task for NetBinox project. That is why I want to have it in harness.

MK01 - If we are discussing linkage errors, then this is nothing new. Whoever uses autoupdate on dev builds runs into these problems. With "ant update" we just make these problems more visible to developers that use the "link against binary mode". This is overall good thing as they may finally start to care about proper versioning. In case this creates unbearable problem, I am ready to deal with it in http://wiki.netbeans.org/SignatureTest and detect API change without spec version change.

MK02 - The AutoUpdate task is created to help people deal with NBM catalogs. Having Maven repository populated daily and easy Ant tasks in harness to communicate with it, would be good thing, but orthogonal to dealing with NBM catalogs. If you feel you can make it, start another project, please.
Comment 10 Jesse Glick 2009-11-23 12:25:36 UTC
JG04 - currently none but trivial to create one (<parseprojectxml codenamebaseproperty="code.name.base">).


JG09 - so define autoupdate.catalog.url in projectized.xml#build-init, and in common.xml#update fail with an informative message if undefined - better than defaulting it to a nonsensical value for external modules.
Comment 11 rmichalsky 2009-11-24 02:28:37 UTC
It's not clear to me from the patch what folder do you want to unpack downloaded NBMs into. Suite.xml defines '<property name="netbeans.dest.dir" value="${cluster}"/>', which looks like you want to unpack it to suite's build dir (although you'd probably need 'value="${cluster}/.."' to get proper cluster structure). That's what I would expect, however  if I assume correctly that NBM unpacks into its own cluster, eg. 'ide', it will not be on cluster.path when suite is run.

OTOH common.xml defines 'netbeansdestdir="${netbeans.dest.dir}"', which means that when used for standalone module, NBM would unpack into module's binary platform, which looks wrong to me, or at least inconsistent with suite modules.
Comment 12 Jaroslav Tulach 2009-11-24 04:18:52 UTC
Thanks for your recent comments. I addressed them in new version:
http://hg.netbeans.org/prototypes/rev/701300b7e2d8
Now the AutoUpdate task can operate in two modes. Either one can specify installation directory or one can specify directly the cluster location. The common.xml uses the latter now.
Comment 13 Jaroslav Tulach 2009-11-26 01:18:49 UTC
Thanks for review. I'll integrate tomorrow:
http://hg.netbeans.org/prototypes/rev/ff26471d3f19
Comment 14 Jesse Glick 2009-11-26 11:13:34 UTC
[JG10] nbbuild/build.xml#update can now (0ee63533c1d1) delete everything up to and including <sortsuitemodules>, since sorted.modules and so on is already defined in init-module-list, so just go straight into <subant-junit>.


[JG11] Default autoupdate.catalog.url for nb.org modules can be defined in nbbuild/build.properties rather than nbbuild/default-properties.xml.


[JG12] "Please specify HTML of catalog" is surely a typo. Did you mean "URL"?


[JG13] For an external suite, where would you specify autoupdate.catalog.url? All modules in the suite would probably use the same update site, but use of <subant inheritall="false"> means that each module must be able to "see" this property. $suite/nbproject/platform.properties may work - if so, document that.


[JG14] suite.xml#update should not use sorttests="true". (I'm not sure it's even necessary to sort the module to be updated at all, though perhaps this is useful in case the Ant process is interrupted between modules and so only some got updated?)
Comment 15 Jaroslav Tulach 2009-11-27 00:49:36 UTC
Re. JG11: Ignoring. Such change yields 
nbbuild/templates/common.xml:173: Please specify URL of catalog...
when running ant -f php.editor/build.xml update

Re. JG13: I am propagating the property now.

Re. JG14: Looks harmless. Skipping.


The branch has been merged as core-main#da3a413640b0
Comment 16 Jesse Glick 2009-11-27 09:20:48 UTC
JG13 - this is probably OK, though it does make it impossible to select different update sites for certain modules in the suite.


JG14 - may or may not be harmless, but it is wrong and should be corrected.
Comment 17 Jesse Glick 2009-11-27 10:06:47 UTC
Does not seem to be working correctly - always downloads NBMs, ignoring local build. E.g.

ant -Dcluster.config=basic clean build-nozip && ant -Dcluster.config=enterprise update

downloads every module, for example:

Entering: /space/src/nb/core-main/projectuiapi

projectuiapi.default.init:
Updating property file: /space/src/nb/core-main/nbbuild/netbeans/moduleCluster.properties

projectuiapi.update:
Using existing module item cache http://deadlock.netbeans.org/hudson/job/nbms-and-javadoc/lastSuccessfulBuild/artifact/nbbuild/nbms/updates.xml.gz
org.netbeans.modules.projectuiapi is not present, downloading version 1.41.0.6
Getting: http://deadlock.netbeans.org/hudson/job/nbms-and-javadoc/lastSuccessfulBuild/artifact/nbbuild/nbms/ide12/org-netbeans-modules-projectuiapi.nbm
To: /tmp/org-netbeans-modules-projectuiapi2416370651889064615.nbm
...................................................
Exiting: /space/src/nb/core-main/projectuiapi
Comment 18 Jaroslav Tulach 2009-11-27 10:59:25 UTC
Re. JG14. I see nothing wrong on sorting the modules.

You are right: cd projectuiapi; ant clean netbeans; ant update downloads something. I'll fix that.

Meanwhile use following workaround for your usecase:
ant -Dcluster.config=enterprise clean update && ant -Dcluster.config=basic build-nozip
Comment 19 Jesse Glick 2009-11-27 11:04:05 UTC
JG14 - sorting modules is OK, but do not use sorttests="true" which uses test dependencies that are irrelevant here.
Comment 20 Jaroslav Tulach 2009-11-27 11:11:38 UTC
core-main#74fa6bdbfa9d
Comment 21 Jesse Glick 2010-01-22 15:50:42 UTC
*** Bug 125522 has been marked as a duplicate of this bug. ***


By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2014, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo