Bug 174846

I have created initial prototype and published it as a branch autoupdate-ant-174846 in the 
http://hg.netbeans.org/prototypes/. Right now I can write tests and can connect to AU center and get list of its NBM 
files. The next step is to really install something.

I guess it is a time for review. I need Jesse and Richard to look at the patch anyway. Whether we shall make this a public API and expose it in harness is slightly separate question, but I am inclined to say yes. At least I know that I could quite readily use the autoupdate task in NetBinox project (http://wiki.apidesign.org/wiki/Netbinox).

However the primary goal is to allow "binary builds" of NetBeans. Just do:

$ cd nb_all
$ ant clean
$ ant update
$ ant -f your.own.module/build.xml clean netbeans

Presummably this shall be faster than full clean build. Moreover one does not need to do clean. It is possible to just use:

$ cd nb_all
$ ant update

and the system downloads just available modules with newer spec version.

Created attachment 91449 [details]
hg diff -r 3f43f5c29bc3:eb41855a794b

if the updating is based on spec versions, it will not work reliably and a clean download will be often necessary.
unless we change the current build login and instead of an arbitrary spec version in dependency declaration we always use the one from the dependency project (so everything depends on the latest spec of everything)

[JG01] The branch shows
 autoupdate.services/build.xml                                                      |   27 
 autoupdate.services/nbproject/project.properties                                   |    6 
 autoupdate.services/nbproject/project.xml                                          |    6 

which I guess is bogus since the Ant task is in fact in nbbuild. So revert all modifications to autoupdate.services.


[JG02] "netbeans.dest.dir must existing be directory" is mangled.


[JG03] Do not rely on config/Modules/*.xml for findExistingModules, since these entries will not exist for {lib,core}/*.jar pseudomodules; use update_tracking/*.xml, which should exist for any NBM project and anyway is what AU is supposed to deal with. Use /module/@codename (remove /\d+$) and /module/module_version/@specification_version, or just try org.netbeans.nbbuild.UpdateTracking which probably does most of what you need already.


[JG04] <modules pattern="..."> should not expect dashes; CNBs are separated by dots (dashes are used only in filenames).


[JG05] Delete f (the temp file) in a finally block when done; deleteOnExit should only be a fallback for VM crashes. Also close the ZipFile in a finally block, as well as the InputStream and OutputStream.


[JG06] Need to deal gracefully with NBMs which fail to specify a target cluster (probably a BuildException is appropriate here).


[JG07] This code fails to delete files from the old NBM which are no longer present in the new NBM. (If you followed JG03 then you would already have a parse of update_tracking/*.xml, which is what you need to determine this information.)


[JG08] You cannot use <autoupdate> in common.xml unless it is available in the harness, which means in apisupport.harness you would need to include AutoUpdate.class in ${bundled.tasks} and define it in taskdefs.properties.


[JG09] Defining autoupdate.url to deadlock.netbeans.org is inappropriate in common.xml. Suggest simply moving the update target to projectized.xml; it does not really make sense in common.xml.

MK01 - it is currently up to module developers to increment spec versions when they introduce new APIs, or for that matter perform any changes which they would want to see pushed (such as P1 bug fixes). Bug #70917 suggests more or less what you said, but I think it is largely orthogonal to this issue.

MK1: I believe the feature proposed here will make the problem more visible. that's why I mentioned it.

MK2: was use of maven ant tasks considered? one reason for doing so would be possible use of repository managers, like Nexus that provide repository proxying. Use of repository managers at each remote office would significantly reduce the download times and also conserve bandwidth on our public servers. Additionally it could help with new binary staging etc.

Created attachment 91499 [details]
hg diff -r 73039bf235f9:c14eff0412a5

New patch that fixes JG01, JG02, JG03, JG05, JG07.

JG04 - tell we what is the name of a property with cnb during build. I cannot find any. So I could do pathconvert equilibristic, but accepting "-" is simpler.

JG08 & JG09 - The target and task is included in harness now. One needs to specify -Dautoupdate.catalog.url= to make it work however. I am sure I will use the task for NetBinox project. That is why I want to have it in harness.

MK01 - If we are discussing linkage errors, then this is nothing new. Whoever uses autoupdate on dev builds runs into these problems. With "ant update" we just make these problems more visible to developers that use the "link against binary mode". This is overall good thing as they may finally start to care about proper versioning. In case this creates unbearable problem, I am ready to deal with it in http://wiki.netbeans.org/SignatureTest and detect API change without spec version change.

MK02 - The AutoUpdate task is created to help people deal with NBM catalogs. Having Maven repository populated daily and easy Ant tasks in harness to communicate with it, would be good thing, but orthogonal to dealing with NBM catalogs. If you feel you can make it, start another project, please.

JG04 - currently none but trivial to create one (<parseprojectxml codenamebaseproperty="code.name.base">).


JG09 - so define autoupdate.catalog.url in projectized.xml#build-init, and in common.xml#update fail with an informative message if undefined - better than defaulting it to a nonsensical value for external modules.

It's not clear to me from the patch what folder do you want to unpack downloaded NBMs into. Suite.xml defines '<property name="netbeans.dest.dir" value="${cluster}"/>', which looks like you want to unpack it to suite's build dir (although you'd probably need 'value="${cluster}/.."' to get proper cluster structure). That's what I would expect, however  if I assume correctly that NBM unpacks into its own cluster, eg. 'ide', it will not be on cluster.path when suite is run.

OTOH common.xml defines 'netbeansdestdir="${netbeans.dest.dir}"', which means that when used for standalone module, NBM would unpack into module's binary platform, which looks wrong to me, or at least inconsistent with suite modules.

Thanks for your recent comments. I addressed them in new version:
http://hg.netbeans.org/prototypes/rev/701300b7e2d8
Now the AutoUpdate task can operate in two modes. Either one can specify installation directory or one can specify directly the cluster location. The common.xml uses the latter now.

Thanks for review. I'll integrate tomorrow:
http://hg.netbeans.org/prototypes/rev/ff26471d3f19

[JG10] nbbuild/build.xml#update can now (0ee63533c1d1) delete everything up to and including <sortsuitemodules>, since sorted.modules and so on is already defined in init-module-list, so just go straight into <subant-junit>.


[JG11] Default autoupdate.catalog.url for nb.org modules can be defined in nbbuild/build.properties rather than nbbuild/default-properties.xml.


[JG12] "Please specify HTML of catalog" is surely a typo. Did you mean "URL"?


[JG13] For an external suite, where would you specify autoupdate.catalog.url? All modules in the suite would probably use the same update site, but use of <subant inheritall="false"> means that each module must be able to "see" this property. $suite/nbproject/platform.properties may work - if so, document that.


[JG14] suite.xml#update should not use sorttests="true". (I'm not sure it's even necessary to sort the module to be updated at all, though perhaps this is useful in case the Ant process is interrupted between modules and so only some got updated?)

Re. JG11: Ignoring. Such change yields 
nbbuild/templates/common.xml:173: Please specify URL of catalog...
when running ant -f php.editor/build.xml update

Re. JG13: I am propagating the property now.

Re. JG14: Looks harmless. Skipping.


The branch has been merged as core-main#da3a413640b0

JG13 - this is probably OK, though it does make it impossible to select different update sites for certain modules in the suite.


JG14 - may or may not be harmless, but it is wrong and should be corrected.

Does not seem to be working correctly - always downloads NBMs, ignoring local build. E.g.

ant -Dcluster.config=basic clean build-nozip && ant -Dcluster.config=enterprise update

downloads every module, for example:

Entering: /space/src/nb/core-main/projectuiapi

projectuiapi.default.init:
Updating property file: /space/src/nb/core-main/nbbuild/netbeans/moduleCluster.properties

projectuiapi.update:
Using existing module item cache http://deadlock.netbeans.org/hudson/job/nbms-and-javadoc/lastSuccessfulBuild/artifact/nbbuild/nbms/updates.xml.gz
org.netbeans.modules.projectuiapi is not present, downloading version 1.41.0.6
Getting: http://deadlock.netbeans.org/hudson/job/nbms-and-javadoc/lastSuccessfulBuild/artifact/nbbuild/nbms/ide12/org-netbeans-modules-projectuiapi.nbm
To: /tmp/org-netbeans-modules-projectuiapi2416370651889064615.nbm
...................................................
Exiting: /space/src/nb/core-main/projectuiapi

Re. JG14. I see nothing wrong on sorting the modules.

You are right: cd projectuiapi; ant clean netbeans; ant update downloads something. I'll fix that.

Meanwhile use following workaround for your usecase:
ant -Dcluster.config=enterprise clean update && ant -Dcluster.config=basic build-nozip

JG14 - sorting modules is OK, but do not use sorttests="true" which uses test dependencies that are irrelevant here.

core-main#74fa6bdbfa9d

*** Bug 125522 has been marked as a duplicate of this bug. ***