Bug 206752 - Consider shipping the Maven central index as part of the product
Consider shipping the Maven central index as part of the product
Status: NEW
Product: installer
Classification: Unclassified
Component: Code
7.0.1
All All
: P3 with 1 vote (vote)
: TBD
Assigned To: Libor Fischmeistr
issues@installer
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2011-12-27 22:00 UTC by _ mriem
Modified: 2014-02-10 14:15 UTC (History)
6 users (show)

See Also:
Issue Type: ENHANCEMENT
:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description _ mriem 2011-12-27 22:00:13 UTC
Since the download for the Maven central index is of a substantial size why not include it with the NetBeans install?
Comment 1 Jesse Glick 2011-12-27 22:09:21 UTC
Precisely because it is so big, and it would be wasted download overhead for those users who do not use Maven, or for those who do but already have a mirror (e.g. Nexus). I recommend WONTFIX.
Comment 2 _ mriem 2011-12-27 22:23:23 UTC
(In reply to comment #1)
> Precisely because it is so big, and it would be wasted download overhead for
> those users who do not use Maven, or for those who do but already have a mirror
> (e.g. Nexus). I recommend WONTFIX.

While I understand where you are coming from, I think from a usability perspective
it sucks. The whole point of using Maven and NetBeans together is so that you can
reap the benefits from both worlds. 

If a personal developer cannot get the index downloaded then the repository browser within NetBeans and completion of dependencies within POM files becomes totally unusable.

Note this is related to the other issue I filed about the index download never finishing. And please do not recommend a WONTFIX there either since both are
usability issues.
Comment 3 Jesse Glick 2011-12-28 00:48:46 UTC
Agreed that for most Maven users it would be most convenient to have the Central index in place from the start, but this would be a large addition to the download package (60Mb last I checked) and would probably have to somehow be optional.
Comment 4 _ mriem 2011-12-28 02:44:19 UTC
(In reply to comment #3)
> Agreed that for most Maven users it would be most convenient to have the
> Central index in place from the start, but this would be a large addition to
> the download package (60Mb last I checked) and would probably have to somehow
> be optional.

I agree it should be optional. 

Question is how can we get the out-of-the-box experience to be so that NetBeans is able to offer POM completion and repository searching without having to wait for the Central index to download?
Comment 5 Antonin Nebuzelsky 2011-12-28 12:05:50 UTC
As mentioned the central index is too large to be included in the installers by default.

I tried downloading nexus-maven-repository-index.gz (39.1MB) from a web browser and it takes quite long. About 10 minutes for me. It seems like the maven server is limiting the download speed intentionally.

How about hosting the index also on our download servers to speed the download for NetBeans users?

Download of the index could also be considered as an extra step of the installation process, so that after the first start of NetBeans the index is ready.
Comment 6 Jesse Glick 2011-12-28 15:38:42 UTC
Also seems much slower to me than my general network bandwidth would permit; either they throttle downloads or their pipe is just saturated.

Can also check known mirrors [1], though I doubt any (possibly excepting the official UK mirror) are as carefully tuned as the default.

BTW the Central index when bzip2 compressed (rather than gzip) is just 29Mb (a 28% savings). The JRE includes no decompressor for this format, but Ant does (org.apache.tools.bzip2.CBZip2InputStream, also extracted as e.g. [2]), as does jbzip2 [3] and Commons Compress [4].

Note that once you have some snapshot of nexus-maven-repository-index.gz downloaded and unpacked, nexus-maven-repository-index-updater.properties keeps track of which version it was, so in subsequent weeks (or whatever the update frequency is set so) you just check the remote nexus-maven-repository-index.properties and download incremental chunks when available.

If IDE download size is under consideration, we should review the open-source libraries we bundle in the IDE (where not linked against by IDE modules) to see if they duplicate artifacts in Central that could rather be downloaded on demand. They are useless to Maven users, and for a given Ant user may or may not ever be used (especially if you want a different version). For example, enterprise/modules/ext/metro/ is 17Mb (not sure if this gets pack200 compressed in the installer). Worth investigating whether Library Manager can accept definitions of JARs which refer to Central artifacts, e.g. using 'm2' URL protocol, and download them if and when the lib is added to a project. (Certainly this would help solve our JUnit conundrum!)

[1] http://repo1.maven.org/maven2/.meta/repository-metadata.xml
[2] http://www.kohsuke.org/bzip2/
[3] http://code.google.com/p/jbzip2/
[4] http://commons.apache.org/compress/
Comment 7 Milos Kleint 2011-12-28 15:49:20 UTC
also please note that in the case the index is part of the installers (as opposed to be downloaded by installers from central), with time passing after the netbeans release the shipped index will be too old and the diffs will not be available anymore, only a complete index (I don't know exactly what the time frame is but I if I'm correct it's there)


having a nexus proxy/ies just for distribution of the index could eventually work.. *if* nexus can actually cope with the volume. AFAIK the index on central is on a totally separate server from the rest of central, in order to better cope with the networking volume..
Comment 8 Jesse Glick 2011-12-28 18:00:08 UTC
(In reply to comment #7)
> in the case the index is part of the installers [...] with time passing after
> the netbeans release the shipped index will be too old and the diffs will not
> be available anymore, only a complete index

So incremental chunks are only offered for a sliding window prior to the current snapshot? FWIW, the current properties file lists 27 versions; I cannot remember if the Central index is refreshed weekly or biweekly (HTTP timestamps on the chunks all correspond to the last refresh).

> having a nexus proxy/ies just for distribution of the index could eventually
> work

If we are to offer a current snapshot of the index for separate download, I would just put it on a static HTTP server (*), and configure maven.indexer to look for it when servicing an initial request for 'central' to the standard unmirrored location. Any subsequent updates would come from the official site.

(*) BitTorrent would be even better, but then we would need a Java BT client - either in the IDE (if part of maven.indexer) or the installer (if done eagerly right after regular installation).
Comment 9 _ mriem 2012-02-11 01:42:16 UTC
Note I have finally tracked down the issue to be a JDK bug that seems to affect more applications that just NetBeans

Forcing Maven to use IP4 seems to solve the problem, using -Djava.net.preferIPv4Stack=true)
Comment 10 Jesse Glick 2012-02-14 19:07:40 UTC
(In reply to comment #9)
> Forcing Maven to use IP4 seems to solve the problem, using
> -Djava.net.preferIPv4Stack=true)

I assume this was meant for some other issue.
Comment 11 _ mriem 2012-02-14 19:46:24 UTC
Ehhh, yea I should have been more clear. It solves the stalling out completely. Ob viously the size of the initial download is still substantial and the main repo is not always the fastest.
Comment 12 _ gtzabari 2012-10-24 15:25:08 UTC
I use Maven all the time, and I'd vote against this issue. The Netbeans 7.3 installer is already too big (235MB for the JavaEE version).

I suggest three alternatives:

1. Ensure downloads never stall (it used to happen a lot, I'm not sure if it still does).

2. Try to automatically locate and download the index from a faster mirror.

I'm pretty sure most people would be happy to download the index on-demand if it actually downloaded at a decent speed. That's not a bundling problem. It's Maven Central's problem.

3. Mirror the index file on Netbeans' servers and download from there. Obviously, this is less than ideal. This would be as fast as bundling the index with the download, but make it optional. Still, it doesn't make much sense for us to work around what is essentially someone else's server problem (and pay extra bandwidth costs on their behalf).
Comment 13 Petr Jiricka 2012-11-13 14:50:48 UTC
I must say that for me it takes a significant amount of time to unpack the index once it's downloaded (on a SDD drive, so it's not a IO issue). Could that be sped up? Are we aware of this problem; should I file a separate report?
Comment 14 Milos Kleint 2012-11-14 05:50:28 UTC
(In reply to comment #13)
> I must say that for me it takes a significant amount of time to unpack the
> index once it's downloaded (on a SDD drive, so it's not a IO issue). Could that
> be sped up? Are we aware of this problem; should I file a separate report?

out of our capacity, it's all hidden behind one library method call. Originally the nexus index shipped with the lucene index itself and download was all that was necessary (if I remember right), nowadays it's a compressed format that is processed and new lucene documents are created one by one.


By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo