Bug 225678 - Paralelize Maven local repo indexing
Paralelize Maven local repo indexing
Product: projects
Classification: Unclassified
Component: Maven
PC Linux
: P3 (vote)
Assigned To: Tomas Stupka
Depends on:
  Show dependency treegraph
Reported: 2013-02-04 00:19 UTC by pekarna
Modified: 2016-07-07 08:38 UTC (History)
1 user (show)

See Also:


Note You need to log in before you can comment on or make changes to this bug.
Description pekarna 2013-02-04 00:19:20 UTC
Indexing local repo takes considerable amount of time.
For classical disks, the limiting factor is I/O speed.
But for SSD disks, it's processor. There's almost no I/O wait and one processor is at 90 %.

Making the indexing paralelized on more CPU cores could make indexing dramatically faster.
Comment 1 Milos Kleint 2013-02-13 15:03:03 UTC
one reason why it takes so long is that the local repository index contains more entries than  the indexes downloaded from official places. Among other things we construct the MavenProject model of the artifact to be able to populate the dependencies field. That one is used in the "Find usages" action in Services/Maven Repositories artifact nodes. (and maybe other places)
I'm not convinced the feature is used widely enough to justify the slowdown. It might be worth measuring what is the speed improvement with the feature in index turned off.
Comment 2 Milos Kleint 2013-09-03 09:14:01 UTC
for my local repository count of 7114 artifacts, the time spent in
ClassDependencyIndexCreator is 32% and ArtifactDependencyIndexCreator is 17%, so in total arount 50% time spent indexing the local repository.
The time spent is measured in creating documents in lucene index (excluded scanning the directories in the beginning)

ClassDependencyIndexCreator opens the artifact and creates a list of classes in the jar.
ArtifactDependencyIndexCreator loads a MavenProject model for the artifact in question.
Comment 3 Milos Kleint 2014-01-17 10:05:53 UTC
please see issue 239915, we've disabled osgi related processor that could in some cases account for almost 50% of the size of the entire index (apparently bundle-import/export entries are fairly long).

That processor was in effect for both local and remote repository indexes.
Comment 4 Martin Balin 2016-07-07 08:38:57 UTC
This old bug may not be relevant anymore. If you can still reproduce it in 8.2 development builds please reopen this issue.

Thanks for your cooperation,
NetBeans IDE 8.2 Release Boss

By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2014, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo