This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
[ BUILD # : 201401020002 ] [ JDK VERSION : 1.7.0_45 ] STEPS: * Sometimes Netbeans index' a Maven repository ACTUAL: The progress bar stays at 100 % for a long time saying "Unpacking index for [your repo]". EXPECTED: Unpacking should not take that long, it does not do that in IDEA.
Forget the 100 % part. That was the progress bar before that, but the bug is still valid, the Unpacking is extremely slow.
(In reply to kimsp from comment #0) > [ BUILD # : 201401020002 ] > [ JDK VERSION : 1.7.0_45 ] > > > EXPECTED: > Unpacking should not take that long, it does not do that in IDEA. I'm not really sure what we can improve here. We basically call one method from maven-indexer library that both downloads the index (or it's part) and populates the lucene index. We do a bit of additional work for local repository (about 30-50% overhead if I recall correctly from the time I measured it) Please note that the slowness can have multiple causes including not having enough memory available to the IDE's JVM, resulting in excessive GC.
I would like to fix this problem myself (I have a masters degree in CS and code Java professionally for a living). Please send me a mail and we can talk. Thanks and regards Kim
issue 225678 is related
http://hg.netbeans.org/core-main/rev/5cc878ddb9e7 see issue 240150 for details. Could at least partially help with the slowness, in the sense that less gc will be done and likely the lucene index will be smaller.
Integrated into 'main-silver', will be available in build *201401160001* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main-silver/rev/5cc878ddb9e7 User: Milos Kleint <mkleint@netbeans.org> Log: #240150, #239915 avoid slow search for groupids, it's a memory bottleneck. Also avoid storing and retriving osgi related document fields, we do not query them and they can occupy a lot of memory
the diff between with and without osgi fields in documents is following in terms of size: Miloss-iMac:mavenindex mkleint$ du my-local-nexus/ 1036216 my-local-nexus/ Miloss-iMac:mavenindex mkleint$ rm -rf my-local-nexus/ Miloss-iMac:mavenindex mkleint$ du my-local-nexus/ 1885560 my-local-nexus/ that's on my mac with my local nexus proxy of central we're down close to half the size, which is great especially given we're not using these fields at all. That was not that severe when the feature was introduced (I think I measured it back then). Apparently more projects now include osgi headers to artifacts.
in human readable sizing it's 921M vs. 506M
not sure what more can be done without additional information. constructing the index is an expensive operation with high requirements on cpu and io. considered fixed.
What puzzles me is why are other processes affected? My machine is literally not usable while this "Unpacking index for Sonatype Repository" is running. I checked the system processes. I have HP Z600 workstation here, with 12GB RAM and 6-core i7. Whenever NetBeans starts indexing, I have to take a coffee break... Could this be done in an external process (separate JVM maybe) perhaps with extremely low priority?
I second what dejlek said, I understand not much can be done, but this is the one and only application bringing my PC to its knees. The problem is that it directly hits the hard drive, and even though the CPU Could this index be split in small batches, or deprioritized, or even better, could you put a HD transfer rate limitation? For instance, one third of the disk's max transfer rate (if one such information can be retrieved in Java) or a manual setting.
> the one and only application bringing my PC to its knees. Same here with the latest release version. This is not resolved. I'm using a brand new i7 with 32 GB of RAM and an SSD. It's been this way for over 90 minutes and counting. Admittedly I just added a huge dependency to my pom.xml file (Spring Framework) and I'm runny a crappy OS(Windows), but the system is nearly unusable. I'm even having trouble typing in this simple textbox.
Hi Milos, I hope you don't mind, I have decided to reopen this issue as it is reproducible with the latest GA build (Netbeans 8.0.1-javaee-windows.exe) and renders the computer inoperable. I have also changed the Hardware field to "Windows 7 x64" (was Windows XP) Environment: - Windows 7 - 6GB RAM - OS is up to date. - Antivirus is up to date. Steps to reproduce: - Download Netbeans javaee 8.0.1 - Import a Maven project. - Use Netbeans normally for a few hours. (used maven to compile a project deploy it on a web server for instance without any problems/delays) Expected behavior: - Computer should remains be reponsive Actual behavior: - After a few hours of usage the following message shows in the bottom corner of the Netbeans window: "Unpacking index for Nexus Public Mirror" - The whole computer becomes nearly unresponsive . - Netbeans itself is excruciatingly slow. - System has been like this for well over an hour. Notes: - Unzipping take a lot longer on my Windows machine compared to a a Linux box with similar horse power. I am thinking that the underlying unzipping algorithm could be the key. - Interestingly the overall CPU usage as per the "Windows task manager" is only between 5% and 9%. - Netbeans memory usage went up to 750Mb and then back down to 422Mb. (Memory usage climbing again.) - Netbeans CPU usage is only between 1% and 4% - It seems to me that Netbeans is waiting for something. Maybe some decompression library at the OS level that is not listed by the "Windows Task Manager"? Question: - I will let the machine run overnight but I don't want to loose hours of work whenever the nexus re-indexing kicks-in. Can this feature be turned of? If so, how? Best regards Michel
Yesterday, shortly after posting the above comment, a message appeared in Netbeans indicating that there was no space left on the C:\ drive. (only about 2Gb left) This was strange because I had checked my C:\ drive for free space that morning and I had over 25Gb free... I found a series of huge files that were created by Netbeans under C:\Users\<nmyUserName>\AppData\Local\Temp Specifically there was a folder called nexus-maven-index.gz879764343.... that was 19.8 Gb There were other nexus-maven-index folders on the system that were created yesterday but they were smaller in size. I have deleted these files/folders as well as the .m2 folder and I am restarted Netbeans. Will keep an eye on it today. Regards Michel
Well, the indexing issue is back. My computer was unusable for the whole afternoon and I let it run overnight. When I came back this morning and I had only 19Gb free on my C:\ (I had 37Gb free before the maven index unpacking started.) Maybe I am missing something here but I can't see why the index would take so much space (18Gb). Since the unpacking is finished the computer is responsive again but Netbeans feels very sluggish. To me this issue is a P2, not a P3... Please take action. Regards, Michel
I have to chime in here, because this problem is getting very annoying and ought to be fixed: Approx. once a week, Netbeans starts thrashing my SSD drive with huge amounts of written data (many many gigabytes written over and over), while doing "Unpacking index for ..<some Maven repo>". It hits my laptop so hard it makes it unusable for other tasks. It looks like something is constantly optimizing (merging segments) of the underlying Lucene index at each step of the "Unpacking index" process. That's what it looks like in the Lucene index directory under ~/.cache/netbeans/8.0.1/mavenindex/<repo>/. Huge segments files of many gigabytes are written as fast as possible and then deleted (merged). This goes on for a good 10 minutes. Lucene usually memory-maps these segment files, and it causes crippling heavy I/O, even on a modern laptop with fast SSD and 16GB RAM. (And it sucks life out of SSD as well !) Something is broken with the Maven [Lucene] index integration.
I'm on Linux 64bit and using Oracle JDK8 to run Netbeans btw. (same issues also with JDK7).
Somewhere, in some code, one should make sure that Maven Lucene indices aren't optimized (force merged) all the time (after every modification ?). Optimizing multi-GB Lucene indexes has a high I/O cost. Though I am not familiar with the details of usage in Netbeans/Maven, so it might be caused by some strange Lucene merge policy or similar. Is it possible to make sure that Maven Lucene indices are only merged down once at the end of the "Unpacking index .." processes ? Or even not at all ?
Since it may be a hint/trigger of this problem, I should mention that it only seems to happen when "Unpacking" our internal Sonatype Nexus Maven repository, which for some reason has a largeish index of about 7GiB on disk in Netbeans cache (with ~4.4 million Lucene docs in it).
I've met the same problem when add new dependency in my pom.xml . I have an MacBook Pro with i7 and 8GR, and nearly stuck with Netbean unpacking index from Sonatype repo. I'd like to know if this bug get fixed. Product Version: NetBeans IDE 8.0 (Build 201408251540) Updates: Updates available to version NetBeans 8.0.1 Patch 1.1 Java: 1.7.0_60; Java HotSpot(TM) 64-Bit Server VM 24.60-b09 Runtime: Java(TM) SE Runtime Environment 1.7.0_60-b19 System: Mac OS X version 10.8.5 running on x86_64; UTF-8; en_US (nb)
Is there an option to disable this indexing altogether? I build from the command prompt, so netbeans doesn't need these indexes at all (in my case) [ build # :201408251540 ] [ jdk version: 1.8.0_25 ] [ Linux 3.11.10 x68_64 ] [ 16 Gb RAM 12 CPU's ]
> Is there an option to disable this indexing altogether? see options > java > maven > index
in case somebody wants to avoid index handling for a particular repository - added switch to suppress automatic repository indexing for a specified list of repos -J-Dmaven.indexing.doNotAutoIndex=[semicolon serated list of repo id-s] core-main #c9284e7d40df
core-main #c62c1bc19f68 several fixes to reduce indexing time - down to 35-50% currently the downloaded gz file from maven central has ~ 150mb and results into a lucene index of ~ 750mb after aprox 2 minutes of indexing time on a moderate hardware (macbook, 2.4 GHz Intel Core i5, RAM 8 GB, SSD) (download time not considered) so closing for now. it is always possible to bring the IDE down at some point - to process a many gigabytes big index file which results into an 5 times bigger lucene idx just can't be done in a few seconds. An index of 7 or 18Gb (like reported), just seems to huge. in case you still want to follow up it would be good to know if - this was caused by a proportionally big index file (e.g. 4GB???) downloaded from a repository or if things went wrong on the client (and we still have a chance to do something about it) - if possible please also attach a profiler snapshot
one more thing - in case there is anybody how observed a too big (1gb+) lucene cache (see in {userdir}/var/cache/mavenindex/{repo_name}) on a public repository, please let us know.
Integrated into 'main-silver', will be available in build *201412090001* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main-silver/rev/c62c1bc19f68 User: Tomas Stupka <tstupka@netbeans.org> Log: issue #239915 - Unpacking index is extremely slow
See #239704 regarding compressed indices with newer Lucene as well. (Decreases space requirements, increases indexing speed)
Tested with this repository: https://maven.atlassian.com/content/groups/public/ Could not upload the snapshot; link below. these are the netbeans start parameters: netbeans_default_options="-J-server -J-Xss2m -J-Xms32m -J-Djava.io.tmpdir=/home/alied/.tmp/ -J-Dnetbeans.logger.console=true -J-ea -J-Dapple.laf.useScreenMenuBar=true -J-Dapple.awt.graphics.UseQuartz=true -J-Dsun.java2d.noddraw=true -J-Dsun.java2d.dpiaware=true -J-Dsun.zip.disableMemoryMapping=true -J-Dplugin.manager.check.updates=false -J-Dnetbeans.extbrowser.manual_chrome_plugin_install=yes" index downloaded: alied@development:~/.tmp$ ls -lh total 636M -rw-r--r-- 1 alied alied 184 Dec 11 00:59 fallback1809026237901536194.netbeans.pom -rw-r--r-- 1 alied alied 182 Dec 11 00:59 fallback2910971787650671001.netbeans.pom drwxr-xr-x 2 alied alied 4.0K Dec 11 00:59 jarfscachealied drwxr-xr-x 2 alied alied 4.0K Dec 11 00:52 jna-92903101 -rw-r--r-- 1 alied alied 3.2K Dec 11 01:03 loading3418414702126207214.html drwxr-xr-x 2 alied alied 4.0K Dec 11 01:14 nexus-maven-repository-index.gz2310359650455394803.dir -rw-r--r-- 1 alied alied 635M Dec 11 01:14 nexus-maven-repository-index.gz8630440163568623504 -rw-r--r-- 1 alied alied 132K Dec 11 01:05 output1418270720970 -rw-r--r-- 1 alied alied 595K Dec 11 01:03 selfsampler3879241942145247205.npss -rw-r--r-- 1 alied alied 1.3K Dec 11 01:02 uigesture4331769078722541539.html HDD: ATA device, with non-removable media Model Number: WDC WD2500HHTZ-04N21V0 Firmware Revision: 04.06A00 Transport: Serial, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6, SATA Rev 3.0 Processor: AMD FX(tm)-8320 Eight-Core Processor RAM: 8GB Max. uncompressed size: 5.9GB snapshot: https://drive.google.com/open?id=0B81H8YEUuTY6V1VvNEhpOVZ2S3c&authuser=0
Same for atlassian public repository: https://m2proxy.atlassian.com/repository/public/ root@development:/home/alied/.tmp# ls -lh total 637M -rw-r--r-- 1 alied alied 184 Dec 11 00:59 fallback1809026237901536194.netbeans.pom -rw-r--r-- 1 alied alied 182 Dec 11 00:59 fallback2910971787650671001.netbeans.pom drwxr-xr-x 2 alied alied 4.0K Dec 11 00:59 jarfscachealied drwxr-xr-x 2 alied alied 4.0K Dec 11 00:52 jna-92903101 -rw-r--r-- 1 alied alied 3.2K Dec 11 01:03 loading3418414702126207214.html -rw-r--r-- 1 alied alied 635M Dec 11 01:47 nexus-maven-repository-index.gz237556840577647767 drwxr-xr-x 2 alied alied 12K Dec 11 01:49 nexus-maven-repository-index.gz8483188642343373115.dir -rw-r--r-- 1 alied alied 136K Dec 11 01:30 output1418270720970 -rw-r--r-- 1 alied alied 1.2M Dec 11 01:28 selfsampler2253156228267891884.npss -rw-r--r-- 1 alied alied 595K Dec 11 01:03 selfsampler3879241942145247205.npss -rw-r--r-- 1 alied alied 1.3K Dec 11 01:02 uigesture4331769078722541539.html Max. uncompressed size: 6.2GB Product Version: NetBeans IDE Dev (Build 201412100001) Java: 1.8.0_25; Java HotSpot(TM) 64-Bit Server VM 25.25-b02 Runtime: Java(TM) SE Runtime Environment 1.8.0_25-b17 System: Linux version 3.17.6 running on amd64; UTF-8; en_US (nb) User directory: /home/alied/.netbeans/dev Cache directory: /home/alied/.cache/netbeans/dev snapshot: https://drive.google.com/open?id=0B81H8YEUuTY6V1VvNEhpOVZ2S3c&authuser=0 P.S. I could run these tests for #239704 as well.
ERRATA: actual link for previous snapshot is: https://drive.google.com/open?id=0B81H8YEUuTY6bGctVWpja0piVUE&authuser=0 Anyway, you should be able to access both in https://drive.google.com/folderview?id=0B81H8YEUuTY6RThFeWVpLVNBUm8&usp=sharing
*** Bug 250158 has been marked as a duplicate of this bug. ***
*** Bug 252851 has been marked as a duplicate of this bug. ***
Why can't NetBeans simply use Nexus incremental indexes (like Elipse)? I understand that the first time is has to download the full index but on subsequent weeks it could use the incremenental chunks!
Issue is back. My machine is literally not usable more then 3 hours while this "Unpacking index for Sonatype Repository" is running. Product Version: NetBeans IDE 8.1 (Build 201510222201) Java : 1.7.0_75; Java HotSpot(TM) Client VM 24.75-b04 Runtime : Java(TM) SE Runtime Environment 1.7.0_75-b13 System : Windows 7 Professionnel 64-bit (6.1, Build 7601) Service Pack 1 Memory : 4GB Processor : Intel(R) Core(TM) i5-3470 CPU @ 3.20GHz (4 CPUs), ~3.2GHz size of "C:\Users\badi.mohammedtahar\AppData\Local\NetBeans\Cache\8.1\mavenindex" = 11.2 GO
Extremely high memory usage (>5GB) and system unusable for hours if not canceled. Windows 7 64-bit Product Version: NetBeans IDE 8.1 (Build 201510222201) Java: 1.8.0_45; Java HotSpot(TM) 64-Bit Server VM 25.45-b02 Runtime: Java(TM) SE Runtime Environment 1.8.0_45-b15 System: Windows 7 version 6.1 running on amd64; Cp1252; en_US (nb) Maven 3.3.9 Memory: 8GB Processor: Intel Core i7
Out of curiosity, is it possible to index across multiple cores? It's pretty frustrating waiting over 30 minutes for "unpacking index" with CPU usage under 12%.
When I disable the the maven index download via the Index Update Frequency setting, then there is an issue with the test connection button of the proxy settings as follows: The test does not finish with a failure even when I wait. The Cancel button on the dialog just hides the dialog, and when I open the Options dialog later, then it is still in its previous state, on the connection test with the progress bar animated.
In the above case, the IDE is again frozen to the degree that I cannot close it, I cannot interact with the menu, anything. JVisualVM waits indefinitely trying to connect to NetBeans which is now shown in it. Could you please try to make an improvement to the NetBeans networking code in general. It does not matter what. Just get started with it. I am confident that anything will help. I hope that I have demonstrated that the code is lacking robustness. As I have suggested elsewhere in this system, it would help to apply a strategy of post mortem analysis. The user needs feedback regarding the the source of the problem. A contemporary computer program should be able to sort out these type of issues without the requirement for support action.
Created attachment 162209 [details] Thread dump for the latest scenario JVisualVM finally allowed me to take a thread dump, but its progress bar is still showing "Opening NetBeans ...", being animated.
Comment on attachment 162209 [details] Thread dump for the latest scenario Sorry, wrong bug
Sorry, my comments are for the wrong bug. This is because bugzilla moves to another bug after an update - annoying.
Any update on this? :)
This is still a major problem. I am using a 64-bit Windows 10 Pro machine, i7 quad core with 4.2Ghz, 64 GB of RAM, and a SSD. I'm running NetBeans 8.2 (build 201609300101) with NetBeans 8.2 Patch 2. I have Java 1.8_131. When NetBeans starts to index a maven repository, it kills my computer. Windows Task Manager shows that I have plenty of CPU and Memory available. But, my Disk I/O is 100% used. I increased my minimum Page file size to 64 GB without any improvement. I sorted the Resource Monitor by Read (B/sec), and it shows that netbeans64.exe has one PID with 12 files open in the AppData\Local\NetBeans\Cache\8.2\mavenindex\sonatype-public-repository directory. It is reading from these files at a combined rate of 20,000,000 B/sec. Thank you for offering a free development environment. But, I cannot use NetBeans because of this problem.
I cannot use netbeans anymore these days. been a user for 10 years! This is killing my disk. Its always 100%.
For troubleshooting purpose, could you take screenshots or capture a video of following: - free disk space - slow index unpacking - performance stats / graphs and info of: - O/S version - JDK version
For a while now this has been preventing me from using Netbeans, every time I start it, after around five minutes, my disk I/O usage jumps to 100% and my system locks up. I always ends up having to kill NetBeans, even if I leave it to do its thing for hours, it never seems to end. Sometimes I've even had to shutdown my laptop the hard way because nothing worked. I'm using an HDD, but I don't even know if an SSD would improve things, and with the number of I/O operations that seem to be going on in the background, I don't know if that'd be a very good idea. Removing my Maven directory helps for a day or so, but then I have to redownload all dependencies for all of my projects, it's not very convenient. I'm switching over to IntelliJ until this is fixed.
Agreed. This is a showstopper issue. The mere presence of files in the Maven indexer directly causes Netbeans to silently (no UI feedback provided) start thrashing the hard-drive. I actually have an SSD and I can tell you it doesn't make a difference. 1. Netbeans is taking longer than expected to process Maven index files. Much longer than Intellij by multiple orders of magnitude. 2. Background tasks should always show up in the status bar. The fact that this does not is a bug in my opinion. It took me forever to figure out that Netbeans was killing my machine. I initially thought it was some random Windows background service. Upon clearing the Maven index directory, the problem went away. At least until the next time the index was downloaded...
Still very slow in Netbeans 8.2, MBPR 13 2015, take over 30 mins.