This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 253322 - Unpacking index for OSS Sonatype Staging consuming 8Gb of RAM
Summary: Unpacking index for OSS Sonatype Staging consuming 8Gb of RAM
Status: REOPENED
Alias: None
Product: projects
Classification: Unclassified
Component: Maven (show other bugs)
Version: 8.0.2
Hardware: PC Windows 7 x64
: P3 normal with 1 vote (vote)
Assignee: Tomas Stupka
URL:
Keywords:
Depends on:
Blocks: 269796
  Show dependency tree
 
Reported: 2015-07-06 18:35 UTC by FiruzzZ
Modified: 2018-02-14 16:35 UTC (History)
1 user (show)

See Also:
Issue Type: DEFECT
Exception Reporter:


Attachments
IDE log (437.55 KB, text/plain)
2015-07-06 18:36 UTC, FiruzzZ
Details

Note You need to log in before you can comment on or make changes to this bug.
Description FiruzzZ 2015-07-06 18:35:48 UTC
Everytime that I open a web application, not too big yet, a process of index unpacking begin and it starts to eat all the memory, progress detail shows: "com.sleepcamel.bsoneer:bsonner-parent", tooltip says "Transfering Maven Repository index: OSS Sonatype Staging"
and freezes the pc

this is the POM
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<groupId>com.mycompany</groupId>
	<artifactId>SindicatoWeb</artifactId>
	<packaging>war</packaging>
	<version>1.0-SNAPSHOT</version>
	<name>SindicatoWeb</name>

	<properties>
		<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
		<vaadin.version>7.4.0.alpha7</vaadin.version>
		<vaadin.plugin.version>${vaadin.version}</vaadin.plugin.version>
	</properties>
	<repositories>
		<repository>
			<id>vaadin-addons</id>
			<url>http://maven.vaadin.com/vaadin-addons</url>
		</repository>
		<repository>
			<id>vaadin-snapshots</id>
			<url>http://oss.sonatype.org/content/repositories/vaadin-snapshots/</url>
			<releases>
				<enabled>false</enabled>
			</releases>
			<snapshots>
				<enabled>true</enabled>
			</snapshots>
		</repository>
                <repository>
			<id>oss.sonatype.org</id>
			<name>OSS Sonatype Staging</name>
			<url>https://oss.sonatype.org/content/groups/staging</url>
		</repository>
               
	</repositories>
        
        
	<pluginRepositories>
		<pluginRepository>
			<id>vaadin-snapshots</id>
			<url>http://oss.sonatype.org/content/repositories/vaadin-snapshots/</url>
			<releases>
				<enabled>false</enabled>
			</releases>
			<snapshots>
				<enabled>true</enabled>
			</snapshots>
		</pluginRepository>
	</pluginRepositories>
	<dependencies>
<!--            <dependency>
                <groupId>Utilitiez</groupId>
                <artifactId>Utilitiez</artifactId>
                <version>1.0</version>
            </dependency>-->
<!--                para instalar el jar en el repositorio local 
                mvn install:install-file -Dfile=/home/nsteinhilber/NetBeansProjects/libs/utilitiez.jar -DgroupId=Utilitiez -DartifactId=Utilitiez -Dversion=1.0 -Dpackaging=jar
                -->
                
                
        <dependency>
            <groupId>utilities</groupId>
            <artifactId>utilities</artifactId>
            <version>1.0</version>
            <scope>system</scope>
            <systemPath>D:/libs/utilitiez.jar</systemPath>
            <type>jar</type>
        </dependency>
		<dependency>
			<groupId>com.vaadin</groupId>
			<artifactId>vaadin-server</artifactId>
			<version>${vaadin.version}</version>
		</dependency>
		<dependency>
			<groupId>com.vaadin</groupId>
			<artifactId>vaadin-client-compiled</artifactId>
			<version>${vaadin.version}</version>
		</dependency>
                <dependency>
                	<groupId>javax.transaction</groupId>
                        <artifactId>jta</artifactId>
                        <version>1.1</version>
                </dependency>
                <dependency>
                        <groupId>com.vaadin.addon</groupId>
                        <artifactId>vaadin-charts</artifactId>
                        <version>1.1.7</version>
                </dependency>
                <dependency>
                        <groupId>org.vaadin</groupId>
                        <artifactId>suggestfield</artifactId>
                        <version>0.0.3</version>
                </dependency>
                <dependency>
                    <groupId>org.vaadin.addon</groupId>
                    <artifactId>confirmdialog</artifactId>
                    <version>2.1.2</version>
                 </dependency>

                

		<!--
		  Needed when using the widgetset optimizer (custom ConnectorBundleLoaderFactory).
		  
          For widgetset compilation, vaadin-client-compiler is automatically added on the
          compilation classpath by vaadin-maven-plugin so normally there is no need for an
          explicit dependency.
        -->
        <!--
        <dependency>
                <groupId>com.vaadin</groupId>
                <artifactId>vaadin-client-compiler</artifactId>
                <version>${vaadin.version}</version>
                <scope>provided</scope>
        </dependency>
        -->
        <dependency>
            <groupId>com.vaadin</groupId>
            <artifactId>vaadin-client</artifactId>
            <version>${vaadin.version}</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>com.vaadin</groupId>
            <artifactId>vaadin-push</artifactId>
            <version>${vaadin.version}</version>
        </dependency>
        <dependency>
            <groupId>com.vaadin</groupId>
            <artifactId>vaadin-themes</artifactId>
            <version>${vaadin.version}</version>
        </dependency>
        <dependency>
            <groupId>javax.servlet</groupId>
            <artifactId>javax.servlet-api</artifactId>
            <version>3.0.1</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>org.eclipse.persistence</groupId>
            <artifactId>eclipselink</artifactId>
            <version>2.5.1</version>
            <exclusions>
                <exclusion>
                    <groupId>org.eclipse.persistence</groupId>
                    <artifactId>commonj.sdo</artifactId>
                </exclusion>
            </exclusions>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>postgresql</groupId>
            <artifactId>postgresql</artifactId>
            <version>9.1-901.jdbc4</version>
            <scope>provided</scope>
        </dependency>
        
        <dependency>
            <groupId>org.eclipse.persistence</groupId>
            <artifactId>org.eclipse.persistence.jpa.modelgen.processor</artifactId>
            <version>2.5.1</version>
            <scope>provided</scope>
        </dependency>
        <dependency>
            <groupId>com.vaadin</groupId>
            <artifactId>vaadin-cdi</artifactId>
            <version>1.0.0.alpha2</version>
            <type>jar</type>
        </dependency>
        <dependency>
            <groupId>net.sf.jasperreports</groupId>
            <artifactId>jasperreports</artifactId>
            <version>5.5.1</version>
            <type>jar</type>
            <exclusions>
                <exclusion>
                    <groupId>bouncycastle</groupId>
                    <artifactId>bcmail-jdk14</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.bouncycastle</groupId>
                    <artifactId>bcmail-jdk14</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.bouncycastle</groupId>
                    <artifactId>bcprov-jdk14</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>bouncycastle</groupId>
                    <artifactId>bcprov-jdk14</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.bouncycastle</groupId>
                    <artifactId>bctsp-jdk14</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>org.codehaus.castor</groupId>
                    <artifactId>castor</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>com.fasterxml.jackson.core</groupId>
                    <artifactId>jackson-annotations</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>com.fasterxml.jackson.core</groupId>
                    <artifactId>jackson-databind</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>com.fasterxml.jackson.core</groupId>
                    <artifactId>jackson-core</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>log4j</groupId>
            <artifactId>log4j</artifactId>
            <version>1.2.17</version>
            <type>jar</type>
        </dependency>
        <dependency>
            <groupId>javax</groupId>
            <artifactId>javaee-web-api</artifactId>
            <version>6.0</version>
            <type>jar</type>
        </dependency>
        <dependency>
            <groupId>net.sf.jasperreports</groupId>
            <artifactId>jasperreports</artifactId>
            <version>5.1.0</version>
        </dependency>
    
         
    </dependencies>
        
      
 

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <configuration>
                    <source>1.7</source>
                    <target>1.7</target>
                </configuration>
            </plugin>
            <!-- As we are doing "inplace" GWT compilation, ensure the widgetset -->
            <!-- directory is cleaned properly -->
            <plugin>
                <artifactId>maven-clean-plugin</artifactId>
                <version>2.4.1</version>
                <configuration>
                    <filesets>
                        <fileset>
                            <directory>src/main/webapp/VAADIN/widgetsets</directory>
                        </fileset>
                    </filesets>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-war-plugin</artifactId>
                <version>2.2</version>
                <configuration>
                    <failOnMissingWebXml>false</failOnMissingWebXml>
                </configuration>
            </plugin>
            <plugin>
                <groupId>com.vaadin</groupId>
                <artifactId>vaadin-maven-plugin</artifactId>
                <version>${vaadin.plugin.version}</version>
                <configuration>
                    <gwtSdkFirstInClasspath>true</gwtSdkFirstInClasspath>
                    <extraJvmArgs>-Xmx2048M -Xss4096k</extraJvmArgs>
                    <!-- <runTarget>mobilemail</runTarget> -->
                    <!-- We are doing "inplace" but into subdir VAADIN/widgetsets. This 
                    way compatible with Vaadin eclipse plugin. -->
                    <webappDirectory>${basedir}/src/main/webapp/VAADIN/widgetsets
                    </webappDirectory>
                    <hostedWebapp>${basedir}/src/main/webapp/VAADIN/widgetsets
                    </hostedWebapp>
                    <!-- Most Vaadin apps don't need this stuff, guide that to target -->
                    <persistentunitcachedir>${project.build.directory}</persistentunitcachedir>
                    <deploy>${project.build.directory}/gwt-deploy</deploy>
                    <!-- Compile report is not typically needed either, saves hunreds of mb disk -->
                    <compileReport>false</compileReport>
                    <noServer>true</noServer>
                    <!-- Remove draftCompile when project is ready -->
                    <draftCompile>false</draftCompile>
					
                    <style>OBF</style>
                    <strict>true</strict>
                    <runTarget>http://localhost:8080/ProyectoUno</runTarget>
                </configuration>
                <executions>
                    <execution>
                        <configuration>
                            <!-- if you don't specify any modules, the plugin will find them -->
                            <!-- <modules> <module>com.vaadin.demo.mobilemail.gwt.ColorPickerWidgetSet</module> 
                            </modules> -->
                        </configuration>
                        <goals>
                            <goal>clean</goal>
                            <goal>resources</goal>
                            <goal>update-theme</goal>
                            <goal>update-widgetset</goal>
                            <goal>compile-theme</goal>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>org.mortbay.jetty</groupId>
                <artifactId>jetty-maven-plugin</artifactId>
                <configuration>
                    <webApp>
                        <contextPath>/SindicatoWeb</contextPath>
                    </webApp>
                    <scanIntervalSeconds>5</scanIntervalSeconds>
                </configuration>
            </plugin>
        </plugins>
        <pluginManagement>
            <plugins>
                <!--This plugin's configuration is used to store Eclipse m2e settings only. It has no influence on the Maven build itself.-->
                <plugin>
                    <groupId>org.eclipse.m2e</groupId>
                    <artifactId>lifecycle-mapping</artifactId>
                    <version>1.0.0</version>
                    <configuration>
                        <lifecycleMappingMetadata>
                            <pluginExecutions>
                                <pluginExecution>
                                    <pluginExecutionFilter>
                                        <groupId>com.vaadin</groupId>
                                        <artifactId>
                                            vaadin-maven-plugin
                                        </artifactId>
                                        <versionRange>
                                            [7.4.0.alpha7,)
                                        </versionRange>
                                        <goals>
                                            <goal>resources</goal>
                                            <goal>update-widgetset</goal>
                                            <goal>compile</goal>
                                            <goal>update-theme</goal>
                                            <goal>compile-theme</goal>
                                        </goals>
                                    </pluginExecutionFilter>
                                    <action>
                                        <ignore></ignore>
                                    </action>
                                </pluginExecution>
                            </pluginExecutions>
                        </lifecycleMappingMetadata>
                    </configuration>
                </plugin>
                <plugin>
                    <artifactId>maven-eclipse-plugin</artifactId>
                    <configuration>
                        <wtpversion>2.0</wtpversion>
                        <additionalProjectnatures>
                            <projectnature>com.vaadin.integration.eclipse.widgetsetNature</projectnature>
                        </additionalProjectnatures>
                        <additionalBuildcommands>
                            <buildcommand>com.vaadin.integration.eclipse.widgetsetBuilder</buildcommand>
                            <buildcommand>com.vaadin.integration.eclipse.addonStylesBuilder</buildcommand>
                        </additionalBuildcommands>
                    </configuration>
                </plugin>
            </plugins>
        </pluginManagement>
    </build>
</project>


Product Version = NetBeans IDE 8.0.2 (Build 201411181905)
Operating System = Windows 7 version 6.1 running on amd64
Java; VM; Vendor = 1.8.0_45
Runtime = Java HotSpot(TM) 64-Bit Server VM 25.45-b02
Comment 1 FiruzzZ 2015-07-06 18:36:04 UTC
Created attachment 154493 [details]
IDE log
Comment 2 Tomas Stupka 2015-07-14 10:26:28 UTC
two problems:
1.) indexing performance 
wasn't able to reproduce any detrimental effects with a current dev build. The oss sonatype index isn't small, but the indexing itself took a few minutes on my box and  the memory footprint never got over 300mb. Please note that there was a bunch of fixed some time ago - e.g. issue #239915 . Even though it was more speed related, it might have also some effect on the freezes you mentioned in your report. 

2.) indexing triggers on every open 
- what do you have set under options > java > maven > index > index update frequency ? 
- what does it mean on every open - also subsequent opens during one netbeans session or do you open the project only once in a while and there are NetBeans restarts between?
Comment 3 FiruzzZ 2015-07-22 10:28:17 UTC
Maven frequency = Once a week
The project is not always opened in Netbeans, but when pass a long time and I have to make it a Git pull to update it, after that this indexing starts, I figured that if I close the project immediately after update the branches, it stops. Reopening the project in the same instances of the IDE doesn't trigger this indexing again, restarting the IDE it does, but the Repository updates normally (slow but normal)
Comment 4 Tomas Stupka 2015-07-22 10:35:07 UTC
> I figured that if I close the project immediately after update the branches,
> it stops. 
so the indexing never runs through? because then reopening the project is going to trigger the indexing again

(in case you want to avoid index handling for a particular repository - use the switch to suppress automatic repository indexing for a specified list of repos
-J-Dmaven.indexing.doNotAutoIndex=[semicolon serated list of repo id-s])
Comment 5 cipriandobra 2015-08-05 18:56:33 UTC
I have seen the same behavior if I don't open Netbeans for a while (1-2 weeks). It seems though that after a few stops-starts Netbeans is able to finish the "Unpacking index for OSS Sonatype Staging".

I have not seen a more annoying bug in Netbeans since I've been using it (Netbeans 5)
Comment 6 thundernet 2017-02-08 16:50:52 UTC
I've come across the same issue in 8.0.2 and 8.2 (not tried 8.1 or other intermediate releases). 

I'm running on Windows 7. 

Several Maven based war projects. I can post effective POMs if need be although I'm not sure they'll be terribly useful. 

I think a clean and build tries to load the sonatype-oss-public repository index.
This takes an awfully long time to start with.

However, when the status bar shows 'Unpacking index for sonatype-oss-public' memory usage jumps up and up until it's using over 6GB and I have to kill the Netbeans64.exe process. 

This will usually take place over a period of a few hours. 

I've added -J-Dmaven.indexing.doNotAutoIndex=sonatype-oss-public in netbeans.conf (on the 8.2 release) and memory usage seems stable at just over 2GB although I'll be keeping an eye on it. Usage is creeping up but that could just just be normal caching. 

I think this is definitely a live issue and warrants a bit more investigation. 
Perhaps index unpacking for certain repo's should be switched off by default? 

It does pretty much make netbeans useless when it's happening and I considered switching IDEs.
Comment 7 MrQuan 2017-05-09 10:31:39 UTC
I'm experiencing this issue today after adding the Sonatype OSS Maven Repository to a project.

My environment:
NetBeans IDE 8.2 (Build 201609300101)
NetBeans IDE is updated to version NetBeans 8.2 Patch 1
Windows 10

This is what I added to my pom.xml that caused the issue:
        <repository>
            <id>oss-snapshots-repo</id>
            <name>Sonatype OSS Maven Repository</name>
            <url>https://oss.sonatype.org/content/groups/public</url>
            <snapshots>
                <enabled>true</enabled>
                <updatePolicy>always</updatePolicy>
            </snapshots>
        </repository>

I needed this repo for a snapshot version of pi4j:
        <dependency>
            <groupId>com.pi4j</groupId>
            <artifactId>pi4j-core</artifactId>
            <version>1.2-SNAPSHOT</version>
        </dependency>

Process netbeans64.exe Uses 6+GB of memory on my machine (8GB RAM), this in turn causes the HDD to max out (I assume disk RAM/compression of some sort), and my whole PC becomes unresponsive until I'm able to kill the Netbeans process.

This is very frustrating and makes Netbeans unusable until I move this dependency to a locally hosted repo, which is not convenient at all.

Please comment if more details are required from my installation.
Comment 8 MrQuan 2017-05-09 12:13:31 UTC
Update from the previous comment.

I've now opened this on another Windows 10 PC with same version of Netbeans, but better hardware specs:
CPU:      i7 4770K @ 3.5 GHz
Memory:   32 GB
HDD:     512 GB SSD

Netbeans status bar message is: "Unpacking index for Sonatype OSS Maven Repository". Memory usage is crazy big and still climbing...

In Resource Monitor netbeans64.exe memory usage is:
Commit (KB):        1,232,532
Working Set (KB):  16,300,632
Shareable (KB):    15,440,388
Private (KB):         860,244

In Task Manager it shows:
CPU:      22.3 %
Memory:  840.1 MB
Disk:    231.0 MB/s

My OS is becoming unresponsive, so I had to kill the Netbeans process again.

Something about that Sonatype OSS Maven Repository is really upsetting Netbeans.
Comment 9 tobidelbruck 2018-02-14 14:58:40 UTC
Same problem here. Netbeans 8.2, JDK 1.8. endless indexing of oss.sonatype.org from some included maven repo library coming from ivy.

Product Version: NetBeans IDE 8.2 (Build 201609300101)
Updates: Updates available to version NetBeans 8.2 Patch 2
Java: 1.8.0_121; Java HotSpot(TM) 64-Bit Server VM 25.121-b13
Runtime: Java(TM) SE Runtime Environment 1.8.0_121-b13
System: Windows 10 version 10.0 running on amd64; Cp1252; en_US (nb)
User directory: C:\Users\Tobi Delbruck\AppData\Roaming\NetBeans\8.2
Cache directory: C:\Users\Tobi Delbruck\AppData\Local\NetBeans\Cache\8.2
Comment 10 tobidelbruck 2018-02-14 16:35:45 UTC
AFter I updated to the version below, and manually ran indexing from the Options/Java/Maven/Index tab, it seemed to fix the problem. Version info below:

Product Version: NetBeans IDE 8.2 (Build 201705191307)
Updates: NetBeans IDE is updated to version NetBeans 8.2 Patch 2
Java: 1.8.0_121; Java HotSpot(TM) 64-Bit Server VM 25.121-b13
Runtime: Java(TM) SE Runtime Environment 1.8.0_121-b13
System: Windows 10 version 10.0 running on amd64; Cp1252; en_US (nb)
User directory: C:\Users\Tobi Delbruck\AppData\Roaming\NetBeans\8.2
Cache directory: C:\Users\Tobi Delbruck\AppData\Local\NetBeans\Cache\8.2