Bug 267969 - Running single simple unit test in debug takes 6+ seconds
Running single simple unit test in debug takes 6+ seconds
Status: RESOLVED FIXED
Product: updatecenters
Classification: Unclassified
Component: Pluginportal
Dev
PC Windows 7
: P2 (vote)
: TBD
Assigned To: attila.kelemen
issues@updatecenters
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2016-09-11 17:36 UTC by NukemBy
Modified: 2016-10-22 17:35 UTC (History)
1 user (show)

See Also:
Issue Type: DEFECT
:


Attachments
debug-unit-test-in6sec-1.nps (62.05 KB, application/octet-stream)
2016-09-11 17:36 UTC, NukemBy
Details
debug-unit-test-in6sec-2.nps (70.33 KB, application/octet-stream)
2016-09-11 17:37 UTC, NukemBy
Details
Self-profiler with stubbed out implementations of loadProject() and isProject() (56.63 KB, application/octet-stream)
2016-09-11 22:07 UTC, NukemBy
Details
debug-unit-test-in1.5sec.nps (33.76 KB, application/octet-stream)
2016-09-11 22:58 UTC, NukemBy
Details

Note You need to log in before you can comment on or make changes to this bug.
Description NukemBy 2016-09-11 17:36:14 UTC
Created attachment 162000 [details]
debug-unit-test-in6sec-1.nps

Hello,

I'm using NB built from recent sources (i.e. latest dev one) and Gradle plugin to integrate my Gradle projects into Netbeans.

It appears that execution of single simple (int i = 1 + 1) unit test takes 6..7+ seconds in IDE. 
Running the same without debug finishes in __less than 1sec__ - initialization of debugger takes ~6 seconds ... that's ahhh bad :(

I guess additional important note is that I have 30 modules opened simultaneously and each of them has 10-20 dependencies loaded from local gradle/maven repositories.

After debugging and self-profiling of NetBeans I noticed that NB excessively refreshes all dependencies and for some reason tries to load them and  as project. According to selfprofiler - verification of "if dependency is project or not" takes really much time and is invoked by netbeans really excessively.

For example - org.netbeans.modules.ide.ergonomics.fod.FeatureInfo.isProject() mentioned in hotspots takes 2.7 sec or 50% all time (BTW - i just have no idea what functionality "ide.ergonomics.fod" may add to NetBeans, can it be disabled somehow?).

Here is one of the sub-trees from self profiler (loadProject() occurs in other substrees/threads too - and probably they wait on each other)

	org.netbeans.modules.debugger.jpda.projectsui.MainProjectManager.getMainProject ()
	  org.netbeans.modules.parsing.spi.indexing.support.QuerySupport.findDependentRoots ()
		org.netbeans.modules.java.project.ProjectSourceForBinaryQuery.findSourceRoots ()
		   org.netbeans.modules.projectapi.SimpleFileOwnerQueryImplementation.getOwner ()
			  org.netbeans.modules.projectapi.nb.NbProjectManager.findProject ()
				 org.netbeans.modules.projectapi.nb.NbProjectManager.createProject ()
					 (... here it tries to load all kinds of supported projects ...)
					 org.netbeans.modules.ide.ergonomics.fod.FeatureProjectFactory.loadProject () - 807ms
					 org.netbeans.modules.project.ui.convertor.ProjectConvertorFactory.loadProject () - 557ms
					 org.netbeans.modules.project.ant.AntBasedProjectFactorySingleton.loadProject () - 347ms
					 org.netbeans.modules.groovy.grailsproject.GrailsProjectFactory.loadProject () - 118 ms
					 org.netbeans.modules.maven.NbMavenProjectFactory.loadProject () - 30ms
					 org.netbeans.gradle.project.NbGradleProjectFactory.loadProject () - 10ms


    Major reason of the slowdown here (as far as I understand) is that ...
    - out of few Projects explicitly created by Gradle plugin (30 projects) ...
    - Netbeans generates source root entry for each JAR dependency and its parent paths plus each sub-folder in each project

    I've logged what NB is doing - it looks like following

      "RepositoryUpdater.worker": NbProjectManager.createProject  "C:/Users/User/.gradle/caches/modules-2/files-2.1/org.gradle/gradle-base-services-groovy/2.13/e1fc0f8d0bb9f9ab3eff1fea7c6060c297c2de4"
      "RepositoryUpdater.worker": NbProjectManager.createProject  "C:/Users/User/.gradle/caches/modules-2/files-2.1/org.gradle/gradle-base-services-groovy/2.13"
      "RepositoryUpdater.worker": NbProjectManager.createProject  "C:/Users/User/.gradle/caches/modules-2/files-2.1/org.gradle/gradle-base-services-groovy"
      "RepositoryUpdater.worker": NbProjectManager.createProject  "C:/Users/User/.gradle/caches/modules-2/files-2.1/org.gradle"
      "RepositoryUpdater.worker": NbProjectManager.createProject  "C:/Users/User/.gradle/caches/modules-2/files-2.1"
      "RepositoryUpdater.worker": NbProjectManager.createProject  "C:/Users/User/.gradle/caches/modules-2"
      "RepositoryUpdater.worker": NbProjectManager.createProject  "C:/Users/User/.gradle/caches"
      "RepositoryUpdater.worker": NbProjectManager.createProject  "C:/Users/User/.gradle"
      "RepositoryUpdater.worker": NbProjectManager.createProject  "C:/Users/User"
      "RepositoryUpdater.worker": NbProjectManager.createProject  "C:/Users"
      "RepositoryUpdater.worker": NbProjectManager.createProject  "C:/"
      "RepositoryUpdater.worker": NbProjectManager.createProject  ""
      "RepositoryUpdater.worker": NbProjectManager.createProject  "C:/Users/User/.m2/repository/org/eclipse/jetty/jetty-util/8.1.14.v20131031"
      "RepositoryUpdater.worker": NbProjectManager.createProject  "C:/Users/User/.m2/repository/org/eclipse/jetty/jetty-util"
      "RepositoryUpdater.worker": NbProjectManager.createProject  "C:/Users/User/.m2/repository/org/eclipse/jetty"
      "RepositoryUpdater.worker": NbProjectManager.createProject  "C:/Users/User/.m2/repository/org/eclipse"
      "RepositoryUpdater.worker": NbProjectManager.createProject  "C:/Users/User/.m2/repository/org"
      "RepositoryUpdater.worker": NbProjectManager.createProject  "C:/Users/User/.m2/repository"
      "RepositoryUpdater.worker": NbProjectManager.createProject  "C:/Users/User/.m2"
      ...
      "RepositoryUpdater.worker": NbProjectManager.createProject  "C:/Prj/Client/DevGit/MainApp/site/src/site-main/site-main-demo/src/main/webapp/WEB-INF/controls/admin"
      "RepositoryUpdater.worker": NbProjectManager.createProject  "C:/Prj/Client/DevGit/MainApp/site/src/site-main/site-main-demo/src/main/webapp/WEB-INF/controls"
      "RepositoryUpdater.worker": NbProjectManager.createProject  "C:/Prj/Client/DevGit/MainApp/site/src/site-main/site-main-demo/src/main/webapp/WEB-INF"
      "RepositoryUpdater.worker": NbProjectManager.createProject  "C:/Prj/Client/DevGit/MainApp/site/src/site-main/site-main-demo/src/main/webapp/WEB-INF/controls/demo"
      "RepositoryUpdater.worker": NbProjectManager.createProject  "C:/Prj/Client/DevGit/MainApp/site/src/site-main/site-main-demo/src/main/webapp/WEB-INF/pages/admin"
      "RepositoryUpdater.worker": NbProjectManager.createProject  "C:/Prj/Client/DevGit/MainApp/site/src/site-main/site-main-demo/src/main/webapp/WEB-INF/pages"
      "RepositoryUpdater.worker": NbProjectManager.createProject  "C:/Prj/Client/DevGit/MainApp/site"
      "RepositoryUpdater.worker": NbProjectManager.createProject  "C:/Prj/Client/DevGit/MainApp"
      "RepositoryUpdater.worker": NbProjectManager.createProject  "C:/Prj/Client/DevGit"
      "RepositoryUpdater.worker": NbProjectManager.createProject  "C:/Prj/Client"
      "RepositoryUpdater.worker": NbProjectManager.createProject  "C:/Prj"
      ...
	  
	In total it is about 800 directories. Each of them is fed into each of 5 Project*Factory'ies - so ... one cycle of "RepositoryUpdater.worker" takes rather long.
	Note: NbGradleProjectFactory consumes only 10ms - that is because major set of folder are cut off by that check. I thinks it has sense to make better-performing isProject() verification for other projects factories too

			public boolean isProject(FileObject projectDirectory) {
				String path = projectDirectory.getPath();
				
				if (path.contains(".gradle/caches") || path.contains(".m2/repository"))

	OK, and that is not the end ... it appears that clicking/switching between window causes rescanning of that huge number of 'fake' sources roots in other threads:

		"Editor Parsing Loop (20160910-b14d1f3f39a0)"
		"Refresh Editor Context"
		"org.netbeans.modules.java.hints.declarative.DeclarativeHintRegistry

	... and finally when it comes to running a unit test - all this has contention with debugger threads, which also scan source roots 
	
		"Debugger attach listener-thread-3"
		"Debugger operator thread"

	=========
    QUESTION: Why all these threads are doing same thing (NbProjectManager.createProject())	in parallel? 
	          Can it be done in some really "worker" thread? And others just consume status from worker thread?
	=========
		
One more thing, there are some absolutely dumb source roots - point to folders I never opened in last 2-3 month and they are not mentioned in any of the currently open projects.
These 2 source roots come from thread named "org.openide.text Editor Initialization"

	"org.openide.text Editor Initialization": NbProjectManager.createProject  "D:/test/project/code/filter-dataset/src/main/java/test/filter/dataset"
	

------------------------------------------------------------------------------
   APPENDIX	- some call stacks coming into NbProjectManager.createProject()
------------------------------------------------------------------------------

"RepositoryUpdater.worker"
	at org.netbeans.modules.projectapi.nb.NbProjectManager.createProject(NbProjectManager.java:401)
	at org.netbeans.modules.projectapi.nb.NbProjectManager.access$300(NbProjectManager.java:94)
	at org.netbeans.modules.projectapi.nb.NbProjectManager$2.run(NbProjectManager.java:314)
	at org.netbeans.modules.projectapi.nb.NbProjectManager$2.run(NbProjectManager.java:243)
	at org.netbeans.modules.openide.util.DefaultMutexImplementation.readAccess(DefaultMutexImplementation.java:209)
	at org.openide.util.Mutex.readAccess(Mutex.java:250)
	at org.netbeans.modules.projectapi.nb.NbProjectManager.findProject(NbProjectManager.java:243)
	at org.netbeans.api.project.ProjectManager.findProject(ProjectManager.java:167)
	at org.netbeans.modules.projectapi.SimpleFileOwnerQueryImplementation.getOwner(SimpleFileOwnerQueryImplementation.java:161)
	at org.netbeans.modules.projectapi.SimpleFileOwnerQueryImplementation.getOwner(SimpleFileOwnerQueryImplementation.java:113)
	at org.netbeans.api.project.FileOwnerQuery.getOwner(FileOwnerQuery.java:150)
	at org.netbeans.modules.java.project.ProjectSourceForBinaryQuery.findSourceRoots2(ProjectSourceForBinaryQuery.java:79)
	at org.netbeans.api.java.queries.SourceForBinaryQuery.findSourceRoots2(SourceForBinaryQuery.java:126)
	at org.netbeans.modules.parsing.impl.indexing.PathRegistry.createResources(PathRegistry.java:750)
	at org.netbeans.modules.parsing.impl.indexing.PathRegistry.getSources(PathRegistry.java:297)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$RootsWork.getDone(RepositoryUpdater.java:4903)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Work.doTheWork(RepositoryUpdater.java:3435)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Task._run(RepositoryUpdater.java:6198)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Task.access$4300(RepositoryUpdater.java:5849)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Task$2$1.run(RepositoryUpdater.java:6114)
	at org.openide.util.lookup.Lookups.executeWith(Lookups.java:304)
	at org.netbeans.modules.parsing.impl.RunWhenScanFinishedSupport.performScan(RunWhenScanFinishedSupport.java:106)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Task$2.call(RepositoryUpdater.java:6110)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Task$2.call(RepositoryUpdater.java:6106)
	at org.netbeans.modules.masterfs.filebasedfs.utils.FileChangedManager.priorityIO(FileChangedManager.java:176)
	at org.netbeans.modules.masterfs.providers.ProvidedExtensions.priorityIO(ProvidedExtensions.java:360)
	at org.netbeans.modules.parsing.nb.DataObjectEnvFactory.runPriorityIO(DataObjectEnvFactory.java:141)
	at org.netbeans.modules.parsing.impl.Utilities.runPriorityIO(Utilities.java:88)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Task.run(RepositoryUpdater.java:6106)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1443)
	at org.netbeans.modules.openide.util.GlobalLookup.execute(GlobalLookup.java:68)
	at org.openide.util.lookup.Lookups.executeWith(Lookups.java:303)
	at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:2058)
	
	
"Refresh Editor Context"  ( jpda.projectsui.RunToCursorActionProvider? )
	at org.netbeans.modules.projectapi.nb.NbProjectManager.createProject(NbProjectManager.java:401)
	at org.netbeans.modules.projectapi.nb.NbProjectManager.access$300(NbProjectManager.java:94)
	at org.netbeans.modules.projectapi.nb.NbProjectManager$2.run(NbProjectManager.java:314)
	at org.netbeans.modules.projectapi.nb.NbProjectManager$2.run(NbProjectManager.java:243)
	at org.netbeans.modules.openide.util.DefaultMutexImplementation.readAccess(DefaultMutexImplementation.java:209)
	at org.openide.util.Mutex.readAccess(Mutex.java:250)
	at org.netbeans.modules.projectapi.nb.NbProjectManager.findProject(NbProjectManager.java:243)
	at org.netbeans.api.project.ProjectManager.findProject(ProjectManager.java:167)
	at org.netbeans.modules.projectapi.SimpleFileOwnerQueryImplementation.getOwner(SimpleFileOwnerQueryImplementation.java:161)
	at org.netbeans.modules.projectapi.SimpleFileOwnerQueryImplementation.getOwner(SimpleFileOwnerQueryImplementation.java:113)
	at org.netbeans.api.project.FileOwnerQuery.getOwner(FileOwnerQuery.java:150)
	at org.netbeans.modules.java.project.ProjectSourceForBinaryQuery.findSourceRoots(ProjectSourceForBinaryQuery.java:67)
	at org.netbeans.api.java.queries.SourceForBinaryQuery.findSourceRoots(SourceForBinaryQuery.java:96)
	at org.netbeans.modules.parsing.spi.indexing.support.QuerySupport.findBinaryRootsForSourceRoot(QuerySupport.java:871)
	at org.netbeans.modules.parsing.spi.indexing.support.QuerySupport.findDependentRoots(QuerySupport.java:194)
	at org.netbeans.api.java.source.SourceUtils.getDependentRoots(SourceUtils.java:791)
	at org.netbeans.api.java.source.SourceUtils.getDependentRoots(SourceUtils.java:769)
	at org.netbeans.modules.debugger.jpda.projectsui.MainProjectManager.isDependent(MainProjectManager.java:173)
	at org.netbeans.modules.debugger.jpda.projectsui.MainProjectManager.getMainProject(MainProjectManager.java:121)
	at org.netbeans.modules.debugger.jpda.projectsui.RunToCursorActionProvider.shouldBeEnabled(RunToCursorActionProvider.java:210)
	at org.netbeans.modules.debugger.jpda.projectsui.RunToCursorActionProvider.access$300(RunToCursorActionProvider.java:81)
	at org.netbeans.modules.debugger.jpda.projectsui.RunToCursorActionProvider$Listener.propertyChange(RunToCursorActionProvider.java:282)
	at org.openide.util.WeakListenerImpl$PropertyChange.propertyChange(WeakListenerImpl.java:215)
	at java.beans.PropertyChangeSupport.fire(PropertyChangeSupport.java:335)
	at java.beans.PropertyChangeSupport.firePropertyChange(PropertyChangeSupport.java:327)
	at org.netbeans.spi.debugger.ui.EditorContextDispatcher.firePropertyChange(EditorContextDispatcher.java:614)
	at org.netbeans.spi.debugger.ui.EditorContextDispatcher.access$2000(EditorContextDispatcher.java:122)
	at org.netbeans.spi.debugger.ui.EditorContextDispatcher$EventFirer.run(EditorContextDispatcher.java:1011)
	at org.netbeans.spi.debugger.ui.EditorContextDispatcher$EditorRegistryListener$1.run(EditorContextDispatcher.java:808)
	at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1443)
	at org.netbeans.modules.openide.util.GlobalLookup.execute(GlobalLookup.java:68)
	at org.openide.util.lookup.Lookups.executeWith(Lookups.java:303)
	at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:2058)
	
"Debugger attach listener-thread-9"
	at org.netbeans.modules.projectapi.nb.NbProjectManager.createProject(NbProjectManager.java:401)
	at org.netbeans.modules.projectapi.nb.NbProjectManager.access$300(NbProjectManager.java:94)
	at org.netbeans.modules.projectapi.nb.NbProjectManager$2.run(NbProjectManager.java:314)
	at org.netbeans.modules.projectapi.nb.NbProjectManager$2.run(NbProjectManager.java:243)
	at org.netbeans.modules.openide.util.DefaultMutexImplementation.readAccess(DefaultMutexImplementation.java:209)
	at org.openide.util.Mutex.readAccess(Mutex.java:250)
	at org.netbeans.modules.projectapi.nb.NbProjectManager.findProject(NbProjectManager.java:243)
	at org.netbeans.api.project.ProjectManager.findProject(ProjectManager.java:167)
	at org.netbeans.modules.projectapi.SimpleFileOwnerQueryImplementation.getOwner(SimpleFileOwnerQueryImplementation.java:161)
	at org.netbeans.modules.projectapi.SimpleFileOwnerQueryImplementation.getOwner(SimpleFileOwnerQueryImplementation.java:113)
	at org.netbeans.api.project.FileOwnerQuery.getOwner(FileOwnerQuery.java:150)
	at org.netbeans.modules.java.project.ProjectSourceForBinaryQuery.findSourceRoots(ProjectSourceForBinaryQuery.java:67)
	at org.netbeans.api.java.queries.SourceForBinaryQuery.findSourceRoots(SourceForBinaryQuery.java:96)
	at org.netbeans.modules.parsing.spi.indexing.support.QuerySupport.findBinaryRootsForSourceRoot(QuerySupport.java:871)
	at org.netbeans.modules.parsing.spi.indexing.support.QuerySupport.findDependentRoots(QuerySupport.java:194)
	at org.netbeans.api.java.source.SourceUtils.getDependentRoots(SourceUtils.java:791)
	at org.netbeans.api.java.source.SourceUtils.getDependentRoots(SourceUtils.java:769)
	at org.netbeans.modules.debugger.jpda.projectsui.MainProjectManager.isDependent(MainProjectManager.java:173)
	at org.netbeans.modules.debugger.jpda.projectsui.MainProjectManager.getMainProject(MainProjectManager.java:121)
	at org.netbeans.modules.debugger.jpda.projectsui.StepIntoActionProvider.shouldBeEnabled(StepIntoActionProvider.java:152)
	at org.netbeans.modules.debugger.jpda.projectsui.StepIntoActionProvider.access$200(StepIntoActionProvider.java:77)
	at org.netbeans.modules.debugger.jpda.projectsui.StepIntoActionProvider$Listener.doSetEnabled(StepIntoActionProvider.java:223)
	at org.netbeans.modules.debugger.jpda.projectsui.StepIntoActionProvider$Listener.engineAdded(StepIntoActionProvider.java:213)
	at sun.reflect.GeneratedMethodAccessor194.invoke
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.openide.util.WeakListenerImpl$ProxyListener.invoke(WeakListenerImpl.java:487)
	at com.sun.proxy.$Proxy11.engineAdded
	at org.netbeans.api.debugger.DebuggerManager.fireEngineAdded(DebuggerManager.java:1455)
	at org.netbeans.api.debugger.DebuggerManager.addEngine(DebuggerManager.java:1679)
	at org.netbeans.api.debugger.Session.addLanguage(Session.java:375)
	at org.netbeans.api.debugger.DebuggerManager.startDebugging(DebuggerManager.java:381)
	at org.netbeans.api.debugger.jpda.JPDADebugger.startListeningAndGetEngines(JPDADebugger.java:270)
	at org.netbeans.api.debugger.jpda.JPDADebugger.startListening(JPDADebugger.java:239)
	at org.netbeans.gradle.project.java.tasks.DebuggerServiceFactory$1.execute(DebuggerServiceFactory.java:88)
	at org.jtrim.concurrent.AbstractTaskExecutorService$1.execute(AbstractTaskExecutorService.java:142)
	at org.jtrim.concurrent.AbstractTaskExecutorService$FunctionWrapper.execute(AbstractTaskExecutorService.java:270)
	at org.jtrim.concurrent.AbstractTaskExecutorService$TaskOfAbstractExecutor.execute(AbstractTaskExecutorService.java:340)
	at org.jtrim.concurrent.Tasks$RunOnceCancelableTask.execute(Tasks.java:342)
	at org.jtrim.concurrent.ThreadPoolTaskExecutor$ThreadPoolTaskExecutorImpl$QueuedItem.runTask(ThreadPoolTaskExecutor.java:1213)
	at org.jtrim.concurrent.ThreadPoolTaskExecutor$ThreadPoolTaskExecutorImpl$Worker.executeTask(ThreadPoolTaskExecutor.java:1049)
	at org.jtrim.concurrent.ThreadPoolTaskExecutor$ThreadPoolTaskExecutorImpl$Worker.run(ThreadPoolTaskExecutor.java:1179)
	at org.jtrim.concurrent.ThreadPoolTaskExecutor$ThreadPoolTaskExecutorImpl$Worker$1.run(ThreadPoolTaskExecutor.java:998)
	at java.lang.Thread.run(Thread.java:745)

	
"Debugger operator thread"
	at org.netbeans.modules.projectapi.nb.NbProjectManager.createProject(NbProjectManager.java:401)
	at org.netbeans.modules.projectapi.nb.NbProjectManager.access$300(NbProjectManager.java:94)
	at org.netbeans.modules.projectapi.nb.NbProjectManager$2.run(NbProjectManager.java:314)
	at org.netbeans.modules.projectapi.nb.NbProjectManager$2.run(NbProjectManager.java:243)
	at org.netbeans.modules.openide.util.DefaultMutexImplementation.readAccess(DefaultMutexImplementation.java:209)
	at org.openide.util.Mutex.readAccess(Mutex.java:250)
	at org.netbeans.modules.projectapi.nb.NbProjectManager.findProject(NbProjectManager.java:243)
	at org.netbeans.api.project.ProjectManager.findProject(ProjectManager.java:167)
	at org.netbeans.modules.projectapi.SimpleFileOwnerQueryImplementation.getOwner(SimpleFileOwnerQueryImplementation.java:161)
	at org.netbeans.modules.projectapi.SimpleFileOwnerQueryImplementation.getOwner(SimpleFileOwnerQueryImplementation.java:113)
	at org.netbeans.api.project.FileOwnerQuery.getOwner(FileOwnerQuery.java:150)
	at org.netbeans.modules.java.project.ProjectSourceForBinaryQuery.findSourceRoots(ProjectSourceForBinaryQuery.java:67)
	at org.netbeans.api.java.queries.SourceForBinaryQuery.findSourceRoots(SourceForBinaryQuery.java:96)
	at org.netbeans.modules.parsing.spi.indexing.support.QuerySupport.findBinaryRootsForSourceRoot(QuerySupport.java:871)
	at org.netbeans.modules.parsing.spi.indexing.support.QuerySupport.findDependentRoots(QuerySupport.java:194)
	at org.netbeans.api.java.source.SourceUtils.getDependentRoots(SourceUtils.java:791)
	at org.netbeans.api.java.source.SourceUtils.getDependentRoots(SourceUtils.java:769)
	at org.netbeans.modules.debugger.jpda.projectsui.MainProjectManager.isDependent(MainProjectManager.java:173)
	at org.netbeans.modules.debugger.jpda.projectsui.MainProjectManager.getMainProject(MainProjectManager.java:121)
	at org.netbeans.modules.debugger.jpda.projectsui.StepIntoActionProvider.shouldBeEnabled(StepIntoActionProvider.java:152)
	at org.netbeans.modules.debugger.jpda.projectsui.StepIntoActionProvider.access$200(StepIntoActionProvider.java:77)
	at org.netbeans.modules.debugger.jpda.projectsui.StepIntoActionProvider$Listener.doSetEnabled(StepIntoActionProvider.java:223)
	at org.netbeans.modules.debugger.jpda.projectsui.StepIntoActionProvider$Listener.engineRemoved(StepIntoActionProvider.java:217)
	at sun.reflect.GeneratedMethodAccessor184.invoke
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.openide.util.WeakListenerImpl$ProxyListener.invoke(WeakListenerImpl.java:487)
	at com.sun.proxy.$Proxy11.engineRemoved
	at org.netbeans.api.debugger.DebuggerManager.fireEngineRemoved(DebuggerManager.java:1506)
	at org.netbeans.api.debugger.DebuggerManager.removeEngine(DebuggerManager.java:1691)
	at org.netbeans.api.debugger.Session.removeEngine(Session.java:443)
	at org.netbeans.api.debugger.DebuggerEngine$Destructor.killEngine(DebuggerEngine.java:241)
	at org.netbeans.modules.debugger.jpda.JPDADebuggerImpl.finish(JPDADebuggerImpl.java:1514)
	at org.netbeans.modules.debugger.jpda.actions.StartActionProvider$4.run(StartActionProvider.java:281)
	at org.netbeans.modules.debugger.jpda.util.Operator$1.run(Operator.java:271)
	at org.netbeans.modules.openide.util.GlobalLookup.execute(GlobalLookup.java:68)
	at org.openide.util.lookup.Lookups.executeWith(Lookups.java:303)
	at org.netbeans.modules.debugger.jpda.util.Operator$2.run(Operator.java:282)
	at java.lang.Thread.run(Thread.java:745)	
	
"Editor Parsing Loop (20160910-b14d1f3f39a0)"
	at org.netbeans.modules.projectapi.nb.NbProjectManager.createProject(NbProjectManager.java:401)
	at org.netbeans.modules.projectapi.nb.NbProjectManager.access$300(NbProjectManager.java:94)
	at org.netbeans.modules.projectapi.nb.NbProjectManager$2.run(NbProjectManager.java:314)
	at org.netbeans.modules.projectapi.nb.NbProjectManager$2.run(NbProjectManager.java:243)
	at org.netbeans.modules.openide.util.DefaultMutexImplementation.readAccess(DefaultMutexImplementation.java:209)
	at org.openide.util.Mutex.readAccess(Mutex.java:250)
	at org.netbeans.modules.projectapi.nb.NbProjectManager.findProject(NbProjectManager.java:243)
	at org.netbeans.api.project.ProjectManager.findProject(ProjectManager.java:167)
	at org.netbeans.modules.projectapi.SimpleFileOwnerQueryImplementation.getOwner(SimpleFileOwnerQueryImplementation.java:161)
	at org.netbeans.api.project.FileOwnerQuery.getOwner(FileOwnerQuery.java:116)
	at org.netbeans.modules.projectapi.ProjectFileEncodingQueryImplementation.getEncoding(ProjectFileEncodingQueryImplementation.java:72)
	at org.netbeans.api.queries.FileEncodingQuery.getEncoding(FileEncodingQuery.java:96)
	at org.netbeans.modules.parsing.api.Source.createSnapshot(Source.java:368)
	at org.netbeans.modules.java.source.parsing.SourceFileObject.createContent(SourceFileObject.java:158)
	at org.netbeans.modules.java.source.parsing.AbstractSourceFileObject.getContent(AbstractSourceFileObject.java:384)
	at org.netbeans.modules.java.source.parsing.AbstractSourceFileObject.getCharContent(AbstractSourceFileObject.java:149)
	at org.netbeans.modules.java.source.parsing.AbstractSourceFileObject.getCharContent(AbstractSourceFileObject.java:84)
	at com.sun.tools.javac.main.JavaCompiler.readSource(JavaCompiler.java:592)
	at com.sun.tools.javac.main.JavaCompiler.parse(JavaCompiler.java:654)
	at com.sun.tools.javac.main.JavaCompiler.parseFiles(JavaCompiler.java:976)
	at com.sun.tools.javac.api.JavacTaskImpl.parseInternal(JavacTaskImpl.java:283)
	at com.sun.tools.javac.api.JavacTaskImpl.access$400(JavacTaskImpl.java:74)
	at com.sun.tools.javac.api.JavacTaskImpl$2.call(JavacTaskImpl.java:275)
	at com.sun.tools.javac.api.JavacTaskImpl$2.call(JavacTaskImpl.java:272)
	at com.sun.tools.javac.api.JavacTaskImpl.handleExceptions(JavacTaskImpl.java:139)
	at com.sun.tools.javac.api.JavacTaskImpl.parse(JavacTaskImpl.java:272)
	at com.sun.tools.javac.api.JavacTaskImpl.parse(JavacTaskImpl.java:252)
	at org.netbeans.modules.java.source.TreeLoader.loadTreeFor(TreeLoader.java:199)
	at com.sun.tools.javac.code.Symbol$ParamSymbol.getSimpleName(Symbol.java:1476)
	at com.sun.tools.javac.code.Symbol$ParamSymbol.getSimpleName(Symbol.java:1443)
	at org.netbeans.modules.java.navigation.ElementScanningTask.createHtmlHeader(ElementScanningTask.java:327)
	at org.netbeans.modules.java.navigation.ElementScanningTask.element2description(ElementScanningTask.java:270)
	at org.netbeans.modules.java.navigation.ElementScanningTask.addMembers(ElementScanningTask.java:230)
	at org.netbeans.modules.java.navigation.ElementScanningTask.addMembers(ElementScanningTask.java:236)
	at org.netbeans.modules.java.navigation.ElementScanningTask.runImpl(ElementScanningTask.java:147)
	at org.netbeans.modules.java.navigation.ElementScanningTask.run(ElementScanningTask.java:105)
	at org.netbeans.modules.java.navigation.ElementScanningTask.run(ElementScanningTask.java:86)
	at org.netbeans.modules.java.source.JavaSourceAccessor$CancelableTaskWrapper.run(JavaSourceAccessor.java:298)
	at org.netbeans.modules.parsing.impl.TaskProcessor.callParserResultTask(TaskProcessor.java:584)
	at org.netbeans.modules.parsing.impl.TaskProcessor$RequestPerformer.run(TaskProcessor.java:809)
	at org.openide.util.lookup.Lookups.executeWith(Lookups.java:304)
	at org.netbeans.modules.parsing.impl.TaskProcessor$RequestPerformer.execute(TaskProcessor.java:725)
	at org.netbeans.modules.parsing.impl.TaskProcessor$CompilationJob.run(TaskProcessor.java:686)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1443)
	at org.netbeans.modules.openide.util.GlobalLookup.execute(GlobalLookup.java:68)
	at org.openide.util.lookup.Lookups.executeWith(Lookups.java:303)
	at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:2058)
Comment 1 NukemBy 2016-09-11 17:37:41 UTC
Created attachment 162001 [details]
debug-unit-test-in6sec-2.nps
Comment 2 NukemBy 2016-09-11 20:04:31 UTC
I've done some more debugging of NB to resolve my curiosity regarding why NbGradleProjectFactory.loadProject() is called for resource folders - it appears that traces go from various places and overall design of invocation of rather expensive NbProjectManager.createProject() in plenty of weird use-cases is bad:

"RepositoryUpdater.worker"
	at org.netbeans.gradle.project.NbGradleProjectFactory.isProject(NbGradleProjectFactory.java:176)
	at org.netbeans.gradle.project.NbGradleProjectFactory.loadProject(NbGradleProjectFactory.java:225)
	at org.netbeans.modules.projectapi.nb.NbProjectManager.createProject(NbProjectManager.java:401)
	at org.netbeans.modules.projectapi.nb.NbProjectManager.access$300(NbProjectManager.java:94)
	at org.netbeans.modules.projectapi.nb.NbProjectManager$2.run(NbProjectManager.java:314)
	at org.netbeans.modules.projectapi.nb.NbProjectManager$2.run(NbProjectManager.java:243)
	at org.netbeans.modules.openide.util.DefaultMutexImplementation.readAccess(DefaultMutexImplementation.java:209)
	at org.openide.util.Mutex.readAccess(Mutex.java:250)
	at org.netbeans.modules.projectapi.nb.NbProjectManager.findProject(NbProjectManager.java:243)
	at org.netbeans.api.project.ProjectManager.findProject(ProjectManager.java:167)
	at org.netbeans.modules.projectapi.SimpleFileOwnerQueryImplementation.getOwner(SimpleFileOwnerQueryImplementation.java:161)
	at org.netbeans.api.project.FileOwnerQuery.getOwner(FileOwnerQuery.java:116)
	at org.netbeans.modules.projectapi.ProjectFileEncodingQueryImplementation.getEncoding(ProjectFileEncodingQueryImplementation.java:72)
	at org.netbeans.api.queries.FileEncodingQuery.getEncoding(FileEncodingQuery.java:96)
	at org.netbeans.modules.parsing.api.Source.createSnapshot(Source.java:368)
	at org.netbeans.modules.parsing.impl.SourceCache.createSnapshot(SourceCache.java:180)
	at org.netbeans.modules.parsing.impl.SourceCache.getSnapshot(SourceCache.java:155)
	at org.netbeans.modules.parsing.api.ResultIterator.getSnapshot(ResultIterator.java:109)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Work$2.run(RepositoryUpdater.java:3172)
	at org.netbeans.modules.parsing.impl.TaskProcessor.callUserTask(TaskProcessor.java:609)
	at org.netbeans.modules.parsing.api.ParserManager$UserTaskAction.run(ParserManager.java:153)
	at org.netbeans.modules.parsing.api.ParserManager$UserTaskAction.run(ParserManager.java:137)
	at org.netbeans.modules.parsing.impl.TaskProcessor$2.call(TaskProcessor.java:204)
	at org.netbeans.modules.parsing.impl.TaskProcessor$2.call(TaskProcessor.java:201)
	at org.netbeans.modules.masterfs.filebasedfs.utils.FileChangedManager.priorityIO(FileChangedManager.java:176)
	at org.netbeans.modules.masterfs.providers.ProvidedExtensions.priorityIO(ProvidedExtensions.java:360)
	at org.netbeans.modules.parsing.nb.DataObjectEnvFactory.runPriorityIO(DataObjectEnvFactory.java:141)
	at org.netbeans.modules.parsing.impl.Utilities.runPriorityIO(Utilities.java:88)
	at org.netbeans.modules.parsing.impl.TaskProcessor.runUserTask(TaskProcessor.java:201)
	at org.netbeans.modules.parsing.api.ParserManager.parse(ParserManager.java:104)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Work.indexEmbedding(RepositoryUpdater.java:3169)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Work.doIndex(RepositoryUpdater.java:2885)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Work.access$800(RepositoryUpdater.java:2177)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Work$1.run(RepositoryUpdater.java:2659)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Work$1.run(RepositoryUpdater.java:2657)
	at org.netbeans.modules.parsing.impl.indexing.errors.TaskCache.refreshTransaction(TaskCache.java:565)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Work.index(RepositoryUpdater.java:2657)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$AbstractRootsWork$4.call(RepositoryUpdater.java:5729)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$AbstractRootsWork$4.call(RepositoryUpdater.java:5637)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$4.run(RepositoryUpdater.java:2150)
	at org.openide.util.lookup.Lookups.executeWith(Lookups.java:304)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater.runInContext(RepositoryUpdater.java:2146)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater.runInContext(RepositoryUpdater.java:2127)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater.access$1500(RepositoryUpdater.java:159)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$AbstractRootsWork.scanSource(RepositoryUpdater.java:5764)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$AbstractRootsWork.scanSources(RepositoryUpdater.java:5434)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$RootsWork.getDone(RepositoryUpdater.java:5053)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Work.doTheWork(RepositoryUpdater.java:3435)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Task._run(RepositoryUpdater.java:6198)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Task.access$4300(RepositoryUpdater.java:5849)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Task$2$1.run(RepositoryUpdater.java:6114)
	at org.openide.util.lookup.Lookups.executeWith(Lookups.java:304)
	at org.netbeans.modules.parsing.impl.RunWhenScanFinishedSupport.performScan(RunWhenScanFinishedSupport.java:106)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Task$2.call(RepositoryUpdater.java:6110)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Task$2.call(RepositoryUpdater.java:6106)
	at org.netbeans.modules.masterfs.filebasedfs.utils.FileChangedManager.priorityIO(FileChangedManager.java:176)
	at org.netbeans.modules.masterfs.providers.ProvidedExtensions.priorityIO(ProvidedExtensions.java:360)
	at org.netbeans.modules.parsing.nb.DataObjectEnvFactory.runPriorityIO(DataObjectEnvFactory.java:141)
	at org.netbeans.modules.parsing.impl.Utilities.runPriorityIO(Utilities.java:88)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Task.run(RepositoryUpdater.java:6106)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1443)
	at org.netbeans.modules.openide.util.GlobalLookup.execute(GlobalLookup.java:68)
	at org.openide.util.lookup.Lookups.executeWith(Lookups.java:303)
	at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:2058)"PackageViewChildren-accessibility"
	at org.netbeans.modules.projectapi.nb.NbProjectManager.createProject(NbProjectManager.java:401)
	at org.netbeans.modules.projectapi.nb.NbProjectManager.access$300(NbProjectManager.java:94)
	at org.netbeans.modules.projectapi.nb.NbProjectManager$2.run(NbProjectManager.java:314)
	at org.netbeans.modules.projectapi.nb.NbProjectManager$2.run(NbProjectManager.java:243)
	at org.netbeans.modules.openide.util.DefaultMutexImplementation.readAccess(DefaultMutexImplementation.java:209)
	at org.openide.util.Mutex.readAccess(Mutex.java:250)
	at org.netbeans.modules.projectapi.nb.NbProjectManager.findProject(NbProjectManager.java:243)
	at org.netbeans.api.project.ProjectManager.findProject(ProjectManager.java:167)
	at org.netbeans.modules.projectapi.SimpleFileOwnerQueryImplementation.getOwner(SimpleFileOwnerQueryImplementation.java:161)
	at org.netbeans.api.project.FileOwnerQuery.getOwner(FileOwnerQuery.java:116)
	at org.netbeans.modules.java.project.ProjectAccessibilityQuery.isPubliclyAccessible(ProjectAccessibilityQuery.java:63)
	at org.netbeans.api.java.queries.AccessibilityQuery.isPubliclyAccessible(AccessibilityQuery.java:97)
	at org.netbeans.spi.java.project.support.ui.PackageViewChildren$PackageNode$1.run(PackageViewChildren.java:1117)
	at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1443)
	at org.netbeans.modules.openide.util.GlobalLookup.execute(GlobalLookup.java:68)
	at org.openide.util.lookup.Lookups.executeWith(Lookups.java:303)
	at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:2058)

"ErrorAnnotator worker"
	at org.netbeans.modules.projectapi.nb.NbProjectManager.createProject(NbProjectManager.java:401)
	at org.netbeans.modules.projectapi.nb.NbProjectManager.access$300(NbProjectManager.java:94)
	at org.netbeans.modules.projectapi.nb.NbProjectManager$2.run(NbProjectManager.java:314)
	at org.netbeans.modules.projectapi.nb.NbProjectManager$2.run(NbProjectManager.java:243)
	at org.netbeans.modules.openide.util.DefaultMutexImplementation.readAccess(DefaultMutexImplementation.java:209)
	at org.openide.util.Mutex.readAccess(Mutex.java:250)
	at org.netbeans.modules.projectapi.nb.NbProjectManager.findProject(NbProjectManager.java:243)
	at org.netbeans.api.project.ProjectManager.findProject(ProjectManager.java:167)
	at org.netbeans.modules.projectapi.SimpleFileOwnerQueryImplementation.getOwner(SimpleFileOwnerQueryImplementation.java:161)
	at org.netbeans.api.project.FileOwnerQuery.getOwner(FileOwnerQuery.java:116)
	at org.netbeans.modules.java.project.ProjectClassPathProvider.findClassPath(ProjectClassPathProvider.java:68)
	at org.netbeans.api.java.classpath.ClassPath.getClassPath(ClassPath.java:676)
	at org.netbeans.modules.parsing.impl.indexing.errors.Utilities.getSourceClassPathFor(Utilities.java:86)
	at org.netbeans.modules.parsing.ui.indexing.errors.ErrorAnnotator$1.run(ErrorAnnotator.java:313)
	at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1443)
	at org.netbeans.modules.openide.util.GlobalLookup.execute(GlobalLookup.java:68)
	at org.openide.util.lookup.Lookups.executeWith(Lookups.java:303)
	at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:2058)

"Git.cache"
	at org.netbeans.modules.projectapi.nb.NbProjectManager.createProject(NbProjectManager.java:401)
	at org.netbeans.modules.projectapi.nb.NbProjectManager.access$300(NbProjectManager.java:94)
	at org.netbeans.modules.projectapi.nb.NbProjectManager$2.run(NbProjectManager.java:314)
	at org.netbeans.modules.projectapi.nb.NbProjectManager$2.run(NbProjectManager.java:243)
	at org.netbeans.modules.openide.util.DefaultMutexImplementation.readAccess(DefaultMutexImplementation.java:209)
	at org.openide.util.Mutex.readAccess(Mutex.java:250)
	at org.netbeans.modules.projectapi.nb.NbProjectManager.findProject(NbProjectManager.java:243)
	at org.netbeans.api.project.ProjectManager.findProject(ProjectManager.java:167)
	at org.netbeans.modules.projectapi.SimpleFileOwnerQueryImplementation.getOwner(SimpleFileOwnerQueryImplementation.java:161)
	at org.netbeans.modules.projectapi.SimpleFileOwnerQueryImplementation.getOwner(SimpleFileOwnerQueryImplementation.java:113)
	at org.netbeans.api.project.FileOwnerQuery.getOwner(FileOwnerQuery.java:150)
	at org.netbeans.modules.projectapi.ProjectSharabilityQuery2.getSharability(ProjectSharabilityQuery2.java:63)
	at org.netbeans.api.queries.SharabilityQuery.getSharability(SharabilityQuery.java:177)
	at org.netbeans.modules.git.utils.GitUtils.isIgnored(GitUtils.java:250)
	at org.netbeans.modules.git.FileStatusCache$IgnoredFilesHandler.run(FileStatusCache.java:874)
	at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1443)
	at org.netbeans.modules.openide.util.GlobalLookup.execute(GlobalLookup.java:68)
	at org.openide.util.lookup.Lookups.executeWith(Lookups.java:303)
	at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:2058)
Comment 3 NukemBy 2016-09-11 22:06:41 UTC
After some more debugging of NB I expect that startup of the debugger for unit test can be done with CLOSE-TO-ZERO LATENCY by fixing implementation of just two methods:

  * MainProjectManager.getMainProject()
  * SimpleFileOwnerQueryImplementation.getOwner() - this also significantly speeds up initial load of projects & background scan

See details below ...

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

To fix issue with NbProjectManager.createProject() I stubbed out implementation of loadProject() and isProject() - by just returning 'null' and 'false' for implementation of these projects I currently do not use actually:

	org.netbeans.modules.ide.ergonomics.fod.FeatureProjectFactory.loadProject () - 807ms
	org.netbeans.modules.project.ui.convertor.ProjectConvertorFactory.loadProject () - 557ms
	org.netbeans.modules.project.ant.AntBasedProjectFactorySingleton.loadProject () - 347ms
	org.netbeans.modules.groovy.grailsproject.GrailsProjectFactory.loadProject () - 118 ms

    these ones are in use, but they are "quick"
	/* org.netbeans.modules.maven.NbMavenProjectFactory.loadProject () - 30ms */
	/* org.netbeans.gradle.project.NbGradleProjectFactory.loadProject () - 10ms */

After that unit test started to execute in around 2.5sec (1sec goes to Gradle itself - 1.5sec goes to NB), was 6.5 secs. I've started self profiler to see what's happening now (text dump is provided below) ...

I see that now ALL time goes to a single method - MainProjectManager.getMainProject() and consecutive MainProjectManager.isDependent(), invoked multiple times.
Current implementation checks if one project is dependent on another by recalculating intersection of source roots EACH time getMainProject() is called.
I actually see no reason to do this - project dependency changes only when some project changes. This needs to be calculated only once.

Source code of mention functions - for quick reference:

    public class SimpleFileOwnerQueryImplementation implements FileOwnerQueryImplementation {
        public Project getOwner(FileObject f) {
            ...
            while (f != null) {
                        Project p;
                        try {
                            p = ProjectManager.getDefault().findProject(f);
                        } catch (IOException e) {
                            // There is a project here, but we cannot load it...
                            if (warnedAboutBrokenProjects.add(f)) { // #60416
                                LOG.log(Level.FINE, "Cannot load project.", e); //NOI18N
                            }
                            return null;
                        }
                        if (p != null) {
                f = f.getParent();
            }
            return null;

    public class MainProjectManager implements ProjectActionPerformer, PropertyChangeListener {

        public Project getMainProject () {
            final Project lastSelectedProject;
            final Project current;
            final boolean isMain;
            synchronized (this) {
                lastSelectedProject = lastSelectedProjectRef.get();
                current = currentProject.get();
                isMain = isMainProject;
            }
            if (isMain && lastSelectedProject != null &&
                lastSelectedProject != current &&
                !isDependent(lastSelectedProject, current)) {
                // If there's a main project set, but the current project has no
                // dependency on it, return the current project.
                //System.err.println("getMainProject() = (LS) "+lastSelectedProject);
                return lastSelectedProject;
            } else {
                return current;
            }
            //System.err.println("getMainProject() = "+currentProject);
        }


        @SuppressWarnings("DMI_COLLECTION_OF_URLS")
        private static boolean isDependent(Project p1, Project p2) {
            Set<URL> p1Roots = getProjectRoots(p1);
            Set<URL> p2Roots = getProjectRoots(p2);

            for (URL root : p2Roots) {
                Set<URL> dependentRoots = SourceUtils.getDependentRoots(root);
                for (URL sr : p1Roots) {
                    if (dependentRoots.contains(sr)) {
                        return true;
                    }
                }
            }
            return false;
        }

Self-profiler with stubbed out implementations of loadProject() and isProject()
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 

Debugger attach listener-thread-4                                                                              (total time 1,350 ms)

  org.netbeans.api.debugger.DebuggerManager.fireEngineAdded ()                                            1,330 ms (48.1%) 1,309 ms (97%)
    ...
      org.netbeans.modules.debugger.jpda.projectsui.MainProjectManager.getMainProject ()                    896 ms (32.4%) 896 ms (66.4%)
        org.netbeans.modules.debugger.jpda.projectsui.MainProjectManager.isDependent ()                     896 ms (32.4%) 896 ms (66.4%)
          org.netbeans.modules.parsing.spi.indexing.support.QuerySupport.findDependentRoots ()              886 ms (32%)   886 ms (65.6%)
	...  
      org.netbeans.modules.debugger.jpda.projectsui.RunToCursorActionProvider$Listener.propertyChange ()    433 ms (15.7%) 413 ms (30.6%)
	    org.netbeans.modules.debugger.jpda.projectsui.MainProjectManager.getMainProject ()              433 ms (15.7%) 413 ms (30.6%)
          org.netbeans.modules.debugger.jpda.projectsui.MainProjectManager.isDependent ()                   433 ms (15.7%) 413 ms (30.6%)
             org.netbeans.modules.parsing.spi.indexing.support.QuerySupport.findDependentRoots ()           433 ms (15.7%) 413 ms (30.6%)	
		 

Debugger operator thread                                                                                       (total time 1,679 ms)

  org.netbeans.api.debugger.DebuggerEngine$Destructor.killEngine ()	                                  1,800 ms (63.9%) 1,659 ms (98.8%)
    org.netbeans.api.debugger.DebuggerManager.removeEngine ()	                                          1,033 ms (36.7%) 1,033 ms (61.5%)
	  org.netbeans.modules.debugger.jpda.projectsui.StepIntoActionProvider.shouldBeEnabled ()	  1,033 ms (36.7%) 1,033 ms (61.5%)
	    org.netbeans.modules.debugger.jpda.projectsui.MainProjectManager.getMainProject ()	          1,033 ms (36.7%) 1,033 ms (61.5%)
		  org.netbeans.modules.debugger.jpda.projectsui.MainProjectManager.isDependent ()         1,033 ms (36.7%) 1,033 ms (61.5%)
		    org.netbeans.api.java.source.SourceUtils.getDependentRoots ()                         1,033 ms (36.7%) 1,033 ms (61.5%)
			  org.netbeans.api.java.source.SourceUtils.getDependentRoots ()                   1,033 ms (36.7%) 1,033 ms (61.5%)
			  
	java.beans.PropertyChangeSupport.firePropertyChange ()                                              767 ms (27.2%) 626 ms (37.3%)
	  org.netbeans.modules.debugger.jpda.projectsui.SourcesTabs.setSessionTitle ()                      767 ms (27.2%) 626 ms (37.3%)
	     org.netbeans.modules.debugger.jpda.projectsui.MainProjectManager.getMainProject ()             767 ms (27.2%) 626 ms (37.3%)
		   org.netbeans.modules.debugger.jpda.projectsui.MainProjectManager.isDependent ()          767 ms (27.2%) 626 ms (37.3%)
		     org.netbeans.api.java.source.SourceUtils.getDependentRoots ()                          767 ms (27.2%) 626 ms (37.3%)
Comment 4 NukemBy 2016-09-11 22:07:59 UTC
Created attachment 162003 [details]
Self-profiler with stubbed out implementations of loadProject() and isProject()
Comment 5 NukemBy 2016-09-11 22:57:25 UTC
And to check last idea - I've commented out 'isDependent(lastSelectedProject, current)' from MainProjectManager.getMainProject() - now unit test runs in 1.5 sec, CPU usage in "Debugger attach listener-thread" and "Debugger operator thread" is around 50ms - all time is spent on waiting for other actions to complete. Such timing is expected behaviour.
Comment 6 NukemBy 2016-09-11 22:58:31 UTC
Created attachment 162004 [details]
debug-unit-test-in1.5sec.nps
Comment 7 NukemBy 2016-09-18 04:51:56 UTC
So ... finally all problems are identified and, after applying dirty hacks in my local build, unit test executes in around 1.5 - 1.7 sec, of which 95% is taken by the Gradle itself, i.e. Netbeans adds around 100ms. 

Below are reference for sub-issues listed down in the order of impact on performance:

These problems are probably 'root' cause, because they cause general slowdown in operations with files - and this consequently affects other functionality

    "assert assertNormalized()" in FileUtil.toFile() is the cause of general slowness of NetBeans
    https://netbeans.org/bugzilla/show_bug.cgi?id=267990

    SimpleFileOwnerQueryImplementation.getOwner() is another cause of major NetBeans slownes
    https://netbeans.org/bugzilla/show_bug.cgi?id=268024

    Excessive calls to File.isDirectory() cause performance problems
    https://netbeans.org/bugzilla/show_bug.cgi?id=65135

    Performance issue in RepositoryForBinaryQueryImpl.getJarMetadataCoordinatesIntern()
    https://netbeans.org/bugzilla/show_bug.cgi?id=268045

    Performance issue: FileUtil.normalizedRef cache is cleaned too frequently
    https://netbeans.org/bugzilla/show_bug.cgi?id=268027

    Performance issue in BaseFileObj.getPath --> FileName.getName
    https://netbeans.org/bugzilla/show_bug.cgi?id=268026

This one will probably self-fix after resolving above mentioned issues

    https://netbeans.org/bugzilla/show_bug.cgi?id=268028
    Performance issue in (jpda) MainProjectManager.getMainProject() -> isDependent(lastSelectedProject, current)
Comment 8 Tomas Hurka 2016-10-17 12:14:05 UTC
NukemBy, thanks for the self-sampler snapshot and outline of the steps to reproduce the problem.

You correctly identified that the bottleneck is in SimpleFileOwnerQueryImplementation.getOwner(). However in this situation, it should not be called at all. The problem is in incorrect position of several Gradle queries, which should have precedence over general ProjectSourceForBinaryQuery.findSourceRoots2() query. We found out that org.netbeans.gradle.project.query.GradleCacheSourceForBinaryQuery and org.netbeans.gradle.project.query.GradleHomeSourceForBinaryQuery should be invoked before ProjectSourceForBinaryQuery.findSourceRoots2(). Similar logic is used in maven for org.netbeans.modules.maven.queries.MavenFileOwnerQueryImpl 

Reassigning to gradle plugin. Please change position of GradleCacheSourceForBinaryQuery and GradleHomeSourceForBinaryQuery. Please also evaluate position of other queries implemented by gradle plugin.
Comment 9 attila.kelemen 2016-10-17 12:57:49 UTC
My problem is that there is no specification / documentation (that I know of) for what position should be, so I cannot - in general - review all queries (so, I left them all on default). Do you have any doc on what positions should be?
Comment 10 attila.kelemen 2016-10-19 12:06:57 UTC
I have looked at the sources of the Maven plugin and it does not specify the position for its SourceForBinaryQueryImplementation2 as well. Is this wrong there as well? If so, can someone tell me what the position for such queries should be?
Comment 11 Tomas Stupka 2016-10-20 09:22:43 UTC
> I have looked at the sources of the Maven plugin and it does not specify 
> the position for its SourceForBinaryQueryImplementation2 as well. 
there are two SourceForBinaryQueryImplementation2 impls in maven - MavenForBinaryQueryImpl and RepositoryForBinaryQueryImpl where: 
- MavenForBinaryQueryImpl is bound to a project, available through the project lookup (registered via ProjectServiceProvider instead of ServiceProvider) and it indeed does not specify a position
- on the other hand RepositoryForBinaryQueryImpl is made available in the global lookup and has a position which ensures that the query kicks in before the default/fallback ProjectSourceForBinaryQuery. If i understand right, the purpose of GradleCacheSourceForBinaryQuery and GradleHomeSourceForBinaryQuery is to provide a -sources.jar for a given jar/binary from a gradle file cache/storage. I such a case those queries should function similar to mavens RepoForBinaryQImpl. The same seems also to aply to GradleCacheJavadocForBinaryQuery, maybe others as well (don't now the plugins internal so well).


> If so, can someone tell me what the position for such queries should be?
unfortunatelly, there are no precise guidelines how to deal with it generaly, but in the above mentioned case it is desirable for gradles SFBQImplementation-s to be called before the fallback ProjectSourceForBinaryQuery, so position should be lower than that from PSFBQ. 

tzezula on cc, maybe he has somethin to add

Regarding the particular value - you can check the SourceForBinaryQueryImplementation2 impls in the NB codebase. Hopefullly you want clash with som toher plugins choice and even if - those two queries would than be on the "same" position with one taking precence over the other, non deterministicly between ide sessions.
Comment 12 attila.kelemen 2016-10-20 09:37:53 UTC
@tstupka Thanks for the answer, I have missed that query for some reason.

I have set the position to 70 which is after Maven but before 100 which seems to be the query to be preceded.

@NukemBy Can you build the plugin from the sources and see if it solves the issue for you: https://github.com/kelemen/netbeans-gradle-project ?
Comment 13 NukemBy 2016-10-22 16:41:01 UTC
I've checked performance for sources before/after this commit - https://github.com/kelemen/netbeans-gradle-project/commit/344cf5cff7c75aa8a3dcb587d65e649baf4bfb19.

The difference is - (before) ~5 sec / (after) less than 2 sec. So it looks like that change fixed the problem.

Few notes: I've tried verifying performance on fresh new instance of NetBeans - and surprisingly got same "less than 2 sec" on old code. After that I've loaded more projects, as in regular environments, NetBeans background scanned them for ~10 minutes - as known it now, results are stored in local Lucene cache in 'workspace\cache\index' - after that I've got performance close to the one I've had before - 5+ seconds (as far as i remember many lookups where made against cached/indexed classpathes). So it may happen that other people just do not have such heavy-loaded workspace.
Comment 14 attila.kelemen 2016-10-22 17:35:03 UTC
Thanks. I'm closing this as fixed then.


By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2014, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo