Bug 162612

Created attachment 80058 [details]
Attempted patch; cancel() is called on PCA.MU, but perhaps something else swallows thread interrupt status

Created attachment 80059 [details]
Log & thread dumps

I've also noticed that selecting a project in the Open project dialog can produce an effect of slow response. Even
though the project loading is done in non-awt thread. Maybe it needs to be delayed a bit, and rescheduled if selection
changes often. The interrupt is probably swallowed by the method that loads the maven project instance. 

I'll look into it.

http://hg.netbeans.org/main/rev/5cda33da0063
integrated, the cancelling seemed to work for me.

Integrated into 'main-golden', will be available in build *200904170201* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main-golden/rev/5cda33da0063
User: Milos Kleint <mkleint@netbeans.org>
Log: #162612 delay loading of display name a bit for improved scrolling perfomance, allow cancelling the job.
also for mavenprojects allow cancelling the getsubprojects routine to skip unnecessary executions and project loadings..

I'm afraid it's still reproducible for me in 47e7f2710e5d which includes the attempted fix. Will attach some thread dumps.

My suspicion is that the thread does get interrupted and some process is ended, but then something else starts a new
blocking process. If true, a proper fix might require RequestProcessor.Task.cancel to be more aggressive and keep on
interrupting the thread until it really dies.

Created attachment 80379 [details]
More thread dumps

P2 is justified if this happens to anyone besides me (still not sure what the true trigger is); it is impossible to open
any Maven project using Open Project unless you are willing to restart the IDE right afterwards. I have been using
--open on the CLI, Open Project context menu item in Favorites, etc. to avoid hitting this bug. Seems to be a recent
regression because I do not recall having such issues in the past.

Again in bca1f4cb6e3b. BTW I encounter these issues working with Hudson sources.

"org.netbeans.modules.project.ui.ProjectChooserAccessory$ModelUpdater" daemon prio=10 tid=0xae434c00 nid=0x7f3a runnable
[0xb11fc000..0xb11fdf30]
   java.lang.Thread.State: RUNNABLE
	at org.apache.maven.model.io.xpp3.MavenXpp3Writer.writeOrganization(MavenXpp3Writer.java:1329)
	at org.apache.maven.model.io.xpp3.MavenXpp3Writer.writeModel(MavenXpp3Writer.java:1091)
	at org.apache.maven.model.io.xpp3.MavenXpp3Writer.write(MavenXpp3Writer.java:99)
	at org.apache.maven.project.interpolation.RegexBasedModelInterpolator.interpolate(RegexBasedModelInterpolator.java:140)
	at org.apache.maven.project.DefaultMavenProjectBuilder.processProjectLogic(DefaultMavenProjectBuilder.java:993)
	at org.apache.maven.project.DefaultMavenProjectBuilder.buildInternal(DefaultMavenProjectBuilder.java:812)
	at org.apache.maven.project.DefaultMavenProjectBuilder.buildFromRepository(DefaultMavenProjectBuilder.java:237)
	at org.apache.maven.project.artifact.MavenMetadataSource.retrieve(MavenMetadataSource.java:132)
	at org.apache.maven.artifact.resolver.DefaultArtifactCollector.recurse(DefaultArtifactCollector.java:509)
	at org.apache.maven.artifact.resolver.DefaultArtifactCollector.recurse(DefaultArtifactCollector.java:539)
	at org.apache.maven.artifact.resolver.DefaultArtifactCollector.collect(DefaultArtifactCollector.java:132)
	at org.apache.maven.artifact.resolver.DefaultArtifactResolver.resolveTransitively(DefaultArtifactResolver.java:347)
	at org.apache.maven.artifact.resolver.DefaultArtifactResolver.resolveTransitively(DefaultArtifactResolver.java:321)
	at org.netbeans.modules.maven.embedder.NbArtifactResolver.resolveTransitively(NbArtifactResolver.java:144)
	at org.apache.maven.artifact.resolver.DefaultArtifactResolver.resolveTransitively(DefaultArtifactResolver.java:289)
	at org.apache.maven.plugin.DefaultPluginManager.getPluginArtifacts(DefaultPluginManager.java:436)
	at org.apache.maven.plugin.DefaultPluginManager.addPlugin(DefaultPluginManager.java:279)
	at org.apache.maven.plugin.DefaultPluginManager.verifyVersionedPlugin(DefaultPluginManager.java:211)
	at org.apache.maven.plugin.DefaultPluginManager.verifyPlugin(DefaultPluginManager.java:186)
	at org.apache.maven.extension.DefaultExtensionManager.addPluginAsExtension(DefaultExtensionManager.java:218)
	at org.netbeans.modules.maven.embedder.NbExtensionManager.addPluginAsExtension(NbExtensionManager.java:111)
	at
org.apache.maven.extension.DefaultBuildExtensionScanner.checkModelBuildForExtensions(DefaultBuildExtensionScanner.java:389)
	at org.apache.maven.extension.DefaultBuildExtensionScanner.scanInternal(DefaultBuildExtensionScanner.java:187)
	at org.apache.maven.extension.DefaultBuildExtensionScanner.scanForBuildExtensions(DefaultBuildExtensionScanner.java:120)
	at org.apache.maven.embedder.MavenEmbedder.readProject(MavenEmbedder.java:377)
	at org.apache.maven.embedder.MavenEmbedder.readProjectWithDependencies_aroundBody0(MavenEmbedder.java:416)
	at org.apache.maven.embedder.MavenEmbedder.readProjectWithDependencies_aroundBody1$advice(MavenEmbedder.java:304)
	at org.apache.maven.embedder.MavenEmbedder.readProjectWithDependencies(MavenEmbedder.java:1)
	at org.netbeans.modules.maven.NbMavenProjectImpl.getOriginalMavenProject(NbMavenProjectImpl.java:328)
	- locked <0x72d28cc0> (a org.netbeans.modules.maven.NbMavenProjectImpl)
	at org.netbeans.modules.maven.NbMavenProjectImpl.<init>(NbMavenProjectImpl.java:212)
	at org.netbeans.modules.maven.NbMavenProjectFactory.loadProject(NbMavenProjectFactory.java:113)
	at org.netbeans.api.project.ProjectManager.createProject(ProjectManager.java:354)
	at org.netbeans.api.project.ProjectManager.access$300(ProjectManager.java:80)
	at org.netbeans.api.project.ProjectManager$2.run(ProjectManager.java:275)
	at org.netbeans.api.project.ProjectManager$2.run(ProjectManager.java:227)
	at org.openide.util.Mutex.readAccess(Mutex.java:327)
	at org.netbeans.api.project.ProjectManager.findProject(ProjectManager.java:227)
	at org.netbeans.modules.maven.SubprojectProviderImpl.processOneSubproject(SubprojectProviderImpl.java:184)
	at org.netbeans.modules.maven.SubprojectProviderImpl.addProjectModules(SubprojectProviderImpl.java:155)
	at org.netbeans.modules.maven.SubprojectProviderImpl.addProjectModules(SubprojectProviderImpl.java:163)
	at org.netbeans.modules.maven.SubprojectProviderImpl.getSubprojects(SubprojectProviderImpl.java:103)
	at org.netbeans.modules.project.ui.ProjectChooserAccessory$ModelUpdater.addSubprojects(ProjectChooserAccessory.java:786)
	at org.netbeans.modules.project.ui.ProjectChooserAccessory$ModelUpdater.run(ProjectChooserAccessory.java:698)
	at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:573)
	at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:1005)

I should mention that if you wait long enough, it does eventually quiet down, but this can take several minutes. If I am
running on battery power I usually do not have the patience. :-)

do you use mouse or keyboard to navigate? is the project you open an aggregating one? (with modules)

The issue was first about cancelling the open dialog, now you mention various ways to open a project.. 

There's nothing wrong with the stacktrace, just loading subprojects. I suppose you keep triggering the loading of the
subprojects of the complete set of hudson projects, I don't, when I open the core project. I guess we have different
ways of using the dialog.

Using keyboard, quickfilechooser. hudson/trunk does get momentarily selected as I use code completion to move into
subdirs; maybe hudson/trunk/plugins is the problem. But the project chooser is supposed to stop loading subprojects when
you move on to another project selection or close the dialog.

BTW: I've figured out some performance optimizations in the last weeks that should significantly reduce the time for
loading projects in batches (eg. when loading subprojects)

Reproduced in 090707 (cluster.config=full), not using quickfilechooser. Opened .../hudson/trunk/plugins/mercurial using
standard directory chooser (keyboard only). Heavy CPU, and I saw some exceptions in the log about plugins/perforce,
indicating that the IDE was loading every plugin project.

Maybe need SubprojectProvider2 which would return some kind of cancellable future object?

as long as you traverse from top directory into the child dirs, all the project dirs on the way get loaded with all
their children. Sort of constraint of the chooser dialog's accessory.. If it were not loading subprojects, we would be
fine..

Reassigning to default owner.

Tested by selecting hudson, hudson/main and hudson/main/core with build 100812-174de290da8c, but saw the thread active only for a few seconds.

Cannot reproduce the heavy CPU use for the reported long time.

(In reply to comment #17)
> Tested by selecting hudson, hudson/main and hudson/main/core

Try navigating through hudson/trunk/plugins/, and try using quickfilechooser as well as standard dirchooser.

Integrated into 'main-golden'
Changeset: http://hg.netbeans.org/main-golden/rev/949724cb4301
User: Jesse Glick <jglick@netbeans.org>
Log: Refining #65349 implementation of asynch project open to better support large source trees.
- show meaningful messages in the progress handle, and make it cancelable
- do not show a blocking modal dialog
- permit subproject calculation to be interrupted when supported (#162612)

what is there more to do?

I've tried with jenkins source, unfortunately it includes the plugins sources no more. same operation with maven/plugins makes it clear that all plugin's MavenProject instances are loaded via the plugins/ folder selection, happens both with quickfilechooser and regular chooser when using keyboard. 

the CPU utilization is a function of number of projects, complexity of the project (number of parents, dependencies, plugins). 2.1 vs current 3.x embedder is also a factor in the equation.

A fix of bug #210465 would I guess render this obsolete - submodules would be scanned for only if the user explicitly requested this, say with a context menu item, but not just from selecting the root of the reactor in Open Project.

A new SPI could also provide an explicit Cancellable object that would be more reliable than thread interrupted status.