Can you attach a threaddump?

Created attachment 112594 [details]
Thread dump

Similar problem were in Ant projects long time ago (issue 68118).

I've found a way how to deal with this, but I'm not sure if the solution is correct. If I wrap applyChanges code to SwingUtilities.invokeLater(), then everything runs just fine. Similar code can be found for Ant projects.

Do you think that it's correct to do so even if the running code isn't AWT related only?

*** Bug 204428 has been marked as a duplicate of this bug. ***

Reassigning to Core team. CloneableEditorSupport code (concretely method openDocumentImpl()) leads to deadlock in some cases. I'm attaching thread dump of those two threads causing the deadlock. The situation why is it happening:

First, the thread with nid=0x1874 is started. It happens when Project properties panel is created and it gets a ProjectManager.mutex().writeAccess right after the creation so it has exclusive access (see CustomizerDialog class, line 294).

When some frameworks are added in the Frameworks customizer panel, there is a call for each concrete framework extenders. Those extenders need to create project files, modify configuration files and so on. Everything goes fine if there is no need for extenders to read some project data, BUT if extender needs to find project and read some information as in this case, then the application gets stucked.

In such a case second thread (nid=0x1aa8) is started inside of CloneableEditorSupport at the line 682 (the call was made from line 889). Right after creation of the second thread, the first thread is going to wait to the second one to finish it's operations (line 902). Here is the point when everything goes fine if there is no need to read some project data inside of framework extender. But if we need to do so, second thread is going to wait for ProjectManager.mutex().readAccess, but that LOCK is holden by first thread.

As a result the first thread has PM.mutex().writeAcces LOCK and it is waiting for the second one to finish, while the second thread is trying to get PM.mutex().readAccess to be able to finish.

Created attachment 112669 [details]
Thread dump - only important threads

*** Bug 204840 has been marked as a duplicate of this bug. ***

Guys can you evaluate this?.. It has to be fixed for 7.1. Either on your side or if there is no time do so for 7.1 I can provide a workarround for now.

The jsploader.BaseJspEditorSupport.loadFromStreamToKit() does some non-trivial processing that finally leads to the deadlock. Reassigning to web module for evaluation.

fixed in web-main#c4e35a2cca9f

Martine, please verify the fix since it is your are now.

Your fix looks really well but it will stuck later (createSyntax?). If I hacked that and removed the createSyntax we figured out that it's locking still somewhere else. It's going to be really complicated. :/

Integrated into 'main-golden'
Changeset: http://hg.netbeans.org/main-golden/rev/c4e35a2cca9f
User: Marek Fukala <mfukala@netbeans.org>
Log: #204427 - Deadlock while adding Spring MVC framework into Maven Web Application

the other deadlock in JspKit.createSyntaxSupport() waiting on project's lock fixed in web-main#b8d4d0280c8c

now we have another one:

"Parsing & Indexing Loop (20111111-c15db42ed827)" daemon prio=1 tid=000000000217c400 nid=0xb58e6000 in Object.wait() [00000000b58e5000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:485)
	at org.openide.util.Mutex$QueueCell.sleep(Mutex.java:1612)
	- locked <000000000f7ef3e0> (a org.openide.util.Mutex$QueueCell)
	at org.openide.util.Mutex.enterImpl(Mutex.java:727)
	at org.openide.util.Mutex.enter(Mutex.java:634)
	at org.openide.util.Mutex.readEnter(Mutex.java:617)
	at org.openide.util.Mutex.readAccess(Mutex.java:327)
	at org.netbeans.api.project.ProjectManager.findProject(ProjectManager.java:233)
	at org.netbeans.modules.projectapi.SimpleFileOwnerQueryImplementation.getOwner(SimpleFileOwnerQueryImplementation.java:131)
	at org.netbeans.api.project.FileOwnerQuery.getOwner(FileOwnerQuery.java:104)
	at org.netbeans.modules.csl.core.ProjectClassPathProvider.findClassPath(ProjectClassPathProvider.java:66)
	at org.netbeans.api.java.classpath.ClassPath.getClassPath(ClassPath.java:632)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater.getClassPathEntry(RepositoryUpdater.java:1431)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater.access$1900(RepositoryUpdater.java:149)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Work.scanFiles(RepositoryUpdater.java:2419)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$FileListWork.getDone(RepositoryUpdater.java:2719)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Work.doTheWork(RepositoryUpdater.java:2505)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Task._run(RepositoryUpdater.java:4518)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Task.run(RepositoryUpdater.java:4455)
	at org.netbeans.modules.parsing.impl.TaskProcessor.callParserResultTask(TaskProcessor.java:618)
	at org.netbeans.modules.parsing.impl.TaskProcessor$CompilationJob.run(TaskProcessor.java:728)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:680)


versus

"Default RequestProcessor" daemon prio=1 tid=000000005200bc00 nid=0xb2a2d000 waiting on condition [00000000b2a2c000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <000000000a9ab858> (a java.util.concurrent.locks.ReentrantLock$FairSync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
	at java.util.concurrent.locks.ReentrantLock$FairSync.lock(ReentrantLock.java:201)
	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
	at org.netbeans.modules.parsing.impl.TaskProcessor.acquireParserLock(TaskProcessor.java:508)
	at org.netbeans.modules.parsing.impl.Utilities.acquireParserLock(Utilities.java:76)
	at org.netbeans.modules.java.source.save.Reformatter$2.lock(Reformatter.java:330)
	at org.netbeans.modules.editor.indent.TaskHandler$MimeItem.lock(TaskHandler.java:541)
	at org.netbeans.modules.editor.indent.TaskHandler.lock(TaskHandler.java:277)
	at org.netbeans.modules.editor.indent.IndentImpl.reformatLock(IndentImpl.java:205)
	- locked <0000000006549d90> (a java.lang.Object)
	at org.netbeans.modules.editor.indent.api.Reformat.lock(Reformat.java:114)
	at org.netbeans.modules.editor.EditorModule.reformat(EditorModule.java:643)
	at org.netbeans.modules.editor.EditorModule.access$100(EditorModule.java:108)
	at org.netbeans.modules.editor.EditorModule$5.operationCreateFromTemplate(EditorModule.java:330)
	at org.openide.loaders.DataLoaderPool.fireOperationEvent(DataLoaderPool.java:240)
	at org.openide.loaders.DataObject.fireOperationEvent(DataObject.java:866)
	at org.openide.loaders.DataObject.createFromTemplate(DataObject.java:839)
	at org.openide.loaders.DataObject.createFromTemplate(DataObject.java:818)
	at org.netbeans.modules.spring.webmvc.SpringWebModuleExtender$CreateSpringConfig.createFromTemplate(SpringWebModuleExtender.java:418)
	at org.netbeans.modules.spring.webmvc.SpringWebModuleExtender$CreateSpringConfig.run(SpringWebModuleExtender.java:364)
	at org.openide.filesystems.EventControl.runAtomicAction(EventControl.java:127)
	at org.openide.filesystems.FileSystem.runAtomicAction(FileSystem.java:566)
	at org.netbeans.modules.spring.webmvc.SpringWebModuleExtender.extend(SpringWebModuleExtender.java:223)
	at org.netbeans.modules.maven.j2ee.customizer.CustomizerFrameworks.applyChanges(CustomizerFrameworks.java:129)
	at org.netbeans.modules.maven.j2ee.customizer.MavenCompositePanelProvider$2.actionPerformed(MavenCompositePanelProvider.java:155)
	at org.netbeans.modules.project.uiapi.CustomizerDialog$OptionListener.storePerformed(CustomizerDialog.java:338)
	at org.netbeans.modules.project.uiapi.CustomizerDialog$OptionListener.access$400(CustomizerDialog.java:242)
	at org.netbeans.modules.project.uiapi.CustomizerDialog$OptionListener$2$1.run(CustomizerDialog.java:300)
	at org.openide.util.Mutex.writeAccess(Mutex.java:397)
	at org.netbeans.modules.project.uiapi.CustomizerDialog$OptionListener$2.run(CustomizerDialog.java:294)
	at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1411)
	at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:1991)

"Default RequestProcessor" thread under project's mutex write lock tries to acquire parser lock whereas respository updater does the opposite.

Likely the spring support should not create the templates under project's MUTEX write access.

Understand. Spring is not the one who acquires the projects mutex write lock. It is acquired by CustomizerDialog.OptionListener#actionPerformed, so I'm not pretty sure how to workaround that problem. 

If I would omit calling project extending in the EDT... It could be done on the background but in such case the customizer's progress bar would disappear earlier than the whole Web framework support would be included into the project. Or I could perform only templates creation in the background but I would still need to wait for their creation and it would still lead to deadlock.

Tomas or Jesse, please could you advice the right way? Thanks...

I do not see why EditorModule.reformat (syntactic level) should be waiting on the parser (semantic level), especially since this is being called from the rather low-level DataObject.createFromTemplate. A bug in editor IMO.

Integrated into 'main-golden'
Changeset: http://hg.netbeans.org/main-golden/rev/b8d4d0280c8c
User: Marek Fukala <mfukala@netbeans.org>
Log: #204427 - fixed another deadlock

>I do not see why EditorModule.reformat (syntactic level) should be waiting on
>the parser (semantic level)
Parse is syntax, attribute is semantic level.

There are two problems:
1st) The SpringWebModuleExtender$CreateSpringConfig.createFromTemplate calls createFromTemplate (reformat) under project mutex.

2nd) The BaseJspEditorSupport.loadFromStreamToKit calls FileEncodingQuery (FileOwnerQuery) under editor lock

The CreateSpringConfig.createFromTemplate can be probably rescheduled (PM.mutex.pRA).

(In reply to comment #19)
> 1st) The SpringWebModuleExtender$CreateSpringConfig.createFromTemplate calls
> createFromTemplate (reformat) under project mutex.

It does not call reformat. It just calls cFT, which is a simple and commonplace call that ought to be safe, and for which the return value is generally critical for subsequent code. If anything, it is EditorModule$5.operationCreateFromTemplate which ought to be invoke its body asynch - there is no contract saying that newly created files should be silently reformatted, and calling into complex APIs from a listener callback is always dangerous.

> 2nd) The BaseJspEditorSupport.loadFromStreamToKit calls FileEncodingQuery
> (FileOwnerQuery) under editor lock

Suppose you are speaking about the original attached thread dumps. It's already solved by Marek and the new one is in comment #14.

> 
> The CreateSpringConfig.createFromTemplate can be probably rescheduled
> (PM.mutex.pRA).

Hmm probably not as a ideal solution. As I mentioned when I would reschedule that, the Saving properties notificator probably couldn't wait for these files creation (to prevent another deadlock). It looked to me that there isn't done nothing wrong (to call DO.cFT by saving project customizer) for now.

> It looked to me that there isn't done nothing wrong (to call DO.cFT by saving project customizer) for now.
Except calling 3rd party code under lock.

Ok, after offline discussion with editor guys we come to following conclusion. The safest way, especially when the solution has to be transplanted into NB71, is really to run project extending in AWT EDT.

Fixed in web-main #f9f9e02059de.

BTW, just this last changeset should be enough for fixing that deadlock, so I will backport just this one. Jirko, could you verify that deadlock doesn't happen any more once #f9f9e02059de will be in the dev build?

Thanks you all for your comments...

*** Bug 205315 has been marked as a duplicate of this bug. ***

Regardless of what is done as a conservative fix for 7.1, I still think EditorModule.reformat was at fault here; the operationCreateFromTemplate handler should call it asynch.

Fix verified in custom dev build 20111122-9a8edc0fe2de.

Please, could anybody (i.e. mjanicek as a owner of maven.j2ee) review the changeset web-main #f9f9e02059de to comply with HR process. Thanks.

Yup, patch looks OK.

Integrated into 'main-golden'
Changeset: http://hg.netbeans.org/main-golden/rev/f9f9e02059de
User: Martin Fousek <marfous@netbeans.org>
Log: #204427 - Deadlock while adding Spring MVC framework into Maven Web Application

Transplanted as changeset: releases #d64decfd7680

Integrated into 'releases'
Changeset: http://hg.netbeans.org/releases/rev/d64decfd7680
User: Martin Fousek <marfous@netbeans.org>
Log: #204427 - Deadlock while adding Spring MVC framework into Maven Web Application
(transplanted from f9f9e02059de6bc3163646785e5f42b26557cd15)

Verified in NetBeans IDE 7.1 RC1 (Build 201111232200).