Bug 38582

Created attachment 12808 [details]
The deadlock is attached

I do not see tomcat in any thread.

I could not reproduce this on Windows XP.  Maybe platform specific?

The stack trace seems to show that j2eeserver does nothing wrong -
could this be caused by openide/nodes?

The code was changed in the last week. I'm not able to reproduce it as
well now. So we can close the bug.

The problem is in the j2ee ServerNode, which, when asked for a subnode
(under the Children.MUTEX.readAccess()) tries to reenter the mutex in
writeAccess (mutex upgrade), which is forbidden and deadlock prone (if
two threads try to upgrade , they'll surely deadlock). Pity the Mutex
doesn't correctly throw IllegalState (but it tries to, in some code
paths).

The important part of trace:
at oou.Mutex$Privileged.enterWriteAccess(1200) - ask writeAccess
at oon.Node.setChildren(353)
at j2ee.diu.FilterXNode.<init>(49)
at j2ee.diu.InstanceTargetXNode.<init>(47)
at j2ee.diu.InstanceTargetXNode.<init>(43)
at j2ee.diu.RegistryNodeProvider.createInstanceTargetNode(76)
at j2ee.diu.ServerNode$PluginChildren.createNodes(99)
at oon.Children$Keys$KE.nodes(1987)
at oon.ChildrenArray.nodesFor(109)
at oon.Children$Info.nodes(1083)
at oon.Children.justComputeNodes(589)
at oon.ChildrenArray.nodes(54)
at oon.Children.getNodes(325) - holds readAccess()

Btw: see issue 10778

Petr N., thanks for explanation, although I still don't understand the
issue exactly. ServerNode does not use the Children.MUTEX class at
all. Who exactly acquires the read lock?

Inside Children.getNodes, the Children.MUTEX is read-locked
(it is not visible in the thread dump, because Children internally
uses Mutex.Privileged).
Look at Children around line 325
ServerNode is called already under readLock (generally,
Children.Keys.createNodes(Object keyFor) is always called under 
Children.MUTEX's readLock.

The node created by the ServerNode.PluginChildren calls setChildren
from inside of its constructor, which is the problem, as setChildren
needs writeAccess.
If you can't resolve the children type before calling
superconstructor, you can wrap the setChildren with
Mutex.postWriteAccess().
But looking at the FilterXNode source, it would be enough to rewrite
the constructor to:
super(extendsChildren ? new XChildren(xnode) : original);

Petr N., thanks for the analysis and suggestion.  Eventhough I could
not reproduce the problem to say for sure whether its fixed, the
suggested change make sense so I will do it.

I changed code so to avoid invoking setChildren in the constructor of
InstanceTargetXNode.

Petr N., thanks for the thorough analysis and suggestions. I think it
would be useful to document in the Javadoc of
Children.Keys.createNodes(Object keyFor), that this method is called
under read lock. Without this, module writers have no way to find out
that they are doing something wrong. Thanks.