Created attachment 12268 [details]
Thread dump

BTW: current dev build; I was closing several documents by pressing
Ctrl-F4 repeatedly. It it matters, is possible that the one I was
about to close had been externally modified and was due to be reloaded.

The dump is a bit of mystery for me either.
It seems the JDK have misreported the locks held, so the dump should
look like:
"AWT-EventQueue-1" prio=1 tid=0x08367590 nid=0x6bc3 runnable
oot.CloneableEditorSupport.closeDocument(CloneableEditorSupport.java:1391)
- relocked lock <0x46ce9358> (oow.CloneableOpenSupport$Listener)
oot.CloneableEditorSupport.notifyClosed(CloneableEditorSupport.java:1244)
oot.DataEditorSupport.notifyClosed(DataEditorSupport.java:217)
oot.EditorSupport$Del.superNotifyClosed(EditorSupport.java:551)
oot.EditorSupport.notifyClosed(EditorSupport.java:426)
onm.java.JavaEditor.notifyClosed(JavaEditor.java:271)
- locked <0x46ce9358> (a oow.CloneableOpenSupport$Listener)
oot.EditorSupport$Del.notifyClosed(EditorSupport.java:536)

The lock is already held by the thread, so reenter should be seamless.
Moreover, the thread is (acc. to line number) already inside the sync
section, just calling openDocumentImpl()
(so the external modification status matters!)

Finally, the thread is marked as runnable so it should (in the next
CPU tick) enter the oDImpl method.

Random thought: Was it really deadlocked? (i.e. no CPU activity
and the dump was the same several times?)
(Can it be an infinite loop instead? ;-)

Huh, now I see it. It *IS* infinite loop...

More details:
The AWT queue spin-waits for the document to be loaded, but keeps the
lock, so it can't be loaded. Two bad things:
1. Doesn't release the lock so the loading can't proceed
2. It spins!

Hard to cut this one:
I can:
A) either replan the actual closing task, which would mean the close()
call would finish before actually closing (the reload will happen
after the call, and then the real close would take place).
B) or I can move the lock inside the loop, which would keep spinning
and racing for the lock with the reload task
C) I could also alter the state machine to perform the real close
under the lock before reloading and let the reload task recognize such
state as soon as it finally acquires the lock.
The reload task would know it is irrelevant and give up.

C) seems the best.

CCing Mila, who wrote the state machine.

Yes, I agree with C as well. The thread doing document closing should
change the state to closed and the reloading task must check state
right after acquiring the lock. I'm attaching a diff of the possible
fix but I did not test it at all. Just an idea.

Created attachment 12292 [details]
Possible fix

OK, I applied the fix and tested it thoroughly. It seems OK to me and
to the tests. 

Petre, thanks a lot for applying and testing the patch.

closed