the problem is that ProgressUtils's dialog and DialogDisplayer do not know about each other. 
So, the following sequence cause deadlock:
- non EDT thread calls ProgressUtils.showProgressDialogAndRun(runnable)
- at the same time EDT thread should use DialogDisplayer 
=>
dlg created from DialogDisplayer is shown below progress dialog and no way to press buttons in it.
Looks like dlg does not use progress's windows as a parent, because it's not in NbPresenter.currentModalDialog field, because progress hasn't put it's dialog in this field

Attach a thread dump showing the DL. Thanks

Created attachment 145076 [details]
thread dump

"AWT-EventQueue-0" doing NbLifecycleManager.exit and calling ProjectOpenedHook$1.projectClosed
"On Start/Stop" calls RunOffEDTImpl.showProgressDialogAndRun, but runner can not run, because projectClosed displayed dialog which can not be closed by user. I will attach screenshot

Created attachment 145077 [details]
dialog with "OK" below progress dlg

and how should it work? We discussed this with Standa and we agreed there's no way for us to fix this.
To be honest i am not following your locking mechanism. I assume the thread started by the progress dialog is:
"org.netbeans.modules.progress.ui.AbstractWindowRunner" - waiting on openedProjects.wait(). And who calls notify on that object? It gets notified after user clicks on OK in the underlying dialog, right?

But because:
1) the dialog expecting user input gets opened first
2) then you explicitely call ProgressUtils.showProgressAndRun which according to javadoc "blocks all user input and waits for the background thread to finish". So it does what it is expected to do.

So in our opinion you should fix this on your side and should not call showProgressDialogAndRun when you're currently showing a dialog requesting input.

Created attachment 145083 [details]
sample application

Ondrej, please, evaluate attached sample project

in the attached sample project Progress is displayed and then user is asked for input, but input dlg is below progress dlg

Created attachment 145092 [details]
patch proposal

This patch may help you. However i will not commit it into the repo, i don't trust it fully, it may cause unexpected regressions during IDE startup. If it helps i suggest you push it to your branch you will build studio from.
Or, again, fix the deadlock in your code as you should. Once more ProgressUtils.showProgressDialogAndRun does what it's supposed to do: blocks all input until the background thread finishes - and because your background thread never finishes, the progress dialog will block until your notebook is out of power.

Ondrej,

Thanks for the patch. Looks interesting. Additionally we can protect from start up regressions by "closed" flag and do it only during close phase.

As shown in my sample project. The situation is not specific to our code. It's just incorrect windows layout due to z-order.

(In reply to Vladimir Voskresensky from comment #12)
> As shown in my sample project. The situation is not specific to our code.
> It's just incorrect windows layout due to z-order.
Again, there is nothing such as incorrect z-order. The blocking progress dialog works as expected. It blocks literally *all* user input throughout NetBeans main window, all dialogs, popups etc.

(In reply to Ondrej Vrabec from comment #13)
> (In reply to Vladimir Voskresensky from comment #12)
> > As shown in my sample project. The situation is not specific to our code.
> > It's just incorrect windows layout due to z-order.
> Again, there is nothing such as incorrect z-order. 
Do you mean, that "if" DialogDisplayer would have had progress's window as parent then I still will not be able to click buttons in DialogDisplayer's dialog?

btw, javadoc of showProgressDialogAndRun claims that it blocks main window, not all other dialogs

Integrated into 'main-silver', will be available in build *201402130001* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)

Changeset: http://hg.netbeans.org/main-silver/rev/c3a4c1a53bbd
User: Vladimir Voskresensky <vv159170@netbeans.org>
Log: fixing #241533 - deadlock on IDE closing (OnStop ProgressUtils.showProgressDialogAndRun will block any AWT dialog)
- try cancellable runner

patch doesn't help. 
What is the problem to remember progress's dialog and use it as the parent for DialogDisplayer and exception dialogs?

(In reply to Vladimir Voskresensky from comment #17)
> What is the problem to remember progress's dialog and use it as the parent
> for DialogDisplayer and exception dialogs?
The problem is that by the time the DialogDisplayer opens the modal dialog, the progress dialog is not even opened yet.

What helps is:
        if (!mainWindow.isVisible()) {
            result.setModalityType(JDialog.ModalityType.MODELESS);
        }

Btw, if main window is not visible anymore, then modality makes not a lot of sense, isn't it?

if needed, we can try to protect ourselves to:
  if (!mainWindow.isVisible() && closingIDE) {

(In reply to Vladimir Voskresensky from comment #17)
> patch doesn't help. 
Well, it helped in the test app you provided. I was able to click on Yes/No in the other dialog.(In reply to Vladimir Voskresensky from comment #19)
> What helps is:
>         if (!mainWindow.isVisible()) {
>             result.setModalityType(JDialog.ModalityType.MODELESS);
>         }
> 
> Btw, if main window is not visible anymore, then modality makes not a lot of
> sense, isn't it?
Sure, you can set it to modeless if you want. But as good would be not to call showProgressAndRun during shutdown from your code.

interesting observation. I've added action on toolbar doing:

    @Override
    public void actionPerformed(ActionEvent e) {
        ProgressUtils.showProgressDialogAndRun(new Runnable() {

            @Override
            public void run() {
                try {
                    Thread.sleep(2000);
                } catch (InterruptedException ex) {
                    Exceptions.printStackTrace(ex);
                }
                
                Object notify = DialogDisplayer.getDefault().notify(new NotifyDescriptor.Confirmation("Throw exception?", NotifyDescriptor.YES_NO_OPTION));
                if (notify == NotifyDescriptor.YES_OPTION) {
                    try {
                        Thread.sleep(2000);
                    } catch (InterruptedException ex) {
                        Exceptions.printStackTrace(ex);
                    }
                    Exceptions.printStackTrace(new IllegalStateException("Hello"));
                }
                
            }
        }, "Long task");
    }

And when main window is visible => 
- DialogDisplayer is visible on top of progress panel
- Exceptions is also displayed and clickable (progress panel is disappeared)

Vladimír wanted me to say something: As far as http://bits.netbeans.org/dev/javadoc/org-netbeans-api-progress/org/netbeans/api/progress/ProgressUtils.html#showProgressDialogAndRun(org.netbeans.api.progress.ProgressRunnable, java.lang.String, boolean) 
method goes, it currently says that it "blocks the main window", but as far as my knowledge of EDT goes, this in fact means to block all windows/dialogs that were shown so far. If this is part of the problem, then just update the javadoc to clarify that. In addition to that, if a new dialog is shown after, it should be operational.

(In reply to Vladimir Voskresensky from comment #21)
> interesting observation. I've added action on toolbar doing:
> 
>     @Override
>     public void actionPerformed(ActionEvent e) {
>         ProgressUtils.showProgressDialogAndRun(new Runnable() {
> 
>             @Override
>             public void run() {
>                 try {
>                     Thread.sleep(2000);
>                 } catch (InterruptedException ex) {
>                     Exceptions.printStackTrace(ex);
>                 }
>                 
>                 Object notify = DialogDisplayer.getDefault().notify(new
> NotifyDescriptor.Confirmation("Throw exception?",
> NotifyDescriptor.YES_NO_OPTION));
>                 if (notify == NotifyDescriptor.YES_OPTION) {
>                     try {
>                         Thread.sleep(2000);
>                     } catch (InterruptedException ex) {
>                         Exceptions.printStackTrace(ex);
>                     }
>                     Exceptions.printStackTrace(new
> IllegalStateException("Hello"));
>                 }
>                 
>             }
>         }, "Long task");
>     }
> 
> And when main window is visible => 
> - DialogDisplayer is visible on top of progress panel
> - Exceptions is also displayed and clickable (progress panel is disappeared)
If i'm getting this right then you're saying that when:
1) the blocking progress dialog is displayed (that's what showProgressDialogAndRun does
2) after that a dialog is displayed (DD.getDefault().notify)
=> then everything works?
Well that's what i've been talking about all the time. In your case your dialog asking the user is displayed earlier than the progress one.

I updated javadoc of ProgressUtils to say correctly what it does: blocks all displayed frames and dialogs: core-main #d506abe6e98a

(In reply to Vladimir Voskresensky from comment #1)
> the problem is that ProgressUtils's dialog and DialogDisplayer do not know
> about each other. 

Would it help if they knew about each other? The only cooperation between dialogs I am aware of is "isLeaf": http://bits.netbeans.org/dev/javadoc/org-openide-dialogs/org/openide/DialogDescriptor.html#isLeaf() but this helps only when dialogs are being closed - otherwise all dialogs follow Swing "nesting" rules.

> So, the following sequence cause deadlock:
> - non EDT thread calls ProgressUtils.showProgressDialogAndRun(runnable)
> - at the same time EDT thread should use DialogDisplayer 
> =>
> dlg created from DialogDisplayer is shown below progress dialog and no way
> to press buttons in it.

Yeah, that is something that can happen. As far as I know there is no support in the infrastructure to avoid that.

> Looks like dlg does not use progress's windows as a parent, because it's not
> in NbPresenter.currentModalDialog field, because progress hasn't put it's
> dialog in this field

I am not sure if this matters. Imho, it depends which dialog is shown sooner. If DialogDisplayer runs first and then ProgressUtils.showProgressDialogAndRun is executed - the first dialog will not be accessible.

why can't we have:
        if (!mainWindow.isVisible()) {
            result.setModalityType(JDialog.ModalityType.MODELESS);
        }
?
I solves all problems

Created attachment 145197 [details]
patch in cnd???

workaround on our side...
http://hg.netbeans.org/cnd-main/rev/3dc53f562f46

(In reply to Ondrej Vrabec from comment #26)
> Created attachment 145197 [details]
> patch in cnd???
The whole purpose is to show dialog informing user, that IDE is really wasn't existed, jut windows was hidden and IDE have to finish flushing it's internal data

Integrated into 'main-silver', will be available in build *201402150001* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)

Changeset: http://hg.netbeans.org/main-silver/rev/d506abe6e98a
User: Ondrej Vrabec <ovrabec@netbeans.org>
Log: #241533 - deadlock on IDE closing (OnStop ProgressUtils.showProgressDialogAndRun will block any AWT dialog)
Updating javadoc to be in accordance with the implementation. The blocking progress dialog blocks all user input to all main window and displayed dialogs or frames.

Related to deadlock on shutdown:
- may be we can introduce method for 'modeless' progress dlg?
Then in OnClose we can use it and no deadlocks can happen

(In reply to Vladimir Voskresensky from comment #30)
> Related to deadlock on shutdown:
> - may be we can introduce method for 'modeless' progress dlg?
> Then in OnClose we can use it and no deadlocks can happen
Yes, that makes sense.