I found what is going on:

Main window is created far sooner during startup of netbeans then custom LF is
set. This is wrong order and the result is, that main window doesn't know about
Substance LF decorations, because Substance LF isn't set at this moment yet.

I tried to reset decorations to the main window later, but it breaks whole
painting and other behaviour of main window, so apparently Swing doesn't allow this.

I fixed part of the problem:
/cvs/core/windows/src/org/netbeans/core/windows/view/ViewHierarchy.java,v  <-- 
ViewHierarchy.java
new revision: 1.40; previous revision: 1.39

But the real order needs to be fixed in startup code:
        at java.lang.Thread.dumpStack(Thread.java:1158)
        at org.netbeans.swing.plaf.Startup.run(Startup.java:363)
        at org.netbeans.core.CoreBridgeImpl.initializePlaf(CoreBridgeImpl.java:9
2)
        at org.netbeans.core.startup.Main.initUICustomizations(Main.java:172)
        at org.netbeans.core.startup.Main.start(Main.java:366)

must be called before any call like below:
        at org.netbeans.core.windows.view.ui.MainWindow.<init>(MainWindow.java:9
9)
        at org.netbeans.core.windows.view.ViewHierarchy.getMainWindow(ViewHierar
chy.java:90)
        at org.netbeans.core.windows.view.DefaultView.getMainWindow(DefaultView.
java:86)
        at org.netbeans.core.windows.ViewRequestor.getMainWindow(ViewRequestor.j
ava:74)
        at org.netbeans.core.windows.Central.getMainWindow(Central.java:1400)
        at org.netbeans.core.windows.WindowManagerImpl.getMainWindow(WindowManag
erImpl.java:127)
        at org.netbeans.modules.autoupdate.Autoupdater$2.run(Autoupdater.java:87
)
        at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:209)
        at java.awt.EventQueue.dispatchEvent(EventQueue.java:461)
        at java.awt.EventDispatchThread.pumpOneEventForHierarchy(EventDispatchTh
read.java:242)
        at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThre
ad.java:163)
        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:157)

        at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:149)

        at java.awt.EventDispatchThread.run(EventDispatchThread.java:110)
Turning on modules:
        org.openide.util [6.5 051011]
        org.openide.modules [6.4 051011]

As you can see, MainWindow is created even during modules restoring, and I found
several such places. Because we don't have proper API for modules to know when
GUI is ready, such calls are inevitable.

To conclude: 
        at org.netbeans.core.startup.Main.initUICustomizations(Main.java:172)
should be moved before modules loading, to have Swing UI properly set before
modules may touch it.

I'm not familiar with startup code and I'm not sure if I woudl do it right, so
passing to Jesse, kindly asking for help. At the same time I'm raising priority,
because it looks like major startup flaw for me when we are constructing main
window with not-yet-set look and feel. Many other issues like wrong fonts etc
may be connected to this.

Jesse please, do you agree with the change and can you do it? Thanks.

I don't know anything about this area of code. Maybe Yarda does? I don't
remember who works on this.

I'll register the UI before ModuleInstalls are called. Probably I introduce PreInstall   
interface which instances will be found in lookup as soon as classloaders are   
ready, but before layers are constructed and  ModuleInstalls called. 

Or simply add to ModuleInstall:

/**
 * Called as early as possible, before layers and so on are ready.
 * Normally better to use {@link #restored}.
 */
public void prerestored() {}

?

We're about to introduce new API. 

IMO the suggested way is not the right one. I think that noone should use winsys
during module system loading. Load modules first then init and show winsys (with
UI init at the beggining) and finally run other codes that can be queued into
end of startup sequence like AU client, resolve broken refs dialog and many similar.

PreInstall/prerestored() adds yet another layer/level into a place that seems to
be well structured in current startup although we cannot be convinced that there
are too many use cases to justify this.

Do that mean that we should resuscitate the old idea of   
  
void WindowManager.runExclusive(Runnable r);  
  
with the semantic being that only one of these runnables can run at one time  
and all of them are executed only when window system is ready? If so, I can 
provide such patch and also fix autoupdate to behave "in the right way" 

Created attachment 26122 [details]
API change in openide/windows and changes in javacore and autoupdate

I'd like to add new method  
 
void WindowManager.invokeExclusively(Runnable) 
 
that could be used to run code as soon as window system is "ready". That is 
going to fix our infamous getMainWindow().addWindowListener trick. 
 
I've attached the patch which in my opinion solves everything and also this 
problem. It contains new TCK in openide/windows and test that verifies that 
core/windows correctly implement that.  
 
As author of the change I've 'selected' Dafe, as he will know much better when 
the window system is "ready". For now I put the code into 
WindowManagerImpl.setVisible, but maybe there is better place for it. That is 
why I am reassigning to him. 

I like this much more than previous idea. 

The only remaining comment from me is priority of these tasks. Is it important
or not? If it is it would be better to register either some runnable in XML
layer or service in META-INF/services.

Wellm I hoped for different solution, simply moving init of LF and netbeans LF
initialization to earlier place of startup process.
Remember we are trying to solve the problem, that UI is initialized too late. We
are *not* solving the problem that window system is accessed too early.
IMO it is wrong to tie LF initialization and winsys together. No, LF init should
be already done when modules can do anything with Swing, open their frame or
dialog. I know two such cases - updater and licence check (correct me if I'm wrong).

My conclusion: Suggested solution is a solution for another problem, not this
one, so I can't finish it and commit. Please consider and try to implement (or
advice me) my idea from above, if you agree.

However I agree that solution suggested by jtulach can be useful and we can
finetune it in separate enhancement.

"priority of these tasks" - or just have invokeExclusively(Runnable, int        
priority), if really needed, which I do not think.  
        
"Remember we are trying to solve the problem" - We are solving the problem of 
main window  without correct UI - as far as I can tell my patch solves it.      
      
Moreover I would argue that the reason for all these problems is that people      
are using WindowListener for showing a UI at reasonable time after the startup      
as there is nothing better to use. The invokeExclusively solves the problem      
for one and all.      
      
"updater and licence check" - they form a separate issue, they are run before  
the module system is initialized and at such early time there is no way to  
call the CoreBridgeImpl.initializePlaf    
    
"LF init should be already done when modules can do anything with Swing, open   
their frame or dialog" - I fully agree. That is why I suggest that we tell   
module writers to not touch swing in ModuleInstall.restored, but instead call  
invokeExclusively and handle the rest of the stuff there.  
  

Yes, your solution fixes the problem for now in our own modules. Generally, any
module (our, 3rd party, point products) can freely call
WindowSystem.getMainWindow() (with no warning sign that they do smt wrong) and
break things again. And so they will for sure, not good. Perhaps jarda's
solution can be improved by adding a check in WindowSystem.getMainWindow that
would scream on console if called too early. Hmm although implementation of this
won't be that straighforward.

"That is why I suggest that we tell module writers to not touch swing in
ModuleInstall.restored" - you mean we will tell them in documentation. Fine, but
again nothing stops modules to break things accidentally at any time.

"updater and licence check are different issue" - yes, and I hoped for solution
that would fix this problem as well. I remember Marek having significant
troubles when coding licence check. Isn't it really possible to turn our PLAF
module into somewhat else that would be called right at the time when system
realizes that it's going to use same GUI (Swing)? I can understand that this can
be overly complicated and not doable for this release, but this is the only way
how to fix UI startup correctly IMO.

For those 'too early' dialogs we now use 'default' system L&F which is the same
as IDE L&F provided user does not change IDE L&F using --laf or otherwise. Not
difficult but not necessary the same as actual IDE L&F.

Along the same lines as Dafe's comment:
1. IMO, the patch itself does not solve this problem. It would only allow us to
solve the problem. All modules that do getMainWindow().addWindowListener() has
to be rewritten to fix this problem (java/project is not fixed by your patch).
2. I do not know about any reason for calling getMainWindow except adding the
listener. Is there any other valid reason? If no and this patch is applied, I
think we should issue a warning each time it is called.
3. IMO, the priorities (or some kind of order) may prove to be necessary.

It seems to me too that this patch solves a different problem and solves this
problem only as a side-effect (and does not enforce correct approach from the
modules).

(This does not mean that I think that invokeExclusively is very usefull, but
should be fine-tuned. BTW: maybe using it for autoupdate is not the correct
thing to do.)

Adding java module to cc list. We register WindowListener in
JavaCoreModule.restored().

Interesting discussion, but please don't forget that it has been triggered by a
report of a problem encountered when using an *unsupported* LAF (sorry). If
anyone can reproduce a similar problem with any of the standard LAFs, that would
be a P2 bug. Until then -> P3.

I think we should definitely improve the NetBeans startup procedure, possibly by
introducing a new API, but not for NB 5.0, because the potential clients won't
be able to migrate to it anyway.

About the "unsupported" - i have seen quite a few mentions of running NetBeans
under custom LAF (mainly JGoodies) over the past two years. As NetBeans *is* a
Swing application, there shouldn't be a distancing from them. Of course, this is
a question of priorities, but still. Continuing declarations of official lack of
support for third-party LAFs are not good. Why allow third-party plugins and not
third-party LAFs?

As you say, this is a matter of priority. Currently, the priority for the
NetBeans UI is to look as "native" as possible on all target platforms.
Third-party LAFs definitely aren't disallowed, we just don't put much effort
into testing they actually work. If bugs are filed and are easy to fix, they get
fixed. More complicated bug fixes may get postponed.

It's probably worth noting that NetBeans uses a couple of custom components that
are designed for the standard LAFs, but are likely to look a little out-of-place
with some 3rd-party LAFs.

About custom components - it takes a little time, but it's quite possible to
provide consistent UI delegates for those ones. See [1]

[1] https://substance-netbeans.dev.java.net/

Very nice!

Problem was fixed by Yarda (I don't know exact bug number) and I verified that
it works in todays build with Substance LF.

I also created new enhancement 67934 with a link to patch that Yarda wrote, to
not waste the work, because the idea of mentioned API is good and should be
integrated in future release. 

without additional comments for long time - verified