Created attachment 60884 [details]
Full thread dump

Although the stack trace is different, this is actually the same bug.  I applied a change suggested by Andrei as to how
we do Lookup in DatabaseServerManager, and this propagates the lookup failure to a different place in our code.

Sorry, please ignore the last comment.  I realized I was wrong as I was typing, but forgot to clear the comments field.

Can you provide instructions on how to run the functional tests?  My NB goes to 100% CPU every time I try to open the
functional tests package. Meanwhile I'll try to determine the issue from the thread dump, and if necessary I'll fix
without being able to test, and you can verify for me.

I have a strong suspicion this is related to the other lookup error we are seeing, Issue 134121, where the lookup for
DatabaseServer is not returning anything.  I noticed that there is a warning saying MultiFileObject couldn't load my
layer file.

Here what is happening is two threads - the module install thread and the thread initializing the Database Explorer node
tree, are both in the middle of looking up the db.mysql layer file.  I think there is a locking issue inside of lookup
that is being hit by two threads both going after the same layer file.

I can't see anything obvious I'm doing wrong except for the fact that I'm trying to talk to the DB Explorer from a
module that the DB Explorer then is talking back to through an SPI, which is a somewhat circular dependency even though
it's happening through an SPI.

OK, after a lot more inspection, here's what I think is happening:

"Asynch children creator org.openide.nodes.AsynchChildren@195afdb" owns a lock on the DatabaseServerManager class.  This
is because I made the method that does the lookup a synchronized static method.  This thread then is waiting for a
Request Processor task to finish

Two other threads, both request processors, are also trying to do a lookup on the db.mysql layer file, and are hanging
in Class.forName().  I am pretty sure it is because the thread above owns the lock on the DatabaseServerManager class.

My method does not actually need to be synchronized.  It's OK if I do the lookup twice, and I am sure the lookup code
does its own synchronization.

I'm removing the synchronization, and I will see if issue 134121 goes away, as I can't reproduce this one.  If it does,
I'll commit and then maybe mschovanek you can verify to see if this behavior goes away too - they seem very much related.

I think removing the synchronization won't fix this issue. The deadlock isn't caused by the synchronized keyword on
lookupDatabaseServer(), but by the fact that this method is called from the static initializer. Only one thread can call
the static initializer -- when another thread needs to access the class while it is being initialized, it waits (like
the folder recognizer thread in Class.forName() here). And since the lookup implementation called by
lookupDatabaseServer() waits on the folder recognized to finish, the two threads will wait for each other forever.

Another thing is that DatabaseNodeInfo.refreshChildren() should probably be called asynchronously when the list of
connection changes, to ensure that it doesn't do work in the folder recognizer thread.

I need to think about this more, so taking over.

Andrei is going to act as my advisor, but I am taking the responsibility for cleaning up this mess :)

I checked in a fix, see http://hg.netbeans.org/main?cmd=changeset;node=7611258ac0fc

But I need to make one more change and make the looked up reference volatile.  Once I have that done, I'll mark this bug
as fixed, as well as issue 134121.

http://hg.netbeans.org/main?cmd=changeset;node=7611258ac0fc

Still reproducible with CB build #20080507052947, attaching the fresh TD.

Created attachment 61113 [details]
Thread Dump 2

*** This issue has been marked as a duplicate of 134577 ***