Created attachment 68811 [details]
stacktrace

The problem is in the MySQL module, cf. stack trace.

Build: NetBeans IDE Dev (Build 20080903152018)
VM: Java HotSpot(TM) Client VM, 11.0-b13, Java(TM) SE Runtime Environment, 1.6.0_10-rc-b26
OS: Windows XP, 5.1, x86

User Comments: 
all zip distribution on initial IDE start, fresh userdir

Stacktrace: 
org.netbeans.api.db.explorer.DatabaseException: java.util.ConcurrentModificationException
        at org.netbeans.modules.db.explorer.infos.RootNodeInfo.initChildren(RootNodeInfo.java:170)
        at org.netbeans.modules.db.explorer.infos.DatabaseNodeInfo.loadChildren(DatabaseNodeInfo.java:710)
        at org.netbeans.modules.db.explorer.infos.DatabaseNodeInfo.getChildren(DatabaseNodeInfo.java:698)
        at org.netbeans.modules.db.explorer.infos.RootNodeInfo.addConnection(RootNodeInfo.java:235)
        at org.netbeans.api.db.explorer.ConnectionManager.addConnection(ConnectionManager.java:145)
        at org.netbeans.modules.derby.api.DerbyDatabases.registerDatabase(DerbyDatabases.java:366)

Created attachment 68985 [details]
stacktrace

*** Issue 144245 has been marked as a duplicate of this issue. ***

*** Issue 146160 has been marked as a duplicate of this issue. ***

Looks like this one is also related to changes to MySQL connection changes

If not, please assign back to me

This issue has already 5 duplicates 
see http://statistics.netbeans.org/exceptions/detail.do?id=103180

Build: NetBeans IDE Dev (Build 200809091401)
VM: Java HotSpot(TM) Client VM, 10.0-b19, Java(TM) SE Runtime Environment, 1.6.0_05-b13
OS: Windows Vista, 6.0, x86

User Comments: 
First run of the IDE

Stacktrace: 
org.netbeans.api.db.explorer.DatabaseException: java.util.ConcurrentModificationException
        at org.netbeans.modules.db.explorer.infos.RootNodeInfo.initChildren(RootNodeInfo.java:170)
        at org.netbeans.modules.db.explorer.infos.DatabaseNodeInfo.loadChildren(DatabaseNodeInfo.java:710)
        at org.netbeans.modules.db.explorer.infos.DatabaseNodeInfo.getChildren(DatabaseNodeInfo.java:698)
        at org.netbeans.modules.db.explorer.infos.RootNodeInfo.addConnection(RootNodeInfo.java:235)
        at org.netbeans.api.db.explorer.ConnectionManager.addConnection(ConnectionManager.java:145)
        at org.netbeans.modules.derby.api.DerbyDatabases.registerDatabase(DerbyDatabases.java:366)

Created attachment 69534 [details]
stacktrace

Build: NetBeans IDE Dev (Build 200809081401)
VM: Java HotSpot(TM) Client VM, 11.0-b11, Java(TM) SE Runtime Environment, 1.6.0_10-beta-b14
OS: Linux, 2.6.24-19-generic, i386

User Comments: 
just started IDE

Stacktrace: 
org.netbeans.api.db.explorer.DatabaseException: java.util.ConcurrentModificationException
        at org.netbeans.modules.db.explorer.infos.RootNodeInfo.initChildren(RootNodeInfo.java:170)
        at org.netbeans.modules.db.explorer.infos.DatabaseNodeInfo.loadChildren(DatabaseNodeInfo.java:710)
        at org.netbeans.modules.db.explorer.infos.DatabaseNodeInfo.getChildren(DatabaseNodeInfo.java:698)
        at org.netbeans.modules.db.explorer.infos.RootNodeInfo.addConnection(RootNodeInfo.java:235)
        at org.netbeans.api.db.explorer.ConnectionManager.addConnection(ConnectionManager.java:145)
        at org.netbeans.modules.derby.api.DerbyDatabases.registerDatabase(DerbyDatabases.java:366)

Created attachment 69535 [details]
stacktrace

Build: NetBeans IDE Dev (Build 200809120201)
VM: Java HotSpot(TM) 64-Bit Server VM, 1.6.0_05-b13-52, Java(TM) SE Runtime Environment, 1.6.0_05-b13-120
OS: Mac OS X, 10.5.4, x86_64

User Comments: 
Started IDE from command line.  Exception thrown when IDE launched

Stacktrace: 
org.netbeans.api.db.explorer.DatabaseException: java.util.ConcurrentModificationException
        at org.netbeans.modules.db.explorer.infos.RootNodeInfo.initChildren(RootNodeInfo.java:174)
        at org.netbeans.modules.db.explorer.infos.DatabaseNodeInfo.loadChildren(DatabaseNodeInfo.java:710)
        at org.netbeans.modules.db.explorer.infos.DatabaseNodeInfo.getChildren(DatabaseNodeInfo.java:698)
        at org.netbeans.modules.db.explorer.infos.RootNodeInfo.addConnection(RootNodeInfo.java:268)
        at org.netbeans.api.db.explorer.ConnectionManager.addConnection(ConnectionManager.java:145)
        at org.netbeans.modules.visualweb.dataconnectivity.utils.SampleDatabaseCreator.registerDatabase(SampleDatabaseCreator.java:144)

Created attachment 69765 [details]
stacktrace

The problem is one thread tries to modify a Collection (of nodes) while another thread is iterating over it (line 118 of org.netbeans.modules.db.mysql.impl.InstallationManager.detectInstallation(...).

I think this exception occurs when importing 6.1 settings the first time starting 6.5, where MySQL had been previously
registered in 6.1.

I have a fix but I'm not sure if it's the right fix

1bfcf4e8d964

The fix does address this particular problem, however, it doesn't address the core issue: getInstallations() can be
called in multiple threads, which can result in the installations being added multiple times.

Integrated into 'main-golden', will be available in build *200809161401* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/1bfcf4e8d964
User: John Baker <jbaker@netbeans.org>
Log: #145839 org.netbeans.api.db.explorer.DatabaseException: java.util.ConcurrentModificationException

Thanks for catching this, Andrei.  

Here's one suggested solution.  First, synchronize the check for null at the beginning of the method, e.g.

synchronized(this) {
  if (INSTALLATIONS != null) {
    return INSTALLATIONS;
  }
}


Then when we instantiate and modify the INSTALLATIONS variable, synchronize this, and also check for null again, in case
another thread has come in and begun initialization of INSTALLATIONS:

synchronized(this) {
  if (INSTALLATIONS == null) {
    INSTALLATIONS = new ArrayList<Installation>();
    INSTALLATIONS.addAll(stackInstalls);
    INSTALLATIONS.addAll(stdInstalls);
  }
}

You don't want to synchronize the whole think because synchronizing a call to Lookup is a dangerous thing...

Just by making  INSTALLATIONS volatile should be sufficient.  Once INSTALLATIONS is set to some installation then there
shouldn't be any more installations. 

or better yet, add a new volatile flag and set to true when an installation is added

You're right, *if* you check for null both before you look up installations and sort them and *after* (just before you
add the list to the INSTALLATIONS variable)

Using a volatile flag would work too, you're right, and would be less expensive because there's less data that's not
being cached in registers.

50d570a813ae

Hi, John.  Sorry, this doesn't work.  It's possible for two different threads to detect that isInstalled is null and
proceed to initialize the INSTALLATIONS list.

Thread 1                      Thread 2
----------------------------  -------------------------------
isInstalled: false
                              isInstalled: false
initialize INSTALLATIONS
                              initialize INSTALLATIONS
set isInstalled to true
                              set isInstalled to true

volatile guarantees that you read the latest data from the field when multiple threads are modifying it, but it doesn't
guarantee atomic behavior like what I think you're assuming.

I am pretty sure you need to check again to see if isInstalled is true just before you modify the INSTALLATIONS list.  

Sorry, let me clarify: you need to set isInstalled to true just after you check it.  If you do anything after checking
isInstalled without set isInstalled to true then you run the risk of two threads doing it

so even if you do

if (! isInstalled) {
  INSTALLATIONS.addAll(x);
  INSTALLATIONS.addAll(y);
  isInstalled = true;
}

then two different threads could call the addAll() methods and you'll have duplicate installations in the list.

I think you need to do

if (! isInstalled) {
  isInstalled = true;
  INSTALLATIONS.addAll(x);
  INSTALLATIONS.addAll(y);
}

David

Or you could do:

if (! isInstalled) {
  isInstalled = true;
  
  // do lookup

  // sort lists into stack-based and non-stack-based

  // add lists to INSTALLATIONS
}

That would work, without having to check twice...

David

Good catch.  Right after I checked in the change I thought about setting the flag before rather than after registering
the installation, but I'm not certain if that's the best approach.

bf97fb7dab3e

Still not good enough I think. Imagine two threads executing getInstallations():

1. Thread T1 executes "if (!isInstalled)", but it is preempted by thread T2 just before executing the block of the if
statement.

2. Thread T2 executes "if (!isInstalled)", preempted by T1.

3. T1 executes first two statements of the if block.

4. T2 executes first two statements of the if block, rewriting the INSTALLATIONS field.

5. T1 executes last two statements of the if block.

6. T2 executes last two statements of the if block.

Both stackInstalls and stdInstalls have just been added twice.

Another problems are that:

- INSTALLATIONS should be volatile, otherwise the Java memory model doesn't guarantee threads other than the one that
set it will see the last set value.

- a thread executing detectInstallation() can still experience CME if it iterates INSTALLATIONS just during step 5 or 6
(when INSTALLATIONS is being modified). If the thread is really unlucky it can even experience a NPE.

- a thread executing detectInstallation() can see an empty INSTALLATIONS field if it iterates the field just before step 5.

Simply, you need to make sure to wait until getInstallations() has finished initializing the field. David's first
solution (splitting the initialization into two synchronized block) looks like a good approach.

By the way, I misread changeset 1bfcf4e8d964. It doesn't seem to do anything meaningful, please consider reverting it.

The original error was a ConcurrentModificationException.
The problem is one thread tries to modify a Collection (of nodes) while another thread is iterating over it (line 118 of
org.netbeans.modules.db.mysql.impl.InstallationManager.detectInstallation(...).

The other issue about concurrent problems with INSTALLATIONS could be filed as a separate issue.

btw, the synchronized blocks mentioned in desc21 won't work since the method is static - "this" is not permitted.

The original problem is not fixed IMHO:

> - a thread executing detectInstallation() can still experience CME 
> if it iterates INSTALLATIONS just during step 5 or 6
> (when INSTALLATIONS is being modified). 

maybe so, but not evident from all the attached stack traces

a67dc340d3b7

Should be better.  Synchronized the method passing the loadedInstallations

Integrated into 'main-golden', will be available in build *200809170201* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/50d570a813ae
User: John Baker <jbaker@netbeans.org>
Log: #145839 org.netbeans.api.db.explorer.DatabaseException: java.util.ConcurrentModificationException

I remember the my OS classes at the uni where our professor (one of the authors of HelenOS - microkernel based OS,
contributor to Linux kernel) told us:

"Locks do NOT work for synchronization of two threads, never use them, use Peterson instead."

Does the volatile keyword really help (on multi-processors)? I believe that "volatile X" means that "variable X can be
modified at any time by other thread, do not cache it in registers, always fetch again from memory".

If so, how does this fix prevent 2 threads from being executed inside the if block (what prevents 2 threads from
fetching variable isInstalled while it is false yet)?

Sorry for asking (maybe it is a really stupid question), I am just curious, lame student;)... (and I didn't use Java too
much)...

joshis: It doesn't.  Volatile only works to guarantee consistent reads.  It doesn't guarantee atomic writes.

John: if you are not going to fix the synchronization issue with INSTALLATION, please open a new issue.

Integrated into 'main-golden', will be available in build *200809171401* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/a67dc340d3b7
User: John Baker <jbaker@netbeans.org>
Log: #145839 org.netbeans.api.db.explorer.DatabaseException: java.util.ConcurrentModificationException

 
please see desc36

Verified

Product Version: NetBeans IDE Dev (Build 200810090201)
Java: 1.6.0_07-rev; Java HotSpot(TM) Client VM 10.0-b24
System: Linux version 2.6.24-19-generic running on i386; UTF-8; en_US (nb)