Bug 214647

Created attachment 121233 [details]
messages.log

What JDK do you use? We have recently seen some mysterious AWT related bugs.

From the threadump - obviously fallback keyring cannot proceed as it is waiting for AWT thread.

_However_ AWT thread is holding only one monitor and it is waiting on the same monitor which it already owns. Really strange. I do not see any reason why AWT should not proceed.

   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00000000d1398b40> (a java.util.WeakHashMap)
	at java.lang.Object.wait(Object.java:485)
	at org.netbeans.api.project.ProjectManager$2.run(ProjectManager.java:254)
	- locked <0x00000000d1398b40> (a java.util.WeakHashMap)
	at org.netbeans.api.project.ProjectManager$2.run(ProjectManager.java:234)
	at org.openide.util.Mutex.readAccess(Mutex.java:332)

Is this really thread dump from the moment IDE is deadlocked?

Ok now I got it (so we can ignore comment #2 ;) ). AWT is waiting in loop (on wait()) in ProjectManager for project to be loaded. "ErrorAnnotator worker" seems to be loading the project and it needs keyring and the keyring needs AWT -> deadlock.

It has never been a good idea to call Keyring on startup. If possible, it should be avoided.

Could this org.netbeans.modules.cloud.oracle.OracleInstanceManager.load be called lazily from org.netbeans.modules.cloud.oracle.serverplugin.OracleOptionalDeploymentManagerFactory.finishServerInitialization?

Created attachment 121255 [details]
possible untested fix

This could be one possible way of fixing this. It is a blind fix which has not been tested.

Patch looks good to me - delaying keyring access makes sense as it is not needed at that moment. Martin could you apply the patch and verify that it solves your problem? If so just push it please. Thx.

One thing that came to my mind later is that perhaps it would make sense to load username and password in the patch together as the common case is these are need together.

Created attachment 121265 [details]
alternative

I mean something like this. Though the original patch should work as well.

Are there some exact steps how to reproduce it? What triggers UI dialog in keyring API? From stack trace it look like some window with a deployment descriptor are being deserialized during IDE startup. I wonder how many user are impacted by this problem and whether we should try to resolve this in 7.2 or not. I tried to reproduce it but without success.

(In reply to comment #9)
> Are there some exact steps how to reproduce it? What triggers UI dialog in
> keyring API? 

Exact steps? Not sure on your environment. I have opened at least one project with Oracle Cloud Remote instance and broken native keyrings, see:

FINEST [org.netbeans.modules.keyring]: reading: org.netbeans.modules.cloud.oracle.username.Oracle Cloud
FINE [org.netbeans.modules.keyring.kde.KWalletProvider]: executing [qdbus, org.kde.kwalletd, /modules/kwalletd, org.kde.KWallet.isEnabled]
FINE [org.netbeans.modules.keyring.kde.KWalletProvider]: application exit with code 2 for commandString: [qdbus, org.kde.kwalletd, /modules/kwalletd, org.kde.KWallet.isEnabled]; errVal: Service 'org.kde.kwalletd' does not exist.
FINE [org.netbeans.modules.keyring.gnome.GnomeProvider]: no GNOME_KEYRING_* environment variable set
FINE [org.netbeans.modules.keyring.win32.Win32Protect]: not running on Windows

> From stack trace it look like some window with a deployment
> descriptor are being deserialized during IDE startup.

Sorry, really don't know about anything special. But agree that the issue shouldn't be facing plenty of users.

Anyway I tried both patches and both of them worked well and prevent the IDE from freezing. I'll settle the pushing with phejl today.

From the thread dump I would expect these are preconditions:
- cloud project
- -J-Dnetbeans.keyring.no.native=true
- opened task list (or what's the current name for the old task list)

Fixed in web-main 3892095d33f7.

Jirko can you verify?

As discussed with Jirka the patch unfortunately introduced the "Resolve missing server dialog" (and by looking to the code, perhaps WhiteListUpdater would not be properly initialized due to missing J2eePlatform).

I investigated the higher parts of call stack and noticed that WhiteListUpdater is registering itself as a listener from WebProject constructor. Perhaps it's too early so I tried to postpone it to phase when the project is really going to be opened. I can't verify right now whether this really helps (as Matej requires my attention :) ). However it might be a way. Attaching the patch against 8497d0a2f749 (the parent of unsuccessful patch) so try with "hg up -r 8497d0a2f749 && patch -p1 < the.patch".

Created attachment 121317 [details]
web project patch

(In reply to comment #13)
> As discussed with Jirka the patch unfortunately introduced the "Resolve missing
> server dialog" (and by looking to the code, perhaps WhiteListUpdater would not
> be properly initialized due to missing J2eePlatform).

Really? How?? I cannot see any functional change in the patch which could cause 'Resolve ...' dialog nor impact WhiteListUpdater. We just delayed loading of data from keyring. Am I missing something?

If there is a 'Resolve ...' dialog problem then it must be a independent issue I think which existed before.

Integrated into 'main-golden', will be available in build *201206260002* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main-golden/rev/3892095d33f7
User: Petr Hejl <phejl@netbeans.org>
Log: #214647 Reproducible deadlock oracle.cloud vs. non-native keyrings?

Verified in dev build 201206260002 using following steps:

- start IDE with empty userdir and -J-Dnetbeans.keyring.no.native=true
- register Oracle cloud
- create java web project with Oracle Cloud Remote target server (add JSF
framework in wizard)
- open Window|Action Items
- restart IDE
- it does not freeze but "Resolve missing server" dialog is opened and you have
to provide master password to get rid of it which is not pleasant. But if there
is no other possibility how to fix it we can live with that.

(In reply to comment #15)
> (In reply to comment #13)
> > As discussed with Jirka the patch unfortunately introduced the "Resolve missing
> > server dialog" (and by looking to the code, perhaps WhiteListUpdater would not
> > be properly initialized due to missing J2eePlatform).
> 
> Really? How?? I cannot see any functional change in the patch which could cause
> 'Resolve ...' dialog nor impact WhiteListUpdater. We just delayed loading of
> data from keyring. Am I missing something?
I've investigated bit more. We have delayed it and the keyring appears when initializing j2ee api instances OracleJ2EEServerInstanceProvider.refreshServersSynchronously(). So before it stopped the j2ee ServerRegistry (with possibility of deadlock). Now the j2ee server registry init proceeds and later IDE stops at asynch j2ee instances initalization.

> 
> If there is a 'Resolve ...' dialog problem then it must be a independent issue
> I think which existed before.
Yes I think, though it was not visible.

Created attachment 121346 [details]
j2ee instances fix

This patch "solves" the Missing server dialog issue. The dialog do not appear although I believe there is still possible race. The patch removes the duplication of username and password in j2ee part. As an additional benefit the keyring password is not requested until it is really needed (so no keyring dialog after IDE start).

David as you are the owner you should decide whether this should be part of the solution or whether this is separate P3.

I tested the solution and it seems to work ok.

(In reply to comment #19)
> Created attachment 121346 [details]
> j2ee instances fix
> 
> This patch "solves" the Missing server dialog issue. The dialog do not appear
> although I believe there is still possible race. The patch removes the
> duplication of username and password in j2ee part. As an additional benefit the
> keyring password is not requested until it is really needed (so no keyring
> dialog after IDE start).
> 
> David as you are the owner you should decide whether this should be part of the
> solution or whether this is separate P3.
> 
> I tested the solution and it seems to work ok.

I filed separate issue #214827.

changeset:   232295:6bb30be02ab3
branch:      release72
user:        Petr Hejl <phejl@netbeans.org>
date:        Mon Jun 25 10:05:51 2012 +0200
files:       cloud.oracle/src/org/netbeans/modules/cloud/oracle/OracleInstance.java cloud.oracle/src/org/netbeans/modules/cloud/oracle/OracleInstanceManager.java
description:
#214647 Reproducible deadlock oracle.cloud vs. non-native keyrings?
(transplanted from 3892095d33f77c5ff9259dc58f9c4bf47284e487)

Integrated into 'releases', will be available in build *201206272359* or newer. Wait for official and publicly available build.
Changeset: http://hg.netbeans.org/releases/rev/6bb30be02ab3
User: Petr Hejl <phejl@netbeans.org>
Log: #214647 Reproducible deadlock oracle.cloud vs. non-native keyrings?
(transplanted from 3892095d33f77c5ff9259dc58f9c4bf47284e487)

Verified in NetBeans IDE 7.2 RC1 (Build 201206272359).