Bug 110074 - java editor periodically freezes for seconds at a time.
java editor periodically freezes for seconds at a time.
Product: ide
Classification: Unclassified
Component: Performance
Sun Solaris
: P2 (vote)
Assigned To: Petr Nejedly
: 108831 (view as bug list)
Depends on:
  Show dependency treegraph
Reported: 2007-07-17 22:23 UTC by _ pcw
Modified: 2011-05-25 11:37 UTC (History)
8 users (show)

See Also:
Issue Type: DEFECT

thread stack djm (17.48 KB, application/octet-stream)
2007-07-17 22:24 UTC, _ pcw
Second occurence, same IDE instance. (24.18 KB, text/plain)
2007-07-17 23:21 UTC, _ pcw
trace 3 - NB6 build from around 8/17 (21.26 KB, text/plain)
2007-08-21 18:58 UTC, _ pcw
trace 4, same instance as trace 3 but 40 minutes later. (20.59 KB, text/plain)
2007-08-21 18:58 UTC, _ pcw
Output of 'jstack -F' when "waking up the ide" to get trace 4, above. (1.27 KB, text/plain)
2007-08-21 18:59 UTC, _ pcw
Dtrace script (733 bytes, text/plain)
2007-08-22 12:38 UTC, Tomas Zezula
output of pstack -F for another lockup. (60.71 KB, application/octet-stream)
2007-08-23 02:32 UTC, _ pcw
messages.log for last session (~8/17 ide build) (53.05 KB, text/plain)
2007-08-23 22:48 UTC, _ pcw
Possbibly related exception (3.50 KB, text/plain)
2007-08-23 22:49 UTC, _ pcw
pstack and jstack output. (47.02 KB, text/plain)
2007-08-23 23:51 UTC, _ pcw
dtrace output (122.57 KB, text/plain)
2007-08-23 23:52 UTC, _ pcw
thread dump. (20.53 KB, text/plain)
2007-08-23 23:53 UTC, _ pcw
pstack/jstack dump (51.84 KB, application/octet-stream)
2007-08-24 21:16 UTC, _ pcw
pstack/jstack dump (51.84 KB, text/plain)
2007-08-24 21:16 UTC, _ pcw
dtrace log (long) (1023.28 KB, text/plain)
2007-08-24 21:17 UTC, _ pcw
^\ thread stack dump from console (21.37 KB, text/plain)
2007-08-24 21:18 UTC, _ pcw
Modified netbeans.conf (1.09 KB, text/plain)
2007-08-28 16:46 UTC, Tomas Zezula
JavaSource with rewritten locking (2.20 MB, application/octet-stream)
2007-08-29 17:48 UTC, Tomas Zezula

Note You need to log in before you can comment on or make changes to this bug.
Description _ pcw 2007-07-17 22:23:32 UTC
I've found that when it works, the new java editor is very fast and very nice.  However, I've also found that it freezes
solid for seconds at a time quite often.  I've attached a thread dump I snapped when this happened just now (taken via
^\) that shows 3 separate threads (including AWT, hence the frozen editor) all blocked by a lock, but no clear owner of
the lock.  Any idea what's up?
Comment 1 _ pcw 2007-07-17 22:24:00 UTC
Created attachment 45260 [details]
thread stack djm
Comment 2 _ pcw 2007-07-17 23:21:54 UTC
Created attachment 45265 [details]
Second occurence, same IDE instance.
Comment 3 Tomas Zezula 2007-08-21 12:47:42 UTC
Seems very strange, the threads are waiting on lock which no one holds, the lock is private to JavaSource and no other
thread has JavaSource on stack trace. It may be related to biased locking, can you try to add -XX:-UseBiasedLocking to
your netbeans.conf?
Comment 4 _ pcw 2007-08-21 18:58:21 UTC
Created attachment 46995 [details]
trace 3 - NB6 build from around 8/17
Comment 5 _ pcw 2007-08-21 18:58:48 UTC
Created attachment 46996 [details]
trace 4, same instance as trace 3 but 40 minutes later.
Comment 6 _ pcw 2007-08-21 18:59:22 UTC
Created attachment 46997 [details]
Output of 'jstack -F' when "waking up the ide" to get trace 4, above.
Comment 7 Tomas Zezula 2007-08-22 12:38:06 UTC
Created attachment 47053 [details]
Dtrace script
Comment 8 Tomas Zezula 2007-08-22 12:55:23 UTC
I've attached a DTrace script monitoring enter|exit from JavaSource internal lock, it needs a build from today (first
build having the change: http://deadlock.netbeans.org/hudson/job/trunk/2701/). You need to modify the
netbeans/etc/netbeans.conf and add -J-XX:+DTraceMonitorProbes to the netbeans_default_options property. The you need to
run the d script with PID of the java proces and store its output into log (./monitor-enter-time.d PID | tee log), the
log may be long but it may help.

According to the response from the hotspot team, there is a bug in the JDK 1.6 (already fixed) causing wrong thread
status in thread dump (http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6471736). It would be helpful if you can
generate both the java thread dump and also the pstack when the IDE freezes.

Can you try to run the IDE on JDK 1.5 if it has the same problem, if so the thread dump from the JDK 1.5 should be
correct. Probably the fastest test.


Comment 9 _ pcw 2007-08-23 02:31:44 UTC
Thanks, I'll see what data I can get with that.

I attached a pstack trace from a lockup that occurred to day... don't have the corresponding java dump because I forgot
to hit ^\ first, but I can get a matched pair later if you want.

I don't see anything obviously wrong with it except that the thread that appears to be AWT (not certain) was stopped at
at a call labeled "__1cFStall6F_v_" which was a bit unexpected. I wonder what it does.
Comment 10 _ pcw 2007-08-23 02:32:21 UTC
Created attachment 47122 [details]
output of pstack -F for another lockup.
Comment 11 _ pcw 2007-08-23 22:48:39 UTC
Created attachment 47220 [details]
messages.log for last session (~8/17 ide build)
Comment 12 _ pcw 2007-08-23 22:49:08 UTC
Created attachment 47221 [details]
Possbibly related exception
Comment 13 _ pcw 2007-08-23 22:52:03 UTC
I don't know if it will help, but before migrating to the new build that has DTrace support, I attached my messages.log
(from IDE session that generated last round of lockups).

I also attached an IllegalStateException stack trace that I often see either just before or just after a lockup (not
sure which).  I do not know if there is a connection.  It does not always appear, but may be an issue on it's own as it
is fairly common.
Comment 14 _ pcw 2007-08-23 23:50:17 UTC
Alright - new IDE build - 8/23 sources, new userdir, dtrace enabled.
Attached is the following:

pstack/jstack output (one file, both traces)
dtrace log from the script mentioned above.
^\ thread dump from console + a few extra lines that might provide context.

Comment 15 _ pcw 2007-08-23 23:51:32 UTC
Created attachment 47227 [details]
pstack and jstack output.
Comment 16 _ pcw 2007-08-23 23:52:43 UTC
Created attachment 47228 [details]
dtrace output
Comment 17 _ pcw 2007-08-23 23:53:12 UTC
Created attachment 47229 [details]
thread dump.
Comment 18 Petr Nejedly 2007-08-24 15:20:37 UTC
I don't see much in the logs.
Thread #29 is the (blocked) AWT thread, native dump says:

 fed186b0 __1cFStall6F_v_ (0, 83b7b2c, fee32000, fc828e48, cd3c7108, fed24300) + c
 feaa9208 __1cSObjectSynchronizerHinflate6FpnHoopDesc__pnNObjectMonitor__ (d1e6be10) + 11c
 fea8d514 __1cSObjectSynchronizerKslow_enter6FnGHandle_pnJBasicLock_pnGThread__v_ (83b7b2c, cd3c71ec, 83b7800) + f8
 fed18d19 __1cSObjectSynchronizerKfast_enter6FnGHandle_pnJBasicLock_bpnGThread__v_ (83b7b2c, cd3c71ec, 1, 83b7800) + 75
 feabfb41 __1cSInterpreterRuntimeMmonitorenter6FpnKJavaThread_pnPBasicObjectLock__v_ (83b7800, cd3c71ec, fc610de1, 3,
d1e6be10, cd3c71ec) + 161
so it is still inside monitorEnter. Nobody owns the lock and the thread is already marked as runnable, but also as
"waiting on condition [0xcd3c6000..0xcd3c7bf0]".

The CMS thread seems to be idle in sleepBeforeNextCycle(), but the oldgen is ~79% full.

In the dtrace log, the thread #29 is never mentioned at all even though it should have entered/exited many locks during
the log period (unless the dtrace was started after the hang).
Comment 19 _ pcw 2007-08-24 16:24:25 UTC
The dtrace process was started at least 20 minutes if not an hour before the hang (and snapshotted immediately after, as
with all other attachments.)

However, it was not started simulataneously with the JVM.  I will modify the start process to do that if necessary.  I
didn't think it would matter for this sort of problem.
Comment 20 _ pcw 2007-08-24 21:14:26 UTC
Here's another round of attachments.  This time, when the editor locked up, I went to get coffee and chit-chat with
people... editor was still frozen when I got back 10 minutes later and took the snapshots.
Comment 21 _ pcw 2007-08-24 21:16:31 UTC
Created attachment 47354 [details]
pstack/jstack dump
Comment 22 _ pcw 2007-08-24 21:16:42 UTC
Created attachment 47355 [details]
pstack/jstack dump
Comment 23 _ pcw 2007-08-24 21:17:35 UTC
Created attachment 47356 [details]
dtrace log (long)
Comment 24 _ pcw 2007-08-24 21:18:21 UTC
Created attachment 47357 [details]
^\ thread stack dump from console
Comment 25 Tomas Zezula 2007-08-26 18:57:25 UTC
Thanks Pete for the new logs I will send them to hotspot team and we will see, unfortunately I don't see the reason why
the threads are blocked even no one hods the lock. It may be something related to concurrent mark and sweep which was
turned on. You can try to disable it by deleting -J-XX:+UseConcMarkSweepGC from the netbeans/etc/netbeans.conf. However
this is just a try.
Comment 26 _ pcw 2007-08-27 06:14:28 UTC
I switched to JDK 5 later on Friday and ran w/o incident.  A couple of brief glitches, but no clear cause, could easily
have been saturated I/O bus reading disk files, I have no idea.

Might be a bug in JDK 6 locking.  I'll run with JDK 5 for a few more days and then try the Biased locking switch you

Unfortunately, on this system, I cannot run NetBeans with any version of JDK 6 other than JDK 1.6.0_00 because of
insufficient Solaris patch level and non-upgrade / non-patchable status of the particular Solaris Express build I
installed.  Upgrading will just have to wait until I have time to install a new Solaris build.
Comment 27 Tomas Zezula 2007-08-28 16:45:24 UTC
Hi Pete,
it's a JDK bug, in the JDK bug 6518490 (http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6518490) there is a
workaround which should help, can you try it if it works for you? I believe that the other Solaris users may have the
same problem. The workaround is adding the -XX:-UseThreadPriorities option. I would rather also disable the biased
locking by adding the -XX:-UseBiasedLocking option. I will attach the modified netbeans.conf file. Can you try to
replace the original netbeans/etc/netbeans.conf by the attached one and run the IDE if it helps.
Comment 28 Tomas Zezula 2007-08-28 16:46:16 UTC
Created attachment 47635 [details]
Modified netbeans.conf
Comment 29 _ pcw 2007-08-28 16:58:01 UTC
Thanks, I read the reports regarding UseThreadPriorities.  I'll try that.  I will say, after using JDK 5 for 2 days, JDK
6 is _so_ much faster.  It'd be nice to go back, if that works.
Comment 30 Tomas Zezula 2007-08-28 17:35:20 UTC
I will also try to split the lock in JavaSource into 2 independent locks (as it was originally), maybe it will help too.
Comment 31 _ pcw 2007-08-28 17:52:14 UTC
Thanks for your help in tracking this down.
Comment 32 Tomas Zezula 2007-08-29 17:48:24 UTC
Created attachment 47738 [details]
JavaSource with rewritten locking
Comment 33 Tomas Zezula 2007-08-29 17:54:30 UTC
Hi Pete,
did the options help?
I've also created an java source module with a bit changed locking. Can you try it without the options I've attached
yesterday? I doubt it will help but I am looking for most painless way how to solve it in beta 1. If the rewritten
locking helps I will commit it otherwise performance team has to change the startup script and put the -XX options into
in when it's started on Solaris or at least document it.
Comment 34 _ pcw 2007-08-29 19:07:54 UTC
The runtime options seem to have helped, but I haven't been doing the kind of heavy editing in the past 2 days that I
was doing last week so it's difficult to be certain just yet.  I will run with just the option changes for a little
while longer and then add your patch and turn the options off to see if the issue comes back.

When the problem does not occur, it is very difficult to tell if that is because it's really gone or because it just
hasn't had the right opportunity to appear.
Comment 35 Tomas Zezula 2007-08-30 12:33:02 UTC
Pete the modified java-source.jar doesn't work,in addition to this it introduces new kind of problems. So you don't have
to test it. The bug should be fixed in the JDK 1.6 update 3, until the fix we have to use the -XX flags on Solaris. I
will reassign this issue to Radim who maintains the netbeans launcher scripts.
Comment 36 Tomas Zezula 2007-08-30 12:38:37 UTC
The hang is caused by the problem in the jvm which should be fixed in the JDK 1.6.0 update 3. In the hotspot bugreport
#6518490 contains a workaround how to prevent it which seems to work fine. The netbeans start up script should add the
-XX:-UseThreadPriorities -XX:-UseBiasedLocking option when running on Solaris and JDK 1.6 older than update 3.
Comment 37 Tomas Zezula 2007-09-20 08:21:42 UTC
*** Issue 108831 has been marked as a duplicate of this issue. ***
Comment 38 Tomas Zezula 2007-10-10 14:49:28 UTC
What is the status of this issue? Were the options already included in the launcher script? The owner should be probably
changed since Radim no more works in performance team.
Comment 39 Antonin Nebuzelsky 2007-10-16 14:12:56 UTC
Reassigning to Petr.

I am not sure we want to include -XX:-UseThreadPriorities -XX:-UseBiasedLocking startup flags only for the case when the
user has Solaris and an older JDK than the recent one. The more corner cases like this in the launcher code, the more
maintenance problems in the future.

Also adding these options for all configurations is not a good idea too. These options are not completely harmless, are
they? Otherwise they would be on by default.

Petr, what's your opinion about these startup flags?
Comment 40 Petr Nejedly 2007-10-16 14:33:24 UTC
OK,it is possible to add the flags on Solaris.
It is not possible to add them only for certain JDK version (we don't have this information during startup).
Another option would be to add them to the netbeans.conf from the installer.

Wouldn't it be enough to just RELNOTE this?
Comment 41 Antonin Nebuzelsky 2007-10-17 10:48:11 UTC
Talked to Petr about the flags and we decided to do nothing in launcher nor netbeans.conf.

The problem appears on Solaris if JDK 1.6.0 or 1.6.0_01 or 1.6.0_02 is used.

The problem is not there with JDK 1.5.x and with JDK 1.6.0_03.

Adding RELNOTE keyword. I will ask docs to include a mention of this problem in release notes. The message could be:

"Issue #110074: Editor periodically freezes on Solaris.
Description: When running NetBeans on Solaris with JDK 6, 6u1 or 6u2, threads may become blocked for a long time and
editor unresponsive.
Workaround: Upgrade your JDK 6 installation to the Update 3 or newer."
Comment 42 Petr Nejedly 2007-10-23 15:30:48 UTC
After talking to the docs team, I'm closing this as WONTFIX/JDK issue. Docs team will relnote the workaround (newer JDK).
Comment 43 Petr Nejedly 2007-11-22 13:46:34 UTC
There are some new reports of similar problem even with JDK1.6.0_03.
If they prove to be this problem, we should reopen this and add the flags to the launcher when running on Solaris.

By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo