I've found that when it works, the new java editor is very fast and very nice. However, I've also found that it freezes
solid for seconds at a time quite often. I've attached a thread dump I snapped when this happened just now (taken via
^\) that shows 3 separate threads (including AWT, hence the frozen editor) all blocked by a lock, but no clear owner of
the lock. Any idea what's up?
Created attachment 45260 [details]
thread stack djm
Created attachment 45265 [details]
Second occurence, same IDE instance.
Seems very strange, the threads are waiting on lock which no one holds, the lock is private to JavaSource and no other
thread has JavaSource on stack trace. It may be related to biased locking, can you try to add -XX:-UseBiasedLocking to
Created attachment 46995 [details]
trace 3 - NB6 build from around 8/17
Created attachment 46996 [details]
trace 4, same instance as trace 3 but 40 minutes later.
Created attachment 46997 [details]
Output of 'jstack -F' when "waking up the ide" to get trace 4, above.
Created attachment 47053 [details]
I've attached a DTrace script monitoring enter|exit from JavaSource internal lock, it needs a build from today (first
build having the change: http://deadlock.netbeans.org/hudson/job/trunk/2701/). You need to modify the
netbeans/etc/netbeans.conf and add -J-XX:+DTraceMonitorProbes to the netbeans_default_options property. The you need to
run the d script with PID of the java proces and store its output into log (./monitor-enter-time.d PID | tee log), the
log may be long but it may help.
According to the response from the hotspot team, there is a bug in the JDK 1.6 (already fixed) causing wrong thread
status in thread dump (http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6471736). It would be helpful if you can
generate both the java thread dump and also the pstack when the IDE freezes.
Can you try to run the IDE on JDK 1.5 if it has the same problem, if so the thread dump from the JDK 1.5 should be
correct. Probably the fastest test.
Thanks, I'll see what data I can get with that.
I attached a pstack trace from a lockup that occurred to day... don't have the corresponding java dump because I forgot
to hit ^\ first, but I can get a matched pair later if you want.
I don't see anything obviously wrong with it except that the thread that appears to be AWT (not certain) was stopped at
at a call labeled "__1cFStall6F_v_" which was a bit unexpected. I wonder what it does.
Created attachment 47122 [details]
output of pstack -F for another lockup.
Created attachment 47220 [details]
messages.log for last session (~8/17 ide build)
Created attachment 47221 [details]
Possbibly related exception
I don't know if it will help, but before migrating to the new build that has DTrace support, I attached my messages.log
(from IDE session that generated last round of lockups).
I also attached an IllegalStateException stack trace that I often see either just before or just after a lockup (not
sure which). I do not know if there is a connection. It does not always appear, but may be an issue on it's own as it
is fairly common.
Alright - new IDE build - 8/23 sources, new userdir, dtrace enabled.
Attached is the following:
pstack/jstack output (one file, both traces)
dtrace log from the script mentioned above.
^\ thread dump from console + a few extra lines that might provide context.
Created attachment 47227 [details]
pstack and jstack output.
Created attachment 47228 [details]
Created attachment 47229 [details]
I don't see much in the logs.
Thread #29 is the (blocked) AWT thread, native dump says:
fed186b0 __1cFStall6F_v_ (0, 83b7b2c, fee32000, fc828e48, cd3c7108, fed24300) + c
feaa9208 __1cSObjectSynchronizerHinflate6FpnHoopDesc__pnNObjectMonitor__ (d1e6be10) + 11c
fea8d514 __1cSObjectSynchronizerKslow_enter6FnGHandle_pnJBasicLock_pnGThread__v_ (83b7b2c, cd3c71ec, 83b7800) + f8
fed18d19 __1cSObjectSynchronizerKfast_enter6FnGHandle_pnJBasicLock_bpnGThread__v_ (83b7b2c, cd3c71ec, 1, 83b7800) + 75
feabfb41 __1cSInterpreterRuntimeMmonitorenter6FpnKJavaThread_pnPBasicObjectLock__v_ (83b7800, cd3c71ec, fc610de1, 3,
d1e6be10, cd3c71ec) + 161
so it is still inside monitorEnter. Nobody owns the lock and the thread is already marked as runnable, but also as
"waiting on condition [0xcd3c6000..0xcd3c7bf0]".
The CMS thread seems to be idle in sleepBeforeNextCycle(), but the oldgen is ~79% full.
In the dtrace log, the thread #29 is never mentioned at all even though it should have entered/exited many locks during
the log period (unless the dtrace was started after the hang).
The dtrace process was started at least 20 minutes if not an hour before the hang (and snapshotted immediately after, as
with all other attachments.)
However, it was not started simulataneously with the JVM. I will modify the start process to do that if necessary. I
didn't think it would matter for this sort of problem.
Here's another round of attachments. This time, when the editor locked up, I went to get coffee and chit-chat with
people... editor was still frozen when I got back 10 minutes later and took the snapshots.
Created attachment 47354 [details]
Created attachment 47355 [details]
Created attachment 47356 [details]
dtrace log (long)
Created attachment 47357 [details]
^\ thread stack dump from console
Thanks Pete for the new logs I will send them to hotspot team and we will see, unfortunately I don't see the reason why
the threads are blocked even no one hods the lock. It may be something related to concurrent mark and sweep which was
turned on. You can try to disable it by deleting -J-XX:+UseConcMarkSweepGC from the netbeans/etc/netbeans.conf. However
this is just a try.
I switched to JDK 5 later on Friday and ran w/o incident. A couple of brief glitches, but no clear cause, could easily
have been saturated I/O bus reading disk files, I have no idea.
Might be a bug in JDK 6 locking. I'll run with JDK 5 for a few more days and then try the Biased locking switch you
Unfortunately, on this system, I cannot run NetBeans with any version of JDK 6 other than JDK 1.6.0_00 because of
insufficient Solaris patch level and non-upgrade / non-patchable status of the particular Solaris Express build I
installed. Upgrading will just have to wait until I have time to install a new Solaris build.
it's a JDK bug, in the JDK bug 6518490 (http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6518490) there is a
workaround which should help, can you try it if it works for you? I believe that the other Solaris users may have the
same problem. The workaround is adding the -XX:-UseThreadPriorities option. I would rather also disable the biased
locking by adding the -XX:-UseBiasedLocking option. I will attach the modified netbeans.conf file. Can you try to
replace the original netbeans/etc/netbeans.conf by the attached one and run the IDE if it helps.
Created attachment 47635 [details]
Thanks, I read the reports regarding UseThreadPriorities. I'll try that. I will say, after using JDK 5 for 2 days, JDK
6 is _so_ much faster. It'd be nice to go back, if that works.
I will also try to split the lock in JavaSource into 2 independent locks (as it was originally), maybe it will help too.
Thanks for your help in tracking this down.
Created attachment 47738 [details]
JavaSource with rewritten locking
did the options help?
I've also created an java source module with a bit changed locking. Can you try it without the options I've attached
yesterday? I doubt it will help but I am looking for most painless way how to solve it in beta 1. If the rewritten
locking helps I will commit it otherwise performance team has to change the startup script and put the -XX options into
in when it's started on Solaris or at least document it.
The runtime options seem to have helped, but I haven't been doing the kind of heavy editing in the past 2 days that I
was doing last week so it's difficult to be certain just yet. I will run with just the option changes for a little
while longer and then add your patch and turn the options off to see if the issue comes back.
When the problem does not occur, it is very difficult to tell if that is because it's really gone or because it just
hasn't had the right opportunity to appear.
Pete the modified java-source.jar doesn't work,in addition to this it introduces new kind of problems. So you don't have
to test it. The bug should be fixed in the JDK 1.6 update 3, until the fix we have to use the -XX flags on Solaris. I
will reassign this issue to Radim who maintains the netbeans launcher scripts.
The hang is caused by the problem in the jvm which should be fixed in the JDK 1.6.0 update 3. In the hotspot bugreport
#6518490 contains a workaround how to prevent it which seems to work fine. The netbeans start up script should add the
-XX:-UseThreadPriorities -XX:-UseBiasedLocking option when running on Solaris and JDK 1.6 older than update 3.
*** Issue 108831 has been marked as a duplicate of this issue. ***
What is the status of this issue? Were the options already included in the launcher script? The owner should be probably
changed since Radim no more works in performance team.
Reassigning to Petr.
I am not sure we want to include -XX:-UseThreadPriorities -XX:-UseBiasedLocking startup flags only for the case when the
user has Solaris and an older JDK than the recent one. The more corner cases like this in the launcher code, the more
maintenance problems in the future.
Also adding these options for all configurations is not a good idea too. These options are not completely harmless, are
they? Otherwise they would be on by default.
Petr, what's your opinion about these startup flags?
OK,it is possible to add the flags on Solaris.
It is not possible to add them only for certain JDK version (we don't have this information during startup).
Another option would be to add them to the netbeans.conf from the installer.
Wouldn't it be enough to just RELNOTE this?
Talked to Petr about the flags and we decided to do nothing in launcher nor netbeans.conf.
The problem appears on Solaris if JDK 1.6.0 or 1.6.0_01 or 1.6.0_02 is used.
The problem is not there with JDK 1.5.x and with JDK 1.6.0_03.
Adding RELNOTE keyword. I will ask docs to include a mention of this problem in release notes. The message could be:
"Issue #110074: Editor periodically freezes on Solaris.
Description: When running NetBeans on Solaris with JDK 6, 6u1 or 6u2, threads may become blocked for a long time and
Workaround: Upgrade your JDK 6 installation to the Update 3 or newer."
After talking to the docs team, I'm closing this as WONTFIX/JDK issue. Docs team will relnote the workaround (newer JDK).
There are some new reports of similar problem even with JDK1.6.0_03.
If they prove to be this problem, we should reopen this and add the flags to the launcher when running on Solaris.