Bug 197158

Summary: NB 7.0 (NetCAT) becomes sluggish after stopping debugger
Product: debugger Reporter: aquaglia <aquaglia>
Component: JavaAssignee: Martin Entlicher <mentlicher>
Status: VERIFIED FIXED QA Contact: issues <issues.netbeans.org>
Priority: P2 CC: mentlicher, mmirilovic, tzezula
Version: 7.0   
Target Milestone: 7.0.1   
Hardware: PC   
OS: Windows XP   
Whiteboard: 70patch1-fixed
Issue Type: DEFECT Exception Report:
Attachments: NB 7 RC1 Threrad dump
Profile Me results
Profiler view
ide log
nps file
memory usage of my PC

Description aquaglia 2011-03-28 15:09:39 UTC
Created attachment 107340 [details]
NB 7 RC1 Threrad dump

This problem has been there for a while.
After interrupting a debugging session, NB becomes sluggish and a single operation (even just a click on a button) can require a few seconds

The w/a is to restart NB.

I have attached a Thread Dump.
Comment 1 Martin Entlicher 2011-04-06 15:38:09 UTC
The debugger session is not fully stopped yet - there's still visible activity of retrieval of variable values that are probably displayed in Variables or Watches tab. It'd be better to have a profiler snapshot.
AWT thread is doing repaint of a debugger table with data, do you still have some debugger window opened?
Do you have some concrete steps how we can reproduce it?
Comment 2 aquaglia 2011-04-08 15:35:11 UTC
I will try to find a reproducible test case using the just release 7.0 RC2.
Comment 3 aquaglia 2011-04-15 10:23:14 UTC
Created attachment 107766 [details]
Profile Me results
Comment 4 aquaglia 2011-04-15 10:24:21 UTC
Created attachment 107767 [details]
Profiler view
Comment 5 aquaglia 2011-04-15 10:27:12 UTC
NOW I AM USING NB 7.0 released to NetCAT members.

It is not due to an interruption of the debugger but happens after running debugging sessions to completion.

Whatever the reason is, I must restart the IDE because every UI action requires seconds to execute. Hence, I have raised the priority to 2.
Comment 6 Martin Entlicher 2011-04-15 12:56:35 UTC
Can you please attach the original .nps or .npss file? I'm not able to visualize the .xml file. Thanks.
Comment 7 Martin Entlicher 2011-04-19 14:40:05 UTC
It's a pain to search for the problem in 40MB XML file. Where I do not see the CPU time.
However the problem does not seem to be in debugger. The time spent in debugger classes is under 10%.
AWT is busy with repaints of editor and other windows, but the time spent there is also low.
There's an ongoing Java parsing, which seems to consume most of the time.
25% of the time is spent in DiffSidebar

Moving to parsing for evaluation...
Comment 8 Tomas Zezula 2011-04-19 16:16:35 UTC
Does not have anything in common with parsing api.
There is some java parsing which Martin pointing on but it seems to be just file reparse (blocked by VCS). It's really hard to find something in 40 MB XML file which nothing except vi can open.
It may be a memory leak and time is spent in full gc.
Please attach the nps file as Martin requested before and netbeans log file.
The memory usage will be also welcomed (how much of the heap is used?).
Without this info it's even impossible to assign the bug to correct category.
Thanks in advance
Comment 9 aquaglia 2011-04-19 18:22:46 UTC
I will be very happy to give you the nps file if you just explain to me how to get it.
In order to produce the xml file I folowed the instructions published here:
http://wiki.netbeans.org/GenerateThreadDump

I have no problem opening the file. I am on windows. Use TextPad for example. It is very fast.


This problem happens very, very often and is very annoying.

Please note that I open bugs in the interest of the NetBeans Community because I think NetBeans is a great product.

The NetBeans Community is all of us.
Comment 10 Martin Entlicher 2011-04-19 22:07:54 UTC
When you generate the Profiler snapshot, please press the save icon (Save Snapshot to Project), then go to Info tab and copy the File path. Attach that file here please. Or use Export to... but be sure to select "Profile Snapshot File (*.nps)" file type.
Thanks.
Comment 11 Tomas Zezula 2011-04-20 07:24:03 UTC
Please don't forgot also to attach the memory usage and ide.log (View/IDE log).
Thanks
Comment 12 aquaglia 2011-04-20 08:59:26 UTC
Created attachment 107853 [details]
ide log
Comment 13 aquaglia 2011-04-20 08:59:53 UTC
Created attachment 107854 [details]
nps file
Comment 14 aquaglia 2011-04-20 09:03:37 UTC
Created attachment 107855 [details]
memory usage of my PC
Comment 15 aquaglia 2011-04-20 09:03:57 UTC
Many thanks for the detailed isntructions.
I have uploaded the requested files.
The memory usage displayed by the IDE was 386.2/396.4 MB and I have uploaded a screen shot about the memory of my PC.
Comment 16 Tomas Zezula 2011-04-20 13:35:51 UTC
Thank you very much for the data.
It seems that my estimate was right the debugger session created a memory leak. When the heap usage is over ~ 70% the hot spots switches from concurrent gc to stop the world gc (when gc is running all the other threads are stopped). As there is a leak and hot spot is not able to free any reasonable amount of memory it repeats the gc cycles.

As I sad before it has nothing in common with parsing.api the debugger or something leaked the memory there is only 10MB free on the heap (386.2/396.4 MB). There is no parsing.api running there is only java hints running on single file which took 12sec but it's cancellable task which is canceled as soon as you start type in the editor. It takes so long because the gc cycles.

Now I will need a heap dump which is a bit harder task.
First you need to get the IDE into the sluggish state

Then from command line find out the netbeans process id:
jps

it should print something like:

578 Main
3583 Jps

where the Main should be the IDE

After it run:

jmap -dump:format=b,file=heap.bin PID

in my case PID = 578
it will create a file heap.bin which contains the heap dump.

Then you need to share the file somehow. Unfortunately the issuzilla does not
allow such a big attachments (it will be quite big file) but you can use magashare or some other
site.

If you have any questions feel free to ask.
Thanks
Comment 17 aquaglia 2011-04-21 10:29:18 UTC
Hi,
thanks for the detailed instructions.

The file is available here:

http://dl.dropbox.com/u/4422938/heap.bin

The file (~400MB) was generated but at some point jmap made NetBeans crash.
Comment 18 Tomas Zezula 2011-04-21 10:40:06 UTC
Thank you very much!
I am now downloading the file, hopefully it's complete and profiler will be able to open it.
Comment 19 Tomas Zezula 2011-04-21 11:45:44 UTC
The heap is complete the crash of VM happened after the dump was created.
Thanks
Comment 20 Tomas Zezula 2011-04-21 12:09:14 UTC
The most of memory is occupied by subversions DiskMapTurboProvider what I have no idea what it is. There is also 8 javac instances I will look at javac instances and then reassign (or create a new issue) for the subversion.
Comment 21 Tomas Zezula 2011-04-21 16:48:08 UTC
There are also lots of Strings hold by debugger on the stack frame of  "JDI Internal Event Handler"
Comment 22 aquaglia 2011-04-21 17:56:16 UTC
Thanks for the update.

I have put together a second testcase.

The IDE was taking a few seconds between actions which made it practically unusable.

http://dl.dropbox.com/u/4422938/nbsluggish2/snapshot-1303407463823.nps
http://dl.dropbox.com/u/4422938/nbsluggish2/heap.bin
Comment 23 Tomas Zezula 2011-04-22 07:42:56 UTC
Thanks.

The second heap dump is a bit different.
There is also lots of string hold by subversion (80MB) and there are also strings hold by Tomcat support.
The debugger still holds lots of sting in "Debugger operator thread" by Operator class, which seems to me as a bug as you ended the debugger.

It seems that the biggest problem is in Subversion and debugger.
If you have some time can you create application snapshot (it shows the VM statistics)?

You will need a VisualVM, you can download it here: http://visualvm.java.net/
Then you start the visual vm in the Applications (Local) select NetBeans and choose "Open".
Then you start debugging and finishes it and do some work to see if the IDE is slow.
Finally in VisualVM you invoke "Application Snapshot" on NetBeans (in pop up menu)
In VisualVM in Applications under the Snapshots node a new sub node appears and it has "Save As..." action.
Please attach the snapshot.

Thanks!
Comment 24 aquaglia 2011-04-22 13:22:23 UTC
VisualVM is a great tool. Thanks a lot for pointing me to it.

The Application Snapshot is available here:
http://dl.dropbox.com/u/4422938/nbsluggish3/application-1303478096740.apps

The second testcase I uploaded refers to a different set of projects.
The second one is about Java and Java Web applications built with Ant.

The first one was about several Maven projects with thousands of files.

All project files are saved in Subversion repositories.

The degradation of performance is gradual and appears after a couple of hours of work with a few debugging sessions. I cannot tell if the same problem would appear without debugging sessions since these are an integral part of my daily work.
However, it is my impression that it gets suddenly worse just after a debugging session.

Many thanks for your assistance,
Angelo
Comment 25 aquaglia 2011-04-22 16:30:41 UTC
One mroe application snapshot:

http://dl.dropbox.com/u/4422938/nbsluggish3/application-1303489871562.apps
Comment 26 aquaglia 2011-04-26 08:18:24 UTC
This is morning NB turned sluggish immediately after the first debugging session.

Almost every click is subject to a wait of about 1-2 seconds.

http://dl.dropbox.com/u/4422938/nbsluggish3/application-1303805808423.apps
Comment 27 aquaglia 2011-04-26 08:21:43 UTC
Let me try with the release 7.0...
Comment 28 Tomas Zezula 2011-04-26 08:39:51 UTC
The problem with hold source is already fixed 8445662e0bfe  by Marek issue #197458.
So the problem is debugger and subversion cache.
The debugger holds string on stack and VCS holds 70-80MB in some cache.
I will look to the newest snapshot and then reassign to debugger which should evaluate it is could not clean the stings when debugger connection is closed and reassign to subversion.
Comment 29 Tomas Zezula 2011-04-26 12:56:24 UTC
One more problem is in EditorContextImpl:

 sourceHandles = new WeakHashMap<JavaSource, JavaSourceUtil.Handle>();

Which holds 4 JavaSourceUtil.Handles but there is just one JavaSource (other are gced).
Comment 30 aquaglia 2011-04-27 07:51:50 UTC
It affects usability quite heavily.
Comment 31 Martin Entlicher 2011-04-27 12:40:25 UTC
The WeakHashMap should be cleared by SessionsListener, which was already introduced by http://hg.netbeans.org/main/rev/d27b2e964744

One problem is, that the last debugger session did not really finish.
There is still "JDI Target VM Interface" thread waiting at SharedMemoryConnection.receivePacket0(Native Method) and "JDI Internal Event Handler" waiting at EventQueueImpl.removeUnfiltered(EventQueueImpl.java:189).
Probably due to this fact "Debugger operator thread" can not be finished and is waiting at EventQueueImpl.removeUnfiltered(EventQueueImpl.java:189). On the stack it's holding JPDADebuggerImpl, whose state is still STATE_RUNNING.

That last debugger session holds all previous debugger sessions through lookup maps. This is something that needs to be fixed.
Comment 32 Martin Entlicher 2011-04-28 12:36:46 UTC
I've solved the memory leak in debugger by changeset:   193131:852caaee8478
http://hg.netbeans.org/main/rev/852caaee8478

If there are some strings being held by "JDI Internal Event Handler", it's a JDK problem and it's not solvable in NetBeans.

Passing to Subversion to solve the 80MB subversion cache.
Comment 33 Ondrej Vrabec 2011-04-28 13:30:21 UTC
uff, you seem to store thousands of files in a versioned context. And what is worse, they are not yet committed but "locally new", which means they have an interesting status for us/you. Such files are offered in the commit dialog when you try to commit your project.
You see, subversion keeps information about such interesting files in a cache and if the number of them grows to a certain limit, well, then the memory gets stuffed.
The files lie in E:\quaglan\My Documents\projects\subversion\sdi3\Geoportal\trunk\Services\MavenProjects\InspireWebServices\*. You can inspect those files if you run Subversion -> Status on the folder in Favorites. Do you keep them intentionally in the versioned context? I suggest you either:
1) commit them if they're needed and intended to share with others
2) svn ignore the folder
3) delete all those files

When you chose and performed one of my suggestions, please delete also ~\.netbeans\7.0\var\cache\svncache and restart the IDE.
If you're not happy or satisfied with the suggested solutions, please provide more information about the files, why you keep them, what are they used for, etc.
Comment 34 aquaglia 2011-04-28 15:12:46 UTC
I understand, your explanation is fine for me.

I generate the files during the unit testing but I delete them before committing.
So I would say this is OK.

I would like to know when and where I will be able to download a build containing the debugger fix as explained above so that I can give a feedback about it.

Many thanks indeed,
Angelo
Comment 35 Ondrej Vrabec 2011-04-28 15:58:50 UTC
Reassigning back to martin so he can close the bug and get a + point to the Resolved Bugs table
Comment 36 Martin Entlicher 2011-04-28 16:02:43 UTC
O.K. so hopefully fixed now.
After the bugfix propagates to a dev build, notification with the build number should be written into this issue.
Comment 37 Quality Engineering 2011-04-29 04:40:56 UTC
Integrated into 'main-golden', will be available in build *201104290000* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/852caaee8478
User: mentlicher@netbeans.org
Log: #197158 Debugger memory leak fixed.
Comment 38 aquaglia 2011-04-29 11:42:02 UTC
This was a GREAT FIX!

Now NetBeans is incredibly stable regardless of how many times I interrupt the debugging session.

You save me hours and hours of time.

Really many, many thanks Martin.
Comment 39 Jiri Kovalsky 2011-05-03 14:35:30 UTC
Adding 70patch keyword to get the fix into 7.0 Patch. Angelo, if you are satisfied with the fix, can you please change status of this bug to VERIFIED? Thanks!
Comment 40 aquaglia 2011-05-03 15:55:29 UTC
I am EXTREMELY satisfied with the fix.
Comment 41 Martin Entlicher 2011-05-16 15:07:31 UTC
Thank you, the fix is integrated in release70_fixes:
changeset:   190996:a54866fabbdb
changeset:   190998:0ad390b0bdc6
http://hg.netbeans.org/releases/rev/a54866fabbdb
http://hg.netbeans.org/releases/rev/0ad390b0bdc6
By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2014, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo