This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
Created attachment 133703 [details] Log file and thread dump in zip file Product Version: NetBeans IDE 7.3 (Build 201302132200) Java: 1.7.0_15; Java HotSpot(TM) Client VM 23.7-b01 Runtime: Java(TM) SE Runtime Environment 1.7.0_15-b03 System: Windows XP version 5.1 running on x86; Cp1252; en_NZ (nb) Happens simply after copying a wedium-larg size web project using project node|right click|Copy The IDE takes some time to copy the project while it scans it. I close the source project after completion. Souce files are broken. The IDE tries to fix some. During that time, the IDE becomes fairly unresponsive. Then the operation ends with: error in log: Not enough memory to compile folder
Heap dump: http://deadlock.netbeans.org/hudson/job/upload/224/
Reassign to performance team for evaluation ....
It is no surprise that there were warnings about not enough memory to compile project, because you have more than 35 opened projects and more than 50 opened editor tabs and various other memory consuming topcomponents like output, usages, search, etc open. Even without those the warning by itself is harmless, it is just information for you that if you want the scanning to be more efficient you should increase memory (xmx (which you shouldn't do because you are using 32-bit JVM)). From the messages.log it seems there is something wrong with the installation - there are numerous exceptions like java.lang.ClassNotFoundException: org.netbeans.modules.php.editor.indent.ui.FmtUses, etc But nonetheless there is something seriously wrong and that is that there are 22 "org.glassfish.tools.ide.server.FetchLogPiped...... "zombie" (abandoned) threads which are somehow hanging on in the memory with all the baggage... reassigning to glassfish to somehow deal with them.
Please see bug 225157 for GlassFish threads
I was already fixing zombie glassfish threads issue related to FetchLogPiped. But please open another bug for zombie glassfish threads, it's not a good idea to mix everything together. I will check if it's already fixed in 7.3.1. Passing this bug back.
I have had these issues for some time. While the number of projects and files in editor are a contributions, they have never been most significant. I could reproduce this with no files open and 5 open projects. Whenever I had these problems, the number of threads was excessive (up to 140) and GlassFish thread count was always high. Debugger threads were culprits, now partially fixed, so next big issue seems to be GlassFish, and still thread count can be reduced further, please see bug 225418
(In reply to comment #5) > I was already fixing zombie glassfish threads issue related to FetchLogPiped. > But please open another bug for zombie glassfish threads, it's not a good idea > to mix everything together. I will check if it's already fixed in 7.3.1. > > Passing this bug back. We are not mixing everything together - these threads have to be fixed for us to see what else is wrong- either assign this issue as a duplicate of the one where the zombie glassfish threads are fixed or do it in this issue if there is none yet created. All the data is present in the heap dump. Another problem with the 22 glassfish threads is that for each of them there is another "destroyer" thread ... so it is really 44 threads just being there consuming memory. @bht: The important thing is that the "Not enough memory to compile folder" message is not very important - it is just a message that informs you that there is not enough memory to use the more effective compiler.
Well, 22 "org.glassfish.tools.ide.server.FetchLogPiped" zombie threads means there was some huge activity around server logs. Usually just 1 thread is used for each GlassFish server to read logs. First of all I need to find some way how to reproduce this. Threads are started with starting GlassFish and then with opening log window which was previously closed. Old threads should die ...but looks like they are becoming undeads.
I played a bit with latest 7.3.1 build but I can't get org.glassfish.tools.ide.server.FetchLogPiped threads in ZOMBIE state. I see only active threads for individual running servers. bht: Please can you try to reproduce this org.glassfish.tools.ide.server.FetchLogPiped zombie threads with 3.1.1 build and givem me some reproduction scenario.
Oops, 3.1.1 -> 7.3.1. It's available on http://bertram2.netbeans.org:8080/job/javaee7/
I have downloaded as advised http://bertram2.netbeans.org:8080/job/javaee7/lastStableBuild/artifact/nbbuild/NetBeans-dev-javaee7-297-on-20130509-full.zip I will use the IDE with VisualVM open and connected to it. Any other ideas are welcome. I can't promise that I will detect the condition soon enough without any alert so that I can start thinking about a test case. Can the NetBeans team provide a little JMX program that runs alongside or inside the IDE and alerts the user when a particular condition is met e.g. the number of org.glassfish.tools.ide.server.FetchLogPiped zombie threads exceeds a limit?
How to reproduce: - Start NetBeans - Start VisualVM and open NetBeans, threads table, sort by Thread - Create sample project ServletStateless - Start GlassFish server in debug mode - Undeploy all projects Repeat the following cycle - Run project ServletStateless creates one thread FetchLogPiped - Clean and build project creates one thread FetchLogPiped. So if you run this cyle 3 times you should have 6 FetchLogPiped threads.
*** Bug 225157 has been marked as a duplicate of this bug. ***
OK, thanks a lot. I was not testing project build/run last time.
Problem was in build task which is killing all child threads at the end. org.glassfish.tools.ide.server.FetchLogPiped was killed at the end of each build and it remained in memory as zombie. Checked into GlassFish Tooling SDK: ----------------------------------- changeset: 502:280234be9cb0 branch: 0.3-b036 tag: tip summary: Added state change events and optional create method to allow passing own ExecutorService hangeset: 512:8f2d63db8de8 summary: Added state change events and optional create method to allow passing own ExecutorService Those changes in library allows me to better handle FetchLogPiped task termination (NetBeans LogViewMgr will be notified in listener) and also to provide ExecutorService which is running threads with both setDaemon(true) and ThreadGroup outside groups being killed by build task thread. I have working solution in former javaee7 branch for 7.3.1. But before porting this into 7.3.1 I have to fix it in trunk.
Checked into web-main trunk: ---------------------------- changeset: 253015:70350e3b09dc summary: #228886 - Glassfish threads are no more being killed in build thread Ported into trunk.
Checked into web-main trunk: ---------------------------- changeset: 253069:422df5694675 summary: #228886 - Fixed comments
Integrated into 'main-golden', will be available in build *201305152300* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main-golden/rev/70350e3b09dc User: Tomas Kraus <TomasKraus@netbeans.org> Log: #228886 - Glassfish threads are no more being killed in build thread
Verified in NetBeans IDE Dev (Build web-main-10534-on-20130516). With steps in comment 12 I see only one thread named "org.netbeans.modules.glassfish.common.LogViewMgr$LoggerRunnable - java.io.BufferedInputStream@76ddbc89".
org.netbeans.modules.glassfish.common.LogViewMgr$LoggerRunnable is old code that I did not touch. Zombie thread was org.glassfish.tools.ide.server.FetchLogPiped... and is renamed to "GlassFish Log Reader" now. Trunk contains better code than 7.3/7.3.1 here so you probably won't see this thread when working with local server. You may see one for each running GF server when testing 7.3.1. But I need to get approval for high resistance push first. :)
Checked into web-main: ---------------------- changeset: 253094:1ad1e811d955 summary: #228886 - Fixed wrong condition for resizing thread pool Petr Hejl found mistake in pool resizing condition. So here is a fix, This can't be verified by QA, because such a problem won't be seen until some very special conditions are met (maximum size attribute overflow). I verified this very carefully with attached debugger to see that code is stable now.
(In reply to comment #20) > Zombie thread was org.glassfish.tools.ide.server.FetchLogPiped... and is > renamed to "GlassFish Log Reader" now. > Trunk contains better code than 7.3/7.3.1 here so you probably won't see this > thread when working with local server. You may see one for each running GF > server when testing 7.3.1. Once more I tried 7.3.1 with local GlassFish and there is always only one FetchLogPiped thread. For trunk build with your patches (web-main-10534-on-20130516) there was no thread called "GlassFish Log Reader" nor FetchLogPiped. So what are steps to see the problem?
changeset: 253176:e0be6fab29ed summary: #228886 - Simplified thread pool code with Integer.MAX_VALUE as max size An advice from Petr Hejl to simplify pool code and remove resizing. Added into trunk.
FetchLogPiped thread in current 7.3.1 IS "GlassFish Log Reader" in trunk. Trunk contains some new enhancements so this thread is not needed all the time as in 7.3.1 - it works with GF process stdout/stderr until you close log window and reopen it with services->Servers->GF Instance->View domain Server Log. Reproduction scenario is in comennt#12 - you have to clean/build and run sample/Servlet Stateless project to get zombie threads.
Integrated into 'main-golden', will be available in build *201305170640* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main-golden/rev/1ad1e811d955 User: Tomas Kraus <TomasKraus@netbeans.org> Log: #228886 - Fixed wrong condition for resizing thread pool
Created attachment 134564 [details] Diff to be applied on release73 branch. All 4 changesets (253015:70350e3b09dc, 253069:422df5694675, 253094:1ad1e811d955 and 253176:e0be6fab29ed) applied on release73 branch. GF Tooling SDK changes applied on 0.3-b036 branch which is used in 73/7.3.1. Module-Specification-Version incremented for glassfish.common and libs.glassfish.sdk. This is what I sent to Jiri for testing.
Integrated into 'main-golden', will be available in build *201305172300* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main-golden/rev/e0be6fab29ed User: Tomas Kraus <TomasKraus@netbeans.org> Log: #228886 - Simplified thread pool code with Integer.MAX_VALUE as max size
Tomas, I do not see changesets in releases/release73 ... could you please transplant all related changes (including changes in libs) into releases/release73 ASAP ? Thanks in advance.
We had some internal discussion around this fix and Petr Hejl had some objections against this solution. I asked Peter Benedikovic to check it again and if he's ok with this solution I'll push it into 7.3.1. For 7.4 I have to make it a bit better.
Reviewed. Looks OK.
Finally 7.3.1 will contain simple fix: ====================================== GlassFish Tooling SDK: ---------------------- changeset: 529:57c1cb118fba branch: 0.3-b036 summary: Bug# 228886 - Fixed local executor handling changeset: 529:d6d9e4d945b8 summary: Bug# 228886 - Fixed local executor handling This was pushed into trunk too because it made code more safe. NetBeans releases: ------------------ changeset: 252336:a53720ec2414 branch: release73 summary: #228886 - Multiple Glassfish threads hanging in memory - simple fix for 7.3.1 Those are 7.3.1 only changes - pointer on proper GF Tooling Library and one library API change.
Pushed into release73 branch ---------------------------- changeset: 261751:830a31ecdb96 branch: release73 summary: #228886 - Multiple Glassfish threads hanging in memory - simple fix for 7.3.1
Integrated into 'releases', will be available in build *201305232200* or newer. Wait for official and publicly available build. Changeset: http://hg.netbeans.org/releases/rev/830a31ecdb96 User: Tomas Kraus <TomasKraus@netbeans.org> Log: #228886 - Multiple Glassfish threads hanging in memory - simple fix for 7.3.1