This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
Summary: | [65cat] Sluggish performance on a clean/build | ||
---|---|---|---|
Product: | ide | Reporter: | Unknown <non_migrated_user> |
Component: | Performance | Assignee: | issues@performance <issues> |
Status: | CLOSED WORKSFORME | ||
Severity: | blocker | CC: | issues, jkovalsky, jlahoda, mmirilovic, pjiricka, tzezula |
Priority: | P2 | Keywords: | PERFORMANCE |
Version: | 3.x | ||
Hardware: | PC | ||
OS: | Linux | ||
Issue Type: | DEFECT | Exception Reporter: | |
Attachments: |
log file
Console and StackTrace of issue More sluggishness thread dump jmap histo j2se project module Modified version of org-netbeans-modules-java-source-ant.jar Sample Project for Testing Clean and Build bash script to generate test ant scripts. |
Description
Unknown
2008-08-07 22:04:38 UTC
Created attachment 66859 [details]
log file
What is the size of your project : - number of source roots - number of classes Thanks in advance. Please be more specific. What exactly did you do? Clean and build on which project? How many files? how many dependent projects? How long did the build run? Attached dump does not show, that UI is frozen. clean build. NB eventually did come back, but I couldn't edit, etc while the build was going on. The "hang" occurred in the last build. Here's a typical clean/build output. init: deps-clean: init: deps-clean: Deleting directory /home/sasbeb/projects/CMP-v920/build Deleting directory /home/sasbeb/projects/CMP-v920/dist clean: init: deps-clean: init: deps-clean: clean: init: deps-clean: init: deps-clean: clean: Deleting directory /home/sasbeb/projects/ETSCommon-v920/build Deleting directory /home/sasbeb/projects/ETSCommon-v920/dist clean: Deleting directory /home/sasbeb/projects/FCmpFuncETSModelEditor-v920/build Deleting directory /home/sasbeb/projects/FCmpFuncETSModelEditor-v920/dist clean: init: deps-clean: init: deps-clean: clean: clean: init: deps-clean: init: deps-clean: clean: init: deps-clean: init: deps-clean: clean: init: deps-clean: init: deps-clean: clean: clean: clean: init: deps-clean: init: deps-clean: clean: clean: Deleting directory /home/sasbeb/projects/sas.analytics.etsmodeleditor/build Deleting directory /home/sasbeb/projects/sas.analytics.etsmodeleditor/dist clean: Deleting directory /home/sasbeb/projects/RiskDimensions-d2rskd52/build Deleting directory /home/sasbeb/projects/RiskDimensions-d2rskd52/dist clean: init: deps-jar: init: deps-jar: Created dir: /home/sasbeb/projects/CMP-v920/build/classes Compiling 153 source files to /home/sasbeb/projects/CMP-v920/build/classes Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. Note: Some input files use unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. Copying 40 files to /home/sasbeb/projects/CMP-v920/build/classes Copied 9 empty directories to 5 empty directories under /home/sasbeb/projects/CMP-v920/build/classes compile: Created dir: /home/sasbeb/projects/CMP-v920/dist Building jar: /home/sasbeb/projects/CMP-v920/dist/sas.analytics.cmp.jar Copy libraries to /home/sasbeb/projects/CMP-v920/dist/lib. To run this application from the command line without Ant, try: java -jar "/home/sasbeb/projects/CMP-v920/dist/sas.analytics.cmp.jar" jar: init: deps-jar: init: deps-jar: compile: Copy libraries to /home/sasbeb/projects/CMP-v920/dist/lib. To run this application from the command line without Ant, try: java -jar "/home/sasbeb/projects/CMP-v920/dist/sas.analytics.cmp.jar" jar: init: deps-jar: init: deps-jar: compile: Copy libraries to /home/sasbeb/projects/CMP-v920/dist/lib. To run this application from the command line without Ant, try: java -jar "/home/sasbeb/projects/CMP-v920/dist/sas.analytics.cmp.jar" jar: Created dir: /home/sasbeb/projects/ETSCommon-v920/build/classes Compiling 73 source files to /home/sasbeb/projects/ETSCommon-v920/build/classes Note: Some input files use unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. Copying 8 files to /home/sasbeb/projects/ETSCommon-v920/build/classes compile: Created dir: /home/sasbeb/projects/ETSCommon-v920/dist Building jar: /home/sasbeb/projects/ETSCommon-v920/dist/sas.etscommon.jar Copy libraries to /home/sasbeb/projects/ETSCommon-v920/dist/lib. To run this application from the command line without Ant, try: java -jar "/home/sasbeb/projects/ETSCommon-v920/dist/sas.etscommon.jar" jar: Created dir: /home/sasbeb/projects/FCmpFuncETSModelEditor-v920/build/classes Compiling 126 source files to /home/sasbeb/projects/FCmpFuncETSModelEditor-v920/build/classes Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. Note: Some input files use unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. Copying 38 files to /home/sasbeb/projects/FCmpFuncETSModelEditor-v920/build/classes Copied 11 empty directories to 6 empty directories under /home/sasbeb/projects/FCmpFuncETSModelEditor-v920/build/classes compile: Created dir: /home/sasbeb/projects/FCmpFuncETSModelEditor-v920/dist Building jar: /home/sasbeb/projects/FCmpFuncETSModelEditor-v920/dist/sas.analytics.cmpfuncetsmdl.jar Copy libraries to /home/sasbeb/projects/FCmpFuncETSModelEditor-v920/dist/lib. To run this application from the command line without Ant, try: java -jar "/home/sasbeb/projects/FCmpFuncETSModelEditor-v920/dist/sas.analytics.cmpfuncetsmdl.jar" jar: init: deps-jar: init: deps-jar: compile: Copy libraries to /home/sasbeb/projects/CMP-v920/dist/lib. To run this application from the command line without Ant, try: java -jar "/home/sasbeb/projects/CMP-v920/dist/sas.analytics.cmp.jar" jar: compile: Copy libraries to /home/sasbeb/projects/ETSCommon-v920/dist/lib. To run this application from the command line without Ant, try: java -jar "/home/sasbeb/projects/ETSCommon-v920/dist/sas.etscommon.jar" jar: init: deps-jar: init: deps-jar: compile: Copy libraries to /home/sasbeb/projects/CMP-v920/dist/lib. To run this application from the command line without Ant, try: java -jar "/home/sasbeb/projects/CMP-v920/dist/sas.analytics.cmp.jar" jar: init: deps-jar: init: deps-jar: compile: Copy libraries to /home/sasbeb/projects/CMP-v920/dist/lib. To run this application from the command line without Ant, try: java -jar "/home/sasbeb/projects/CMP-v920/dist/sas.analytics.cmp.jar" jar: init: deps-jar: init: deps-jar: compile: Copy libraries to /home/sasbeb/projects/CMP-v920/dist/lib. To run this application from the command line without Ant, try: java -jar "/home/sasbeb/projects/CMP-v920/dist/sas.analytics.cmp.jar" jar: compile: Copy libraries to /home/sasbeb/projects/ETSCommon-v920/dist/lib. To run this application from the command line without Ant, try: java -jar "/home/sasbeb/projects/ETSCommon-v920/dist/sas.etscommon.jar" jar: compile: Copy libraries to /home/sasbeb/projects/FCmpFuncETSModelEditor-v920/dist/lib. To run this application from the command line without Ant, try: java -jar "/home/sasbeb/projects/FCmpFuncETSModelEditor-v920/dist/sas.analytics.cmpfuncetsmdl.jar" jar: init: deps-jar: init: deps-jar: compile: Copy libraries to /home/sasbeb/projects/CMP-v920/dist/lib. To run this application from the command line without Ant, try: java -jar "/home/sasbeb/projects/CMP-v920/dist/sas.analytics.cmp.jar" jar: compile: Copy libraries to /home/sasbeb/projects/ETSCommon-v920/dist/lib. To run this application from the command line without Ant, try: java -jar "/home/sasbeb/projects/ETSCommon-v920/dist/sas.etscommon.jar" jar: Created dir: /home/sasbeb/projects/sas.analytics.etsmodeleditor/build/classes Compiling 172 source files to /home/sasbeb/projects/sas.analytics.etsmodeleditor/build/classes Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. Note: Some input files use unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. Copying 30 files to /home/sasbeb/projects/sas.analytics.etsmodeleditor/build/classes compile: Created dir: /home/sasbeb/projects/sas.analytics.etsmodeleditor/dist Building jar: /home/sasbeb/projects/sas.analytics.etsmodeleditor/dist/sas.analytics.etsmodeleditor.jar jar: Created dir: /home/sasbeb/projects/RiskDimensions-d2rskd52/build/classes Compiling 905 source files to /home/sasbeb/projects/RiskDimensions-d2rskd52/build/classes Copying 136 files to /home/sasbeb/projects/RiskDimensions-d2rskd52/build/classes Copied 4 empty directories to 1 empty directory under /home/sasbeb/projects/RiskDimensions-d2rskd52/build/classes compile: Created dir: /home/sasbeb/projects/RiskDimensions-d2rskd52/dist Building jar: /home/sasbeb/projects/RiskDimensions-d2rskd52/dist/sas.analytics.risk.jar Copy libraries to /home/sasbeb/projects/RiskDimensions-d2rskd52/dist/lib. To run this application from the command line without Ant, try: java -jar "/home/sasbeb/projects/RiskDimensions-d2rskd52/dist/sas.analytics.risk.jar" jar: BUILD SUCCESSFUL (total time: 47 seconds) It does not seem to be a problem of javac compiler. Reassigning. More sluggishness...Attaching a thread dump/console Created attachment 67173 [details]
Console and StackTrace of issue
Created attachment 67577 [details]
More sluggishness thread dump
Can somebody from java team take a look at the newly attached dumps again? Thanks a lot! The most probable cause of the problems is a memory leak (judging from the thread dumps, the old gen is pretty full). Currently, there is no clue what and where would leak. I am afraid that the Java team does not have resources to evaluate the problem unless there is a reason to believe that the problem is in the Java support. To reporter: to evaluate the problem, a heap dump would be perfect (jmap -dump:live,format=b,file=<some-file> <id>; would be pretty big ~450MB). But heap histogram (jmap -histo:live <id>) could also provide some leads. Could you please try to provide at least one of these. Thanks. I'll set it up and run with it today. ..And I have a 300gig drive;) *** Issue 144444 has been marked as a duplicate of this issue. *** Note the histogram should be much smaller txt file, could be attached here and would also give us some data - number of instances, occupied space. We'll let you know if we need full heap dump. Thanks. (Full heap dump is much bigger, containing all references so it can be used to find out why the suspicious instances are held in memory.) jmap does not work for me. The format that Honza Lahoda posted is not recognized by my JDK, I have JDK 1.5 (on Mac OS X). Could this be 1.6-specific format? When I tried without any options, it does not work either. dhcp-eprg06-21-224:~ petrjiricka$ jmap 4932 Attaching to process ID 4932, please wait... Error attaching to process: Error attaching to process, or no such process Looks like a wiki page with exact instructions would be very useful. jmap -histo <pid> worked for me... It appears that the ":live" part is really 1.6 specific - sorry for that. Created attachment 68470 [details]
jmap histo
I think I know what is the problem (unless I misunderstood something): there is a cache in javac that keeps a map from jar file to its entries (ZipFileIndex[Entry]). This cache is realized by a static hash map, and it is never cleared by itself (it can be cleared externally though), and so is kept in memory as long as the ZipFileIndex class. Part of the problem is that the ZipFileIndex class is being load by the application classloader, and so it is never unloaded, meaning the cache is never cleared (speaking about the "internal compilation" - see below). Some random notes: -although the number of the ZFIE in the provided dump is ~420000 (which is quite high), this represents only ~25 instances of rt.jar (the one from a JDK1.6 contains ~17000 entries). Also, the reporter's heap size is set to ~400MB, the minimal heap our ergonomics will set is (IIRC) 96MB, and problems could then occur with much smaller projects/jars. -this is a semi-cumulative leak - the cache should not grow if you build one project again and again, but building different projects with different jars on classpath will grow the cache. -in JDK 1.6, this cache was not present in 1.6update5, but is in 1.6update7 -the cache of course causes problems (to the IDE) only if "internal compilation" is used - ant is executed in the same JVM as the IDE (alway), and ant executes the javac in the same JVM too (i.e. not spawn a new process) - depends on the build script. Given that the user might experience the problem even during the first build done in the IDE, the most appropriate workaround, I see so far, would be to fork (spawn a new process) the javac from the build script. Either do it automatically for all users or manually for the users that experience the problems. Even if the javac itself would crash with OOME, it wouldn't crash the IDE and the heap can be increased specifically for the javac. Spawning a new process of course makes the build slightly slower. Reporter, in a different issue we discussed an experiment with forking the compiler - did you have chance to test it yet? Did it help? Other workarounds are also possible, like clearing the cache (using reflection) after each ant build, but this won't really solve the non-cumulative aspects of the problem. Unless someone else wants to do it, I will contact the javac folks on this (as time permits). Thanks to the reporter for the histogram and heap dump. >> Reporter, in a different issue we discussed an experiment with forking the compiler
>> - did you have chance to test it yet? Did it help?
If memory serves, this is the forked compiler. I'll double check when I get into the office.
This was from the forked compiler Hm, strange. One thing I probably forgot to mention - the platform setting (and so whether the compiler is or is not forked) is a project setting, and so if there are more interdependent projects, the use of "internal" or "external" (forked) compiler during build depends on the sub-project that is being currently built, not on the "main" project. Did you change the platform only for the "main" project or for all? If you have too many projects to change them manually, we might be able to produce either patch for the IDE or a script that would change that automatically for all. Thanks. *** Issue 144298 has been marked as a duplicate of this issue. *** check the netcat list. I uploaded another jmap heap dump to megashares From the log, we have found that external compiler is not used for building all projects. With Honza we have prepared modified module, which should use external process for compiling as default, I'll attach it. Created attachment 69428 [details]
j2se project module
Bryan, could you, please: i) shutdown your ide, ii) decompress the gz attachement to {netbeans-dir}/java2/modules/ directory, iii) start your IDE again. The issue shouldn't be reproducible with the modified module. Thanks. Done.... Running with it now. Very sluggish.... arggg.... almost unworkable... Hm, strange. Bryan, could you, please, check that fork attribute is set in build-impl.xml. Look to project directory, which was cleaned/built: {project}/nbproject/build-impl.xml There should be a javac tag: <javac debug ... fork="yes" ...> Please, ensure, that the fork attribute is present. Thanks. I am going to attach a modified version of an ant extension library that tries to clean the javac caches after each compilation. Could you please try to replace the original version of org-netbeans-modules-java-source-ant.jar in ${netbeans_install}/java2/ant/nblib with the attached version (please backup the original version) and try to work with it? After each invocation of javac in the ant build script, it should print something like: Clearing javac caches Original caches: <number> Do clearing javac caches After clearing caches: 0 Normal verbosity should be enough to see the messages. Thanks. Created attachment 69684 [details]
Modified version of org-netbeans-modules-java-source-ant.jar
Bryan, did you have chance to test the patch? Did it improve the situation? Thanks. No, it didn't improve performance. Friday when I left for the weekend, the IDE was so sluggish it took about 3-5 minutes to compilem when it normally (under 6.1) takes less than a minute to compile the same sources. Can it be that clean/build all (I have to do it a lot due to http://www.netbeans.org/issues/show_bug.cgi?id=144855) I'm not sure I'm helping a ton jumping into this issue this late, but I've seen some reproducable stuff recently using NetBeans IDE Build 200809151401. I will attach a ZIP file of a sample Java Web App I've used to demo this issue. It contains a couple dozen Java source files that I've fluffed up to 3000-4000 lines per class to simulate "real" code. When I do a fresh open of the IDE, I open the project and run a Clean and Build. The memory spikes a little, but not too bad. The Clean&Build finishes. However, if I open all the java source files in the Source Editor, the memory/cpu does not move much. If I NOW do a Clean & Build, the memory utilization seems to jump significantly more than the first Clean & Build. If I leave the Java source files open without doing anything and wait a few seconds and do another Clean and Build, the memory goes up and then stabilizes. After a half dozen or dozen cycles of this the memory usage stays quite high and doesn't go back down. Not sure if this is a similar situation to this thread, but hopefully the sample project can help. Created attachment 69998 [details]
Sample Project for Testing Clean and Build
Sorry, forgot to add basic details on my setup : Product Version: NetBeans IDE Dev (Build 200809151401) Java: 1.6.0_02; Java HotSpot(TM) Client VM 1.6.0_02-b05 System: Windows XP version 5.1 running on x86; Cp1252; en_US (nb) Thanks for you information. Unfortunately, I'm not able to reproduce it on my machine. I've tried both Linux and WinXP platform. I will ask someone else to try to reproduce. Dmitry Lipin has a VM of sidux. (Debian sid) I'm running 64-bit. He might be running 32-bit, but he might be able to help. Bryan, could you please: -check that the ant log contains the debugging info I wrote about above: Clearing javac caches Original caches: <number> Do clearing javac caches After clearing caches: 0 What are the numbers in your case? (The <number> represents how many .jar files are cached, not entries.) -is the IDE as a whole slow, or only the build process? -could you please try to run the IDE on an older JDK (1.5 or 1.6 update 5 or older)? -could you please double-check that you are using the same JDK to run the 6.1 and 6.5 (in the IDE log or under Help/About) Thanks. I have performed a little experiment over the weekend: I have a build script, that executes other build scripts that are trying to compile a trivial class, with some jars on the classpath. Executing the script on JDK1.5 and JDK1.6 update 5 passed without problems. The same script failed with OOME on JDK1.6 update 10. I have 'export ANT_OPTS="-Xmx256m"' in my ~/.antrc. I have executed the build script in the IDE with JDK1.5 and with JDK1.6 update 10, with and without the "clearing caches" patch. The only case where I encountered problems (extraordinary memory consumption, slow response, OOME, etc.) was JDK1.6 update 10 without the "clearing caches" patch. In case someone wanted to reproduce the experiment, I am attaching a bash script that creates the build scripts+jars that I used. Notes: -the script may misbehave (I am not an expert on bash scripts), so please verify the build script yourself to make sure it won't hurt your computer :-) -place the script into an empty directory and execute it from there - it will create directory "build" which will contain the test build script (please note that the script deletes the "build" directory as its first step) -at the beginning there are four variables you can try to modify and see what happens. Make sure that: dirs * files < 65536 -to perform the experiment, execute "ant" inside the "build" directory (or, inside NB, add "build" directory to the Favorites view and execute "depend" target on the "build.xml"). Created attachment 70203 [details]
bash script to generate test ant scripts.
I have found a simple way to disable the static cache in the javac: http://hg.netbeans.org/main?cmd=changeset;node=8c3e1d7dd54a Bryan (and others), could you please try a build with that patch? Continuous trunk builds contain that fix since build #3842: http://deadlock.netbeans.org/hudson/job/trunk/ A link to a daily build containing that fix will be added to this issue when available. Thanks. Integrated into 'main-golden', will be available in build *200809231435* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main/rev/8c3e1d7dd54a User: Jan Lahoda <jlahoda@netbeans.org> Log: #143234: disabling javac zip file caches. Any news? -- Does last Honza's integration help? For the time being, we do not have any other idea what to do to improve behavior as we are unable to reproduce the problem. There hasn't been any response last week, considering issue as fixed. Thanks. HMMM... If you guys didn't make a code change to address the problem, then it can't be fixed. I was out of the office last week. I'm downloading it now, but in the meantime, I'm re-opening it. Honza applied the patch which clears the caches. -- Such a change removed memory leak occupied by javac -- this leak was originally identified as a cause of your problem. If it doesn't help, without any new thread dump and heap dumps (or reproducible test-case) we are not able to provide any other patch. That is all I can say for the time being, if there is any volunteer with better understanding of this problem, he can fix it correctly. Great... I should know something by the end of the day. I'm running the new stuff now. Seems to work as adverrtised. Thanks for your hard work. I'll file another defect if the problem arises again. |