This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
[ JDK VERSION : 1.7.0_09 ] since a couple biulds background scanning of files is quite slow and CPU/memory eating. I'm using a webapp project with Vaadin. Since I work in the pure backend services, I made the Vaadin widgets to be compiled and keep it in the project.
Created attachment 129327 [details] snapshot of a extremelly slow sesion I cannot provide my project, but I believe it's because the 40M vaadin's js
groovy and javascript show up in the repository updater thread. reassigning to groovy, please evaluate and reassign to javascript afterwards
I'll be searching my slowness reports. as soon as I get them I'll add them to this issue.
oh, yes, there are some groovy scripts as well. I don't use them my self, that's why I missed it
Almost certainly two independent problems: 1] About 1/3 of the time is slow down by groovy parser - there are still some things/ideas that could be improved to make this faster, but they are not planned for 7.3 (see issue 180230 comment 62 for some details) 2] The rest of the slowness is caused by javascript editor indexing, but can't say if anything could be done to make this faster --> reassining for evaluation
PHP version - clean installation of 7.3 beta2 - on starting for the first time imported user settings from 7.2 installation - tried to include windows breakpoints and variables and rearranged them plus some others - meanwhile the background scanning was getting out of hand and I clicked on the graph to release some memory - but the memory was going up much faster already at 500 MB with just two drupal projects open and no other activity - bug reporter opened up to send bug report but it also froze and I had to force exit from task manager
Could you please try a newer build. I have made few improvements in memory consumption. The latest build can be find here http://bits.netbeans.org/download/trunk/nightly/latest/ .
Created attachment 130315 [details] .npss file attached from NetBeans .npss file
I don't see in the npss file from Alied any javascript. There are java hints. Alied, what kind of application is scanned in your case? Thanks. I'm ccing Tomas. Tomas I send you an ide snapshot with the javascript scanning, where 48% of the time was spend in the save method in cleaning a cache. If you have time, please look at this. Thanks.
I've already have prepared fix for integration. After running tests I will integrate it to dev then transplant to NB 7.3
I've created an subissue on parsing.api, #225021.
Thanks Tomas. I will test in my usecase.
(In reply to comment #9) > I don't see in the npss file from Alied any javascript. There are java hints. > Alied, what kind of application is scanned in your case? Thanks. > > I'm ccing Tomas. Tomas I send you an ide snapshot with the javascript scanning, > where 48% of the time was spend in the save method in cleaning a cache. If you > have time, please look at this. Thanks. Maybe the javascrip is no longer the issue (Hurray on that!), and I need to say that performance has improved a lot in the latest builds. My project is a Vaadin-based webapp with a lot of REST services to be used by external applications. After installed RC-1 it took a lot of time (creating tha cache, I pressume). After that the behaviopur is as the snapshot. some update: with both dev and RC-1 at start netbeans takes about 50%CPU, while (not sure if related) services.exe takes about 25%. It lasts a while (the time the snapshot takes) and then only from time to time gets sluggish. some stats: $ find src/main/java/ -name *.java | wc -l 530 $ find src/main/webapp/ -name *.js | wc -l 243 $ find src/main/webapp/VAADIN/ -name *.js | wc -l 243 $ du src/main/java/ -cshx 3.2M src/main/java/ 3.2M total $ du src/main/webapp/ -cshx 47M src/main/webapp/ 47M total $ du src/main/webapp/VAADIN/ -cshx 47M src/main/webapp/VAADIN/ 47M total
one more thing in case it gives more hints; I'm using this start up parameters: netbeans_default_options="-J-server -J-Xmx512m -J-XX:MaxPermSize=128m -J-ea -J-Dnetbeans.logger.console=true -J-XX:+UseG1GC -J-XX:+CMSClassUnloadingEnabled -J-XX:+CMSPermGenSweepingEnabled -J-Xverify:none -J-Dsun.java2d.noddraw=false -J-Dswing.aatext=true -J-Dawt.useSystemAAFontSettings=lcd -J-Dsun.zip.disableMemoryMapping=true -J-Xverify:none -J-XX:+TieredCompilation -J-Dnetbeans.extbrowser.manual_chrome_plugin_install=yes"
The -J-XX:+TieredCompilation is default on JDK 7. The -J-XX:MaxPermSize=128m is less then default for 4GB RAM but with features on demand or non full IDE it may be enough. The -J-XX:+UseG1GC may be problematic, as far as I remember the G1GC was causing some OOM in NB. The -J-XX:+CMSClassUnloadingEnabled and the -J-XX:+CMSPermGenSweepingEnabled don't have effect for Garbage First GC they are Concurrent Mark and Sweep options. What is the machine configuration? Number of cores (threads), memory? Thanks
(In reply to comment #15) I'll take into account your hints; I've been dragging these parameters for a long time so it's a good time to update them. > The -J-XX:+TieredCompilation is default on JDK 7. ok > The -J-XX:MaxPermSize=128m is less then default for 4GB RAM but with features > on demand or non full IDE it may be enough. I had high memory usage issues (not by netbeans) on my machine so I decreased it > The -J-XX:+UseG1GC may be problematic, as far as I remember the G1GC was > causing some OOM in NB. never have had one; > The -J-XX:+CMSClassUnloadingEnabled and the -J-XX:+CMSPermGenSweepingEnabled > don't have effect for Garbage First GC they are Concurrent Mark and Sweep > options. so it should be time to remove them > > What is the machine configuration? > Number of cores (threads), memory? Microsoft Windows XP Professional 32bit Intel(R) Core(TM)2 Duo CPU E7500 @ 2.93GHz 4GB RAM (say 3.21GB) > Thanks
Thanks Tomas for your fix. I have measure my case and there is now big improvement. Before your fix in my case the org.netbeans.modules.javascript2.editor.index.JsIndexer.storeObject() took 160 seconds and after the fix it takes only 67 seconds.
Created attachment 130367 [details] .npss file attached from NetBeans .npss file
Hopefully latest snapshot. This one is with Groovy module installed and a clean cache dir. Covers the startup time until project opening and background scanning finishes.
I've spent some time looking at the first and the second snapshot, I don't get t the last one yet. The second one (from comment #8) is just up to date check where everything is probably up to date as the scan took just 7 seconds (binaries 2 seconds, sources 5 seconds). The rest of time splits to 2 parts. The first part is waiting for the project to be opened 27 seconds. In the second part is the background task suspended by some priority task like editor, project or something ad does nothing just waits until the priority task(s) is done, it's approximately 40 seconds. So the second snapshot is OK, there is no problem in it. Just the 7 seconds the IDE spends in scanning, the rest is related to project loading which needs to be done anyway. The first one is much more interesting and can be split also to 2 parts. The first one 106 seconds is spent in indexEmbedding (indexing of javascript, html, css). Most of the time is spent in o.n.m.javascript2.editor.model.Model.getGlobalObject(). This is the part Petr is working on. The second part is 76 seconds spent in the groovy parsing. I've took a look more closely on this part as I was fixing groovy parsing performance in NB 7.2, the issue #180230. I did a measurement on the "gravel-0.4" project and found that parts of the fix of the issue #180230 was broken. The fix reduced indexing time of gravel from hour+ in NB 7.1 to 9 minutes in NB 7.2. But trying the same in the NB 7.3 the times grows to 30 minutes, so there is 200% regression compared to NB 7.2. One problem, taking approximately 8 minutes, is caused by java script which registers allstubs.zip from indexer and causes the currently running task to be canceled. My question to Petr Cyhelsky. Do we have some performance test(s) for groovy indexing similar to those we have for java?
The NB 7.3 groovy ClassNode hit ration is 44.2% while for NB 7.2 is 44.8%. Also NB 7.3 much earlier goes out of cache slots. Did you do some changes in groovy type resolution or did you integrate a new groovy parser in NB 7.3?
(In reply to comment #21) > Did you do some changes in groovy type resolution or did you integrate a new > groovy parser in NB 7.3? Only changes I'm aware of were related to annotation type resolution. Tickets: issue 167284: web-main #ddd83054b1d4 issue 224956: web-main #6728c157fd89 ..but these are most probably unrelated (as far as I can judge). What might be related (but only in Grails projects) is fix for the issue 206199: web-main #64d25569108c The changeset add some dependencies to the project classpath based on the BuildConfig.groovy file. It's just another way how to declare dependencies in Grails project, but it could caused some slowdown.
Performance comparison of Nb 7.2 and Nb 7.3 (including fixes of dependent issues) on gravl-0.4 project. NB 7.3 (with fixes) Work.index 290,347 JavaCustomIndexer.index 166,640 GroovyVFP.translateVirtualSources 155,460 SuperOnePassCompileWorker.compile 9,810 RepositoryUpdater.indexEmbedding 122,509 GroovyParser.parse 75,648 JsIndexer.index 16,277 TLIndexer.index 19,624 HtmlErrorFilter 12,332 CssParserResult.getDiagnostics 5,520 Work.scanFinished 4,191 NB 7.2 Work.index 237,009 JavaCustomIndexer.index 131,802 GroovyVFP.translateVirtualSources 123,949 SuperOnePassCompileWorker.compile 7,227 RepositoryUpdater.indexEmbedding 104,350 GroovyParser.parse 76,437 JsIndexer.index 2,817 TLIndexer.index 16,605 HtmlErrorFilter 12,100 CssParserResult.getDiagnostics 4,181 Work.scanFinished 1,255 Other measured projects: Project scanned Scanning time in 7.2 Scanning Time in 7.3 Tomcat 17,344 s 17,715 s BigWebProject(Frankioski) 9,548 s 11,947 s JEdit 14,208 s 12,129 s Mediawiki 55,236 s 68940 s There is still some regression in NB 7.3 in web, grails and PHP projects (gravl, BigWebProject, Mediawiki). There is some performance improvement in J2SE (JEdit).
Although performance has improved lately, there are some moments when the IDE gets very Slugish (even in my new Corei5). I have noticed that it happens specially when I create a new class, or some how I make changes that affect the public interface of a class (modify/add/remove public methods, etc.) As soon as I have a snapshot I'll upload it.
In fact when you change the public "API" of the class, the IDE needs to recheck all the files depending on the class which was changed to add or remove possible error badges (the red icons in project view). The default scope in which the check is done is all projects depending on the project where you did the change. The scope can be changed in case of Java in Tools/Options/Editor/Hints/Java/Dependency Scanning (disable | all projects | project only| source root).
I have pushed a few improvements in JavaScript scanning area. http://hg.netbeans.org/web-main/rev/4947695e6da2 http://hg.netbeans.org/web-main/rev/325b5675b49b http://hg.netbeans.org/web-main/rev/c703821c8801 I hope that I will be able to find more cases, where improve it.
Integrated into 'main-golden', will be available in build *201302020001* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main-golden/rev/4947695e6da2 User: Petr Pisl <ppisl@netbeans.org> Log: #223776 - poor performance in background scanning of files Small improvement in the js doc support
Thanks Petr, I transplanted your changes to release73 so they make it into tonight's build: http://hg.netbeans.org/releases/rev/843b9547ce56 http://hg.netbeans.org/releases/rev/fc08565813b1 http://hg.netbeans.org/releases/rev/adaa93e1bec9
Ok, so please close this bug as FIXED and report new one for pending cases, thanks in advance.
I think we should do more. I'm closing this issue, but I would like to put into 7.3 at least one more fix that I'm working on.
Integrated into 'releases', will be available in build *201302032200* or newer. Wait for official and publicly available build. Changeset: http://hg.netbeans.org/releases/rev/843b9547ce56 User: Petr Pisl <ppisl@netbeans.org> Log: #223776 - poor performance in background scanning of files Small improvement in the js doc support
v. Product Version: NetBeans IDE 7.3 (Build 201302132200) Java: 1.7.0_15; Java HotSpot(TM) Client VM 23.7-b01 Runtime: Java(TM) SE Runtime Environment 1.7.0_15-b02 System: Linux version 3.2.0-35-generic-pae running on i386; UTF-8; en_US (nb)
I have the habit of save files every single comma I change. In Netbeans 7.3 the slowness of saving files is driving me crazy and it is not in background, the user interface freezes when I press Ctrl+S for about 3 seconds.