Bug 223776 - poor performance in background scanning of files
poor performance in background scanning of files
Status: VERIFIED FIXED
Product: javascript
Classification: Unclassified
Component: Editor
7.3
PC Windows XP
: P1 with 1 vote (vote)
: 7.3
Assigned To: Petr Pisl
issues@javascript
: 73_HR_FIX, PERFORMANCE
Depends on: 225021 225181 225186 225260 225674
Blocks:
  Show dependency treegraph
 
Reported: 2012-12-13 13:54 UTC by alied
Modified: 2013-03-15 20:22 UTC (History)
8 users (show)

See Also:
Issue Type: DEFECT
:


Attachments
snapshot of a extremelly slow sesion (2.07 MB, application/octet-stream)
2012-12-13 13:56 UTC, alied
Details
.npss file attached from NetBeans (2.55 MB, application/x-npss)
2013-01-16 23:02 UTC, alied
Details
.npss file attached from NetBeans (3.31 MB, application/x-npss)
2013-01-18 14:43 UTC, alied
Details

Note You need to log in before you can comment on or make changes to this bug.
Description alied 2012-12-13 13:54:18 UTC
[ JDK VERSION : 1.7.0_09 ]

since a couple biulds background scanning of files is quite slow and CPU/memory
eating. I'm using a webapp project with Vaadin. Since I work in the pure
backend services, I made the Vaadin widgets to be compiled and keep it in the
project.
Comment 1 alied 2012-12-13 13:56:56 UTC
Created attachment 129327 [details]
snapshot of a extremelly slow sesion

I cannot provide my project, but I believe it's because the 40M vaadin's js
Comment 2 Milos Kleint 2012-12-13 14:16:20 UTC
groovy and javascript show up in the repository updater thread. reassigning to groovy, please evaluate and reassign to javascript afterwards
Comment 3 alied 2012-12-13 14:37:28 UTC
I'll be searching my slowness reports. as soon as I get them I'll add them to this issue.
Comment 4 alied 2012-12-13 14:38:42 UTC
oh, yes, there are some groovy scripts as well. I don't use them my self, that's why I missed it
Comment 5 Martin Janicek 2012-12-13 14:47:05 UTC
Almost certainly two independent problems:

1] About 1/3 of the time is slow down by groovy parser - there are still some things/ideas that could be improved to make this faster, but they are not planned for 7.3 (see issue 180230 comment 62 for some details)

2] The rest of the slowness is caused by javascript editor indexing, but can't say if anything could be done to make this faster --> reassining for evaluation
Comment 6 GiorgosK 2012-12-21 11:25:56 UTC
PHP version
- clean installation of 7.3 beta2
- on starting for the first time imported user settings from 7.2 installation
- tried to include windows breakpoints and variables and rearranged them plus some others
- meanwhile the background scanning was getting out of hand and I clicked on the graph to release some memory
- but the memory was going up much faster already at 500 MB with just two drupal projects open and no other activity
- bug reporter opened up to send bug report but it also froze and I had to force exit from task manager
Comment 7 Petr Pisl 2013-01-03 15:19:22 UTC
Could you please try a newer build. I have made few improvements in memory consumption. The latest build can be find here http://bits.netbeans.org/download/trunk/nightly/latest/ .
Comment 8 alied 2013-01-16 23:02:18 UTC
Created attachment 130315 [details]
.npss file attached from NetBeans

.npss file
Comment 9 Petr Pisl 2013-01-17 11:31:16 UTC
I don't see in the npss file from Alied any javascript. There are java hints. Alied, what kind of application is scanned in your case? Thanks.

I'm ccing Tomas. Tomas I send you an ide snapshot with the javascript scanning, where 48% of the time was spend in the save method in cleaning a cache. If you have time, please look at this. Thanks.
Comment 10 Tomas Zezula 2013-01-17 11:41:19 UTC
I've already have prepared fix for integration.
After running tests I will integrate it to dev then transplant to NB 7.3
Comment 11 Tomas Zezula 2013-01-17 11:48:15 UTC
I've created an subissue on parsing.api, #225021.
Comment 12 Petr Pisl 2013-01-17 11:51:09 UTC
Thanks Tomas. I will test in my usecase.
Comment 13 alied 2013-01-17 13:34:11 UTC
(In reply to comment #9)
> I don't see in the npss file from Alied any javascript. There are java hints.
> Alied, what kind of application is scanned in your case? Thanks.
> 
> I'm ccing Tomas. Tomas I send you an ide snapshot with the javascript scanning,
> where 48% of the time was spend in the save method in cleaning a cache. If you
> have time, please look at this. Thanks.
Maybe the javascrip is no longer the issue (Hurray on that!), and I need to say that performance has improved a lot in the latest builds. My project is a Vaadin-based webapp with a lot of REST services to be used by external applications. After installed RC-1 it took a lot of time (creating tha cache, I pressume). After that the behaviopur is as the snapshot.
some update:
with both dev and RC-1 at start netbeans takes about 50%CPU, while (not sure if related) services.exe takes about 25%. It lasts a while (the time the snapshot takes) and then only from time to time gets sluggish.
some stats:
$ find src/main/java/ -name *.java | wc -l
530
$ find src/main/webapp/ -name *.js | wc -l
243
$ find src/main/webapp/VAADIN/ -name *.js | wc -l
243
$ du src/main/java/ -cshx
3.2M    src/main/java/
3.2M    total
$ du src/main/webapp/ -cshx
47M     src/main/webapp/
47M     total
$ du src/main/webapp/VAADIN/ -cshx
47M     src/main/webapp/VAADIN/
47M     total
Comment 14 alied 2013-01-17 13:52:52 UTC
one more thing in case it gives more hints; I'm using this start up parameters:
netbeans_default_options="-J-server -J-Xmx512m -J-XX:MaxPermSize=128m -J-ea -J-Dnetbeans.logger.console=true -J-XX:+UseG1GC -J-XX:+CMSClassUnloadingEnabled -J-XX:+CMSPermGenSweepingEnabled -J-Xverify:none -J-Dsun.java2d.noddraw=false -J-Dswing.aatext=true -J-Dawt.useSystemAAFontSettings=lcd -J-Dsun.zip.disableMemoryMapping=true -J-Xverify:none -J-XX:+TieredCompilation -J-Dnetbeans.extbrowser.manual_chrome_plugin_install=yes"
Comment 15 Tomas Zezula 2013-01-17 15:34:26 UTC
The -J-XX:+TieredCompilation is default on JDK 7.
The -J-XX:MaxPermSize=128m is less then default for 4GB RAM but with features on demand or non full IDE it may be enough.
The -J-XX:+UseG1GC may be problematic, as far as I remember the G1GC was causing some OOM in NB.
The -J-XX:+CMSClassUnloadingEnabled and the -J-XX:+CMSPermGenSweepingEnabled don't have effect for Garbage First GC they are Concurrent Mark and Sweep options.

What is the machine configuration?
Number of cores (threads), memory?
Thanks
Comment 16 alied 2013-01-17 16:11:38 UTC
(In reply to comment #15)
I'll take into account your hints; I've been dragging these parameters for a long time so it's a good time to update them.
> The -J-XX:+TieredCompilation is default on JDK 7.
  ok
> The -J-XX:MaxPermSize=128m is less then default for 4GB RAM but with features
> on demand or non full IDE it may be enough.
  I had high memory usage issues (not by netbeans) on my machine so I decreased it
> The -J-XX:+UseG1GC may be problematic, as far as I remember the G1GC was
> causing some OOM in NB.
  never have had one;
> The -J-XX:+CMSClassUnloadingEnabled and the -J-XX:+CMSPermGenSweepingEnabled
> don't have effect for Garbage First GC they are Concurrent Mark and Sweep
> options.
  so it should be time to remove them
> 
> What is the machine configuration?
> Number of cores (threads), memory?

Microsoft Windows XP Professional 32bit
Intel(R) Core(TM)2 Duo CPU E7500 @ 2.93GHz
4GB RAM (say 3.21GB)
> Thanks
Comment 17 Petr Pisl 2013-01-18 09:46:10 UTC
Thanks Tomas for your fix. I have measure my case and there is now big improvement. Before your fix in my case the org.netbeans.modules.javascript2.editor.index.JsIndexer.storeObject() took 160 seconds and after the fix it takes only 67 seconds.
Comment 18 alied 2013-01-18 14:43:52 UTC
Created attachment 130367 [details]
.npss file attached from NetBeans

.npss file
Comment 19 alied 2013-01-18 14:49:37 UTC
Hopefully latest snapshot. This one is with Groovy module installed and a clean cache dir. Covers the startup time until project opening and background scanning finishes.
Comment 20 Tomas Zezula 2013-01-18 17:12:34 UTC
I've spent some time looking at the first and the second snapshot, I don't get t the last one yet.

The second one (from comment #8) is just up to date check where everything is probably up to date as the scan took just 7 seconds (binaries 2 seconds, sources 5 seconds). The rest of time splits to 2 parts. The first part is waiting for the project to be opened 27 seconds. In the second part is the background task suspended by some priority task like editor, project or something ad does nothing just waits until the priority task(s) is done, it's approximately 40 seconds. So the second snapshot is OK, there is no problem in it. Just the 7 seconds the IDE spends in scanning, the rest is related to project loading which needs to be done anyway.


The first one is much more interesting and can be split also to 2 parts.
The first one 106 seconds is spent in indexEmbedding (indexing of javascript, html, css). Most of the time is spent in o.n.m.javascript2.editor.model.Model.getGlobalObject(). This is the part Petr is working on.

The second part is 76 seconds spent in the groovy parsing. I've took a look more closely on this part as I was fixing groovy parsing performance in NB 7.2, the issue #180230.
I did a measurement on the "gravel-0.4" project and found that parts of the fix of the issue #180230 was broken. The fix reduced indexing time of gravel from hour+ in NB 7.1 to 9 minutes in NB 7.2. But trying the same in the NB 7.3 the times grows to 30 minutes, so there is 200% regression compared to NB 7.2.
One problem, taking approximately 8 minutes, is caused by java script which registers allstubs.zip from indexer and causes the currently running task to be canceled.

My question to Petr Cyhelsky. Do we have some performance test(s) for groovy indexing similar to those we have for java?
Comment 21 Tomas Zezula 2013-01-18 18:14:52 UTC
The NB 7.3 groovy ClassNode hit ration is 44.2% while for NB 7.2 is 44.8%.
Also NB 7.3 much earlier goes out of cache slots.
Did you do some changes in groovy type resolution or did you integrate a new groovy parser in NB 7.3?
Comment 22 Martin Janicek 2013-01-21 12:35:43 UTC
(In reply to comment #21)
> Did you do some changes in groovy type resolution or did you integrate a new
> groovy parser in NB 7.3?

Only changes I'm aware of were related to annotation type resolution. Tickets:
issue 167284: web-main #ddd83054b1d4
issue 224956: web-main #6728c157fd89

..but these are most probably unrelated (as far as I can judge).

What might be related (but only in Grails projects) is fix for the issue 206199: web-main #64d25569108c
The changeset add some dependencies to the project classpath based on the BuildConfig.groovy file. It's just another way how to declare dependencies in Grails project, but it could caused some slowdown.
Comment 23 Tomas Zezula 2013-01-29 09:54:33 UTC
Performance comparison of Nb 7.2 and Nb 7.3 (including fixes of dependent issues) on gravl-0.4 project.

NB 7.3 (with fixes)
Work.index                                                     290,347
       JavaCustomIndexer.index                                166,640
               GroovyVFP.translateVirtualSources      155,460
               SuperOnePassCompileWorker.compile                9,810
       RepositoryUpdater.indexEmbedding                       122,509
               GroovyParser.parse                              75,648
               JsIndexer.index                                 16,277
               TLIndexer.index                                 19,624
                       HtmlErrorFilter                         12,332
                       CssParserResult.getDiagnostics           5,520
Work.scanFinished                                                4,191



NB 7.2
Work.index                                                     237,009
       JavaCustomIndexer.index                                131,802
               GroovyVFP.translateVirtualSources      123,949
               SuperOnePassCompileWorker.compile                7,227
       RepositoryUpdater.indexEmbedding                       104,350
               GroovyParser.parse                              76,437
               JsIndexer.index                                  2,817
               TLIndexer.index                                 16,605
                       HtmlErrorFilter                         12,100
                       CssParserResult.getDiagnostics           4,181
Work.scanFinished                                                1,255


Other measured projects:

Project scanned                 Scanning time in 7.2    Scanning Time in 7.3


Tomcat                                          17,344 s                17,715 s
BigWebProject(Frankioski)              9,548 s                   11,947 s
JEdit                                               14,208 s                 12,129 s
Mediawiki                                       55,236 s                  68940 s


There is still some regression in NB 7.3 in web, grails and PHP projects (gravl, BigWebProject, Mediawiki). There is some performance improvement in J2SE (JEdit).
Comment 24 alied 2013-01-30 14:57:29 UTC
Although performance has improved lately, there are some moments when the IDE gets very Slugish (even in my new Corei5). I have noticed that it happens specially when I create a new class, or some how I make changes that affect the public interface of a class (modify/add/remove public methods, etc.) As soon as I have a snapshot I'll upload it.
Comment 25 Tomas Zezula 2013-01-30 17:20:50 UTC
In fact when you change the public "API" of the class, the IDE needs to recheck all the files depending on the class which was changed to add or remove possible error badges (the red icons in project view). The default scope in which the check is done is all projects depending on the project where you did the change. The scope can be changed in case of Java in Tools/Options/Editor/Hints/Java/Dependency Scanning (disable | all projects | project only| source root).
Comment 26 Petr Pisl 2013-02-01 10:54:45 UTC
I have pushed a few improvements in JavaScript scanning area. 

http://hg.netbeans.org/web-main/rev/4947695e6da2
http://hg.netbeans.org/web-main/rev/325b5675b49b
http://hg.netbeans.org/web-main/rev/c703821c8801

I hope that I will be able to find more cases, where improve it.
Comment 27 Quality Engineering 2013-02-02 02:34:45 UTC
Integrated into 'main-golden', will be available in build *201302020001* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main-golden/rev/4947695e6da2
User: Petr Pisl <ppisl@netbeans.org>
Log: #223776 - poor performance in background scanning of files
	Small improvement in the js doc support
Comment 28 Petr Jiricka 2013-02-03 20:32:48 UTC
Thanks Petr, I transplanted your changes to release73 so they make it into tonight's build:
http://hg.netbeans.org/releases/rev/843b9547ce56
http://hg.netbeans.org/releases/rev/fc08565813b1
http://hg.netbeans.org/releases/rev/adaa93e1bec9
Comment 29 Marian Mirilovic 2013-02-03 20:40:09 UTC
Ok, so please close this bug as FIXED and report new one for pending cases, thanks in advance.
Comment 30 Petr Pisl 2013-02-03 23:38:37 UTC
I think we should do more. I'm closing this issue, but I would like to put into 7.3 at least one more fix that I'm working on.
Comment 31 Quality Engineering 2013-02-04 01:16:46 UTC
Integrated into 'releases', will be available in build *201302032200* or newer. Wait for official and publicly available build.
Changeset: http://hg.netbeans.org/releases/rev/843b9547ce56
User: Petr Pisl <ppisl@netbeans.org>
Log: #223776 - poor performance in background scanning of files
	Small improvement in the js doc support
Comment 32 Vladimir Riha 2013-02-18 09:06:39 UTC
v.


Product Version: NetBeans IDE 7.3 (Build 201302132200)
Java: 1.7.0_15; Java HotSpot(TM) Client VM 23.7-b01
Runtime: Java(TM) SE Runtime Environment 1.7.0_15-b02
System: Linux version 3.2.0-35-generic-pae running on i386; UTF-8; en_US (nb)
Comment 33 melanke 2013-03-15 20:22:39 UTC
I have the habit of save files every single comma I change. In Netbeans 7.3 the slowness of saving files is driving me crazy and it is not in background, the user interface freezes when I press Ctrl+S for about 3 seconds.


By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo