This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 179073 - slow indexing for big projects
Summary: slow indexing for big projects
Status: RESOLVED FIXED
Alias: None
Product: editor
Classification: Unclassified
Component: Parsing & Indexing (show other bugs)
Version: 6.x
Hardware: All All
: P2 normal with 1 vote (vote)
Assignee: Vitezslav Stejskal
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2009-12-28 06:09 UTC by Vladimir Voskresensky
Modified: 2010-05-11 16:49 UTC (History)
3 users (show)

See Also:
Issue Type: DEFECT
Exception Reporter:


Attachments
CND Project for mozilla (158.37 KB, application/x-bzip)
2009-12-28 06:23 UTC, Vladimir Voskresensky
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Vladimir Voskresensky 2009-12-28 06:09:33 UTC
We are working on C++ support for enterprise level applications(i.e. mozilla-size projects) and even with CND-only IDE "Scanning" phase finishes only after one hour.

TaskList is closed in the beginning. file system is local.

INFO [org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater]: Indexing of: file:/net/elif/export1/sside/av202691/mozilla-1.9.1/ took: 2423755 ms (New or modified files: 56081, Deleted files: 0)
INFO [org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater]: Indexing of: jar:file:/net/d-espb04-127-81/export/devarea/trunk/nbbuild/netbeans/ide12/jsstubs/allstubs.zip!/ took: 2539 ms (New or modified files: 189, Deleted files: 0)
INFO [org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater]: Indexing of: file:/net/elif/export1/sside/av202691/mozilla-1.9.1_1_vv/ took: 94 ms (New or modified files: 11, Deleted files: 0)INFO [org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater]: Complete indexing of 3 source roots took: 2426388 ms (New or modified files: 56281, Deleted files: 0)

At the same time full C++ parse takes:
 === Stopping parser queue stopwatch:    146044 ms
Comment 1 Vladimir Voskresensky 2009-12-28 06:11:12 UTC
There are a lot of exceptions in log with the same text:
java.lang.IllegalStateException: FAILED ASSERTION
        at org.mozilla.nb.javascript.Kit.codeBug(Kit.java:449)
        at org.mozilla.nb.javascript.Node.getExistingIntProp(Node.java:769)
        at org.mozilla.nb.javascript.Parser.getterSetterProperty(Parser.java:3625)
        at org.mozilla.nb.javascript.Parser.primaryExpr(Parser.java:3328)
        at org.mozilla.nb.javascript.Parser.memberExpr(Parser.java:2786)
        at org.mozilla.nb.javascript.Parser.unaryExpr(Parser.java:2477)
        at org.mozilla.nb.javascript.Parser.mulExpr(Parser.java:2380)
        at org.mozilla.nb.javascript.Parser.addExpr(Parser.java:2353)
        at org.mozilla.nb.javascript.Parser.shiftExpr(Parser.java:2324)
        at org.mozilla.nb.javascript.Parser.relExpr(Parser.java:2289)
        at org.mozilla.nb.javascript.Parser.eqExpr(Parser.java:2236)
        at org.mozilla.nb.javascript.Parser.bitAndExpr(Parser.java:2216)
        at org.mozilla.nb.javascript.Parser.bitXorExpr(Parser.java:2196)
        at org.mozilla.nb.javascript.Parser.bitOrExpr(Parser.java:2176)
        at org.mozilla.nb.javascript.Parser.andExpr(Parser.java:2156)
        at org.mozilla.nb.javascript.Parser.orExpr(Parser.java:2136)
        at org.mozilla.nb.javascript.Parser.condExpr(Parser.java:2111)
        at org.mozilla.nb.javascript.Parser.assignExpr(Parser.java:2072)
        at org.mozilla.nb.javascript.Parser.variables(Parser.java:1874)
        at org.mozilla.nb.javascript.Parser.statementHelper(Parser.java:1518)
        at org.mozilla.nb.javascript.Parser.statement(Parser.java:1001)
        at org.mozilla.nb.javascript.Parser.parse(Parser.java:566)
        at org.mozilla.nb.javascript.Parser.parse(Parser.java:494)
        at org.netbeans.modules.javascript.editing.JsParser.parseBuffer(JsParser.java:922)
        at org.netbeans.modules.javascript.editing.JsParser.sanitize(JsParser.java:817)
        at org.netbeans.modules.javascript.editing.JsParser.parseBuffer(JsParser.java:947)
        at org.netbeans.modules.javascript.editing.JsParser.parse(JsParser.java:104)
        at org.netbeans.modules.parsing.impl.SourceCache.getResult(SourceCache.java:198)
        at org.netbeans.modules.parsing.api.ResultIterator.getParserResult(ResultIterator.java:134)
        at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Work$2.run(RepositoryUpdater.java:1799)
        at org.netbeans.modules.parsing.api.ParserManager$UserTaskAction.run(ParserManager.java:129)
        at org.netbeans.modules.parsing.api.ParserManager$UserTaskAction.run(ParserManager.java:113)
        at org.netbeans.modules.parsing.impl.TaskProcessor.runUserTask(TaskProcessor.java:190)
        at org.netbeans.modules.parsing.api.ParserManager.parse(ParserManager.java:99)
        at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Work.indexEmbedding(RepositoryUpdater.java:1782)
        at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Work.index(RepositoryUpdater.java:1658)
        at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$AbstractRootsWork.scanSource(RepositoryUpdater.java:3085)
        at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$AbstractRootsWork.scanSources(RepositoryUpdater.java:2987)
        at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$RootsWork.getDone(RepositoryUpdater.java:2762)
        at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$InitialRootsWork.getDone(RepositoryUpdater.java:3158)
        at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Work.doTheWork(RepositoryUpdater.java:1899)
        at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Task._run(RepositoryUpdater.java:3457)
        at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Task.run(RepositoryUpdater.java:3399)
        at org.netbeans.modules.parsing.impl.TaskProcessor$CompilationJob.run(TaskProcessor.java:574)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269)
        at java.util.concurrent.FutureTask.run(FutureTask.java:123)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
        at java.lang.Thread.run(Thread.java:595)
Comment 2 Vladimir Voskresensky 2009-12-28 06:22:28 UTC
folders organization:
mozilla-1.9.1 - built sources 
mozilla-1.9.1_1_vv - NB project
Comment 3 Vladimir Voskresensky 2009-12-28 06:23:35 UTC
Created attachment 92957 [details]
CND Project for mozilla
Comment 4 Vladimir Voskresensky 2009-12-28 06:27:38 UTC
to reproduce:
- exclude all clusters except needed for CND:
platform11
nb6.8
ide12
cnd3
dlight2
- extract mozilla-1.9.1 sources and put on the same level as extracted mozilla-1.9.1_1_vv folder from attachement
Comment 5 Vladimir Voskresensky 2009-12-28 06:28:50 UTC
to get sources:
hg clone http://hg.mozilla.org/releases/mozilla-1.9.1
Comment 6 Vladimir Voskresensky 2009-12-28 06:33:09 UTC
Please, forgive mine ""Scanning" phase finishes only after one hour" :-) It was checked on NFS folder and took 66min. Local time is better - 40min, but it is too slow...
Comment 7 David Strupl 2010-01-05 08:16:24 UTC
Vita and/or Tomas please have a look.
Comment 8 Tomas Zezula 2010-01-05 11:07:22 UTC
The reported exceptions come from indexing of the allstubs.zip (javascript which is not associated with project), but it tooks just 2539 ms.
The most time is spent in the indexing of the mozilla-1.9.1/ 2423755 ms, but the parser took only 146044 ms. I will run it under profiler to find out where the time is spent.
Thanks Vladimir for the test case.
Comment 9 Vladimir Voskresensky 2010-01-05 11:14:51 UTC
Hi Tomas,
I think exceptions could be from indexing js files in mozilla-1.9.1 as well. Mozilla has huge number of java script files.
mozilla-1.9.1$ find . -name *.js | wc -l
4970
Comment 10 Tomas Zezula 2010-01-05 11:19:25 UTC
This may be the reason of this slowness.
The javascript parser is rather slow as it solves embeddings.
I will run it under profiler and let you know.
Comment 11 Vladimir Voskresensky 2010-01-05 13:43:12 UTC
Another possible reason could be html parser:
mozilla-1.9.1$  find . -name *.html | wc -l
10572
Comment 12 Vitezslav Stejskal 2010-01-06 06:19:58 UTC
I'll have a look. Tomas is now fighting hard battles on the javafx front.
Comment 13 Vladimir Voskresensky 2010-01-20 04:09:42 UTC
discussed the idea of a special marker "skipt javascript" in ClassPath
Comment 14 Vitezslav Stejskal 2010-05-11 16:49:13 UTC
I just tried the scenario described here and confirmed that no scanning is performed for C/C++ projects now.