This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
Summary: | Big multimodule MAVEN projects – slow to open, weirdly slow background scan | ||
---|---|---|---|
Product: | projects | Reporter: | NukemBy |
Component: | Maven | Assignee: | Tomas Stupka <tstupka> |
Status: | RESOLVED WORKSFORME | ||
Severity: | normal | CC: | markiewb |
Priority: | P2 | ||
Version: | 8.1 | ||
Hardware: | PC | ||
OS: | Windows 7 | ||
Issue Type: | DEFECT | Exception Reporter: | |
Attachments: |
pack of selfsampler
test project 20 seconds of self sampler while bg scan was running for 10 minutes IDE log for the same Hot points from the npss attached before Hot points from the npss attached before - dig deeper Capture from process monitor - see: https://technet.microsoft.com/en-us/library/bb896645.aspx selfsampler with "Invalidating facelets libraries due to a library descriptor change" selfsampler with "Invalidating facelets libraries due to a library descriptor change" (screenshot) |
Description
NukemBy
2015-03-02 19:09:03 UTC
Created attachment 152330 [details]
test project
Created attachment 152348 [details]
20 seconds of self sampler while bg scan was running for 10 minutes
Created attachment 152349 [details]
IDE log for the same
Created attachment 152350 [details]
Hot points from the npss attached before
Created attachment 152351 [details]
Hot points from the npss attached before - dig deeper
Things sometime are getting worse - for no obvious reason today that same "test apache project" was bg-scanned for 10+ minutes. The only difference between previous test is the way i get to opening that project: 1. Start NB with project group containing my production MAVEN projects - ~300 modules, 7000 files. 2. Switch to "(none)" Project Group 3. Switch to "Apache Test" project group - bg scan was running for more than 10 minutes Somewhere in-between I've started SelfProfiler - i was not surprised that almost all time goes into MAVEN-related classes - see https://bugzilla-attachments-250833.netbeans.org/bugzilla/attachment.cgi?id=152350. Created attachment 152355 [details] Capture from process monitor - see: https://technet.microsoft.com/en-us/library/bb896645.aspx According to process monitor - during 15 seconds of bg scan NB queried attributes of various folders 213000 times - 14200/second (All of these are calls to parent operating system). Such implementation cannot be quick. Which version of the JDK are you using? Can you use the latest one and try this process again? https://jdk8.java.net/download.html On the daily basis I'm using Jdk7 (1.7.0_72), because my production project is still on Java7. I've started NB with Java8-u32/x64 and have got absolutely same behavior as on Java7, and by the way have reproduced another strange problem - it looks like in-memory "indexes" keep alive after switching to another Project group. Here are the steps i did right now: 1. Started NB (with Java8) 2. It opened last "Project Group" - it was the "my production one" 3. I switched to the "(none)" Project Group, waited until all processes are stopped 4. I switched to "big Apache" test Project Group - background scan is currently still running for somewhat 10+ minutes and ... I see plenty of messages "INFO [FaceletsLibrarySupport]: Invalidating facelets libraries due to a library descriptor change." in IDE log. I've never seen this message with "Apache sources" when I open that project in fresh IDE (on startup). This means that my initial "production project" which was opened initially still somehow affects scanning of currently open "Apache" project, but they are compleeetly unrelated. Created attachment 152378 [details]
selfsampler with "Invalidating facelets libraries due to a library descriptor change"
Created attachment 152379 [details]
selfsampler with "Invalidating facelets libraries due to a library descriptor change" (screenshot)
BTW: my sources have nothing to do with JSF - I have no idea why JSF comes in during background scan. And one more weird thing, just got into that use-case ... If I close NB with Project Group full of Maven modules and then start it - initial opening and bg scan of that project group takes at least 2+ minutes. (this is my typical performance test). If I close NB with "(none)" Project Group (i.e. NB is empty), Start empty NB and only then switch to Project Group with plenty of Maven Modules - "opening of the projects" takes 35 seconds (vs. typical 60-65 seconds) + "bg scan" completes in 15 seconds (vs 1:00..1:30 minutes) - with total of 50 seconds. Tested this twice with the same result. Isn't it strange? Why same action has such different performance? We will look at this problem. Not sure, this problem can be partially related - https://netbeans.org/bugzilla/show_bug.cgi?id=251571 JSF related staff with methods like 'scanFinished()' in stack trace cause performance problems also in JavaScript intellisense thanks for the thorough report lets summarize the particular problems: 1.) external mvn install causes a long background scan in a subsequent IDE start at least in the given example the mvn install command generates java files which then have to be scanned. This of course takes longer as when there are no new or modified files. You can also compare the particular log output ... e.g. a bg scan right after mvn install was executed before IDE start Complete indexing of 1,615 source roots took: 293,052 ms (New or modified files: 1,823, Deleted files: 0) vs a bg scan after a restart when everything was already scanned and no files were changed Complete indexing of 1,613 source roots took: 31,013 ms (New or modified files: 0, Deleted files: 0) 2.) closing project group with many mvn projects takes a long time i believe this is tracked in issue #250869 3.) some long background scans correlate with many JSF related logs even though there are no JSF related projects opened in the IDE seems like JFS is jumping into scanning even though it doesn't have to. Filed a separate issue #251973 as a workaround - you can try to deactivate the JSF in your IDE 4.) sometimes refactorings cause an unusually long background scan the time necessary to rescan depends on how many files were affected by the particular change. If a significant part of your projects was touch by the refactoring then also the scan will take a while. In case you think there is something specific to it and that it perhaps isn't addressed by other points at this place then feel free to file a new issue. Please, if possible, also attach a complete snapshot and message.log from such refactoring action. 5.) switching big project groups triggers an unusually long scan unfortunately, we weren't able to reproduce - each try resulted into times in accord with the times listed in 1.) - if there were no touched files then the overall scan time did not overreach what would be expected 6.) project open and scanning times differ when starting NB with a project group vs starting NB with no project opened and switching to a project group afterwards similar as in 5.) we weren't able to properly reproduce. There was a slight difference between those two times (starting with none and switching to PG was aprox 13%-25% faster then starting directly with a PG). Please note that while it seems to be the same actions, things might differ on implementation level as well as in interference with other IDE processes during and right after startup (actually this action was also a big longer). Do far hard to say what could be the particular problem ... 7.) MavenForBinaryQueryImpl is heavy on work during bg scan (https://bugzilla-attachments-250833.netbeans.org/bugzilla/attachment.cgi?id=152350) There might be some potential for a faster performance but not sure about possible side effects and afaik those particular calls are happening only during some phases and it probably won't have any dramatic effect on the overall time. Filed issue #251990 to cover that will close this as worksforme as the to be fixed problems are now tracked in separated issues thanks Thanks for attention to this issue. Will recheck behavior after MavenForBinaryQueryImpl$BinResult.getRoots() problem will get fixed. I've checked current status. It seems that fixing MavenForBinaryQueryImpl.getRoots() reduced time of bgscan from 1.5 minutes to around 40 seconds. Thanks. Meanwhile overall 1.5 minutes to open the project still seems to me to be too long for that action. I've analysed current selfprofiler on my real project and filed 2 other issues: Optimize 'Open Projects' phase of the 'Open Project Group' action https://netbeans.org/bugzilla/show_bug.cgi?id=252068 Optimize 'Background Scan' phase of the 'Open Project Group' action https://netbeans.org/bugzilla/show_bug.cgi?id=252069 |