I'm having issues which I find very annoying and prevents me from using the "Go to File" popup. I'm currently working on a Maven project with quite a lot of modules, but I only have 5 or 6 projects opened atm. There's also a lot of jar dependencies which could cause a lot of files to index.
I'm facing 2 problems:
- if I start typing a file name, the results area always show "No Files Found" and seem to do nothing unless I press the backspace key. At this moment it shows "Searching...". The strange thing is, if I start typing a file name and wait, it will eventually show up results even if I had no feedback about any actual search being in progress...
- the second problem is much more annoying, with both options checked (case sentitive and prefer current project), searching for a file name which will give a single result takes up to 25s!!
This is taking so long that I prefer searching the files by myself in the project structure :/
Netbeans 7.0 beta2 on Windows XP SP3, Intel Core2 duo @2.66GHz, 3.23Gb of RAM.
FYI, I have the same problem with netbeans 6.9.1 when I work on the same project.
Created attachment 106302 [details]
Profiler snapshot taken during go to file
The speed of indexed part was improved in the NB 7.0 by index caches, unfortunately part of it comes not from index and is searched on disk. If you attache the output of self profile? Thanks
It's already attached. :-) Thanks
Could it be related to my antivirus (McAfee) which is slowing down file scanning?
Unfortunately I don't have the necessary permissions to shut it down to make a test...
Finally I've got to the snapshot.
It's as I've described above the nearly whole time is spent in searching in non indexed parts of the project.
Here are the times:
Complete search: 76857ms
Search in indexed part: 2526ms
Search in non indexed part: 74331ms (The non indexed search is just going through the project folders ignoring the indexed folders (source roots) and listing them, the same as UNIX find command).
Yes, the Antivirus may have affect on it, but most of antivirus are activated by write not by read.
I am not Maven expert but it seems that the project has significant part which is not source and it's unindexed. Can you describe more the project structure? What is the biggest part of the project (excluding source roots - these are searched by parsing API in 2.5 s)?
Created attachment 106657 [details]
mvn dependency:tree extract
Indeed it is a maven project with quite a lot of dependencies (apache commons, ibatis, hibernate, spring etc.). I've attached a sample of dependency tree so that you can have an idea of the number of jars included. I cut the output because other modules have more or less the same dependencies.
For example our "core" project has about 160 dependencies (including transitive ones), 1600 classes (source code files) and several hundreds of JUnit test classes, XML files etc. The snapshot I provided initially was a search in this project.
I've tried using Intellij on the same project, and the go to file function finds instantly what I'm looking for. The problem is that NetBeans has a better maven integration IMHO, so I'd really like to continue using it without worrying about performance issues :)
What could I do to provide more information and help fixing this issue?
Any chance to see a performance improvement on this feature in NB7 final ?
Sorry I didn't got to it yet.
I will look at it later this week.
The dependency graph is not much helpful, also the number of sources is not helpful.
The snapshot shows that the indexed search is fast but significant time is spent in the non indexed part (non sources).
I've added a performance logging into jet-main b0eb35fa13ce.
When it becomes part of daily build can you try to run netbeans with -J-Dorg.netbeans.modules.jumpto.file.FileSearchAction.level=FINE to find out the times of indexed, providers and non indexed search.
Integrated into 'main-golden', will be available in build *201103160400* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
User: Tomas Zezula <email@example.com>
Log: #195814 - performance logging
Created attachment 107039 [details]
Logs with o.n.m.jumpto.file.FileSearchAction.level=FINE
All right, I just downloaded the zip from hudson and did a few "go to file" with either indexed files (.java) or unindexed files (.xml or .jspx if I did understand correctly). The first result should be quite irrelevant because NB started scanning projects at the same time, but for the next searches there was no background task going on.
It seems that you are right, most of the time is spent searching in unindexed files (see attachment for full logs):
FINE [org.netbeans.modules.jumpto.file.FileSearchAction]: Worker for DealTermBusiness - started 3 391 ms.
FINE [org.netbeans.modules.jumpto.file.FileSearchAction]: Indexed Search: 0ms
FINE [org.netbeans.modules.jumpto.file.FileSearchAction]: Providers Search: 0ms
FINE [org.netbeans.modules.jumpto.file.FileSearchAction]: Unindexed search: 6 187ms
FINE [org.netbeans.modules.jumpto.file.FileSearchAction]: Worker for text DealTermBusiness finished after 9 578 ms.
FINE [org.netbeans.modules.jumpto.file.FileSearchAction]: Real search time 9578 ms.
I have one question though. How are compiled classes in JARs considered as? Are they indexed or unindexed resources?
The whole time is spent in the non indexed search (outside of sources).
The goto file does not search inside of jar files.
I cannot help much as the languages infrastructure does not know about non source folders.
That's quite interesting: I was wondering how to reduce the number of non-indexed files to scan, assuming that the search was done in opened projects. I closed my projects except one, which is quite small. The results were displayed instantly. I reopened the core project and I had no performance problem either. But when I opened the root project (which is only a pom project with multiple submodules), I get a huge search time again.
I only use this project to open submodules when needed, so I don't need it very often. I can get a satisfying workaround by opening only submodules I need...
The root project probably has lots of non indexed files.
I will try to experiment with it.
I don't know how the maven support works internally but I would guess that when I open the root project, NB also takes into consideration all the non-indexed files located in the submodules.
The root project has no direct non-indexed file, but the submodules do.
Yes, it seems so.
*** Bug 196960 has been marked as a duplicate of this bug. ***
Affects only non sources part of the project.
I will try to create a module which will pretend that the non sources part of the project is a "special" source which uses only file indexer.
Set target-milestone of open issue to TBD (was < 7.3, f.e. 6.8), so the issue doesn't get lost.
Set target-milestone of open issue to TBD (was < 7.3, f.e. 6.8), so the issue
doesn't get lost. This time the target milestone is really set.
I have this problem too, our project is a huge free form project and it takes too long to find any file. Any chance to have this addressed for next release?
The problem of the current GTF implementation is that it waits for the slowest provider before it displays the results. The slowest provider is the file crawling of all folders which are not under source root. Making the dialog to display partial results before the slowest provider completes will improve the search time for files under source roots.
>Any chance to have this addressed for next release?
Hard to say, there are no plans for next release yet.
Can you attach the nps snap shot.
Created attachment 131909 [details]
I hope it helps.
Can you add an option to 'Goto File' also in the Favourites tab. In this tab usually all the folder for database scripts are mounted, since they are not really (maven, ant) projects.
(In reply to comment #24)
> Can you add an option to 'Goto File' also in the Favourites tab. In this tab
> usually all the folder for database scripts are mounted, since they are not
> really (maven, ant) projects.
tomzi, can you file a new issue about it? I agree that this option could be helpful, but it is a different problem and should be treated as well. If you don't mind, I can do this for you, just let me know.
(In reply to comment #22)
> >Any chance to have this addressed for next release?
> Hard to say, there are no plans for next release yet.
Now that we have some plans for the next release, any chance to consider some improvements on this area? IMHO, it is a critical usability problem which could be considered a P2, but since it is just a matter of interpretation, you can disagree with me :)
Works without slowness for me in a medium-sized maven project.
Product Version: NetBeans IDE Dev (Build 201312190002)
Java: 1.7.0_45; Java HotSpot(TM) 64-Bit Server VM 24.45-b08
Runtime: Java(TM) SE Runtime Environment 1.7.0_45-b18
System: Windows 7 version 6.1 running on amd64; Cp1252; pt_BR (nb)
Okay, time for some real data. I tested it with the same hardware and project, but in Windows and Linux I had fairly different results. We have a project with 17.619 files where 10.509 are not indexed AFAIK. Our structure is something like that:
. other_files (10.509 files which are not mounted as source root in the project)
. src (default Maven structure with 6.982 files)
. (here more files like pom.xml etc)
Given this structure, AFAIK 'other_files' folder will not be index (as discussed before in this issue) and is probably the source of the issue because it is not mounted as a source root in the project.
I tested it in my Linux workstations searching for a single js file and it took something like 3 seconds, which is acceptable. But when we tested it in a Windows workstation it took more than 20 seconds, which now is not acceptable. I should highlight that the searched file is inside a folder which is mounted as source root in the project, so it probably is index.
Above is the configuration of the workstations.
Product Version: NetBeans IDE 8.0 RC1 (Build 201402242200)
Java: 1.7.0_51; Java HotSpot(TM) 64-Bit Server VM 24.51-b03
Runtime: Java(TM) SE Runtime Environment 1.7.0_51-b13
System: Windows 7 version 6.1 running on amd64; Cp1252; en_US (nb)
Product Version: NetBeans IDE Dev (Build 20140227-8aabb71e3f6d)
Updates: Updates available
Java: 1.7.0_40; Java HotSpot(TM) 64-Bit Server VM 24.0-b56
Runtime: Java(TM) SE Runtime Environment 1.7.0_40-b43
System: Linux version 3.4.63-2.44-desktop running on amd64; UTF-8; en_US (nb)
I hope this issue can have some attention soon.
20 votes ... at least P2
Could this somehow be related to https://netbeans.org/bugzilla/show_bug.cgi?id=241602 ?
Got the same problem with a big AngularJS project generated with yeoman (yo angular). The node_modules folder has a lot of files maybe ~10.000 and when my cursor is in the current file, for me for example Gruntfile.js I typed Shift + Alt + O, Go To File opens, shows No results until 3 - 4 seconds and this is not acceptable for me and this will happens after close and open the dialog again.
I use netbeans for daily work and some other developers, who do pair programming, shows me how fast IntelliJ is and how slow netbeans, but I dont want to switch to IntelliJ. Ok we have different machines and maybe IntelliJ will be slow too, so I will try it with the same project on my machines.
AngularJS -> cc'ing Petr Pisl.
Created attachment 147218 [details]
Took 10 seconds to open java.lang.Object
Attached profile for Netbeans taking 10 seconds to open java.lang.Object. Usually this is "instant" when the IDE has recently been restarted.
In rely to Comment #28:
Thanks Michel Graciano for measurement!
The Go To File was rewritten in dev (NB 8.0.1) to be asynchronous like Mac Spotlight, the files are added into the result immediately as that are found. No need to wait for the slowest search provider (unindexed files).
The sorting of providers is:
1) Index for current project
2) Index for other opened projects
3) Unindexed for current project
4) Unindex for other opened projects
so the file in current project sources is displayed very soon, then files in sources of other project, etc...
The commits were:
(In reply to Tomas Zezula from comment #34)
> In rely to Comment #28:
> Thanks Michel Graciano for measurement!
> The Go To File was rewritten in dev (NB 8.0.1) to be asynchronous like Mac
> Spotlight, the files are added into the result immediately as that are
> found. No need to wait for the slowest search provider (unindexed files).
> The sorting of providers is:
> 1) Index for current project
> 2) Index for other opened projects
> 3) Unindexed for current project
> 4) Unindex for other opened projects
> so the file in current project sources is displayed very soon, then files in
> sources of other project, etc...
Thank you very much. I have tested it and is really really better. I really appreciate we have this issue finally fixed. It is already available for 8.0.1 or it is yet to be released? I usually works with daily builds, so I already have the fix, I will check 8.X soon.
v. Build 20140715-6b3e5e696bec
>It is already available for 8.0.1 or it is yet to be released?
It's in the trunk (dev) now, the 8.0.1 will be branched form the trunk, so it will be a part of NB 8.0.1
Awesome, this is a really big improvement, thx guys for the work :)
Will enhancement available for go to symbol and type too?
(In reply to ChrisLE from comment #39)
> Will enhancement available for go to symbol and type too?
For 'Go to File' it is already optimized with this strategy for a while already, and AFAIK 'Go to Symbol' too.
Sorry for later answer I was on vacation.
The Go To Type and Go To Symbol is a bit different.
All the Go To File, Type, Symbol are plugable.
The Go To File plugins are hosted in the same module (easy to rewrite) and they all do the same (searching files) but in different way (using language index creed during scan of sources, doing disk search) on the set of opened projects.
The Go To File plugins were changed to first query the active project and then the rest, updating the result after each root.
The Go to Symbol and Go To Type plugins are different. They are language specific, for example there is JavaTypeProvider, PHPTypeProvider, etc. All these providers collect all the result for opened projects and then return them. Showing partial results will help as the Java types will be displayed before the PHP types will be computed. But much better will be to query individual roots and display the results after each root as it's done now in Go To File. The jumpto module has no official API and has lots of copied code. If time permits I will get rid of the jumpto friend API create a new performance aware API, remove duplicated code and allow partial results for all the Go To. However I am an "owner" of jumpto as it has no owner and the other parts I am working on like project system, java core, generic languages infrastructure have higher priority. But I will try to.