Bug 195814 - "Go to file" is really slow and does search files unless backspace is pressed
"Go to file" is really slow and does search files unless backspace is pressed
Status: VERIFIED FIXED
Product: editor
Classification: Unclassified
Component: Navigation
7.0
All All
: P2 with 20 votes (vote)
: 8.0.1
Assigned To: Tomas Zezula
issues@editor
: 8.0_WAIVER_APPROVED, PERFORMANCE, PLAN
: 196960 (view as bug list)
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2011-02-22 15:25 UTC by nebelmann
Modified: 2014-08-19 09:44 UTC (History)
10 users (show)

See Also:
Issue Type: DEFECT
:


Attachments
Profiler snapshot taken during go to file (139.94 KB, application/octet-stream)
2011-02-22 15:35 UTC, nebelmann
Details
mvn dependency:tree extract (40.86 KB, text/plain)
2011-03-03 15:15 UTC, nebelmann
Details
Logs with o.n.m.jumpto.file.FileSearchAction.level=FINE (12.47 KB, application/octet-stream)
2011-03-16 10:03 UTC, nebelmann
Details
NPS file (105.09 KB, application/octet-stream)
2013-02-26 17:54 UTC, Michel Graciano
Details
Took 10 seconds to open java.lang.Object (248.61 KB, application/octet-stream)
2014-05-11 14:08 UTC, Leperous
Details

Note You need to log in before you can comment on or make changes to this bug.
Description nebelmann 2011-02-22 15:25:15 UTC
Hello,

I'm having issues which I find very annoying and prevents me from using the "Go to File" popup. I'm currently working on a Maven project with quite a lot of modules, but I only have 5 or 6 projects opened atm. There's also a lot of jar dependencies which could cause a lot of files to index.

I'm facing 2 problems:

 - if I start typing a file name, the results area always show "No Files Found" and seem to do nothing unless I press the backspace key. At this moment it shows "Searching...". The strange thing is, if I start typing a file name and wait, it will eventually show up results even if I had no feedback about any actual search being in progress...

 - the second problem is much more annoying, with both options checked (case sentitive and prefer current project), searching for a file name which will give a single result takes up to 25s!!

This is taking so long that I prefer searching the files by myself in the project structure :/

Config:
Netbeans 7.0 beta2 on Windows XP SP3, Intel Core2 duo @2.66GHz, 3.23Gb of RAM.

FYI, I have the same problem with netbeans 6.9.1 when I work on the same project.
Comment 1 nebelmann 2011-02-22 15:35:34 UTC
Created attachment 106302 [details]
Profiler snapshot taken during go to file
Comment 2 Tomas Zezula 2011-02-22 15:39:25 UTC
The speed of indexed part was improved in the NB 7.0 by index caches, unfortunately part of it comes not from index and is searched on disk. If you attache the output of self profile? Thanks
Comment 3 Tomas Zezula 2011-02-22 15:39:57 UTC
It's already attached. :-) Thanks
Comment 4 nebelmann 2011-02-22 16:41:33 UTC
Could it be related to my antivirus (McAfee) which is slowing down file scanning? 

Unfortunately I don't have the necessary permissions to shut it down to make a test...
Comment 5 Tomas Zezula 2011-02-23 12:55:04 UTC
Finally I've got to the snapshot.
It's as I've described above the nearly whole time is spent in searching in non indexed parts of the project. 

Here are the times:
Complete search: 76857ms
Search in indexed part: 2526ms

Search in non indexed part: 74331ms (The non indexed search is just going through the project folders ignoring the indexed folders (source roots) and listing them, the same as UNIX find command).

Yes, the Antivirus may have affect on it, but most of antivirus are activated by write not by read.
I am not Maven expert but it seems that the project has significant part which is not source and it's unindexed. Can you describe more the project structure? What is the biggest part of the project (excluding source roots - these are searched by parsing API in 2.5 s)?
thanks
Comment 6 nebelmann 2011-03-03 15:15:43 UTC
Created attachment 106657 [details]
mvn dependency:tree extract

Indeed it is a maven project with quite a lot of dependencies (apache commons, ibatis, hibernate, spring etc.). I've attached a sample of dependency tree so that you can have an idea of the number of jars included. I cut the output because other modules have more or less the same dependencies.

For example our "core" project has about 160 dependencies (including transitive ones), 1600 classes (source code files) and several hundreds of JUnit test classes, XML files etc. The snapshot I provided initially was a search in this project.

I've tried using Intellij on the same project, and the go to file function finds instantly what I'm looking for. The problem is that NetBeans has a better maven integration IMHO, so I'd really like to continue using it without worrying about performance issues :)

What could I do to provide more information and help fixing this issue?
Comment 7 nebelmann 2011-03-15 11:08:31 UTC
Any chance to see a performance improvement on this feature in NB7 final ?
Comment 8 Tomas Zezula 2011-03-15 12:13:26 UTC
Sorry I didn't got to it yet.
I will look at it later this week.
Comment 9 Tomas Zezula 2011-03-15 17:29:25 UTC
The dependency graph is not much helpful, also the number of sources is not helpful.
The snapshot shows that the indexed search is fast but significant time is spent in the non indexed part (non sources).
I've added a performance logging into jet-main b0eb35fa13ce.
When it becomes part of daily build can you try to run netbeans with -J-Dorg.netbeans.modules.jumpto.file.FileSearchAction.level=FINE to find out the times of indexed, providers and non indexed search.
Comment 10 Quality Engineering 2011-03-16 09:42:10 UTC
Integrated into 'main-golden', will be available in build *201103160400* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/b0eb35fa13ce
User: Tomas Zezula <tzezula@netbeans.org>
Log: #195814 - performance logging
Comment 11 nebelmann 2011-03-16 10:03:00 UTC
Created attachment 107039 [details]
Logs with o.n.m.jumpto.file.FileSearchAction.level=FINE

All right, I just downloaded the zip from hudson and did a few "go to file" with either indexed files (.java) or unindexed files (.xml or .jspx if I did understand correctly). The first result should be quite irrelevant because NB started scanning projects at the same time, but for the next searches there was no background task going on.

It seems that you are right, most of the time is spent searching in unindexed files (see attachment for full logs):

FINE [org.netbeans.modules.jumpto.file.FileSearchAction]: Worker for DealTermBusiness - started 3 391 ms.
FINE [org.netbeans.modules.jumpto.file.FileSearchAction]: Indexed Search: 0ms
FINE [org.netbeans.modules.jumpto.file.FileSearchAction]: Providers Search: 0ms
FINE [org.netbeans.modules.jumpto.file.FileSearchAction]: Unindexed search: 6 187ms
FINE [org.netbeans.modules.jumpto.file.FileSearchAction]: Worker for text DealTermBusiness finished after 9 578 ms.
FINE [org.netbeans.modules.jumpto.file.FileSearchAction]: Real search time 9578 ms.

I have one question though. How are compiled classes in JARs considered as? Are they indexed or unindexed resources?
Comment 12 Tomas Zezula 2011-03-16 10:52:43 UTC
The whole time is spent in the non indexed search (outside of sources).
The goto file does not search inside of jar files.
I cannot help much as the languages infrastructure does not know about non source folders.
Comment 13 nebelmann 2011-03-16 11:14:37 UTC
That's quite interesting: I was wondering how to reduce the number of non-indexed files to scan, assuming that the search was done in opened projects. I closed my projects except one, which is quite small. The results were displayed instantly. I reopened the core project and I had no performance problem either. But when I opened the root project (which is only a pom project with multiple submodules), I get a huge search time again.

I only use this project to open submodules when needed, so I don't need it very often. I can get a satisfying workaround by opening only submodules I need...
Comment 14 Tomas Zezula 2011-03-16 12:51:55 UTC
The root project probably has lots of non indexed files.
I will try to experiment with it.
Comment 15 nebelmann 2011-03-16 13:58:08 UTC
I don't know how the maven support works internally but I would guess that when I open the root project, NB also takes into consideration all the non-indexed files located in the submodules.

The root project has no direct non-indexed file, but the submodules do.
Comment 16 Tomas Zezula 2011-03-16 14:35:00 UTC
Yes, it seems so.
Comment 17 Tomas Zezula 2011-03-22 16:52:58 UTC
*** Bug 196960 has been marked as a duplicate of this bug. ***
Comment 18 Tomas Zezula 2011-03-23 13:04:01 UTC
Affects only non sources part of the project.
I will try to create a module which will pretend that the non sources part of the project is a "special" source which uses only file indexer.
Comment 19 markiewb 2012-12-17 18:52:57 UTC
Set target-milestone of open issue to TBD (was < 7.3, f.e. 6.8), so the issue doesn't get lost.
Comment 20 markiewb 2012-12-17 19:00:12 UTC
Set target-milestone of open issue to TBD (was < 7.3, f.e. 6.8), so the issue
doesn't get lost. This time the target milestone is really set.
Comment 21 Michel Graciano 2013-02-26 14:19:04 UTC
I have this problem too, our project is a huge free form project and it takes too long to find any file. Any chance to have this addressed for next release?
Comment 22 Tomas Zezula 2013-02-26 16:43:20 UTC
The problem of the current GTF implementation is that it waits for the slowest provider before it displays the results. The slowest provider is the file crawling of all folders which are not under source root. Making the dialog to display partial results before the slowest provider completes will improve the search time for files under source roots.
>Any chance to have this addressed for next release?
Hard to say, there are no plans for next release yet.
Can you attach the nps snap shot.
Thanks!
Comment 23 Michel Graciano 2013-02-26 17:54:33 UTC
Created attachment 131909 [details]
NPS file

I hope it helps.
Comment 24 tomzi 2013-03-07 12:49:33 UTC
Can you add an option to 'Goto File' also in the Favourites tab. In this tab usually all the folder for database scripts are mounted, since they are not really (maven, ant) projects.
Comment 25 Michel Graciano 2013-03-08 17:41:59 UTC
(In reply to comment #24)
> Can you add an option to 'Goto File' also in the Favourites tab. In this tab
> usually all the folder for database scripts are mounted, since they are not
> really (maven, ant) projects.

tomzi, can you file a new issue about it? I agree that this option could be helpful, but it is a different problem and should be treated as well. If you don't mind, I can do this for you, just let me know.
Comment 26 Michel Graciano 2013-05-27 19:24:17 UTC
(In reply to comment #22)
> >Any chance to have this addressed for next release?
> Hard to say, there are no plans for next release yet.

Now that we have some plans for the next release, any chance to consider some improvements on this area? IMHO, it is a critical usability problem which could be considered a P2, but since it is just a matter of interpretation, you can disagree with me :)
Comment 27 ceklock 2013-12-19 19:28:28 UTC
Works without slowness for me in a medium-sized maven project.



Product Version: NetBeans IDE Dev (Build 201312190002)
Java: 1.7.0_45; Java HotSpot(TM) 64-Bit Server VM 24.45-b08
Runtime: Java(TM) SE Runtime Environment 1.7.0_45-b18
System: Windows 7 version 6.1 running on amd64; Cp1252; pt_BR (nb)
Comment 28 Michel Graciano 2014-03-11 11:56:38 UTC
Okay, time for some real data. I tested it with the same hardware and project, but in Windows and Linux I had fairly different results. We have a project with 17.619 files where 10.509 are not indexed AFAIK. Our structure is something like that:

project_root
  . other_files (10.509 files which are not mounted as source root in the project)
  . src (default Maven structure with 6.982 files)
  . (here more files like pom.xml etc)

Given this structure, AFAIK 'other_files' folder will not be index (as discussed before in this issue) and is probably the source of the issue because it is not mounted as a source root in the project.

I tested it in my Linux workstations searching for a single js file and it took something like 3 seconds, which is acceptable. But when we tested it in a Windows workstation it took more than 20 seconds, which now is not acceptable. I should highlight that the searched file is inside a folder which is mounted as source root in the project, so it probably is index.

Above is the configuration of the workstations.

Product Version: NetBeans IDE 8.0 RC1 (Build 201402242200)
Java: 1.7.0_51; Java HotSpot(TM) 64-Bit Server VM 24.51-b03
Runtime: Java(TM) SE Runtime Environment 1.7.0_51-b13
System: Windows 7 version 6.1 running on amd64; Cp1252; en_US (nb)

Product Version: NetBeans IDE Dev (Build 20140227-8aabb71e3f6d)
Updates: Updates available
Java: 1.7.0_40; Java HotSpot(TM) 64-Bit Server VM 24.0-b56
Runtime: Java(TM) SE Runtime Environment 1.7.0_40-b43
System: Linux version 3.4.63-2.44-desktop running on amd64; UTF-8; en_US (nb)

I hope this issue can have some attention soon.
Comment 29 Marian Mirilovic 2014-03-13 10:20:38 UTC
20 votes ... at least P2
Comment 30 everflux 2014-03-13 10:40:55 UTC
Could this somehow be related to https://netbeans.org/bugzilla/show_bug.cgi?id=241602 ?
Comment 31 ChrisLE 2014-04-25 21:33:10 UTC
Got the same problem with a big AngularJS project generated with yeoman (yo angular). The node_modules folder has a lot of files maybe ~10.000 and when my cursor is in the current file, for me for example Gruntfile.js I typed Shift + Alt + O, Go To File opens, shows No results until 3 - 4 seconds and this is not acceptable for me and this will happens after close and open the dialog again.

I use netbeans for daily work and some other developers, who do pair programming, shows me how fast IntelliJ is and how slow netbeans, but I dont want to switch to IntelliJ. Ok we have different machines and maybe IntelliJ will be slow too, so I will try it with the same project on my machines.


Regards

Chris
Comment 32 Petr Jiricka 2014-04-28 07:42:27 UTC
AngularJS -> cc'ing Petr Pisl.
Comment 33 Leperous 2014-05-11 14:08:37 UTC
Created attachment 147218 [details]
Took 10 seconds to open java.lang.Object

Attached profile for Netbeans taking 10 seconds to open java.lang.Object. Usually this is "instant" when the IDE has recently been restarted.
Comment 34 Tomas Zezula 2014-07-17 09:17:54 UTC
In rely to Comment #28:
Thanks Michel Graciano for measurement!
The Go To File was rewritten in dev (NB 8.0.1) to be asynchronous like Mac Spotlight, the files are added into the result immediately as that are found. No need to wait for the slowest search provider (unindexed files).
The sorting of providers is:

1) Index for current project
2) Index for other opened projects
3) Unindexed for current project
4) Unindex for other opened projects

so the file in current project sources is displayed very soon, then files in sources of other project, etc...
Comment 36 Michel Graciano 2014-07-18 14:19:59 UTC
(In reply to Tomas Zezula from comment #34)
> In rely to Comment #28:
> Thanks Michel Graciano for measurement!
> The Go To File was rewritten in dev (NB 8.0.1) to be asynchronous like Mac
> Spotlight, the files are added into the result immediately as that are
> found. No need to wait for the slowest search provider (unindexed files).
> The sorting of providers is:
> 
> 1) Index for current project
> 2) Index for other opened projects
> 3) Unindexed for current project
> 4) Unindex for other opened projects
> 
> so the file in current project sources is displayed very soon, then files in
> sources of other project, etc...

Thank you very much. I have tested it and is really really better. I really appreciate we have this issue finally fixed. It is already available for 8.0.1 or it is yet to be released? I usually works with daily builds, so I already have the fix, I will check 8.X soon.

Thanks again.
v. Build 20140715-6b3e5e696bec
Comment 37 Tomas Zezula 2014-07-21 10:35:18 UTC
>It is already available for 8.0.1 or it is yet to be released?
It's in the trunk (dev) now, the 8.0.1 will be branched form the trunk, so it will be a part of NB 8.0.1
Comment 38 ChrisLE 2014-07-21 10:53:22 UTC
Awesome, this is a really big improvement, thx guys for the work :)
Comment 39 ChrisLE 2014-08-08 11:56:33 UTC
Will enhancement available for go to symbol and type too?
Comment 40 Michel Graciano 2014-08-09 01:32:16 UTC
(In reply to ChrisLE from comment #39)
> Will enhancement available for go to symbol and type too?

For 'Go to File' it is already optimized with this strategy for a while already, and AFAIK 'Go to Symbol' too.
Comment 41 Tomas Zezula 2014-08-19 09:44:57 UTC
Sorry for later answer I was on vacation.

The Go To Type and Go To Symbol is a bit different.
All the Go To File, Type, Symbol are plugable.
The Go To File plugins are hosted in the same module (easy to rewrite) and they all do the same (searching files) but in different way (using language index creed during scan of sources, doing disk search) on the set of opened projects.
The Go To File plugins were changed to first query the active project and then the rest, updating the result after each root.

The Go to Symbol and Go To Type plugins are different. They are language specific, for example there is JavaTypeProvider, PHPTypeProvider, etc. All these providers collect all the result for opened projects and then return them. Showing partial results will help as the Java types will be displayed before the PHP types will be computed. But much better will be to query individual roots and display the results after each root as it's done now in Go To File. The jumpto module has no official API and has lots of copied code. If time permits I will get rid of the jumpto friend API create a new performance aware API, remove duplicated code and allow partial results for all the Go To. However I am an "owner" of jumpto as it has no owner and the other parts I am working on like project system, java core, generic languages infrastructure have higher priority. But I will try to.


By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2014, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo