Bug 195814 - "Go to file" is really slow and does search files unless backspace is pressed
"Go to file" is really slow and does search files unless backspace is pressed
Status: NEW
Product: editor
Classification: Unclassified
Component: Navigation
7.0
All All
: P2 with 23 votes (vote)
: TBD
Assigned To: Svata Dedic
issues@editor
: 8.0_WAIVER_APPROVED, PERFORMANCE, PLAN
: 196960 (view as bug list)
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2011-02-22 15:25 UTC by nebelmann
Modified: 2014-03-13 10:40 UTC (History)
9 users (show)

See Also:
Issue Type: DEFECT
:


Attachments
Profiler snapshot taken during go to file (139.94 KB, application/octet-stream)
2011-02-22 15:35 UTC, nebelmann
Details
mvn dependency:tree extract (40.86 KB, text/plain)
2011-03-03 15:15 UTC, nebelmann
Details
Logs with o.n.m.jumpto.file.FileSearchAction.level=FINE (12.47 KB, application/octet-stream)
2011-03-16 10:03 UTC, nebelmann
Details
NPS file (105.09 KB, application/octet-stream)
2013-02-26 17:54 UTC, Michel Graciano
Details

Note You need to log in before you can comment on or make changes to this bug.
Description nebelmann 2011-02-22 15:25:15 UTC
Hello,

I'm having issues which I find very annoying and prevents me from using the "Go to File" popup. I'm currently working on a Maven project with quite a lot of modules, but I only have 5 or 6 projects opened atm. There's also a lot of jar dependencies which could cause a lot of files to index.

I'm facing 2 problems:

 - if I start typing a file name, the results area always show "No Files Found" and seem to do nothing unless I press the backspace key. At this moment it shows "Searching...". The strange thing is, if I start typing a file name and wait, it will eventually show up results even if I had no feedback about any actual search being in progress...

 - the second problem is much more annoying, with both options checked (case sentitive and prefer current project), searching for a file name which will give a single result takes up to 25s!!

This is taking so long that I prefer searching the files by myself in the project structure :/

Config:
Netbeans 7.0 beta2 on Windows XP SP3, Intel Core2 duo @2.66GHz, 3.23Gb of RAM.

FYI, I have the same problem with netbeans 6.9.1 when I work on the same project.
Comment 1 nebelmann 2011-02-22 15:35:34 UTC
Created attachment 106302 [details]
Profiler snapshot taken during go to file
Comment 2 Tomas Zezula 2011-02-22 15:39:25 UTC
The speed of indexed part was improved in the NB 7.0 by index caches, unfortunately part of it comes not from index and is searched on disk. If you attache the output of self profile? Thanks
Comment 3 Tomas Zezula 2011-02-22 15:39:57 UTC
It's already attached. :-) Thanks
Comment 4 nebelmann 2011-02-22 16:41:33 UTC
Could it be related to my antivirus (McAfee) which is slowing down file scanning? 

Unfortunately I don't have the necessary permissions to shut it down to make a test...
Comment 5 Tomas Zezula 2011-02-23 12:55:04 UTC
Finally I've got to the snapshot.
It's as I've described above the nearly whole time is spent in searching in non indexed parts of the project. 

Here are the times:
Complete search: 76857ms
Search in indexed part: 2526ms

Search in non indexed part: 74331ms (The non indexed search is just going through the project folders ignoring the indexed folders (source roots) and listing them, the same as UNIX find command).

Yes, the Antivirus may have affect on it, but most of antivirus are activated by write not by read.
I am not Maven expert but it seems that the project has significant part which is not source and it's unindexed. Can you describe more the project structure? What is the biggest part of the project (excluding source roots - these are searched by parsing API in 2.5 s)?
thanks
Comment 6 nebelmann 2011-03-03 15:15:43 UTC
Created attachment 106657 [details]
mvn dependency:tree extract

Indeed it is a maven project with quite a lot of dependencies (apache commons, ibatis, hibernate, spring etc.). I've attached a sample of dependency tree so that you can have an idea of the number of jars included. I cut the output because other modules have more or less the same dependencies.

For example our "core" project has about 160 dependencies (including transitive ones), 1600 classes (source code files) and several hundreds of JUnit test classes, XML files etc. The snapshot I provided initially was a search in this project.

I've tried using Intellij on the same project, and the go to file function finds instantly what I'm looking for. The problem is that NetBeans has a better maven integration IMHO, so I'd really like to continue using it without worrying about performance issues :)

What could I do to provide more information and help fixing this issue?
Comment 7 nebelmann 2011-03-15 11:08:31 UTC
Any chance to see a performance improvement on this feature in NB7 final ?
Comment 8 Tomas Zezula 2011-03-15 12:13:26 UTC
Sorry I didn't got to it yet.
I will look at it later this week.
Comment 9 Tomas Zezula 2011-03-15 17:29:25 UTC
The dependency graph is not much helpful, also the number of sources is not helpful.
The snapshot shows that the indexed search is fast but significant time is spent in the non indexed part (non sources).
I've added a performance logging into jet-main b0eb35fa13ce.
When it becomes part of daily build can you try to run netbeans with -J-Dorg.netbeans.modules.jumpto.file.FileSearchAction.level=FINE to find out the times of indexed, providers and non indexed search.
Comment 10 Quality Engineering 2011-03-16 09:42:10 UTC
Integrated into 'main-golden', will be available in build *201103160400* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/b0eb35fa13ce
User: Tomas Zezula <tzezula@netbeans.org>
Log: #195814 - performance logging
Comment 11 nebelmann 2011-03-16 10:03:00 UTC
Created attachment 107039 [details]
Logs with o.n.m.jumpto.file.FileSearchAction.level=FINE

All right, I just downloaded the zip from hudson and did a few "go to file" with either indexed files (.java) or unindexed files (.xml or .jspx if I did understand correctly). The first result should be quite irrelevant because NB started scanning projects at the same time, but for the next searches there was no background task going on.

It seems that you are right, most of the time is spent searching in unindexed files (see attachment for full logs):

FINE [org.netbeans.modules.jumpto.file.FileSearchAction]: Worker for DealTermBusiness - started 3 391 ms.
FINE [org.netbeans.modules.jumpto.file.FileSearchAction]: Indexed Search: 0ms
FINE [org.netbeans.modules.jumpto.file.FileSearchAction]: Providers Search: 0ms
FINE [org.netbeans.modules.jumpto.file.FileSearchAction]: Unindexed search: 6 187ms
FINE [org.netbeans.modules.jumpto.file.FileSearchAction]: Worker for text DealTermBusiness finished after 9 578 ms.
FINE [org.netbeans.modules.jumpto.file.FileSearchAction]: Real search time 9578 ms.

I have one question though. How are compiled classes in JARs considered as? Are they indexed or unindexed resources?
Comment 12 Tomas Zezula 2011-03-16 10:52:43 UTC
The whole time is spent in the non indexed search (outside of sources).
The goto file does not search inside of jar files.
I cannot help much as the languages infrastructure does not know about non source folders.
Comment 13 nebelmann 2011-03-16 11:14:37 UTC
That's quite interesting: I was wondering how to reduce the number of non-indexed files to scan, assuming that the search was done in opened projects. I closed my projects except one, which is quite small. The results were displayed instantly. I reopened the core project and I had no performance problem either. But when I opened the root project (which is only a pom project with multiple submodules), I get a huge search time again.

I only use this project to open submodules when needed, so I don't need it very often. I can get a satisfying workaround by opening only submodules I need...
Comment 14 Tomas Zezula 2011-03-16 12:51:55 UTC
The root project probably has lots of non indexed files.
I will try to experiment with it.
Comment 15 nebelmann 2011-03-16 13:58:08 UTC
I don't know how the maven support works internally but I would guess that when I open the root project, NB also takes into consideration all the non-indexed files located in the submodules.

The root project has no direct non-indexed file, but the submodules do.
Comment 16 Tomas Zezula 2011-03-16 14:35:00 UTC
Yes, it seems so.
Comment 17 Tomas Zezula 2011-03-22 16:52:58 UTC
*** Bug 196960 has been marked as a duplicate of this bug. ***
Comment 18 Tomas Zezula 2011-03-23 13:04:01 UTC
Affects only non sources part of the project.
I will try to create a module which will pretend that the non sources part of the project is a "special" source which uses only file indexer.
Comment 19 markiewb 2012-12-17 18:52:57 UTC
Set target-milestone of open issue to TBD (was < 7.3, f.e. 6.8), so the issue doesn't get lost.
Comment 20 markiewb 2012-12-17 19:00:12 UTC
Set target-milestone of open issue to TBD (was < 7.3, f.e. 6.8), so the issue
doesn't get lost. This time the target milestone is really set.
Comment 21 Michel Graciano 2013-02-26 14:19:04 UTC
I have this problem too, our project is a huge free form project and it takes too long to find any file. Any chance to have this addressed for next release?
Comment 22 Tomas Zezula 2013-02-26 16:43:20 UTC
The problem of the current GTF implementation is that it waits for the slowest provider before it displays the results. The slowest provider is the file crawling of all folders which are not under source root. Making the dialog to display partial results before the slowest provider completes will improve the search time for files under source roots.
>Any chance to have this addressed for next release?
Hard to say, there are no plans for next release yet.
Can you attach the nps snap shot.
Thanks!
Comment 23 Michel Graciano 2013-02-26 17:54:33 UTC
Created attachment 131909 [details]
NPS file

I hope it helps.
Comment 24 tomzi 2013-03-07 12:49:33 UTC
Can you add an option to 'Goto File' also in the Favourites tab. In this tab usually all the folder for database scripts are mounted, since they are not really (maven, ant) projects.
Comment 25 Michel Graciano 2013-03-08 17:41:59 UTC
(In reply to comment #24)
> Can you add an option to 'Goto File' also in the Favourites tab. In this tab
> usually all the folder for database scripts are mounted, since they are not
> really (maven, ant) projects.

tomzi, can you file a new issue about it? I agree that this option could be helpful, but it is a different problem and should be treated as well. If you don't mind, I can do this for you, just let me know.
Comment 26 Michel Graciano 2013-05-27 19:24:17 UTC
(In reply to comment #22)
> >Any chance to have this addressed for next release?
> Hard to say, there are no plans for next release yet.

Now that we have some plans for the next release, any chance to consider some improvements on this area? IMHO, it is a critical usability problem which could be considered a P2, but since it is just a matter of interpretation, you can disagree with me :)
Comment 27 ceklock 2013-12-19 19:28:28 UTC
Works without slowness for me in a medium-sized maven project.



Product Version: NetBeans IDE Dev (Build 201312190002)
Java: 1.7.0_45; Java HotSpot(TM) 64-Bit Server VM 24.45-b08
Runtime: Java(TM) SE Runtime Environment 1.7.0_45-b18
System: Windows 7 version 6.1 running on amd64; Cp1252; pt_BR (nb)
Comment 28 Michel Graciano 2014-03-11 11:56:38 UTC
Okay, time for some real data. I tested it with the same hardware and project, but in Windows and Linux I had fairly different results. We have a project with 17.619 files where 10.509 are not indexed AFAIK. Our structure is something like that:

project_root
  . other_files (10.509 files which are not mounted as source root in the project)
  . src (default Maven structure with 6.982 files)
  . (here more files like pom.xml etc)

Given this structure, AFAIK 'other_files' folder will not be index (as discussed before in this issue) and is probably the source of the issue because it is not mounted as a source root in the project.

I tested it in my Linux workstations searching for a single js file and it took something like 3 seconds, which is acceptable. But when we tested it in a Windows workstation it took more than 20 seconds, which now is not acceptable. I should highlight that the searched file is inside a folder which is mounted as source root in the project, so it probably is index.

Above is the configuration of the workstations.

Product Version: NetBeans IDE 8.0 RC1 (Build 201402242200)
Java: 1.7.0_51; Java HotSpot(TM) 64-Bit Server VM 24.51-b03
Runtime: Java(TM) SE Runtime Environment 1.7.0_51-b13
System: Windows 7 version 6.1 running on amd64; Cp1252; en_US (nb)

Product Version: NetBeans IDE Dev (Build 20140227-8aabb71e3f6d)
Updates: Updates available
Java: 1.7.0_40; Java HotSpot(TM) 64-Bit Server VM 24.0-b56
Runtime: Java(TM) SE Runtime Environment 1.7.0_40-b43
System: Linux version 3.4.63-2.44-desktop running on amd64; UTF-8; en_US (nb)

I hope this issue can have some attention soon.
Comment 29 Marian Mirilovic 2014-03-13 10:20:38 UTC
20 votes ... at least P2
Comment 30 everflux 2014-03-13 10:40:55 UTC
Could this somehow be related to https://netbeans.org/bugzilla/show_bug.cgi?id=241602 ?


By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo