This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 114195 - Speed up folder expansion
Summary: Speed up folder expansion
Status: RESOLVED DUPLICATE of bug 121913
Alias: None
Product: platform
Classification: Unclassified
Component: Explorer (show other bugs)
Version: 6.x
Hardware: All All
: P2 blocker (vote)
Assignee: t_h
URL: http://xtest-db.czech/AllInOnePerform...
Keywords: PERFORMANCE
: 115944 (view as bug list)
Depends on: 117278 117279 117280 117288 117292 117319 117426 117781
Blocks:
  Show dependency tree
 
Reported: 2007-08-30 08:59 UTC by Oleg Khokhlov
Modified: 2008-12-22 14:53 UTC (History)
7 users (show)

See Also:
Issue Type: TASK
Exception Reporter:


Attachments
Expand java files graph (22.56 KB, application/octet-stream)
2007-08-30 11:39 UTC, Oleg Khokhlov
Details
Expand jsp files graph (29.10 KB, application/octet-stream)
2007-08-30 11:39 UTC, Oleg Khokhlov
Details
Expand txt files graph (33.63 KB, application/octet-stream)
2007-08-30 11:39 UTC, Oleg Khokhlov
Details
Expand xml files graph (23.59 KB, application/octet-stream)
2007-08-30 11:40 UTC, Oleg Khokhlov
Details
profiler snapshot (617.19 KB, application/octet-stream)
2007-09-03 10:43 UTC, Oleg Khokhlov
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Oleg Khokhlov 2007-08-30 08:59:07 UTC
UI responsiveness automated tests show that expanding folders is out of boundaries:
E.g.
Expand folder with 100 java files:
One of the measuredTime(s) [ 3948 633 241 245 ] > expectedTime[1000] - performance issue (it's ok if the first usage is
in boundary <0,2*expectedTime>)

Expand folder with 100 xml files.
One of the measuredTime(s) [ 2745 259 252 241 ] > expectedTime[1000] - performance issue (it's ok if the first usage is
in boundary <0,2*expectedTime>)

Expand folder with 100 JSP files
One of the measuredTime(s) [ 2977 2820 3525 ] > expectedTime[1000] - performance issue (it's ok if the first usage is in
boundary <0,2*expectedTime>)
Comment 1 David Simonek 2007-08-30 11:09:13 UTC
Question for reporter ovk: Is the change in folder opening speed recent, I mean change is between builds 20070827 and
20070826? Could you attach previous results? (I'm unable to access xtest-db.czech pages, don't know why).
Comment 2 Oleg Khokhlov 2007-08-30 11:39:02 UTC
Created attachment 47800 [details]
Expand java files graph
Comment 3 Oleg Khokhlov 2007-08-30 11:39:28 UTC
Created attachment 47801 [details]
Expand jsp files graph
Comment 4 Oleg Khokhlov 2007-08-30 11:39:48 UTC
Created attachment 47802 [details]
Expand txt files graph
Comment 5 Oleg Khokhlov 2007-08-30 11:40:06 UTC
Created attachment 47803 [details]
Expand xml files graph
Comment 6 Oleg Khokhlov 2007-08-30 11:41:52 UTC
I think it happend earlier. Graphs attached.
Comment 7 David Simonek 2007-08-31 09:13:53 UTC
OK, graphs are interesting but they are quite varied and I can't judge much from them. It's clear that there is some
slowdown, but quite varied across platforms and time and types of opened projects.

Let's try it from another corner - you have and know the sources for your ui responsiveness tests, right? We would need
results from netbeans profiler to know what eats most time and where. Could you profile at least one of the test for us
and attach results? Thanks.
Or give us test code and detailed run instructions to profile on our side...
 
Comment 8 Oleg Khokhlov 2007-09-03 10:42:20 UTC
attaching profiler snapshot with expanding folder with 100 java files.
Hope this helps.
Comment 9 Oleg Khokhlov 2007-09-03 10:43:06 UTC
Created attachment 47960 [details]
profiler snapshot
Comment 10 Milan Kubec 2007-09-03 11:12:56 UTC
From the snapshot I don't think that projects/ui code is involved. Reassigning for evaluation to openide/loaders. Please
evaluate. Thanks.
Comment 11 Jaroslav Tulach 2007-09-06 17:09:17 UTC
The only hotspot I can see from the snaphost is 11% of time being spent in MasterFO.isValid(). Passing to Radek.
Comment 12 rmatous 2007-09-07 11:56:27 UTC
11% gain in FS wouldn't be sufficient enough to bring us nearer to expectedTime[1000]. Moreover those 11% cannot be
solved  by simple isolated fix without impact on quality.

Petr, please look at it as the whole (not just FS part) and put in your opinion. Regression against 5.5 or 5.0?
Comment 13 Petr Nejedly 2007-09-26 15:30:36 UTC
Well, it seems to me this can be related to the integration of CND pack into the official build of the IDE.
This is in line both with:
* my profiling measurements, where nontrivial amount of time is spent inside several CND-provided mime resolvers
* my previous observation that CND's mime resolvers are quite expensive, not fully necessary, and the recognition is
performed twice for each (otherwise unrecognized) file - once during mime recognition from the first mime-aware data
loader, second time directly from particular CND's data loader.

I believe I have reported the CND's problem before, but I can't find the issue.
I'll update this with the CND's issue.
Comment 14 Jaroslav Tulach 2007-09-26 22:41:05 UTC
*** Issue 115944 has been marked as a duplicate of this issue. ***
Comment 15 Petr Nejedly 2007-09-27 13:55:35 UTC
In case of many java files in a folder, another few problems are caused by:
*) J2ME loader, which, in case of _each_ java file, iterates all the parent folders up to the project root, and then
checks whether it is J2ME project type.
*) JsfJavaDataLoader, which seems to do much more complicated logic, but also traverses the folder hierarchy up to root.

Radku, as a small optimization, we could also cache the result of getExt() in CachedFileObject. It gets called many
times during mime recognition and it is quite expensive (it boils down to getting full name somewhere deep and then
stripping the right part).
Comment 16 Petr Nejedly 2007-09-27 17:21:02 UTC
> Issue 115944 has been marked as a duplicate of this issue. 
Actually, 115944 is the DataLoader subset of this problem. Another contributor in the folder expansion slowdown is nodes
layer.
JavaNode ctor seems to be quite slow (66% of node processing), where most of the time goes into the FileBuiltQuery usage
(>90%). The built status computation should IMO be rescheduled and displayed later.
 
Another small problem (from time point of view, can be much bigger problem when looked at from different angles) is that
there was a JavaEditorSupport immediately created for each of the displayed nodes (through Favorites view, can be
different through Projects view) as a result of FilterNode.getCookie(DataObject.class)
Comment 17 Petr Nejedly 2007-09-28 23:31:37 UTC
Another big component, when recognizing bigger folder full of java files is the fact that some sniffing resolvers(namely
CND's executable resolver) is before the java resolver, which means that the infrastructure has to _read_ each file
several times before recognizing it (well, the content is cached, but has to be fetched at least once).
In my profile, it makes 1000ms out of 4000ms for whole recognition of the folder.

Reorder of the mime resolvers might improve the situation, but there are currently at least two fixed resolvers - ruby
and shell (META-INF registered, coded, not declared, it seems).
Comment 18 Petr Nejedly 2007-10-01 23:06:06 UTC
Turning this umbrella issue into a TASK.
Comment 19 Lukas Hasik 2008-04-10 21:16:12 UTC
moving opened issues from TM <= 6.1 to TM=Dev
Comment 20 t_h 2008-10-20 09:39:18 UTC
http://wiki.netbeans.org/FitnessViaLaziness

*** This issue has been marked as a duplicate of 121913 ***