This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 81765

Summary: Computing of NB project packages is slow
Product: profiler Reporter: Jesse Glick <jglick>
Component: BaseAssignee: Jiri Sedlacek <jis>
Status: RESOLVED FIXED    
Severity: blocker Keywords: PERFORMANCE, THREAD
Priority: P3    
Version: 6.x   
Hardware: All   
OS: All   
Issue Type: DEFECT Exception Reporter:
Attachments: Thread dumps

Description Jesse Glick 2006-08-03 01:43:01 UTC
I downloaded profiler-60dev-linux.bin from
http://profiler.netbeans.org/download/60qbuild.html and installed in a current
NB dev build. With nb_all/nbbuild open as the main project, I selected Profile
Main Project. The IDE then froze at 100% CPU usage until I killed it. Attaching
thread dumps.
Comment 1 Jesse Glick 2006-08-03 01:43:28 UTC
Created attachment 32473 [details]
Thread dumps
Comment 2 Jesse Glick 2006-08-03 01:55:50 UTC
Profiling a module project directly does not appear to produce this problem.
Comment 3 Jesse Glick 2006-08-03 02:14:20 UTC
Profiling a module project with more deps does produce this problem, but the IDE
unfreezes itself about thirty seconds or so. You should not block EQ
indefinitely like this.
Comment 4 Tomas Hurka 2006-08-03 06:59:45 UTC
According to the thread dumps the real problem is caused by slow project infrastructure. Almost all time 
is spend in org.netbeans.spi.project.support.ant.SourcesHelper$SourcesImpl.getSourceGroups()

FYI: indefinitely != thirty seconds
Comment 5 ehucka 2006-08-03 09:32:04 UTC
But a progress dialog would be much prettier than blocked ui there. :)
Comment 6 Tomas Hurka 2006-08-03 11:58:32 UTC
Sure.
Comment 7 Jesse Glick 2006-08-03 21:53:23 UTC
It's "indefinitely" in the sense that the more projects you have as subprojects
of the selected one, the longer the freeze (as far as I can make out).

BTW do you really need to get source groups for every recursive subproject to
begin with? (Cf. my other bug report re. meaning of "only this project" filter.)

Optimizing SourcesHelper is probably also possible, though I can't get the
profiler to work yet so it will be tough to know which part is slow...
Comment 8 Jesse Glick 2006-08-04 00:02:18 UTC
So I did some performance measurements in a standalone app.

public static void main(String[] args) throws Exception {
    Project p =
ProjectManager.getDefault().findProject(FileUtil.toFileObject(new
File("/space/src/nb_all/nbbuild")));
    Set<Project> shown = new HashSet<Project>();
    long start = System.currentTimeMillis();
    show(p, shown);
    long end = System.currentTimeMillis();
    System.err.println("time: " + (end - start) + "msec");
}
private static void show(Project p, Set<Project> shown) throws Exception {
    if (!shown.add(p)) {
        return;
    }
    for (SourceGroup g :
ProjectUtils.getSources(p).getSourceGroups(JavaProjectConstants.SOURCES_TYPE_JAVA))
{
        System.out.println(FileUtil.toFile(g.getRootFolder()));
    }
    SubprojectProvider spp = p.getLookup().lookup(SubprojectProvider.class);
    if (spp != null) {
        for (Project sub : spp.getSubprojects()) {
            show(sub, shown);
        }
    }
}

prints all Java source roots in the NB standard distro. It runs for me in
between 7 and 11 seconds. Probably could be somewhat faster, but remember that
this is loading cca. 200 module projects including project.properties and
project.xml. The freeze I saw was certainly longer than this, so maybe something
else is going on.
Comment 9 Jiri Sedlacek 2006-08-09 13:20:07 UTC
Please read my comment in Issue 81769. Could you suggest better Profile Only 
Project Classes filter behavior or other improvements that would fit your needs?

Note that profiling all NetBeans modules at once this way is not a good idea in 
general, you should use Part of Application profiling (define explicit root 
methods) and as much restrictive instrumentation filter as possible for 
profiling large applications.
Comment 10 Tomas Hurka 2006-08-11 15:24:38 UTC
Right, profiling nb_all/nbuild as main project is not a good idea and probably something you don't want 
anyway. You should select module you want to profile as main project or select it and use context menu to 
invoke 'Profile Project'. 
Comment 11 Tomas Hurka 2006-08-11 15:28:39 UTC
Computing of java source roots was improved. Lowering priority to P3.
Comment 12 Jesse Glick 2007-06-22 19:05:51 UTC
"you should select module you want to profile" - I have a list of around 10 modules I want to profile. I don't know
which is primarily responsible for the slow behavior that has been reported. (If I knew that much, I probably would not
need the profiler!)
Comment 13 Jesse Glick 2007-06-27 20:14:20 UTC
I have gotten it to work by selecting a couple of root methods in java/project and selecting Profile Project on that;
apparently the profiler will gather timing information from methods in other projects. It is not clear from the UI what
I would do if I had root methods in several projects; would I just pick one of those projects to "profile" arbitrarily?
Does it actually make any difference which project I select to profile, other than that if I select a low-level project
the dialog is responsive whereas if I select nbbuild it is unusable? A bit confusing.
Comment 14 Jiri Sedlacek 2007-06-28 09:05:39 UTC
Once you invoke "Profile Project", all packages of this project and all its subprojects recursively are computed, that's
why it's so slow and the dialog is blocked. This will be fixed in this issue. If you select just java/project, it's
faster than invoking profiling on all NetBeans modules because it contains just a small subset of all NetBeans packages.

You can define any root methods from any opened projects for profiling in context of a project. These root methods are
stored in context of the profiled project - this means that if you want to profile whole NetBeans, you have to define
separate set of root methods for the suite, root method's aren't the sum of all defined root methods from all
subprojects (modules). The same applies for your case.

Root methods define where profiling starts, not where it "ends", that's why you can see data from other modules. This is
what the instrumentation filter does - it defines which classes will be instrumented (== profiled). If you want to
profile just java/project, then invoke profiling of this module and use Profile Only Project Classes filter.

Could you please describe in more details what's not clear from UI about the root methods? There is a list of root
methods that will be used for the profiling session, do you think it's not enough?

If you think that current profiler documentation is not good enough, could you please file an issue about it and list
the areas which should be improved? Thanks.
Comment 15 Jesse Glick 2007-07-10 01:42:36 UTC
To repeat my question of Jun 27, it is not clear to me what I would do if I wanted root methods in, say, both
java/project and openide/masterfs. Which one would I select for profiling? Whichever one I choose, the IDE will start up
with the same arguments, but it is not obvious whether the set of root methods will be affected, assuming I have already
selected root methods using the editor's context menu. The only natural-seeming project to profile is nbbuild, but as I
have found, that is unacceptably slow.

I have no idea what you mean by "root method's aren't the sum of all defined root methods from all subprojects
(modules)" but apparently I am not understanding something.
Comment 16 Jiri Sedlacek 2007-07-10 12:30:37 UTC
For each project a separate set of root methods for profiling is defined. If you are adding root methods from editor, 
the root method is added to the project the current source belongs to. If you want to add root methods from different 
projects, you have to use the Select Root Methods dialog with Show All Projects checkbox selected (Profile Project/CPU 
Part of Application/define.../Add From Project...).

When using CPU Part of Application mode, it doesn't matter which NB module you choose to profile unless you want to use 
Profile Only Project Classes instrumentation filter - that's because the owning suite is always used to start NB.
Comment 17 Jiri Sedlacek 2007-07-12 16:32:21 UTC
Projects, subprojects, source roots and all packages are now processed lazily after selecting profiling task & settings 
in separate thread. New instrumentation filter "Profile Project & Subprojects Classes" introduced.

As for your computation of resolving source roots - the profiler is searching for all packages in all subprojects, 
that's why it takes more time than your example. All possible optimizations have been implemented on profiler side but 
it will always take some time for large projects like NetBeans.

Fixed for Beta1.