This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 156183

Summary: Make projects usable during "project initializing"
Product: apisupport Reporter: _ wadechandler <wadechandler>
Component: ProjectAssignee: rmichalsky <rmichalsky>
Status: RESOLVED FIXED    
Severity: blocker CC: hmichel, jtulach, mkleint, rmichalsky
Priority: P2    
Version: 6.x   
Hardware: All   
OS: All   
Issue Type: DEFECT Exception Reporter:

Description _ wadechandler 2008-12-31 06:23:16 UTC
I have the form module, ireport, and some other Java SE projects opened in the IDE. I restart it. I want to work with
one of the small projects for a starter. Should be no big deal really as there isn't much in it. I can't however because
the projects are being initialized, and apparently the one I want to work with first must wait until all others are
scanned and initialized before I can see the nodes of the project. This is horrible on startup because all one can do is
wait around. 

The IDE needs to be smarter. It should notice I'm trying to use this or that project, switch to initializing it, and try
to get that one going first. That is probably doable, but might take some real coordination with all the different
project types to make the thread loading them sleep or for their logic to save state or something.

Anyways, there is no workaround other than to sit waiting to use the IDE. At this point it is loaded, but is having to
do all this other work before I can use any of the projects. This is why I have set this as a P2. Surely as files which
were opened when the IDE closed are opened again and have their syntax highlighting working and which I can at least
type, although the project loading stuff is eating my disk and CPU which makes it harder (that is another issue too),
then if one could see the file nodes they could be loaded in the editor as well...maybe.
Comment 1 Milos Kleint 2009-01-01 09:43:44 UTC
just a side note: "all the different project types" equals to something like 50 nowadays..

what is the project type in your case that takes so much time to initialize? any netbeans.org's own projects around?
Comment 2 _ wadechandler 2009-01-01 19:01:40 UTC
The form module (Matisse or NB UI editor) and ireport are NB module projects. iReport is a suite and a set of modules:
http://jasperforge.org/scm/?group_id=241

However, this has happened with different web projects as well as they can be quite large. But, for examples you can
open the form module project and download and open iReport. For iReport you'll need to get and setup a 6.0.1 platform
from the netbeans.org web site.

Still, whether 50 or 10 types of projects, currently a single project initializing can suck up the time to load, and
these block others from loading. Seems a better threading model is needed for loading projects all the way around.
Something with call backs which projects can use between doing different things and while running through their code.
Then the system can hand over resources to things needing them as they are needed versus relying on some project type to
load as fast as possible. I mean, the  project types taking so long needs fixed regardless, any of them, but a better
model that can support better resource management would be best to have in the long run.

If the project types are using FileSystem and FileObject instead of java.io.File directly something could possibly be
done at the base level to block a given module's project loading and move work to other threads to give other projects a
chance to load; then as needed resources can be handed out to projects. That way ones the user is trying to use can get
most of the resources some how. Essentially a scheduler.

I just closed all the ireport things, and now I have a JSE project using JPA, the form module, and a simple JSE project
I created to test some simple code. I restarted the IDE. Technically, I closed it, did some disk clean up, and then
reopened it, so I don't know what kind of disk caching this would have impacted. 

Opening projects was taking a while, and the UI was being blocked by something, possibly too much disk and cpu access,
when it finally unlocked the form module project was stuck trying to initialize, and this took a good while. Other times
though it may be the simple JSE project. I would have to look and see how the opening of projects takes place; how the
projects are loaded per their order. Maybe that persistence is not ordered (???) so that sometimes form loads first or
last and when loading first it still tells me on the JSE project that the project is initializing? I'll have to try it a
few times and see what the form module says when the JSE project has the initializing node under it. If I can get that
particular state to repeat itself.

It took maybe 5-10 minutes or so to get to "scanning projects". That of course takes a while to start every time I start
the IDE with these projects opened. Scanning projects of course blocks code completion, for all projects, but that is a
different issue entirely which I have either filed or commented on in another IZ issue; that may be specific to Java
projects. 

Maybe this one is specific to Java projects on a given level, but it really seems a better model is needed to help these
things be more centrally managed. Sure, a given module could not use those things and mess things up regardless, unless
it could be made to work on the file system level, but that can be documented in the projects support and possibly some
analysis checks could be built into the module project support to check those things at development time and guide
developers down the right path to protect the environment.

Anyways, I'll do some more digging to see what other information I can find and see if I can get some thread dumps too.
Comment 3 Milos Kleint 2009-01-02 09:03:45 UTC
if you have the form module from hg.netbeans.org open, you effectively load (or at least scan) all netbeans.org
projects. That has been an issue since the beginning of time, but always closed as wontfix by apisupport maintainers.
suite projects or single modules are behaving better.

I'm actually in favour of going in opposite direction and stop showing a list of opened project if they are in effect
not loaded. a simple "loading projects" text in the projects view shall do and is more straightforward way of showing
what's happening. The list of "fake" projects just give people a false impression that things are already ready to be
used, while they are not.

just for the record, in some situations the form module does not need to be even opened to trigger the loading of itself
and other netbeans.org module projects. Opened projects provide more information about themselves, but for example if a
file from form module is opened, the project itself gets also loaded.
Comment 4 _ wadechandler 2009-01-02 09:34:25 UTC
Even at that if one could work on some other projects while those that take a longer, much longer in this case, time to
load are loading it would be bearable. At the moment if a small project which should load fast is not initialized and
ready to go/not initialized before the big ones you have to wait on it as well. This is where something could be done in
a generic sense at the IDE level I think. Too, during scanning. For instance, code completion is blocked in a totally
unrelated project while some other large project is scanning. That makes no sense. Certainly something there needs to be
rearranged.
Comment 5 _ wadechandler 2009-01-07 04:23:45 UTC
Seems project initialization is random on what gets loaded first unless it is alphabetically ordered. I had form opened
when I closed the IDE. I also had a couple test projects opened. One was a UML project called UMLProject1 and a Java
project called JavaApplication5. The test projects are as simple as they can get. Basically a single class and a single
diagram. They were unusable until form usable. The only Nodes they had were Initializing and close I think. So, that is
one thing, some how get projects which can be initialized faster initialized first or in between as needed versus a
specific order; that needs to be some kind of a generic change to the way things work.

RCP projects then need to be more scalable. As far as dependencies go, the form module has 35 dependencies versus all
the projects in NB. That isn't really a lot of dependencies. NB being able to scale in the RCP realm is important. RCP
is where the Eclipse project has gotten all its partners. This is the same case for NB. Two high profile things lately,
"Visual Paradigm" for UML and ICEFaces for Visual Web JSF, point to a simple fact; without partners NB won't be able to
keep competing. This has to be on the list of priorities.
Comment 6 Milan Kubec 2009-01-07 09:31:09 UTC
The way it works now is basically following:

1) Main Project is being loaded first

2) Then projects without any defined order are being opened with exception of project owning document opened in editor,
such project is opened as preferred

3) Then any other project touched by user (e.g. expanded in Explorer, document switch in editor) is moved in the list of
projects to be open to first position

The problem in your case is that one of the first loaded projects (because of document opened in editor) is netbeans
module project, and such project loading is simply very costly because it needs to load all its dependencies, which is
basically whole IDE. This might be fixed a bit by moving all netbeans project loading to the end of loaded project list
by default or by making netbeans projects load faster, e.g. by using cache of loaded projects.

Reassigning to apisupport guys to let them express opinion about speeding up apisupport project loading.
Comment 7 greggwon 2009-01-07 15:43:20 UTC
Overall, I think the single, large issue, is that netbeans is trying to make more information "correct" and "visible"
than it is possible for people to consume.  In the time that it takes to do all of this, I could, for example, go out
and summarily delete, resync to a different revision or do countless other things to the filesystem, again, completely
invalidating what netbeans is trying to show me.

What matters most to me, as a developer, is that I must be able to click and open anything in the IDE immediately and
not be distracted by any gathering task which the IDE is doing.  

I personally, like to edit and compile in an IDE, and I like to use all the click through features for find references,
definitions etc.  

The IDE wants to have a single copy of things visible, so that seems to enable some very straightforward "find" logic
that would not be very involved.  

What exactly is all of this scanning actually collecting which is needed "all the time".  If it's collecting stuff
that's needs later on, on only for certain features, than it should be going on in the background, or being triggered at
the point those features are used it seems to me.

Maybe there could be some elaboration here of all the issues from the nb development teams perspective so that the
community could know more about how to vote?
Comment 8 _ wadechandler 2009-01-07 15:54:28 UTC
Yes and there are different things affected similarly. For instance, code completion. I have 3 or more project
opened...pick a number. I have one initialized. Some others are still initializing. I get a message in code completion
in files for that project that some scanning is taking place. Seemingly related to those other projects. Seems a general
issue overall. This has happened to me before on large EE applications, but nothing as long as the apisupport projects,
but long enough to be a pain in the rump.

As far as backgrounding goes, it seems things are running in the background, but blocking certain features until
something is finished, and the threads are not friendly. They use way too much disk too fast and too much CPU. 

Seems as it relates to code completion, navigation, etc, that 1) processors, being fair to the CPU and disk and sleeping
the threads just a bit here and there, can run in the background adjusting a model. 2) on demand processing can occur in
between so that a model which may be filled in the background can be pulled out into an on demand processor which
finishes that part as the user needs it. When it finishes with the on demand processing it updates the model the
background will be processing, and then the background never has to touch that piece again as the on demand did it for
it. 3) Cache those things and only update them when the size of the original or the date has changed, and possibly give
the users the option of having the IDE make a CRC or SHA comparison on the files to check for changes. That way the
cache can be updated more efficiently.
Comment 9 rmichalsky 2009-01-08 09:47:06 UTC
Regarding NB.org module projects: nbbuild cache is used when opening NB.org projects that should be fast enough. I'll
take a look if it actually works and doesn't get invalidated too often.
Comment 10 _ wadechandler 2009-01-08 15:30:27 UTC
Does that mean that if I have built NB from Ant on the command line already that opening should be faster than if I just
opened the sources in NB without building? If so, then that doesn't seem to work either. I'll check it out on Linux
sometime this week. I need to try out your patch for the new RCP project enhancements as well, so that may be a good
time to test it out on my Linux box.
Comment 11 Jesse Glick 2009-01-08 17:00:23 UTC
"if I have built NB from Ant on the command line already th[en] opening should be faster than if I just
opened the sources in NB without building" - yes, this is the idea of the cache.
nbbuild/nbproject/private/scan-cache-*.ser should have been created, and should be read by apisupport.project (assuming
you have not since invalidated the cache by e.g. updating sources). Should be able to get logging about this with

-J-Dorg.netbeans.modules.apisupport.project.universe.ModuleList.level=450
Comment 12 _ wadechandler 2009-01-08 23:52:38 UTC
I may have performed an update. I don't know. Either way, I'll try to test it out one day on the weekend, and will use
the logging property you have given me. I'll report more then. Thanks all.
Comment 13 rmichalsky 2009-02-11 10:14:07 UTC
Nbbuild cache hasn't been used due to mismatched change in org.netbeans.nbbuild.ModuleListParser$Entry. Fixed in
core-main, changeset #b44569b64956
Comment 14 Jesse Glick 2009-02-11 15:34:07 UTC
Aha. Will such mismatches now be logged so as to be more visible? Would be nice if there were some kind of test to
ensure that it was working.


You can use NbCollections.checkedMapByFilter rather than @SuppressWarnings("unchecked").


For easier hyperlinking by

http://wiki.netbeans.org/BrowserTools#section-BrowserTools-LinkToMercurialChangesets

please format the comment as

"core-main #b44569b64956"

not

"core-main, changeset #b44569b64956"
Comment 15 Michel Graciano 2009-02-11 15:47:12 UTC
Why messages are not internationalized?
Comment 16 Jesse Glick 2009-02-11 15:54:31 UTC
Only messages displayed in the GUI are I18N'd.
Comment 17 rmichalsky 2009-02-11 16:11:27 UTC
> Aha. Will such mismatches now be logged so as to be more visible? Would be nice if there were some kind of test to
> ensure that it was working.
Cache misses are currently logged on FINE level, will raise to INFO so that it appears in default log. I plan to write
test to it when I have more time (hopefully in 2 weeks).

> You can use NbCollections.checkedMapByFilter rather than @SuppressWarnings("unchecked").
Will do.

> please format the comment as
> "core-main #b44569b64956"
Sorry, I keep forgetting that.
Comment 18 Quality Engineering 2009-02-12 08:21:24 UTC
Integrated into 'main-golden', will be available in build *200902120201* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/b44569b64956
User: Richard Michalsky <rmichalsky@netbeans.org>
Log: #156183: fixed using nbbuild cache when opening NBM projects
Comment 19 Quality Engineering 2009-02-13 09:26:00 UTC
Integrated into 'main-golden', will be available in build *200902130336* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/3551fc4889bb
User: Richard Michalsky <rmichalsky@netbeans.org>
Log: #156183: logging, unchecked warning