This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
Summary: | Make projects usable during "project initializing" | ||
---|---|---|---|
Product: | apisupport | Reporter: | _ wadechandler <wadechandler> |
Component: | Project | Assignee: | rmichalsky <rmichalsky> |
Status: | RESOLVED FIXED | ||
Severity: | blocker | CC: | hmichel, jtulach, mkleint, rmichalsky |
Priority: | P2 | ||
Version: | 6.x | ||
Hardware: | All | ||
OS: | All | ||
Issue Type: | DEFECT | Exception Reporter: |
Description
_ wadechandler
2008-12-31 06:23:16 UTC
just a side note: "all the different project types" equals to something like 50 nowadays.. what is the project type in your case that takes so much time to initialize? any netbeans.org's own projects around? The form module (Matisse or NB UI editor) and ireport are NB module projects. iReport is a suite and a set of modules: http://jasperforge.org/scm/?group_id=241 However, this has happened with different web projects as well as they can be quite large. But, for examples you can open the form module project and download and open iReport. For iReport you'll need to get and setup a 6.0.1 platform from the netbeans.org web site. Still, whether 50 or 10 types of projects, currently a single project initializing can suck up the time to load, and these block others from loading. Seems a better threading model is needed for loading projects all the way around. Something with call backs which projects can use between doing different things and while running through their code. Then the system can hand over resources to things needing them as they are needed versus relying on some project type to load as fast as possible. I mean, the project types taking so long needs fixed regardless, any of them, but a better model that can support better resource management would be best to have in the long run. If the project types are using FileSystem and FileObject instead of java.io.File directly something could possibly be done at the base level to block a given module's project loading and move work to other threads to give other projects a chance to load; then as needed resources can be handed out to projects. That way ones the user is trying to use can get most of the resources some how. Essentially a scheduler. I just closed all the ireport things, and now I have a JSE project using JPA, the form module, and a simple JSE project I created to test some simple code. I restarted the IDE. Technically, I closed it, did some disk clean up, and then reopened it, so I don't know what kind of disk caching this would have impacted. Opening projects was taking a while, and the UI was being blocked by something, possibly too much disk and cpu access, when it finally unlocked the form module project was stuck trying to initialize, and this took a good while. Other times though it may be the simple JSE project. I would have to look and see how the opening of projects takes place; how the projects are loaded per their order. Maybe that persistence is not ordered (???) so that sometimes form loads first or last and when loading first it still tells me on the JSE project that the project is initializing? I'll have to try it a few times and see what the form module says when the JSE project has the initializing node under it. If I can get that particular state to repeat itself. It took maybe 5-10 minutes or so to get to "scanning projects". That of course takes a while to start every time I start the IDE with these projects opened. Scanning projects of course blocks code completion, for all projects, but that is a different issue entirely which I have either filed or commented on in another IZ issue; that may be specific to Java projects. Maybe this one is specific to Java projects on a given level, but it really seems a better model is needed to help these things be more centrally managed. Sure, a given module could not use those things and mess things up regardless, unless it could be made to work on the file system level, but that can be documented in the projects support and possibly some analysis checks could be built into the module project support to check those things at development time and guide developers down the right path to protect the environment. Anyways, I'll do some more digging to see what other information I can find and see if I can get some thread dumps too. if you have the form module from hg.netbeans.org open, you effectively load (or at least scan) all netbeans.org projects. That has been an issue since the beginning of time, but always closed as wontfix by apisupport maintainers. suite projects or single modules are behaving better. I'm actually in favour of going in opposite direction and stop showing a list of opened project if they are in effect not loaded. a simple "loading projects" text in the projects view shall do and is more straightforward way of showing what's happening. The list of "fake" projects just give people a false impression that things are already ready to be used, while they are not. just for the record, in some situations the form module does not need to be even opened to trigger the loading of itself and other netbeans.org module projects. Opened projects provide more information about themselves, but for example if a file from form module is opened, the project itself gets also loaded. Even at that if one could work on some other projects while those that take a longer, much longer in this case, time to load are loading it would be bearable. At the moment if a small project which should load fast is not initialized and ready to go/not initialized before the big ones you have to wait on it as well. This is where something could be done in a generic sense at the IDE level I think. Too, during scanning. For instance, code completion is blocked in a totally unrelated project while some other large project is scanning. That makes no sense. Certainly something there needs to be rearranged. Seems project initialization is random on what gets loaded first unless it is alphabetically ordered. I had form opened when I closed the IDE. I also had a couple test projects opened. One was a UML project called UMLProject1 and a Java project called JavaApplication5. The test projects are as simple as they can get. Basically a single class and a single diagram. They were unusable until form usable. The only Nodes they had were Initializing and close I think. So, that is one thing, some how get projects which can be initialized faster initialized first or in between as needed versus a specific order; that needs to be some kind of a generic change to the way things work. RCP projects then need to be more scalable. As far as dependencies go, the form module has 35 dependencies versus all the projects in NB. That isn't really a lot of dependencies. NB being able to scale in the RCP realm is important. RCP is where the Eclipse project has gotten all its partners. This is the same case for NB. Two high profile things lately, "Visual Paradigm" for UML and ICEFaces for Visual Web JSF, point to a simple fact; without partners NB won't be able to keep competing. This has to be on the list of priorities. The way it works now is basically following: 1) Main Project is being loaded first 2) Then projects without any defined order are being opened with exception of project owning document opened in editor, such project is opened as preferred 3) Then any other project touched by user (e.g. expanded in Explorer, document switch in editor) is moved in the list of projects to be open to first position The problem in your case is that one of the first loaded projects (because of document opened in editor) is netbeans module project, and such project loading is simply very costly because it needs to load all its dependencies, which is basically whole IDE. This might be fixed a bit by moving all netbeans project loading to the end of loaded project list by default or by making netbeans projects load faster, e.g. by using cache of loaded projects. Reassigning to apisupport guys to let them express opinion about speeding up apisupport project loading. Overall, I think the single, large issue, is that netbeans is trying to make more information "correct" and "visible" than it is possible for people to consume. In the time that it takes to do all of this, I could, for example, go out and summarily delete, resync to a different revision or do countless other things to the filesystem, again, completely invalidating what netbeans is trying to show me. What matters most to me, as a developer, is that I must be able to click and open anything in the IDE immediately and not be distracted by any gathering task which the IDE is doing. I personally, like to edit and compile in an IDE, and I like to use all the click through features for find references, definitions etc. The IDE wants to have a single copy of things visible, so that seems to enable some very straightforward "find" logic that would not be very involved. What exactly is all of this scanning actually collecting which is needed "all the time". If it's collecting stuff that's needs later on, on only for certain features, than it should be going on in the background, or being triggered at the point those features are used it seems to me. Maybe there could be some elaboration here of all the issues from the nb development teams perspective so that the community could know more about how to vote? Yes and there are different things affected similarly. For instance, code completion. I have 3 or more project opened...pick a number. I have one initialized. Some others are still initializing. I get a message in code completion in files for that project that some scanning is taking place. Seemingly related to those other projects. Seems a general issue overall. This has happened to me before on large EE applications, but nothing as long as the apisupport projects, but long enough to be a pain in the rump. As far as backgrounding goes, it seems things are running in the background, but blocking certain features until something is finished, and the threads are not friendly. They use way too much disk too fast and too much CPU. Seems as it relates to code completion, navigation, etc, that 1) processors, being fair to the CPU and disk and sleeping the threads just a bit here and there, can run in the background adjusting a model. 2) on demand processing can occur in between so that a model which may be filled in the background can be pulled out into an on demand processor which finishes that part as the user needs it. When it finishes with the on demand processing it updates the model the background will be processing, and then the background never has to touch that piece again as the on demand did it for it. 3) Cache those things and only update them when the size of the original or the date has changed, and possibly give the users the option of having the IDE make a CRC or SHA comparison on the files to check for changes. That way the cache can be updated more efficiently. Regarding NB.org module projects: nbbuild cache is used when opening NB.org projects that should be fast enough. I'll take a look if it actually works and doesn't get invalidated too often. Does that mean that if I have built NB from Ant on the command line already that opening should be faster than if I just opened the sources in NB without building? If so, then that doesn't seem to work either. I'll check it out on Linux sometime this week. I need to try out your patch for the new RCP project enhancements as well, so that may be a good time to test it out on my Linux box. "if I have built NB from Ant on the command line already th[en] opening should be faster than if I just opened the sources in NB without building" - yes, this is the idea of the cache. nbbuild/nbproject/private/scan-cache-*.ser should have been created, and should be read by apisupport.project (assuming you have not since invalidated the cache by e.g. updating sources). Should be able to get logging about this with -J-Dorg.netbeans.modules.apisupport.project.universe.ModuleList.level=450 I may have performed an update. I don't know. Either way, I'll try to test it out one day on the weekend, and will use the logging property you have given me. I'll report more then. Thanks all. Nbbuild cache hasn't been used due to mismatched change in org.netbeans.nbbuild.ModuleListParser$Entry. Fixed in core-main, changeset #b44569b64956 Aha. Will such mismatches now be logged so as to be more visible? Would be nice if there were some kind of test to ensure that it was working. You can use NbCollections.checkedMapByFilter rather than @SuppressWarnings("unchecked"). For easier hyperlinking by http://wiki.netbeans.org/BrowserTools#section-BrowserTools-LinkToMercurialChangesets please format the comment as "core-main #b44569b64956" not "core-main, changeset #b44569b64956" Why messages are not internationalized? Only messages displayed in the GUI are I18N'd. > Aha. Will such mismatches now be logged so as to be more visible? Would be nice if there were some kind of test to > ensure that it was working. Cache misses are currently logged on FINE level, will raise to INFO so that it appears in default log. I plan to write test to it when I have more time (hopefully in 2 weeks). > You can use NbCollections.checkedMapByFilter rather than @SuppressWarnings("unchecked"). Will do. > please format the comment as > "core-main #b44569b64956" Sorry, I keep forgetting that. Integrated into 'main-golden', will be available in build *200902120201* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main/rev/b44569b64956 User: Richard Michalsky <rmichalsky@netbeans.org> Log: #156183: fixed using nbbuild cache when opening NBM projects Integrated into 'main-golden', will be available in build *200902130336* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main/rev/3551fc4889bb User: Richard Michalsky <rmichalsky@netbeans.org> Log: #156183: logging, unchecked warning |