This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 168306 - Fiximports blocks while scanning is occuring
Summary: Fiximports blocks while scanning is occuring
Status: REOPENED
Alias: None
Product: java
Classification: Unclassified
Component: Editor (show other bugs)
Version: 6.x
Hardware: All All
: P2 blocker with 14 votes (vote)
Assignee: Dusan Balek
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2009-07-08 22:58 UTC by greggwon
Modified: 2013-02-09 16:44 UTC (History)
4 users (show)

See Also:
Issue Type: ENHANCEMENT
Exception Reporter:


Attachments
Thread dump of Fix-imports blocking EDT while scanning is occuring (34.26 KB, text/plain)
2009-07-08 22:59 UTC, greggwon
Details
Stack trace from netbeans startup with similar block of opening projects on simply using the IDE for a new, standalone application. (20.88 KB, text/plain)
2013-01-14 20:17 UTC, greggwon
Details

Note You need to log in before you can comment on or make changes to this bug.
Description greggwon 2009-07-08 22:58:31 UTC
If indexing/compiling is happening in the background fix-imports seems to hang the EDT while looking for class
information.  It should just use what is available, and it could even hint that it couldn't get up to date information,
and provide the user with the ability to say just use what you have, or I'll wait for you to finish, or cancel.

I've attached a stack trace to show where the EDT is stuck at.
Comment 1 greggwon 2009-07-08 22:59:32 UTC
Created attachment 84514 [details]
Thread dump of Fix-imports blocking EDT while scanning is occuring
Comment 2 Max Sauer 2009-07-10 12:59:31 UTC
Please tell us your build number, some fixes has been made in this area recently, thanks.
Comment 3 Peter Pis 2009-08-19 18:31:12 UTC
No additional information has been provided, closing as WORKSFORME. In a case you will encounter this issue, please feel free to reopen it.
Comment 4 greggwon 2009-08-20 18:54:51 UTC
Which fix is associated with making sure that under all circumstances, fix imports runs immediately on user request?  I
have seen this happening in 6.7.1 production downloaded from the web this week.
Comment 5 Max Sauer 2009-08-21 08:40:51 UTC
Currently, fix imports gets into queue and waits for the scan to finish, in order not to block AWT. The fixes that guarantee this are:

http://hg.netbeans.org/jet-main/rev/d3e6ac8495bc
http://hg.netbeans.org/jet-main/rev/105223f02b54

Under current infrastructure, this is the only possible option (no use-what-you-have, sorry). We can track this as an ENHANCEMENT for the future.
Comment 6 greggwon 2009-08-26 00:32:06 UTC
When I ask for fiximports to be done, it's because I am editing a file.  The chance that I have edited/touched another
file of interest (where the import would come from) in the recent past, is going to be slim to none.  What this means is
that it really doesn't make sense for fiximports to "wait for the cache" to be "up to date".  I'm going to be looking to
import stuff that should be fairly static and unchanging.  The worse thing is that in this case, I wanted to import
something from the JSE platform, where nothing is changing, ever in my environment.

It really has to be possible to use the "old version" of the cache.  I think many of us just don't understand why this
isn't a 3 line change to add

    if( cacheUpdateInProgress ) {
         oldCache.lookup(...)
    } else {
         currentCache.lookup(...)
    }

Maybe it's just hard and we don't understand the complexities?  The concurrency control, right now, seems to be
synchronized() use of Object.wait().  There is a great pattern for updating largish global data structures that
completely eliminates concurrent access issues by just initializing a new "copy" into a local variable and then
assigning that to the global reference atomically.

Cache current;
public void updateCache() {
    Cache newCache = new Cache( current );
    // newCache can read from current just like everyone
    // else because its not mutating
    newCache.checkForUpdates();
    synchronized(this) {
        current = newCache;
    }
}

public Data lookup( Parameters p ) {
    return current.lookup( p );
}

this way the visible cache is never mutating and never needs to be locked.  It can always be read, and only when the new
updated cache is ready, does it become visible for use.

What is the motivation behind refusing to provide this kind of facility?
Comment 7 Jan Lahoda 2009-08-26 12:16:26 UTC
First of all, thanks for your proposal, and your patch fixing this problem is very welcome!

A few things from the top of my head to be aware of while implementing this:
-there are currently two modules: parsing.api and java.source. Most of the threading is in parsing.api, but some changes
will most likely be needed in java.source too.
-beware of memory issues - the parse tree can be quite big, and if parsing for editor and indexing will run
concurrently, it will be necessary to guard more carefully against OOMEs.
-there is no in-memory cache, the cache is only on disk. The Java indexing deletes some of the obsolete data of this
cache before indexing. This needs to be changed, but as a consequence, it will be necessary to introduce filtering in
the JavaFileManagers (see JSR199).
-it will probably be necessary to adjust the features (like Fix Imports) to include some warning about not up-to-date
caches.

Please let me know if you have any questions. Thanks for your contribution!
Comment 8 greggwon 2009-12-14 09:56:33 UTC
I am raising this up to P1 because it really does keep me from using netbeans.  This is a very, very important issue it seems to me with all kinds of recurring themes of appearance.  It happens to everyone using the IDE at some point.  The continued insistence that scanning can and should be a gate into the meta-data is just not realistic.  Sure you want to make sure you provide the right answer to the developer.  

The javac compiler always gets it right.  Every time I use it, it tells me exactly what is wrong, and I can go fix it.  It is successful because while I am compiling, the compiler sees the files as they exist at a particular moment.  If I see a build error, I can go back and edit that file, save it, and then recompile again.  Javac never keeps me from editing and I can always make progress and get work done.

Netbeans is trying to view the project meta-data as an atomic view, continuously.  It is not atomically consistent, ever.  It is continuously mutating, and netbeans does not have access to all the gates that control access to the mutation, and thus it will always be playing catchup and there are not computing resources at its disposal, to ever succeed.

The complexity of what you are trying to achieve is out stripping your ability to construct solutions to all the issues.

It is time to reconsider how this is working, and really make an effort to eliminate the meta-data as a gating mechanism for editing and development work.

I am using 6.8 today, and the whole time I went looking for this issue (I almost opened a new one, but my browser field entry history showed me that I had already opened an issue with this title), have been entering this text, and thinking about how to say something constructive, scanning has continued and I've been unable to do a "fix imports", and not only that, but added Serializable as an implemented interface, and am hoping to get a refactoring hint to add serialVersionUID as a field, but that won't happen till the scanning finishes either.

There are just too, too many gates that scanning is closing right now.  It must be rethought and there must be a "last view" and "current view" mechanism that allows you to create more up-to-date information without blocking tasks that don't need that information.  

Adding a "reason" mechanism into the mix which would detail what file change or other event triggered the scan might help you make better use of new scanning to say, we are interested in "type XX" or "file yy" and seeing that it is being scanned, you can add a listener to that "Future" result, and wait for just that to be available.  Or, use the old data, provide a hint that it is old information, and when the new information becomes available, use that, or instruct the user that new information might change what they want.
Comment 9 David Strupl 2009-12-14 10:33:21 UTC
This is ENH request IMHO. And it is not P1 for us (sorry about that).

You are right that it is not that easy as it might sound. The compile on save feature makes it rather tricky to have some "old" data stored somewhere. Please note that (as Jan Lahoda mentioned) the data are large set of files on disk currently.
Comment 10 twolf2919 2009-12-14 15:20:43 UTC
While from a technical perspective, this may not be a P1, it should be of prime importance from a usability perspective.  I'm guessing that Gregg just marked it as a P1 to draw attention to it.  Same thing with "bug" vs. "enhancement".  To me it's actually a bug - since this problem didn't used to exist before 6.x.

I suppose in the end, it doesn't matter whether it's a P1 or whatever - hopefully NB developers will look it based on the number of votes it has.

Personally, in almost 10 years of using NB, I've never had as many problems with NB getting in my way as now with 6.7 & 6.8.  Sure, I can still be productive with it, but tools are supposed to be come better, not worse, over time.  This is definitely an important area where NB has gotten worse.

We all know that NB doesn't have as many resources as it once did.  But that's why it's doubly important to try to address this as soon as possible - if you don't, you're going to spend more and more time evaluating bug reports on the "user experience" side that all have their root in this problem.

Just my $.02
tom
Comment 11 jessholle 2009-12-14 15:34:23 UTC
I added my vote to this issue, but the problem is *far* more general.

Attach Debugger is blocked until scanning is complete!?!

So is Ctrl-[ navigation to a balancing tag in a JSP.

So is....

The list goes on and on.  This list should be *empty* -- period.  One could argue that refactoring during a scan should be disallowed, but frankly I'd rather get a warning and be allowed to proceed.

Furthermore, at no point during re-scanning should one be left with navigation, refactoring, etc, having access to either the old or new scan data for a given class being re-scanned!  There should be atomic updates as new scan data becomes available.

Until (1) nothing is blocked by scanning and (2) scan updates are atomic with zero regression in available data throughout re-scanning a repository, NetBeans is going to continue to suffer horribly in terms of real world usability for large, complex projects.  If the NetBeans team does not make this their top priority soon I think it'll just be time to give up on NetBeans.
Comment 12 jessholle 2009-12-14 15:48:47 UTC
I meant to say "should one be left without..." in my last comment, i.e. one should always have either the new or old data available at any point during a re-scan of existing source and/or class/jar repositories.  Navigation, etc, should never suddenly be unaware of a class that is there the whole time.
Comment 13 greggwon 2013-01-14 20:17:08 UTC
Created attachment 130208 [details]
Stack trace from netbeans startup with similar block of opening projects on simply using the IDE for a new, standalone application.

Here is the blockage which happens when I startup the IDE.  I was able to create a new project (Java application) and typed into the *App.java created file, some code.  I then tried to "fix imports" after adding a line which was "Socket s = new Socket("some.host", port);".  It beeped and said no imports to fix.  I then selected run file, and it paused as shown in this back-trace.  I let it sit for the next 10 minutes while it finished opening all of the projects (I don't close projects and I don't have project groups, I have everything I need to use day-in and day-out visible in the projects pane).   When it finished opening all projects, it ran the file and then reported a compile exception 'cannot find symbol".

At this point, in the great timeline of this problem, this is just absolutely ridiculous to still be an issue.  There is nothing modular or attractive about the netbeans platform, if something like this can be so pervasive, so abusive to the user experience, and not fixable in this amount of time.

I truely hate to just get on the bug list and beat around about this, but the frustration level is so high and the change so small compared to my expectations for how import to address this issue is, that I just have a difficult time not lashing out, trying to push the right buttons to get this addressed.  There are so many other great things about netbeans, and it could be such a wonderful demonstration of Java technology.  But, clearly, you guys are having to work to much on other things, or aren't getting the right support or something else is keeping real world, user productivity from being addressed.