Bug 194147

Most of what you are describing is not a bug, but feature request. If you want to have it, you have to write it yourself.

However if you want to disable the native listeners completely, there is already a way: ergonomics#6e636dc627ec

Focus stealing is a bug.  And AFAIK there is no way to write such a feature using existing APIs.

Focus stealing is indeed a bug. But not a bug in filesystems. Please reassign the issue to the part of the system that steals the focus.

Focus stealing is not a bug in filesystems. Until properly reassigned, closing as invalid.

> Until properly reassigned

== never?

The simple fix for focus stealing is:  Don't respond to file events while the main window is inactive, just collect a list of them to respond to on reactivation.

If you don't want to report individual bugs against focus stealing, then probably never.

You suggestion to defer events is an enhancement from the point of view of filesystems and there is nobody who will implement it any time soon.

Agreed with Tim that update-on-activation would be the better behavior. It's not just about stealing focus; the IDE should not be doing work (e.g. parsing files) that might be invalidated by another file change before the user switches back. Queueing up events until the MW gets back focus would make more sense for most purposes.

"either we trust file change reporting from the OS and don't need to rescan; or we don't trust it and use the old behavior" - see bug #190462 comment #16.

> that might be invalidated by another file change before the user switches back

Indeed - in fact, this is an extremely common occurrence if you use Git:
 - Switch to shell
 - Change branch from 'work' to 'master' (all local changes disappear)
 - Merge changes from 'work' (all local changes reappear)
 - Push to remote repo
 - Change branch back to 'work' (if you have Git support installed, triggers more work)

This sequence of work is truly annoying if, after each of the above shell commands, NetBeans suddenly fronts itself, which is often the case.

Report bug against the piece that fronts itself, but stop complaining here. I am going close this enhancement soon, unless an enhancement patch is provided.

Those who are not satisfied with current native fs watches, I have repeat: use what has been described at http://hg.netbeans.org/ergonomics/rev/6e636dc627ec

(In reply to comment #8)
>  - Change branch from 'work' to 'master' (all local changes disappear)
>  - Merge changes from 'work' (all local changes reappear)

Also very common when using Mercurial. Just running 'hg fetch' when you have both outgoing and incoming changes will result in files in your recently modified project being switched to another version and then back a few seconds later.

> This sequence of work is truly annoying if, after each of the above shell
> commands, NetBeans suddenly fronts itself

Or initiates a rescan of sources, even if it does not front itself.

(In reply to comment #9)
> Those who are not satisfied with current native fs watches [disable them]

That would leave us as badly off as in 6.9. The point is to make some minor adjustments to the behavior of native FS watches so that everything works smoothly in 7.0. If you don't have the time to work on it right now that's fine, but I think the issue itself should not be controversial.

The proposed change is not a fix, but a workaround. Rather, the versioncontrol system and parsing infrastructure shall be aware that there are branches which get switched from time to time. We touched the topic with Tomáš briefly about three months ago, but nothing like that is going to happen for 7.0.

The proposed change is indeed a fix, and preferable to the one you're proposing, as follows:
 - The IDE should really not be doing work when it is not in use by the user
 - Reparsing files that are going to change in a few seconds makes no sense


If you force everyone else to write their own code to notice that the IDE is inactive and stop paying attention to changes (which, being the simplest way to handle this case, is probably exactly how such a fix will be implemented), the only thing you'll guarantee is many slightly different variations of the same code, some of which work and some of which don't - not to mention forcing anyone who wants to do it right to depend on openide.windows.

Wouldn't a centralized fix make more sense?  I'd be willing to bet it could be implemented in 10 minutes by someone familiar with the new native listening code.

The Tim's proposal should be easy just waiting the dispatch thread when IDE has no focus. But there is a problem that the queue may grow very large when IDE is inactive for long time. While Mac OS X core services can solve this problem there is no simple solution on other operating systems.

Integrated into 'main-golden', will be available in build *201101140000* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/6e636dc627ec
User: Jaroslav Tulach <jtulach@netbeans.org>
Log: #194147: Documenting way to disable native file changes listening

Nice that the property is documented, although I wouldn't call it a fix.

IMO, there is a missing API for application active/inactive state - I've had to write code that checks this in my own modules (I forget exactly where, probably somewhere in javacard), which is annoying because it forces anything that wants to know this to depend on the window system (or guess that Frame.getFrames()[0] is the main window).

There is also a missing API for "application is shutting down" (and observably, all of the ways to run something after startup are another similar state).  A truly nice fix for this would unify starting/started/shutting-down and active/inactive into a simple API for that, and then use that to affect behavior.

I don't plan any work on this. Whoever wants to donate a patch (and possibly pass API review), feel free to reopen and assign to yourself.

Created attachment 106225 [details]
API for application state change notification added to openide.windows

I'm attaching the API I would want for tracking activated state changes, and an implementation (all in openide.windows - the implementation doesn't really need much except RequestProcessor).

The next step is making masterfs use it.

What this does:
 - Adds org.openide.windows.ApplicationActivationManager, a listener interface and an enum of possible application states
 - When adding a listener, the caller specifies whether it should be notified in the event thread or on a background thread, and the set of states it is interested in
 - The application only becomes "active" when it has received focus and not lost focus again for some period of time

You'll notice (and no doubt complain) that this also supports an "idle" state (application has focus, but no keyboard or mouse input for 2 minutes).  Where this would be very useful:  The Maven module's remote repository indexing routinely brings my machine to its knees once a week - anything like that - large amounts of remote I/O - could be more usefully done when the application knows the user really is not doing anything (not to mention, stop or postpone such work if the application leaves the IDLE state).

Created attachment 106226 [details]
Updated patch

Moved new API to openide.util to avoid creating a dependency from masterfs -> openide.windows - it has no external dependencies other than Lookup and RequestProcessor anyway and can work in any Swing application.

Patched Watcher in masterfs to use it.  Tried it against a large git checkout, opening all projects, then switching to a branch in which everything is deleted and back - no scanning and no focus stealing.  Would have also patched maven to use IDLE, but could not find the code that schedules repository updating.

[JG01] /** for subclasses */ protected void ApplicationActivationManager() {}


[JG02] @since


[JG03] What is the purpose of defining this as a service, when the only expected impl is in the same module? Probably harmless, just wondering.


[JG04] The isActive/Idle convenience methods could likely be deleted.


[JG05] The use of Set<State> is confusing. Presumably you cannot be BECOMING_ACTIVE and BECOMING_INACTIVE at the same time! Would be clearer if all possible state combinations were enumerated directly: inactive, becoming active, becoming inactive, idle?


[JG06] Need dep incr in masterfs/nbproject/project.xml.


(In reply to comment #18)
> Would have also patched maven to
> use IDLE, but could not find the code that schedules repository updating.

maven/src/org/netbeans/modules/maven/ProjectOpenedHookImpl.java

> [JG03] What is the purpose of defining this as a service, when the only
> expected impl is in the same module? Probably harmless, just wondering.

Originally expected the implementation to need a dependency on the window system - but it turned out that KeyboardFocusManager does enough.  Longer-term, though, there are other similar lifecycle state changes in NetBeans which are addressed through various hacks and dissimilar APIs, such as:
 - Main window first shown (warm up tasks, etc.) - how you do this sort of thing is pretty non-obvious, and the need for it is clear
 - Main window shown and warm up complete (i.e. do background work for one-time installation of things that cannot be done declaratively - such as the way the Java Card RI bundle registers RI as a platform)
 - Shutting down - there are still various issues with things attempting to parse files during shutdown - there should be a way to detect this state at low cost and abort or not queue a Task or similar

So I'd anticipate adding enum constants to State, and the implementation eventually needing to live somewhere else to satisfy these dependencies.

> [JG04] The isActive/Idle convenience methods could likely be deleted.

Probably.  I could imagine wanting a lockless isActive check, for which you'd want a volatile boolean.  If you wanted to quickly add active checks to an existing codebase, this might be less painful.

> [JG05] The use of Set<State> is confusing. Presumably you cannot be
> BECOMING_ACTIVE and BECOMING_INACTIVE at the same time! Would be clearer if all
> possible state combinations were enumerated directly: inactive, becoming
> active, becoming inactive, idle?

Agreed, but it opens itself to a combinatoric explosion if more constants are added.  BECOMING_ACTIVE (perhaps it could have a better name) really means "the application received focus", and is different from using a listener directly only in that the thread the event will be delivered on is specified.

If this is a huge issue, I would just make it opaque:
public final class ApplicationState {
  private final Set<State> state;
  ...
  public boolean isActive() {  return state.contains(ACTIVE); }
}
and return that from the public getState() method.

But I suspect that Set<State> does a better job of communicating that states exist in combination with each other and that it's worth thinking about whether you want to do something any time state.contains(ACTIVE) versus only when state.equals(EnumSet.of(ACTIVE)).  Documenting what states are mutually exclusive is probably enough (in practice, you won't ever see BECOMING_ACTIVE and BECOMING_INACTIVE together unless a window can manage to be focused and not focused at the same time).

Y01 I don't like Listener nested class. Consider using PropertyChangeListener, as getState is getter anyway. The fact that it is hold weakly is not proper according to JavaBean specification, but probably correct. 

Y02 I don't understand why some methods are static and why some are instance one. Probably make all of them instance one (and final).

Y03 There shall be no state change on dialogs being open. If you can't do that in openide.util, move proper implementation into core.windows, there you'll be able to implement that properly. 

Y04 I see no unit tests (and I don't mean clickable unreliable ones, but rather those that check behavior of runWhenNextActive, addListener, etc.

Y05 Now I think that runWhenNextActive & addListener could be merged together, if you create public void runWhenState(State state, Runnable run)....

PS: I still believe masterfs shall not use this at all, and react properly to all changes.

> Y01 I don't like Listener nested class. Consider using PropertyChangeListener,

If the JDK adds generics to PropertyChangeListener I'd consider it.

> Y03 There shall be no state change on dialogs being open. If you can't do that
> in openide.util, move proper implementation into core.windows, there you'll be
> able to implement that properly. 

Actually, that's fairly unlikely.  This has been an invisible issue for years - something we do internally with focus event processing causes NetBeans to behave differently than a normal Swing application.  In a plain Swing app, moving focus from a frame to a child dialog and back causes a two focus events - frame -> dialog -> frame.  The same thing, in NetBeans, will generate five:  frame -> null -> dialog -> null -> frame.  I have looked for the culprit a few times, with no luck.

> Y04 I see no unit tests (and I don't mean clickable unreliable ones, but rather
> those that check behavior of runWhenNextActive, addListener, etc.

Hey, I have to leave something for you guys :-)

> Y05 Now I think that runWhenNextActive & addListener could be merged together,
> if you create public void runWhenState(State state, Runnable run)....

Possibly.  If you want to go that route, why not just put Listener instances in the default lookup?

> PS: I still believe masterfs shall not use this at all, and react properly to
> all changes.

Why?

The original problem of focus stealing is reported as bug 197349. I will keep that one as a bug and I am changing this issue to represent one possible enhancement that might address bug 197349 as well.

No.  A desktop application doing work when it is idle and the user did not initiate the activity is a bug.  A desktop application should do nothing when it is idle.

The current behavior *is* correct for an OS-level service, or some low-priority thing like a tray applet which monitors a directory or other file-like thing (proc fs, whatever).

Both behaviors are right for some kinds of applications;  the current behavior is not right for an IDE.


Furthermore, I don't think we will *ever* reach a state where there are no bugs related to spurious file disappearance and reappearance due to, for example, changing branches in version control.  In my experiments I got quite a few exceptions and strange behavior while making the current set of open projects disappear and reappear, with and without giving focus back to NetBeans.  

I do not think every case where every project type assumes that project.xml or project.properties or their parent dirs must exist if the project exists will be found;  the same for various editors for different file types;  and since File.exists() is a bottleneck on Windows, it is not even desirable to put that kind of test (FileObject.isValid() or whatever) in every single piece of code which interacts with any file - to make that provably foolproof, you would need to add that test before every method call on FileObject throughout the codebase, or make sure that every block of code which touches a FileObject catches FileStateInvalidException.  That is the only way you would get to a state where nothing *can* fail, no matter how radically the disk contents change under the running IDE.  And without that, change monitoring will be an endless source of very hard-to-reproduce bugs.

At least with blocking notifications when the IDE is idle, we can rely on the user to put the filesystem back into a sane state before returning to the IDE.

Agreed with Tim; focus stealing is just one symptom. I described another symptom already in comment #10.

Badly burned by this today. Had two copies of the IDE open, one with a few projects open, the other with many files open from many projects and in the process of scanning; with an MQ patch applied. Went to command line to refresh patch and noticed that I had an unadded file from an unrelated commit ('?' status). So I needed to hg qref the patch, hg qpop it, hg qadd the forgotten file, hg fetch to merge with remote changes, hg push the add, then hg qpush to go back to my patch. Obviously the qpop, fetch, and qpush all involve changing a number of source files, some of which are in the transitive dependency closure of open projects/files; so both IDEs went crazy rescanning everything after each command, even though I was working from a shell without either IDE visible (i.e. comment #11 misses the mark). The whole system basically locked up for several minutes.

Workaround: killall -STOP java before doing any CLI operations, and killall -CONT java afterwards.

*** Bug 197349 has been marked as a duplicate of this bug. ***

1. The here-in attached patch represents an enhancement, not a bug.

(In reply to comment #26)
> Badly burned by this today. Went to command line to hg qref the patch, hg qpop it, hg qadd, so IDEs went crazy rescanning everything

2. True, I am running into this problem often too, especially when it is close to a release. "hg up release71_base; hg ci; hg pull; hg up -C default; hg merge; hg ci" just makes the IDE completely mad. The problem should be reported as separate issue, as the here-in provided enhancement is just one way to workaround it. 

3. The problem with parsing API and its interaction with Hg tags should be analyzed by an UI expert. Basing a "fix" on focus is inappropriate as it is quite easy to accidentally switch to the IDE (especially when using Alt-Tab) and run into the problems anyway.

> Workaround: killall -STOP java before doing any CLI operations, and killall
> -CONT java afterwards.

4. If we get serious about fixing the Hg related problems, then we should come up with a robust solution, not a hack based on patch provided here-in. Rather than that use the STOP workaround as it at least does not require API changes which I am not comfortable with.

(In reply to comment #24)
> No.  A desktop application doing work when it is idle and the user did not
> initiate the activity is a bug.  

I doubt anybody can claim this is a general rule. My Amarok refreshes collection of mp3s as soon as I copy them into the collection folders. All file explorers I know (konqueror, dolphin) redraw content of a displayed folder as soon as I change it.

> 4. If we get serious about fixing the Hg related problems, 

For f**k's sake, this is not an Hg-related problem, it is an any-other-program-on-the-user's-computer-which-modifies-files-NB-cares-about problem.  Git has the same problem, and absolutely anything which modifies, or worse, deletes and replaces, files NetBeans has open will trigger this completely avoidable problem.

> then we should come up with a robust solution, not a hack based on patch provided
> here-in

What on earth does "robust" mean in this context?

In what way is detecting a change in the state of the application in the context of the desktop environment a hack?

>> No.  A desktop application doing work when it is idle and the user did not
>> initiate the activity is a bug.  
> I doubt anybody can claim this is a general rule. My Amarok refreshes
> collection of mp3s as soon as I copy them into the collection folders. All file
> explorers I know (konqueror, dolphin) redraw content of a displayed folder as
> soon as I change it.

And there is the mistake:  NetBeans is not a file explorer, it is an IDE - something which is much more complex, uses far more system resources, and does much lot more with files than display icons and launch external processes with selected files as arguments.  They do not have editors which open files.  They are not applications which, in order to operate normally, need to assume that a file disappearing or changing underneath them is an exceptional situation.

It is not that Amarok, Konqueror and Dolphin do not have this problem;  it is that they haven't discovered they have it.  Since they are simpler, it would probably take more dramatic filesystem changes to cause them to make the user's machine unusable due to background activity in response to those changes.

We simply are in the position of needing to define the correct behavior of a desktop application with this problem.  It's pretty unlikely that the correct behavior is "let the user suffer" simply because we don't like the fact that there is a problem.

In reply to comment #28.
> The problem with parsing API and its interaction with Hg tags should be analyzed by an UI expert.
It's not hg specific problem. In parsing.api we have no special handling of  events from external processes, we depend on FileSystem events. We do some things to improve the performance by handling events from single atomic action by one transaction with correct ordering of event types.
We also have a special contract with VCS module to suspend listening (events are buffered) when VCS operation is running (VCS operation within the IDE, we don't know about the external VCS).
Yes, we can do the same (buffer the events) when the IDE has no focus. But it resolves a problem on the side of the single client and other clients will have the same problem. Ideally it should be done in Notifier class to prevent wasting of CPU caches of the foreground app.

(In reply to comment #31)
> In reply to comment #28.
> > The problem with parsing API and its interaction with Hg tags should be analyzed by an UI expert.
> In parsing.api we have no special handling of 
> events from external processes, we depend on FileSystem events. We do some
> things to improve the performance by handling events from single atomic action
> by one transaction with correct ordering of event types.
> We also have a special contract with VCS module to suspend listening (events
> are buffered) when VCS operation is running 

What you are describing is a problem and yes, it is a VCS related bug. parsing.api should not start reparsing when there is an VCS operation in progress. Right now this works for operations done from inside of the IDE. It does not work for operation performed externally (which I believe is a bug; fixable in Hg case at least) and if VCS guys agree, we should report it as separate issue.

But that is not the usecase from comment #26 and #26@2. We would like to perform sequence of VCS operations without the IDE reacting. That is the case where I am calling for an an advice of a HIE.

> It is not that Amarok, Konqueror and Dolphin do not have this problem;

Actually I think these applications don't have a problem. Redrawing UI to be up-to-date when being visible (and not focused) is completely OK. Should at the same time the application steal focus, it would be immediately discovered as a problem. That is why I am calling for closing the file, but not stealing focus in bug 197349.

However the usecase in comment #26 relates to much bigger activity than redrawing. It talks about complete reconstruction of internal database that may take minutes...

> We simply are in the position of needing to define the correct behavior of a
> desktop application with this problem.  It's pretty unlikely that the correct
> behavior is "let the user suffer" simply because we don't like the fact that
> there is a problem.

...performing such heavy activity on background (against user will) is not acceptable. I just don't believe focus gained is universtally agreed UI gesture when such operation should start (automatically).

It is not just parsing API - projects API is affected as well. Develop a new
project on a branch;  switch to the default branch (new project disappears); 
merge the branch.  This can cause all sorts of mayhem in the IDE.

> We would like to perform sequence of VCS operations without the IDE reacting. That is
> the case where I am calling for an an advice of a HIE.

If you can, with or without an HIE, figure out a way to detect whether a
sequence of external VCS operations in progress - i.e. the IDE is predicting whether the
user is going to type another VCS command in an external shell, I will be
pretty impressed - I look forward to the Psychic API.

You have exactly one bit of information to decide if the user could be
performing external VCS operations:  the fact that the IDE does not have focus.
 And in fact, there is exactly one way to do it:  Don't respond to changes
until the IDE regains focus.  If the user didn't perform filesystem-affecting
operations, no harm is done.  If the user did perform such operations, the IDE
only reacts to the filesystem state after all of them have been performed.

(In reply to comment #34)

> You have exactly one bit of information to decide if the user could be
> performing external VCS operations:  the fact that the IDE does not have focus.

This won't do for two reasons:

- we now have terminasl running inside the IDE and users can initiate 
  massive filesystem operations while IDE has focus.

- focus-follows-mouse. It's still popular with unix people. I had filed a bug
  where everytime I moved the mouse the IDE would wake up and do file scans
  and all kinds of expensive stuff.

One obvious question I haven't seen discussed ... can the external file
events be batched in order to avoid being nickel and dimed?
This would require a hysteresis on reaction time.

(In reply to comment #35)
> - we now have terminal running inside the IDE and users can initiate 
>   massive filesystem operations while IDE has focus.

True, though this is a far less common use case. Could in principle treat focus in the terminal TopComponent like a loss of focus from the main window for purposes of file event delivery.

> - focus-follows-mouse. It's still popular with unix people. I had filed a bug
>   where everytime I moved the mouse the IDE would wake up

This would only apply if you moved the mouse _over_ the IDE, and again this is a much less common use case.

> can the external file
> events be batched in order to avoid being nickel and dimed?
> This would require a hysteresis on reaction time.

You could indeed batch them and try to delay delivery if there have been recent events. The problem is that the IDE's reaction to file events can be nonlinear - a modification to a single source file at the bottom of your dependency graph can trigger a massive rescan - so even if you successfully figure out that you are in the middle of an event storm and try to delay further deliveries, it may be too late. Anyway this is useless for cases where you make one set of external changes, wait a minute, and then make another set, without using the IDE in between.

I think that the intensity of the comments on this issue demonstrate the wide range of "expectations" and "desires" that everyone has, regarding what the IDE should and should not do for them.

I still believe that the most reasonable behavior would work as follows.

1.  Anytime that a file changes outside of the editor, the IDE should remember
    it, and at most, put a ? annotation on the navigator/project view entry(s).
2.  Anytime that the editor "reveals" the contents of such a file, only then,
    should the IDE "process" that files changed state.
3.  If there are known dependencies between that file, and anything else in the
    "set" of changed files, then the IDE might "process" just that next layer
    to see if there is an "interface" issue or some other "syntax" or "semantic"
    error to report.  But, I am not really sure that this matters.

With this limited behavior, developers get feedback on what they are working on now.  For me, that's all my brain needs to deal with.

Is there a survey somewhere, which shows that "most" developers just look at the project tree display to decide where to fix errors?  From a point of efficiency of work, for me, it is orders of magnitude more efficient to just run the compiler, and click on the error "links" provided by the compiler output.

In the 7 or so years that I've used netbeans, there has been exactly ZERO releases where the project tree and editor annotations have been foolproof.  They've never worked reliably, and the overall processing efficiency of the IDE has been abysmal with regard to trying to make the project tree and editor annotations "work."

Anytime that I make changes to a "tree" in the ways described in other comments, with an SCM command line for example, I don't want the IDE to "reprocess" everything.  Instead, I want to select "build", and have the compiler tell me what is wrong.  I can then click through the error and warning messages, to the associated files, and take charge of fixing things, pronto.

Right now, there are moments that I can't run the compiler, for minutes, at times.  There are moments when I can't edit a file to fix it, for minutes, at times.  My computer system, and the applications that I am running at the same time, are part of that behavior problem.  

But, even with just netbeans running by itself, on a 16GB machine with dual, quad-core processors, the broken annotations and corrupted .../cache directory trees still represent no value added and loss of productivity.

(In reply to comment #37)
> I think that the intensity of the comments on this issue demonstrate the wide
> range of "expectations" and "desires" that everyone has, regarding what the IDE
> should and should not do for them.

Right, the topics discussed here were broad. With your recent contribution this issue can easily turn into another rant about parsing and Java editor. 

Should that happen, or should the API enhancement be miscategorized as a defect, I'll close the issue as invalid (due to loosing any relavance to the proposed API change).

(In reply to comment #32)
> does not work for operation performed externally (which I believe is a bug;
> fixable in Hg case at least) and if VCS guys agree, we should report it as
> separate issue.

Reported as issue 206255.

Overall, I think that taking this step, is running from the issues presented here.  I find it very hard to understand why the direction of the Java editor and parsing, for which the IDE has a large target audience, is so out of line with so many comments and complaints in issues which represent the most predominate problem areas.

I am one of those constant complainers because I really believe there is a much better choice of direction to go, and I really am serious about trying to understand what the problems are with changing the direction.  

But, we never seem to have those discussions, but instead argue about the fine points of proposals or suggestions with many conversations just stopping because we all tire of the incessant arguments.

You can, of course do exactly what you want, and Oracle, can pick the direction most desirable for them, I suppose.  But, I will still assert, without any real numbers, just observations, that this single issue, Java Parsing and Editor interactions, is the largest source of "hate" posts to the users list, as well as being the largest creator of heated debates. 

I think it's vitally important to the health of the Netbeans community, and I think that it, more than anything else, should have some focused consideration by the governance side of netbeans to formulate a community opinion on how important it is, and then to see how it might be possible to get the community to decide on a direction.

I want to make it clear, that I do enjoy a large number of the actions that the IDE provides based on the meta-data that it collects.  What I don't enjoy, is when the collection of that meta-data is keeping me from actually getting work done.  

This specific issue, in particular is one of those background actions that actually expands the RSS of the IDE process by bringing in all kinds of classes which where not being used.  Any EDT code that causes classes to be loaded is putting itself in a position to be a problem for the users productivity.  There have been lots of changes in how the EDT is used, but, I still see the "work on something in the background which has locked the database, sorry you get to wait again" popup.  

In particular, right after I start the IDE, the background parsing takes off, and I can't do "fix imports" and many other things like that, until it finishes.  I have ~75 projects that I keep opened in the IDE.  Many of those are libraries, or modules.  There are ~10 that are applications which have thousands of source files.  Most of the projects have dependencies on at least 4 modules/projects.  

It really should be possible to "read" the database at any time, regardless of what parsing is happening.  Doing this single thing, would eliminate so many sources of "IDE isn't responding" problems.  Removing actions that cause class loading from the EDT threading, would be another big time saver.  It might also be interesting to think about what would happen to put in an EDT queue that interacted with classloading, and treated entry to the .loadClass() as we do "opening a dialog", and create a new EDT thread while that is going on.

Comment #40 belongs on nbdev, as it is not closely related to this issue, which is about the IDE's response to events that occur when it is not in the foreground. Such event delivery is new as of 7.0, and in various circumstances including but not limited to indexable source file modifications, can result in substantial CPU/disk activity by the IDE - including but not limited to source file rescanning.

Improving the IDE's response to file events generally is orthogonal; and scanning strategy after opening projects, or using a full read-write lock on the parser database (already under investigation in 7.2), is completely unrelated.

I experience exceptions daily due to switching git branches on a daily basis.

It is unrealistic to expect every module owner that uses FileObject or DataObject to insert a test of whether the file exists or not before calling any method on it.  And that is the only way external changes will ever not cause random things to break.

Can we do something about this?

As I expressed multiple times, I am against integrating this change. Closing as won'tfix.

(In reply to comment #43)
> As I expressed multiple times, I am against integrating this change. Closing as
> won'tfix.

I agree with Jesse and Tim that the current behaviour is wrong and performance-wise suboptimal.

I do feel this is a hard one, but I haven't looked at the suggested fix yet. What if one wants to build a system without a UI based on NB and it uses the filesystem? What if the main window is to be hidden and things processed in the background for desktop applications built on the platform? How will this impact this if it is impacted?

When I say the UI is hidden, imagine an action in the users taskbar or notification area which reshows the UI. So, there are two use cases. 1) no UI at all. 2) where the UI may be completely hidden, but things should be able to be processed as fast as possible behind the scenes in case the user comes back they don't have to wait a long time for the app to catch up.

@Wade:  With attached patch, you would simply provide an alternate implementation of ApplicationActivationManager which does not ever go into the inactive state, and you would get the present behavior.

@Jarda:  As long as it is trivial to produce exceptions in the IDE by changing branches (rolling back a repository to its first commit sometime is a reliable way to trigger all sorts of chaos);  and as long as restoring the state of the filesystem does not restore the state of the IDE (i.e., in a shell, you can roll back to an empty repository, and then back to where you were, and when you return to the IDE there is no indication that something happened), this bug stays open.

(In reply to comment #44)
> I agree with Jesse and Tim that the current behaviour is wrong and
> performance-wise suboptimal.

+1. I've experience IDE responding to my external changes too and was surprised by it. In my case I was moving some files/folders around in OS explorer so you cannot blame it on a bug in VCS support.

Jarda, what is your technical objection against this change? I do not think it is fair to just close the issue despite all the feedback supporting this change.

I think part of the issue is that Jarda is primarily a Mercurial, not Git user.  Git users switch between local branches many times a day as a way of working on a few things at once.  Mercurial has "bookmarks", but I have never met someone who uses them.

I think if you don't experience this problem several times a day, it doesn't seem as much like a problem.

Regardless, the odds that every piece of code in NetBeans will ever be prepared to handle any file disappearing and reappearing underneath it are extremely low.  For example, what should happen if an open project temporarily disappears and then reappears?

Won'tfix.

Bear in mind that it's standard git workflow to:
 - Do a pull
 - Switch to a local branch (newly pulled stuff disappears)
 - Merge master to local branch (newly pulled stuff reappears)
 - Open a bunch of files and do some work
 - Commit
 - Switch to master branch (open files change, any new ones [which are probably
open] disappear)
 - Merge the local branch to master (files that just disappeared reappeared)
 - Push

This is pretty much how you use Git.  And it means you are *guaranteed* that
you will have problems in NetBeans every time you pull and every time you push,
if the IDE is open.

I'm a Mercurial user, but use mq a lot to switch between several patches I'm working on. Everyday I call hg qgoto to go here and there multiple times, and when I want to sync with the master, I hg qpop --all and hg fetch and hg qgoto the patch I'm working on.

I'm glad that NetBeans does not steal the focus anymore, but yes it still scans too often and too much.

well, even if you just build netbeans itself on cmd line doing hg fetch; ant clean; ant you get pretty much dragged down by the ide rescanning wildly. by quitting netbeans I'm able to cut the build time to half!

*** Bug 215131 has been marked as a duplicate of this bug. ***

> by quitting netbeans I'm able to cut the build time to half!

Are you saying that NetBeans causes Ant build to be 100% slower? Well in that case we are talking about a P2 defect, not a P3 enhancement. I suggest let's file that as a separate defect (with crisp steps to reproduce), and focus on fixing that.

(In reply to comment #55)
> > by quitting netbeans I'm able to cut the build time to half!
> 
> Are you saying that NetBeans causes Ant build to be 100% slower? Well in that
> case we are talking about a P2 defect, not a P3 enhancement. I suggest let's
> file that as a separate defect (with crisp steps to reproduce), and focus on
> fixing that.

Well, it's not really causing it. it's just competing for the same resources at the same time. CPU, IO operations,  to lesser amount memory. The base of the problem is described here already. The IDE should not initiate actions consuming common resources when it's not the active application.

(In reply to comment #56)
> (In reply to comment #55)
> > > by quitting netbeans I'm able to cut the build time to half!
> > 
> > Are you saying that NetBeans causes Ant build to be 100% slower? Well in that
> > case we are talking about a P2 defect, not a P3 enhancement. I suggest let's
> > file that as a separate defect (with crisp steps to reproduce), and focus on
> > fixing that.
> 
> Well, it's not really causing it. it's just competing for the same resources at
> the same time. CPU, IO operations,  to lesser amount memory. The base of the
> problem is described here already. The IDE should not initiate actions
> consuming common resources when it's not the active application.

form me it's more visible on my windows 7 notebook, almost unnoticable on my iMac with more than double the notebook's specs..

I tried to build the IDE (EE cluster only) from command line on my Lenovo laptop running Ubuntu 12.04 and see what difference the IDE running on background makes:

Build time with the IDE running on background: 13 minutes 33 seconds
Build time without the IDE running: 10 minutes 8 seconds

And one more test with clean and rebuild (previous test was just build on cleaned repo):
With the IDE running on background: 12 minutes 55 seconds
Without the IDE running: 10 minutes 41 seconds

That's a decent slowdown of my OS by a program I'm not using! Isn't this what we call P2 DEFECT?

I can also see in my IDE log

WARNING [org.netbeans.modules.masterfs.watcher.Watcher]: Cannot add filesystem watch for /home/dev/main/nbbuild/netbeans/.....: java.io.IOException: addWatch on /home/dev/main/nbbuild/netbeans/..... errno: 2

logged about 10000 times.

Closing this issue repeatedly as WONTFIX is unprofessional and arrogant. Could we please stop doing that and discuss it instead? Thanks.

Jarda, you should be senior enough to be able to drive this discussion towards some constructive outcome which satisfies all involved parties. You yourself admitted in the comment #33 that this issue is "not acceptable".

Problem described in this issue is simple and non-controversial: inactive IDE listens to external file changes and responds to them by aggressive use of common OS resources. That in turn slows down all other running applications.

The issue can be triggered by many different actions, for example external VCS operations, command line build, moving folders in OS explorer, etc. The issue 206255 filed to address VCS subset of the problem will not solve anything unfortunately - it can help for a single external VCS operation but that's not the problem here, *multiple* external VCS operations are the problem and that cannot be fixed in the VCS subsystem.

If we can identify which part of IDE is causing aggressive use of the resources (which I think we can; "scanning" right?) then we could design a solution around that - expensive operations should not be performed but queued when IDE is inactive; cheap operations like updating list of files in a folder (ala Amarok, Konqueror, Dolphin) should be still permitted and should not cause problems.

My two cents.

> expensive operations should not be performed but queued when IDE

Incidentally, the attached patch makes it trivial for any module to do exactly that (and is stable - I ran NetBeans with it for several months).

Definitely not ide/UI

(In reply to comment #59)
> expensive operations [such as scanning] should not be performed
> but queued when IDE is inactive

If there were a way to follow state changes (Tim’s ApplicationActivationManager, hackish listening on WindowManager.mainWindow focus listener, etc.), then project.indexingbridge.IndexingBridge could be used to suppress scanning while out of focus.

I would still be inclined to recommend just disabling all file events while out of focus, since it is likely to be difficult to enumerate all file event listeners which might trigger something expensive under some circumstance. A project folder could be deleted (causing ProjectManager to mark the Project as invalid and close it), then immediately recreated with slightly different contents. A pom.xml could be updated to refer to a new plugin version not locally present, causing the project to be marked as unloadable, shortly followed by a Maven build which downloads the new plugin and triggers another project reload. One file deeply nested in a build folder (but shown in the Files tab) may be deleted by a clean build, followed by another, after a few seconds followed by the whole folder being deleted. The benefit of reacting to every such file event in an IDE which might not even be visible, much less in use, seems very marginal compared to the known problems.

(And even “cheap” operations like refreshing a folder node may trigger GC, which can swap in big chunks of NB’s address space when you would rather reserve most of your RAM for something else. Linux file managers are not a fair comparison since they use far less memory.)

My worry is that disabling events at FileSystem level when IDE is out of focus may not be desirable in all cases. There might be a client in future which could benefit from immediate delivery of external file changes. And such client would be suddenly out of luck.

With a platform API to observe application state changes (active, inactive, idle)(and for how long) we are getting more flexibility but also more work as expensive operations have to be hand-pick. But in long term I think this should pay off. And enable other useful features as Tim noted eg. start indexing Maven's remote repo if IDE is idle for 5 minutes.

As for the scope of this issue I would suggest we put the "scanning" on hold while IDE is in inactive state and see what improvement it makes.

> There might be a client in future which
> could benefit from immediate delivery of external file changes. And such client
> would be suddenly out of luck.

Couldn't MasterFS or Filesystems provide an API for this if it is really needed?  In my patch, the code that collects the events is in MasterFS.

<sigh> reopening

You really want to keep closing this as WONTFIX?

I can play this game as long as you want...

Looking at the Tim's patch. The patch relays on the current non optimal implementation where the recursive listener needs FS traversal on all OS and holds references in FileObjKeeper. This is not optimal on Win32(64) or Mac OS X where the FS listener is recursive and native (JNA) impl listens on a root. On these systems it can be reimplemented in a way that adding a recursive listener practically costs nothing. However if we do it the queue in the Watcher will grow significantly when the IDE has no focus or it will not work. This can be solved in a way how Mac OS X CoreServices::FSEventStream handles it, it merges the events into common sub tries, when there is a big number of events in two folders the common parent is taken and single recursive event is used.

One of the reason, why we had implemented the native listening, was complains that the IDE starts to do expensive operations when it gets focus (exactly the moment user wants to start to use it). We replaced the "checking for external changes" done when the IDE got a focus by native listening. Unfortunately due to linux inotify limitations we need to return the "checking for external changes" back. Currently the IDE uses both.

I apologize but I cannot resist to express an opinion on this:

When the IDE does not have focus it can either do nothing and do a heavy operation when it gains focus OR it can listen and update itself while not having focus. We have invested a lot of effort into implementing the first and now you are suggesting that we revert the behaviour? Than it will automatically reopen the issue "IDE does long refresh after gaining focus" - while I don't have the bug number at hand this was at least P2 if I remember correctly.

So the short of it is: people you cannot have both of these issues resolved. It is simply impossible. I agree with Jarda about closing the issue (or start repeatedly closing the complaints that the IDE "does something horrible on focus gained").

comment #33 had Jarda's last words on the real issue of this report:

> ...performing such heavy activity on background (against user will) is
> not acceptable. I just don't believe focus gained is universtally agreed
> UI gesture when such operation should start (automatically).

Unfortunately, the discussion got diverted after that comment.

In last year's comments there seems to be much agreement that the current state is not correct. And look at comment 42 which restarted this year's discussion of this bug:

> I experience exceptions daily

I wonder if the "UI gesture" Jarda refers to that indicates you want the IDE to ignore things could be to minimize the IDE's main window. That would give a pretty well defined procedure
    - minimize IDE
    - work with VCS or whatever
    - maximize IDE

(And it would be interesting to observe what happens after VCS actions to remove everything and return everything to the original state. After maximize, there should be no indexing, parsing...)

> When the IDE does not have focus it can either do nothing and do a heavy
> operation when it gains focus OR it can listen and update itself while not
> having focus. 

> So the short of it is: people you cannot have both of these issues resolved. It
> is simply impossible. 

Here's the thing:  With "live" notifications, you get all sorts of problems, from the IDE randomly fronting itself (fixable, but do we believe we can find every place in the code where this might happen?), to exceptions being thrown, to projects being put in an unusable state, spontaneously closed, etc.

All of those things are also possible, but much less likely, if the IDE ignores changes when it doesn't have focus - since those problems are all caused by normal workflow in Git and Hg.

So ask yourself, which is more achievable:
 - Throttling scanning activities when the IDE regains focus, so they do not interfere with doing work, OR
 - Finding and fixing every possible case where code touches a FileObject which might be invalid by adding an isValid() test (and remember that File.exists() is an expensive, blocking call on Windows that we've had problems with before, and that's what FileObject.isValid() must call)

As this is a non-trivial issue with considerable impact on usability I am CC-ing UIRreviews@netbeans.org in order to start a UI review process at NetBeans UI Steering committee. We need to decide what to do to improve IDE usability in this use case; the point is to find way how to prevent excessive triggering of file change events in the common situation when users apply a series of versioning system queues from external command line what eventually results in minor change only, but throughout the process is detected by IDE as a major change in repository. The problem is that IDE does not recognize that the major part of the change is temporary only (caused by switching to another queue what changes lots of files, but quickly followed by quick switch back to the same queue after typically a minor update).
Possible approaches to the problem: add possibility to switch off temporarily the file change listeners on demand (automatic detection of the use case above seems difficult to define), or the possibility to suspend on demand the ongoing lengthy tasks (=extending the UI of progress bars in bottom right of the IDE). Any solution like that might lead to difficulties (e.g., user may suspend scanning but forget about it soon afterwards what would leave IDE in unwanted state)..

Is there any change made regarding to this recently?

I'm using Dev (Build 201211140001) and I've just ran "hg qpop" to update a file, NetBeans on the background jumps to the top and asks me if I want to reload the file. I thought that (197349) was resolved long time ago.

My previous version was 20121031 so it happens this month.

Discussion keeps going on offline but I guess I can report for the UI Steering committee that the solution based on permitting manual "Pause IDE" has been agreed as acceptable.

(In reply to comment #58)
> I tried to build the IDE (EE cluster only) from command line on my Lenovo
> laptop running Ubuntu 12.04 and see what difference the IDE running on
> background makes:
> 
> Build time with the IDE running on background: 13 minutes 33 seconds
> Build time without the IDE running: 10 minutes 8 seconds

I don't think this test is relevant  - just the fact that the IDE is running on the background and consuming resources is slowing down the build. In my opinion the more relevant test is to run IDE on background in both cases - just in one case to include the switch to turn off the listening to filesystem changes.

I've made some measurements, the setup is: two systems - relatively fast ubuntu and old slow XP. On both of them three scenarios were measured : 
1. build all clusters with nothing else running. 
2. build all clusters with netbeans with filesystem listening turned off running on background 
3. build all clusters with regular netbeans running on background.
In both 2. and 3. java.source is open in the ide running on the background.

Results:
           1.        2.         3.
Ubuntu   12:18     12:49      13:01
XP       28:39     39:02      40:15          

> That's a decent slowdown of my OS by a program I'm not using! Isn't this what
> we call P2 DEFECT?

It seems that most of the slowdown is caused just by IDE consuming some resources and similar slowdown could be caused (and is) by any other memory intensive process (ie. firefox with multiple tabs open) running on background.

Returning priority to P3.

(In reply to comment #75)
> (In reply to comment #58)
> > That's a decent slowdown of my OS by a program I'm not using! Isn't this what
> > we call P2 DEFECT?
> 
> It seems that most of the slowdown is caused just by IDE consuming some
> resources and similar slowdown could be caused (and is) by any other memory
> intensive process (ie. firefox with multiple tabs open) running on background.

I can't resist asking if you actually tried with multiple firefox tabs or you are just making assumptions here. Sure it's a memory hog (especially chrome) but when you have enough of that, the CPU usage is fairly low most of the time.

BTW your results align with my own observation that the issue is worse on windows.

(In reply to comment #74)
> Discussion keeps going on offline but I guess I can report for the UI Steering
> committee that the solution based on permitting manual "Pause IDE" has been
> agreed as acceptable.

It seems a little odd that something called a "UI Steering Committee" would recommend adding actions to the UI for a user to (if they remember) manually say "I'm not using you now", when that state can be easily detected with no new actions.

(In reply to comment #76)
> (In reply to comment #75)
> > (In reply to comment #58)
> > > That's a decent slowdown of my OS by a program I'm not using! Isn't this what
> > > we call P2 DEFECT?
> > 
> > It seems that most of the slowdown is caused just by IDE consuming some
> > resources and similar slowdown could be caused (and is) by any other memory
> > intensive process (ie. firefox with multiple tabs open) running on background.
> 
> I can't resist asking if you actually tried with multiple firefox tabs or you
> are just making assumptions here. Sure it's a memory hog (especially chrome)
> but when you have enough of that, the CPU usage is fairly low most of the time.
> 
> BTW your results align with my own observation that the issue is worse on
> windows.

Of course I just assumed the firefox, but now I've actually tried it so:
firefox with 30 open tabs instead of netbeans:

Ubuntu 13:21
XP     49:27

In hindsight it was an overkill with too many too complex tabs, because the ff consumed more resources than netbeans normally do, but the dependency on resources, not on the netbeans specifics seems clear to me...

The Petr's results are very similar what I've measured on my Mac i5 (4 stands CPU).

Thanks for the measurements Petr. It is useful to see more concrete numbers.

(In reply to comment #75)
> In both 2. and 3. java.source is open in the ide running on the background.

If you still have the setup in place could you please try one more test? I assume that amount of scanning done by the IDE depends on projects opened and their dependencies. Could you please open bunch of projects from different clusters and see what impact it makes on the numbers? In my test I remember having open openide.lookup, web.common, web.project, web.clientproject, web.jsf, html.editor and few others. I also had open bunch of files from different modules opened in the editor. Thanks.

It would also be a good idea to have a lot of files from those projects open in the editor, so there are definitely live FileObjects for some files which are modified.

>It would also be a good idea to have a lot of files from those projects open in
>the editor, so there are definitely live FileObjects for some files which are
>modified.
Is irrelevant as the FileObjectKeeper hods all folders under the source roots, as explained several times above.

Petr, thanks for the measurements in comment 75; a couple comments/questions about that.

The difference between scenario 2. and 3. is small, which is good, i.e. the source scanning does not slow down builds too much. On the other hand, the difference between scenario 1. and 2. on the Windows machine is significant. According to an off-line conversation with Petr, the Linux machine has 8GB of RAM, whereas the Windows machine has 2GB. I suspect that the slowdown on the Windows machine between 1. and 2. is because of the small amount of memory (not because of Windows vs. Linux efficiency), but to confirm this theory, it would be useful to also do the measurement on an old underpowered Linux machine with little RAM, and/or on a new powerful Windows machine with lots of RAM. Would we be able to run this test on such machine(s)? This would highlight the need to continue the focus on decreasing the memory usage of the IDE. Thanks.

Now, when enhancement 222193 is in, I am able to provide some trivial UI to "give user control over the IDE": 

http://hg.netbeans.org/main/contrib/rev/16184ed8f1e0
ergonomics#f1f91d6d6904

Although not ideal, it solves the original problem of switching branches temporarily from a command line. I feel I can't do more in masterfs and thus I am marking the issue as fixed.

UI and performance teams (and other contributors) may of course continue to think about and design better ways to "give user control".

Integrated into 'main-golden', will be available in build *201301150001* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main-golden/rev/f1f91d6d6904
User: Jaroslav Tulach <jtulach@netbeans.org>
Log: #194147: Including masterfs.suspend module on beta update center