Bug 152749 - Scanning of sources and compiling should be removed
Scanning of sources and compiling should be removed
Status: NEW
Product: editor
Classification: Unclassified
Component: Parsing & Indexing
All All
: P2 with 1 vote (vote)
: 6.x
Assigned To: Tomas Zezula
Depends on:
  Show dependency treegraph
Reported: 2008-11-08 00:46 UTC by greggwon
Modified: 2017-09-17 05:19 UTC (History)
7 users (show)

See Also:


Note You need to log in before you can comment on or make changes to this bug.
Description greggwon 2008-11-08 00:46:06 UTC
The changes in netbeans for 6.0 and later included changes in the compiler usage in a direction which is not as helpful
as it is hurtful for the overall operation of the IDE.  The use of background scanning of sources when dependency
changes are detected seemed to be aimed only at providing a visual indicator of problems that the developer needs to pay
attention to.

These changes seem to be aimed as a useful behavior when open projects are perhaps all interrelated modules.  That is, a
broken API interface in one module will result in another module being a problem.  While many types of projects do have
this type of behavior, it is not clear to me, that this is an overwhelming reason for all users and all project types to
be burdened by the overwhelming slowdown this activity has on the IDE.

As a general issue of the value this can have overall on "correctness", I am not convinced that everything can be
detected and noted.  In particular, background activities of SCMs, can alter the visible source tree, and depending on
settings of the SCM, checked out files might either have the current timestamp, or the timestamp of the submission.

So, for example, if I revert a file that had changes built and visible, and that file is not opened in the IDE, will the
IDE know to always check this file?  SCM activity can touch 100s if not 1000s of files which can take a long time to "scan".

My general usage of the IDE, when this scanning gets in my way, is that I am building one project that is a library
because I need new functionality from that library that one other project uses.  When the installation of the library
into the directory where my "library" reference points causes 10 other projects to be rebuilt and validated, and I don't
need that to happen, the activities of the IDE are not helpful to me.

When bootstrapping a project, I find myself in this mode of operation continuously as I flesh out the base code and it
is extremely frustrating that I have to work at 1/10th the speed I might work.

Please consider completely removing this scanning and just let the builds take care of the issues in other projects.
Comment 1 Tomas Danek 2008-11-10 16:24:41 UTC
I'm pretty sure that performance team has arguments why is this useful and points you to documents what has been in 6.5
improved in this area..
Comment 2 greggwon 2008-11-10 18:31:42 UTC
The particulars of fixes to this mechanism are not as interesting as how the scanning makes it faster for the IDE to
operate over all.  I don't see the IDE operating faster when I now have to wait 5mins after rebuilding a library, for
the IDE to rescan all my projects and be happy that I didn't break any APIs.  I updated the library, rescan the library,
not all of my projects.  When I open a source file, scan it.  When I search a project, scan it.  When I search all
source files for some text, keep track of what you learn then.

I should only pay for the time it takes to search globally, when I issue for global searches.

Rescanning all of my projects after the library build is not something I need the IDE to do.  Instead, it is something
at a feature of the IDE wants to do, to "color" my project tab with information that I don't need.  So, it should do
that either completely in the background, without impact to the IDE editing, build and run mechanisms, or it should not
do it at all until I ask specifically for that information.
Comment 3 Tomas Pavek 2008-11-10 19:39:45 UTC
> ... I now have to wait 5mins after rebuilding a library, for the IDE to rescan all my projects ...

I think NB java team could answer, I believe it is possible to have such a setup that projects are not rescanned after 
rebuilding a library.

Note that for NB 7.0 we plan to do an experiment by providing a special plugin which when installed would disable 
automatic scanning and let the user invoke it explicitly if needed. It was planned mainly for turning off checking 
external changes after IDE restart (project reopen) but perhaps it could be extended also to some cases of changes on 
project classpath.
Comment 4 rdelaplante 2008-11-10 20:16:59 UTC
I was analyzing a heap dump when I decided to look in the source of a web project.  I clicked the + beside it, and it
said "Initializing project...".  It said that for all web projects.  I waited a minute and it seemed stuck.  I restarted
the IDE a few times, rebooted, and it still said that.  Just before I was ready to delete my .netbeans folder, I noticed
the status bar said it was scanning all of my projects!  I made a meal, ate it, and when I got back NetBeans was done
scanning and my projects became usable again. 
Comment 5 rdelaplante 2008-11-10 20:17:26 UTC
Added self to CC list
Comment 6 greggwon 2008-11-11 19:07:23 UTC
I just had another episode of this behavior.  It occurs to me that netbeans should really just be checking the date on
the library jars when the user invokes a build, and ignore .class files older than the libraries in the classpath to
facilitate a robust build.  Error badges are just not adding value to my experience given the limited amount of
information that they actually convey.  There is no list of errors or anything else accessible after all of that work. 
The fastest way to see the errors I have to deal with is to run the compiler and look at the error messages.  That
provides clickable links to the source files etc, which IS helpful in finding out what needs to be fixed.

But guess what, this is already how I work.  When I first start working each day, I sync the SCM to my machine, look for
library updates from my coworkers, build those libraries, and then any project I will work on, I select "Clean and
Build" on to make sure I can see exactly what the state of affairs are.  It's that tree of dependencies I know netbeans
developers are trying to assist me with.  But, the simple problem is that I can't make use of all the work that they are
trying to do, instantaneously, and the time it takes to do that work is just so counter productive compared to the tasks
that I actually need to focus on.
Comment 7 rdelaplante 2008-11-11 19:17:53 UTC
> Error badges are just not adding value to my experience given the limited 
> amount of information that they actually convey.  There is no list of 
> errors or anything else accessible after all of that work. 

If you keep the Tasks pane open it will show you a list of all errors, which file, line, etc. If you double click on one
of the tasks it will open the source file and put the cursor on the line with an error.
Comment 8 greggwon 2008-11-11 20:03:06 UTC
Okay, so that was not an obvious, to me, place to look.  Now I look there and see all kinds of things to click on.  I
click on some of them that describe unresolved external class references.  I click on various ones and I don't get any
margin markings for errors, and no orange/yellow warnings either.

For example, here is one of the nonsense messages:

Error	cannot find symbol symbol  : method print(java.lang.String) location: class System.out	Gsm_State.java	71

It's telling me (indirectly, why isn't the same message here that is on the "resolve reference problems" dialog?) that
the project is referencing a non-existent JVM.  There are hundreds of these errors (and associated badges) when only one
is needed.

Comment 9 greggwon 2008-11-12 01:42:18 UTC
I had another case just now where I rebuilt one of my core libraries.  NB(version 6.1) went off to rescan/compile all of
the dependent projects.  I noticed another edit I needed to make on a library source file.  I started making that edit
and the editor noted a syntax error as I opened a try{}catch.  But, it did not note when the braces were correctly
matched.  I was able to finish my edit, start another build, and that finished as well.  It then kicked off another set
of scans/compiles, which I believe are now happening in parallel.  The library built without error, but my editor is
still full of red bars in the margin and red lines in the source because it is waiting for use of the compiler or
something to make progress.
Comment 10 rdelaplante 2008-11-12 02:33:57 UTC
For each of your projects, right click the project tree node, select properties.  Click the Libraries tree node, and at
the bottom of th escreen un-check the "Build projects on classpath" checkbox, and click OK.  Does this solve your problems?
Comment 11 greggwon 2008-11-12 14:10:21 UTC
Most of my large projects, where it seems a lot of the time is spent, are freeform and this option is not present on the
java sources classpath pane.
Comment 12 Jan Becicka 2008-11-13 10:51:36 UTC
Jarda wanted to do some experiments with disabling startups scan and file events.
Comment 13 David Strupl 2008-12-04 13:07:29 UTC
This sounds like a performance enhancement. If this requires something from java team please create sub issues and
assign only the subissues to us.
Comment 14 Vitezslav Stejskal 2009-06-16 14:00:30 UTC
There is an experimental module on AU which attempts to turn off automatic scanning and adds File -> 'Refresh All' menu
item. See http://wiki.netbeans.org/ScanOnDemand for more details. 
Comment 15 David Strupl 2009-06-18 13:07:49 UTC
One possible way to implement this is to be able to specify a list of excludes of source (or binary) roots. By exclude I
mean to be able to say that these roots should not be scanned and/or that these roots should not be watched for
modifications. This would allow the users that generate new source code during the build to generate the code into the
folders that would not trigger the otherwise enabled automatic scan. Of course there would have to be an UI that would
allow to index such roots on demant (e.g. action on the root "Index" or similar). Similarly it might be possible to
exclude a .jar file from indexing, again when someone e.g. generates new jars (with the same data) when doing build.
Comment 16 Tomas Zezula 2009-06-30 18:25:33 UTC
Rather than removing the initial scan and letting the IDE to work on staled data it seems to me better to speed up the indexing (compiling) itself.
1) Is it needed to do the full attribution? Yes-when compile on save is turned on, but enter + symbol index (like JDT) may be enough otherwise.  Attr is about 
20% of java initial scan.
2) There are also possible improvements in the lucene usage. I've some experiments which improve the performance of LuceneIndex by 30%. The lucene takes 
also about 20% of the whole java indexing. Also when doing the indexing using JSR166 on 4 cpu (2*2 cores) machine the performance improvement is even 
better, but probably not yet main stream.
Comment 17 matthies 2009-06-30 19:01:50 UTC
I'm surprised to learn that Lucene is being used for the source index. Does this make any sense? Source code, unlike 
document text, is highly structured, and source code index lookups have quite different requirements from full text 
Comment 18 nleck 2009-06-30 22:43:08 UTC
Hi Guys, 

I don't think the increasing the speed will resolve the issue. We need to make the indexing non blocking. The initial
index is not so much of the concern as it happens only once. It's all the features that are blocked while the
incremental scans are done that's the biggest killer. 

Comment 19 RAM518 2017-02-19 11:04:36 UTC
bump. I would really, really appreciate the option to completely disable background scanning of projects, and error indicators in my project, completely. I have other tools I use to determine errors, I don't need, nor do I want, visual indicators about errors while I'm writing code.
Comment 20 anon123123 2017-09-17 05:19:40 UTC
I'm not interested in scanning or compilation of my code. I just want Netbeans to act like a simple code highlighter. I find 'error badges' to be extremely distracting flashing error notifications, and I'd like the ability to turn them off, completely.

Literally Netbeans flashes red warnings anytime you type one single character, and won't take no for an answer.

By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2014, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo