This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 162635 - Scanning project too frequent
Summary: Scanning project too frequent
Status: RESOLVED FIXED
Alias: None
Product: editor
Classification: Unclassified
Component: Parsing & Indexing (show other bugs)
Version: 6.x
Hardware: All All
: P3 blocker (vote)
Assignee: Vitezslav Stejskal
URL:
Keywords:
Depends on: 162706
Blocks:
  Show dependency tree
 
Reported: 2009-04-14 21:06 UTC by mjreged
Modified: 2009-05-15 10:09 UTC (History)
4 users (show)

See Also:
Issue Type: DEFECT
Exception Reporter:


Attachments
netbean scanning maven projects (10.39 KB, image/png)
2009-04-14 21:07 UTC, mjreged
Details
during scanning (137.62 KB, application/octet-stream)
2009-04-15 00:38 UTC, mjreged
Details
First start up - no source changes (311.04 KB, text/plain)
2009-04-16 14:58 UTC, mjreged
Details
Restart the IDE with the same project group then scanning begins for long time (375.92 KB, text/plain)
2009-04-16 15:02 UTC, mjreged
Details

Note You need to log in before you can comment on or make changes to this bug.
Description mjreged 2009-04-14 21:06:35 UTC
Nb7.6 build 200904131401
I find that project scanning is very frequent and performance intrusive. Every 2nd or third time i open the same group of 
maven projects i end up waiting for about 7 min for scanning projects.. to finish. It looks like it scans local maven 
repo, my projects, jdk rt.jar etc... I don't understand why so frequently.  I would accept that first time project can 
potentially take long time as it needs to learn about the project structure. I can see that this can become very painful 
as more dependencies are introduces and more projects are added. It's rescanning way to often or the process is too much 
of a performance drain maybe someone can set my expectation strait here, but i feel it a legitimate concern.
Comment 1 mjreged 2009-04-14 21:07:27 UTC
Created attachment 80076 [details]
netbean scanning maven projects
Comment 2 mjreged 2009-04-15 00:37:22 UTC
Just want to emphasize on the duration and speed as well, i just compared this operation with Netbeans 6.5, nb6.5 is much 
better and faster at loading and scanning maven projects.  So right now as it stands in NB6.7 it's definitely a step 
backward in term of performance. I am also attaching a heap trace during scanning. What really strikes me JaxWsOpenHook$WebservicesChangeListener.updateJaxWs() is taking so much time.  In my case I find this feature dead wrong, 
please don't touch my jaxws configurations i use CXF and i configure everything with spring. I raised other issues with 
this feature, i wish someone would pull this feature out. 
Comment 3 mjreged 2009-04-15 00:38:13 UTC
Created attachment 80090 [details]
during scanning
Comment 4 Milos Kleint 2009-04-15 06:44:38 UTC
reassigning to java for evaluation of repeated scanning. I've noticed the same and often the error marks stay even
though they shuld disappear and single files get rescanned upon opening..

as far as I can tell the jaxws open hook is only waiting for teh scanning to finish and is not performing any tasks on
it's own.
mkuchtiak: please evaluate the jaxws part of the issue. I've noticed you have spawned a req. processor job for each
project and there's like 10+ running concurrently. Maybe you should create your own RP queue and put the jobs there..
Comment 5 mjreged 2009-04-15 14:45:14 UTC
You hit it right on the nail with the error marks.
I was embarrassed by this already in front of another developer. I Introduced him to Netbeans
Opened all projects, build it and deployed in on server, he saw me do it all. But i could not explain to him
why on almost all projects, netbeans is showing error marks even thought maven built and deployed.
I had to shutdown netbeans and bring it back up, explaining these error marks are out of synch. Sometimes
the restart would not alway be effective.  For a new user, this a nightmare because they are not sure if they coded 
something wrong, or is the IDE getting confused.
Comment 6 mjreged 2009-04-15 16:35:13 UTC
Just want to add, that the error marks also are out of synch even in Netbeans 6.5  as far as maven is concerned.
Comment 7 Milos Kleint 2009-04-16 07:28:34 UTC
can we get this fixed in 6.7 timeframe? I consider it crucial to the success of maven support in 6.7. There have been
problems of this kind since forever, for some reason they are more common for maven projects (my suspicion is the deep
directory structure of the local repository). It might get overlooked in simple setups, but any non-trivial real life
project suffers from it time to time.

Can we increase to P2?
Comment 8 Rastislav Komara 2009-04-16 09:26:16 UTC
@all: Do you have any reproducible scenario? If not this is really hard to fix this issue.

@mkleint: I don't think this issue is correct candidate for P2. But we can discuss this.
Comment 9 Vitezslav Stejskal 2009-04-16 10:05:18 UTC
Ok, I see two issues here - (a) scanning performance (unneeded/unexpected/? scanning) and (b) stale error badges.

For the first one could you please run the IDE with
-J-Dorg.netbeans.modules.parsing.impl.indexing.RepositoryUpdater.level=FINE and attach the IDE's log file
(<ide-userdir>/var/log/messages.log) here. That should show us what was the IDE indexing and perhaps we we will the be
able to guess why.

In general scanning is always done when a project is opened (ie IDE started), but it should not take too long _if_
nothing changed since the last time the scanning ran. So for example just opening/closing projects or restarting the ide
_without_ any changes in files on your disk (timestamp check) the initial scanning should be fast as it only needs to
check that nothing really changed. On the other hand if for _any_ reason local files changed (eg. you run a build,
update repo, download new versions of your libraries, etc) the scanning will have to physically reindex all the changed
stuff. And it may take some time.

Finally, the indexing was under heavy development in Nb6.7 and we are still fixing bugs. Please always use recent dev
builds for testing/verifying problems/fixes. Thanks

As for (b) stale error badges - Milos demonstrated the problem on a simple testcase and we are looking into it. Thanks Milos
Comment 10 mjreged 2009-04-16 14:58:14 UTC
Created attachment 80264 [details]
First start up - no source changes
Comment 11 mjreged 2009-04-16 15:02:44 UTC
Created attachment 80266 [details]
Restart the IDE with the same project group then scanning begins for long time
Comment 12 Milan Kuchtiak 2009-04-16 17:09:42 UTC
I've implemented some improvement in @WebService annotation processing.

See:
http://hg.netbeans.org/main?cmd=changeset;node=50b123a52883
Comment 13 mjreged 2009-04-21 18:13:33 UTC
Well this issue definitely has bent my will to work on our projects with the latest build.  Other issues were manageable 
,i could still get a lot of work done. But this one had me change back to NB6.5.  I will try newer builds later see it 
this improved.  
Comment 14 mjreged 2009-04-21 21:31:34 UTC
Just Tried build NB6.7  200904210201

I start the IDE load all 14 maven projects. Let it scan for the first time. ( takes about 12min).  Wait so that it's done

Restart the IDE and now have to wait again for scanning to be over. Also really long time.

I thought of something funny.  If you guys had to develop the entire netbeans modules using maven nb integration instead 
of ant. I wonder how long would you have to wait every you opened The IDE. (Days perhaps ?)
Comment 15 mjreged 2009-05-14 19:30:03 UTC
NB6.7 200905130201

Seems to work very well in respect to project scanning.
The Initial long over 10 min scan is gone.

I am really happy about that. Great progress.
Comment 16 Vitezslav Stejskal 2009-05-15 10:09:08 UTC
ad a. scanning performance - thanks for your testing and confirmation that things have improved. The main slowdown was
due to problems described in issue #162706 (fixed).

ad b. error badges - there were several problems in checking classpath dependencies and rescanning affected roots, the
main relevant to this was probably issue #163385 (fixed). So, this should now behave better, but there are still number
of problems with error badges in general (most of them not introduced in 6.7 and unlikely to get fixed in 6.7), please
see the umbrella issue #121950 and its dependencies.