This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 208277 - Scanning after projects have been closed
Summary: Scanning after projects have been closed
Status: RESOLVED WORKSFORME
Alias: None
Product: editor
Classification: Unclassified
Component: Parsing & Indexing (show other bugs)
Version: 7.2
Hardware: All All
: P2 normal with 1 vote (vote)
Assignee: Svata Dedic
URL:
Keywords:
: 184773 202135 207215 207539 208298 (view as bug list)
Depends on: 211163 223951
Blocks:
  Show dependency tree
 
Reported: 2012-02-10 21:13 UTC by Jesse Glick
Modified: 2012-12-22 14:16 UTC (History)
17 users (show)

See Also:
Issue Type: DEFECT
Exception Reporter: 184357


Attachments
stacktrace (1.93 KB, text/plain)
2012-02-10 21:13 UTC, Jesse Glick
Details
stacktrace (2.01 KB, text/plain)
2012-06-14 10:49 UTC, Jiri Skrivanek
Details
stacktrace (2.58 KB, text/plain)
2012-06-18 12:13 UTC, stefan79
Details
stacktrace (2.02 KB, text/plain)
2012-08-21 07:44 UTC, Jaroslav Tulach
Details
stacktrace (2.02 KB, text/plain)
2012-08-31 16:44 UTC, Jaroslav Tulach
Details
ide log file (129.59 KB, text/plain)
2012-12-05 17:14 UTC, bht
Details
thread dump while IDS scans closed project (25.59 KB, application/octet-stream)
2012-12-17 11:04 UTC, bht
Details
Log file, last entries relate to reprucing case (237.66 KB, text/plain)
2012-12-17 11:07 UTC, bht
Details
Log file, last entries relate to reproducing case (806.39 KB, text/plain)
2012-12-17 11:25 UTC, bht
Details
Tread dump relating to the last case (39.89 KB, text/plain)
2012-12-17 11:26 UTC, bht
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Jesse Glick 2012-02-10 21:13:25 UTC
Build: NetBeans IDE Dev (Build 20120208-5b0f6249ef8c)
VM: Java HotSpot(TM) Client VM, 22.0-b10, Java(TM) SE Runtime Environment, 1.7.0_02-b13
OS: Linux

User Comments:
jglick: There are no projects open. I switched Hg branches on the command line.

GUEST: Scanning projects was stuck at 60% for a long time.

gleyverg: I closed a maven project and there were no projects in the projects explorer and the scanning projects message started to happen.




Stacktrace: 
java.lang.Exception: Scan canceled.
   at java.lang.Thread.getStackTrace(Thread.java:1567)
   at org.netbeans.modules.parsing.impl.indexing.LogContext.create(LogContext.java:74)
   at org.netbeans.modules.parsing.impl.indexing.LogContext.create(LogContext.java:67)
   at org.netbeans.modules.parsing.impl.indexing.PathRegistry.scheduleFirer(PathRegistry.java:813)
   at org.netbeans.modules.parsing.impl.indexing.PathRegistry.resetCacheAndFire(PathRegistry.java:808)
   at org.netbeans.modules.parsing.impl.indexing.PathRegistry.access$500(PathRegistry.java:84)
Comment 1 Jesse Glick 2012-02-10 21:13:27 UTC
Created attachment 115607 [details]
stacktrace
Comment 2 Exceptions Reporter 2012-02-14 09:03:56 UTC
This bug already has 5 duplicates 
see http://statistics.netbeans.org/exceptions/detail.do?id=184357
Comment 3 Svata Dedic 2012-02-14 19:35:17 UTC
re. gleyverg (exception report #556675)
not reproduced (in dev build). When closing all mvn projects, the IDE reports that 0 source+binary roots were scanned. The uilog shows several times the sequence "open projects, clean & build, close projects"; 
did ALL the projects closed each time?


re. Jesse (exception report #559796)
not reproduced either (dev build). Tried: 
1/ closed all the projects; nothing was reindexed

INFO [org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater]: Complete indexing of 0 binary roots took: 0 ms
INFO [org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater]: Complete indexing of 0 source roots took: 0 ms (New or modified files: 0, Deleted files: 0) [Adding listeners took: 0 ms]
INFO [org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater]: Resolving dependencies took: 0 ms
INFO [org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater]: Complete indexing of 0 binary roots took: 0 ms
INFO [org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater]: Complete indexing of 0 source roots took: 0 ms (New or modified files: 0, Deleted files: 0) [Adding listeners took: 0 ms]

2/ closed SOME of projects, watching out for what the Indexer actually rescans - it took just non-existing dirs from left projects, total time spent in scanning close to nothing.

3/ closed ALL  the projects, and while still running the IDE, switched hg branches (release71 to default, then back) from cmdline; no scanning happenned at all

re. report 560219: automatic scanning setting is irrelevant to project open/close action, which changes paths in GPR. Ignoring.

re. report 560143: the hang-up may be related to the exception thrown from Facelets indexer - see the logs.
Comment 4 Jesse Glick 2012-02-14 20:06:47 UTC
(In reply to comment #3)
> re. Jesse (exception report #559796)
> not reproduced either (dev build). Tried: 
> 1/ closed all the projects; nothing was reindexed

Well, it happens from time to time for me, not consistently. Maybe the logging needs to be improved to diagnose better.
Comment 5 Svata Dedic 2012-02-14 20:14:33 UTC
So your scenario is to close everything (no scan is running), then sometimes the indexer starts some real work although there are no projects at all ? Some logging is already in place, so it could be possible to just raise logger verbosity when a 'invalid' state is detected.
Comment 6 Jesse Glick 2012-02-14 20:52:45 UTC
(In reply to comment #5)
> So your scenario is to close everything (no scan is running), then sometimes
> the indexer starts some real work although there are no projects at all?

Right.
Comment 7 Tomas Zezula 2012-02-16 09:43:05 UTC
*** Bug 208298 has been marked as a duplicate of this bug. ***
Comment 8 Tomas Zezula 2012-02-16 09:49:44 UTC
There are 2 possible things that may cause this problem:
1st) the GlobalPathRegistry still contains some paths.
2nd) The unknownRoots is not empty. Unknown roots are roots which are not from opened projects or dependencies but were accessed during IDE run (for example through favorites or open file).  As there is no relation of these roots to opened projects they are active until the ClassPaths holding them are gced.

Logging showing which one is true needs to be added.
Comment 9 Svata Dedic 2012-02-16 15:06:18 UTC
More logging added. When the situation happens again, please attach your messages.log - try to also note:

* what project(s) you had been working on (assuming you work on NB sources)
* whether some files from closed projects, or even non-project files were opened
* whether the scanning was running just prior the close operation


I won't close the defect, as the logging introduced another unwanted dependnecy between parsing.api -> project.uiapi, which should be removed before release.
Comment 10 Tomas Zezula 2012-02-16 15:18:56 UTC
Thanks Svato!
Comment 11 Svata Dedic 2012-02-17 14:53:27 UTC
*** Bug 207215 has been marked as a duplicate of this bug. ***
Comment 12 Quality Engineering 2012-02-18 09:26:33 UTC
Integrated into 'main-golden', will be available in build *201202180400* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main-golden/rev/2f76718810bb
User: Svata Dedic <sdedic@netbeans.org>
Log: #208277: Additional logging of PathRegistry and GlobalPathRegistry, when all projects are closed
Comment 13 Jesse Glick 2012-02-20 19:01:19 UTC
(In reply to comment #9)
> whether some files from closed projects, or even non-project files were opened

In the cases when this has happened to me, no - no files or projects were open.
Comment 14 Svata Dedic 2012-02-27 16:09:08 UTC
Please attach portion of log when the defect happens again.
Comment 15 Jesse Glick 2012-03-02 17:22:07 UTC
Exception #563405 includes a log file. I had just closed openide.util and a j2seproject in /tmp/jaxp-7u4-regr (leaving no files or projects open); scanning then began and ran for about ten seconds, during which I tried to cancel it.
Comment 16 Jesse Glick 2012-03-09 23:36:00 UTC
Continues to happen to me pretty regularly.
Comment 17 Quality Engineering 2012-04-05 01:19:54 UTC
Integrated into 'releases', will be available in build *201204042205* or newer. Wait for official and publicly available build.
Changeset: http://hg.netbeans.org/releases/rev/8f11c11baa35
User: Svata Dedic <sdedic@netbeans.org>
Log: #208277: Additional logging of PathRegistry and GlobalPathRegistry, when all projects are closed
Comment 18 Tomas Zezula 2012-04-13 13:58:41 UTC
The /tmp/jaxp-7u4-regr project was not opened at all or was closed before the openide.util was closed?
Was some file from it opened during the IDE run?
Comment 19 Jesse Glick 2012-04-13 16:47:06 UTC
(In reply to comment #18)
> The /tmp/jaxp-7u4-regr project was not opened at all or was closed before the
> openide.util was closed?
> Was some file from it opened during the IDE run?

They were both opened, probably with some files open; I selected them both in the Projects tab and chose File > Close Projects, which of course closed any open files as well. Then scanning started.
Comment 20 Svata Dedic 2012-04-25 08:19:12 UTC
*** Bug 207539 has been marked as a duplicate of this bug. ***
Comment 21 Exceptions Reporter 2012-06-14 02:37:50 UTC
This bug already has 100 duplicates 
see http://statistics.netbeans.org/exceptions/detail.do?id=184357
Comment 22 Jiri Skrivanek 2012-06-14 10:49:50 UTC
Created attachment 120828 [details]
stacktrace

Just need to revert changes in some file.
Comment 23 Svata Dedic 2012-06-15 08:40:56 UTC
Not really a P1, made P1 automatically by exception reporter based on number of reports. However after brief scan many of the reports are not actually related to scanning of closed projects, but to canceled scans. 

I'll try to categorize them and assign to appropriate issues.
Comment 24 stefan79 2012-06-18 12:13:50 UTC
Created attachment 120979 [details]
stacktrace
Comment 25 Jaroslav Tulach 2012-08-21 07:44:24 UTC
Created attachment 123333 [details]
stacktrace

I have few projects from my netbeans hg repository openned. I also had one independent suite and three its projects openned. I selected the suite and its projects and closed them. The complete reparse of remaining projects started - imho, for no reason.
Comment 26 Exceptions Reporter 2012-08-22 14:35:33 UTC
This bug already has 100 duplicates 
see http://statistics.netbeans.org/exceptions/detail.do?id=184357
Comment 27 bht 2012-08-24 20:32:39 UTC
I found scanning of closed projects without seeing an exception. I noticed this because of access denied to files of the closed projects that I tried to delete after the projects were closed. There was evidence in the IDE log that the projects were scanned after they were closed.

I have also seen multiple times that scanning of closed projects started after they were already closed.
Comment 28 Jaroslav Tulach 2012-08-31 16:44:03 UTC
Created attachment 123786 [details]
stacktrace

I don't think there is anything to scan, I've just closed all my projects, but I still see the progress bar (not moving). Now it finished, before I finished typing this text.
Comment 29 Svata Dedic 2012-11-07 12:56:59 UTC
*** Bug 184773 has been marked as a duplicate of this bug. ***
Comment 30 Svata Dedic 2012-11-28 12:06:38 UTC
*** Bug 202135 has been marked as a duplicate of this bug. ***
Comment 31 Svata Dedic 2012-12-05 14:48:56 UTC
Note that there's no report from 7.3 (dev, beta1, beta2). Exception reports from 7.2 #607015, #608948, #630758 are irrelevant (there are opened projects still)

Closing as worksforme until a 7.3 report arrives.
Comment 32 bht 2012-12-05 17:13:00 UTC
Reproduces with 7.3 dev Build 201211300001

- Start NetBeans in separate userdir - do not import settings
- Menu|Tools|Options|Miscellaneous|Files|Uncheck "Enable auto-scanning of sources"
- Open project WicketExamples from
  Issue http://netbeans.org/bugzilla/show_bug.cgi?id=181173
  file attachement http://netbeans.org/bugzilla/attachment.cgi?id=94576
- Wait until scanning completes
- Copy the project Projects Window|Copy
- Close the project Projects Window|Close
- In Windows, Delete the folder of the closed project while scanning is still in progress

Windows complains that it cannot delete the folder because it is in use.

Reproduces sometimes not always - depends on whether scanning continues or not after the project is closed.
This can be seen at the status bar.
Comment 33 bht 2012-12-05 17:14:54 UTC
Created attachment 128917 [details]
ide log file
Comment 34 Svata Dedic 2012-12-14 16:02:36 UTC
Please, provide some clarifications:

(In reply to comment #32)
> - Wait until scanning completes
> - Copy the project Projects Window|Copy
Where did you copy the project ? Please specify the original and the copied project path(s).

> - Close the project Projects Window|Close
Which one ?

> - In Windows, Delete the folder of the closed project while scanning is still
> in progress
> 
> Windows complains that it cannot delete the folder because it is in use.
> 
And what is the complaint ? From the above it is not obvious that NO project is opened. If the 2nd project is still opened & scanning, it's OK.

Windows complain when a file anywhere in the subtree is opened at the time delete is invoked; not necessarily connected with background scan.
Comment 35 bht 2012-12-14 17:09:11 UTC
In response to #34:

a) The directory structure:
parent_dir|
          |-WicketExamples
          |-copy|
                |-WicketExamples

This should be consistent with the log entries

b) Closed the copied project with the aim of deleting it later. But in other cases I closed both projects and deleted both.

I agree there could be any reason for Windows not letting me delete a file but the fact that it works after scanning ends has been consistent.

I don't know whether the copying of a project from within NetBeans has anything to do with this but it may be a convenient way to get NetBeans busy enough to trigger the issue. Perhaps you can create multiple copies fast and close them fast to reproduce. As I wrote this reproduces only sometimes.

On Windows in a different scenario I used Process Explorer to identify which process locks which file.
Comment 36 bht 2012-12-15 03:21:51 UTC
I have just reproduced the test case first time. I closed the copied project while initial scanning for it was still in progress after it was copied. Then scanning continued for a long time, and deleting the copied project was not possible during that scanning, only after a few seconds delay after scanning of the copied, closed project completed.

The remaining issue here is perhaps primarily that scanning is not canceled if a project is closed.

I would prefer to have an option that lets the user decide whether to let NetBeans open a project or not after NetBeans copies it. Just a check box with default checked in the existing copy dialog.
Comment 37 Svata Dedic 2012-12-17 09:42:10 UTC
Cannot reproduce using the provided steps. Scan of the copied project takes some noticeable time (~10 secs) on my machine; if I step in and close the new copy during the scan, the scan appears to be terminated almost immediately (progress bar hides).

Please re-run the IDE with -J-Dorg.netbeans.modules.parsing.impl.indexing.RepositoryUpdater.level=400, to make the RepoUpdater log what it is doing. During the time the scan is still hanging around (after you've closed the new copy being scanned), please:
* make a thread dump manually (http://wiki.netbeans.org/GenerateThreadDump)
* cancel the scan (it will log certain information, and also a thread dump)
Maybe do both, please.

There were issues with timely termination of a scan/indexer after an event obsoleted the running scan, but they should be solved already in your dev version (see issue #219578)
Comment 38 bht 2012-12-17 11:04:58 UTC
Created attachment 129448 [details]
thread dump while IDS scans closed project
Comment 39 bht 2012-12-17 11:07:40 UTC
Created attachment 129449 [details]
Log file, last entries relate to reprucing case

This time the issue reproduced on the second attempt. At the top of the log file there would be messages from the first attempt also.
Comment 40 bht 2012-12-17 11:25:42 UTC
Created attachment 129450 [details]
Log file, last entries relate to reproducing case
Comment 41 bht 2012-12-17 11:26:45 UTC
Created attachment 129451 [details]
Tread dump relating to the last case
Comment 42 bht 2012-12-17 11:29:30 UTC
Perhaps I can reproduce this quite easily because my XP computer is slower than most PCs. Sorry only the last log file has the option -J-Dorg.netbeans.modules.parsing.impl.indexing.RepositoryUpdater.level=400
Comment 43 Svata Dedic 2012-12-17 13:54:09 UTC
bht - thanks for all the support. Yes, it could be caused by some timing/improper synchronization. I was hoping that the thread dump shows some code which does not check the 'interrupted' flag. Sadly this is not the case.

Each of the logs contains approx. 5 iterations of adding /copy/ resources (apparently Copy project) and then removing them (close project). Can you confirm that you performed the copy/close cycle more times during the IDE run ? The states of path registry seems OK after each updater task (contain all or none relevant /copy/ paths).

Could you please perform the scenario just once (so any excess parsing will show up) and - instead of capturing a thread-dump manually - try to cancel the scan after the project is closed.

The IDE should then also log stacktraces of event which triggered the extra (unneeded) scan. File that as an exception report. Just for case, please not the exception report number (or link to it) here, I don't know whether the exception reporter associates the stacktrace with this defect.

Thanks!
Comment 44 bht 2012-12-17 18:29:16 UTC
With the latest version Build 201212170919 I do not get an exception when I try to cancel the scanning in the status bar. The IDE just refuses to cancel the scan with a message dialog.

So how should I cancel the scan?

Otherwise with my previous build 201211300001 see exception report  #640512: "It has been classified as a duplicate of report #195833. This bug was already fixed. Please update your build of NetBeans to the latest build of 7.3 where bug #222940 is fixed"

Re logs with attachment id 129450 I may have performed the cycle 5 times. It did not reproduce as reliably as expected.

Regarding why you may not see the expected problem, I have NOT noticed that the status bar reported an end to the first scan after copy and then reported a start of a new scan after close.

So this might ba a case of the IDE being too busy to cancel the first scan in time.

What I have seen at times is a delayed checking for external changes, perhaps because I deleted the copy, and HTML editor initialization.

I currently need to run the test 5 or more times within the same session to catch the scanning after closing before it finishes. However in all cases I needed multiple attempts in Windows Explorer to delete the copied files.

It is a bit difficult because the scan starts early while copying is still in progress which slows down the copy part and shortens the remaining scan time after the GUI is unblocked.
Comment 45 bht 2012-12-17 19:04:10 UTC
There is more going on. After the copy closed in the IDE, and after the copy is deleted in Explorer then the IDE status bar reports:

Background scanning of projects
Scanning for external changes

two times I think, with "Scanning for external changes" suspended at some stage. This is strange because after the project is closed then why does the IDE care about the used deleting it?

This seems to get sticky during a session - it now takes minutes after the scan completes and the files are finally deletable.
Comment 46 bht 2012-12-17 19:50:18 UTC
Filed issue 223951
Comment 47 bht 2012-12-22 14:16:36 UTC
Looks good now. I cannot reproduce this with the latest version. The IDE has become a little faster :)