This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 221837 - Background scanning of projects takes too long
Summary: Background scanning of projects takes too long
Status: RESOLVED FIXED
Alias: None
Product: javaee
Classification: Unclassified
Component: JSF Editor (show other bugs)
Version: 7.2
Hardware: All All
: P3 normal with 2 votes (vote)
Assignee: Martin Fousek
URL:
Keywords: PERFORMANCE
: 210600 (view as bug list)
Depends on: 210526
Blocks:
  Show dependency tree
 
Reported: 2012-11-09 15:11 UTC by Exceptions Reporter
Modified: 2014-05-23 07:09 UTC (History)
5 users (show)

See Also:
Issue Type: DEFECT
Exception Reporter: 194827


Attachments
stacktrace (2.00 KB, text/plain)
2012-11-09 15:11 UTC, Exceptions Reporter
Details
patch to web jsf editor JsfIndex.java to stop querying the index too much times (4.98 KB, patch)
2013-02-11 16:02 UTC, windli
Details | Diff
patched class file for you to try (8.10 KB, application/x-zip-compressed)
2013-02-11 16:10 UTC, windli
Details
Background process endless loop cause (650.22 KB, patch)
2014-01-27 14:30 UTC, erkancengiz
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Exceptions Reporter 2012-11-09 15:11:19 UTC
This issue was reported manually by sdedic.
It already has 8 duplicates 


Build: NetBeans IDE 7.2.1 (Build 201210100934)
VM: Java HotSpot(TM) Client VM, 20.4-b02, Java(TM) SE Runtime Environment, 1.6.0_29-b11
OS: Windows Vista

User Comments:
GUEST: I killed successfully this task.

nicu2005: just open a netbeans 7.1.2 project and create the missing project custom  library

GUEST: Background scanning of projects doesn
Comment 1 Exceptions Reporter 2012-11-09 15:11:21 UTC
Created attachment 127484 [details]
stacktrace
Comment 2 Svata Dedic 2012-11-09 15:17:52 UTC
Please check the processing of JEE metadata - see the threaddumps in the exception reports. The call chain from TLIndexer, which takes most of the time in those reports is:

TLindexer > errors (html) > MIME type > metadata > JSF library > facelets > JSFIndex > filesystems

may indicate bad performance, maybe some cache could be created, if the info is fetched repeatedly ?

Thanks.
Comment 3 Martin Fousek 2012-11-13 08:47:36 UTC
Jsf.editor or html.editor related, Marek will know better/faster the purpose and way to improve it.
Comment 4 Marek Fukala 2012-11-13 10:41:56 UTC
The sore point is the up-to-date check of the JSF libraries. It is necessary if we need 100% correct result, but OTOH some reasonable caching is apparently necessary.

...
org.netbeans.modules.web.jsf.editor.index.JsfIndex.getAllFaceletsLibraryDescriptors(JsfIndex.java:250)
org.netbeans.modules.web.jsf.editor.facelets.FaceletsLibrarySupport.checkLibraryDescriptorsUpToDate(FaceletsLibrarySupport.java:177)
org.netbeans.modules.web.jsf.editor.facelets.FaceletsLibrarySupport.getLibraries(FaceletsLibrarySupport.java:158)
...
Comment 5 windli 2013-02-11 15:56:41 UTC
It's a blocker issue. It's too painful to use an IDE with bug like this. I nearly switch to another IDE just for this issue. I have to for hours for it to finish in a big project.

I have to create a patch to continue use NetBeans.
Comment 6 windli 2013-02-11 16:02:45 UTC
Created attachment 131250 [details]
patch to web jsf editor JsfIndex.java to stop querying the index too much times

The issues here are getAllFaceletsLibraryDescriptors() and getAllCompositeLibraryNames() are called too many times during be background scanning even each call need not too much time.

This is not perfect patch, but it helps me continue use NetBeans as IDE.
You are fell free to improve.

Regards
Wind
Comment 7 windli 2013-02-11 16:10:34 UTC
Created attachment 131253 [details]
patched class file for you to try

You can unzip the attached file to get the patched class and use zip tools to replace org/netbeans/modules/web/jsf/editor/index/JsfIndex.class in enterprise/modules/org-netbeans-modules-web-jsf-editor.jar 

Regards
Wind
Comment 8 Martin Fousek 2013-02-12 15:36:45 UTC
Thanks a lot for attached patch. As in the case of the issue #226002, we will focus on that for the next release since the NB7.3 is on the way.

BTW, aren't you able to provide also testing project suitable for the performance measurement? I understand if it's not possible, I'm just trying that...
Comment 9 Marek Fukala 2013-02-12 15:41:26 UTC
Thank you for the patch, but I won't apply it. I already work on a little bit more effective solution and fill fix it today.
Comment 10 Marek Fukala 2013-02-12 17:11:11 UTC
fixed in web-main#b665cc9a0622

the original sore point - libraries consistency check called from FaceletsLibrarySupport.getLibraries() - now runs only if any of the JSF indexers reindexes a source root which belongs to the classpath of the corresponding JsfSupport.

I've slightly tested the functionality, haven't double checked the performance impact, but the original issue is now gone so I believe the performance will be much better. 

Still it'd be nice if anyone checked it.
Comment 11 Quality Engineering 2013-02-13 01:52:08 UTC
Integrated into 'main-golden', will be available in build *201302122300* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main-golden/rev/b665cc9a0622
User: Marek Fukala <mfukala@netbeans.org>
Log: #221837 - Background scanning of projects takes too long
Comment 12 windli 2013-02-13 05:56:33 UTC
Thanks for the fix.
It works great.
I just tried it with  a big project with 900+ java files and 700+ xhtml files. It was opened in 1 minutes for the first time. It just needs a few seconds to open/close the project after the first time.
Comment 13 vsiologas 2013-07-04 08:09:57 UTC
If anyone still has a problem with background scanning projects getting stuck, make sure that you don't have any almost unending directory trees in your project folders. You can imagine how this can cause the project scanner to get stuck.
Here is a quick script to delete the directory tree, as it is impossible to do it with dos/windows7 as it fails with an error "Source path too long" or "The source file name(s) are larger than is supported by the file system."

In my case the build/classes folder continued like this:
build/classes/build/classes/build/classes/build/classes/build/classes/build/classes/build/classes/build/classes/build/classes/build/classes/build/classes/build/classes/
, seemingly forever. 

Use this recursive batch (delRecur.bat)  to delete the tree:

-------------------------------------------------------

ren D:\Development\Projects\MyProject\build\classes1234\build x
ren D:\Development\Projects\MyProject\build\classes1234\classes x
move D:\Development\Projects\MyProject\build\classes1234\x\build D:\Development\Projects\MyProject\build\classes1234\
move D:\Development\Projects\MyProject\build\classes1234\x\classes D:\Development\Projects\MyProject\build\classes1234\

rd D:\Development\Projects\MyProject\build\classes1234\x /s /q

echo removed

delRecur.bat

-------------------------------------------------------
Comment 14 Martin Fousek 2013-07-12 13:12:25 UTC
*** Bug 210600 has been marked as a duplicate of this bug. ***
Comment 15 erkancengiz 2014-01-27 14:30:21 UTC
Created attachment 144415 [details]
Background process endless loop cause

This is the project folder that causes the background process to go in loop.
I recognised that the class colder repeats itself as follows

class/class/class/class/....../class/...
Comment 16 Martin Fousek 2014-01-31 08:49:26 UTC
(In reply to erkancengiz from comment #15)
> Created attachment 144415 [details]
> Background process endless loop cause
> 
> This is the project folder that causes the background process to go in loop.
> I recognised that the class colder repeats itself as follows
> 
> class/class/class/class/....../class/...

How are you able to create such build directory with recursive nesting of "classes" folders? Don't you have any link within your source/test directory or similarly?
Comment 17 Martin Fousek 2014-02-03 10:10:27 UTC
Reported please could complete requested information from #comment16 and reopen this issue? Thanks a lot.
Comment 18 _ hair 2014-05-22 21:01:40 UTC
I see it with our "sql" submodules.

A normal build doesn't do this, but netbeans generates directories like

<project>/sql/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target/classes/target


These sql subprojects use liquidbase.
 eg the liquibase-maven-plugin in maven, 
They also include "." as a resource directory.
Comment 19 Martin Fousek 2014-05-23 07:09:19 UTC
Sorry for closing as resolved/fixed, but the original issue was resolved as confirmed in the comment #12.

The additional comments and reopens are about recursively nested built artefacts which causes slowness. These issues should be reported against Maven/Ant project's support if it's cause by NetBeans.

The issue from comment #18 is the user issue. When you place resource folder into the project root, maven itself will recursively nest target folder - that's no issue of NetBeans IDE, that's your setup in pom.xml. If you would like to do something like this, you should exclude the target folder in this way:
...
<resources>
    <resource>
        <directory>.</directory>
        <filtering>true</filtering>
        <excludes>
            <exclude>./target</exclude>
        </excludes>
    </resource>
</resources>
...

Thanks for understanding.