I had this problem from NB versions 6.x, 7.X, and again with 8.0.
OK, the first fault is mine as I never reported it until today.

With default heap, and version 8, I have a outofmemoryerror after nb startup. Il I increase Xmx (>4Gb), I just spend tens of minutes in "background scanning of project", many times in the day, every days.

All my java projects are in /data/prj/java, and all my dépendencies are in /data/prj/java_lib.
I have many many other non java related files in /data/prj

I noticed that the "background scanning" thing is scanning *all* the files in /data/prj

I guess that it has something to do with my problem. But why is it looking in places outsides project directories ???

----
stdout of the outofmemory error : 
$ /data/prj/netbeans-8.0beta/bin/netbeans
...
Diagnostic information
Input arguments:
	-Djdk.home=/usr/java
	-Dnetbeans.default_userdir_root=/home/scachat/.netbeans
	-Dnetbeans.running.environment=gnome
	-Dnetbeans.dirs=/data/prj/netbeans-8.0beta/nb:/data/prj/netbeans-8.0beta/ergonomics:/data/prj/netbeans-8.0beta/ide:/data/prj/netbeans-8.0beta/extide:/data/prj/netbeans-8.0beta/java:/data/prj/netbeans-8.0beta/apisupport:/data/prj/netbeans-8.0beta/webcommon:/data/prj/netbeans-8.0beta/websvccommon:/data/prj/netbeans-8.0beta/enterprise:/data/prj/netbeans-8.0beta/mobility:/data/prj/netbeans-8.0beta/profiler:/data/prj/netbeans-8.0beta/python:/data/prj/netbeans-8.0beta/php:/data/prj/netbeans-8.0beta/identity:/data/prj/netbeans-8.0beta/harness:/data/prj/netbeans-8.0beta/cnd:/data/prj/netbeans-8.0beta/dlight:/data/prj/netbeans-8.0beta/groovy:/data/prj/netbeans-8.0beta/extra:/data/prj/netbeans-8.0beta/javacard:/data/prj/netbeans-8.0beta/javafx:
	-Dnetbeans.home=/data/prj/netbeans-8.0beta/platform
	-Dnetbeans.importclass=org.netbeans.upgrade.AutoUpgrade
	-Dnetbeans.accept_license_class=org.netbeans.license.AcceptLicense
	-XX:MaxPermSize=384m
	-Xmx1024m
	-Xss2m
	-Xms32m
	-XX:PermSize=32m
	-Dnetbeans.logger.console=true
	-ea
	-Dapple.laf.useScreenMenuBar=true
	-Dapple.awt.graphics.UseQuartz=true
	-Dsun.java2d.noddraw=true
	-Dsun.java2d.dpiaware=true
	-Dsun.zip.disableMemoryMapping=true
	-Dnetbeans.extbrowser.manual_chrome_plugin_install=yes
	-XX:+HeapDumpOnOutOfMemoryError
	-XX:HeapDumpPath=/home/scachat/.netbeans/8.0beta/var/log/heapdump.hprof
Compiler: HotSpot 64-Bit Tiered Compilers
Heap memory usage: initial 32,0MB maximum 910,5MB
Non heap memory usage: initial 34,4MB maximum 432,0MB
Garbage collector: PS Scavenge (Collections=14 Total time spent=0s)
Garbage collector: PS MarkSweep (Collections=2 Total time spent=0s)
Classes: loaded=7704 total loaded=7704 unloaded 0
INFO [org.netbeans.core.ui.warmup.DiagnosticTask]: Total memory 16 535 904 256
INFO [org.netbeans.modules.subversion]: Finished indexing svn cache with 2 863 entries. Elapsed time: 108 ms.
INFO [glassfish]: Requested Entity: public id = -//Sun Microsystems, Inc.//DTD Application Server 9.0 Resource Definitions //EN, system id = http://www.sun.com/software/appserver/dtds/sun-resources_1_3.dtd
INFO [glassfish]: Requested Entity: public id = -//Sun Microsystems, Inc.//DTD Application Server 9.0 Resource Definitions //EN, system id = http://www.sun.com/software/appserver/dtds/sun-resources_1_3.dtd
WARNING [org.netbeans.TopSecurityManager]: use of system property netbeans.user has been obsoleted in favor of InstalledFileLocator/Places at org.netbeans.modules.java.api.common.project.ProjectHooks$ProjectOpenedHookImpl.updateUserBuildPropertiesRef(ProjectHooks.java:573)
INFO [org.netbeans.libs.svnclientadapter]: Javahl client adapter is not available
Failed to load JavaHL Library.
These are the errors that were encountered:
no libsvnjavahl-1 in java.library.path
no svnjavahl-1 in java.library.path
no svnjavahl in java.library.path
java.library.path = /usr/java/jre/lib/amd64:/usr/java/jre/lib/i386::/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
INFO [org.netbeans.modules.subversion.client.SvnClientFactory]: JavaHL not available. Falling back on SvnKit.
INFO [org.netbeans.modules.subversion.client.SvnClientFactory]: svnClientAdapter running on svnkit
INFO [glassfish]: Requested Entity: public id = -//Sun Microsystems, Inc.//DTD Application Server 9.0 Resource Definitions //EN, system id = http://www.sun.com/software/appserver/dtds/sun-resources_1_3.dtd
INFO [glassfish]: Requested Entity: public id = -//Sun Microsystems, Inc.//DTD Application Server 9.0 Resource Definitions //EN, system id = http://www.sun.com/software/appserver/dtds/sun-resources_1_3.dtd
INFO [glassfish]: Requested Entity: public id = -//Sun Microsystems, Inc.//DTD Application Server 9.0 Resource Definitions //EN, system id = http://www.sun.com/software/appserver/dtds/sun-resources_1_3.dtd
INFO [org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater]: Resolving dependencies took: 1 665 ms
INFO [org.openide.filesystems.FileUtil]: Cannot read symbolic link /data/prj/java/TrafxRegul/testFiles/526a4074e1844
usage: InstallPlugins
WARNING [org.netbeans.TopSecurityManager]: use of system property netbeans.home has been obsoleted in favor of InstalledFileLocator/Places at org.netbeans.Clusters.relativeDirsWithHome(Clusters.java:137)
INFO [org.netbeans.modules.subversion]: Subversion.cleanupTask: Scanning in progress, trying again in 10 000ms
INFO [org.netbeans.modules.subversion.FileStatusCache]: FileStatusCache.refreshTask: Scanning in progress, trying again in 10 000ms
INFO [org.netbeans.modules.subversion]: Subversion.cleanupTask: Scanning in progress, trying again in 10 000ms
INFO [org.netbeans.modules.subversion.FileStatusCache]: FileStatusCache.refreshTask: Scanning in progress, trying again in 10 000ms
INFO [org.netbeans.modules.subversion.FileStatusCache]: FileStatusCache.refreshTask: Scanning in progress, trying again in 10 000ms
INFO [org.netbeans.modules.subversion]: Subversion.cleanupTask: Scanning in progress, trying again in 10 000ms
INFO [org.netbeans.modules.subversion]: Subversion.cleanupTask: Scanning in progress, trying again in 10 000ms
INFO [org.netbeans.modules.subversion.FileStatusCache]: FileStatusCache.refreshTask: Scanning in progress, trying again in 10 000ms
INFO [org.netbeans.modules.subversion.FileStatusCache]: FileStatusCache.refreshTask: Scanning in progress, trying again in 10 000ms
INFO [org.netbeans.modules.subversion]: Subversion.cleanupTask: Scanning in progress, trying again in 10 000ms
INFO [org.netbeans.modules.subversion.FileStatusCache]: FileStatusCache.refreshTask: Scanning in progress, trying again in 10 000ms
INFO [org.netbeans.modules.subversion]: Subversion.cleanupTask: Scanning in progress, trying again in 10 000ms
INFO [org.netbeans.modules.subversion.FileStatusCache]: FileStatusCache.refreshTask: Scanning in progress, trying again in 10 000ms
INFO [org.netbeans.modules.subversion]: Subversion.cleanupTask: Scanning in progress, trying again in 10 000ms
INFO [org.netbeans.modules.subversion.FileStatusCache]: FileStatusCache.refreshTask: Scanning in progress, trying again in 10 000ms
INFO [org.netbeans.modules.subversion]: Subversion.cleanupTask: Scanning in progress, trying again in 10 000ms
INFO [org.netbeans.modules.subversion]: Subversion.cleanupTask: Scanning in progress, trying again in 10 000ms
INFO [org.netbeans.modules.subversion.FileStatusCache]: FileStatusCache.refreshTask: Scanning in progress, trying again in 10 000ms
java.lang.OutOfMemoryError: GC overhead limit exceeded
Dumping heap to /home/scachat/.netbeans/8.0beta/var/log/heapdump.hprof ...
Heap dump file created [1375991915 bytes in 8,934 secs]
WARNING [org.netbeans.core.TimableEventQueue]: too much time in AWT thread org.netbeans.modules.sampler.InternalSampler@7c113d10
SEVERE [org.openide.util.Exceptions]
java.lang.OutOfMemoryError: GC overhead limit exceeded
	at java.io.UnixFileSystem.resolve(UnixFileSystem.java:108)
	at java.io.File.<init>(File.java:367)
	at org.netbeans.modules.masterfs.filebasedfs.naming.FileName.getFile(FileName.java:109)
	at org.netbeans.modules.masterfs.filebasedfs.naming.NamingFactory.getReference(NamingFactory.java:347)
	at org.netbeans.modules.masterfs.filebasedfs.naming.NamingFactory.registerInstanceOfFileNaming(NamingFactory.java:259)
	at org.netbeans.modules.masterfs.filebasedfs.naming.NamingFactory.registerInstanceOfFileNaming(NamingFactory.java:212)
	at org.netbeans.modules.masterfs.filebasedfs.naming.NamingFactory.fromFile(NamingFactory.java:91)
	at org.netbeans.modules.masterfs.filebasedfs.fileobjects.FileObjectFactory.create(FileObjectFactory.java:486)
	at org.netbeans.modules.masterfs.filebasedfs.fileobjects.FileObjectFactory.getOrCreate(FileObjectFactory.java:437)
	at org.netbeans.modules.masterfs.filebasedfs.fileobjects.FileObjectFactory.issueIfExist(FileObjectFactory.java:408)
	at org.netbeans.modules.masterfs.filebasedfs.fileobjects.FileObjectFactory.getFileObject(FileObjectFactory.java:226)
	at org.netbeans.modules.masterfs.filebasedfs.fileobjects.FolderObj.computeChildren(FolderObj.java:191)
	at org.netbeans.modules.masterfs.filebasedfs.fileobjects.FolderObj.getChildren(FolderObj.java:157)
	at org.openide.filesystems.FileObject$1WithChildren.process(FileObject.java:905)
	at org.openide.filesystems.FileObject$1WithChildren.process(FileObject.java:901)
	at org.openide.util.Enumerations$QEn.nextElement(Enumerations.java:434)
	at org.netbeans.modules.parsing.impl.indexing.ProxyBinaryIndexerFactory.supports(ProxyBinaryIndexerFactory.java:195)
	at org.netbeans.modules.parsing.impl.indexing.ProxyBinaryIndexerFactory.scanStarted(ProxyBinaryIndexerFactory.java:132)
[catch] at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$Work.binaryScanStarted(RepositoryUpdater.java:2799)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$AbstractRootsWork.scanBinary(RepositoryUpdater.java:4972)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$AbstractRootsWork$1.apply(RepositoryUpdater.java:4917)
	at org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater$AbstractRootsWork$1.apply(RepositoryUpdater.java:4914)
	at org.netbeans.modules.parsing.impl.indexing.IndexBinaryWorkPool$Task.call(IndexBinaryWorkPool.java:202)
	at org.netbeans.modules.parsing.impl.indexing.IndexBinaryWorkPool$Task.call(IndexBinaryWorkPool.java:185)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
	at org.openide.util.RequestProcessor$Task.run(RequestProcessor.java:1423)
	at org.openide.util.RequestProcessor$Processor.run(RequestProcessor.java:2033)
WARNING [org.netbeans.JarClassLoader]: Opening /data/prj/netbeans-8.0beta/platform/modules/org-netbeans-core.jar took 1 898 ms
WARNING [org.netbeans.core.TimableEventQueue]: no snapshot taken
INFO [org.netbeans.modules.mercurial]: version: null
WARNING [org.netbeans.JarClassLoader]: Opening /data/prj/netbeans-8.0beta/nb/modules/locale/org-netbeans-modules-autoupdate-ui_nb.jar took 1 428 ms
INFO [org.netbeans.core.ui.focus]: External Changes Refresh on focus gain disabled

Please upload zipped heap dump (/home/scachat/.netbeans/8.0beta/var/log/heapdump.hprof.old) here: <http://deadlock.netbeans.org/job/upload/build>. Thanks.

Done !

Thanks for the heap dump. 
You are right that NetBeans scans your /data/prj folder. There is 1,200,000 FileObjects on the heap and the most of them refers to files in /data/prj folder. It looks like /data/prj folder (ClassPath$Entry#3215) is part of ClassPath#205 and this is the reason, why it is scanned by NetBeans. Reassigning to java/project for further investigation why /data/prj folder is part of ClassPath#205.

*** Bug 241217 has been marked as a duplicate of this bug. ***

The indexing never touches anything outside of project's classpath.
>(ClassPath$Entry#3215) is part of ClassPath#205
The /data/prj/ folder is a part of the project's classpath.
What kind of project are you using (Ant or Maven) (J2SE or Web)?
Can you attach the project metadata (for ant based project the nbproject folder)?
Thanks!

Created attachment 145213 [details]
main project metadata (the problem occurs even when only this project is open)

I've analysed the heap dump and found the problem.
The problem is not in the project which metadata were attached but in its dependency (/home/prj/java/oneWire/).
The oneWire sources (/home/prj/java/oneWire/src) have configured endorsed classpath, you can find it in the /home/prj/java/oneWire/nbproject/project.properties file.
endorsed.classpath=${file.reference.javaws.jar}:${file.reference.javaws.jar-1}
The problem is with reference: ${file.reference.javaws.jar-1}
which is overridden in the /home/prj/java/oneWire/nbproject/private/private.properties to point to /data/prj.
The private.properties have higher priority to project.properties.

The fix is easy just delete the broken reference file.reference.javaws.jar-1=...
from the /home/prj/java/oneWire/nbproject/private/private.properties file.

Thanks a lot for this resolution. 

Is there a way to find the classpath value from the nb ui, so we can fix this type of problem ourself ?

Yes.
In the Properties of the java file there is ClassPath section having Boot ClassPath & Compile Classpath. The /data/prj should be found in the Boot ClassPath. In general a source root + dependencies (boot + classpath) are scanned.