Bug 207189 - Parsing API needs to tell recursive listener which folders to skip
Parsing API needs to tell recursive listener which folders to skip
Status: RESOLVED FIXED
Product: platform
Classification: Unclassified
Component: Filesystems
7.1
All All
: P1 (vote)
: 7.2
Assigned To: Jaroslav Tulach
issues@platform
EXCEPTIONS_REPORT
: API_REVIEW_FAST
Depends on:
Blocks: 207214 209557
  Show dependency treegraph
 
Reported: 2012-01-11 19:23 UTC by MartySchechtman
Modified: 2012-04-19 10:01 UTC (History)
3 users (show)

See Also:
Issue Type: DEFECT
:


Attachments
stacktrace (2.72 KB, text/plain)
2012-01-11 19:23 UTC, MartySchechtman
Details
Using SAM interface for FileUtil.addRecursiveListener (12.22 KB, patch)
2012-04-05 18:54 UTC, Jaroslav Tulach
Details | Diff
I'd like to integrate tomorrow (17.76 KB, patch)
2012-04-10 11:15 UTC, Jaroslav Tulach
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description MartySchechtman 2012-01-11 19:23:33 UTC
Build: NetBeans IDE 7.1 (Build 201112071828)
VM: Java HotSpot(TM) 64-Bit Server VM, 20.4-b02-402, Java(TM) SE Runtime Environment, 1.6.0_29-b11-402-11M3527
OS: Mac OS X

User Comments:
martyschechtman: I had just started the application.




Stacktrace: 
java.lang.OutOfMemoryError: Java heap space
   at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:232)
   at java.lang.StringCoding.encode(StringCoding.java:272)
   at java.lang.String.getBytes(String.java:946)
   at java.io.UnixFileSystem.getBooleanAttributes0(UnixFileSystem.java:0)
   at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:228)
   at java.io.File.isDirectory(File.java:754)
Comment 1 MartySchechtman 2012-01-11 19:23:37 UTC
Created attachment 114809 [details]
stacktrace
Comment 2 Jaroslav Tulach 2012-01-12 09:49:21 UTC
The last messages in your log say:
INFO [org.netbeans.modules.uihandler.Installer]: heap dump was not created at /Users/marty/.netbeans/7.1/var/log/heapdump.hprof
INFO [org.netbeans.modules.parsing.impl.indexing.RepositoryUpdater]: Indexing of: file:/usr/local/www/brendahereth/ took: 661903 ms (New or modified files: 1, Deleted files: 0) [Adding listeners took: 661718 ms]


First of all, it would be good to have a heap dump to know what objects are wasting your memory.

The only suspect that comes to my mind is content of /usr/local/www/brendahereth/ what kind of disk is that? Local/remote? How much directories is under it? It seems that just listing the files took 11 min, so there must be something wrong in there.

CCing Tomáš, to see if the problem is related to MacOSX native file listener implementation.
Comment 3 MartySchechtman 2012-01-12 15:56:20 UTC
Hi Jaroslav,


/usr/local/www/brendahereth is a folder on my local machine.


It is the root of a website I'm developing.


I'm using a softlink to a directory on a remote drive.  The softlink is named images.  The actual directory is huge!


But I have the folder (/usr/local/www/brendahereth/images) marked as one for Netbeans to ignore.


To reduce the scanning time, I deleted the softlink.  And that did the trick.  There is no delay with the scanning when Netbeans starts.  But this not a good solution, because when I debug, I need access to those image files.


So I'm wondering if there is problem telling Netbeans to ignore folders that are actually softlinks.


Thanks very much for your help!
Comment 4 Tomas Zezula 2012-01-12 16:00:04 UTC
For java project you can set up excludes in project properties/sources to exclude the soft linked folder.
But I am afraid it does not help with recursive listener as it does not use this info (only Parsing.API and project system is using it).
Comment 5 Jaroslav Tulach 2012-01-27 09:30:15 UTC
The original assumption was that the cause of these problems are cyclic symlinks, which recursive listener does not handle well. That has been fixed in ergonomics#3997f88e67a1.

But the problem with ignored folder (which is not being ignored) is slightly different topic. Tomáš needs an API to tell the recursive listener what folders to ignore as well. Versioning systems have such API, but it would not work on remotefs, so we probably need new one.

What about:
FileObject.addRecursiveListener(FileChangeListener, Callable<Boolean>, Set<FileObject> ignoredFolders)
where javadoc will just say it calls only "get" on the Set - e.g. it can be computed dynamically.
Comment 6 Tomas Zezula 2012-01-27 13:10:25 UTC
Such an API is fine for me.
Comment 7 Jesse Glick 2012-02-13 17:49:00 UTC
(In reply to comment #5)
> FileObject.addRecursiveListener(FileChangeListener, Callable<Boolean>,

What is the Callable<Boolean>?

> Set<FileObject> ignoredFolders)
> where javadoc will just say it calls only "get" on the Set - e.g. it can be
> computed dynamically.

Better to have a lambda-friendly SAM interface (FileObject) -> boolean; and true should mean included vs. false for excluded, since negatively phrased APIs are harder to understand:

ClassPath.Entry e = ...;
e.getRoot().addRecursiveListener(listener, (folder) -> e.includes(folder));
Comment 8 Quality Engineering 2012-02-14 02:32:25 UTC
Integrated into 'releases', will be available in build *201202132200* or newer. Wait for official and publicly available build.
Changeset: http://hg.netbeans.org/releases/rev/db6e571452ee
User: Jaroslav Tulach <jtulach@netbeans.org>
Log: #207189: Avoiding multiple listeners due to cyclic symlinks
Comment 9 Jaroslav Tulach 2012-04-05 18:54:11 UTC
Created attachment 117906 [details]
Using SAM interface for FileUtil.addRecursiveListener

Not much javadoc and no versioning, but the functionality should be there. If it works, I polish it and integrate during next week.
Comment 10 Tomas Zezula 2012-04-06 07:34:14 UTC
The patch is OK for me.
Thanks Jardo
Comment 11 Jaroslav Tulach 2012-04-10 11:15:43 UTC
Created attachment 118045 [details]
I'd like to integrate tomorrow
Comment 12 Jesse Glick 2012-04-10 23:43:09 UTC
Looks OK to me.
Comment 13 Jaroslav Tulach 2012-04-11 16:17:52 UTC
ergonomics#c1caafa64bd8
Comment 14 Quality Engineering 2012-04-19 10:01:18 UTC
Integrated into 'main-golden', will be available in build *201204190400* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main-golden/rev/c1caafa64bd8
User: Jaroslav Tulach <jtulach@netbeans.org>
Log: #207189: addRecursiveListener with FileFilter


By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo