This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 192259 - Remote SFTP mount endless project scanning
Summary: Remote SFTP mount endless project scanning
Status: RESOLVED FIXED
Alias: None
Product: platform
Classification: Unclassified
Component: Filesystems (show other bugs)
Version: 7.0
Hardware: PC Linux
: P2 normal (vote)
Assignee: Jaroslav Tulach
URL:
Keywords: PERFORMANCE
Depends on:
Blocks:
 
Reported: 2010-11-22 08:53 UTC by achapman
Modified: 2011-07-29 13:17 UTC (History)
2 users (show)

See Also:
Issue Type: DEFECT
Exception Reporter:


Attachments
snapshot of stuck project scanning (28.59 KB, application/octet-stream)
2010-11-22 18:03 UTC, achapman
Details
IDE Exception (3.10 KB, text/plain)
2011-06-28 13:08 UTC, alex707
Details

Note You need to log in before you can comment on or make changes to this bug.
Description achapman 2010-11-22 08:53:34 UTC
I have a gvfs mounted SFTP connection to a remote server and have built a project using this as a local path. 

Whenever I start Netbeans 7.0 is just scans this project endlessly. Eventually the IDE becomes totally unresponsive.

This isn't happening on 6.9.x .... 

I prefer using a mounted directory to a remote server instead of creating a remote connection because I don't want a synced copy of all the source files locally as this is problematic with more than one person working on the sources.

The remote directory is rather large but since I use the settings from my 6.9.x folder for 7.0 I don't understand why it wants to go through the entire scanning process again.
Comment 1 Petr Pisl 2010-11-22 13:36:00 UTC
Please provide a snapshot of the scanning. How you can do it is described here: http://blogs.sun.com/netbeansphp/entry/is_the_ide_slow_send

Also could you try to run NB 7.0 with clean user dir? http://blogs.sun.com/netbeansphp/entry/userdir . Just to be sure that the issue is not in importing settings from NB 6.9. 

Anyway it's not good approach to work through network, it's always many time slower than local copy.
Comment 2 achapman 2010-11-22 18:03:59 UTC
Created attachment 103199 [details]
snapshot of stuck project scanning

Here is a snapshot, as requested.

The project has been scanning for 4 hours now and I have shut down the IDE.

I started with a new profile and replaced OpenJDK with Sun JDK and still the same behaviour.
Comment 3 Petr Pisl 2010-12-10 10:01:53 UTC
Where so you have user dir of nb? On the local disk or on the network one?

Could you try to switch off autoscanning? Tools -> Options -> Miscellaneous -> Files -> Enable auto-scanning of seources. 

According the snapshot the file based operations are too slow. For example 1100 ms took check whether a file object exist, which is really slow. 

One question, why you don't use a VCS instead sharing network drive? I thought that sharing network drive is not used for team development anymore for 10 years.
Comment 4 Petr Pisl 2011-01-20 14:16:57 UTC
I'm marking this issue as incomplete. We should have more data. Please reopen, if you have still the problem.
Comment 5 alex707 2011-06-28 05:59:58 UTC
7 months and countless updates later and this is still not resolved. Is there anyone who can take this seriously? There are users who still use SFTP mounts
Comment 6 Petr Pisl 2011-06-28 09:15:39 UTC
If I understand correctly, you mount remote folder through SFTP mount and you develop directly on the remote server. According the attached snapshot, the main problem is in the file system responsive time, which is very slow (according the snapshot). We can not do anything with this. I don't understand how we could make faster your remote file system. 

You can probably use Scan on Demand http://wiki.netbeans.org/ScanOnDemand, that can help.

Also important is, where do you have the userdir, which I have asked in December 2010. Because you didn't answered, I closes the issue in January 2011. So 7 mount was the issue closed, this is why nobody took care about this.


If you really develop on remote FS, I'm marking this as want fix, because I don't really know how to fix it.
Comment 7 Petr Pisl 2011-06-28 09:18:19 UTC
One more question. Are there a simlinks in the source structure? If yes, is there a simlink recursive?
Comment 8 alex707 2011-06-28 13:07:28 UTC
There are simlinks, not recursive, the problem is that 6.9.x works fine on mounted drives, so it's obviously a problem that is on 7.0. 
After the latest updates the IDE Log reports an error and scanning projects just hangs there until it becomes unresponsive. Attached the error from the log.
Comment 9 alex707 2011-06-28 13:08:47 UTC
Created attachment 109150 [details]
IDE Exception
Comment 10 alex707 2011-06-28 13:39:19 UTC
Userdir is stored locally on the windows partition.
Comment 11 Petr Pisl 2011-06-28 14:23:57 UTC
The exception explain probably the root of the problem. It looks like native windows file system watcher (it was added into NB 7.0 in November (my guess)) is trying to register a listener to listen file events. 

Jardo, the windows watcher shouldn't be registered on a remote disks. IMHO you are not getting the events anyway from a remote FS. 

To alex707: I'm sorry, I didn't noticed that you are not the original reporter:), but the bug was close six month, because I didn't have enough informations.  Thanks for the exception.

Could you run NB 7.0 with property org.netbeans.modules.masterfs.watcher.disable=true ? It should disable registering the watcher. You can run NB from command line with -J-Dorg.netbeans.modules.masterfs.watcher.disable
=true or add it into the netbeans.conf file, as one of the netbeans_default_options. Let us know, whether it helps.
Comment 12 alex707 2011-06-30 05:42:19 UTC
I added the line in the conf file and it solved the thrown exception. It no longer becomes unstable so i'm guessing a memory leak was involved. First time the project was scanned it took quite a while:
 
Complete indexing of 4 source roots took: 3403969 ms (New or modified files: 6795, Deleted files: 0) [Adding listeners took: 59380 ms]

But now it's quite ok:

Complete indexing of 3 source roots took: 141805 ms (New or modified files: 6, Deleted files: 4) [Adding listeners took: 67369 ms]

I am however seeing that NB 6.9.1 is many times faster than NB 7.0 when it comes to scanning the project. Any ideas why this happens?

Anyway your solution solved the problem, for a more permanent solution perhaps a check might be implemented whether the listener can be registered on the sources to avoid that crash.

Thank you for your time, Petr.
Comment 13 Petr Pisl 2011-06-30 12:58:54 UTC
This bug should be fixed in the Windows watcher, so I'm reopening this. Guys from file systems should take care about this.

Now I don't know why it's so slower against 6.9. Could you please write how many files and which file types are there? I mean how many .php, .html, .js etc files?
Comment 14 Jaroslav Tulach 2011-06-30 14:24:01 UTC
Workaround exists, downgrading to P2.

(In reply to comment #9)
> Created an attachment (id=109150) [details]
> IDE Exception

Is the exception thrown just once or were there multiple instances of the same exception in your log file?
Comment 15 alex707 2011-06-30 15:08:04 UTC
(In reply to comment #14)
> Workaround exists, downgrading to P2.
> 
> (In reply to comment #9)
> > Created an attachment (id=109150) [details] [details]
> > IDE Exception
> 
> Is the exception thrown just once or were there multiple instances of the same
> exception in your log file?

In the saved log it seems to be just once, but I didn't wait until it became unresponsive so I don't know if any more exceptions were thrown. After I noticed the exception I saved the log and closed the IDE.
Comment 16 alex707 2011-06-30 15:26:25 UTC
(In reply to comment #13)
> This bug should be fixed in the Windows watcher, so I'm reopening this. Guys
> from file systems should take care about this.
> 
> Now I don't know why it's so slower against 6.9. Could you please write how
> many files and which file types are there? I mean how many .php, .html, .js etc
> files?

Complete indexing of 4 source roots took: 3403969 ms (New or modified files:
6795, Deleted files: 0) [Adding listeners took: 59380 ms]

The above is the first indexing of the project. There were actually 2 open projects but one was a small local test project with a few files. The main project contains a total of 6795 files(without svn files): 1956 .html files 369 .php files 769 .js files and others the rest (images, css, txt, modules, includes a.s.o.)
Comment 17 Jiri Rechtacek 2011-07-11 11:53:04 UTC
No disagreement with waiving this one, it's approved.
Comment 18 Jaroslav Tulach 2011-07-29 13:17:00 UTC
If the  exception is reported only once (Alex you can disagree), then there is not much to fix, I guess.