This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
I was checking netbeans performance after uninstalling many unneeded plugins that are on by default and I found that for some reason, java process started by netbeans seems to be scanning lots of odd directories. For example my skype directory with chat logs, then Chrome's preferences, cookies, history etc. It also acesses the ntuser.dat constantly, but since it's part of the registry I suppose it could be somewhere deep in the java logic that needs to check and keep internally in sync all the system preferences. But all that chrome and skype directory scanning seems rather fishy... Is this somehow intentionally done in netbeans and if yes, how can I disable it: first for security reasons and secondly for performance hit of scanning directories that are not relevant to writing PHP code in IDE and pressing save. I did not do any php debugging or anything at the time of screenshot, just waited for a bit and that activity showed up in procmon. And the PID of javaw is exact same as the netbeans's process, so this is not some other rogue process on the system.
Created attachment 107872 [details] skype directory accessing, chrome directory accessing
same as for bug 197925 ... should be evaluated carefully
It is certainly not a normal behaviour and not done intentionally. There is not much that can be investigated from just one screenshot of procmon - please provide more information - It would be best to use the self-sampling tool which is on the memory toolbar to sample the ide activity while this is happening and send us the snapshot. Also the messages.log file would be nice to have. For now until there is more information closing as INCOMPLETE
Created attachment 108521 [details] cpu spanshot of netbeans
Created attachment 108522 [details] log from netbeans log folder
Created attachment 108523 [details] another example of javaw.exe accessing skype and chrome directories
I provided logfiles and porfiler snapshot and also new screenshot from procmon. Is there anything else needed that I can provide ?
The snapshot does show anything which can explain this issue. So next logical explanation is that javaw.exe is not NetBeans. In fact NetBeans java process is reported as netbeans.exe on Windows in most cases. Can you check (for example via Java VisualVM) NetBeans process id. Lowering priority to P3. I don't see a reason to have it as P1.
As you can see from the screenshot, PID for the process that accesses skype and chrome folders is the same one that keeps reading in netbeans's uihandler.properties. netbeans.exe on my machine just starts up javaw.exe and then later on uses 888KB memory. So the main netbeans process is in that javaw.exe.
(In reply to comment #9) > As you can see from the screenshot, PID for the process that accesses skype and > chrome folders is the same one that keeps reading in netbeans's > uihandler.properties. This looks to me like general scan, so accessing uihandler.properties does not really mean that this is NetBeans. > netbeans.exe on my machine just starts up javaw.exe and then later on uses > 888KB memory. So the main netbeans process is in that javaw.exe. Maybe because you are using 64bit JVM and netbeans.exe is 32bit application. It would be great if you can provide steps how to reproduce it, since the snapshot does not provide any clue. Can you try to reproduce it on different computer?
This is 100% netbeans hosting java runtime. I don't have multiple javaw processes in processlist and that one the screenshot is also the only one that appears after I have started netbeans. netbeans.exe itself stays at 888KB ram usage while javaw.exe consumes 150MB-500MB RAM, depending what I have open in netbeans. And when I close the netbeans, that javaw.exe process goes away. So according to deduction logic, that javaw.exe is the process that actually runs netbeans, netbeans.exe is just a bootstrap exe to start that javaw process. Also constantly opening uihandler.properties is another netbean's issue (which I have reported in), it occurs after some intervals PLUS it occurs always when I switch back to netbeans from another window. So possibly there is some timer involved, that causes netbeans to read in uihandler.properties after certain intervals PLUS there is some logic that reads in uihandler when netbeans main window gains focus. Similar case was with spellchecker plugin in netbeans, which kept checking directories for language files on each main window activation + after some certain interval. So, IMHO, netbeans code does something funky: keeps readin in properties files after certain intervals and on main window activation. PLUS the same runtime for some reason accesses skype and chrome directories. Is there some certain way I can get more detailed process dump from netbeans to send to you ? One that would list exactly what files were opened and why. Or some way to make netbeans log more information to the logfiles ? I'll try to check for similar behaviour on a 32bit win7 machine also, will be reporting back in few hours.
(In reply to comment #11) > This is 100% netbeans hosting java runtime. > I don't have multiple javaw processes in processlist and that one the > screenshot is also the only one that appears after I have started netbeans. > netbeans.exe itself stays at 888KB ram usage while javaw.exe consumes > 150MB-500MB RAM, depending what I have open in netbeans. And when I close the > netbeans, that javaw.exe process goes away. So according to deduction logic, > that javaw.exe is the process that actually runs netbeans, netbeans.exe is just > a bootstrap exe to start that javaw process. VisualVM will tell you which java processes are running and you can be sure which process is NetBeans process. > Is there some certain way I can get more detailed process dump from netbeans to > send to you ? Maybe a heap dump will help to discover what is going on. > One that would list exactly what files were opened and why. Or > some way to make netbeans log more information to the logfiles ? It could be possible on Solaris, but on Windows it will require modification to JDK. I would like to have reproducible test case. This way we can take a look at it and we can also be sure that the issue is fixed.
Created attachment 108534 [details] screen of visualvm Hm, perhaps this is the culprit: NotifyChangeDirectory The parameter seems to be always drive C:\ I closed Skype and skype directory related events went away. But the events themselves do indeed come from javaw.exe, skype.exe generates its own file access events. As I understand, java for some reason asks to get all directory change notifications for drive C:\, thus causing it to check every file that gets reported. And after each chek it triggers "NotifyChangeDirectory C:\" again. And so it goes around and around for each and every file that changes on C:\ disk. As seen from visualvm, the PID belongs indeed to netbeans's process. Although for some reason visualvm thinks netbeans to be 6.9+, and not 7.0.
Created attachment 108535 [details] snapshot from visualvm
(In reply to comment #13) > Created an attachment (id=108534) [details] > screen of visualvm > > Hm, perhaps this is the culprit: NotifyChangeDirectory > The parameter seems to be always drive C:\ > I closed Skype and skype directory related events went away. But the events > themselves do indeed come from javaw.exe, skype.exe generates its own file > access events. > > As I understand, java for some reason asks to get all directory change > notifications for drive C:\, thus causing it to check every file that gets > reported. And after each chek it triggers "NotifyChangeDirectory C:\" again. > And so it goes around and around for each and every file that changes on C:\ > disk. Yes, NotifyChangeDirectory C:\ is the root of the problem. It explains why you see access of skype related files, when skype is running. Those are the files changed by skype and notified by NotifyChangeDirectory c:\ to netbeans, which in turn checks the changed files. > As seen from visualvm, the PID belongs indeed to netbeans's process. Yes, it does. > Although > for some reason visualvm thinks netbeans to be 6.9+, and not 7.0. 6.9+ covers also 7.0. So you know why the files are accessed but we still don't know why there is NotifyChangeDirectory C:\ - this is definitely wrong. With the information you have, can you put together a step-by-step test case, so we can reproduce it. Thanks.
One more thing: if the theory is correct. The following system property should disable file notifications. Run netbeans with -J-Dorg.netbeans.modules.masterfs.watcher.disable=true
Yes, that option improved the behaviour. I tested around a bit and hopefully got a few decent test cases: CASE 1 1) start netbeans with no projects open: no NotifyChangeDirectory events should happen 2) open up a PHP project: NotifyChangeDirectory events start occuring for C:\ 3) close PHP project: NotifyChangeDirectory remain happening for C:\ CASE 2 1) start netbeans with no projects open: no NotifyChangeDirectory events should happen 2) open up a Java project (in my case java applet): no NotifyChangeDirectory events should happen 3) close Java project: no NotifyChangeDirectory events should happen CASE 3 1) start netbeans with no projects open: no NotifyChangeDirectory events should happen 2) open up a Java project (in my case java applet): no NotifyChangeDirectory events should happen 3) close Java project: no NotifyChangeDirectory events should happen 4) open up a PHP project: NotifyChangeDirectory events start occuring for C:\ 5) close PHP project: NotifyChangeDirectory remain happening for C:\ CASE 4 -J-Dorg.netbeans.modules.masterfs.watcher.disable=true: 1) start netbeans with no projects open: no NotifyChangeDirectory events should happen 2) open up a PHP project: no NotifyChangeDirectory events should happen 3) close PHP project: no NotifyChangeDirectory events should happen So, common case is opening up a PHP project. It does not matter if the project has remote connection (auto save to server) set up or not.
PHP is doing something wild that Java does not? Interesting.
I have played with it and I'm able to reproduce it with NetBeans 7.0.1 JavaSE build. Just I have created new Java project and then the Process Monitor claims that NetBeans are notified about changes of files that are not relevant to a project or NetBeans at all. Basically all changes on the disk C: are announced to NetBeans which is probably OK, but NetBeans are doing some operation with the files. Every file event from the Windows system is proceed by the WindowsNotifier class, where in the notify() method asks whether the event is about file or directory. So for every file event is executed isDirectory() and getAbsolutePath(). The method File.isDirectory() touches the disk, this is probably why we see in the Process Monitor the file operations.
*** Bug 193539 has been marked as a duplicate of this bug. ***
Thanks for the investigation. Yes, calling isDirectory is really inefficient and I will remove it.
*** Bug 195288 has been marked as a duplicate of this bug. ***
core-main#8673d8210e02
Merged to release701 branch as 2284393a5a82
Ok, so the patch can be seen at: http://hg.netbeans.org/core-main/rev/8673d8210e02 Judging from the patch, the isDirectory() is removed and getAbsolutePath() is replaced with getPath() But, IMHO, the core problem still remains. The problem that netbeans is monitoring all files on C:\ disk in the first place. All my projects are on D:\ disk. Only relevant thing to netbeans on C:\ disk is netbeans' installation itself and configuration/log files in AppData folder. After checking the code, it is visible that all notified file object are kept in some sort of cache as judged from lines: FileObject fo = factory.getCachedOnly(file); and map.containsKey(fo); Is this correct: for every event netbeans caches the file information internally and stores in a map ? So when I happen to move around few thousands files on C:\ then the memory consumed by netbeans goes up a bit for every file ? Is this mainly used to detect external changes in project files ? If yes, then how come public @Override Void addWatch(String path) gets called with C:\ instead of D:\Projects\php\projectfolder ?? Passing in the path for the project root would keep the notifications minimal and 100% relevant. Right now the isDirectory() removal fixes the file locking issue as seen on other bugs marked as duplicate of this, but even with that fix, netbeans is still asking notifications for too many files (and in my case also on the wrong disk)... Also, after I closed the opened project, netbeans did not stop watching C:\. Correct me if I'm wrong but shouldn't the logic be like that: 1) open project X that is located in D:\Projects\PHP\projectX 2) netbeans tells windows to notify itself of changes to that directory, D:\Projects\PHP\projectX 3) close project X 4) netbeans tells windows to stop notifying itself of changed to that directory, D:\Projects\PHP\projectX Right now the behaviour seems to be (incorrectly/suboptimally) like this: 1) open project X that is located in D:\Projects\PHP\projectX 2) netbeans tells windows to notify itselt of changes to C:\ instead, getting irrelevant file events, not related to project at all 3) close project X 4) netbeans does not tell windows to stop the notifications about C:\ and keeps receiving them (purely unnecessary IO) Should theis be createad as a new bug ?
I probably know why is checking in your case also dick C:. When you want to find a project where belongs file xyz, the implementation is going through SimpleFileOwnerQueryImplementation.getOwner(). This call ProjectManager.findProject(folder) and this adds listener for the folder, where the project should be find. It doesn't matter whether there is project or not. If there is no project, then SimpleFileOwnerQueryImplementation.getOwner() call the ProjectManager.findProject(folder.getParent()), for the parent of the previous folder. This is done until is not a project find or the root of the FS is reached. The php project has on classpath signature files to the editor provide code completion for php runtime staff. These signature files are in a zip file, that is located in the NetBeans installation. When is composed the classpath for the project, then is called the SimpleFileOwnerQueryImplementation also for this zip file. For the zip file the project is not find and then a listener is add for C:\. I don't know how to solve this now. IMHO there is wrong the SimpleFileOwnerQueryImplementation. But I can be wrong.
(In reply to comment #26) > SimpleFileOwnerQueryImplementation.getOwner(). This call > ProjectManager.findProject(folder) and this adds listener for the folder, where > the project should be find. It doesn't matter whether there is project or not. This looks like a bug in ProjectManager: core-main #d5e6902b5920 Anyway this listener is not recursive and so is not currently implemented using native watches anyway (bug #197985).
(In reply to comment #25) > But, IMHO, the core problem still remains. The problem that netbeans is > monitoring all files on C:\ disk in the first place. All my projects are on D:\ > disk. Only relevant thing to netbeans on C:\ disk is netbeans' installation > itself and configuration/log files in AppData folder. The core of the P1 problem and its duplicates is that people cannot work with files due to NetBeans touching them instantly. This one is hopefully fixed by my patch. > Should theis be createad as a new bug ? You can try to report a bug, but with the changes done for bug 197985, the current state may be the most effective: It may be cheaper to throw away unrelated events than add/remove new directory watchers all the time.
(In reply to comment #28) > You can try to report a bug, but with the changes done for bug 197985, the > current state may be the most effective: It may be cheaper to throw away > unrelated events than add/remove new directory watchers all the time. May be cheaper or IS cheaper ? If there are constant changes to the C:\ or whatever partition (installing some program, bunch of regitry file accesses (ntuser.dat) and writing files to multiple location) then netbeans get notifixations for all of them, has to process them etc. The OS has to also spend some effort to check event pollers and keep sending in all those events (proably not much, but still something) and netbeans/java has to handle those events and check against the projects. Judging from the process, wouldn't it be more efficient if the OS itself would decide not so send in event for unrelated files, thus eliminating the event sending to program and event handling in the program. How many directory watchers do currently get added/removed all the time ? I don't know for sure, but shouldn't the process be like: add watcher for project dir when opening a project, remove that watcher when closing project. Not sure how many people keep opening and closing projects all the time... I think I'll wait for when the fixes get into update and test the things again on my machine so I will have stats to back up my story.
*** Bug 198492 has been marked as a duplicate of this bug. ***
Integrated into 'main-golden', will be available in build *201106021001* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main/rev/8673d8210e02 User: Jaroslav Tulach <jtulach@netbeans.org> Log: #197927: Don't call isDirectory (or other attribute queries) unless we know we watch given file
sazzer, ynov, cschlichtherle, musilt2, bushychris, velodiver - anyone able to consistently reproduce the problem in 7.0 who can try to verify the fix in build 201106021001?
> This looks like a bug in ProjectManager: core-main #d5e6902b5920 Should this also be transplanted to 7.0.1?
changeset: 199746:2284393a5a82 branch: release701 parent: 199742:3c04a00990b3 parent: 199741:8673d8210e02 user: Jaroslav Tulach <jtulach@netbeans.org> date: Wed Jun 01 15:15:41 2011 +0200 summary: Merge of #197927 to release701 branch
I don't understand, is your comment a reply to my previous comment? Changeset 2284393a5a82 in release701 does not correspond to changeset core-main #d5e6902b5920 that I was inquiring about. Does the release701 branch contain a changeset corresponding to core-main #d5e6902b5920 ? Thanks.
TM=Dev until this is clarified.
This bug is not about ProjectManager, but File.isDirectory() call for every changed file on disk. You can create another bug for ProjectManager, if you think it is important.
Well, Jesse (who made this change) should tell whether it's important for 7.0.1.
(In reply to comment #38) > Well, Jesse should tell whether it's important for 7.0.1. Sure, Jesse can express his opinion. However the truth is that his change does not affect the "nosing around" at all. It is completely unrelated and associated with this issue just by accident (or rather because we did not understand the real cause then).
Ok, thanks for explaining.
(In reply to comment #38) > Jesse (who made this change) should tell whether it's important for 7.0.1. I do not think so; it is a bug in 7.0.1 but as mentioned in comment #27 it has nothing to do with native listeners in that release (if not fixed in 7.1 it _would_ have involved native listeners, due to bug #197985). Whether there is any measurable performance penalty in 7.0.1, I have no idea - the issue only came up while debugging something else.