This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
Steps to reproduce: - copy a new folder with 1000 java files into the open project (in a external file manager) - refresh the project in Files view - expand the newly added folder w/ 1000 javas - delete the folder w/ 1000 javas Dell Precision 220, PIII 800MHz, 512MB RAM, Windows 2000, JDK 1.4.2_04 trunk 200405131800: 6406ms, 6375ms, 6125ms refactoring 040513: 15750ms, 15031ms, 15109ms With the refactoring build the contents of the folder is repainted as deleted in Files view after approx. 6 seconds but an exception is raised (see the attachment) and the delete action continues for next 9 seconds and only then the folder's node is deleted in the Files view. The whole delete action thus takes 15 seconds.
Created attachment 14870 [details] AIOOBE exception
AIOOBE filed as a separate issue 43280. Regarding your numbers: Deleting java file will take longer in refactoring build, because we must delete also Resource from java metamodel to ensure consistency.
suggestion: perhaps you can quickly mark the deleted entries as invalid and do the real update in the background later.
OK. I move metamodel update to background thread.
Metamodel update moved to background thread. Tondo, can you provide fresh numbers? Checking in RepositoryUpdater.java; /cvs/java/javacore/src/org/netbeans/modules/javacore/Attic/RepositoryUpdater.java,v <-- RepositoryUpdater.java new revision: 1.1.2.5.2.14; previous revision: 1.1.2.5.2.13 done
The numbers are now affected by a trunk regression (see issue 44003). Please, keep this bug open until 44003 is fixed and the numbers can be compared again.
Tondo, can you compare it again? Thanks
The same configuration and procedure as before. I have done more measurements this time because the behavior was quite unstable (see the numbers below). Each invocation of delete on the folder of 1000 java files is listed as two numbers (A/B) where A is the time when the nodes of the 1000 java files disappeared in the explorer and B is the time when also the node of the folder disappeared. Both A and B are measured from the beginning of delete action. trunk 200406031800: 6828ms/7406ms, 7765ms/12000ms, 7719ms/8531ms, 6484ms/10359ms, 6734ms/6921ms You can see that especially the very end of the delete action (time B) varies a lot. This behavior should be investigated in a profiler and explained.
Created attachment 15559 [details] OptimizeIt snapshot of the second part of delete action (when files' nodes disappeared but folder's node is still visible)
One possible improvement found. Quite a lot time is spent in org.netbeans.api.java.classpath.ClassPath$RootsListener.processEvent(). Tomas is going to integrate an improvement which minimizes the time spent in this method for deleted files other than the first one in the set.
*** Issue 41154 has been marked as a duplicate of this issue. ***
Changing summary of this bug.
The trunk regression is fixed. Checking in api/nbproject/project.properties; /cvs/java/api/nbproject/project.properties,v <-- project.properties new revision: 1.4; previous revision: 1.3 done Processing log script arguments... More commits to come... Checking in api/src/org/netbeans/api/java/classpath/ClassPath.java; /cvs/java/api/src/org/netbeans/api/java/classpath/ClassPath.java,v <-- ClassPath.java new revision: 1.19; previous revision: 1.18 done Processing log script arguments... More commits to come... Checking in api/test/unit/src/org/netbeans/api/java/classpath/ClassPathTest.java; /cvs/java/api/test/unit/src/org/netbeans/api/java/classpath/ClassPathTest.java,v <-- ClassPathTest.java new revision: 1.4; previous revision: 1.3 done
Tondo, please close this bug, or reassign it to proper module. It does not look like refactoring issue any more. Thanks
Tomas fixed it. Please reopen this issue if deleting is still slower than your expactation.
I tried it for 1800 files on dev build 20050718 and it took several minutes. I'm using Linux FC2, on P4 1GB ram. I concurently watched the folder in MC and the files was beeing deleted slowly about 3 in a second, but the cpu usage was only about 50%. During the deleting it can be worked with IDE but it had long responses. When i tried it second time the exception occures: java.lang.Error: org.openide.loaders.DataObjectNotFoundException: MasterFileObject@53d29b[file:/home/jp159440/JavaApplication10/src/pack/AbstractJ2eeFile.java] at org.netbeans.spi.project.support.ant.GlobFileBuiltQuery$StatusImpl.<init>(GlobFileBuiltQuery.java:201) at org.netbeans.spi.project.support.ant.GlobFileBuiltQuery.createStatus(GlobFileBuiltQuery.java:184) Please try it also on some performance computer, and if the response time is ok feel free to close it again.
Created attachment 23141 [details] exception
Can you attach few thread dumps for me to see what is going on during delete process? Thanks
This bug is hardly refactoring issue. Tondo, can you reevaluate this issue and attach fresh performance team opinion? Thanks
With a folder with 1000 files I measured 35s on local filesystem and 69s on my NFS network home. OptimizeIt shows that majority of the time is spent in org.openide.loaders.FileEntry.delete() |- 40% - org.openide.loaders.MultiDataObject$Entry.takeLock() |- 35% - org.netbeans.modules.masterfs.MasterFileObject.delete() |- 21% - org.netbeans.modules.masterfs.Delegate$FileLockImpl.releaseLock() Radku, please have a look on this and try to optimize deleting and the operations with the locks needed for that.
Created attachment 23323 [details] BTW, here is a snapshot from OptimizeIt 6
Locking is slower since 4.1 which affect delete bulk operations For every locked FileObject is created temporary file which is deleted after releasing of lock. This is probably the cause. This concept was introduced because of reliability of locking, preventing from data loss in editor and was approved by DevRev. I don't recommend to fix it for 5.0 (definitely not for beta) because of stability of the whole product. On the other hand I would vote to solve this issue as a prority issue for next release.
I think that lock on folder could mean locking of the whole subtree, which would bring semantical API change that we could afford.
Radku, please plan for the API change soon enough for the next release. For 5.0, after the discussion we had, I agree we have to waive this issue.
/cvs/openide/loaders/src/org/openide/loaders/DataFolder.java,v <-- new revision: 1.41; previous revision: 1.40 /cvs/openide/masterfs/src/org/netbeans/modules/masterfs/Delegate.java,v <-- new revision: 1.20; previous revision: 1.19 /cvs/openide/masterfs/src/org/netbeans/modules/masterfs/filebasedfs/fileobjects/BaseFileObj.java,v new revision: 1.19; previous revision: 1.18 /cvs/openide/masterfs/src/org/netbeans/modules/masterfs/filebasedfs/fileobjects/FileObj.java,v new revision: 1.16; previous revision: 1.15 /cvs/openide/masterfs/src/org/netbeans/modules/masterfs/filebasedfs/fileobjects/FolderObj.java,v new revision: 1.15; previous revision: 1.14 /cvs/openide/masterfs/src/org/netbeans/modules/masterfs/filebasedfs/fileobjects/WriteLock.java,v new revision: 1.10; previous revision: 1.9 /cvs/openide/masterfs/src/org/netbeans/modules/masterfs/filebasedfs/fileobjects/WriteLockFactory.java,v new revision: 1.3; previous revision: 1.2 /cvs/openide/masterfs/test/unit/src/org/netbeans/modules/masterfs/filebasedfs/fileobjects/FolderObjTest.java,v new revision: 1.8; previous revision: 1.7 /cvs/openide/masterfs/test/unit/src/org/netbeans/modules/masterfs/filebasedfs/fileobjects/WriteLockTest.java,v new revision: 1.4; previous revision: 1.3 Although not very nice I hope its fixed. Jarda, please review.
Verified. It now takes <5s on my local filesystem to delete a folder with 1000 java files.