This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
Description: RC2 build. Win 2k and Win XP (tested on 2 systems, jdk1_4_1_02b6 and beta 1.4.2) now the the javacvs is gone, we need to switch to the generic CVS filesystem mounting system. As a plugin developer, we always mount the root of the app server s1as source code (pretty big, but tha't not the issue). After mounting the source code, the IDE starts some background activity (I presume to create the code completion database for the cvs FS). The IDE is barely usable for at least 40 minutes (maybe much more): it takes between 70 and 98% of the CPU and the user can't do much... xxx@xxxx 2003-05-06: I've put the thread dumps into an attachment. BT+ has a problem with too long texts in it's fields. This was a recommentation of the BT+ people after I filed a servicedesk ticket. Also it looks like the dumps were not completely included here, maybe Ludo has the remainder somewhere? -- Marco Work Around: Do not use the CVS filesystem. This defeats the whole value proposition of Nevada around VCS systems... xxx@xxxx 2003-05-05 xxx@xxxx 2003-05-05 Suggested Fix: According to my tests it seems, that the root cause of the performance problems are I/O operation when reading/storing the file attribute. The DefaultAttributes are not efficient for storing of big objects. Thus I've created a patch that works as described in the Evaluation: The object integrity database is saved in a raw binary form (a serialized Java object) in a file .nbintdb next to .nbattrs. The patch Patch4858578.jar is attached in the Attachments. To apply, put the jar into <IDE-install>/modules/autoload/patches/org-netbeans-modules-vcscore/ folder. I've done some measurments of a test working directory with a bunch of modules from cvs.netbeans.org: The time of building of the parser database after the filesystem was mounted: 13 min NetBeans 3.5 RC2, VCS Generic filesystem with CVS profile, 4 min NetBeans 3.5 RC2 with the attached patch applied, VCS Generic FS, 2.5 min NetBeans 3.5 RC2, Local Filesystem The Generic VCS filesystem will always be slower, but it's evident, that the patch improved the performance substantially. Also the IDE was subjectively more responsive. xxx@xxxx 2003-05-07 Please verify whether the attached patch helps. It needs to be decided whether it should be included into NetBeans 3.5 / S1S 5.0 or not. xxx@xxxx 2003-05-07 Will try today. Could we decrease the thread priority also? xxx@xxxx 2003-05-07 Evaluation: xxx@xxxx 2003-05-06: how does this work iff you simply mount this as a `normal' source directory structure? The CodeCompletion DB would have to updated as well? -- Marco It works fine if you do not use VCS FS... (If it did not, we could not use Studio at all to develop the plugin;-) The issue is really around the VCS FS: not usable for decent repository. You can try it on you side with a Studio Module (from you CVS repository) and do some measures. xxx@xxxx 2003-05-06 On another machine: win2000, Dual CPU 600Mhz, the code completion on S1AS VCS fileystem took one entire CPU for around 15 minutes to complete the database. After that, the IDE was usable again. xxx@xxxx 2003-05-06 I tried mounting the s1s 5 source tree using local and CVS filesystem on a SunBlade 2000 machine. For local filesystem, to took ~5 1/2 minutes for the IDE to finish creating the code completion DB. For VCS filesystem, it took ~12 minutes to finish. From the thread dumps , it appears that the IDE is scanning the entire source tree, parsing files and creating data objects . VCS filesystem exacerbates the problem by doing additional things. I will bugbridge this bug to Issuezilla tomorrow. Bugbridge is currently down. xxx@xxxx 2003-05-06 It seems like this problem can be caused by the implementation of issue #31056. This was not implemented in JavaCVS, only in VCS Generic. However in the attached thread dumps I didn't find anything from VCS that can cause some performance problems. The first thread dump does not contain anything from VCS, the second contains just reading of an integrity support attribute and the third writing of the integrity support attribute. If this is still causing problems, the whole feature would probably have to be completely redesigned or removed. xxx@xxxx 2003-05-07 One possible performance improvement that came to my mind is that we don't use FileObject attributes for the Object Integriry database. Therefore it will not be necessary to parse a long XML file. The database can be saved in a raw binary form (a serialized Java object) in a file next to .nbattrs. It would have to be measured whether it will lead to a substantial performance improvements. xxx@xxxx 2003-05-07 A comment.
Created attachment 10247 [details] The binary patch, that should improve the performance. To apply, put the jar into <IDE-install>/modules/autoload/patches/org-netbeans-modules-vcscore/ folder.
Copied the further comments: We can not decrease the thread priority any more. It's already '1'. Martin Entlicher 2003-05-07 I tried on my laptop (dell 650Mh, 512Mb): without the patch, yeterday, I stopped the IDE after 45 minutes and it was still constructing the database. With the patch, the database is done in 18 minutes. its size is 3.8Mb. So this is MUCH better. Is is enough? I don't know;-) 18 minutes of CPU is a lot... xxx@xxxx 2003-05-07 You will never reach zero! The building of the parser database takes some time, which is proportional to the filesystem size. This will happen when you mount a local filesystem as well. My goal is not to prolog that time too much. VCS does have to do some work due to the implementation of issue #31056, but the CPU consumption should not be bigger than is really necessary. Anyway the IDE should stay responsive, which should be assured by the lowest possible thread priority. So someone would have to decide whether this is acceptable and whether it's worth to integrate into FCS. My opinion is, that it should be integrated, the fix is relatively safe and gives a substantial performance improvement. It would need a throughout analysis to find out whether an even better performance improvement is possible. Martin Entlicher 2003-05-07
The bug is fixed in the main trunk after it was verified that the attached patch really improves the performance: /cvs/vcscore/src/org/netbeans/modules/vcscore/objectintegrity/IntegritySupportMaintainer.java,v <-- IntegritySupportMaintainer.java new revision: 1.4; previous revision: 1.3
I just tried the patch and it did improve the performance for mount CVS filesystem. However, the performance as a whole is still horrible. This is not a VCS-only issue. The problem really has to do with the fact that we are scanning the entire tree. I know it is probably too late to make anymore changes. Shouldn't the code completion DB be built on demand as the user access each directory?
Created attachment 10255 [details] The contextual diff of changes that fix this problem.
I've found out, that in some cases the database file could be corrupted (even when the IDE was not killed, but exited via Exit). Thus the database needs to be written to a temp file first: .nbintdb~ and later renamed to .nbintdb Also an adaptive schedule time was introduced. This improves the performance a little bit more. This is fixed in the main trunk: /cvs/vcscore/src/org/netbeans/modules/vcscore/objectintegrity/IntegritySupportMaintainer.java,v <-- IntegritySupportMaintainer.java new revision: 1.5; previous revision: 1.4
Created attachment 10258 [details] The new contextual diff of all changes.
Created attachment 10259 [details] The new binary patch.
As was pointed out by the reviewer, when reading the database file we should try to read the backup when the original does not exist (the process could be killed before it renamed the .nbintdb~ to .nbintdb): /cvs/vcscore/src/org/netbeans/modules/vcscore/objectintegrity/IntegritySupportMaintainer.java,v <-- IntegritySupportMaintainer.java new revision: 1.6; previous revision: 1.5
Code reviewed without objections.
The fix is merged into release35 branch upon a preliminary approval by the release coordinator: /cvs/vcscore/src/org/netbeans/modules/vcscore/objectintegrity/IntegritySupportMaintainer.java,v <-- IntegritySupportMaintainer.java new revision: 1.1.2.4; previous revision: 1.1.2.3
yes, I gave verbal approval to Martin. thx for the fix
I have a comment here. The patch introduced new file .nbintdb that is not being ignored by default. Can we avoid displaying this file ? Therefore reopening to get this regression fixed. As for the fix of the performance issue, it looks good so far. I am confirming that it helped to reduce full CPU usage approximately to 20 - 30% for large filesystems with 10000+ files. For filesystems with less than 5000 files it does not help nearly at all. Tested in Sun ONE Studio 5.0 build #030502 and Patch4858578.jar.
Nobody has found it earlier, because there's no periodic refresh :-( The fix is simple - filter .nbintdb in VcsFileSystem.children().
Created attachment 10268 [details] The new binary patch, that filters .nbintdb. Use with the latest build from May 11. Put into <IDE-install>/modules/autoload/patches/org-netbeans-modules-vcscore/ folder.
Created attachment 10273 [details] The contextual diff.
The filtering is fixed in the main trunk: /cvs/vcscore/src/org/netbeans/modules/vcscore/objectintegrity/IntegritySupportMaintainer.java,v <-- IntegritySupportMaintainer.java new revision: 1.7; previous revision: 1.6 Checking in VcsFileSystem.java; /cvs/vcscore/src/org/netbeans/modules/vcscore/VcsFileSystem.java,v <-- VcsFileSystem.java new revision: 1.226; previous revision: 1.225
The additional diff reviewed, looks OK.
Yes, the patch fixes the problem. No accessory file is visible. Verified in RC1 build #200305112350 of NetBeans 3.5 with Patch4858578_2.jar.
Integrated in release35 branch: /cvs/vcscore/src/org/netbeans/modules/vcscore/objectintegrity/IntegritySupportMaintainer.java,v <-- IntegritySupportMaintainer.java new revision: 1.1.2.5; previous revision: 1.1.2.4 Checking in VcsFileSystem.java; /cvs/vcscore/src/org/netbeans/modules/vcscore/VcsFileSystem.java,v <-- VcsFileSystem.java new revision: 1.207.2.14; previous revision: 1.207.2.13
Verified in Sun ONE Studio 5.0 Standard Edition build #030528.