This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 156330

Summary: All module JAR metadata is accessed despite pnejedly's cache
Product: platform Reporter: Jaroslav Tulach <jtulach>
Component: Module SystemAssignee: Jaroslav Tulach <jtulach>
Status: RESOLVED FIXED    
Severity: blocker CC: issues, jglick, jskrivanek, pnejedly
Priority: P2 Keywords: PERFORMANCE
Version: 6.x   
Hardware: All   
OS: All   
Issue Type: DEFECT Exception Reporter:
Bug Depends on: 199320    
Bug Blocks:    
Attachments: Slight improvement to work on windows

Description Jaroslav Tulach 2009-01-06 09:24:32 UTC
Look at ide.kit/test/qa-functional/src/org/netbeans/test/ide/ReadAccessTest.java in revision 2b6c4462b1aa

The IDE seems to access the disk ~4500 during startup while searching for various files on second startup. Especially 
during cold startup it, this has to be really slow. Some improvement in all-resources.dat cache seems desirable.

A lot of disk touches comes from findExtensionsAndVariants:

java.io.File.isFile(File.java:776)
org.netbeans.Util.findLocaleVariantsWithSuffixesOf(Util.java:118)
org.netbeans.Util.findLocaleVariantsOf(Util.java:99)
org.netbeans.StandardModule.findExtensionsAndVariants(StandardModule.java:329)
org.netbeans.StandardModule.<init>(StandardModule.java:133)
org.netbeans.ModuleFactory.create(ModuleFactory.java:65)
org.netbeans.ModuleManager.create(ModuleManager.java:496)
org.netbeans.core.startup.ModuleList$1.run(ModuleList.java:287)
org.openide.filesystems.EventControl.runAtomicAction(EventControl.java:120)
org.openide.filesystems.FileSystem.runAtomicAction(FileSystem.java:502)
org.netbeans.core.startup.ModuleList.readInitial(ModuleList.java:166)
org.netbeans.core.startup.ModuleSystem.readList(ModuleSystem.java:262)
org.netbeans.core.startup.Main.getModuleSystem(Main.java:162)
org.netbeans.core.startup.Main.start(Main.java:304)

Can we change the system to scan for module JARs only if the cache missed during classloading? Similar, case is very 
likely:

java.io.File.exists(File.java:731)
org.netbeans.StandardModule.findExtensionsAndVariants(StandardModule.java:344)
org.netbeans.StandardModule.<init>(StandardModule.java:133)
org.netbeans.ModuleFactory.create(ModuleFactory.java:65)
org.netbeans.ModuleManager.create(ModuleManager.java:496)
org.netbeans.core.startup.ModuleList$1.run(ModuleList.java:287)
org.openide.filesystems.EventControl.runAtomicAction(EventControl.java:120)
org.openide.filesystems.FileSystem.runAtomicAction(FileSystem.java:502)
org.netbeans.core.startup.ModuleList.readInitial(ModuleList.java:166)
org.netbeans.core.startup.ModuleSystem.readList(ModuleSystem.java:262)
org.netbeans.core.startup.Main.getModuleSystem(Main.java:162)
org.netbeans.core.startup.Main.start(Main.java:304)
org.netbeans.core.startup.TopThreadGroup.run(TopThreadGroup.java:110)
java.lang.Thread.run(Thread.java:619)

The other quite frequent touch comes from JarClassLoader constructor and again, shall probably be delayed:

org.netbeans.test.ide.CountingSecurityManager.checkRead(CountingSecurityManager.java:176)
java.io.File.isDirectory(File.java:752)
org.netbeans.JarClassLoader$Source.create(JarClassLoader.java:314)
org.netbeans.JarClassLoader.<init>(JarClassLoader.java:143)
org.netbeans.StandardModule$OneModuleClassLoader.<init>(StandardModule.java:696)
org.netbeans.StandardModule.classLoaderUp(StandardModule.java:629)
org.netbeans.ModuleManager.enable(ModuleManager.java:803)
org.netbeans.core.startup.ModuleList.installNew(ModuleList.java:428)
org.netbeans.core.startup.ModuleList.trigger(ModuleList.java:364)
org.netbeans.core.startup.ModuleSystem.restore(ModuleSystem.java:276)

And finally I can see a bit of URLConnection touches which could probably be eliminated as well, if we correctly 
implemented BinaryFS.lastModified:

org.netbeans.test.ide.CountingSecurityManager.checkRead(CountingSecurityManager.java:175)
java.io.File.isDirectory(File.java:752)
sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:65)
sun.net.www.protocol.file.FileURLConnection.initializeHeaders(FileURLConnection.java:90)
sun.net.www.protocol.file.FileURLConnection.getLastModified(FileURLConnection.java:151)
org.netbeans.core.startup.layers.BinaryFS$BFSFile.lastModified(BinaryFS.java:789)
org.openide.filesystems.MultiFileObject.lastModified(MultiFileObject.java:510)
org.openide.filesystems.MultiFileObject.lastModified(MultiFileObject.java:510)
org.openide.loaders.InstanceDataObject$Ser.instanceOf(InstanceDataObject.java:1214)
org.openide.loaders.InstanceDataObject.instanceOf(InstanceDataObject.java:737)
the call comes either from InstanceDataObject or from
org.netbeans.modules.project.libraries.LibrariesStorage.isUpToDate(LibrariesStorage.java:481)
Comment 1 Jaroslav Tulach 2009-01-06 09:29:18 UTC
I guess it is not wise to have large discussions in IZ, so I've created a section about this problem at
http://wiki.netbeans.org/FitnessViaCleanup
where we can express and persist our ideas.
Comment 2 Jesse Glick 2009-01-07 02:40:15 UTC
I was not involved in writing the new cache and do not know so much about how it works and what it is expected to do, so
if you have some ideas about what to improve (without breaking things), go ahead.

Certainly avoiding all calls to File.isFile/isDirectory/exists on enabled JARs does not seem feasible to me; just
calling File.toURI will do this! And once you have gotten File inode information, the OS very likely caches it, so I am
not sure that avoiding subsequent calls on the same File gives you much actual performance benefit. If so, we could keep
a simple File -> metadata cache in core somewhere which would be active just during startup.
Comment 3 Jaroslav Tulach 2009-02-19 06:32:27 UTC
ff93758eb7eb
Comment 4 Jesse Glick 2009-02-20 04:57:07 UTC
In JarClassLoader.java, toURI(File) will be incorrect for files whose paths contain special characters (even ' '),
because you are not doing URI encoding. Also remember that if the JRE's impl of File.toURI changes, your copied code
will become incorrect. Anyway I think duplicating JRE code is unnecessary, since File is not final:

import java.io.File;
public class DisklessFile {
    public static void main(String[] args) {
        File f = new File("/tmp/non existent");
        System.out.println(f.toURI());
        f = new File(f.getAbsolutePath()) {
            public @Override File getAbsoluteFile() {
                return this;
            }
            public @Override boolean isDirectory() {
                return true;
            }
        };
        System.out.println(f.toURI());
    }
}

As to the inverse operation later on in this patch, this is also wrong, assuming that the encoding is done correctly.
Just use new File(URI) which should not touch the disk AFAIK; never attempt to parse a file: URI any other way.
Comment 5 Quality Engineering 2009-02-22 07:56:21 UTC
Integrated into 'main-golden', will be available in build *200902220201* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/ff93758eb7eb
User: Jaroslav Tulach <jtulach@netbeans.org>
Log: #156330: Making sure the classes cache is really used on second start
Comment 6 Jaroslav Tulach 2009-03-02 11:59:50 UTC
Created attachment 77556 [details]
Slight improvement to work on windows
Comment 7 Jaroslav Tulach 2009-03-02 12:50:32 UTC
Improved in ergonomics#d19edaf83fa9
Comment 8 Jesse Glick 2009-03-02 20:31:58 UTC
Again, new File(URI) would be preferable to the manual URI parsing around line 750, and AFAIK does not involve any I/O
operations.
Comment 9 Quality Engineering 2009-03-04 09:05:37 UTC
Integrated into 'main-golden', will be available in build *200903040201* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/d19edaf83fa9
User: Jaroslav Tulach <jtulach@netbeans.org>
Log: #156330: Slight mangling with / and \ to make the cache work better on windows