This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
Modules currently contain layers in XML format. These are not intended to be edited by a end user - no human readability is required. It causes performance problems to parse these general XML documents during start up. To eliminate the performance problem we can use Google's "binary xml" search results. Particurarly compile layer's XML into its binary form. XMLFileSystem then will sniff a magic and use faster binary reader instead of XML parser. This approach also can eliminate bugs in layers as the compiler may do some extra checks. Please evaluate the idea. It is worthy for 3.4?
APISupport could provide the compiler.
Is it really worth the bother? According to pnejedly's studies, parsing complete premerged layers, after removing various extra attrs we plan to remove, takes around 300msec. If we did a lot of work to get it down to 100msec (say) using a binary format, is that a good use of effort (relative to more difficult debugging, more complex and maybe buggier code, more complex build procedure for people to learn, etc.)? We can do validity checks much more easily in other ways, build-time or test-time, so I don't think that's a strong argument for this.
My words. Moreover, we'd like to reduce the (startup parsed) content of the lookup using Naming, so it is less appealing even for .settings Reducing the priority, we may evaluate it later but it seems as a workaround of a problem which we should solve some better way.
If the parsing takes 300msec, and expected gain is about 50% then it really does not matter. Besides basic impl is extremely simple. You write a SAX handler that writes down into stream callbacks it gets. Then you simply replay the callbacks from stream instead of parsing XML document. Thanks for the evaluation
#20168's org.netbeans.core.projects.cache.LayerCacheManager would need to be implemented and registered in place of the current XML cache. Here is an idea: rather than using an XML format, use a JAR file to cache the merged layers. Probably a JAR is faster to open (native optimized code). Files can be served on demand. Attributes could be handled by subclassing JarFileSystem and using DefaultAttributes as its attr; so the cache recreation would store .nbattrs files. Less data need be held in memory, e.g. all the hundreds of files added by vcsgeneric and never used in an average IDE session would exist only as JAR entries - assuming JarFileSystem is written correctly to not create FileObject's until they are wanted.
AFAIK JarFS will initially create only CacheEntries, not FileObjects, but I'm not sure it is better when it comes to memory usage...
Set target milestone to TBD
Comparing gain of binary cache with drawbacks, seems to me that I`d rather won`t fix this RFE.
x
I am reconsidering this for 4.0. I don't think the savings would be very large compared to the XML layer cache, but there would be some, and it may be worth the effort - especially for installations with 100+ modules, e.g. S1S EE. It could save a second or so of startup time, I guess. It might also save a certain amount of heap, if most files are not really accessed - consider for example the vcsgeneric command list, hundreds of .class files in the layer which might never be touched. org.netbeans.core.projects.cache.* already has some infrastructure, incl. layer parsing, cache recreation logic, and even unit tests; all that is needed is code to write a binary file (not JAR - some custom optimized format, possibly memory-mapped under 1.4), and code to read it into a FileSystem (AbstractFileSystem is probably unnecessary overhead?).
Note that the cache infrastructure is entirely runtime, not build-time: when started, NB checks for an existing cache and examines JAR timestamps. It is recreated if JARs have been modified (or added or deleted). The layer parser can afford to spend a little time validating the layers, since it is not called on most startups. (Currently it doesn't, but that could be changed.)
I've implemented a BinaryFS, a filesystem over a binary snapshot. The FS is creating the FOs on demand (nearly-zero initialization). I've also written a cache manager for it, but it can't work correctly with current LCM api. I've created a core branch named binary_layer_Oct_02 and added these files to the CVS, you can try it and fix the LCM to allow changing the instance of the FS at runtime (or to fall back to XFS on change) To enable it, use -J-Dnetbeans.cache.layers=org.netbeans.core.projects.cache.BinaryCacheManager It won't work correctly on the first start but the next start should be OK. I'll be on vacations next two weeks so it is your turn now. (BTW: The performance looks good so far)
Petr next time you make a branch, please make a base tag so I can easily work with it...
Got it working. Improvement over XML cache manager: 988 msec 6.16% 1.83 standard deviations Refactored the LayerCacheManager SPI and its callers to accommodate both cache managers that can load from disk into an existing filesystem (e.g. XMLLayerCacheManagerImpl) and those that cannot (e.g. the new BinaryCacheManager). Wrote a unit test for the new manager. Petr, it passed the automated test without modification (only LCM semantic changes were needed). Added a magic header to the binary cache to serve in identifying such files. Refactored ModuleLayeredFileSystem a bit to use a special non-caching manager, rather than lots of if-conditions checking for null. Use of XMLFileSystem completely factored into the non-caching manager. Changed layer-stamp.txt format to include cache manager class name in hash. This was necessary to permit a userdir containing an old all-layers.xml to be used with the binary manager in effect: there is a cache miss, the new all-layers.dat is created, then everything proceeds normally.
Petre - "utf8" is not a valid encoding under JDK 1.3, though JDK 1.4 accepts it. You must use "UTF-8", the canonical name according to the docs.
committed * Up-To-Date 1.23 core/src/org/netbeans/core/projects/ModuleLayeredFileSystem.java committed * Up-To-Date 1.2 core/src/org/netbeans/core/projects/cache/BinaryCacheManager.java committed * Up-To-Date 1.2 core/src/org/netbeans/core/projects/cache/BinaryFS.java committed * Up-To-Date 1.3 core/src/org/netbeans/core/projects/cache/LayerCacheManager.java added * Up-To-Date 1.1 core/src/org/netbeans/core/projects/cache/NonCacheManager.java committed * Up-To-Date 1.7 core/src/org/netbeans/core/projects/cache/ParsingLayerCacheManager.java committed * Up-To-Date 1.4 core/src/org/netbeans/core/projects/cache/XMLLayerCacheManagerImpl.java added * Up-To-Date 1.1 core/test/unit/src/org/netbeans/core/projects/cache/BinaryCacheManagerTest.java added * Up-To-Date 1.1 core/test/unit/src/org/netbeans/core/projects/cache/CacheManagerTestBaseHid.java added * Up-To-Date 1.1 core/test/unit/src/org/netbeans/core/projects/cache/NonCacheManagerTest.java committed * Up-To-Date 1.3 core/test/unit/src/org/netbeans/core/projects/cache/XMLLayerCacheManagerImplTest.java