I implemented a code that copies the content of XMLlayers to the local
filesystem and recieved 9% startup time improvements. I have not
measured the improvements in memory consumption.

Startup without patch took 20.2s on my computer.
Startup with the patch tool 18.4s on my computer.

Created attachment 4573 [details]
The patch that demonstrates the 9% speed up

To try the patch start the ide with plain switches:

-J-Dusecache=false -J-Dorg.netbeans.log.startup=print

this should measure the time without any caching. Then start it with

-J-Dusecache=true -J-Dorg.netbeans.log.startup=print

this will create the cache in /tmp/NB-CACHE directory and also take
longer, I have not tried to optimize it. Then start it once more with

-J-Dusecache=true -J-Dorg.netbeans.log.startup=print

and you should see the performance improvement.

To implement the cache copying in better way, it is necessary not to
copy content of files in XMLFS but just their URLs as described in
issue 20170

Folks I am tired of adding PERFORMANCE keywords for you :-)

Why do you use a LocalFileSystem? I am not sure that is more efficient
on all platforms than a premerged XML file. LFS means more polling,
and some platforms are slow to access many files. Re. keeping the
whole content in memory: with LFS you will have softly held
FileObject's anyway. I suggest to keep an XML cache and to work on
optimizing XMLFileSystem for memory usage.

I assume the patch is supposed to be an experiment, not something to
really apply as is.

The patch does not seem to handle module
installation/uninstallation/upgrading, does it?

No, it was just an experiment if *any* other storage can speed
it up. The problem with XMLFS is that it parses everything at once
although not needed yet (maybe 15% of the XMLFS content is used
during the startup?).
The other part of the problem is that it causes changes in SFS
during the startup and they are very ineffective in layered MFS.
(2/5 of the setXmlUrls time is parsing, 3/5 is the change propagation)
Having the XMLFS with the "right" content from the beginning
is itself quite an improvement.

Re. memory usage: It is not that bad (~300Kb
last time I've checked) but it grows as we're adding more modules.

Re. premerged XML file: Its performance will be comparable
to separate XML files, as most of the time is spent elsewhere
than changing the stream during parsing.

OK, so it sounds like a LocalFileSystem with polling disabled will be
faster to load than a premerged XMLFileSystem, though consumes more
disk space and probably causes heavier OS-level disk usage. (Ideal
solution would be a compact binary format I guess.)

Next question: as far as change propagation goes, how does adding an
LFS to the MFS stack differ from adding an XMLFS to it? Seems to me
like it would be exactly the same situation. During startup, all
module layers are collected into one XMLFS and it is added to the
SystemFileSystem as a layer in one operation, right? With a cache, you
would add one LFS at the same time instead.

For handling module installation, uninstallation, and other changes, I
think we would need to annotate the cache dir with information about
what layers are installed, and what version of each, so we know when
to recreate the cache. Either module spec version (fast to compute but
not robust) or CRC of layer XML (slower to compute but much safer).
Maybe module spec version + layer file size would be reasonably
effective and fast.

1. Hrebejk suggested to use MDR's btree for the compact binary format.
2. Yes, this is only a demo. Real implementation needs to take care of
synchronization of cache & real data (time stamps & module state seem
enough for me).

True, module JAR timestamp might be enough to serve as the layer
version component of the cache key, I did not think of that.

Reusing MDR's btree or in fact MDR sounds technically good, but I know
little about it, and of course it is problematic to reuse parts of MDR
in the core when the module is not even in standard builds yet. I
guess the btree lib could be copied to core, but that is a bit messy.

We can create interface CacheLayers in core and then create new module
that will implement it using mdr's btree. So everything work without
btree but will work much better with it. Moreover the core will not
depend on experimental module.

It think Jarda's idea should be easy. When we were developing the MDR
we didn't have the btree implemented. So we designed a simple API
which was first implemented in memory using Hashtables and then we
just moved to the disk based btree. The API are those few classes in
org.netbeans.mdr.persistence.

> little about it, and of course it is problematic to reuse parts of MDR
> in the core when the module is not even in standard builds yet. I
> guess the btree lib could be copied to core, but that is a bit messy.

can someone lobby mdr team to factor out their btree code into a
standalone library packacged under org.netbeans.lib.btree.  The lib
can live under mdr or under core or under btree.netbeans.org.  I think
it would increase the reuse opportunity for btree.  I suspect it would
be very easy, a small perl script will do.  The question of course is
if it has a sensible API or not.

[I do not say that core will use btree for caching purposes (yet)]

FYI: I've done some additional measurements on the XMLFS in SFS:
Parsing of all standard module layers: ~950ms
explicit refresh + firing changes: ~2050ms
Parsing of premerged layer: ~650ms
above w/o SFS.[loc*|icon]: ~420ms
above w/o ordering attrs: ~350ms

I don't have numbers for separate XMLs as it would be a lot of work
to apply the filters on all of them.

When I tried refresh+firing from a XMLFS plugged into a fully
populated MFS (new MFS(fullXFS,newXFS); newXFS.setXmlUrls()),
it took ~4000ms

The Yarda's hack speeds it up because it places the LFS into
the SFS immediatelly from the beginning so the refresh+firing
step is not there. IMHO we should try to "repair" the MFS
behaviour and then probably use the premerged layer file
(which can be there from the beginning as well, if we'll check
it properly)


BTW: I've generated the premerged layer by a hack inside the XMLFS
implementation. Similar scheme could be used for creating
the real merged layer.

Petr's suggestion makes sense to me. However I'm not sure how easy
it's going to be to get the XMLFS in the SFS layer stack right at
startup: currently just finding out what modules are enabled requires
having a SFS to inspect (Modules/ folder). Probably this can be hacked
around: make a temporary SFS with just Modules/, then throw it out and
create a new one with the premerged layer all ready (assuming we have
a cache hit).

Sounds like some profiling & optimizations in MFS will be useful no
matter how we do this.

The 650ms -> 350ms differential we can make up with other enhancements
I think, which don't depend on this impl. Both possibilities of
removing SFS.* attrs and removing most explicit ordering attrs have
been separately filed. I think some backdoor cooperation between
XMLFileSystem and FolderList could also help reduce overhead of
ordering attributes.

Merging in the event of a cache miss might be effectively done by
direct parsing of the consituent layers. This would remove the
XMLFileSystem dependencies, and might anyway be faster & more reliable.

Look at RFE: http://www.netbeans.org/issues/show_bug.cgi?id=20528 that
should allow to avoid updateAll call as reaction for
fileData/FolderCreated. Then only fileDeleted must be handled using
updateAll because fileDeleted is fired only for FileObject that was
deleted but not for its children in hieararchy.

I'm starting to work on this. Proposed mechanism (please tell me if
this sounds stupid):

1. ModuleLayeredFileSystem will always have two delegates, #0 the
writable and #1 an XMLFileSystem.

2. With caching off, behavior as now: XML delegate initially empty,
and setURLs calls setXMLURLs on the XML delegate, done.

3. With caching on, a directory $userdir/cache/ or similar is
selected. Better than /tmp I think, since the cache is more or less
specific to the userdir's state.

3a. In the cache dir, a preparsed layers are kept, named layers.xml.
The initial XML delegate is loaded from this file.

3b. The cache dir also has a stamp.txt containing all the XML URLs
(jar:file:/..../foo.jar!/.../layer.xml) and also timestamps of the
associated JAR files. Or, just a hash (e.g. MD5) of the same. The hash
is a bit more compact, but the full data should be easier to inspect
when debugging.

3c. When setURLs is called, the new hash is computed.

3c1. If the new hash is in fact the one already loaded, nothing
happens. This should be the usual case.

3c2. If the stamp differs from the loaded one, layers.xml is
regenerated from the XML layers, setXmlURLs is called on the result
(triggering a slow refresh) and a new stamp.txt is written out.

3c3. If it turns out to be a problem to regenerate the cache whenever
modules are enabled or disabled - e.g. due to session support or
similar - it should be possible to keep multiple cache files, so that
an older cache can be returned to, and expire old cache files
according to an LRU mechanism. I don't see a particular need for this
now however.

The trick, if I understand correctly, is that most of the time (user
has not turned modules on or off etc.) the correct XML delegate will
already be there at startup time and no refresh will occur. The fact
that the system file system is full of module-supplied stuff before
you have actually turned on any modules should not much matter; lookup
has not been initialized, so no one is really inspecting the SFS much
yet, and since core is in the classpath and has a layer, it is
guaranteed that setURLs will be called pretty quickly.

The cache generation can I think be done by parsing the constituent
layers using SAX and generating an XML output tree. <file>s originally
having literal CDATA content will continue to; empty files will
continue to be empty; files with url= will get url= contents, though
the URL will be adjusted to not be relative. <attr>s will be copied
exactly as they appeared in the original XML. Such a merge process
should not be difficult to implement, should be faster than creating a
filesystem and doing FileObject operations on it, and should not
require any changes to openide code - so no dependency on issue #20169
nor on issue #20170.

One question I have for people: is there any particular reason why
NbInstaller dumps the layers for $userdir/modules/*.jar into the user
ModuleLayeredFileSystem? Why can't *all* module layers go into the
"installation layer"? I don't see the purpose of separating them;
functionally, user-installed modules should behave the same as
central-installed modules, I think.

I have a working XFS merger as part of XMLFS.setXmlURLs
No need for manual creation of the cache by a special parser,
it should suffice to dump the state of ResourceElems
just after the standart cacheless parse.
Tomorrow I'll attach a diff against XMLFS.

Ad. implementation: Jesse, please try to separate the module's API
with the actual cache implementation by creating interface that will
recieve URLs of layers and will return/generate a cache filesystem. I
still do not want to give up the idea of btree storage - so please
create an interface for communication between SystemFileSystem and the
cache.

Ad. "Why can't *all* module layers go into the installation layer?"
Because it is readonly?

Created attachment 4758 [details]
A testing patch against XmlFS which adds formatted save

Thanks Petr for merging code, will try to use it.

Re. separation of cache impl via interface: will try, but consider
that a secondary priority; can always rewrite later. We hardly need
this to be pluggable; the best option should be selected and used.

Re. installation layer being read-only: yeah; so what? All module
layers are read-only. My point is that currently we have (ignoring
projects module) an installation layer (r/o; $nbhome/system/ + layers
from $nbhome/modules/*.jar + layers from $nbhome/lib/*.jar) and a user
layer (r/w; $nbuser/system/ + layers from $nbuser/modules/*.jar +
layers from other *.jar). Why not change this to inst. layer (r/o;
$nbhome/system/ + layers from all *.jar) and user layer (r/w;
$nbuser/system/)? Why do certain module layers have to be in the user
ModuleLayeredFileSystem and not others? Makes no sense to me.

Layers: Currently administrator may modify the shared installation and
those changes can be overriden by user modules and user system dir. To
make it easier to satisfy this goal the installation and the user
layers are separated (including module jars).

Created attachment 4793 [details]
Proposed patch

See the latest patch - incorporates pieces of both Yarda's and Petr's
patches, as well as some new stuff and corrections. It seems to be
working pretty well; run with -J-Dnetbeans.cache.layers=true and
preferably also -J-Dorg.netbeans.core.projects=0 and perhaps
-J-Dnetbeans.cache.layers.prettyprint=true
-J-Dorg.netbeans.core.modules=0. Cache hit/miss logic is implemented
though not completely tested yet; at least enabling/disabling modules
works as usual.

(Note: you need to first correct a syntax error in the form layer:
gratuitous url= for a <file> with CDATA contents. Also due to a bug in
NbErrorManager which I will correct, [org.netbeans.core.projects] log
messages do not reliably appear in the log file.)

There are a few more things I want to do to it before committing,
including making exact measurements of what the improvement is.

Note to Yarda: there is not an interface for plugging in a different
cache impl, mainly because the current integration is dependent on
using XMLFileSystem as the second delegate. Choosing a different cache
mechanism should not be too hard but you would still need to redesign
the initialization of ModuleLayeredFileSystem a bit, I think. Anyway
in agreement with Petr's measurements, I see cache-load times around
1100ms for stable-with-apisupport (I assume Petr's 950 was from stable
config), which doesn't seem unreasonable considering the number of
files and attributes in the merged layer (~ 400K of XML without
whitespace). If we later want to reduce this time (and heap
consumption) a btree or similar impl should work, but of course then
core would have to do the merging work, not XMLFileSystem.

Startup time improvement: Using moduleconfig 'stable', JDK 1.4.0,
Linux 2.2.12 on Toshiba Tecra 8000 w/ 256MB RAM, running X and Emacs
and Bash only. Methodology: create two fresh user dirs (one for no
cache, one for cache). Go through four priming runs and then ten
measured runs, each consisting of a startup (netbeans.close=true) with
caching off, then one with caching on (interleaved). First priming run
creates cache, of course; others just compensate for disk cache
differences. Avg. time w/o cache: 25.2sec; w/ cache: 23.8sec; delta:
1.4sec = 5.5% (percentage would be higher for JDK 1.3.1_02 because
total startup time is more like 20sec).

I am changing tomcatint/tomcat32/manifest.mf to use
[org.apache.tomcat.core.Response] rather than
[org.apache.tomcat.core.Context]. For some reason I have not been able
to determine, when the cache is in effect (but only then), the package
dependency check on Context takes some 375msec (!) rather than the
usual 25msec or so. [Observed with JDK 1.3.1_02, moduleconfig
stable-with-apisupport.] There is no such differential for Response,
an interface in the same package. No other module's package dependency
check is similar affected. -J-verbose:class and grepping for tomcat or
jasper shows that only three or four such classes are being loaded,
whether or not the cache is on, and Context is always loaded anyway
because something else needs it. I have no idea why the time required
to do a dependency check would be affected by presence of the cache;
the only thing different when the cache is on is that the
SystemFileSystem from early on has a big XMLFileSystem in it rather
than an empty XMLFileSystem (with no URLs). Someone else can
investigate this, maybe.

Done.

committed     Up-To-Date  1.43       
form/src/org/netbeans/modules/form/resources/layer.xml
committed     Up-To-Date  1.5         tomcatint/tomcat32/manifest.mf

committed   * Up-To-Date  1.13       
core/src/org/netbeans/core/projects/ModuleLayeredFileSystem.java
committed   * Up-To-Date  1.25       
core/src/org/netbeans/core/projects/SystemFileSystem.java
committed   * Up-To-Date  1.53       
openide/src/org/openide/filesystems/XMLFileSystem.java

Try -J-verbose:gc
The VM does two full-gc cycles during the startup
for no apparent reason but mostly in the same place each time,
if you keep the memory activity similar.
You've changed the memory activity which may have moved the full gc
to different place where you may not notice it.

Ah, that makes sense re. GC timing. Didn't think of checking that.

Anyway, I still seem to be missing a few hundred milliseconds
somewhere, haven't figured out where yet. I.e. when you compare times
to call setURLs without cache, vs. time to read cache (~1000msec) plus
checking timestamps and misc (~75msec), the differential seems to be
somewhat more than the actual time improvement. So I am guessing that
something else is being slowed down by a few hundred msec, but not
sure what.

In the original way parsing, MFS have created full tree of SFS,
now it does in incrementally and only used part of the SFS is
reflected in the MFS structures. Probably part of the spared time
is re-spent during SFS lookups.

I'm reporting problems with cache:
if you break IDE session by pressing Ctrl+Break, cache is not
consistent and IDE will not start anymore, until you clean the cache
manually. This should be more robust IMO.
----------------------------
java.io.IOException: org.xml.sax.SAXException: Premature end of input.
: file:/c:/Netbeans/Configs/MainTrunk/cache/all-layers.xml
        at
org.netbeans.core.projects.SystemFileSystem.create(SystemFileSystem.java:404)
        at
org.netbeans.core.projects.SessionManager.create(SessionManager.java:72)
        at
org.netbeans.core.NonGui.createDefaultFileSystem(NonGui.java:233)
        at
org.netbeans.core.NbTopManager.getRepository(NbTopManager.java:316)
        at org.netbeans.core.NonGui.run(NonGui.java:447)
        at org.netbeans.core.Main.run(Main.java:213)
        at
org.openide.TopManager.initializeTopManager(TopManager.java:120)
        at org.openide.TopManager.getDefault(TopManager.java:81)
        at org.netbeans.core.Main.main(Main.java:346)
        at org.netbeans.core.TopThreadGroup.run(TopThreadGroup.java:81)
        at java.lang.Thread.run(Thread.java:484)
Cannot add System filesystem: c:\Netbeans\Configs\MainTrunk\system,
exiting...
Press any key to continue . . .

It worked correctly until it was replaced by the binary cache ;-)