View All Attachments for Bug 21675

This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Attachment #6355
Unfinished document	text/html	2002-06-20 17:13:47 UTC	10.63 KB	no flags	Details

<?xml version="1.0" encoding="US-ASCII"?><!-- -*- xhtml -*- -->
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
 <head>
  <title>Caching Module Resources - DRAFT</title>
  <link rel="stylesheet" type="text/css" href="http://www.netbeans.org/netbeans.css"/>
  <meta name="CATEGORY" value="Proposal"/>
  <meta name="DESCRIPTION" value="Proposed mechanism for caching module manifests, layers, classes, and resources during startup."/>
  <meta name="AUTHOR" value="Jesse Glick"/>
  <meta name="INDEXED" value="N"/>
 </head>
 <body>

  <h1>Caching Module Resources - DRAFT</h1>
  <dl>
   <dt><em>Version:</em></dt>
   <dd><!-- hhmts start -->February 19, 2002<!-- hhmts end --></dd>
   <dt><em>Author:</em></dt>
   <dd>Jesse Glick</dd>
   <dt><em>Abstract:</em></dt>
   <dd>
    <p>
     A significant portion of NetBeans' startup time is spent opening module JAR
     files, retrieving classes and resources from these JARs, examining
     manifests, and parsing and merging XML layers, followed by firing changes in
     the layers despite the fact that no files have yet been changed. This
     document proposes a way to collapse these tasks into a streamlined process
     involving opening a limited number of JARs containing classes and resources
     essential to startup and loading a premerged layer before any changes might
     be fired. The optimization should be transparent to module authors and users
     alike and require no API modifications.
    </p>
   </dd>
  </dl>

  <h2>Justification and Filed Issues</h2>
  <p>
   Too much startup time is spent loading certain kinds of information from
   modules, where this information does not typically change between consecutive
   runs.
  </p>

  <h3>Time drains</h3>
  <dl>
   <dt>Opening JARs</dt>
   <dd>
    <p>
     Opening too many JARs during startup is undesirable because it requires some
     operating-system resources to physically open the JARs; time to read the
     full ZIP entry list for all enabled modules; and memory to keep the entry
     lists. Searching through JAR entry lists when loading resources also
     consumes a bit of time.
    </p>
    <p>
     It would be preferable to open only a limited number of JAR files during
     startup. For example, open just JARs in the classpath (as controlled by the
     JVM's application classloader), and one "jumbo jar" containing critical
     resources.
    </p>
    <p>
     Since most modules are not likely to be used during startup, their actual
     JARs should be opened only when initiating an unusual action that requires
     loading classes or resources not normally used. This can be accomplished by
     keeping a special JAR containing only those resources expected to be used
     during the VM session, adaptively based on previous runs.
    </p>
   </dd>
   <dt>Loading layers</dt>
   <dd>
    <p>
     A moderate amount of time is spent loading XML layers. Some of this is
     simply XML parsing time. Layers as written by a module author often include
     comments and whitespace which are not needed at runtime; or CDATA sections
     which need not frequently be loaded. A little time is also spent switching
     parser streams, and some time is spent merging layers together.
    </p>
    <p>
     A cache could keep a precomputed merged master layer containing files from
     all module layers, which could be loaded in one sweep. URL-referenced file
     contents as well as literally included contents could be stored separately
     from the merged layer XML.
    </p>
    <p>
     One optimization not covered here is to minimize the number of file
     attributes actually present, specifically
     <code>SystemFileSystem.localizingBundle</code>,
     <code>SystemFileSystem.icon</code>, and relative ordering constraints. Other
     proposed optimizations cover these improvements.
    </p>
   </dd>
   <dt>Firing changes in the system file system</dt>
   <dd>
    <p>
     A large chunk of time during startup (a couple of seconds) is spent firing
     changes in the system file system after the module layers have been added to
     the layer stack. This is gratuitous.
    </p>
    <p>
     A cache could load merged layers before creating the system file system,
     assuring that no file events need be fired.
    </p>
   </dd>
   <dt>Computing lookup</dt>
   <dd>
    <p>
     A lot of time is spent preparing lookup. Effectively this means parsing
     <samp>*.instance</samp> and especially <samp>*.settings</samp> files to find
     their declared interface types and remembering which files can provide
     instances for a given class.
    </p>
    <p>
     Yarda Tulach already has a proposed way of speeding this up by serializing
     the index portion of lookup between sessions so that instances can be found
     directly. A cache system could help manage expiration and recreation of such
     a serialized lookup, rather than having to implement it as a separate
     component.
    </p>
   </dd>
  </dl>

  <h3>Filed Issues</h3>
  <ul>
   <li>
    <p>
     #17722 generally asks for persistent storage of module state, encompassing a
     number of issues.
    </p>
   </li>
   <li>
    <p>
     #20168 suggests that XML layers be preparsed and premerged and installed
     into the system filesystem when it is created.
    </p>
   </li>
   <li>
    <p>
     #20190 proposes that lookup be persisted across sessions.
    </p>
   </li>
   <li>
    <p>
     <a href="db/index.html">Scalability Index</a>
    </p>
   </li>
   <li>
    <p>
     #20358 suggests to reduce the number of relative ordering constraints.
    </p>
   </li>
   <li>
    <p>
     #18310 suggests that <code>SystemFileSystem.localizingBundle</code> and
     <code>SystemFileSystem.icon</code> attributes could be eliminated.
    </p>
   </li>
   <li>
    <p>
     #20628 proposes storing layer data in a binary format.
    </p>
   </li>
  </ul>

  <h2>Proposed Implementation</h2>
  <p>
   The proposed implementation involves a compact cache kept in the user
   directory which stores module manifests, merged layers, and commonly-used
   classes and resources. The cache is controlled by a single hash key which
   should change if significant changes are made to the execution environment but
   not between typical consecutive runs of the IDE. If there is a cache hit on
   startup, the cache is hit. If there is a miss, resources are loaded the slow
   way, and upon exiting the IDE the cache is recreated according to state at
   shutdown time.
  </p>
  <p>
   It should be possible to dynamically switch off the cache completely.
   Effectively this means that startup behaves exactly like a cache miss, and no
   cache recreation will be done at shutdown time.
  </p>

  <h3>Physical Storage Format</h3>
  <p>
   <strong>XXX</strong> the package mapping will not work, since two modules may
   have their own private copies of some library, etc.
  </p>
  <p>
   The cache should be kept in the user directory, as this provides an assuredly
   writable area, and unlike temporary directories it has a good chance of
   corresponding directly to which user is running NetBeans and in which
   configuration. It should be a single JAR file, say
   <samp>$userdir/cache.jar</samp>. It can serve multiple functions, to avoid
   needing to open more than one JAR.
  </p>
  <p>
   Although it might be a bit faster to use a special binary format to store such
   things, using XML, text files, and a JAR has the advantage of being much
   easier to debug and diagnose problems in. Compared to all the activity that
   takes place during startup, using textual formats will probably not be
   significant overhead. Of course, the cache format could be changed in the
   future to use a binary format such as a B-tree, if this proves desirable
   enough to justify the opacity.
  </p>
  <p>
   The following files in the JAR serve various functions:
  </p>
  <dl>
   <dt><samp>key.txt</samp></dt>
   <dd>
    <p>
     One line containing the cache key, as an integer.
    </p>
   </dd>
   <dt><samp>jars.txt</samp></dt>
   <dd>
    <p>
     An unsorted list of JARs (modules only - not their locale variants,
     extensions, or patches), one per line as an absolute filename.
    </p>
   </dd>
   <dt><samp>packages.txt</samp></dt>
   <dd>
    <p>
     Unsorted list of packages found in module JARs (and extensions and locale
     variants and patches) as well as on the classpath. Each line consists of a
     package name (format: <samp>org/netbeans/modules/foo/</samp>), then a space,
     then an integer which is either <em>either</em> a JAR index starting at one
     from <samp>jars.txt</samp> - in case the package is to be found in a module
     JAR (or its variants or extensions or patches) - <em>or</em> a zero in case
     the package is to be found in the classpath. Note that NetBeans already does
     not permit two modules to split up a package.
    </p>
   </dd>
   <dt><samp>META-INF/<em>pathremainder</em>/<em>integer</em>/<em>integer</em></samp></dt>
   <dd>
    <p>
     All files found in <samp>META-INF/</samp> and its subdirectories, including
     <samp>META-INF/MANIFEST.MF</samp> manifests, are included in the cache JAR,
     with the filename suffixed with a slash, the module number, another slash,
     and a <em>sequence number</em> (starting at zero and allocated incrementally
     for each module number). For a given <samp>META-INF/**</samp> resource name
     found among a module JAR, its variants, extensions, and patches, all
     available instances of that resource are included, each with its own
     sequence number - patches first in an arbitrary order, then the module, then
     extensions and variants in an arbitrary order.
    </p>
   </dd>
   <dt>all other paths</dt>
   <dd>
    <p>
     All other paths (which must be in some package, else they are not loadable
     anyway, and must not be in <samp>META-INF/</samp> because of the above rule)
     represent that exact class or resource in some module JAR (or variant or
     extension or patch). Only classes and resources considered essential by the
     cache are kept this way; others can be found in the original JAR according
     to <samp>packages.txt</samp>.
    </p>
   </dd>
  </dl>

  <h3>Cache Key</h3>
  <p>
   <strong>XXX</strong> hash of $CLASSPATH, locale, branding, file names and
   timestamps of JARs in systemClassLoader (incl. branding- and locale-variants)
  </p>

  <h3>Startup Procedure During Cache Hit</h3>
  <p>
   <strong>XXX</strong>
  </p>

  <h3>Startup Procedure During Cache Miss</h3>
  <p>
   <strong>XXX</strong>
  </p>

  <h3>Cache Recreation</h3>
  <p>
   <strong>XXX</strong>
  </p>

  <h2>Issues</h2>
  <p>
   <strong>XXX</strong> interaction with project &amp; session switching
  </p>

 </body>
</html>