Bug 53898

Without even thinking about usecases and requirements here is one
solution [question is what it is solution to, but I am sure it is
solution]:

Each module JAR file comes with a descriptive XML file in
config/Modules/ directory. It defines the state of the module
(enabled/disable), and type (regular, autoload, eager). This XML file
could be extended with voluntary element describing locations of extra
libraries:

<extra-cp>
  <path location="c:/Program Files/AppServer8/jars/jsr88.jar" />
  <path location="..." />
</extra-cp>

This extra classpath would be read and interprested by the module
system and would be included in the module's runtime classloader.

The cobundle installer would install appserver8 and netbeans and then
modify the files in nb4.1/config/Modules/ to include this extra-cp
element for modules that need to refer to libraries provided by
appserver8.

The UI of Modules node could be extended to allow the user to edit
this path - especially for disabled modules due to defined dependency
on a specific package: javax.help[Help] could allow the user to locate
a jar file on disk, check if the missing classfile is provided and if
so modify the $userdir/config/Modules/... with extra-cp element, so
from now on, the javahelp would get inserted into modules classpath
and the module could be enabled.

Created attachment 19906 [details]
Proposed enhancement in module system

Reviewers, I'd like to get a review. Fast one would be good, as there
is non-zero probability Ludo will insist on a solution to this problem
to be implemented for 4.1.

Ludo, I'd like you to try this patch and confirm that you can use
<extra-cp><path location="..."/></extra-cp> to refer to external
libraries and that it works correctly.

Jesse, can you please review the changes in code and the test and
suggest improvements? I am glad I made it work, but I am sure the code
could look nicer. Also please tell me what documentation/manifest
files to update. Increase version of openide? Etc... Thanks.

Don't like this so much since config/Modules/*.xml is supposed to be
distributed with an NBM and be installation-neutral. This mixes
installation details into the module configuration, where it does not
really belong IMHO.

I would rather suggest a somewhat different approach which is more
general (and probably simpler to implement):

1. Continue to use Class-Path in manifests to declare external
libraries, e.g.

Class-Path: ext/jsr88.jar

However make sure that Module interprets Class-Path using
InstalledFileLocator in case the direct interpretation (URI relative
to module JAR's dir) fails to match. This is what we already do for
e.g. localizing/branding JARs.

2. Modify the core InstalledFileLocatorImpl to search the installation
directories for files named e.g. "links/$code-name-base.properties"
(so in this case e.g. "links/com-sun-appserverplugin.properties"). The
keys are logical paths, the values are physical paths (either
absolute, or relative to the installation directory). E.g.

modules/ext/jsr88.jar=c:\\Program Files\\AppServer8\\jars\\jsr88.jar

The nice thing here is that you can have a regular NBM, which really
works (can include jsr88.jar), and the module can be tested as it
comes in that NBM (or in reload mode etc.); when you want to produce
an installer that works with an external program installation, all it
needs to do is write one properties file (no need to modify existing
XML files etc.).

What do you think? Should be very simple to do. 

Jesse, I like this proposal better. It's great both for internal work,
and the installer team can further customize the installation without
changing existing code/jars...

One tech question now around duplicated class loaders:

let's say 2 modules use ext/jsr88.jar  (symbolic name)

would you emit a warning is the real jar is the same for the 2 modules?
I think this is currently done for regular extension jars.

Also would we support relative links like:

modules/ext/jsr88.jar=../AppServer8/jars/jsr88.jar

This would help installer team to better handle cobundles of products
with NetBeans, in a platform independant way, assuming the bundled
products are installed in a known area, for ex:

C:\\MegaBundle\\Netbeans
C:\\MegaBundle\\AppServer
C:\\MegaBundle\\WebServer
C:\\MegaBundle\\Tomcat
...

Re. duplicated JARs: yes, I think the existing warning about
duplicating JARs in different class loaders should be kept and
continue to apply to actual file path. That should probably work with
no additional code changes. See Javadoc for
org.netbeans.core.modules.Module.extensionOwners.

Re. permitting relative paths to use "../" and thus jump out of the
cluster directory into a neighboring directory: I don't see why not. I
guess the example is not quite right, since you would really have e.g.

---%<--- /opt/NetBeans/platform5/links/com-sun-asplugin.properties
# Points to /opt/AS8/jars/jsr88.jar
# relative to /opt/NetBeans/platform5:
modules/ext/jsr88.jar=../../AS8/jars/jsr88.jar
---%<---

i.e. the base directory used for resolving relative paths is the
cluster. (Technically: ${netbeans.home}, ${netbeans.user}, or an entry
in ${netbeans.dirs}: the same as the roots used by
InstalledFileLocatorImpl.)

I think that this requirement may become very important in the context of
scripting support since the library dependencies are crucial.  We do not want to
be getting into a situation of X copies of RHino and jRUby and others.  It will
be helpful to have a current assessment/swag as to what the effort would be
today wrt to Nb 6.0

I realize that even consideration of this would have no bearing on something
like near term Phobos Tools which are targeting NetBeans 5.5 release.

Commenting on the solutions suggested by Jesse and Jarda: One additional aspect
that I believe needs to be addressed is that it is not sufficient to provide the
external jar path only by the installer - it must be possible to provide this
path at IDE runtime using UI provided by the module. For example, in an
installation without any appserver instance registered, the user can register
the server using the Server Manager (generic UI in module manager is not
sufficient). This also means that the module would be enabled or disabled based
on whether the external jar path is defined. Also, modules would need to have a
way to programatically set the external jar path.

Petr - this feature should not be used for dynamically registered external
libraries. That is best handled in your code, using ClassLoader's to split
module code across a set of interfaces. The feature should only be used for
libraries which we could have bundled statically as Class-Path extensions but
which we would prefer to use from a predefined external location known to
installation/packaging tools. For example, the Ant module permits the user to
select a different version of Ant to run. In such a case it is inappropriate to
use this feature since only the Ant module, not the module system, knows how to
properly handle runtime changes in the Ant installation (certainly the Ant
module should not be reloaded after such a change).

Also I would not propose any UI in the Module Manager! This feature is designed
for invisible use e.g. as part of an RPM/deb/solpkg/MSI package, or from a
standalone installer. The external library linkages should never be changed
while the IDE is running.

If that's the case, then that means that this enhancement request will have
limited usefulness, as in many real world cases (Ant module, and I believe also
the Appserver plugin), we will need to continue to use the current approach with
custom classloaders handcoded in the module. And that may well be the right
solution. 

To return back to the motivation, do we have examples of cases when providing
the external classpath statically at installation/startup time is sufficient,
and it is not necessary to change this path through UI at IDE runtime?

An example of a static use case would be JavaHelp - if it is installed on the
system in a reliable location and we are providing NB as a package using the
same package system (e.g. RPM) we would prefer to simply use the existing
installation. We do not permit the user to configure the JH path now and so this
feature is applicable. Generally - the feature should be useful where a library
is used as part of the IDE's functionality, but not when the library may be used
as part of the user's application.

Let's start once again. I've just met Ludo and he reminding me that this is still opened issue. That is why I coded 
the "external" manifest support. As far as I know this is exactly what our competition does to solve inclusion of JARs 
which are not native or are located elsewhere. 

My diff is missing documentation in apichanges and arch, but otherwise I consider it done. It would be nice to reuse 
it in 3rd party library wrapper one day, but that is definitely not a blocker for Ludo. I know I completely ignore all 
the above suggested tricks with properties, but I do not like them and I will not implement them myself.

If there are no objections, I'll polish the documentation and integrate sometime in second week of May, when J1 is 
over.

Created attachment 60998 [details]
Support for external manifest

Great to see some activity there.
I guess we need to test it with a real case: either GlassFish, or even better the Nb DTrace plugin that needs a jar from
/usr/share/lib/java area...

[JG01] A much simpler (and to my mind preferable) patch would just be to alter the code in the module system which
interprets Class-Path to permit an absolute path (or URL?) to be specified. No other changes to an NBM would be needed:
just change Class-Path to a full path on disk, and do not unpack the modules/ext/*.jar (or whatever). The build harness
ought to enforce simple relative paths in <class-path-extension>s, since a switch to an absolute path should only be
done as part of an installer or native package.

I would also like to take this opportunity to remind people that the applicability of such an RFE is limited to cases in
which runtime or user-directed customization of the library in question is unnecessary and inappropriate. For any kind
of dynamic behavior it is necessary to use a custom ClassLoader, which has always been possible without any support from
the module system.

I agree that one or more concrete use cases would be required in order to properly evaluate any solution.

The new DTrace gui plugin has dependencies on an external jar file (/usr/share/lib/java/dtrace.jar)
which cannot be bundled with the nbm file. The /usr/share/lib/java/dtrace.jar file is shipped with Solaris OS.
We should allow the platform specific plugins such as the DTrace plugin, to specify the absolute pathname 
to an external jar file via the module Class-Path. 

> We should allow the platform specific plugins such as the DTrace plugin, to specify the absolute pathname

What would be the behavior on other platforms? Would the module be disabled? If so, would the user be notified about the
module being disabled? If not, would it throw ClassNotFoundExceptions in the user's face or do something else?

My feeling is that this is not an API-only enhancement, there is also a UI/behavior aspect for which we need to have a spec.

pjiricka: 
As for the DTrace case it's possible to define OS dependency for that module. Then the module would be disabled on other
platforms than Solaris - and it would show the standard "The module blablabla is disabled blablabla..." dialog. So no
CNFE in this case.

What is the status of this feature? The DTrace GUI Plug-in can use this feature if it is ready.

No one is working on this, and it is not clear whether anyone will. You don't *need* it for the DTrace plugin since you
can always load /usr/share/lib/java/dtrace.jar dynamically using URLClassLoader, rather than using Class-Path.

As author of the last patch
http://www.netbeans.org/nonav/issues/showattachment.cgi/60998/X.diff
I think I can donate some time and integrate it. I am not sure if Jesse's yesterday's comment means that he wants to 
block the change or he just does not want to do it himself. Thus, I'll delay the integration till Monday.

I still prefer the solution suggested in JG01. Implementation would be much simpler (merely the removal of three lines
of code from StandardModule.java), and I think usage is more intuitive: rather than

Class-Path: ext/dtrace.jar

you have

Class-Path: /usr/share/lib/java/dtrace.jar


[JG02] As I've said before, I'm not fully convinced it is even desirable to implement this RFE. I would urge module
authors to instead consider dynamic loading of the external JAR, which offers much more flexibility: you can select the
external JAR location from among several alternatives, display a polite dialog in case it is missing (or even offer to
install the native package which contains it), deal with version skew intelligently, etc.

Another alternative would be a post-install action in the NBM which would locate and validate the external JAR, then
create a symlink to it from .../modules/ext/*.jar.

The post-install solution works fine as long as the plugin is installed from the NetBeans Update Center. In fact, the
DTrace GUI uses post-install to copy the dtrace.jar file to the module/ext directory. 

However, the post-install solution doesn't work for the modules that have dependencies on a platform specific jar file
and  they are already installed/integrated with NetBeans.

For the URLClassLoader soultion, I would like to know whether your suggestion works with both the plugin modules and for
the modules that are already integrated with NetBeans. I would appreciate it if you point me to the modules and plugins
that are using URLClassLoader to load a platform specific jar file. 

"the post-install solution doesn't work for the modules that have dependencies on a platform specific jar file and they
are already installed/integrated with NetBeans" - obviously if the module is installed in some other way, not from NBM,
then you would use that mechanism's installation hooks. I don't know about Solaris, but Linux packages can directly
specify symlinks as part of the content of the package, which would trivially satisfy your requirement: your plugin
package would depend on both the DTrace package and some NB package, and include a link from .../modules/ext/dtrace.jar
-> /usr/.../dtrace.jar. Of course Linux packages can also run scripts during installation, in case a static symlink was
not flexible enough.

"For the URLClassLoader solution, I would like to know whether your suggestion works with both the plugin modules and
for the modules that are already integrated with NetBeans." - I'm not sure exactly what the distinction is in your mind,
but it should make no difference.

"I would appreciate it if you point me to the modules and plugins that are using URLClassLoader to load a platform
specific jar file." - I don't know about "platform-specific" JAR files, but the Ant integration module uses (a subtype
of) URLCL to load ant.jar from an arbitrary location. There is no special NB API involved here, just plain Java
programming: you create the class loader, you load some entry point class, you call it using reflection. If there are a
lot of entry points, generally it would be easier to create an interface in the module representing them all, an impl in
a separate JAR which can link against the foreign JAR, then load the impl plus the foreign JAR together, instantiate the
impl, cast to the interface, and continue from there.

Re. Solution JG01 requires repackaging of a JAR file. This is too complicated for my taste. Better to generate simple 
text file, imho.

Re. JG02 own classloading is imho needlessly complicated too. Imho, it is much simpler to generate dtrace.mf with
OpenIDE-Module: com.sun.dtrace
and into config/Modules/com-sun-dtrace.xml add
<manifest>dtrace.mf</manifest>
but if you really do not want me to integrate this change, then remove the keyword by Monday. Thanks.

You do not need to repackage the JAR file if you write

Class-Path: /usr/share/lib/java/dtrace.jar

into ${basedir}/manifest.mf to begin with. You would also need

cp.extra=/usr/share/lib/java/dtrace.jar

to compile. This assumes that you always want to run the module using the external JAR, i.e. that it is designed this
way from the beginning.

Anyway editing the manifest of a packaged JAR is quite trivial. In Emacs, just C-x C-f, select the JAR, move down to
META-INF/MANIFEST.MF, press f, edit, C-s to save, and you are done. Seems simpler & easier than extracting the manifest,
saving it somewhere, editing it, then editing a separate XML file to point to the copy.

As to class loading, it is not "needlessly" complicated; it is a little more work, but in return you get a much more
flexible and robust module, which can safely be installed without precondition onto any system, and decide at runtime
how to handle missing external dependencies. (If your module is installed only by nonstandard mechanisms such as native
packages, then you do not need help from the NB module system anyway, as you can use simple symlinks.)

And any such inconveniences seem to me preferable to introducing a new API and nontrivial implementation into the module
system. So overall I would prefer the patch not be integrated.

I have already tried most of your suggestions few months ago which includes: 

* Creating the link from the module/ext to the /usr/.../dtrace.jar
* Modifying the manifest file manually
* Using the URLClassLoader

None of them worked for me with the exception of the post-install solution. Which means copying the /usr/.../dtrace.jar
file into the module/ext directory.

It is possible that I didn't implement them correctly and that is why I would like you to be more specific and point me
to the modules that are already using any of those solutions successfully.

I already directed you to the Ant integration module which uses URLClassLoader (o.apache.tools.ant.module in NB sources).

I don't see any compelling need that can't be met either by installation/packaging systems. Providing a feature that
would be implemented by the core module system seems inappropriate, since an NBM cannot portably refer to a possibly
nonexistent external file anyway.

Dynamic class loading and communication through stub interfaces is definitely the preferable technique. If there is a
need for some support API that would make it easier to manage the class loaders, that is certainly possible, though I
suspect there is not much to write that is not specific to the situation.