Bug 190992 - NbModuleSuite does not work from Maven projects
NbModuleSuite does not work from Maven projects
Status: VERIFIED FIXED
Product: platform
Classification: Unclassified
Component: NB JUnit
7.0
All All
: P2 with 5 votes (vote)
: 7.0
Assigned To: Jesse Glick
issues@platform
:
Depends on: 195302
Blocks:
  Show dependency treegraph
 
Reported: 2010-10-14 01:09 UTC by Jesse Glick
Modified: 2011-04-21 11:53 UTC (History)
8 users (show)

See Also:
Issue Type: DEFECT
:


Attachments
Application w/ a func test (148.36 KB, application/zip)
2010-10-26 21:26 UTC, Jesse Glick
Details
Alternate setup w/ test in app prj (149.01 KB, application/zip)
2010-10-26 22:02 UTC, Jesse Glick
Details
Functional test with JellyTools (151.97 KB, application/zip)
2010-11-10 20:00 UTC, Jesse Glick
Details
Error if 779ef84872c1 is reverted (10.95 KB, text/plain)
2010-11-10 20:04 UTC, Jesse Glick
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Jesse Glick 2010-10-14 01:09:18 UTC
When you run an NbModuleSuite-based test from an nbm project (or from a nbm-application project in which you have manually set up the Surefire plugin), you just get

junit.framework.ComparisonFailure: .../.m2/repository/org/netbeans/api/org-netbeans-modules-nbjunit/nbdev/org-netbeans-modules-nbjunit-nbdev.jar is in a folder named 'harness' expected:<[harness]> but was:<[org-netbeans-modules-nbjunit]>
        at junit.framework.Assert.assertEquals(Assert.java:81)
        at org.netbeans.junit.NbModuleSuite$S.findPlatform(NbModuleSuite.java:828)

so the module system cannot be loaded. (Ensuring that your dependencies include e.g. org-netbeans-core-startup does not suffice; NbMS refuses to load the module system from the classpath, regardless of enableClasspathModules.)

And of course cluster.path.final is not set, either.

Not clear if NbMS needs to be fixed, or nbm-maven-plugin needs to do something to preconfigure Surefire, or projects need some special configuration, or some combination of these things.
Comment 1 simpatico 2010-10-26 20:36:05 UTC
Cannot do TDD without this.
Comment 2 Jesse Glick 2010-10-26 21:26:12 UTC
Created attachment 102656 [details]
Application w/ a func test
Comment 3 Jesse Glick 2010-10-26 21:34:17 UTC
As of core-main #5c20a1a28c43 it is possible to run NbModuleSuite normally if you do some special configure of surefire in the module with the tests; it needs to define cluster.path.final with the list of clusters in the app. Not very satisfactory for several reasons:

1. You need to hardcode the relative source path to the application project, plus the list of clusters it will include.

2. The application project needs to be built *first*, even though it declares the module as a dependency. So e.g. to test changes, you must build the module, then build the app, then test the module.

It is probably possible to put the functional tests in the application project, which would partially solve #1 and fully solve #2; I can look into this.

In the long run, http://wiki.netbeans.org/DependencyBasedModuleIncludes is probably the way to go; you would have a branding/kit module which would contain the functional tests and express runtime (thus also test-scope) deps on the chunks of the platform it wanted to use. Then NbModuleSuite would need to be modified to simply load whatever modules it found on the test classpath, i.e. runInRuntimeContainer would use ${java.class.path} for bootCP.
Comment 4 Jesse Glick 2010-10-26 22:02:49 UTC
Created attachment 102660 [details]
Alternate setup w/ test in app prj

Arguably better setup, where the functional tests live in the application project, not an individual module. (They can still *refer* to module classes if desired, but this better reflects the reality that what is being tested is the whole app, not an isolated module.)

Cleaner in the sense that tests are now built and run as part of the app project, after module projects have been built and they have been packed into the target clusters. An annoyance is that there are no standard lifecycle bindings for tests in nbm-application projects: http://jira.codehaus.org/browse/MNBMODULE-107
Comment 5 Jesse Glick 2010-10-26 22:28:55 UTC
I also added commented-out instructions to application/pom.xml in the NB app archetype in revision 12936. I think this should suffice to get people up and running.
Comment 6 Jesse Glick 2010-10-26 23:08:19 UTC
Some problems I don't know how to solve:

1. You get various DuplicateException's on console:

WARNING [org.netbeans.core.startup.ModuleList]: Error encountered while reading org.netbeans.bootstrap
org.netbeans.DuplicateException: ~/.m2/repository/org/netbeans/modules/org-netbeans-bootstrap/nbdev/org-netbeans-bootstrap-nbdev.jar is a duplicate of org.netbeans.bootstrap
        at org.netbeans.ModuleManager.subCreate(ModuleManager.java:656)
        at org.netbeans.ModuleManager.create(ModuleManager.java:549)
[catch] at org.netbeans.core.startup.ModuleList$ReadInitial.run(ModuleList.java:1598)
        at org.openide.filesystems.EventControl.runAtomicAction(EventControl.java:125)
        at org.openide.filesystems.FileSystem.runAtomicAction(FileSystem.java:566)
        at org.netbeans.core.startup.ModuleList.readInitial(ModuleList.java:168)
        at org.netbeans.core.startup.ModuleSystem.readList(ModuleSystem.java:272)
        at org.netbeans.core.startup.Main.getModuleSystem(Main.java:171)
        at org.netbeans.core.startup.Main.start(Main.java:308)
        at org.netbeans.core.startup.TopThreadGroup.run(TopThreadGroup.java:114)
        at java.lang.Thread.run(Thread.java:662)

though these seem to be harmless.

2. I cannot get Jemmy/Jellytools to work:

java.lang.NoClassDefFoundError: org/netbeans/jemmy/operators/JDialogOperator while loading org.netbeans.jellytools.NbDialogOperator; see http://wiki.netbeans.org/DevFaqTroubleshootClassNotFound
        at org.netbeans.ProxyClassLoader.selfLoadClass(ProxyClassLoader.java:304)
...
Caused by: java.lang.NoClassDefFoundError: org/netbeans/jemmy/operators/JDialogOperator
...
Caused by: java.lang.ClassNotFoundException: org.netbeans.jemmy.operators.JDialogOperator starting from ModuleCL@7be8c2[org.netbeans.modules.jellytools.platform] with possible defining loaders null and declared parents [org.netbeans.MainImpl$BootClassLoader@111a3a4, ModuleCL@180cf2a[org.netbeans.core.multiview], ..., ModuleCL@1e12f6d[org.netbeans.modules.nbjunit], ...7 more]
        at org.netbeans.ProxyClassLoader.loadClass(ProxyClassLoader.java:264)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
        ... 48 more
Caused by: java.lang.ClassNotFoundException: org.netbeans.jemmy.operators.JDialogOperator
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
...

despite having test-scope deps on both org.netbeans.api:org-netbeans-modules-jellytools-platform and org.netbeans.external:jemmy-2.3.0.0.

3. Some other stack traces (harmless but may be related to cause of #2): http://jira.codehaus.org/browse/MNBMODULE-108
Comment 7 Jesse Glick 2010-10-26 23:30:13 UTC
In fact #2 works OK (need only test dep on jellytools.platform) if you work around MNBMODULE-108 by editing the manifest of org-netbeans-modules-jemmy-nbdev.jar:

Class-Path: ../../../external/jemmy-2.3.0.0/nbdev/jemmy-2.3.0.0-nbdev.jar

i.e. the relative path in the local repository. This seems very un-Mavenly but I am not sure what else to do; NbModuleSuite does not care that jemmy-2.3.0.0-nbdev.jar is on the test classpath, and the module system loads org-netbeans-modules-jemmy-nbdev.jar due to findEnabledModules but cannot find its extension.
Comment 8 Jesse Glick 2010-10-26 23:47:27 UTC
(In reply to comment #7)
> NbModuleSuite does not care that
> jemmy-2.3.0.0-nbdev.jar is on the test classpath

I think fixed in core-main #779ef84872c1. (jtulach please review.)
Comment 9 Jesse Glick 2010-10-27 00:02:43 UTC
(In reply to comment #6)
> You get various DuplicateException's on console

Fixed in core-main #a91111e4b744, though it is pretty hacky.
Comment 10 Jesse Glick 2010-10-27 00:46:35 UTC
(In reply to comment #6)
> Some other stack traces (harmless but may be related to cause of #2):
> http://jira.codehaus.org/browse/MNBMODULE-108

Fixed for next plugin version, applicable for most people to next upload of a NB release to bits.netbeans.org.
Comment 11 Quality Engineering 2010-10-28 02:52:17 UTC
Integrated into 'main-golden', will be available in build *201010280000* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/5c20a1a28c43
User: Jesse Glick <jglick@netbeans.org>
Log: #190992: prefer to use ${cluster.path.final} to find platform, as this does not assume that nbjunit.jar is in any particular place, and is thus better for running from Maven.
Comment 12 Jesse Glick 2010-10-28 14:37:55 UTC
So the executive summary of these changes:

Functional testing using Maven does not work out of the box in 6.9. To make it work, you need to use the 7.0 dev version of nbjunit; you need to create your own copy of at least org.netbeans.api:org-netbeans-modules-jemmy (6.9 or 7.0 dev version irrelevant) using nbm-maven-plugin 3.4-SNAPSHOT (or manually edit org-netbeans-modules-jemmy-RELEASE691.jar to remove Class-Path from META-INF/MANIFEST.MF); and you need to add more or less stuff to the POM depending on whether you use 3.3 or 3.4-SNAPSHOT to build. (In https://svn.codehaus.org/mojo/trunk/mojo/mojo-archetypes/netbeans-platform-app-archetype/src/main/resources/archetype-resources/application/pom.xml you can see what is needed using 3.4-SNAPSHOT; using 3.3 you must also add standard lifecycle bindings for test-compile, test-resources, and test phases using maven-{compiler,resources,surefire}-plugin respectively.) So, possible but not straightforward.

In 7.0, there should be a fixed nbjunit, released 3.4 plugin, an official repo uploaded using 3.4, and simple commented-out blocks in application/pom.xml in the Platform archetype that when activated will let you build & run functional tests using NbModuleSuite (optionally also with Jemmy/Jelly).
Comment 13 johanandren 2010-10-28 16:17:12 UTC
(In reply to comment #12)

Great news. We have been running a patched nbjunit for our qa-tests earlier. I am now trying to migrate our app from 6.7 to 6.9.1 and avoid our own patched nbjunit in preparation of 7.0. 

Following the instructions in your last comment Jesse, i still have problems. With the attached project "test in app prj" and a build of nbjunit from rev 5c20a1a28c43, setting netbeans.version to RELEASE691 and trying to build/run the tests i first get an exception about org-netbeans-insance scanner missing and after adding that I get a ClassDefNotFoundError about NbModuleLogHandler as though it was not picked up on the classpath.


java.lang.NoClassDefFoundError: org/netbeans/junit/internal/NbModuleLogHandler
        at org.netbeans.junit.NbModuleSuite$S.runInRuntimeContainer(NbModuleSuite.java:760)
        at org.netbeans.junit.NbModuleSuite$S.run(NbModuleSuite.java:575)
        at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
        ...


Are you sure it is possible to get it to run on 6.9.*?
Comment 14 Jesse Glick 2010-10-28 17:53:28 UTC
5c20a1a28c43 was only the first of several fixes to NbModuleSuite; for whatever reason the daily build uploader chose to list the first fix rather than the last. You definitely also need 779ef84872c1, and a91111e4b744 is also desirable.
Comment 15 Jesse Glick 2010-11-09 21:40:13 UTC
(In reply to comment #13)
> i first get an exception about org-netbeans-insance scanner missing

This was fixed for the 3.4 plugin as http://jira.codehaus.org/browse/MNBMODULE-100 but since it affects repository generation, not module building, people using bits.netbeans.org/maven2 will not "see" the fix until the next release is uploaded.
Comment 16 Jaroslav Tulach 2010-11-10 18:38:48 UTC
Causes regression. Now the test distribution does not work. The module/tests.jar gets included on classpath and is then loaded by wrong classloader. Steps to reproduce:

ok, here are steps to reproduce locally:

1) wget latest ide full zip distro from smetiste
2) wget latest testdist zip from smetiste
3) unzip both
4) ant -Dnetbeans.dest.dir=$PATH_TO_UNZIPPED_NB -Dtest.config=commit

lot of failures, problems of this type:
------------------8<----------------
junit] Testcase: org.netbeans.junit.NbModuleSuite$S@1e68d7a:        Caused an ERROR
    [junit] org/openide/filesystems/FileObject
    [junit] java.lang.NoClassDefFoundError: org/openide/filesystems/FileObject
    [junit]        at java.lang.Class.getDeclaredMethods0(Native Method)
    [junit]     at java.lang.Class.privateGetDeclaredMethods(Class.java:2427)
    [junit]      at java.lang.Class.getDeclaredMethods(Class.java:1791)
Comment 17 Jesse Glick 2010-11-10 20:00:53 UTC
Created attachment 102893 [details]
Functional test with JellyTools

Complete app using plugin 3.4 running a func test from the app prj using NbModuleSuite and Jemmy/Jellytools. Assumes local repo prepared from core-main #f5bdc1318df4 (i.e. current dev sources) with:

ant -Dcluster.config=platform build-nbms
mvn nbm:populate-repository -DforcedVersion=nbdev -DnetbeansInstallDirectory=nbbuild/netbeans -DnetbeansNbmDirectory=nbbuild/nbms
Comment 18 Jesse Glick 2010-11-10 20:04:40 UTC
Created attachment 102894 [details]
Error if 779ef84872c1 is reverted
Comment 19 Jesse Glick 2010-11-10 21:44:12 UTC
Had to back out the problematic change in core-main #784d5e578aeb; will look for another solution.
Comment 20 Marian Mirilovic 2010-11-10 21:49:59 UTC
Jesse, this is critical. We can't run any functional automated tests now.
Comment 21 Jaroslav Tulach 2010-11-10 22:02:24 UTC
Jesse reverted the change and I can confirm that in 8e2c6a7ac7f6 the testdist seems to run (at least the first few tests).
Comment 22 Filip Zamboj 2010-11-10 22:25:23 UTC
i run php tests on hudson4qe and the result is 

[junit] Testcase: org.netbeans.junit.NbModuleSuite$S@5717e4ff:	Caused an ERROR
[junit] org/netbeans/jellytools/JellyTestCase
[junit] java.lang.NoClassDefFoundError: org/netbeans/jellytools/JellyTestCase

but we probably have to wait till this is propagated so leaving opened for now.
Comment 23 Jaroslav Tulach 2010-11-10 22:58:24 UTC
OK, the temporary hiccups are over. Here are my comments to the whole effort:

Y01 The way to use NbModuleSuite shall be build system independent. There shall not be a reason why to do something in Ant and something different in Maven

Y02 Right now, the system may not always allow use of non-module libraries during test regime. This may be less and less important as various libraries are turned into modules/OSGi bundles, but still this is legitimate goal. To allow that basically add anything on classpath which is not module/bundle into netbeans.systemclassloader.patches variable in preparePatches.

Y03 The solution outline in Y02 does not work for libs.freemarker, probably. Make sure that maven versions of NetBeans module JARs encode information about their classpath in a way that it can be resurrected when Maven puts these JARs and their libraries on a flat classpath.
Comment 24 Jesse Glick 2011-01-15 01:15:06 UTC
For future reference, a clarification of Y01 and the root problem:

Unlike, say, OSGi bundles, NB modules are not in general self-contained. The "naked" module JAR is in many cases sufficient for the module's whole functionality, but sometimes other files packed in the NBM are also needed. The most common case of this is Class-Path extensions, conventionally in modules/ext/*.jar.

Now the issue is not the build system per se, it is the assumptions that NbModuleSuite makes about its execution environment. The current Ant-based module build harness happens to construct the runtime classpath for tests from module JARs located in their final install positions: $cluster/modules/*.jar. NbMS takes advantage of this fact by parsing ${java.class.path} and assuming that any entry with an OpenIDE-Module header is in cluster position; the regular classpath is ignored (under the assumption that everything "of interest" is really a module anyway, with fixed exceptions such as the compiled test classes), and the list of encountered modules is passed to the module system to be loaded in an entirely different way.

When tests are run from Maven, the classpath is constructed from regular artifacts, either target/classes or ~/.m2/repository/**.jar. Class-Path extensions, i.e. plain old dependencies, are inserted into the classpath from similar locations. This is all fine for typical tests, which merely expect all requested classes to be present on the classpath. But before 779ef84872c1, NbMS gets confused: library wrapper modules from the repository are "enabled" but the actual library (in the classpath) is ignored.

(Prior to MNBMODULE-108, the naked module JAR had a Class-Path header pointing to a relative location, which is nonsense for a JAR in the Maven repository, since Maven collects dependencies itself by other means. Fixing that issue prevented the module system from printing a warning about a nonexistent ext/*.jar, but did not point it to the actual location.)

779ef84872c1 changed NbMS to better honor the contents of the classpath, loading any non-module in it in the regular way. This generally fixed the problem with library wrapper modules - their classes were now accessible - but apparently led to issues in certain tests as some classes were loaded from a different loader than was intended.

The desired behavior is then for NbMS to figure out that when given a classpath containing a mixture of module JARs and their non-module extensions, certain JARs belong together. The problem is missing information. A Class-Path header pointing to a nonexistent location could at least be used to guess at the association (assuming no two JARs have the same basename!), but the fix of MNBMODULE-108 must be preserved (and in particular using a relative path within the local repo, such as "../../../../../junit/junit/4.8.2/junit-4.8.2.jar", is unacceptable). JARs built using maven-jar-plugin contain POM metadata which can be used to determine associations, but this metadata is not present for module JARs imported using nbm:populate-repository, not to mention many third-party JARs uploaded to Central such as JUnit.

The only plausible option seems to be for MNBMODULE-108 to be amended to include some other header in the naked module JAR indicating its expected Class-Path extensions in some format, which would be a private contract with NbMS. This is not ideal since the fix is then not easily available to other potential test harnesses - other IDEs, alternate Ant scripts using Ivy, etc.
Comment 25 Jesse Glick 2011-01-28 01:13:57 UTC
Y01/Y03 will be handled in conjunction with: http://jira.codehaus.org/browse/MNBMODULE-132

Deferring Y02 for another day; not needed to solve the immediate issue.
Comment 26 Jesse Glick 2011-01-28 01:27:24 UTC
(In reply to comment #3)
> You need to hardcode the relative source path to the application project,
> plus the list of clusters it will include.

See bug #194794.
Comment 27 Jesse Glick 2011-01-28 01:29:07 UTC
core-main #9fbff2f79c1d
Comment 28 Jesse Glick 2011-01-28 01:32:09 UTC
core-main #39cf30af621f
Comment 29 Jesse Glick 2011-01-28 20:17:27 UTC
Note: http://jira.codehaus.org/browse/MNBMODULE-131 may still cause problems for testing from a CI environment because "mvn clean test" on the reactor root will not work; you need "mvn clean package test".
Comment 30 Jesse Glick 2011-01-28 20:36:05 UTC
(In reply to comment #29)
> http://jira.codehaus.org/browse/MNBMODULE-131

Fixed for 3.5, so you can run simply "mvn clean integration-test".
Comment 31 Quality Engineering 2011-01-29 05:27:21 UTC
Integrated into 'main-golden', will be available in build *201101290000* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/39cf30af621f
User: Jesse Glick <jglick@netbeans.org>
Log: #190992: NbModuleSuite does not work from Maven projects
Comment 32 Jaroslav Tulach 2011-01-30 18:38:22 UTC
Adding a unit test to the NbModuleSuite change would be beneficial from long term perspective.

> XXX any better way to get list of URLs from loader?
URLClassLoader seems to have getURLs() method and class sun.misc.Launcher$AppClassLoader is subclass of URLClassLoader.
Comment 33 Jesse Glick 2011-01-31 15:27:33 UTC
(In reply to comment #32)
> Adding a unit test to the NbModuleSuite change would be beneficial from long
> term perspective.

I'll see what I can do. Writing tests for test infrastructure itself is particularly tricky since you are not always sure that a test result really reflects your code, rather than it being an artifact of the way the test was run.

> URLClassLoader seems to have getURLs() method

Yes, I considered that; the problem is that there is no real guarantee that the loader in question is in fact a URLCL. I guess I could try to cast to URLCL and fall back to the current code if that fails.
Comment 34 Quality Engineering 2011-02-02 06:18:01 UTC
Integrated into 'main-golden', will be available in build *201102020000* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main/rev/473ae33b17b6
User: Jesse Glick <jglick@netbeans.org>
Log: Unit test for #190992. Also minor cleanups: (1) no need to check for spec version anymore; (2) cleaner way of skipping pseudomodules.
End-to-test is more like:
ant -Dcluster.config=platform clean build-nbms
mvn -f .../nbm-maven-plugin/pom.xml install
mvn org.codehaus.mojo:nbm-maven-plugin:...-SNAPSHOT:populate-repository -DnetbeansInstallDirectory=nbbuild/netbeans -DnetbeansNbmDirectory=nbbuild/nbms -DforcedVersion=nbdev
mvn -f .../mvnfunctest190992/pom.xml integration-test
Comment 35 escay 2011-04-21 11:53:28 UTC
Verified fixed on RELEASE70 with nbm-module version 3.5
Netbeans GUI tests now run for our project without any workarounds.
Thanks for the great work!


By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo