This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 219097 - Test Results reports 100% success in spite of failures
Summary: Test Results reports 100% success in spite of failures
Status: RESOLVED FIXED
Alias: None
Product: projects
Classification: Unclassified
Component: Maven (show other bugs)
Version: 7.2
Hardware: All Windows 7
: P3 normal (vote)
Assignee: Milos Kleint
URL: https://jira.codehaus.org/browse/SURE...
Keywords:
Depends on:
Blocks:
 
Reported: 2012-09-27 05:19 UTC by _ gtzabari
Modified: 2012-12-13 17:29 UTC (History)
2 users (show)

See Also:
Issue Type: DEFECT
Exception Reporter:


Attachments
Testcase (3.04 KB, application/octet-stream)
2012-11-28 17:07 UTC, _ gtzabari
Details

Note You need to log in before you can comment on or make changes to this bug.
Description _ gtzabari 2012-09-27 05:19:11 UTC
Product Version: NetBeans IDE 7.2 (Build 201207171143)
Java: 1.7.0_07; Java HotSpot(TM) 64-Bit Server VM 23.3-b01
System: Windows 7 version 6.1 running on amd64; Cp1252; en_CA (nb)
User directory: C:\Users\Gili\AppData\Roaming\NetBeans\7.2
Cache directory: C:\Users\Gili\AppData\Local\NetBeans\Cache\7.2

I defined three classes containing TestNG tests. When I hit ALT+F6 the Test Results window prints: "All 24 tests passed. (6.93 s)" however when I open the Output Window I see:

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running TestSuite
org.apache.maven.surefire.util.SurefireReflectionException: java.lang.reflect.InvocationTargetException; nested exception is java.lang.reflect.InvocationTargetException: null
java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:601)
	at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164)
	at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110)
	at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175)
	at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:107)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68)
Caused by: org.testng.TestNGException: 
DependencyMap::Method "LocationsTest.removeNonExistentLocation()[pri:0, instance:com.foo.location.server.LocationsTest@77c42952]" depends on nonexistent group "postLocation"
	at org.testng.DependencyMap.getMethodsThatBelongTo(DependencyMap.java:45)
	at org.testng.TestRunner.createDynamicGraph(TestRunner.java:1079)
	at org.testng.TestRunner.privateRun(TestRunner.java:734)
	at org.testng.TestRunner.run(TestRunner.java:617)
	at org.testng.SuiteRunner.runTest(SuiteRunner.java:334)
	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:329)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:291)
	at org.testng.SuiteRunner.run(SuiteRunner.java:240)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:51)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:85)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1197)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1122)
	at org.testng.TestNG.run(TestNG.java:1030)
	at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:70)
	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeMulti(TestNGDirectoryTestSuite.java:158)
	at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:98)
	at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:111)
	... 9 more

Results :

Tests run: 0, Failures: 0, Errors: 0, Skipped: 0


Expected behavior: Test Results window should display a failure.

I believe you should be able to reproduce this failure by defining this method:

	@Test(groups = "deleteLocation", dependsOnGroups =
	{
		"postLocation", "getLocation"
	})
	public void removeNonExistentLocation() {}

where group "postLocation" doesn't exist.
Comment 1 Theofanis Oikonomou 2012-10-31 12:43:29 UTC
After some evaluation I have these results:

created maven javase project. removed junit as test dependency, created a testng test class and defined the reporter's method.

1) CoS OFF: everything works as expected. Tried these scenarios:
   a) test project => "No tests executed" message in TRW, exception in OW

      commented (groups = "deleteLocation", dependsOnGroups = {"postLocation", "getLocation"})
   b) test project => "The test passed" message in TRW, no exception in OW

      uncommented (groups = "deleteLocation", dependsOnGroups = {"postLocation", "getLocation"})
   c) test project => "No tests executed" message in TRW, exception in OW
   d) test file => "No tests executed" message in TRW, exception in OW

      commented (groups = "deleteLocation", dependsOnGroups = {"postLocation", "getLocation"})
   e) test file => "The test passed" message in TRW, no exception in OW

2) CoS ON: Test Project does not work as expected. Tried the same scenarios:
   a) test project => "No tests executed" message in TRW, exception in OW

      commented (groups = "deleteLocation", dependsOnGroups = {"postLocation", "getLocation"})
   b) test project => "The test passed" message in TRW, no exception in OW

      uncommented (groups = "deleteLocation", dependsOnGroups = {"postLocation", "getLocation"})
   c) test project => "The test passed" message in TRW, exception in OW => this is wrong!!!
   d) test file => "No tests executed" message in TRW, exception in OW

      commented (groups = "deleteLocation", dependsOnGroups = {"postLocation", "getLocation"})
   e) test file => "The test passed" message in TRW, no exception in OW

So there should be an issue with CoS with test Project action as shown in case 2c. re-assigning for evaluation. Thank you
Comment 2 Milos Kleint 2012-11-05 09:17:27 UTC
I will do some more in depth investigation, but most likely something with surefire itself (internal error)

Tests run: 0, Failures: 0, Errors: 0, Skipped: 0 suggests that we have no failures and no errors, so I assume even the surefire xml file doesn't contain the exception as a failure nor error (IMHO should be error)
Comment 3 Milos Kleint 2012-11-05 12:12:46 UTC
theofanis: it doesn't matter if CoS is on or off.. You probably failed to notice that in some situations the build fails with "nonexistent group "postLocation"" but the test output window actually shows the failure from previous execution.

if all the steps get separated by a mvn clean execution, then the Test project action will always fail the same way. The problem is primarily in surefire itself. in the case of our failure, the target/surefire-reports/TEST-*.xml file is not generated thus we have nothing to populate our test output window from. 

So either it's a problem in surefire itself when it's badly processing this sort of error. Or it's more or less correct, but we don't handle it.. 
which is unlikely IMHO because the maven build doesn't list the failed test as failed, and at the end prints BUILD SUCCESS which is wrong as far as I'm concerned. -> it's a surefire testng failure.
Comment 4 Milos Kleint 2012-11-05 12:15:50 UTC
2.12.4 of surefire maven plugin already fails instead of finishing as successful.
Comment 5 Milos Kleint 2012-11-05 12:39:57 UTC
http://hg.netbeans.org/core-main/rev/053abdf729ee
fixes the problem of previous execution display.. the rest is most likely to be filed against surefire I suppose.
Comment 6 Quality Engineering 2012-11-07 03:42:19 UTC
Integrated into 'main-golden', will be available in build *201211070001* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress)
Changeset: http://hg.netbeans.org/main-golden/rev/053abdf729ee
User: Milos Kleint <mkleint@netbeans.org>
Log: #219097 ignore results from previous invokations.
Comment 7 _ gtzabari 2012-11-14 01:29:38 UTC
NetBeans IDE Dev (Build 201211060001)

The issue doesn't seem to be fixed. Please run the test-case on your end before resolving.
Comment 8 Milos Kleint 2012-11-28 08:52:13 UTC
(In reply to comment #7)
> NetBeans IDE Dev (Build 201211060001)
> 
> The issue doesn't seem to be fixed. Please run the test-case on your end before
> resolving.

What exactly doesn't work? I didn't say I fixed it entirely (see comment 5). I just said that we no longer populate the test output window using results from previous execution. That's what I have tested and I believe it worked (please note that I've fixed/processed like 50 bugs since then and don't exactly remember).  And I said that the rest is surefire plugin problem and should be filed there (http://jira.codehaus.org/browse/SUREFIRE)

questions for you: have you upgraded your project to surefire 2.12.4? when executing, does surefire mention that the affected test failed in maven output? does it generate correct (with failure) xml file in target/surefire-reports/*.xml?

closing as fixed (wontfix is also an alternative but we did fix a problem in our codebase as well), the rest is not a bug in netbeans but external tool.
Comment 9 _ gtzabari 2012-11-28 17:07:02 UTC
Created attachment 128526 [details]
Testcase

I apologize. I should have been more specific. I am running Build 201211270002 with the following repro steps:

1. Open the attached testcase (it uses Surefire 2.12.4 which I agree improves the situation).
2. Hit ALT+F6 to run tests
3a. Scenario 1: If the Test Results tab is open
4a. The Test Results tab reads "No tests executed".
3b. Scenario 2: If the Test Results tab is closed
4b. Netbeans pops up a message: "Tests finished successfully for project: X. Open Test Results Window"
5. Click on the Output tab
6. Notice Surefire says "The forked VM terminated without saying properly goodbye. VM crash or System.exit called ?"
7. If you check the "surefire-reports" directory there is no indication of success or failure, because the JVM terminated before writing to disk.

So there are 3 separate issues here (2 of which are caused by and should be fixed in Netbeans)

1. I believe the TestNG plugin is causing the JVM to exit prematurely. We should investigate what's going on there and fix it.
2. "No tests executed" is technically correct because right now there are no result files. I believe this will get fixed automatically once we fix TestNG.
3. "Tests finished successfully for project: X" is wrong and should be fixed.
Comment 10 Milos Kleint 2012-11-29 08:40:34 UTC
(In reply to comment #9)
> 
> 1. I believe the TestNG plugin is causing the JVM to exit prematurely. We
> should investigate what's going on there and fix it.
> 2. "No tests executed" is technically correct because right now there are no
> result files. I believe this will get fixed automatically once we fix TestNG.
> 3. "Tests finished successfully for project: X" is wrong and should be fixed.

1+2 are clearly out of scope for netbeans integration. please file it against surefire with the attached project.

3. is technically correct. the test output window defines success as no failed tests, which the current scenario fulfills. the fact that the build actually fails within surefire execution (or after) is out of scope for the test output window. This will correct itself as well once the surefire bug is fixed.
Comment 11 _ gtzabari 2012-11-29 16:25:01 UTC
Milos,

I believe #1 is in our scope because we authored the TestNG <-> Netbeans plugin. We should verify whether the JVM exit is caused by TestNG itself or our plugin. If you agree, can you redirect this issue there?
Comment 12 Milos Kleint 2012-11-29 19:00:44 UTC
(In reply to comment #11)
> Milos,
> 
> I believe #1 is in our scope because we authored the TestNG <-> Netbeans
> plugin. We should verify whether the JVM exit is caused by TestNG itself or our
> plugin. 

well, it's caused by either maven-surefire-plugin or testng or both interacting. We have no word to say there. We (netbeans) just parse the output and the surefire's xml files.

> If you agree, can you redirect this issue there?

No, I cannot redirect issue from our bugzilla to maven's JIRA. It has to be filed as new bug there, maybe mentioning this issue. And it's best if it's filed by you as the original reporter (and not me serving as human proxy in the communication).  I sometimes file issues myself but only in cases that either directly involve our codebase or that are generally important. This is not the case IMHO.
Comment 13 _ gtzabari 2012-12-07 19:31:14 UTC
I posted on the testng mailing list: https://groups.google.com/forum/?fromgroups=#!topic/testng-dev/v5k1EupGdus
Comment 14 Milos Kleint 2012-12-13 17:29:52 UTC
filed at surefire plugin at https://jira.codehaus.org/browse/SUREFIRE-931