This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
Product Version: NetBeans IDE 7.2 (Build 201207171143) Java: 1.7.0_07; Java HotSpot(TM) 64-Bit Server VM 23.3-b01 System: Windows 7 version 6.1 running on amd64; Cp1252; en_CA (nb) User directory: C:\Users\Gili\AppData\Roaming\NetBeans\7.2 Cache directory: C:\Users\Gili\AppData\Local\NetBeans\Cache\7.2 I defined three classes containing TestNG tests. When I hit ALT+F6 the Test Results window prints: "All 24 tests passed. (6.93 s)" however when I open the Output Window I see: ------------------------------------------------------- T E S T S ------------------------------------------------------- Running TestSuite org.apache.maven.surefire.util.SurefireReflectionException: java.lang.reflect.InvocationTargetException; nested exception is java.lang.reflect.InvocationTargetException: null java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110) at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175) at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:107) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68) Caused by: org.testng.TestNGException: DependencyMap::Method "LocationsTest.removeNonExistentLocation()[pri:0, instance:com.foo.location.server.LocationsTest@77c42952]" depends on nonexistent group "postLocation" at org.testng.DependencyMap.getMethodsThatBelongTo(DependencyMap.java:45) at org.testng.TestRunner.createDynamicGraph(TestRunner.java:1079) at org.testng.TestRunner.privateRun(TestRunner.java:734) at org.testng.TestRunner.run(TestRunner.java:617) at org.testng.SuiteRunner.runTest(SuiteRunner.java:334) at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:329) at org.testng.SuiteRunner.privateRun(SuiteRunner.java:291) at org.testng.SuiteRunner.run(SuiteRunner.java:240) at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:51) at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:85) at org.testng.TestNG.runSuitesSequentially(TestNG.java:1197) at org.testng.TestNG.runSuitesLocally(TestNG.java:1122) at org.testng.TestNG.run(TestNG.java:1030) at org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:70) at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeMulti(TestNGDirectoryTestSuite.java:158) at org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:98) at org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:111) ... 9 more Results : Tests run: 0, Failures: 0, Errors: 0, Skipped: 0 Expected behavior: Test Results window should display a failure. I believe you should be able to reproduce this failure by defining this method: @Test(groups = "deleteLocation", dependsOnGroups = { "postLocation", "getLocation" }) public void removeNonExistentLocation() {} where group "postLocation" doesn't exist.
After some evaluation I have these results: created maven javase project. removed junit as test dependency, created a testng test class and defined the reporter's method. 1) CoS OFF: everything works as expected. Tried these scenarios: a) test project => "No tests executed" message in TRW, exception in OW commented (groups = "deleteLocation", dependsOnGroups = {"postLocation", "getLocation"}) b) test project => "The test passed" message in TRW, no exception in OW uncommented (groups = "deleteLocation", dependsOnGroups = {"postLocation", "getLocation"}) c) test project => "No tests executed" message in TRW, exception in OW d) test file => "No tests executed" message in TRW, exception in OW commented (groups = "deleteLocation", dependsOnGroups = {"postLocation", "getLocation"}) e) test file => "The test passed" message in TRW, no exception in OW 2) CoS ON: Test Project does not work as expected. Tried the same scenarios: a) test project => "No tests executed" message in TRW, exception in OW commented (groups = "deleteLocation", dependsOnGroups = {"postLocation", "getLocation"}) b) test project => "The test passed" message in TRW, no exception in OW uncommented (groups = "deleteLocation", dependsOnGroups = {"postLocation", "getLocation"}) c) test project => "The test passed" message in TRW, exception in OW => this is wrong!!! d) test file => "No tests executed" message in TRW, exception in OW commented (groups = "deleteLocation", dependsOnGroups = {"postLocation", "getLocation"}) e) test file => "The test passed" message in TRW, no exception in OW So there should be an issue with CoS with test Project action as shown in case 2c. re-assigning for evaluation. Thank you
I will do some more in depth investigation, but most likely something with surefire itself (internal error) Tests run: 0, Failures: 0, Errors: 0, Skipped: 0 suggests that we have no failures and no errors, so I assume even the surefire xml file doesn't contain the exception as a failure nor error (IMHO should be error)
theofanis: it doesn't matter if CoS is on or off.. You probably failed to notice that in some situations the build fails with "nonexistent group "postLocation"" but the test output window actually shows the failure from previous execution. if all the steps get separated by a mvn clean execution, then the Test project action will always fail the same way. The problem is primarily in surefire itself. in the case of our failure, the target/surefire-reports/TEST-*.xml file is not generated thus we have nothing to populate our test output window from. So either it's a problem in surefire itself when it's badly processing this sort of error. Or it's more or less correct, but we don't handle it.. which is unlikely IMHO because the maven build doesn't list the failed test as failed, and at the end prints BUILD SUCCESS which is wrong as far as I'm concerned. -> it's a surefire testng failure.
2.12.4 of surefire maven plugin already fails instead of finishing as successful.
http://hg.netbeans.org/core-main/rev/053abdf729ee fixes the problem of previous execution display.. the rest is most likely to be filed against surefire I suppose.
Integrated into 'main-golden', will be available in build *201211070001* on http://bits.netbeans.org/dev/nightly/ (upload may still be in progress) Changeset: http://hg.netbeans.org/main-golden/rev/053abdf729ee User: Milos Kleint <mkleint@netbeans.org> Log: #219097 ignore results from previous invokations.
NetBeans IDE Dev (Build 201211060001) The issue doesn't seem to be fixed. Please run the test-case on your end before resolving.
(In reply to comment #7) > NetBeans IDE Dev (Build 201211060001) > > The issue doesn't seem to be fixed. Please run the test-case on your end before > resolving. What exactly doesn't work? I didn't say I fixed it entirely (see comment 5). I just said that we no longer populate the test output window using results from previous execution. That's what I have tested and I believe it worked (please note that I've fixed/processed like 50 bugs since then and don't exactly remember). And I said that the rest is surefire plugin problem and should be filed there (http://jira.codehaus.org/browse/SUREFIRE) questions for you: have you upgraded your project to surefire 2.12.4? when executing, does surefire mention that the affected test failed in maven output? does it generate correct (with failure) xml file in target/surefire-reports/*.xml? closing as fixed (wontfix is also an alternative but we did fix a problem in our codebase as well), the rest is not a bug in netbeans but external tool.
Created attachment 128526 [details] Testcase I apologize. I should have been more specific. I am running Build 201211270002 with the following repro steps: 1. Open the attached testcase (it uses Surefire 2.12.4 which I agree improves the situation). 2. Hit ALT+F6 to run tests 3a. Scenario 1: If the Test Results tab is open 4a. The Test Results tab reads "No tests executed". 3b. Scenario 2: If the Test Results tab is closed 4b. Netbeans pops up a message: "Tests finished successfully for project: X. Open Test Results Window" 5. Click on the Output tab 6. Notice Surefire says "The forked VM terminated without saying properly goodbye. VM crash or System.exit called ?" 7. If you check the "surefire-reports" directory there is no indication of success or failure, because the JVM terminated before writing to disk. So there are 3 separate issues here (2 of which are caused by and should be fixed in Netbeans) 1. I believe the TestNG plugin is causing the JVM to exit prematurely. We should investigate what's going on there and fix it. 2. "No tests executed" is technically correct because right now there are no result files. I believe this will get fixed automatically once we fix TestNG. 3. "Tests finished successfully for project: X" is wrong and should be fixed.
(In reply to comment #9) > > 1. I believe the TestNG plugin is causing the JVM to exit prematurely. We > should investigate what's going on there and fix it. > 2. "No tests executed" is technically correct because right now there are no > result files. I believe this will get fixed automatically once we fix TestNG. > 3. "Tests finished successfully for project: X" is wrong and should be fixed. 1+2 are clearly out of scope for netbeans integration. please file it against surefire with the attached project. 3. is technically correct. the test output window defines success as no failed tests, which the current scenario fulfills. the fact that the build actually fails within surefire execution (or after) is out of scope for the test output window. This will correct itself as well once the surefire bug is fixed.
Milos, I believe #1 is in our scope because we authored the TestNG <-> Netbeans plugin. We should verify whether the JVM exit is caused by TestNG itself or our plugin. If you agree, can you redirect this issue there?
(In reply to comment #11) > Milos, > > I believe #1 is in our scope because we authored the TestNG <-> Netbeans > plugin. We should verify whether the JVM exit is caused by TestNG itself or our > plugin. well, it's caused by either maven-surefire-plugin or testng or both interacting. We have no word to say there. We (netbeans) just parse the output and the surefire's xml files. > If you agree, can you redirect this issue there? No, I cannot redirect issue from our bugzilla to maven's JIRA. It has to be filed as new bug there, maybe mentioning this issue. And it's best if it's filed by you as the original reporter (and not me serving as human proxy in the communication). I sometimes file issues myself but only in cases that either directly involve our codebase or that are generally important. This is not the case IMHO.
I posted on the testng mailing list: https://groups.google.com/forum/?fromgroups=#!topic/testng-dev/v5k1EupGdus
filed at surefire plugin at https://jira.codehaus.org/browse/SUREFIRE-931