This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
No notification is sent when unit tests in the daily test runs are not compilable. The failures log is available as an artifact of the build of the binary test distribution, but it is not enough. IMHO if the fact that unit tests do not compile is a fail in itself and should be reported as such.
I agree that it's serious problem. Build masters are now responsible for compilation of tests. The tests are compiled together with daily build. The build of tests passed because errors during tests compilation are ignored. It can be disabled but the daily build wil fail. Testing infrastructure doesn't know anything about compilation problems. This issue can be solved in two ways: 1) To add compilation of binary test distribution to continuous builds and enable failing of builds on compilation errors. I was talking about it few months ago with Jesse. 2) To add statistics with number of tested modules, suites and testcases to test evaluation infrastructure.
This is unbearable. There are at least two modules in the Java EE support whose tests do not compile, and there was no notification for this. Tests become pretty much useless if one cannot be sure that they are run daily, and that a notification is sent if they cannot be run. Petre, please reassign to the build masters if they can fix this, or to whoever else who can.
I think that Michal is the correct person. Building of Binary test distribution tooks just only 10 min. I am at least for to add building of binary test distribution to daily builds on deadlock. If building fails P1 will be reported.
+1 on creating a project on deadlock to attempt compilation of all unit tests. I could try to set it up if someone tells me (1) what target to use to invoke compilation on them all, (2) how to find the list of CVS modules that need to be checked out.
How to build test distribution is described at http://wiki.netbeans.info/wiki/view/TestDistribution It builds tests for fixed modules list: <property name="config.modules.test" value="${config.test.extra},${config.fixedmodules.stable},${config.modules.stable}"/> It should be rewritten to new build configurations (standard, basic, etc...). What do you think?
BTW wiki.netbeans.org is now the preferred hostname.
I am working on setting up such a project on deadlock: http://deadlock.netbeans.org/hudson/job/test-compilation/
Cool! Compilation fails in 50 modules. I will attach list of modules. If no one is against it I will report P1 bugs next week.
Created attachment 48412 [details] Grep of build script
Sure, please go ahead and file bugs for module owners. I am not sure if P1 or P2 is appropriate but I will leave that to you. I already filed the first one I came across, issue #115048. The current script (in nbbuild/hudson/test-compilation) does not use -Dtest.fail.on.error=false, because then the project would falsely succeed. But stopping on the first failure, when there are so many, is not great either. It would be perfect if all module builds would be run, and the Ant build always succeeded, but all failed module builds would be collected in a JUnit-format report, so we could use Hudson's test failure tracking mechanism. I think the key is making a replacement for <subant> which tolerates but reports failures. I will play with it.
I fixed compilation in core and apisupport/refactoring.
Now collecting problems in JUnit format for easy browsing in Hudson: Checking in hudson/test-compilation; /shared/data/ccvs/repository/nbbuild/hudson/test-compilation,v <-- test-compilation new revision: 1.2; previous revision: 1.1 done Checking in build.xml; /shared/data/ccvs/repository/nbbuild/build.xml,v <-- build.xml new revision: 1.834; previous revision: 1.833 done RCS file: /shared/data/ccvs/repository/nbbuild/antsrc/org/netbeans/nbbuild/SubAntJUnitReport.java,v done Checking in antsrc/org/netbeans/nbbuild/SubAntJUnitReport.java; /shared/data/ccvs/repository/nbbuild/antsrc/org/netbeans/nbbuild/SubAntJUnitReport.java,v <-- SubAntJUnitReport.java initial revision: 1.1 done
Were you going to file bugs for module owners about failures? If so, please make them block this issue.
All tests now compilable. Turned on notification to broken_builds@netbeans.org so people will be pestered if necessary.