This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.
| Summary: | Convenient way to mark unit tests unreliable | ||
|---|---|---|---|
| Product: | platform | Reporter: | Jesse Glick <jglick> |
| Component: | NB JUnit | Assignee: | Jesse Glick <jglick> |
| Status: | RESOLVED FIXED | ||
| Severity: | blocker | CC: | apireviews |
| Priority: | P2 | Keywords: | API, API_REVIEW_FAST |
| Version: | 6.x | ||
| Hardware: | All | ||
| OS: | All | ||
| Issue Type: | ENHANCEMENT | Exception Reporter: | |
| Attachments: |
Patch: new annotation, filtering based on it, test for that, sample usage
Revised patch which ensures that Dummy really passes (APIJail is to prevent it from being public to javac) Updated patch with new name @RandomlyFails |
||
Created attachment 62717 [details]
Patch: new annotation, filtering based on it, test for that, sample usage
Please review the attached patch and API change. Before actually committing I would of course increment the nbjunit spec version; add @since and apichanges; and use the new annotation in the remaining modules that need it: autoupdate.services, openide.loaders, etc. Nice, just on question. If the whole test class is marked @Random and NbTestSuite.Dummy class is used instead will not the junit framework fail the suite complaining that there are no test cases? Good point. I'm not sure it actually would have made module tests fail, but better to fix it. See new patch. Created attachment 62759 [details]
Revised patch which ensures that Dummy really passes (APIJail is to prevent it from being public to javac)
Idea is good, the name is somehow strange. What Random means? Random number? Randomized test? Etc. I would use something meaningful like @RandomlyFails, etc. @RandomlyFails would work. Created attachment 62946 [details]
Updated patch with new name @RandomlyFails
core-main #a396be574ea5 Integrated into 'main-golden', available in NB_Trunk_Production #271 build Changeset: http://hg.netbeans.org/main/rev/a396be574ea5 User: Jesse Glick <jglick@netbeans.org> Log: #137069: adding @org.netbeans.nbjunit.Random to mark randomly failing tests. |
Currently we are using the idiom of checking for ${ignore.random.failures} in various unit tests (mostly in the platform) and, if so, skipping over them. This lets us restrict the set of tests to run e.g. on a team builder to those which are thought to be reproducible, while skipping timing-dependent tests etc. However this idiom is pretty clumsy. Would be nicer to be able to use an annotation to indicate tests which are known to not be reliable.