Bug 137069 - Convenient way to mark unit tests unreliable
Convenient way to mark unit tests unreliable
Status: RESOLVED FIXED
Product: platform
Classification: Unclassified
Component: NB JUnit
6.x
All All
: P2 (vote)
: 6.x
Assigned To: Jesse Glick
issues@platform
: API, API_REVIEW_FAST
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2008-06-12 05:32 UTC by Jesse Glick
Modified: 2009-09-29 22:19 UTC (History)
1 user (show)

See Also:
Issue Type: ENHANCEMENT
:


Attachments
Patch: new annotation, filtering based on it, test for that, sample usage (14.25 KB, patch)
2008-06-12 05:36 UTC, Jesse Glick
Details | Diff
Revised patch which ensures that Dummy really passes (APIJail is to prevent it from being public to javac) (14.68 KB, patch)
2008-06-12 16:03 UTC, Jesse Glick
Details | Diff
Updated patch with new name @RandomlyFails (14.83 KB, patch)
2008-06-17 17:33 UTC, Jesse Glick
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Jesse Glick 2008-06-12 05:32:58 UTC
Currently we are using the idiom of checking for ${ignore.random.failures} in various unit tests (mostly in the
platform) and, if so, skipping over them. This lets us restrict the set of tests to run e.g. on a team builder to those
which are thought to be reproducible, while skipping timing-dependent tests etc.

However this idiom is pretty clumsy. Would be nicer to be able to use an annotation to indicate tests which are known to
not be reliable.
Comment 1 Jesse Glick 2008-06-12 05:36:24 UTC
Created attachment 62717 [details]
Patch: new annotation, filtering based on it, test for that, sample usage
Comment 2 Jesse Glick 2008-06-12 05:39:08 UTC
Please review the attached patch and API change. Before actually committing I would of course increment the nbjunit spec
version; add @since and apichanges; and use the new annotation in the remaining modules that need it:
autoupdate.services, openide.loaders, etc.
Comment 3 Vitezslav Stejskal 2008-06-12 06:42:04 UTC
Nice, just on question. If the whole test class is marked @Random and NbTestSuite.Dummy class is used instead will not
the junit framework fail the suite complaining that there are no test cases?
Comment 4 Jesse Glick 2008-06-12 16:01:19 UTC
Good point. I'm not sure it actually would have made module tests fail, but better to fix it. See new patch.
Comment 5 Jesse Glick 2008-06-12 16:03:05 UTC
Created attachment 62759 [details]
Revised patch which ensures that Dummy really passes (APIJail is to prevent it from being public to javac)
Comment 6 Jaroslav Tulach 2008-06-12 18:22:22 UTC
Idea is good, the name is somehow strange. What Random means? Random number? Randomized test? Etc. I would use 
something meaningful like @RandomlyFails, etc.
Comment 7 Jesse Glick 2008-06-12 19:27:22 UTC
@RandomlyFails would work.
Comment 8 Jesse Glick 2008-06-17 17:33:45 UTC
Created attachment 62946 [details]
Updated patch with new name @RandomlyFails
Comment 9 Jesse Glick 2008-06-19 18:53:01 UTC
core-main #a396be574ea5
Comment 10 Quality Engineering 2008-06-20 15:50:24 UTC
Integrated into 'main-golden', available in NB_Trunk_Production #271 build
Changeset: http://hg.netbeans.org/main/rev/a396be574ea5
User: Jesse Glick <jglick@netbeans.org>
Log: #137069: adding @org.netbeans.nbjunit.Random to mark randomly failing tests.


By use of this website, you agree to the NetBeans Policies and Terms of Use. © 2012, Oracle Corporation and/or its affiliates. Sponsored by Oracle logo