This Bugzilla instance is a read-only archive of historic NetBeans bug reports. To report a bug in NetBeans please follow the project's instructions for reporting issues.

Bug 269766 - Possibility to pass test case with issue
Summary: Possibility to pass test case with issue
Status: STARTED
Alias: None
Product: qa
Classification: Unclassified
Component: Synergy (show other bugs)
Version: Dev
Hardware: PC Linux
: P1 normal (vote)
Assignee: Vladimir Riha
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-02-07 09:36 UTC by Jiri Kovalsky
Modified: 2017-02-13 12:16 UTC (History)
0 users

See Also:
Issue Type: ENHANCEMENT
Exception Reporter:

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Jiri Kovalsky 2017-02-07 09:36:13 UTC 
Right now a test case can either pass or fail which requires an issue to be provided. Sometimes a test case basically passes except a minor problem which is already known for some time but was not fixed to date. For situations like this it seems more appropriate to mark such test case(s) as PASSED while still referring the issue number instead of marking strictly as FAILED.

This way the failed test cases would really represent a seriously broken functionality which would lead to an overall NoGo verdict for the tested release.

The overall test run statistic would then look like this:

81% PASSED
14% PASSED with issues
 2% FAILED
 3% SKIPPED
Comment 1 Jiri Kovalsky 2017-02-07 09:38:31 UTC 
Ideally, the 14% and 2% would be links to JIRA listing all issues in question.
Comment 2 Vladimir Riha 2017-02-11 13:41:40 UTC 
Jirko, I'll try to fix the links, but at this moment I'm afraid it is not easily possible to introduce a new test case state (PASSED WITH ISSUES). Would you be OK with having the 3 states as now (passed, failed, skipped) together with option to report issue in passed test case?

Right now what I propose is:
 - user can add issue for cases marked as passed
 - these cases will be reported as passed
 - issues in passed cases will be displayed as usual in statistics but if test assignment has only passed cases (and some of them were added issues during testing), the test cases will be still green and passed in test run page
Comment 3 Jiri Kovalsky 2017-02-13 09:40:01 UTC 
What exactly is the problem with introducing the new TC state? Does it require some heavy refactoring or is it necessity to migrate data or something else? I still find it very important to distinguish between clean passes and "dirty" passes.

If that enhancement is not possible to do, we must reword PASSED to COMPLETED probably. :|
Comment 4 Vladimir Riha 2017-02-13 12:16:12 UTC 
It is possible, but it requires a lot of changes (even in database schema). Renaming to "completed" would imho be more confusing as completed means completed regardless how (I think). Migration is not possible either way as only test cases after such change would be "passed with issues". I'll try to come up with something but I can't promise you when it will be.