The Defect Management Process

4. Triage. Every new report gets a decision. Is it a defect? Is it new or a duplicate? Should it be fixed now, deferred, or does it describe a new requirement, which belongs to a change control board, a "formally chartered group responsible for reviewing, evaluating, approving, delaying, or rejecting changes to a project" (the PMBOK Guide)? This is where the reports that are "something else" leave the defect stream.

5. Resolution. A developer finds the defect behind the failure (Chapter Forty-Seven's debugging) and changes the work product.

6. Verification. A tester runs the confirmation test, the original failing test on the fixed version, and regression tests around the change (Chapter Thirty-Two, on test levels and test types). A fix that fails is reopened.

7. Closure. The report is closed when the fix is verified, or when it is rejected, found to be a duplicate, or turned into a change request, with the reason recorded.

8. Improvement. The defects are studied together: which kinds, from where, found when, and why. The causes found lead to corrective action, "action to eliminate the cause of a nonconformity and to prevent recurrence", and preventive action, "action to eliminate the cause of a potential nonconformity" (ISO/IEC 19770-1). Chapter Eighty, on using defect data to improve the process, takes this stage further.

Florac's generic problem management system draws the middle of the same process as four activities, collection, evaluation, resolution and closure, which his problem statuses (Chapter Seventy-Four, on the defect life cycle) follow: a report is Recognized once collected, Evaluated once its type is known, Resolved once the rules for resolution are met, and then Closed.

Worked example: release 2.0 through the process

Release 2.0 drew 250 incident reports. ExamReg's triage rules decide what happens to each kind, and the program routes all 250 through them, then follows the accepted defects through resolution, verification and closure, and finally picks the target for causal analysis.

from collections import Counter

# release 2.0's 250 incident reports by Florac's problem type and uniqueness (FINDINGS 5.2.3)
reports = {("requirements defect", "original"): 28, ("design defect", "original"): 40,
           ("code defect", "original"): 120, ("operational document defect", "original"): 12,
           ("test case defect", "original"): 9, ("code defect", "duplicate"): 18,
           ("user mistake", "original"): 7, ("operations mistake", "original"): 3,
           ("new requirement/enhancement", "original"): 6,
           ("not repeatable/cause unknown", "value not identified"): 5,
           ("value not identified", "value not identified"): 2}

def triage(problem_type, uniqueness):
    """ExamReg's triage rules: the decision taken on each new report."""
    if uniqueness == "duplicate":
        return "duplicate: linked to the original report, closed"
    if problem_type in ("user mistake", "operations mistake"):
        return "rejected: not a defect, reason recorded"
    if problem_type == "new requirement/enhancement":
        return "change request: sent to the change control board"
    if problem_type == "test case defect":
        return "test case defect: returned to the test team"
    if problem_type in ("not repeatable/cause unknown", "value not identified"):
        return "returned to the reporter for more information"
    return "accepted as a product defect"

decisions = Counter()
for (problem_type, uniqueness), n in reports.items():
    decisions[triage(problem_type, uniqueness)] += n
print(f"TRIAGE of {sum(decisions.values())} reports")
for decision, n in decisions.most_common():
    print(f"   {n:>3}  {decision}")

accepted = decisions["accepted as a product defect"]
deferred = ["DR-311 (major, waiver W-07)", "DR-318 (minor)"]          # FINDINGS 5.2.4
fixed, reopened = accepted - len(deferred), 14
print(f"RESOLUTION of {accepted} product defects: {fixed} fixed, {len(deferred)} deferred to 2.1:"
      f" {', '.join(deferred)}")
print(f"VERIFICATION of {fixed} fixes: {fixed - reopened} passed the first confirmation test,"
      f" {reopened} reopened and fixed again")
print(f"CLOSED by the end of the count: {fixed} product defects; still open, deferred: {len(deferred)}")

# the loop back into the process: the commonest defect type goes to causal analysis (FINDINGS 5.2)
types = {"input validation": 58, "logic and computation": 44, "interface": 30, "user interface": 24,
         "data and database": 18, "documentation": 12, "performance": 8, "security": 6}
top = max(types, key=types.get)
print(f"IMPROVEMENT: {top} ({types[top]} of {sum(types.values())}) is selected for causal analysis")

TRIAGE of 250 reports
   200  accepted as a product defect
    18  duplicate: linked to the original report, closed
    10  rejected: not a defect, reason recorded
     9  test case defect: returned to the test team
     7  returned to the reporter for more information
     6  change request: sent to the change control board
RESOLUTION of 200 product defects: 198 fixed, 2 deferred to 2.1: DR-311 (major, waiver W-07), DR-318 (minor)
VERIFICATION of 198 fixes: 184 passed the first confirmation test, 14 reopened and fixed again
CLOSED by the end of the count: 198 product defects; still open, deferred: 2
IMPROVEMENT: input validation (58 of 200) is selected for causal analysis

Read the output as the process's record.

Triage is where one report in five leaves the defect stream. Of 250 reports, 200 were product defects; the other 50 were something else, and each went somewhere definite. Duplicates were linked to their originals, so that the original's fix closes them and the count of defects is not inflated. Users' and operators' mistakes were rejected with the reason recorded, so the reporter learns why. Test case defects went back to the test team, because the fix is to the test, not the product. Enhancement requests went to the change control board, because a new requirement is a decision about scope, not a repair. And unrepeatable reports went back to their reporters for more information, rather than being closed unexamined.

Resolution fixed 198 defects and deferred 2, the two defects the college accepted at acceptance testing (Chapter Forty-Two): DR-311 under its waiver, and the minor DR-318. Deferred defects stay open, and they appear in the release's report as known defects.

Verification caught 14 fixes that did not work the first time, one in fourteen, and sent them back; all passed at the second attempt. Without that step, 14 defects would have been closed while still present.

Improvement closes the loop: input validation, the commonest defect type at 58 of 200, is selected for causal analysis, the first step of the prevention that starts the next release's process.

Who does what

On ExamReg's project the stages are divided like this; other teams divide them differently, but the verification stays independent of the fix.

What it does not mean

Defect management is not only fixing. Triage, verification and improvement are stages of the process; fixing is one of eight.

Every report does not become a defect. A fifth of release 2.0's reports were duplicates, mistakes, test errors, requests or unrepeatable, and each needed its own decision.

A fix is not the end. The confirmation and regression tests decide whether a defect is really gone.

Improvement is not optional. Without it, the process removes this release's defects and lets the next release's in by the same causes.

Quick revision

• Defect management process (ISTQB): a workflow from discovery to closure and rules for classification; log, analyse and classify, decide a response, close.
• Report objectives (ISTQB): enough information to resolve; tracking product quality; ideas for improving the process.
• Test incident (ISO/IEC/IEEE 29119-3): an event in test execution requiring investigation; the test incident reporting process (29119-2) reports incidents needing action to stakeholders.
• Stages: prevention, discovery, recording, triage, resolution, verification, closure, improvement.
• Triage decisions: accept, reject, duplicate, defer, change request, return for information.
• Causal analysis (ISO/IEC/IEEE 24765); corrective and preventive action (ISO/IEC 19770-1); CMMI: prevention beats detection.
• Worked example: 250 reports; 200 product defects; 198 fixed, 2 deferred; 14 fixes reopened; input validation (58) to causal analysis.

Test yourself

1. What is defect management? List its stages. The process of handling reported problems from discovery to closure, and of using what they show to improve the process. Its stages are prevention, discovery, recording, triage, resolution, verification, closure and improvement.

2. What are the objectives of a defect report? To give those who handle and resolve the defect enough information to resolve it; to provide a means of tracking the quality of the work product; and to provide ideas for improving the development and test process.

3. What happens at triage? Give the possible decisions. Each new report is examined and a decision is taken: accept it as a defect to fix now, defer the fix to a later release, reject it as not a defect with the reason recorded, link it as a duplicate, send it to the change control board as a new requirement, or return it to the reporter for more information.

4. Why is verification a separate stage, and who performs it? Because a fix can fail or break something else; the confirmation test on the fixed version and regression tests around the change show whether the defect is really gone. It is performed by a tester, not by the developer who made the fix.

5. How does defect management prevent defects, not only remove them? By analysing the defects of a release together to find their causes, and taking corrective and preventive actions, such as new checklist items, review focus, coding standards or training, so that the same kinds of defect are not introduced again.

6. In the worked example, what became of the 250 reports? 200 were accepted as product defects, of which 198 were fixed (14 after a failed first fix) and 2 deferred to the next release; 18 were duplicates, 10 were rejected as users' or operators' mistakes, 9 were test case defects returned to the test team, 7 went back to their reporters for more information, and 6 became change requests.