Tracking Defects to Closure

Worked example: release 2.0's ten test weeks

The program tracks release 2.0's 114 test-phase defects over its ten test weeks, then takes the list of defects open at the end of week 6, the midpoint of testing, and analyses it as the review board did that week.

from collections import Counter
from itertools import accumulate

# release 2.0's ten test weeks (FINDINGS 5.2.5): defects reported and closed each week
new    = [10, 16, 18, 17, 14, 12, 10, 8, 5, 4]
closed = [4, 10, 14, 16, 16, 15, 13, 11, 8, 5]
open_at_end = [a - c for a, c in zip(accumulate(new), accumulate(closed))]

print("week  new  closed  open")
for week, (n, c, o) in enumerate(zip(new, closed, open_at_end), 1):
    print(f"{week:>4} {n:>4} {c:>7} {o:>5}  {'#' * o}")
turn = next(w for w, (n, c) in enumerate(zip(new, closed), 1) if c > n)
print(f"closures first exceed new reports in week {turn}; open at the end: {open_at_end[-1]}")

# the defects open at the end of week 6: (id, severity, requirement area, days open)
week6 = [("DR-221", "major", "FEE", 3), ("DR-224", "minor", "FORM", 5),
         ("DR-226", "cosmetic", "HALLTICKET", 5), ("DR-218", "minor", "FEE", 8),
         ("DR-213", "major", "FORM", 9), ("DR-209", "minor", "LOGIN", 12),
         ("DR-207", "cosmetic", "FORM", 13), ("DR-204", "minor", "FEE", 15),
         ("DR-199", "major", "FEE", 20), ("DR-195", "minor", "FORM", 26),
         ("DR-190", "cosmetic", "HALLTICKET", 31), ("DR-183", "minor", "FEE", 40)]
bands = [("0 to 7 days", 0, 7), ("8 to 14 days", 8, 14), ("15 to 30 days", 15, 30), ("over 30 days", 31, 999)]
print(f"week 6: {len(week6)} open, by age:")
for name, low, high in bands:
    ids = [d for d, s, r, age in week6 if low <= age <= high]
    print(f"   {name:<14} {len(ids):>2}  {' '.join(ids)}")
print("   by requirement area:", ", ".join(f"{r} {n}" for r, n in Counter(r for _, _, r, _ in week6).most_common()))
old_major = [d for d, s, r, age in week6 if age > 14 and s in ("critical", "major")]
print("   for the review board, major and open over 14 days:", " ".join(old_major))

week  new  closed  open
   1   10       4     6  ######
   2   16      10    12  ############
   3   18      14    16  ################
   4   17      16    17  #################
   5   14      16    15  ###############
   6   12      15    12  ############
   7   10      13     9  #########
   8    8      11     6  ######
   9    5       8     3  ###
  10    4       5     2  ##
closures first exceed new reports in week 5; open at the end: 2
week 6: 12 open, by age:
   0 to 7 days     3  DR-221 DR-224 DR-226
   8 to 14 days    4  DR-218 DR-213 DR-209 DR-207
   15 to 30 days   3  DR-204 DR-199 DR-195
   over 30 days    2  DR-190 DR-183
   by requirement area: FEE 5, FORM 4, HALLTICKET 2, LOGIN 1
   for the review board, major and open over 14 days: DR-199

The trend. The open count climbs for four weeks, to 17, while testers find defects faster than they are fixed, and the bars show the shape at a glance. In week 5 closures (16) overtake new reports (14) for the first time, and from then on the open count falls every week. By week 10 new reports are down to 4 a week and only 2 defects remain open: DR-311 and DR-318, the two the college accepted as known defects at acceptance testing. A falling arrival rate is only good news if testing has not slowed down; the review board checks it against the test progress figures (Chapter Ten, on test reporting) before reading it as stability.

The snapshot. At the end of week 6, twelve defects were open. Seven were under two weeks old, which is the normal flow. Five had been open more than 14 days, and two more than a month; the review board's rule is to ask about every defect over 14 days old, and to act at once on any that is major or critical. That picks out DR-199, a major defect in the fee area, 20 days old. The others old enough to ask about are minor or cosmetic, which may be why they waited, and the board decides for each whether to fix it now or defer it deliberately rather than by neglect.

The requirement view. Five of the twelve belonged to the fee requirements and four to the exam form. Traced to requirements, open defects say which parts of the product are not yet ready, which the ISTQB syllabus names as one use of traceability: it "can be used to evaluate the level of residual risk in a test object". The fee area was the one to watch.

What a tracking report contains

A weekly tracking report for a defect review board answers five questions, each with a number:

1. How many new defects, and how many closed, this period, and the trend over recent periods?
2. How many are open, by severity and by status (new, assigned, fixed and awaiting verification, reopened, deferred)?
3. How old are the open ones, and which exceed the agreed age for their severity?
4. Which requirements, modules or areas have open defects?
5. What was decided about each defect that needed a decision?

What it does not mean

Tracking is not reporting once. It is a continuing record, read as a trend; one week's numbers say little.

A falling arrival rate does not prove quality. It may mean the product is stabilising, or that testing has slowed; it is read beside the test progress figures.

An old defect is not always a neglected one. It may be deferred by decision; the point of ageing is to make every such decision explicit.

Zero open defects is not the goal. Known, accepted defects with agreed workarounds, like DR-311 and DR-318, may be released; unknown ones are the danger.

Quick revision

• Florac: status dates give "status, problem age, and problem arrival rate", and are "of primary importance in product readiness models".
• Measures: arrival rate, closure rate, open count and its trend, age of open defects.
• Defect review board: triages new reports and reviews the open list, oldest and most severe first.
• Traceability (ISTQB): between test basis, testware, results and defects; open defects by requirement show residual risk.
• Worked example: open count peaked at 17 in week 4; closures overtook arrivals in week 5; 2 open at the end (DR-311, DR-318); week 6: 12 open, 5 over 14 days, one major (DR-199) escalated; fee area 5, form 4.

Test yourself

1. What does defect tracking involve? Keeping a continuing record of every defect's status from report to closure, with the date of each change, and reviewing the defects together over time: how many arrive and close each period, how many are open, how old they are, and which requirements they affect, so that neglected defects and trends are seen and acted on.

2. What are the arrival rate and the closure rate, and what does their crossing mean? The arrival rate is the number of new defects reported per period; the closure rate the number closed per period. When closures first exceed arrivals, the open count starts to fall, a sign that the product is stabilising, provided testing effort has not dropped.

3. Why track the age of open defects? Because a defect can sit open without anyone deciding anything about it. Age shows which defects have waited longest; with a rule such as reviewing every defect open more than two weeks, each old defect is either fixed or deferred by an explicit decision.

4. What is a defect review board, and what does it do? A group, typically the test lead, development lead and product owner, that meets regularly during testing to triage every new defect report and to review the open defects, deciding for each whether to fix, defer, reject or escalate it.

5. How does traceability help in tracking defects? By linking each defect to the requirement and test case it concerns, it shows which requirements still have open defects, and so which parts of the product are not ready and where residual risk remains; and it tells the tester which test confirms a fix.

6. In the worked example, which defect did the board escalate at week 6, and why? DR-199, because it was the only major defect that had been open more than 14 days; the other old defects were minor or cosmetic and were considered for fixing or deliberate deferral.