Using Defect Data to Improve the Process

The first step is the one most often skipped. Release 2.0's defects offer many patterns: the fee and exam form modules' high density (Chapter Seventy-Nine, on defect metrics), the code review's low effectiveness, the documents' long survival. Input validation is chosen here because it is the largest defect type, 58 of 200, so one successful action on it removes more defects than an action on any other type; Chapter One Hundred Four, on Pareto diagrams, sets out the same reasoning for all the types at once.

Root causes and the five whys

A root cause is not the first cause found. The exam form accepted a negative number of backlog papers is a symptom; the developer forgot the check is a cause, but not one a process can change: people forget. The root cause is the reason the process let the forgetting through. ASQ describes the five whys as "a questioning process designed to drill down into the details of a problem or a solution and peel away the layers of symptoms", and notes that it "may take less or more than five times to reach the root cause". On release 2.0's input validation defects:

1. Why were 58 defects input validation? Forms accepted values the rules forbid: negative days, blank names, fractional backlog counts.
2. Why did the forms accept them? Each form's developer wrote that form's checks, and several missed some.
3. Why did each developer write their own? ExamReg had no shared validation for its common field types.
4. Why was there none? The design left validation to each screen; no one owned it.
5. Why did the design leave it? The requirements listed the valid values but never said how invalid ones must be handled; the ruling on negative days (Chapter Eight) arrived late, after the code.

The last answer is a cause the process can change, and so is the fourth. The team's actions follow from them: a shared validation module for ExamReg's field types; a requirements review checklist item, does every input say what happens to an invalid value? (Chapter Sixty-Four, on checklist-based testing); and unit test templates built from equivalence partitions and boundaries (Chapters Fifty-Four and Fifty-Five, on equivalence partitioning and boundary value analysis) for every input.

Orthogonal defect classification: reading the process from the defects

Chillarege and his colleagues at IBM proposed a way to get process feedback from defects without a separate study for each question. ODC classifies each defect by the kind of fix it needed (the types listed in Chapter Two, on errors, faults and failures), and chooses the types so that each "can be associated with a few specific phases in the process". Then the mix of types found at each stage is a signature: "the defect type distribution changes with time, and the distribution provides an indication of where the development is, logically." Function defects, for instance, should be found early; "the bar corresponding to function defects should be diminishing through the process."

The payoff is a diagnosis. "If at system test the profile of the distribution looks like it should be in unit test or integration test, then the distribution indicates that the product is prematurely in system test." And "When a departure in the process is identified by a deviation in the distribution curve, the offending defect type also points to the part of the process that is probably responsible for this departure." Release 2.0's input validation defects, most of them checking defects in ODC's terms, pointing at the design and coding of screens, are exactly the kind of signal ODC formalises.

Worked example: did the action work?

Release 2.1, the next release, was built with the three actions in place. It added or changed 8 KLOC and had 69 defects. Because the two releases differ in size, the program compares defects per KLOC by type; and because many things change between releases, it compares the targeted type with all the others, which the actions did not aim at and which therefore act as a control.

# defects by type: release 2.0 (FINDINGS 5.2, 20 KLOC) and release 2.1 (FINDINGS 5.3, 8 KLOC)
r20 = {"input validation": 58, "logic and computation": 44, "interface": 30, "user interface": 24,
       "data and database": 18, "documentation": 12, "performance": 8, "security": 6}
r21 = {"input validation": 9, "logic and computation": 20, "interface": 13, "user interface": 10,
       "data and database": 7, "documentation": 5, "performance": 3, "security": 2}
KLOC_20, KLOC_21 = 20, 8
TARGET = "input validation"                   # the type causal analysis acted on

print(f"{'defect type':<22}{'2.0 per KLOC':>13}{'2.1 per KLOC':>13}{'change':>8}")
for t in r20:
    before, after = r20[t] / KLOC_20, r21[t] / KLOC_21
    mark = "   <- the action's target" if t == TARGET else ""
    print(f"{t:<22}{before:>13.2f}{after:>13.2f}{100 * (after - before) / before:>7.0f}%{mark}")

def rate(release, kloc, types):
    return sum(release[t] for t in types) / kloc

others = [t for t in r20 if t != TARGET]
print(f"target type:  {rate(r20, KLOC_20, [TARGET]):.2f} -> {rate(r21, KLOC_21, [TARGET]):.2f} per KLOC")
print(f"all others:   {rate(r20, KLOC_20, others):.2f} -> {rate(r21, KLOC_21, others):.2f} per KLOC")
print(f"share of all defects: {100 * r20[TARGET] / sum(r20.values()):.0f}% -> "
      f"{100 * r21[TARGET] / sum(r21.values()):.0f}%")

defect type            2.0 per KLOC 2.1 per KLOC  change
input validation               2.90         1.12    -61%   <- the action's target
logic and computation          2.20         2.50     14%
interface                      1.50         1.62      8%
user interface                 1.20         1.25      4%
data and database              0.90         0.88     -3%
documentation                  0.60         0.62      4%
performance                    0.40         0.38     -6%
security                       0.30         0.25    -17%
target type:  2.90 -> 1.12 per KLOC
all others:   7.10 -> 7.50 per KLOC
share of all defects: 29% -> 13%

Input validation defects fell from 2.90 to 1.12 per KLOC, a drop of 61 per cent, and from 29 to 13 per cent of all defects. The other types together did not fall: 7.10 per KLOC before, 7.50 after. That contrast is the evidence. Had every type fallen by a similar amount, the drop in input validation could have come from anything that changed between the releases (a smaller, simpler release, a more careful team); because only the targeted type fell, the actions are the likeliest explanation.

The practice CMMI calls "Evaluate the Effect of Implemented Actions" asks for exactly this, and the evaluation is also a caution. One release is one measurement, and 9 defects is a small number; the team keeps watching the rate over the next releases before calling the problem solved, and the rise in logic defects, 14 per cent, is itself a candidate for the next causal analysis.

What it does not mean

Improvement is not fixing more defects. It is changing the process so that fewer of them are made or more are caught early.

A root cause is not a person. The developer forgot is where the five whys start, not where they stop; the answer must be something the process can change.

One release is not proof. An effect is confirmed over several releases, and against the defect types that were not targeted.

ODC does not replace judgement. A deviation in the type distribution points at a part of the process; people still have to find out what went wrong there.

Quick revision

• CMMI Causal Analysis and Resolution: "to identify causes of selected outcomes and take action to improve process performance"; select outcomes, analyse causes, implement actions, evaluate the effect, record the data.
• Root cause (ISO/IEC/IEEE 24765): a source whose removal decreases or removes the defect; causal analysis: analysis of a defect to determine its cause.
• Five whys (ASQ): ask why repeatedly, from the symptom to a cause the process can change; "It may take less or more than five times".
• ODC (Chillarege 1992): defect types associated with process phases; the type distribution by stage is a process signature; a deviation points at the responsible part of the process.
• Worked example: input validation 58 of 200 in release 2.0; three actions; release 2.1: 2.90 to 1.12 per KLOC (a 61 per cent drop) while other types rose from 7.10 to 7.50; share 29 to 13 per cent.

Test yourself

1. What is causal analysis and resolution? List its practices. A process, in CMMI, for identifying the causes of selected outcomes, such as a class of defects, and acting on them to improve the process. Its practices are selecting outcomes for analysis, analysing their causes, implementing action proposals, evaluating the effect of the actions, and recording the causal analysis data.

2. Apply the five whys to a class of defects. For input validation defects: forms accepted invalid values; because each developer wrote their own checks and some missed them; because there was no shared validation; because the design left validation to each screen; because the requirements never said how invalid values must be handled. The last two are root causes the process can change: shared validation, and a requirements review item on invalid input.

3. Why is "the developer made a mistake" not a root cause? Because people will always make mistakes; a cause is useful only if changing it prevents the defect, so the analysis continues until it reaches something in the process, such as missing requirements, a missing shared component or a missing review item, that can be changed.

4. How does orthogonal defect classification give feedback on the process? It classifies each defect by the kind of fix it needed, with types associated with particular phases of development. The distribution of types found at each stage is a signature of the process; if it departs from the expected pattern, for example function defects still common at system test, the product may be prematurely in that stage, and the defect type points at the part of the process responsible.

5. How should the effect of a process change be evaluated? By measuring the targeted class of defects in the following release, normalised by size, and comparing it with the classes that were not targeted, which act as a control; a drop only in the targeted class supports the change as the cause. The evaluation continues over several releases.

6. In the worked example, why was the unchanged rate of the other defect types important? Because it rules out explanations that would have lowered every type, such as a simpler release or a more careful team; since only input validation defects fell, the three actions aimed at them are the likeliest cause of the fall.