Statistical Process Control: Control Charts for Software

The individuals chart

Software data often arrive one value at a time: one response time per request, one repair time per defect, one defect count per release. For such data the handbook's individuals chart estimates the process's variation from the moving range, "the absolute value of the first difference (e.g., the difference between two consecutive data points) of the data." With x-bar the mean of the values and MR-bar the mean of the moving ranges:

• centre line = x-bar
• upper control limit (UCL) = x-bar + 3 × MR-bar / 1.128
• lower control limit (LCL) = x-bar - 3 × MR-bar / 1.128

The constant 1.128 is, as the handbook notes, the value of d2 for n = 2: the factor that turns the average range of pairs into an estimate of the standard deviation. The moving ranges can be charted too; their upper limit is 3.267 × MR-bar, the handbook's factor D4 for ranges of two values, and their lower limit is 0.

Reading the chart. The first signal is a point beyond the limits. The handbook adds the Western Electric rules, patterns inside the limits that are about as unlikely as a point outside them in a stable process: 2 of the last 3 points beyond 2 sigma on the same side; 4 of the last 5 beyond 1 sigma on the same side; 8 consecutive points on one side of the centre line; 6 in a row rising or falling. ASQ's page lists the same kinds of signal. The rules have a price: with the single 3-sigma rule a stable process gives a false alarm about "every 371 points on the average", and adding the rules raises that to "about once in every 91.75 points".

Limits need a first phase. Limits computed from data that contain a special cause are distorted by it. ASQ describes the practice: limits from the first points are conditional, and the chart is recomputed once points from a period in control are available. When a point is outside, its cause is investigated; if a cause is found and removed, the point is dropped and the limits recomputed.

Worked example 1: the fee page's forty requests

The program computes the individuals chart for the forty fee page requests of Chapter Forty-Five, on load testing, in the order they were sent; drops the points outside the limits whose causes have been found; recomputes; and applies the Western Electric rules to the result.

from statistics import mean

# Chapter 45's forty requests for ExamReg's fee page, in order: (elapsed ms, HTTP response code)
runs = [(380, 200), (410, 200), (417, 200), (463, 200), (454, 200), (516, 200), (491, 200),
        (569, 200), (528, 200), (622, 200), (565, 200), (675, 200), (602, 200), (728, 200),
        (639, 200), (431, 200), (676, 200), (484, 200), (413, 200), (537, 200), (450, 200),
        (590, 200), (487, 200), (643, 200), (524, 200), (696, 200), (561, 200), (749, 200),
        (598, 200), (452, 200), (635, 200), (505, 200), (672, 200), (558, 200), (409, 200),
        (120, 503), (446, 200), (664, 200), (95, 503), (717, 200)]

def xmr_limits(x):
    """Individuals chart: x-bar +/- 3 MR-bar / 1.128; moving range upper limit 3.267 MR-bar."""
    mr = [abs(b - a) for a, b in zip(x, x[1:])]
    centre, mr_bar = mean(x), mean(mr)
    sigma = mr_bar / 1.128
    return centre, centre - 3 * sigma, centre + 3 * sigma, sigma, 3.267 * mr_bar

times = [ms for ms, _ in runs]
centre, lcl, ucl, sigma, mr_ucl = xmr_limits(times)
print(f"all 40: centre {centre:.1f}, limits {lcl:.1f} to {ucl:.1f} ms")
outside = [i for i, ms in enumerate(times, 1) if not lcl <= ms <= ucl]
print("   outside the limits:", ", ".join(f"request {i} ({times[i - 1]} ms)" for i in outside))

served = [ms for i, ms in enumerate(times, 1) if i not in outside]   # causes found: 503 refusals
centre, lcl, ucl, sigma, mr_ucl = xmr_limits(served)
print(f"without them: centre {centre:.1f}, limits {lcl:.1f} to {ucl:.1f} ms,"
      f" moving range limit {mr_ucl:.1f} ms")

def signals(x, centre, sigma):
    """The Western Electric rules in the NIST/SEMATECH e-Handbook: the point each first fires at."""
    z = [(v - centre) / sigma for v in x]

    def same_side(window, beyond, needed):
        return any(sum(s * v > beyond for v in window) >= needed for s in (1, -1))
    found = {}
    for i in range(len(z)):
        tests = {"a point beyond 3 sigma": abs(z[i]) > 3,
                 "2 of 3 beyond 2 sigma, same side": i >= 2 and same_side(z[i - 2:i + 1], 2, 2),
                 "4 of 5 beyond 1 sigma, same side": i >= 4 and same_side(z[i - 4:i + 1], 1, 4),
                 "8 in a row on one side": i >= 7 and same_side(z[i - 7:i + 1], 0, 8),
                 "6 in a row rising or falling": i >= 5 and same_side(
                     [b - a for a, b in zip(x[i - 5:i], x[i - 4:i + 1])], 0, 5)}
        for rule, fired in tests.items():
            if fired and rule not in found:
                found[rule] = i + 1
    return found

fired = signals(served, centre, sigma)
print("rules that fire on the 38 served pages:", fired or "none")
moving = [abs(b - a) for a, b in zip(served, served[1:])]
print("moving ranges over their limit:", sum(m > mr_ucl for m in moving))

Phase one: two special causes. With all forty requests, the limits run from 130.3 to 928.3 ms, and two points fall below the lower limit: requests 36 and 39, the two 503 responses. The investigation that Chapter Eighty-One, on quality concepts, called for found their cause: a database backup job ran during the test and twice made the database refuse connections for a moment. The job was moved to a quiet hour, and the two points are removed.

Phase two: a stable process. Without them, the chart centres on 551.5 ms with limits from 254.2 to 848.7 ms. Every served page lies inside, no moving range exceeds its limit of 365.1 ms, and none of the Western Electric rules fires. By the control chart, the thirty-eight served times are common-cause variation: the answer to Chapter Eighty-One's open question. Chapter One Hundred Seven, on run charts, puts the same data to a different, more sensitive test.

Figure 91.1 The fee page's individuals chart: the served pages are in control; the two 503 responses are special causes

What the chart promises. A stable process is predictable: as long as nothing changes, the next page will very probably be served in 254 to 849 ms. Whether that is good enough is a different question, answered by the specification; the chart only says that the variation is the process's own. To make every page faster, the process itself has to change.

Worked example 2: repair times, a process measure

Control charts serve software processes as well as products. Chapter Sixty-Eight, on quality, process and test metrics, recorded how long each of release 2.0's twelve after-release defects took from report to a fix in production. The program charts them. A time cannot be negative, so a lower limit below zero is shown as zero.

from statistics import mean

# hours from report to a fix in production for release 2.0's 12 after-release defects (FINDINGS 5.2.2)
hours = [4, 6, 3, 30, 8, 5, 72, 10, 6, 4, 12, 20]

def limits(x):
    sigma = mean(abs(b - a) for a, b in zip(x, x[1:])) / 1.128
    centre = mean(x)
    return centre, max(0.0, centre - 3 * sigma), centre + 3 * sigma    # no time is below zero

for label, x in [("all 12 fixes", hours), ("without the 7th", hours[:6] + hours[7:])]:
    centre, lcl, ucl = limits(x)
    out = [h for h in x if not lcl <= h <= ucl]
    print(f"{label}: centre {centre:.1f} h, limits {lcl:.1f} to {ucl:.1f} h, outside: {out or 'none'}")

all 12 fixes: centre 15.0 h, limits 0.0 to 65.3 h, outside: [72]
without the 7th: centre 9.8 h, limits 0.0 to 32.2 h, outside: none

The seventh fix, 72 hours, is above the upper limit of 65.3 hours. Its cause was outside the team's process: the fix waited for the payment gateway's provider to change its side. With it removed, the remaining eleven fixes centre on 9.8 hours with an upper limit of 32.2, and all lie inside; the 30-hour fix, though long, is within the process's own variation. Twelve points make only conditional limits, as ASQ warns, but the chart already separates the one repair that needs a different response (an agreement with the provider on response times) from the ones that need none.

Using control charts in a software project

Three cautions apply to software. The data must come from one process: charting a fee page and a hall ticket page together mixes two processes and hides both. Many software measures change every release, and a chart spanning a change should be restarted after it, which is exactly what makes a shift visible (Chapter Eighty, on using defect data, measured one). And a signal is a reason to investigate, not a verdict: a rule fires by chance about once in 92 points when all the rules are used.

What it does not mean

In control does not mean good. It means predictable; the fee page is in control between 254 and 849 ms, and whether that meets its requirement is a separate question.

A point outside the limits is not automatically an error. It is a signal to look for a special cause; only when one is found is the point removed.

Control limits are not targets. They describe what the process does, computed from its data; nobody sets them.

More rules are not always better. Each rule adds sensitivity and false alarms; the handbook reports false alarms rising from about 1 in 371 points to about 1 in 92.

Quick revision

• SPC (ISO/IEC/IEEE 24765): statistical analysis of a process to "identify common and special causes of variation" and "maintain process performance within limits".
• Control chart (ASQ): time-ordered data; centre line at the average; upper and lower control limits from historical data.
• Individuals chart (NIST/SEMATECH): moving range = difference between consecutive values; limits x-bar plus or minus 3 × MR-bar / 1.128 (d2 for n = 2); moving range limit 3.267 × MR-bar (D4).
• Signals: a point beyond 3 sigma; 2 of 3 beyond 2 sigma; 4 of 5 beyond 1 sigma; 8 in a row on one side; 6 in a row rising or falling. False alarms: about 1 in 371 points with the first rule alone, about 1 in 92 with all.
• Control against specification limits: the first come from the process, the second from the requirements.
• Worked examples: fee page phase one limits 130.3 to 928.3 ms, requests 36 and 39 (the 503s) below; phase two 254.2 to 848.7 ms around 551.5, no rule fires; repair times: the 72-hour fix beyond 65.3 h, the rest within 32.2 h.

Test yourself

1. What is statistical process control, and what does a control chart show? The use of statistics to analyse a process and its measures of performance, to identify common and special causes of variation and keep performance within limits. A control chart plots the process's data in time order with a centre line and upper and lower control limits computed from the data, showing whether the variation is stable or affected by special causes.

2. How are the limits of an individuals chart computed? From the values' mean, x-bar, and the mean of the moving ranges, MR-bar, the differences between consecutive values: the centre line is x-bar and the limits are x-bar plus or minus 3 × MR-bar / 1.128, where 1.128 is d2 for ranges of two values. The moving range chart's upper limit is 3.267 × MR-bar.

3. State the rules that signal a special cause. A point beyond the 3-sigma limits; 2 of the last 3 points beyond 2 sigma on the same side; 4 of the last 5 beyond 1 sigma on the same side; 8 consecutive points on one side of the centre line; 6 points in a row steadily rising or falling.

4. Distinguish control limits from specification limits. Control limits are computed from the process's own data and show whether it is in statistical control; specification limits come from the requirements and show whether the product will do its job. A process can be in control and still fail its specification.

5. Describe what happened in the fee page example. With all forty requests, the limits were 130.3 to 928.3 ms and the two 503 responses fell below the lower limit; their cause, a database backup job running during the test, was found and removed, so they were dropped. Recomputed on the thirty-eight served pages, the limits were 254.2 to 848.7 ms around 551.5, every point was inside and no rule fired: the served times are common-cause variation.

6. Why can a control chart be useful for a process measure such as repair time? It separates repairs that took long for a special reason from the process's ordinary variation. Release 2.0's 72-hour fix was beyond the upper limit of 65.3 hours and had an outside cause, a wait for the payment gateway's provider, which calls for a specific response; the other repairs, including one of 30 hours, were within the process's own variation.