Quality Concepts: Variation, Design and Conformance

The standards sort its sources into two kinds.

• A common cause is a "source of variation of a process that exists because of normal and expected interactions among components of a process". Every request for ExamReg's fee page crosses the same network, waits behind the same server's other work and reads the same database. Small differences in each of these, from one request to the next, add up to differences in response time, and no single one of them can be named as the cause of a particular page's time.
• A special cause is a "source of variation that is not inherent in the system, is not predictable, and is intermittent". A disk that fails, a setting changed by mistake, a release with a new defect: something happens that is not part of how the process normally works, and it can be found.

ASQ's page on the control chart puts the pair briefly: variation comes from "special causes (non-routine events) or common causes (built into the process)". A process whose variation comes from common causes alone is a stable process, one "from which all special causes of process variation have been removed and prevented from recurring, so that only common causes of process variation of the process remain" (ISO/IEC/IEEE 24765:2017). The same ASQ page says what stability buys: it shows "whether the process variation is consistent (in control) or is unpredictable (out of control, affected by special causes of variation)". Stable means predictable. It does not mean good: a stable process can be predictably too slow.

Worked example 1: the fee page's forty requests

Chapter Forty-Five, on load testing, read the results of forty requests for ExamReg's fee page. The program below takes the same forty, in the order they were sent, and looks at their variation: the spread of the pages that were served, drawn as a histogram with one # for each page, and the requests that stand apart from it.

from statistics import mean, stdev

# Chapter 45's forty requests for ExamReg's fee page, in order: (elapsed ms, HTTP response code)
runs = [(380, 200), (410, 200), (417, 200), (463, 200), (454, 200), (516, 200), (491, 200),
        (569, 200), (528, 200), (622, 200), (565, 200), (675, 200), (602, 200), (728, 200),
        (639, 200), (431, 200), (676, 200), (484, 200), (413, 200), (537, 200), (450, 200),
        (590, 200), (487, 200), (643, 200), (524, 200), (696, 200), (561, 200), (749, 200),
        (598, 200), (452, 200), (635, 200), (505, 200), (672, 200), (558, 200), (409, 200),
        (120, 503), (446, 200), (664, 200), (95, 503), (717, 200)]

served = [ms for ms, code in runs if code == 200]
print(f"{len(served)} pages served: {min(served)} to {max(served)} ms,"
      f" mean {mean(served):.0f}, standard deviation {stdev(served):.0f}")
for low in range(350, 750, 50):
    n = sum(low <= ms < low + 50 for ms in served)
    print(f"   {low}-{low + 49} ms  {'#' * n}")

for i, (ms, code) in enumerate(runs, 1):
    if code != 200:
        print(f"request {i}: no fee page; response code {code} after {ms} ms")

38 pages served: 380 to 749 ms, mean 551, standard deviation 104
   350-399 ms  #
   400-449 ms  ######
   450-499 ms  #######
   500-549 ms  #####
   550-599 ms  ######
   600-649 ms  #####
   650-699 ms  #####
   700-749 ms  ###
request 36: no fee page; response code 503 after 120 ms
request 39: no fee page; response code 503 after 95 ms

The served pages. Thirty-eight pages were served, in times from 380 to 749 milliseconds around a mean of 551. They form one spread with no gaps in it. Nothing in the picture marks any of them out, and nothing in the data says why request 28 took 749 ms while request 1 took 380. Every one of them passed through the same network, server and database, and variation that comes from shared, normal sources like these is common-cause variation by the definition above.

The two failures. Requests 36 and 39 are different in kind. They got no fee page at all: they came back with response code 503, in 120 and 95 ms, faster than any page that was served. They are not the slow end of the spread; they are something that happened, twice in forty requests, and not on the other thirty-eight. That is what the definition of a special cause describes: not inherent in the system, not predictable, intermittent. The test's own load does not explain them: Chapter Forty-Five, on load testing, found that never more than two of its requests were in flight at once. So the tester reports them as a defect, and someone finds the cause.

What the histogram cannot show. A histogram throws away the order in which the pages were served. Whether the thirty-eight served pages are all common-cause variation, or hide a special cause of their own, such as a slow start or a drift over time, depends on that order. The tools that keep the order are the control chart and the run chart, which Chapters Ninety-One and One Hundred Seven, on statistical process control and run charts, apply to ExamReg's data.

Two kinds of cause, two responses

The two kinds of cause need different responses. ASQ lists among the uses of a control chart "determining whether your quality improvement project should aim to prevent specific problems or to make fundamental changes to the process".

• A special cause is found and removed where it happened. The 503 responses have a cause that can be traced, fixed and prevented from recurring, and the people who run and maintain the portal can do it.
• Common-cause variation is reduced only by changing the process. No single served page has a cause of its own to remove. To make pages like request 28 rarer, the system that serves every page has to change: a faster server, a cached fee table, a lighter page. Each of these is a decision about design and money, not a repair.

Confusing the two wastes effort either way. Investigating request 28 as if it had a cause of its own spends time on a difference the process produces all the time, and finds nothing a fix would remove. Filing the 503 responses under the portal is sometimes slow leaves a defect in the product.

The Juran Institute's account of Joseph Juran's trilogy draws a similar line, from the side of cost. It describes a sporadic spike in failures, which "resulted from some unplanned event such as a power failure, process breakdown, or human error", and a chronic level of waste, which "goes on and on until the organization decides to find its root causes and remove it". Against the spike, the people doing the work restore the usual level. Against the chronic waste they cannot act alone: "Under conventional responsibility patterns, the operating forces are unable to get rid of the defects or waste." Chapter Eighty-Two, on Shewhart, Deming and Juran, returns to the trilogy.

Quality of design and quality of conformance

Chapter Twenty-Two, on what quality means, gave ISO's definition of quality and two classic short ones: Juran's "fitness for use" and Philip Crosby's "conformance to requirements". ASQ's glossary sets out two technical meanings side by side: quality is "1) the characteristics of a product or service that bear on its ability to satisfy stated or implied needs; 2) a product or service free of deficiencies."

The two meanings ask two different questions of any product, and this book gives each a name for what it judges.

• Quality of design judges the plan: are the characteristics chosen for the product, its requirements, its design and its grade, the right ones for the people who will use it? A fee page designed for desktop browsers alone would have a low quality of design for students who register from their phones, however well it was built.
• Quality of conformance judges the build: does the product as made match its plan? ExamReg's fee rules say 1 to 7 days late costs Rs 100; a version coded so that the band ends at 6 days, and charges Rs 500 at exactly 7, has a conformance defect, the one Chapter Fifty-Five, on boundary value analysis, found.

A product needs both. A perfect build of the wrong design fails its users as surely as a faulty build of the right one. The two questions are ones this book has asked before: conformance is what verification checks, that "you built it right", and design is what validation checks, that "you built the right thing" (CMMI's plain words, Chapter Twenty-Six, on verification and validation). ASQ's page on the cost of quality speaks in the same terms when it describes failures that occur "when the results of work fail to reach design quality standards".

Worked example 2: which checks find which kind of defect

Release 2.0's 200 defects were recorded with the work product each was in and the activity that found it: the table Chapter Seventy-Nine, on defect metrics, worked from. A defect in the requirements or the design is a flaw in the plan, a failure of quality of design. A defect in the code is a departure from the plan, a failure of conformance. (The twelve defects in the user documents are left out: a document can be wrong in either way.) The program asks which checks found each kind.

# release 2.0's defects: the work product each was in, and the activity that found it (FINDINGS 5.2.6)
found_by = ["requirements review", "design review", "code review", "unit testing",
            "integration testing", "system testing", "acceptance testing", "after release"]
origin = {"requirements": [16, 3, 1, 1, 1, 2, 3, 1],
          "design":       [0, 21, 5, 3, 4, 4, 1, 2],
          "code":         [0, 0, 28, 40, 26, 20, 3, 3]}
checks = {"reviews": ["requirements review", "design review", "code review"],
          "tests built from the plan": ["unit testing", "integration testing", "system testing"],
          "the users": ["acceptance testing", "after release"]}

for name, rows in [("flaws in the plan (requirements, design)", ["requirements", "design"]),
                   ("departures from the plan (code)", ["code"])]:
    total = sum(sum(origin[r]) for r in rows)
    print(f"{name}: {total} defects, found by")
    for check, activities in checks.items():
        n = sum(origin[r][found_by.index(a)] for r in rows for a in activities)
        print(f"   {check:<26} {n:>3}  ({n / total:.0%})")

flaws in the plan (requirements, design): 68 defects, found by
   reviews                     46  (68%)
   tests built from the plan   15  (22%)
   the users                    7  (10%)
departures from the plan (code): 120 defects, found by
   reviews                     28  (23%)
   tests built from the plan   86  (72%)
   the users                    6  (5%)

The two kinds were found by different checks. Unit, integration and system tests, which are built from the requirements and the design, found 86 of the 120 departures from the plan, 72 per cent, but only 15 of the 68 flaws in it, 22 per cent. That is what their construction predicts: a test derived from a wrong requirement expects the wrong result, and passes when the code faithfully does the wrong thing. The plan's flaws were found mostly by reviews, 46 of the 68. And twice the share of them got as far as the users, at acceptance testing or after release: 10 per cent, against 5 per cent of the code's defects.

The lesson is the one the two names carry. Conformance can be checked against the plan; the plan itself has to be checked against the users' needs, by reviews of the requirements and the design and by validation with the people who will use the product. Chapter Ninety-Six, on why reviews pay, prices the difference between finding a flaw in the plan early and finding it late.

Quality control: holding the variation within limits

Quality control is where variation and conformance meet. ExamReg's fee page will never be served in the same time twice, and its code will never be written without defects; what matters is whether the variation stays within what the design allows, and whether the departures are caught. ASQ's glossary gives one definition of quality control as "the operational techniques and activities used to fulfill requirements for quality"; an older definition in SEVOCAB describes the loop: "monitoring service performance or product quality, recording results, and recommending necessary changes" (ISO/IEC/IEEE 24765c:2014). Chapter Twenty-Four, on quality control and quality assurance, set out the current definitions and where testing fits.

For software the loop has three steps, each already met in this book.

1. Measure a characteristic of the product: response times from a load test, defects from a review, results from a test run.
2. Compare it with what the design requires: the fee page within 2 seconds, every fee as the rules say.
3. Act on the difference: report the defect, fix it and confirm the fix.

ASQ's glossary adds a warning about the word itself. Control can mean "an evaluation to indicate needed corrective responses", "the act of guiding" or "the state of a process in which the variability is attributable to a constant system of chance causes". The first is the loop above; the last is the stable process of the previous sections. A project can run the loop faithfully, catching and correcting every departure it finds, while its process is not in control in the last sense.

The cost of quality, in outline

Everything in this chapter costs money: the reviews that judge the plan, the tests that check conformance, the fixes, and the failures that get through to the users. The cost of quality counts it. ASQ describes it as a way for an organisation to determine "the extent to which its resources are used for activities that prevent poor quality, that appraise the quality of the organization's products or services, and that result from internal and external failures", and so in four categories.

The standard definition puts the same total in two parts: the costs of "assuring that a product or service conforms to requirements", plus the "failure costs from non-conformance to requirements". Crosby, as ASQ reports, called the measure the "price of nonconformance" and "argued that organizations choose to pay for poor quality". Chapters One Hundred One and One Hundred Two, on the cost of quality and on using quality costs for decision making, draw up ExamReg's statement and use it to decide.

Management's part

Three of this chapter's findings end at the same place. Common-cause variation is reduced only by changing the system. Flaws in the plan are caught by reviews that someone has to schedule and staff. And the cost of quality depends on how the budget is divided between prevention, appraisal and failure. Each is a decision about the process, its people and its money, and the people doing the work cannot take it on their own.

ISO's quality management principles make leadership the second of seven: "Leaders at all levels establish unity of purpose and direction and create conditions in which people are engaged in achieving the organization's quality objectives." W. Edwards Deming gave the reason in the tenth of his fourteen points for management, which asks managers to stop exhorting the work force to zero defects, because "the bulk of the causes of low quality and low productivity belong to the system and thus lie beyond the power of the work force." The Juran Institute's account of the trilogy says what the operating forces can do without such leadership: "What they can do is to carry out control", which is holding the chronic level where it is. Moving it needs more: "Breakthrough requires special methods and leadership support to attain significant changes and results." ASQ's page on total quality management lists among its common difficulties "Insufficient resources or lack of sustained commitment of those resources" and "Management's failure to recognize and/or reward achievements", both of them management's to put right.

On ExamReg the division is plain. The maintenance team can find and remove the cause of the 503 responses. A faster fee page for every student needs the college to pay for a better server or the software house to schedule a redesign. The review checklist of Chapter Eighty, on using defect data to improve the process, needed someone with authority over the project's plan to give reviews the time. Chapter Ninety-Four, on the ISO 9000 family and its seven principles, sets out the rest.

What it does not mean

Variation is not a defect. Every process varies. A defect is a departure from what the design allows, and a special cause is a reason to look, not a verdict.

Stable is not good. A stable process is predictable; it can predictably fall short of its requirements, and then only a change to the process helps.

Conformance is not the whole of quality. A product that conforms perfectly to a poor design is still poor. Quality of design and quality of conformance are judged separately, by different checks.

Quality control is not only testing. Testing is "a major form of quality control", in the ISTQB syllabus's words, and not the only one; a review of a design is quality control too.

Management's part is not a slogan. It is a set of decisions: time for reviews, money for a better system, and changes to how the work is done.

Quick revision

• Variation: every process varies. Common cause (ISO/IEC/IEEE 24765): "normal and expected interactions among components of a process"; special cause: "not inherent in the system, is not predictable, and is intermittent". ASQ: "special causes (non-routine events) or common causes (built into the process)".
• Stable process: only common causes remain; stable means predictable, not good.
• Responses: a special cause is found and removed; common-cause variation is reduced only by changing the process.
• Quality of design: are the chosen requirements, design and grade right for the users? Checked by reviews of the plan and by validation. Quality of conformance: does the product match its design? Checked by verification, mostly by tests built from the plan.
• Quality control: measure, compare with what the design requires, act on the difference.
• Cost of quality: prevention, appraisal, internal failure, external failure (ASQ); the costs of assuring conformance plus the failure costs of non-conformance (ISO/IEC/IEEE 24774:2021).
• Management: owns the decisions that change the system; ISO's leadership principle.
• Worked examples: 38 fee pages served in 380 to 749 ms, and two requests that got no page, with the marks of special causes; release 2.0's 68 flaws in the plan were found 68 per cent by reviews and 22 per cent by tests, its 120 departures from the plan 72 per cent by tests.

Test yourself

1. What is variation, and what are its two kinds of cause? Variation is the difference between one output of a process and the next; every process has it. Common causes are built into the process: the normal interactions of its parts, present all the time, none of which can be singled out as the cause of a particular result. Special causes are not part of the process: unpredictable, intermittent events, such as a failed disk or a faulty change, which can be found and removed.

2. Why does it matter which kind of cause lies behind a variation? Because the responses differ. A special cause is found and removed where it happened. Common-cause variation has no single cause to remove; it is reduced only by changing the process, which is usually a management decision. Treating one as the other either wastes effort on routine differences or leaves a real problem in place.

3. Distinguish quality of design from quality of conformance, with an example. Quality of design is how well the chosen requirements, design and grade suit the users' needs; quality of conformance is how closely the product as built matches that design. A fee page designed for desktop browsers alone has a low quality of design for students on phones, however well it is built; a correctly specified fee page whose code ends the Rs 100 late fee band a day early has a conformance defect.

4. Why do tests find conformance defects more easily than design defects? Because tests are derived from the requirements and the design; a test built from a wrong requirement expects the wrong result and passes. In release 2.0, tests built from the plan found 72 per cent of the code's defects but only 22 per cent of the requirements and design defects, most of which were found by reviews.

5. What is quality control? Describe its loop. The activities that check a product against what its design requires and act on the differences. The loop: measure a characteristic of the product, compare it with the requirement, and act on any difference by reporting, fixing and confirming the fix.

6. What is management's part in quality? Taking the decisions the people doing the work cannot take alone: changing the system that produces common-cause variation, giving reviews the time and people to catch flaws in the plan, and dividing spending between prevention, appraisal and failure. ISO's leadership principle puts it as establishing unity of purpose and direction and creating the conditions in which people achieve the quality objectives.