Software Testing and Quality Assurance Notes | B.Sc. (Computer Science) Semester 5 | Mumbai University | munotes

1. A system or component. Testing can be aimed at a single function, a module, a whole application, or a system made of several applications. What is being tested at any moment is the test item, and choosing it is the first decision in any test.
2. Executed under specified conditions. A test is not "try it and see". The inputs, the starting state of the data, the environment and the order of steps are decided in advance and written down, so that the test can be repeated by someone else and give the same result.
3. The results are observed or recorded. What the software actually did is captured: the value returned, the page shown, the message printed, the record written to the database.
4. An evaluation is made. The observed result is compared with the expected result, which was also decided in advance. A test without an expected result is only a demonstration: nobody can say whether it passed. Chapter Seven, on what a test case is, gives the expected result the attention it needs.

The ISO definition adds two things IEEE 730's does not. It speaks of discovery or evaluation, so testing is not only hunting for bugs but also measuring properties such as speed, ease of use or security. And it speaks of test items, not programs, which is what makes a review of a requirements document a form of testing too.

Two kinds of testing: dynamic and static

The standards name the two kinds precisely. ISO/IEC/IEEE 29119-2 defines dynamic testing as "testing in which a test item is evaluated by executing it", and static testing as "testing in which a test item is examined against a set of quality or other criteria without the test item being executed".

The difference in the "what it finds" row matters. Dynamic testing sees a failure, the software doing the wrong thing, and someone must then work back to the defect that caused it. Static testing looks straight at the work product and sees the defect itself. Chapter Two, on errors, faults and failures, explains those words exactly.

The objectives of testing

Testing is done for more than one reason, and a test plan states which ones apply. The ISTQB syllabus lists the typical objectives as follows; the right-hand column says what each means in practice.

The syllabus adds that objectives vary with context: the work product, the test level, the risks, the development life cycle in use, and business factors such as time to market. A test of a hospital's dosage calculator and a test of a college canteen's menu page do not have the same objectives.

Worked example: the first test in this book

This book follows one invented system throughout, ExamReg, a college's online examination registration portal. Students log in, fill in the semester examination form, pay the fee and download the hall ticket. Every rule of ExamReg is this book's own example, not a rule of the University.

One of its rules is the late fee, and it is the whole specification the tester is given:

A programmer writes the function below. A tester, who has not seen the code, writes five test cases from the rule alone, each with an input and an expected result, and runs them.

def late_fee(days_late):
    """Late fee in rupees for a form submitted days_late days after the last date."""
    if days_late > 15:
        raise ValueError("form not accepted more than 15 days late")
    if days_late <= 7:
        return 100
    return 500

tests = [            # (days late, expected fee), written from the rule alone
    (0, 0),
    (3, 100),
    (7, 100),
    (8, 500),
    (15, 500),
]
for days, expected in tests:
    actual = late_fee(days)
    verdict = "pass" if actual == expected else "FAIL"
    print(f"{days:>2} days late: expected {expected:>3}, got {actual:>3}  {verdict}")

 0 days late: expected   0, got 100  FAIL
 3 days late: expected 100, got 100  pass
 7 days late: expected 100, got 100  pass
 8 days late: expected 500, got 500  pass
15 days late: expected 500, got 500  pass

Four tests pass and one fails. Look at what the failure says: a student who submits the form on time is charged Rs 100. The program runs, it gives a sensible-looking number for every input, and on four of the five inputs it is right. Nothing about it looks broken until a test asks the one question it gets wrong.

Notice also where the failing input came from. The tester chose 0 because the rule has a boundary there, between "on time" and "late". Chapter Fifty-Five, on boundary value analysis, turns that instinct into a method.

Testing and debugging are different activities

The failing test has done its job. It has shown that a defect exists. It has not said where the defect is, and finding it is a different activity with a different name. The ISTQB syllabus describes debugging after a dynamic test fails as three steps: reproduction of the failure, diagnosis (finding the defect), and fixing the defect.

Here the three steps are short. Reproduce: call late_fee(0) again and get 100 again. Diagnose: the condition days_late <= 7 is true for 0, so an on-time form falls into the Rs 100 branch; the rule's first case, "on or before the last date", was never written into the code. Fix: handle that case first.

Then two more kinds of testing follow the fix. Confirmation testing reruns the test that failed, to confirm the fix works. Regression testing reruns the tests that passed, to confirm the fix has not broken anything else. Both are run below.

def late_fee(days_late):
    """Late fee in rupees for a form submitted days_late days after the last date."""
    if days_late > 15:
        raise ValueError("form not accepted more than 15 days late")
    if days_late <= 0:                  # on or before the last date: the missing case
        return 0
    if days_late <= 7:
        return 100
    return 500

tests = [(0, 0), (3, 100), (7, 100), (8, 500), (15, 500)]
print("confirmation test:", "pass" if late_fee(0) == 0 else "FAIL")
failed = [days for days, expected in tests if late_fee(days) != expected]
print("regression run:", len(tests) - len(failed), "of", len(tests), "pass")

confirmation test: pass
regression run: 5 of 5 pass

Debugging has a chapter of its own later in Module 1 (Chapter Forty-Seven), with its techniques.

Dijkstra's warning

The fixed function passes all five tests. Is it now correct? The five tests cannot say. They checked five inputs out of every integer the function might receive, and a defect could be sitting on an input nobody tried.

Edsger Dijkstra put this limit in one sentence in his 1972 ACM Turing Lecture, "The Humble Programmer": "program testing can be a very effective way to show the presence of bugs, but is hopelessly inadequate for showing their absence."

The late-fee function shows both halves. The first test run showed the presence of a bug, effectively and cheaply. The second run showed that five tests pass, which is not the same as showing that no bug remains. What happens, for instance, if days_late is negative? The rule does not say, and the tests did not ask. Chapter Four turns this limit into the first of the seven principles of testing.

A static test of the same rule

Go back to the rule itself and read it as a tester, without running anything. It says what happens on time, 1 to 7 days late, 8 to 15 days late and more than 15 days late. It does not say what the portal should do if the date on a form is before the day the form was opened, or if the number of days is missing. Those are gaps in the requirement.

Noticing them is static testing: a work product was examined against a criterion (is every possible input covered?) without executing anything. It found a defect in the requirement, and it found it before a single line of code depended on the missing answer. The same questions asked after release would arrive as complaints.

What testing does not mean

Testing is not only running the program. Reviewing a requirement, a design or the code is testing too: static testing. The ISTQB syllabus calls the belief that testing only consists of executing tests "a common misconception".

A test that passes does not prove the program correct. It proves the program gave the expected result for that input, in that environment, on that day. Dijkstra's sentence is the reason.

Testing is not debugging. Testing shows that something is wrong; debugging finds out why and removes it. They are often done by different people, and they need different information.

Finding bugs is not the only objective. Measuring quality, checking compliance, building confidence and giving managers the information to decide on a release are objectives too. A test run in which nothing fails is still useful: it is evidence.

A test without an expected result is not a test. Running the program and looking at what comes out, with nothing decided in advance to compare it with, tells nobody whether it passed.

Where testing stops

Testing samples. It runs some inputs out of an enormous number, so it can reduce the risk that a defect reaches users but can never remove it. It also depends on someone knowing the right answer: if the expected result is itself wrong, a correct program fails the test and a wrong one passes. And testing costs time and money, so every project decides how much testing is enough, a question Module 1 returns to in its strategic approach to testing, which asks when testing is complete (Chapter Thirty-One).

Quick revision

• Software testing: a set of activities that evaluates quality and discovers defects in software and its work products.
• ISO/IEC/IEEE 29119-2: testing is a "set of activities conducted to facilitate discovery or evaluation of properties of one or more test items".
• Dynamic testing executes the test item; static testing examines it without executing it (reviews, static analysis).
• A test needs an input, specified conditions and an expected result; the actual result is compared with it.
• ISTQB objectives include evaluating work products, finding defects, coverage, reducing risk, verifying requirements, compliance, informing decisions, building confidence and validating.
• Testing shows failures; debugging reproduces, diagnoses and fixes; then confirmation testing reruns the failed test and regression testing reruns the rest.
• Dijkstra (1972): testing can show the presence of bugs, never their absence.
• Worked example: ExamReg's late_fee(0) returned 100 instead of 0; one test in five found it.

Test yourself

1. Define software testing and name its two kinds. Software testing is a set of activities that evaluates the quality of software and its work products and discovers defects in them. Dynamic testing executes the test item under specified conditions and evaluates the results; static testing examines a work product, such as a requirement or the code, without executing it.

2. Why must a test case have an expected result decided in advance? Because a test is an evaluation: the actual result is compared with the expected one. Without an expected result nobody can say whether the test passed, and a wrong output would be accepted simply because it looked reasonable.

3. Differentiate testing from debugging. Testing shows that failures occur, or finds defects directly in a work product; it starts from a requirement and ends with a pass or fail verdict. Debugging starts from an observed failure, reproduces it, diagnoses the defect and fixes it; it is followed by confirmation testing and regression testing.

4. In the late-fee example, which test failed, and what did the fix require besides changing the code? The test with 0 days late failed: the function returned Rs 100 instead of 0. After the fix, confirmation testing reran that test and regression testing reran the four tests that had passed, to show the fix broke nothing else.

5. Explain Dijkstra's statement about testing, with an example. Testing can show that bugs are present but cannot show that they are absent, because it checks only the inputs actually tried. In the late-fee example, five passing tests say nothing about a negative number of days, which no test tried and the rule does not cover.

6. Is reading a requirements document and finding a gap in it testing? Justify. Yes. It is static testing: a work product is examined against a criterion, here completeness, without anything being executed. It finds the defect directly and earlier than any dynamic test could.

7. State any four objectives of testing. Any four of: evaluating work products; causing failures and finding defects; ensuring required coverage; reducing the risk of inadequate quality; verifying that requirements are fulfilled; verifying compliance with contractual, legal and regulatory requirements; providing information for decisions; building confidence; validating that the product works as stakeholders expect.

Read the table from top to bottom and it is a story in time: the root cause came first, then the mistake, then the defect sat in the code for as long as nobody ran it with zero, and then the failure happened on the first day a student submitted on time.

A defect does not always cause a failure

The ISTQB syllabus is careful here: "Some defects will always result in a failure if executed, while others will only result in a failure in specific circumstances, and some may never result in a failure." The late-fee defect fails every time the input is zero. The next program shows the second kind.

A leap-year check is written with only the first half of the calendar rule: a year divisible by 4 is a leap year. The full Gregorian rule adds that a year divisible by 100 is not, unless it is also divisible by 400. Python's standard library already implements the full rule as calendar.isleap, so it serves as the oracle, the source of the right answer.

import calendar

def is_leap(year):
    return year % 4 == 0          # the defect: the rule for century years is missing

for first, last in [(1901, 2099), (1600, 2400)]:
    wrong = [y for y in range(first, last + 1) if is_leap(y) != calendar.isleap(y)]
    print(f"years {first} to {last}: {last - first + 1} tried, "
          f"{len(wrong)} failure(s) {wrong}")

years 1901 to 2099: 199 tried, 0 failure(s) []
years 1600 to 2400: 801 tried, 6 failure(s) [1700, 1800, 1900, 2100, 2200, 2300]

The defect is present on every run. It causes no failure at all for any year from 1901 to 2099, which includes every year a student's date of birth or an examination could fall in. Over six centuries it fails six times. Tested only on this century's years, the function would pass every test anyone thought to write, and the defect would wait for 2100.

Two lessons follow. A defect's presence and its failure are different facts: testing observes failures, and a defect that cannot be made to fail by the inputs tried stays invisible. And the choice of test inputs decides what can be seen, which is why Module 2 spends a whole row of the syllabus on choosing them (Chapter Fifty-Four onwards, from equivalence partitioning).

Failures without defects, and defects without code

Not every failure comes from a defect in the code. The ISTQB syllabus notes that "Failures can also be caused by environmental conditions, such as when radiation or electromagnetic fields cause defects in firmware." A disk that is full, a network that drops, a clock set to the wrong year or a server configured differently from the test machine can all make correct code fail.

And not every defect is in the code. Defects "can be found in documentation, such as a requirements specification or a test script, in source code, or in a supporting work product such as a build file." The missing answer for a negative number of days, found while testing the late-fee rule in Chapter One, was a defect in the requirement. The syllabus adds the reason this matters so much: defects in work products produced early, "if undetected, often lead to defective work products later in the lifecycle." One gap in a requirement becomes a gap in the design, then in the code, then in the tests written from the same requirement.

Where errors come from

People do not make mistakes at random. The ISTQB syllabus lists typical reasons: "time pressure, complexity of work products, processes, infrastructure or interactions, or simply because they are tired or lack adequate training." To those every project adds a few of its own:

• Miscommunication: the person who knows the rule is not the person who codes it, and the rule loses a clause on the way.
• Changing requirements: the code was right for last month's rule.
• Unfamiliar technology: a new framework or language whose defaults the programmer does not yet know.
• Complex interactions: two modules each correct, whose combination was never thought through (Chapter Thirty-Seven, on integration testing, is about exactly this).

These are the root causes that root cause analysis looks for, and Module 2 returns to them in the chapter on using defect data to improve the process (Chapter Eighty).

The word "bug"

Students are often told that Grace Hopper invented the word "bug" when a moth was found in a computer. The object is real, and it is in the Smithsonian's National Museum of American History, which catalogues it as a 1947 log book from the Mark II computer at Harvard University. The museum's own record says the engineers "taped the insect in their logbook and labeled it 'first actual case of bug being found.'"

The same record corrects the legend in three ways. The word was older: "Thomas Edison talked about bugs in electrical circuits in the 1870s." The joke in the label depends on that: it was the first actual bug, a literal insect, in a machine whose faults were already called bugs. And of the log book itself the museum says it "was probably not Hopper's", though Hopper and the Mark II team "helped popularize the use of the term computer bug and the related phrase 'debug.'"

Other words for the same family

Standards and tools use several more words around a defect. Each has its own shade of meaning, and Module 2's defect management row gives them their full treatment (Chapter Seventy-Three, on what a defect is).

Note the last row. "Error" has two meanings in the standards: the human mistake of the ISTQB sentence, and a wrong value inside a running system. A numerical analyst's "rounding error" is the second meaning. When a textbook says "an error occurred", read the sentence around it to see which is meant.

Classifying defects

A defect that is only counted teaches nothing. Classified, a collection of defects shows where a process is weak. Three classifications are in common use and a defect gets one value in each.

By where it was introduced. Requirements, design, code, documentation, or a supporting item such as a build or configuration file. This tells a team which activity is leaking.

By type. IBM's Orthogonal Defect Classification (Chillarege and six colleagues, IEEE Transactions on Software Engineering, 1992) defines the types by the kind of correction the programmer makes, and distinguishes in each case "between something missing or something incorrect". Its eight types, in the paper's own terms:

By severity. How bad the failure is for the user, from a crash or data loss down to a cosmetic flaw. Severity, and priority, which is a different thing, are taught with the defect report in Module 2 (Chapter Seventy-Six).

Worked example: classifying three ExamReg defects

Three defects are found in one week on ExamReg. Each is classified below, with the reason, because the reason is what an examiner looks for.

The third row is worth a second look. Each module could pass its own tests perfectly, and the pair would still charge a student one hundred times too much. It is a defect of the kind integration testing exists to catch.

When the test is wrong: false positives and false negatives

A test result can itself be wrong, in two ways. If a tester misreads the rule and writes that 7 days late should cost Rs 500, a correct function fails the test: the test reports a defect that does not exist. If the expected result for 0 days had been written as Rs 100, copying the code instead of the rule, the defective function would pass: a real defect goes unreported.

def late_fee(days_late):              # the CORRECTED function from Chapter One
    if days_late > 15:
        raise ValueError("form not accepted more than 15 days late")
    if days_late <= 0:
        return 0
    if days_late <= 7:
        return 100
    return 500

expected_from_a_misread_rule = {7: 500}      # the tester's mistake, not the program's
for days, expected in expected_from_a_misread_rule.items():
    actual = late_fee(days)
    print(f"{days} days late: expected {expected}, got {actual}:",
          "pass" if actual == expected else "FAIL, but the program is right")

7 days late: expected 500, got 100: FAIL, but the program is right

The names for these two cases are a trap, because two current standards use them in opposite senses. ISO/IEC 23643:2020 defines a false positive as an "observed defect which does not correspond to a true defect" and a false negative as a "true defect that has not been observed". ISO/IEC TR 29119-11:2020, on testing AI systems, defines a false positive as "incorrect reporting of a pass when in reality it is a failure", which is the reverse. This book uses the first sense, which is also the ISTQB's: a false positive reports a defect that is not there, and a false negative misses one that is. In an answer, define the term before you use it and no examiner can mark you wrong.

What it does not mean

Every defect does not cause a failure. Some fail every time, some only on particular inputs, and some never. The leap-year defect above failed six times in eight hundred years.

Every failure is not caused by a defect in the code. The environment, a configuration, the data, or a wrong expected result in the test can all produce a failure.

Defects are not only in code. Requirements, designs, test scripts, user manuals and build files can all be defective, and early ones propagate.

"Error" does not always mean a human mistake. In ISO/IEC/IEEE 15026-1 it means a wrong value inside the system.

Grace Hopper did not coin "bug". The Smithsonian's record dates the word to Edison in the 1870s; the 1947 log book is famous because its bug was a real moth.

Quick revision

• Error (mistake): "human action that produces an incorrect result" (ISO/IEC/IEEE 24765:2017).
• Defect (fault, bug): an imperfection in a work product that fails its requirements and must be repaired or replaced (ISO/IEC 23531:2020).
• Failure: the system not performing as required; "departure of system behavior from system requirements" (IEEE 982:2024).
• Root cause: the source whose removal removes or reduces the defect (ISO/IEC/IEEE 24765:2017).
• Chain: root cause, error, defect, failure. A defect may fail always, sometimes or never; failures can also come from the environment.
• Causes (ISTQB): time pressure, complexity, fatigue, lack of training; plus miscommunication and changing requirements.
• Classify by origin, by ODC type (function, assignment, interface, checking, timing/serialization, build/package/merge, documentation, algorithm; missing or incorrect) and by severity.
• Leap-year example: no failure from 1901 to 2099, six from 1600 to 2400.
• False positive (ISTQB sense): a defect reported that is not there. False negative: a real defect not reported.

Test yourself

1. Distinguish error, defect and failure with one example. An error is a human mistake, such as a programmer forgetting the on-time case of a fee rule. It leaves a defect in the work product: the function has no branch for zero days late. When the function runs with zero, the system fails: an on-time student is charged Rs 100.

2. Can a program contain a defect and never fail in testing? Explain. Yes. A defect causes a failure only when the inputs and conditions reach it. The leap-year function missing the century rule gives the right answer for every year from 1901 to 2099, so tests drawn from this century all pass while the defect remains.

3. Give two causes of failure that are not defects in the code. Environmental conditions, such as radiation or electromagnetic fields affecting firmware, a full disk or a misconfigured server; and a wrong expected result in the test itself, which makes a correct program appear to fail.

4. What is a root cause, and why is finding it worth the effort? It is the source of a defect whose removal removes or reduces the defect, such as time pressure or an ambiguous requirement. Fixing only the defect repairs one instance; removing the root cause prevents the same kind of defect from being made again.

5. Name the eight ODC defect types. Function, assignment, interface, checking, timing/serialization, build/package/merge, documentation and algorithm, each recorded as missing or incorrect.

6. Why must you define "false positive" before using it in an answer? Because standards disagree: ISO/IEC 23643:2020 uses it for a reported defect that does not exist, while ISO/IEC TR 29119-11:2020 uses it for a pass reported when the result was really a failure. Defining it removes the ambiguity.

import struct

def to_16_bit(value):
    """Pack a number into a 16-bit signed integer, as the SRI's conversion did."""
    return struct.pack(">h", int(value))

for horizontal_bias in [1250.0, 32767.0, 40000.0]:
    try:
        to_16_bit(horizontal_bias)
        print(f"{horizontal_bias:>9}: fits in 16 bits")
    except struct.error as problem:
        print(f"{horizontal_bias:>9}: conversion refused ({problem})")

   1250.0: fits in 16 bits
  32767.0: fits in 16 bits
  40000.0: conversion refused ('h' format requires -32768 <= number <= 32767)

The three values are this book's, chosen to straddle the limit; the report does not publish the flight's value of BH. What the board does say about testing is the lesson. Equipment testing of the SRI was rigorous, "However, no test was performed to verify that the SRI would behave correctly when being subjected to the count-down and flight time sequence and the trajectory of Ariane 5." And then: "Had such a test been performed by the supplier or as part of the acceptance test, the failure mechanism would have been exposed." The test was never written because the SRI's specification did not contain the Ariane 5 trajectory data as a requirement: a defect in a requirement, exactly the kind Chapter Two, on errors, faults and failures, warned propagates.

The Patriot at Dhahran, 25 February 1991: an error too small to see, run for too long

The United States General Accounting Office (GAO) report IMTEC-92-26 begins: "On February 25, 1991, a Patriot missile defense system operating at Dhahran, Saudi Arabia, during Operation Desert Storm failed to track and intercept an incoming Scud. This Scud subsequently hit an Army barracks, killing 28 Americans."

The Patriot's clock counted time "in tenths of seconds", and to predict where a Scud would appear next, the count had to be converted to a real number. Its registers were 24 bits long, so the conversion "cannot be any more precise than 24 bits." One tenth has no exact binary form, so every tick of the clock was stored very slightly short, and the shortfall accumulated for as long as the system ran. At Dhahran the battery "had been operating continuously for over 100 hours."

How short, exactly? GAO says only that the registers were 24 bits long; it does not print the format in which the tenth was held. So the program below tries two readings and lets GAO's own published figures decide. In the first, the tenth is kept to 24 binary places after the point and the rest chopped off. In the second, it is kept to 23, which is what a 24-bit register leaves for the fraction if one bit holds the sign. The figures GAO printed in its Appendix II are typed in beside the program, from the page image of the report.

from fractions import Fraction

tenth = Fraction(1, 10)

def drift(places, hours):
    """Seconds lost after `hours` if each tenth is chopped to `places` binary places."""
    stored = Fraction(int(tenth * 2**places), 2**places)
    return hours * 3600 * 10 * (tenth - stored)

for places in (24, 23):
    print(f"kept to {places} binary places: drift after 100 h = {float(drift(places, 100)):.4f} s")
print()

# GAO/IMTEC-92-26, Appendix II: hours, calculated time, inaccuracy (seconds)
gao = [(1, "3599.9966", "0.0034"), (8, "28799.9725", "0.0275"),
       (20, "71999.9313", "0.0687"), (48, "172799.8352", "0.1648"),
       (72, "259199.7528", "0.2472"), (100, "359999.6667", "0.3433")]
for hours, gao_time, gao_drift in gao:
    ours = f"{float(drift(23, hours)):.4f}"
    check = hours * 3600 - Fraction(gao_drift)
    note = "" if f"{float(check):.4f}" == gao_time else f"  <- GAO's time should be {float(check):.4f}"
    print(f"{hours:>3} h: drift {ours} s, GAO printed {gao_drift}{note}")

kept to 24 binary places: drift after 100 h = 0.1287 s
kept to 23 binary places: drift after 100 h = 0.3433 s

  1 h: drift 0.0034 s, GAO printed 0.0034
  8 h: drift 0.0275 s, GAO printed 0.0275
 20 h: drift 0.0687 s, GAO printed 0.0687
 48 h: drift 0.1648 s, GAO printed 0.1648
 72 h: drift 0.2472 s, GAO printed 0.2472
100 h: drift 0.3433 s, GAO printed 0.3433  <- GAO's time should be 359999.6567

The first reading gives a drift of about an eighth of a second at 100 hours, which is not what GAO measured, so it is rejected. The second reproduces every figure in GAO's inaccuracy column, to four decimal places: after 100 hours the clock was a third of a second out. In a third of a second a missile flying at "approximately MACH 5" covers hundreds of metres, and GAO's table gives the shift of the radar's range gate as 687 metres, enough for the system to "look in the wrong place for the incoming Scud."

Two small lessons sit inside this one. A model is tested against data, and the first one here failed its test: that is how the 23 was found, not assumed. And the program found something in the report itself. In GAO's last row the calculated time (359999.6667) and the inaccuracy (.3433) cannot both be right, because 360000 minus .3433 is 359999.6567. The inaccuracy column is the one the drift model confirms. It is a small slip in a careful report, and it is why a table of figures is worth recomputing rather than trusting.

The report's account of the timing is the sharpest lesson. Israeli data had shown a loss of accuracy after 8 consecutive hours, and "Army officials modified the software to improve the system's accuracy. However, the modified software did not reach Dhahran until February 26, 1991", the day after the Scud struck. No test had run the system for as long as it was actually used. GAO notes that the Patriot had never before been used against Scud missiles "nor was it expected to operate continuously for long periods of time."

Therac-25, 1985 to 1987: software trusted to be safe

Nancy Leveson's account "Medical Devices: The Therac-25", drawn from her investigation with Clark S. Turner (IEEE Computer, July 1993), begins: "Between June 1985 and January 1987, a computer-controlled radiation therapy machine, called the Therac-25, massively overdosed six people." Eleven machines had been installed, five in the United States and six in Canada.

The design had shifted safety from hardware to software. Earlier machines in the family had "independent protective circuits" and "mechanical interlocks for policing the machine", but "In the Therac-25, software checks were substituted for many of the traditional hardware interlocks." Code was reused from the earlier machines too, and one of the bugs was later found in the older Therac-20's software, where it had hurt nobody because that machine "includes hardware safety interlocks."

Two software defects stand out, and each was invisible to ordinary testing. At Tyler, Texas, the overdose happened only when an experienced operator edited the prescription quickly: "If the prescription data was edited at a fast pace (as is natural for someone who has repeated the procedure a large number of times), the overdose occurred." The hospital physicist had to practise before he could reproduce it, and the manufacturer's engineer at first "could not reproduce the error" at all. At Yakima, Washington, a one-byte variable called Class3 was "incremented by one in each pass" through a setup routine, and a zero value meant a safety check was skipped: "on every 256th pass through the Set Up Test code, the variable will overflow and have a zero value."

The second defect takes four lines to reproduce.

def skipped_checks(passes, fixed=False):
    """Passes on which the upper collimator check is skipped."""
    class3, skipped = 0, []
    for pass_number in range(1, passes + 1):
        class3 = 1 if fixed else (class3 + 1) % 256   # one byte: 255 + 1 wraps to 0
        if class3 == 0:                                 # zero is read as "nothing to check"
            skipped.append(pass_number)
    return skipped

print("as shipped:", skipped_checks(1000))
print("as fixed  :", skipped_checks(1000, fixed=True))

as shipped: [256, 512, 768]
as fixed  : []

The check is skipped on three passes in a thousand, and only when the operator happens to press the button on exactly one of them. The fix, as the account reports it, was "simple: the program is changed so that the Class3 variable is set to some fixed nonzero value each time through Set Up Test instead of being incremented." A defect with a one-line fix killed and injured people because no test drove the setup routine through hundreds of passes, and none drove the keyboard as fast as a practised operator did.

Knight Capital, 1 August 2012: one server missed, and dead code woken

The United States Securities and Exchange Commission's order (Release 70694, 2013) records that on August 1, 2012 Knight Capital's automated order router, SMARS, "routed millions of orders into the market over a 45-minute period, and obtained over 4 million executions in 154 stocks for more than 397 million shares." It had been processing just 212 small retail orders. "Knight lost over $460 million from these unwanted positions."

The chain of events is a lesson in two kinds of testing at once. New code was deployed to replace an old, unused function called Power Peg, and the new code "repurposed a flag that was formerly used to activate the Power Peg code." During deployment "one of Knight's technicians did not copy the new code to one of the eight SMARS computer servers," and "Knight did not have a second technician review this deployment". On the eighth server the repurposed flag woke the old code. And that old code had itself been changed years earlier without a retest: in 2005 a function it relied on was moved, and "Knight did not retest the Power Peg code after moving the cumulative quantity function to determine whether Power Peg would still function correctly if called."

Two things were missing: a regression test after a change to old code, and a review (a static check) of the deployment. Chapter Thirty-Nine, on regression testing, smoke testing and continuous integration, and Chapter Twenty-Eight, on software reviews, are about exactly these.

What failures cost a whole economy

Famous disasters are rare; ordinary failures are constant, and their cost adds up. In 2002 the United States National Institute of Standards and Technology (NIST) surveyed software developers and users and estimated that "the national annual cost estimates of an inadequate infrastructure for software testing are estimated to be $59.5 billion." It estimated that "The potential cost reduction from feasible infrastructure improvements is $22.2 billion." Software users bore about 60 per cent of the cost and developers about 40 per cent.

The figures are more than twenty years old and were for one country, so the exact amounts are history. The shape of the finding is what lasts: most of the cost of poor testing is paid by users, not by the people who wrote the software, and a large part of it could be avoided by testing better and earlier.

Why late is expensive

The failures above were found in operation, the most expensive place to find anything. Barry Boehm and Victor Basili, in their 2001 "Software Defect Reduction Top 10 List", put the first item on the list this way: "Finding and fixing a software problem after delivery is often 100 times more expensive than finding and fixing it during the requirements and design phase."

The reason is easy to see on ExamReg. A missing case in the late-fee rule, caught in a review of the requirement, costs a sentence. Caught in testing, it costs a code change and a retest. Caught after release, it costs refunds to students who were overcharged, a correction to every receipt, a public apology and a patch deployed under pressure, which is exactly the situation in which Knight Capital's technician missed a server. Chapter Ninety-Six returns to this with a defect amplification model and why reviews pay.

What testing contributes

The ISTQB syllabus lists what testing contributes to a project's success, and it is more than finding bugs:

• It "provides a cost-effective means of detecting defects", which debugging then removes, so testing contributes indirectly to higher quality.
• It "provides a means of directly evaluating the quality of a test object", and those measures feed decisions such as whether to release.
• It "provides users with indirect representation on the development project", because testers carry the users' needs into a project users cannot join.
• It "may also be required to meet contractual or legal requirements, or to comply with regulatory standards."

The syllabus also separates testing from quality assurance: testing is "a product-oriented, corrective approach" and "a major form of quality control", while QA is "a process-oriented, preventive approach". That distinction is Module 1's third row, and Chapter Twenty-Four, on quality control and quality assurance, gives it in full.

The four failures, side by side

What it does not mean

Testing does not guarantee a failure-free product. Every one of these systems was tested. Testing reduces risk; the missing test in each case was one that nobody thought to run.

Reused code is not already tested. Ariane 5 reused Ariane 4's software, and Knight reused an old flag. Code tested in one context is untested in a new one.

Testing is not only for rockets and hospitals. NIST's estimate is dominated by ordinary business software. A portal that overcharges students is a failure in the same sense.

More testing is not automatically better. Testing costs money, and the question is always which tests reduce the most risk for the effort. Chapter Four's principle that testing is context dependent says the same.

Quick revision

• Failures cost money, time and reputation, and in extreme cases cause injury or death (ISTQB v4.0.1).
• Ariane 5 (4 June 1996): unprotected 64-bit to 16-bit conversion of BH in reused Ariane 4 code; no test with Ariane 5's trajectory; "Had such a test been performed ... the failure mechanism would have been exposed."
• Patriot (25 February 1991): time converted in 24-bit registers, each tenth of a second held slightly short; after 100 hours, 0.3433 s of drift and a 687 m range-gate shift; 28 killed; the fix arrived the next day.
• Therac-25 (June 1985 to January 1987): six people massively overdosed; software checks replaced hardware interlocks; a race on fast editing, and a one-byte counter that skipped a safety check every 256th pass.
• Knight Capital (1 August 2012): new code missed on one of eight servers, a repurposed flag woke old code never retested; over $460 million lost in 45 minutes.
• NIST 2002: inadequate testing infrastructure cost the US an estimated $59.5 billion a year; $22.2 billion avoidable.
• Boehm and Basili 2001: fixing after delivery is often 100 times more expensive than in requirements and design.
• Testing detects defects, evaluates quality, represents users and meets legal requirements; it is quality control, not quality assurance.

Test yourself

1. Give four reasons why software testing is necessary. Defects are inevitable in human work; defects found late cost far more to fix than those found early; failures in operation cause financial, legal, reputational and human harm; and testing provides the evidence for release decisions and for contractual and regulatory compliance.

2. What caused the Ariane 5 failure, and which test would have exposed it? Software reused from Ariane 4 converted a 64-bit floating-point value, the horizontal bias, into a 16-bit signed integer without protection, and Ariane 5's faster early trajectory made the value too large, raising an exception in both inertial reference units. The inquiry board found that a ground test injecting Ariane 5's own trajectory would have exposed it.

3. Explain how a tiny error made the Patriot miss the Scud at Dhahran. Its clock counted tenths of a second and converted the count to a real number with only 24 bits of precision, so each tick was stored slightly short. The shortfall grew with running time: after about 100 hours of continuous operation it reached about a third of a second, shifting the radar's range gate about 687 metres, so the system looked in the wrong place.

4. What two missing checks allowed the Knight Capital failure? A review of the deployment, which would have noticed that one of eight servers had not received the new code; and a regression test of the old Power Peg code after it was changed in 2005, which was never done.

5. According to Boehm and Basili, how much more does it cost to fix a problem after delivery? Often 100 times more than finding and fixing it during the requirements and design phase.

6. Is testing the same as quality assurance? No. Testing is product-oriented and corrective, a major form of quality control; quality assurance is process-oriented and preventive, working on the basis that a good process produces a good product.

Principle 2: exhaustive testing is impossible

The syllabus: "Testing everything is not feasible except in trivial cases". Exhaustive testing means trying every possible combination of inputs and preconditions, and the numbers grow faster than intuition expects. The program below counts two cases: a function that takes two 32-bit integers, and ExamReg's examination form.

SECONDS_PER_YEAR = 365.25 * 24 * 3600

pairs = 2**32 * 2**32                      # two 32-bit integer inputs
print(f"two 32-bit inputs: {pairs:,} combinations")
print(f"  at a billion tests a second: about {pairs / 10**9 / SECONDS_PER_YEAR:,.0f} years")

form = {                                   # ExamReg's exam form, deliberately simplified
    "semester (1 to 6)": 6,
    "backlog papers (8 tick boxes)": 2**8,
    "days late (0 to 15, or later)": 17,
    "fee concession (yes or no)": 2,
    "payment mode (4 choices)": 4,
}
combos = 1
for field, values in form.items():
    combos *= values
print(f"exam form: {combos:,} combinations")
print(f"  at one manual test a minute: about {combos / 60 / 24:,.0f} days without a break")

two 32-bit inputs: 18,446,744,073,709,551,616 combinations
  at a billion tests a second: about 585 years
exam form: 208,896 combinations
  at one manual test a minute: about 145 days without a break

Even a five-field form outruns a test team, and the form ignores the student's name, the browser, the network and the order in which fields are filled. The syllabus's answer is not to give up but to choose: "test techniques ... test case prioritization ... and risk-based testing ... should be used to focus test efforts." Module 2's techniques exist to choose a few hundred tests that stand in for the millions.

Principle 3: early testing saves time and money

The syllabus: "Defects that are removed early in the process will not cause subsequent defects in derived work products." And therefore "both static testing ... and dynamic testing ... should be started as early as possible."

The mechanism is the one Chapter Two on errors, faults and failures described: a defect in a requirement is copied into the design, the code and the tests written from that requirement. Remove it while it is one sentence and it never multiplies. Boehm and Basili's figure in Chapter Three, on why software must be tested, puts the price of waiting: a fix after delivery is "often 100 times more expensive" than one during requirements and design. This principle is also called shift left, because on a time line drawn left to right it moves testing towards the start.

Applied to ExamReg. The gap in the late-fee rule (what happens with a negative number of days?) was found by reading the rule. Found there, it cost one question to the exam cell. Found after release, it would have cost refunds and a patch.

Principle 4: defects cluster together

The syllabus: "A small number of system components usually contain most of the defects discovered or are responsible for most of the operational failures." It calls this "an illustration of the Pareto principle", the observation that a few causes account for most of an effect. ExamReg's release 2.0 had 200 defects, and the program sorts them by the module they were found in.

defects = {"Exam form": 74, "Fee payment": 58, "Admin reports": 20, "Hall ticket": 18,
           "Login": 12, "Profile": 10, "Notifications": 8}
total = sum(defects.values())
running = 0
for module, count in sorted(defects.items(), key=lambda kv: -kv[1]):
    running += count
    print(f"{module:<14}{count:>4}  {100 * count / total:5.1f}%   cumulative {100 * running / total:5.1f}%")

Exam form       74   37.0%   cumulative  37.0%
Fee payment     58   29.0%   cumulative  66.0%
Admin reports   20   10.0%   cumulative  76.0%
Hall ticket     18    9.0%   cumulative  85.0%
Login           12    6.0%   cumulative  91.0%
Profile         10    5.0%   cumulative  96.0%
Notifications    8    4.0%   cumulative 100.0%

Two modules out of seven hold two thirds of the defects. The two are not an accident: the exam form holds every eligibility and paper-selection rule, and the fee module every calculation and the link to the payment gateway. Complexity attracts defects. The syllabus draws the practical conclusion: known and predicted clusters "are an important input for risk-based testing". The next release's test effort should go where these defects were found. Module 2 turns this into a method with the Pareto diagram (Chapter One Hundred Four).

Principle 5: tests wear out

The syllabus: "If the same tests are repeated many times, they become increasingly ineffective in detecting new defects." The 2018 syllabus called this the pesticide paradox, and explained the name: tests stop finding defects "just as pesticides are no longer effective at killing insects after a while."

It happens because a suite of tests only asks the questions it was written to ask. Once the defects those questions can find are fixed, the suite goes quiet, and a new defect somewhere it does not look is invisible to it. Below, ExamReg's late-fee function is rewritten in release 2 as a lookup table, and one entry is typed wrongly. The five tests written in Chapter One, on what software testing is, are run again, then one new test.

FEE = {0: 0}
FEE.update({day: 100 for day in range(1, 8)})
FEE.update({day: 500 for day in range(8, 16)})
FEE[12] = 50                                   # release 2: a typing slip, 50 for 500

def late_fee(days_late):
    if days_late > 15:
        raise ValueError("form not accepted more than 15 days late")
    return FEE[max(days_late, 0)]

old_suite = [(0, 0), (3, 100), (7, 100), (8, 500), (15, 500)]
passed = sum(late_fee(days) == expected for days, expected in old_suite)
print(f"old suite: {passed} of {len(old_suite)} pass")
print("new test, 12 days late: expected 500, got", late_fee(12))

old suite: 5 of 5 pass
new test, 12 days late: expected 500, got 50

The old suite passes perfectly and the defect ships unless someone writes a new test. The remedy in the syllabus: "existing tests and test data may need to be modified, and new tests may need to be written." It also notes the one case where repetition is the point: in automated regression testing the same tests are rerun deliberately, to prove that nothing that used to work has broken (Chapter Thirty-Nine, on regression testing).

Principle 6: testing is context dependent

The syllabus: "There is no single universally applicable approach to testing. Testing is done differently in different contexts." What changes with context is how much testing is enough, which techniques are used, how formally it is documented, and who does it.

Applying one context's testing to another is a mistake in both directions: a game tested like a pacemaker never ships, and a pacemaker tested like a game should never have shipped.

Principle 7: the absence-of-defects fallacy

The syllabus: "It is a fallacy (i.e., a misconception) to expect that software verification will ensure the success of a system." A system can pass every test of its specification, with every defect found fixed, and still "not fulfill the users' needs and expectations", still fail the customer's business goals, and still be worse than its competitors. Hence: "In addition to verification, validation should also be carried out".

Applied to ExamReg. Suppose the specification described the portal for a desktop browser, and the team built and tested exactly that, flawlessly. If most students fill in their forms on phones, a defect-free ExamReg is still a failure. Verification asked "does it meet the specification?" and got yes; validation, which asks "does it meet the users' need?", was never asked. Verification and validation have a chapter of their own (Chapter Twenty-Six).

The seven, and what each tells a tester to do

What it does not mean

"Exhaustive testing is impossible" does not mean testing is pointless. It means testing must be selective, and selection is a skill Module 2 teaches.

"Defects cluster" does not mean the other modules can be skipped. It sets where the extra effort goes; every module still gets tested.

"Tests wear out" does not mean old tests should be thrown away. Regression suites are rerun on purpose. It means old tests alone are not enough to find new defects.

The principles are not a law of any standard. ISTQB calls them general guidelines. They are widely taught because they are true in practice, not because anyone enforces them.

Quick revision

• ISTQB v4.0.1 section 1.3, seven principles; 2018 names in brackets where they differ.
• 1. Presence, not absence of defects: passing tests cannot prove correctness.
• 2. Exhaustive testing is impossible: two 32-bit inputs give about 1.8 × 10^19 combinations, centuries of testing; use techniques, priority and risk.
• 3. Early testing saves time and money (shift left): static and dynamic testing as early as possible.
• 4. Defects cluster together (Pareto): ExamReg's exam form and fee modules hold 132 of 200 defects, 66 per cent.
• 5. Tests wear out (pesticide paradox): repeated tests stop finding new defects; add and revise tests; repetition is right for regression.
• 6. Testing is context dependent: a therapy machine and a game are not tested alike.
• 7. Absence-of-defects fallacy (absence-of-errors is a fallacy): verified is not the same as useful; validate too.

Test yourself

1. State the seven principles of testing. Testing shows the presence, not the absence of defects; exhaustive testing is impossible; early testing saves time and money; defects cluster together; tests wear out (the pesticide paradox); testing is context dependent; and the absence-of-defects fallacy.

2. Why is exhaustive testing impossible? Give a number. The combinations of inputs and conditions grow multiplicatively. A function of just two 32-bit integers has 2 to the power 64, about 1.8 × 10^19, input pairs, which would take about 585 years at a billion tests a second. So tests are selected with techniques and risk.

3. Explain the pesticide paradox with an example. Tests repeated unchanged stop finding new defects, because they only ask the questions they were written to ask. In the example, a lookup-table rewrite of the late-fee function stored 50 instead of 500 for 12 days; the old suite of five tests still passed, and only a new test at 12 days found the defect.

4. What does "defects cluster together" mean, and how should a test manager use it? A small number of components usually contain most of the defects; in ExamReg two of seven modules held 66 per cent of them. The manager uses past and predicted clusters to direct extra test effort in risk-based testing.

5. What is the absence-of-defects fallacy? The misconception that finding and fixing every defect against the specification ensures the system's success. A system can be defect-free against its specification and still fail its users' needs, so validation must accompany verification.

6. Distinguish principles 1 and 7. Principle 1 is about knowledge: testing cannot prove there are no defects. Principle 7 is about value: even a system truly free of defects against its specification can be the wrong system. The first warns against over-trusting a passed test; the second against over-trusting the specification.

3. Test analysis. Analysing the test basis "to identify testable features", then defining and prioritising test conditions. A test condition is a "testable aspect of a component or system, such as a function, transaction, feature, quality attribute, or structural element" (ISO/IEC/IEEE 29119-2). Analysis also reviews the test basis itself for defects and for testability. The syllabus sums it up: analysis answers "what to test?"

4. Test design. "Elaborating the test conditions into test cases and other testware", identifying coverage items that guide the choice of inputs, defining the test data needed and designing the test environment. Design answers "how to test?". The techniques of Module 2 (equivalence partitioning, boundary values, decision tables and the rest) are used here.

5. Test implementation. Creating or acquiring everything execution will need: test data, manual and automated test scripts, test procedures, which order test cases into steps, and test suites, which group procedures. The procedures are arranged into a test execution schedule, and the test environment is built and checked. ISO/IEC/IEEE 29119-2 defines that environment as the "facilities, hardware, software, firmware, and procedures needed to conduct a test".

6. Test execution. "Running the tests in accordance with the test execution schedule", comparing actual results with expected results, logging the results, and analysing anomalies to find their likely cause, so that failures can be reported. Chapter Nine, on test execution, takes this activity in detail.

7. Test completion. At a milestone such as a release: raising change requests or backlog items for defects still open, archiving testware that will be useful again, returning the test environment to an agreed state, recording lessons learned, and writing the test completion report.

Not a waterfall

The order above is logical, not rigid. The syllabus is explicit: "Although many of these activities may appear to follow a logical sequence, they are often implemented iteratively or in parallel." In an agile team, analysis, design, implementation and execution may all happen inside one two-week cycle, for one user story, and then again for the next. Test planning is revisited when the project changes. What matters is that each activity happens, not that they happen once each in a line.

The testware each activity produces

Two terms in that table deserve a definition now. Entry criteria are the "states of being that have to be present before an effort can begin successfully", and exit criteria those that "have to be present before an effort can end successfully" (ISO/IEC/IEEE 24765:2017). A test level might enter only when the build installs cleanly, and exit only when every high-priority test has passed.

Traceability: the thread through all of it

The syllabus insists on keeping links "between the test basis elements, testware associated with these elements (e.g., test conditions, risks, test cases), test results, and defects." Those links are traceability, and they are what let a test manager answer the questions that matter. Is every requirement covered by a test? Which requirements are affected by a failed test? If a requirement changes, which tests must change?

The program below is a traceability record for ExamReg's fee rules, in its simplest form: each requirement lists its test cases, and each test case has a result from the latest run.

requirements = {                     # requirement -> the test cases that trace to it
    "FEE-1 no late fee on or before the last date": ["TC-01"],
    "FEE-2 Rs 100 for 1 to 7 days late":             ["TC-02", "TC-03"],
    "FEE-3 Rs 500 for 8 to 15 days late":            ["TC-04", "TC-05"],
    "FEE-4 form refused after 15 days":              ["TC-06"],
    "FEE-5 fee shown before payment":                [],
}
results = {"TC-01": "fail", "TC-02": "pass", "TC-03": "pass",
           "TC-04": "pass", "TC-05": "pass", "TC-06": "pass"}

covered = [r for r, tests in requirements.items() if tests]
print(f"requirements with at least one test: {len(covered)} of {len(requirements)}")
for requirement, tests in requirements.items():
    if not tests:
        status = "NOT TESTED"
    elif all(results[t] == "pass" for t in tests):
        status = "all tests pass"
    else:
        status = "FAILING: " + ", ".join(t for t in tests if results[t] != "pass")
    print(f"  {requirement:<46} {status}")

requirements with at least one test: 4 of 5
  FEE-1 no late fee on or before the last date   FAILING: TC-01
  FEE-2 Rs 100 for 1 to 7 days late              all tests pass
  FEE-3 Rs 500 for 8 to 15 days late             all tests pass
  FEE-4 form refused after 15 days               all tests pass
  FEE-5 fee shown before payment                 NOT TESTED

Two findings fall straight out of it, and neither could be seen from a list of test results alone. The failing test is linked to a requirement, so the report can say which rule is broken, not only which test failed. And requirement FEE-5 has no test at all: something the portal must do is simply not being checked. Chapter Forty-One, on validation testing, returns to this with the full traceability matrix.

Worked example: the late-fee rule through the whole process

Follow one small piece of ExamReg, its late-fee rule, through all seven activities.

Test process in context

The same syllabus section adds that the way the process is carried out depends on context: the stakeholders, the team's skills, the business domain and its risks, technical factors, project constraints of time and budget, the organisation, the development life cycle in use, and the tools available. Those factors decide "test strategy, test techniques used, degree of test automation, required level of coverage, level of detail of testware, test reporting". A three-person team building ExamReg for one college and a bank's payments division both follow the seven activities; their test plans differ in length by a factor of fifty.

Who does what: two roles

The syllabus names two roles. The test management role "takes overall responsibility for the test process, test team and leadership of the test activities", and is mainly concerned with planning, monitoring and control, and completion. The testing role "takes overall responsibility for the engineering (technical) aspect of testing", mainly analysis, design, implementation and execution. They are roles, not job titles: a team leader or a development manager can hold the first, and "one person to take on the roles of testing and test management at the same time" is allowed.

The same process in ISO/IEC/IEEE 29119-2

The software testing standard organises the same work into three layers, which is useful when a question uses its vocabulary.

The standard's own definitions make the correspondence clear: the test design and implementation process is the "test process for deriving and specifying test cases and test procedures", and the test execution process executes those procedures "in the prepared test environment" and records the results.

What it does not mean

The process is not strictly sequential. The activities are often iterative and parallel; the order is logical, not a timetable.

Testing does not start at execution. Planning, analysis and design come first, and analysis already finds defects in the test basis before any code runs.

Monitoring and control are not a final step. They run throughout, which is why they are drawn beside the other activities.

Testware is not only test cases. Plans, conditions, data, scripts, environments, logs, reports and lessons learned are all testware.

Quick revision

• Test process: "set of testing activities performed to achieve a test objective" (ISO/IEC/IEEE 29119-2).
• Seven groups (ISTQB v4.0.1): planning; monitoring and control; analysis (what to test); design (how to test); implementation; execution; completion.
• Often iterative or parallel; tailored to context (stakeholders, team, domain, technology, constraints, organisation, SDLC, tools).
• Testware: artefacts produced to plan, design and execute tests; each activity has its own.
• Test basis: information used to design test cases. Test condition: a testable aspect identified as a basis for testing.
• Entry and exit criteria: conditions to begin and to end an effort.
• Traceability links test basis, conditions, cases, results and defects: coverage, impact of change, audit.
• Two roles: test management (planning, monitoring, control, completion) and testing (analysis to execution).

Test yourself

1. List the activities of the basic test process. Test planning, test monitoring and control, test analysis, test design, test implementation, test execution and test completion.

2. Distinguish test analysis from test design. Analysis studies the test basis to decide what to test, producing prioritised test conditions; design turns those conditions into test cases, coverage items, test data requirements and an environment design, deciding how to test.

3. Why are monitoring and control drawn beside the other activities rather than after them? Because they run throughout: progress is checked against the plan continuously, and corrective action, such as moving effort to a module where defects cluster, is taken while the other activities are still going on.

4. Name the testware produced by test implementation. Test procedures, manual and automated test scripts, test suites, test data, the test execution schedule and test environment items such as stubs, drivers and simulators.

5. What does traceability make possible? Give two examples. It shows which requirements are covered by tests and which are not, as with ExamReg's untested requirement FEE-5; and it shows which requirement a failed test affects, so a failure can be reported against the broken rule and the impact of a change can be judged.

6. What happens in test completion? Unresolved defects become change requests or backlog items, reusable testware is archived, the environment is returned to an agreed state, lessons learned are recorded, and a test completion report is written and communicated.

Phase 3: test case development. Test scenarios and test cases are written, with test data and expected results, and reviewed. The requirement traceability matrix is updated so every requirement maps to its test cases. Automated test scripts are written here too.

Phase 4: test environment set-up. The hardware, software, network, browsers, databases, test accounts and permissions are prepared, and the environment is checked, usually with a short smoke test that confirms the build installs and its main functions respond. This phase often runs in parallel with phase 3, and it is often run by a separate team.

Phase 5: test execution. The test cases are run as scheduled, results are compared with expected results and logged, failures are reported as defects with severity and priority, fixed defects are retested, and regression tests are run after fixes. Execution usually takes several test cycles, each on a new build.

Phase 6: test cycle closure. The team checks the exit criteria, prepares the test summary (or completion) report, makes sure every defect is closed or knowingly deferred, archives the testware, returns the environment, and holds a meeting on lessons learned.

Entry criteria, exit criteria and deliverables

The phases are made controllable by their criteria. ISO/IEC/IEEE 24765:2017 defines entry criteria as "states of being that have to be present before an effort can begin successfully" and exit criteria as those that "have to be present before an effort can end successfully". The table gives typical ones; a real test plan states them for its own project.

The six phases against the standard process

Read the table and one difference stands out. The STLC puts requirement analysis before planning, while ISTQB puts planning first and analysis after. In practice both happen together: a plan cannot be made without knowing what the requirements ask for, and the analysis cannot be complete without knowing the plan's scope. The last row matters too. Monitoring and control are not a phase of the STLC, but they run through all six, exactly as in the basic test process of Chapter Five.

Worked example: does execution exit?

ExamReg release 2.0 has 240 planned test cases, 40 of them critical, 80 high priority and 120 medium. The test plan's exit criteria for the execution phase are: every critical test has been run; at least 90 per cent of all planned tests have been run; at least 95 per cent of the tests run have passed; and no critical defect is open. After the first execution cycle the figures are as below, and the program checks each criterion.

planned = {"critical": 40, "high": 80, "medium": 120}
run     = {"critical": 40, "high": 78, "medium": 101}
passed  = {"critical": 38, "high": 72, "medium": 96}
open_defects = {"critical": 1, "major": 3, "minor": 7}

total_planned = sum(planned.values())
total_run = sum(run.values())
total_passed = sum(passed.values())
criteria = [
    ("every critical test run", run["critical"] == planned["critical"]),
    (f"at least 90% of planned tests run ({total_run} of {total_planned}, "
     f"{100 * total_run / total_planned:.1f}%)", total_run >= 0.90 * total_planned),
    (f"at least 95% of tests run passed ({total_passed} of {total_run}, "
     f"{100 * total_passed / total_run:.1f}%)", total_passed >= 0.95 * total_run),
    ("no critical defect open", open_defects["critical"] == 0),
]
for text, met in criteria:
    print(("met     " if met else "NOT MET ") + text)
print("execution phase may close:", all(met for _, met in criteria))

met     every critical test run
met     at least 90% of planned tests run (219 of 240, 91.2%)
NOT MET at least 95% of tests run passed (206 of 219, 94.1%)
NOT MET no critical defect open
execution phase may close: False

The phase does not close. Two criteria are met and two are not: the pass rate is just under the threshold, and one critical defect is still open. Test control now acts: the critical defect is fixed first, the failed tests are rerun on the next build, and a second cycle begins. Nobody argues about whether testing is finished, because the plan said in advance what finished means. That is the whole value of exit criteria.

The STLC inside the SDLC

The STLC is not a separate life cycle running beside development; it runs inside the software development life cycle (SDLC), which Module 1's second row covers from Chapter Thirteen. Its first two phases can begin as soon as requirements exist, long before code, which is principle 3 (early testing) in practice. In the V-model (Chapter Fifteen), each development phase on the left of the V has its test level on the right, and a complete STLC, from analysis to closure, can be run for each test level.

In agile development the six phases compress into each short iteration. Requirement analysis becomes refining a user story and its acceptance criteria, planning becomes the sprint's test tasks, case development and execution happen within the sprint, and closure becomes the sprint review and retrospective. The phases are still there; they are just two weeks long.

What it does not mean

The STLC is not an international standard. It is industry's phase view of the standard test process; ISO/IEC/IEEE 29119-2 and ISTQB define the process it describes.

The STLC does not make a product defect-free. Some descriptions give its objective as a defect-free product; no process can deliver that (principle 1, Chapter Four). Its objective is to find defects and give evidence about quality in a planned, controlled way.

The phases are not strictly sequential. Environment set-up usually overlaps case development, and execution runs in cycles with fixes between them.

The STLC phases are not test levels. Unit, integration, system and acceptance testing are levels (Chapter Thirty-Two); the STLC can be followed within any of them.

Quick revision

• STLC: the phases testing goes through in a project; industry's view of the test process, not an international standard.
• Six phases: requirement analysis, test planning, test case development, test environment set-up, test execution, test cycle closure.
• Each has entry criteria (to begin), exit criteria (to end) and deliverables.
• Deliverables: review comments; test plan; test cases, data, traceability matrix; ready environment and smoke test result; logs, defect reports, execution reports; summary report and lessons learned.
• Maps onto ISTQB's activities and ISO/IEC/IEEE 29119-2's processes; monitoring and control run throughout.
• Worked example: 219 of 240 tests run (91.25 per cent), 206 of them passed (about 94.1 per cent), one critical defect open, so execution does not exit.
• In agile, all six phases happen inside each iteration.

Test yourself

1. What is the software testing life cycle? Name its phases. It is the sequence of phases through which testing proceeds in a project: requirement analysis, test planning, test case development, test environment set-up, test execution and test cycle closure, each with entry criteria, exit criteria and deliverables.

2. Define entry and exit criteria, with one example of each for test execution. Entry criteria are the conditions that must hold before a phase begins, such as approved test cases and an environment that has passed its smoke test. Exit criteria are those that must hold before it ends, such as every critical test run and no critical defect open.

3. What are the deliverables of test cycle closure? The test summary or completion report, confirmation that defects are closed or knowingly deferred, archived testware, the environment returned, and documented lessons learned.

4. How does the STLC relate to the SDLC? It runs inside the SDLC, as its testing thread. Its first phases can begin as soon as requirements exist, before any code, and in the V-model a full testing cycle can be run for each test level.

5. In the worked example, why could the execution phase not close? Two of its four exit criteria were not met: only 94.1 per cent of the tests run had passed against a threshold of 95 per cent, and one critical defect was still open.

6. Is it correct to say the objective of the STLC is a defect-free product? Justify. No. Testing can show the presence of defects but never their absence (principle 1), so no process can guarantee a defect-free product. The STLC's objective is to find defects and give evidence of quality in a planned and controlled way.

Two fields are filled in only when the test runs: the actual result and the status. Everything else is written before, which is what makes the test repeatable. The postcondition is easily forgotten and quietly important: it says what state the test leaves behind, so the next test does not start from a surprise.

The expected result, and where it comes from

The part of a test case that decides pass or fail has a name. ISO/IEC TR 29119-11:2020 defines a test oracle as a "source of information for determining whether a test has passed or failed". For ExamReg's fee test the oracle is the fee table in the requirements. Oracles come from several places, and a tester should know which one each test relies on:

• The specification, as with the fee table.
• An independent calculation, as when Chapter Two, on errors, faults and failures, used Python's own calendar.isleap to judge a leap-year function.
• A previous version of the software, trusted for the features that did not change.
• A comparable product, such as a published calculator for the same rule.
• A person's judgement, for things like readability or layout, where no document can say what "right" is.

The same technical report names the difficulty that sits under all of this, the test oracle problem: the "challenge of determining whether a test has passed or failed for a given set of test inputs and state". When nobody can say what the right answer is, the test cannot be judged, however carefully it was run.

From test condition to test suite

A test case sits in a hierarchy, and the practical's documents use every level of it.

Test scenarios and test cases

In industry, and in the practical of this paper, a test scenario is usually written as one line saying what situation will be tested, and several test cases are then written saying exactly how, each with its own data and expected result. A scenario is quick to write and quick to review; it lets a business analyst check the testers have thought of the right situations before anyone writes a single step.

What makes a good test case

These are principles of practice, each followed because of what goes wrong without it.

1. It is traceable. It names the requirement or risk it checks. Without that link, a failure cannot be reported against a rule and coverage cannot be measured (Chapter Five, on the basic test process).
2. It has one objective. A case that checks the fee, the receipt and the email at once gives one "fail" for three possible causes, and the defect report cannot say which.
3. Its preconditions are explicit. Most "cannot reproduce" arguments between testers and developers are really two different starting states.
4. Its expected result is exact and decided in advance. "The fee is calculated correctly" is not an expected result; "Rs 100" is.
5. It is repeatable and independent. It gives the same result every time on the same build, and does not depend on another test having run first unless its preconditions say so.
6. It includes the unwelcome inputs. Invalid values, empty fields, the maximum, the moment just after a deadline. Most defects live in the cases a developer did not think of; a suite of only valid, typical inputs finds few of them.
7. It is not redundant. Each case should be able to find something the others cannot. Ten cases that all enter 3 days late add effort and no evidence. Module 2's techniques are, in effect, methods for choosing non-redundant cases.
8. Another tester could run it. Clear steps and data, no private knowledge.
9. It is prioritised. ISTQB v4.0.1 section 5.1.5 describes prioritising test cases, most commonly by risk, so that if time runs out the most important ones have already run.

Worked example: a table of test cases, executed

The six late-fee cases of scenario TS-01 are written below as data, one record per case, with the same fields as the table above. A short program then does what a tester does: runs each case, records the actual result, and sets the status. The function under test is the corrected version from Chapter One.

def late_fee(days_late):
    if days_late > 15:
        raise ValueError("form not accepted more than 15 days late")
    if days_late <= 0:
        return 0
    if days_late <= 7:
        return 100
    return 500

test_cases = [   # id, requirement, input (days late), expected result
    ("TC-FEE-01", "FEE-1", 0,  0),
    ("TC-FEE-02", "FEE-2", 1,  100),
    ("TC-FEE-03", "FEE-2", 7,  100),
    ("TC-FEE-04", "FEE-3", 8,  500),
    ("TC-FEE-05", "FEE-3", 15, 500),
    ("TC-FEE-06", "FEE-4", 16, "refused"),
]
print(f"{'ID':<10}{'req':<7}{'input':>6}  {'expected':<9}{'actual':<9}status")
for case_id, requirement, days, expected in test_cases:
    try:
        actual = late_fee(days)
    except ValueError:
        actual = "refused"
    status = "pass" if actual == expected else "FAIL"
    print(f"{case_id:<10}{requirement:<7}{days:>6}  {str(expected):<9}{str(actual):<9}{status}")

ID        req     input  expected actual   status
TC-FEE-01 FEE-1       0  0        0        pass
TC-FEE-02 FEE-2       1  100      100      pass
TC-FEE-03 FEE-2       7  100      100      pass
TC-FEE-04 FEE-3       8  500      500      pass
TC-FEE-05 FEE-3      15  500      500      pass
TC-FEE-06 FEE-4      16  refused  refused  pass

Notice what the records contain and what the program adds. Each record was written before the run: identifier, requirement, input and expected result. The program supplies only the actual result and the status, exactly the two fields left blank in the written test case. Notice also that the test for 16 days has an expected result that is not a number: a refusal is a result too, and a test case that forgot to say what should happen after 15 days would have been incomplete.

What it does not mean

A test scenario is not a test case. A scenario says what situation to test; it cannot be executed or judged until test cases with data and expected results are derived from it.

A test case is not a test script or a procedure. The case says what to check; the procedure says in what order and with what setup to run cases; the script is the document or program that carries the procedure.

More test cases do not mean better testing. Ten redundant cases are worth less than three well-chosen ones; what counts is what the cases can find.

Expected results do not come from running the software. An expected result copied from the program's own output tests nothing: it certifies whatever the program already does, including its defects.

Quick revision

• Test case: "set of preconditions, inputs and expected results, developed to drive the execution of a test item to meet test objectives" (ISO/IEC/IEEE 29119-2:2021).
• Fields: ID, title, requirement traced, priority, preconditions, test data, steps, expected result, actual result, status, postconditions.
• Test oracle: the source that decides pass or fail: specification, independent calculation, previous version, comparable product, human judgement. The oracle problem is not knowing the right answer.
• Hierarchy: test condition, scenario, case, procedure, script, suite.
• Scenario says what situation; case says exactly how, with data and expected result; several cases per scenario.
• A good test case: traceable, one objective, explicit preconditions, exact expected result in advance, repeatable, includes invalid inputs, non-redundant, runnable by others, prioritised.

Test yourself

1. Define a test case and name its essential parts. A set of preconditions, inputs and expected results developed to drive the execution of a test item to meet test objectives. In practice it also carries an identifier, the requirement it traces to, steps, a priority, postconditions, and, after execution, the actual result and a status.

2. Differentiate a test scenario from a test case, with an example. A scenario is a one-line situation to be tested, such as a student submitting the form late and paying the late fee; it cannot be executed as it stands. A test case derived from it is executable and exact, such as a form submitted 7 days late with an expected late fee of Rs 100.

3. What is a test oracle? Give three kinds. The source of information that decides whether a test passed or failed. Examples: the specification, an independent calculation such as a trusted library function, a previous version of the software, a comparable product, or a person's judgement.

4. Why must the expected result be decided before the test is run? Because the test is a comparison, and a result decided after seeing the output simply accepts whatever the program did, defects included.

5. State any five principles of good test case design. Any five of: traceable to a requirement; one objective per case; explicit preconditions; exact expected result decided in advance; repeatable and independent; includes invalid and unexpected inputs; not redundant; clear enough for another tester; prioritised by risk.

6. In what order are these produced during testing: test case, test suite, test condition, test scenario, test procedure? Test condition first (in test analysis), then the test scenario and the test cases derived from it (in test design), then the test procedure that puts cases in execution order with their setup, and the test suite that groups them (both in test implementation).

Family two: white-box, or structure-based

ISO/IEC/IEEE 29119-1 defines structure-based testing as "dynamic testing in which the tests are derived from an examination of the structure of the test item". The box is now transparent: the tester reads the code, finds its statements, decisions and paths, and designs tests to exercise them.

This family has a mirror-image limitation. The syllabus: "As the test cases are dependent on how the software is designed, they can only be created after the design or implementation of the test object." And if the code changes, the tests may need to change with it. The white-box techniques MU names are statement testing (Chapter Fifty-Nine) and branch testing (Chapter Sixty), with structural testing introduced as a whole in Chapter Fifty-Eight.

Family three: experience-based

ISO/IEC/IEEE 29119-4:2021 defines experience-based testing as a "class of test case design techniques based on using the experience of testers to generate test cases". A tester who has seen date fields fail on 29 February, and forms fail when a user double-clicks Submit, tries those things first.

The syllabus is candid about both sides: the effectiveness of these techniques "depends heavily on the tester's skills", and yet they "can detect defects that may be missed using the black-box test techniques and white-box test techniques. Hence, experience-based test techniques are complementary". MU names three: error guessing (Chapter Sixty-Two), exploratory testing (Chapter Sixty-Three) and checklist-based testing (Chapter Sixty-Four).

The three families compared

The "misses" row is the reason all three are used. A black-box tester cannot see a stray line of code that does something nobody asked for. A white-box tester cannot see a requirement the programmer forgot, because there is no code for it to cover. And both follow rules, which is exactly where an experienced tester's instinct for the unexpected earns its place.

Worked example: three families, three defects

Below, ExamReg's late-fee function contains three planted defects of three different kinds. The oracle is the fee table, with one clarification the exam cell gave after the review of the requirement in Chapter One, on what software testing is: a negative number of days, or anything that is not a whole number of days, must be refused cleanly. Each family's tests are chosen the way that family chooses them, and the program reports what each finds.

def late_fee(days_late):
    if days_late > 15:
        raise ValueError("form not accepted more than 15 days late")
    if days_late <= 0:                  # defect C: the input is never validated
        return 0
    if days_late == 13:                 # defect B: a forgotten debugging shortcut
        return 0
    if days_late <= 6:                  # defect A: the rule says 7, not 6
        return 100
    return 500

def expected(days):                     # the oracle: the fee table and the exam cell's ruling
    if not isinstance(days, int) or days < 0 or days > 15:
        return "refused"
    return 0 if days == 0 else (100 if days <= 7 else 500)

def actual(days):
    try:
        return late_fee(days)
    except ValueError:
        return "refused"
    except Exception as problem:        # anything else is a crash, not a clean refusal
        return "crash: " + type(problem).__name__

families = {
    "black-box, from the fee table":    [0, 1, 7, 8, 15, 16],
    "white-box, one test per branch":   [16, 0, 13, 3, 10],
    "experience-based, likely mistakes": [-1, "7", 7.5],
}
for family, inputs in families.items():
    found = [f"{d!r}: expected {expected(d)}, got {actual(d)}"
             for d in inputs if actual(d) != expected(d)]
    print(f"{family}: {len(inputs)} tests, {len(found)} failure(s)")
    for line in found:
        print("   ", line)

black-box, from the fee table: 6 tests, 1 failure(s)
    7: expected 100, got 500
white-box, one test per branch: 5 tests, 1 failure(s)
    13: expected 500, got 0
experience-based, likely mistakes: 3 tests, 3 failure(s)
    -1: expected refused, got 0
    '7': expected refused, got crash: TypeError
    7.5: expected refused, got 500

Each family found what the others could not. The black-box tests were chosen at the edges of the fee table, so 7 days, the boundary the programmer got wrong, was among them; but 13 is an ordinary value in the middle of a band, and no specification-based technique would single it out. The white-box tests were chosen to make every branch of the code run at least once, so the strange == 13 branch was forced to execute and gave itself away; but no branch exists for "exactly 7", so branch testing had no reason to try it. And the experience-based tests tried what users actually do wrong: a negative number, a number typed as text, a fraction. All three failed, and all three are the same defect showing three faces: the function never validates its input, so it charges nothing for minus one day, crashes on text, and charges Rs 500 for seven and a half days. The fee table never mentioned any of these inputs, so no specification-based test would have tried them.

The lesson is the one the ISTQB syllabus gives: the families are complementary. A test plan that uses only one of them has chosen, in advance, which kinds of defect it will not find.

The wider catalogue

MU's syllabus names nine techniques. The software testing standard ISO/IEC/IEEE 29119-4:2021 defines more, and a student meeting their names elsewhere should know which family each belongs to.

Choosing a technique

No technique is best everywhere, which is Chapter Four's sixth principle (testing is context dependent) applied to design. A few rules of thumb from practice:

• Input fields with ranges (marks, days, amounts): equivalence partitioning with boundary value analysis.
• Business rules combining several conditions (fee waivers, eligibility): decision tables.
• Behaviour that depends on history (login attempts, an order's status): state transition testing.
• Safety-critical or complex code: white-box coverage targets, set in the test plan.
• New, poorly specified or rushed features: exploratory testing, guided by checklists.
• Every time: some error guessing, because it costs little and finds what rules do not.

What it does not mean

Black-box does not mean careless or blind. It means the tests are derived from the specification without looking at the code; the design is as systematic as any white-box technique.

White-box testing is not only for developers. Anyone who can read the code can use it, though in practice developers apply it most, at unit level.

Experience-based testing is not random clicking. It is disciplined by charters, checklists and notes, as Module 2 shows; and random testing is a separate, specification-based technique in ISO/IEC/IEEE 29119-4.

100 per cent coverage does not mean no defects remain. It means every coverage item of that technique was exercised. In the worked example, 100 per cent branch coverage missed defect A.

Quick revision

• Test design technique: a "procedure used to create or select a test model, identify test coverage items, and derive corresponding test cases" (ISO/IEC/IEEE 29119-2).
• Test coverage: the percentage of specified coverage items exercised by the tests.
• Black-box (specification-based): from the specification; tests survive code changes; misses code nobody specified. MU: equivalence partitioning, boundary value analysis, decision tables, state transitions.
• White-box (structure-based): from the code; needs the code; misses behaviour never coded. MU: statement and branch testing.
• Experience-based: from the tester's experience; complementary to both. MU: error guessing, exploratory, checklist-based.
• Worked example: each family found exactly one kind of planted defect the others missed.

Test yourself

1. What is a test design technique, and why are techniques used? A procedure for identifying test coverage items and deriving test cases from a test basis. They are used because exhaustive testing is impossible, and they produce a small but sufficient set of tests systematically, so the result is repeatable and its coverage can be measured.

2. Name the three families of techniques and give MU's examples of each. Black-box or specification-based: equivalence partitioning, boundary value analysis, decision table testing and state transition testing. White-box or structure-based: statement testing and branch testing. Experience-based: error guessing, exploratory testing and checklist-based testing.

3. Why can white-box tests not be written at the start of a project? Because they are derived from the internal structure of the code or detailed design, which does not exist until design or implementation has been done.

4. Give one kind of defect each family is good at finding and one it misses. Black-box finds wrong or missing behaviour against the specification but misses code nobody specified; white-box finds untested or unexpected code but misses required behaviour that was never coded; experience-based finds unusual real-world inputs but misses whatever the tester has never encountered.

5. In the worked example, which family found the forgotten debugging shortcut, and why could the others not? White-box branch testing, because it had to execute every branch, including the one for 13 days. Black-box tests had no reason to choose 13, an ordinary value in the middle of a band, and experience-based tests looked at invalid inputs instead.

6. Does 100 per cent branch coverage guarantee a correct program? No. It guarantees each branch ran at least once. In the worked example the branch tests achieved full branch coverage and still missed the wrong boundary at 7 days, because no branch exists for a value the code handles wrongly within a range.

Then comes the verdict. ISO/IEC/IEEE 29119-2 defines a test result as the "indication of whether a specific test case has passed or failed, i.e. if the actual results correspond to the expected results or if deviations were observed". In practice four states are recorded:

A blocked test is neither a pass nor a fail, and it must never be quietly counted as either. A report that shows 95 per cent passing because the blocked tests were left out of the total is misleading.

When the result is unexpected: anomalies and incidents

A failed comparison is not yet a defect in the software. The ISTQB syllabus describes what happens next: "Anomalies are analyzed to identify their likely causes. This analysis allows us to report the anomalies based on the failures observed." A failure can come from the software, from the test itself (a wrong expected result, the false positive of Chapter Two on errors, faults and failures), from the environment (a misconfigured server) or from the test data.

ISO/IEC/IEEE 29119-3:2021 calls the event a test incident: an "event occurring during the execution of a test that requires investigation". The tester records it in an incident report, "documentation of the occurrence, nature, and status of an incident" (ISO/IEC/IEEE 29119-2), which becomes a defect report once the cause is confirmed to be a defect. Writing a defect report has a chapter of its own in Module 2 (Chapter Seventy-Six).

The test log

Everything that happens during execution goes into the test log, which ISO/IEC/IEEE 24765:2017 defines as a "chronological record of relevant details about the execution of tests". ISO/IEC/IEEE 29119-3 calls the document a test execution log, one that "records details of the execution of one or more test procedures". A useful log entry answers: which test, on which build, in which environment, with which data, when, by whom, with what result, and where the evidence is.

Evidence: logs and screenshots

The paired practical sets an exercise called "Screenshot Capture and Logging Mechanism", in which an automated test takes a screenshot when it fails and writes its progress to a log. The idea behind it is older than any tool. A failure that cannot be shown did not, for practical purposes, happen: the developer who receives the report will try to reproduce it, and if they cannot, the report is closed. A screenshot shows what the user saw at the moment of failure. A log shows what the program did on the way there. Together they turn "it went wrong" into evidence.

Logs are written at levels, so that a quiet run and a detailed investigation can use the same code. Apache Log4j 2's manual describes levels "from debug to fatal"; Python's standard logging module has DEBUG, INFO, WARNING, ERROR and CRITICAL. A test run typically logs each test's start at INFO, each failure at ERROR with its evidence, and detailed steps at DEBUG, switched on only when chasing a problem.

One currency point for the practical. Apache's own page for Log4j 1 records that "On August 5, 2015 the Logging Services Project Management Committee announced that Log4j 1.x had reached end of life", and it lists vulnerabilities in Log4j 1 of which it says "none of the issues listed will be fixed". A practical written today should use Log4j 2, as Apache recommends.

Worked example: an execution run with its log and evidence

The program below runs four ExamReg fee tests against release 2.0's function, which still has the on-time defect found in Chapter One, on what software testing is. It logs every step with Python's logging module, marks a test blocked when its precondition fails (the payment sandbox is down), and on a failure saves a text snapshot of what the fee page showed, the stand-in here for a screenshot. The log format leaves out the time so that the output is the same on every run; a real log starts each line with a timestamp.

import logging, os, sys

logging.basicConfig(stream=sys.stdout, level=logging.INFO,
                    format="%(levelname)-7s %(message)s")
log = logging.getLogger("examreg-tests")

def late_fee(days_late):                     # release 2.0, with the on-time defect
    if days_late > 15:
        raise ValueError("form not accepted more than 15 days late")
    return 100 if days_late <= 7 else 500

def fee_page(days_late):                     # what the student would see
    return f"ExamReg fee page\nLate fee: Rs {late_fee(days_late)}\n"

sandbox_up = False                           # the payment gateway's test sandbox
tests = [("TC-FEE-01", 0, 0, None), ("TC-FEE-03", 7, 100, None),
         ("TC-FEE-04", 8, 500, None), ("TC-PAY-21", 3, 100, "sandbox")]
os.makedirs("evidence", exist_ok=True)
tally = {"pass": 0, "fail": 0, "blocked": 0}
log.info("build 2.0.1, environment TEST-2, cycle 1")
for case_id, days, expected, needs in tests:
    if needs == "sandbox" and not sandbox_up:
        log.warning("%s blocked: payment sandbox unreachable", case_id)
        tally["blocked"] += 1
        continue
    actual = late_fee(days)
    if actual == expected:
        log.info("%s pass: %d days late, fee Rs %d", case_id, days, actual)
        tally["pass"] += 1
    else:
        evidence = os.path.join("evidence", case_id + ".txt")
        with open(evidence, "w") as page:
            page.write(fee_page(days))
        log.error("%s FAIL: %d days late, expected Rs %d, got Rs %d; evidence %s",
                  case_id, days, expected, actual, evidence)
        tally["fail"] += 1
log.info("cycle 1 totals: %s", tally)
print(open("evidence/TC-FEE-01.txt").read(), end="")

INFO    build 2.0.1, environment TEST-2, cycle 1
ERROR   TC-FEE-01 FAIL: 0 days late, expected Rs 0, got Rs 100; evidence evidence/TC-FEE-01.txt
INFO    TC-FEE-03 pass: 7 days late, fee Rs 100
INFO    TC-FEE-04 pass: 8 days late, fee Rs 500
WARNING TC-PAY-21 blocked: payment sandbox unreachable
INFO    cycle 1 totals: {'pass': 2, 'fail': 1, 'blocked': 1}
ExamReg fee page
Late fee: Rs 100

Read the log as a developer would. The build and environment come first, so there is no argument about which version was tested. The failure names the test, the input, the expected and actual results and where the evidence is, and the evidence file shows exactly what a student would have seen. The blocked test is reported as blocked, with its reason, and counted separately: it is a problem to fix in the environment, not a defect in the fee rule.

After the fix: retesting and regression testing

When the developer fixes the defect and delivers a new build, two different things are tested, and they have different names.

Retesting, also called confirmation testing, is "testing performed to check that modifications made to correct a fault have successfully removed the fault" (ISO/IEC/IEEE 29119-2). Here: run TC-FEE-01 again and see Rs 0.

Regression testing is "testing performed following modifications to a test item or to its operational environment, to identify whether failures in unmodified parts of the test item occur" (ISO/IEC/IEEE 29119-1). Here: rerun the other fee tests, and the form and payment tests near the changed code, to show the fix broke nothing else.

Stopping and restarting: suspension and resumption

Sometimes testing must stop. ISO/IEC/IEEE 29119-1 defines suspension criteria as "criteria used to (temporarily) stop all or a portion of the testing activities". A test plan states them in advance, together with the resumption requirements that must be met to restart. For ExamReg, for example: suspend if the build fails its smoke test, or if more than a quarter of the scheduled tests are blocked; resume when a build passes the smoke test and the blocking problem is fixed. Deciding this in advance stops testers wasting days on a build that was never going to be usable, and stops a manager pushing testing on regardless.

What it does not mean

A failed test is not automatically a defect in the software. The test, the data or the environment may be wrong; anomalies are analysed first.

Blocked is not failed, and not passed. It means the test could not run, and it is counted separately.

Retesting and regression testing are not the same. One checks the fix; the other checks everything else around it.

Execution is not only for automated tests. Manual execution follows the same rules: exact procedures, recorded actual results, a log and evidence.

Quick revision

• Test execution: "process of running a test on the test item, producing actual results" (ISO/IEC/IEEE 29119-2).
• Before it: entry criteria, environment and data readiness, a passing smoke test.
• Actual results include data and environment, not only the screen. Statuses: pass, fail, blocked, not run.
• Anomalies are analysed: the cause may be the software, the test, the environment or the data. Test incident: an event during execution that needs investigation; recorded in an incident report.
• Test log: "chronological record of relevant details about the execution of tests" (ISO/IEC/IEEE 24765).
• Evidence: logs (by level) and screenshots on failure. Log4j 1 reached end of life on 5 August 2015; use Log4j 2.
• Retesting (confirmation): checks a fix. Regression testing: checks unmodified parts after a change.
• Suspension criteria stop testing; resumption requirements restart it.

Test yourself

1. What is test execution? List its main tasks. Running tests on the test item to produce actual results: executing procedures as scheduled, comparing actual with expected results, logging outcomes, analysing anomalies and reporting incidents, then retesting fixes and running regression tests.

2. Distinguish retesting from regression testing. Retesting reruns the test that failed, to confirm a fix removed the fault. Regression testing reruns tests that previously passed, after any change, to check that failures have not appeared in unmodified parts.

3. What does "blocked" mean as a test status, and why must it be reported separately? The test could not be run because a precondition could not be met, such as a gateway sandbox being down. It is neither a pass nor a fail, and hiding blocked tests in either count distorts the picture of quality.

4. Why is evidence such as a log or a screenshot attached to a failure? So the failure can be shown and reproduced: it records what the user saw and what the program did, with the build, environment and data, which lets the developer find the defect instead of arguing about whether it exists.

5. What are suspension criteria? Give one example. Criteria, decided in advance, for temporarily stopping all or part of testing; for example, suspending when a build fails its smoke test, and resuming when a build passes it.

6. Which Log4j should a practical use today, and why? Log4j 2. Apache announced on 5 August 2015 that Log4j 1.x had reached end of life, and the vulnerabilities found in it since will not be fixed.

What the reports measure

The ISTQB syllabus lists the common test metrics, and a report draws on several of them:

• Project progress: task completion, resource usage, test effort.
• Test progress: test cases implemented; environment readiness; test cases run and not run, passed and failed; execution time.
• Product quality: availability, response time, mean time to failure.
• Defects: number and priorities found and fixed, defect density, defect detection percentage.
• Risk: the residual risk level.
• Coverage: requirements coverage, code coverage.
• Cost: the cost of testing, the organisation's cost of quality.

Most of these have chapters of their own in Module 2, where metrics, defect metrics and quality costs are rows of the syllabus. A report uses them; it does not invent new ones.

The test execution report

The paired practical asks students to prepare a test execution report. It is the name industry uses for a test progress report focused on execution: after a day or a cycle of running tests, it says what was run, what passed, what failed, what was blocked, which defects are open, and what that means. Its usual sections are:

1. Identification: project, release and build, test environment, cycle, reporting period, author.
2. Execution summary: planned, executed, passed, failed, blocked, not run, as numbers and percentages.
3. By module or feature: the same figures broken down, so problems can be located.
4. Defects: open defects by severity, new and closed in the period.
5. Impediments: what blocked testing and what is being done about it.
6. Risks and recommendation: what the figures mean for the release, measured against the exit criteria.

Worked example: ExamReg, cycle 1, day 3

ExamReg release 2.0 has 240 planned test cases across five modules. At the end of day 3 of the first execution cycle, the counts per module are as follows, and the program produces the execution summary from them. Computing it, rather than typing it, is not a formality: percentages typed by hand are one of the commonest errors in real reports.

modules = {        # planned, passed, failed, blocked (not run is what remains)
    "Login":         (30, 29, 1, 0),
    "Exam form":     (80, 61, 9, 0),
    "Fee payment":   (60, 42, 5, 8),
    "Hall ticket":   (40, 12, 1, 0),
    "Admin reports": (30, 0, 0, 0),
}
open_defects = {"critical": 1, "major": 5, "minor": 6, "cosmetic": 3}

def pct(part, whole):
    return f"{100 * part / whole:5.1f}%" if whole else "    -"

print("ExamReg release 2.0, build 2.0.1, environment TEST-2, cycle 1, day 3")
print(f"{'module':<14}{'planned':>8}{'run':>5}{'pass':>6}{'fail':>6}{'blocked':>9}{'not run':>9}{'pass rate':>11}")
totals = [0, 0, 0, 0, 0, 0]
for name, (planned, passed, failed, blocked) in modules.items():
    run = passed + failed
    not_run = planned - run - blocked
    row = [planned, run, passed, failed, blocked, not_run]
    totals = [t + r for t, r in zip(totals, row)]
    print(f"{name:<14}{planned:>8}{run:>5}{passed:>6}{failed:>6}{blocked:>9}{not_run:>9}{pct(passed, run):>11}")
planned, run, passed, failed, blocked, not_run = totals
print(f"{'TOTAL':<14}{planned:>8}{run:>5}{passed:>6}{failed:>6}{blocked:>9}{not_run:>9}{pct(passed, run):>11}")
print()
print("executed", run, "of", planned, "planned:", pct(run, planned).strip())
print("passed", passed, "of", planned, "planned:", pct(passed, planned).strip())
print("open defects:", ", ".join(f"{n} {s}" for s, n in open_defects.items()))

ExamReg release 2.0, build 2.0.1, environment TEST-2, cycle 1, day 3
module         planned  run  pass  fail  blocked  not run  pass rate
Login               30   30    29     1        0        0      96.7%
Exam form           80   70    61     9        0       10      87.1%
Fee payment         60   47    42     5        8        5      89.4%
Hall ticket         40   13    12     1        0       27      92.3%
Admin reports       30    0     0     0        0       30          -
TOTAL              240  160   144    16        8       72      90.0%

executed 160 of 240 planned: 66.7%
passed 144 of 240 planned: 60.0%
open defects: 1 critical, 5 major, 6 minor, 3 cosmetic

The report then says, in words, what the numbers mean:

• Progress. Two thirds of the planned tests have run in three of the five days scheduled. Admin reports have not started; the team is behind schedule there and ahead on login.
• Quality. Nine in ten executed tests pass, but the failures are concentrated: the exam form has 9 of the 16, which matches the defect clustering seen in Chapter Four's principles of testing.
• Impediments. Eight fee-payment tests are blocked because the gateway's test sandbox is down; the vendor has promised it for tomorrow. They are not counted as passed.
• Risk and recommendation. One critical defect is open (on-time students charged a late fee). The exit criteria set in Chapter Six, on the software testing life cycle, cannot be met while it is open, so the recommendation is: fix it first, and move one tester from login, which is finished, to admin reports.

Notice the two different percentages at the end. Passed of executed is 90 per cent and sounds good. Passed of planned is 60 per cent, because 80 tests have not yet run or cannot run. Both are true, and a report that gave only the first would mislead the release manager into thinking the product was nearly ready. The honest report gives both.

Audience, formality and channel

The ISTQB syllabus notes that "Different audiences require different information in the reports and influence the degree of formality and the frequency of test reporting." The developers want the list of failures with evidence; the project manager wants progress against schedule and the blockers; senior management wants the risk and the recommendation, in a sentence.

The channel changes with the audience too. The syllabus lists verbal communication, dashboards (for example continuous integration dashboards, task boards and burn-down charts), email and chat, online documentation, and formal test reports. A team in one room talks; a team spread across offices and time zones writes more down.

Principles of an honest report

1. Numbers come from the data, not from memory. Counts and percentages are computed from the test management records.
2. Blocked and not-run tests are shown. Leaving them out inflates the pass rate.
3. Every percentage names its base. "90 per cent" of what: of the tests run, or of the tests planned?
4. Facts are separated from judgement. The figures first, then the tester's interpretation, clearly labelled as a recommendation.
5. The report is measured against the plan. Exit criteria and the schedule are the yardstick; without them "good" and "bad" have no meaning.
6. Risks are stated, not softened. An open critical defect is written as what it is.

What it does not mean

A test report is not a list of every test. It summarises; the logs and the test management tool hold the detail.

A high pass rate does not mean a product is ready. It depends on how many tests were run, which ones failed, and whether the exit criteria are met.

The completion report is not the last progress report. It evaluates the whole activity against the plan, records deviations and lessons learned, and is written once.

Reporting is not the tester's opinion. It is evidence first; the recommendation that follows must be traceable to that evidence.

Quick revision

• Test reporting: summarising and communicating test information during and after testing.
• Test status (progress) report: regular, during testing, supports control. Test completion report: once, at the end of a level, cycle or project; evaluates against the plan (formerly the IEEE 829 test summary report).
• Progress report contents: period; progress and deviations; impediments; metrics; risks; next period's plan.
• Completion report contents: summary; evaluation against objectives and exit criteria; deviations; impediments; metrics; unmitigated risks and unfixed defects; lessons learned.
• Metric families (ISTQB): project progress, test progress, product quality, defects, risk, coverage, cost.
• Test execution report (the practical): identification, execution summary, by module, defects, impediments, recommendation.
• Worked example: 160 of 240 run, 144 passed: 90 per cent of executed, 60 per cent of planned; 8 blocked; one critical defect open.

Test yourself

1. Distinguish a test progress report from a test completion report. A progress (status) report is produced regularly during testing to support test control, covering progress, deviations, impediments, metrics, risks and the next period's plan. A completion report is produced once when a level, cycle or project ends, and evaluates the testing and the product against the plan's objectives and exit criteria, with deviations and lessons learned.

2. List the contents of a test execution report. Identification (release, build, environment, cycle, period); an execution summary (planned, executed, passed, failed, blocked, not run); a breakdown by module; open, new and closed defects by severity; impediments; and risks with a recommendation against the exit criteria.

3. Why must a report give both "passed of executed" and "passed of planned"? Because they answer different questions. In the worked example 90 per cent of executed tests passed, but only 60 per cent of planned tests have passed so far; quoting only the first hides the 80 tests not yet run or blocked.

4. Name four families of test metrics. Any four of: project progress, test progress, product quality, defect, risk, coverage and cost metrics.

5. How does audience change a test report? Developers need the failures and their evidence; project managers need progress, blockers and schedule; senior management needs the risk and the recommendation. The audience also sets the formality and frequency, from a daily conversation to a formal completion report.

Heading 5 surprises students most. Writing down what will not be tested is not an admission of failure; it is the point at which a manager can object. If the plan says load testing of the payment gateway is not in scope because the vendor certifies it, the principal can either accept that risk or insist otherwise, before the deadline rush rather than during it.

The same content, as ISTQB describes a plan today

The ISTQB syllabus (2024) lists the typical content of a test plan more compactly: the context of testing (scope, objectives, test basis); assumptions and constraints; stakeholders (roles, responsibilities, hiring and training needs); communication (forms, frequency, templates); a risk register of product and project risks; the test approach (levels, types, techniques, deliverables, entry and exit criteria, independence of testing, metrics, test data and environment requirements, deviations from policy and strategy); and budget and schedule. The two lists cover the same ground:

Entry and exit criteria in the plan

The plan is where entry and exit criteria are set, for each test level. The ISTQB syllabus gives typical ones. Entry criteria concern the availability of resources (people, tools, environments, test data, budget, time), of testware (test basis, testable requirements, test cases), and the initial quality of the test object, such as "all smoke tests have passed". Exit criteria are either measures of thoroughness (coverage achieved, unresolved defects, defect density, failed test cases) or yes/no criteria (planned tests executed, static testing performed, all defects reported, regression tests automated).

The syllabus adds a realistic note: "Running out of time or budget can also be viewed as valid exit criteria", provided "the stakeholders have reviewed and accepted the risk to go live without further testing." And in agile teams the same ideas have other names: exit criteria are the Definition of Done, and the entry criteria a user story must meet before work starts are the Definition of Ready.

How much effort? Estimation in the plan

The schedule and budget depend on an estimate of test effort. The ISTQB syllabus describes four techniques. Estimation based on ratios uses figures from past projects, such as a test-to-development effort ratio. Extrapolation measures the current project early and projects forward, for example averaging the last three iterations. Wideband Delphi has experts estimate separately, discuss the outliers and re-estimate until they agree; Planning Poker is its agile variant. Three-point estimation asks for an optimistic estimate a, a most likely m and a pessimistic b, and combines them as E = (a + 4m + b) / 6, with a spread SD = (b - a) / 6.

The program applies three-point estimation to four of ExamReg's test tasks, in person-hours.

tasks = {                                   # optimistic, most likely, pessimistic
    "design fee and form tests":    (16, 24, 44),
    "set up test environment":      (6, 8, 16),
    "execute cycle 1":              (40, 56, 90),
    "execute regression cycle":     (12, 16, 26),
}
total = 0
for task, (a, m, b) in tasks.items():
    estimate = (a + 4 * m + b) / 6
    spread = (b - a) / 6
    total += estimate
    print(f"{task:<28} E = {estimate:5.1f}  SD = {spread:4.1f}  "
          f"(likely {estimate - spread:.1f} to {estimate + spread:.1f})")
print(f"{'total expected effort':<28} E = {total:5.1f} person-hours")

design fee and form tests    E =  26.0  SD =  4.7  (likely 21.3 to 30.7)
set up test environment      E =   9.0  SD =  1.7  (likely 7.3 to 10.7)
execute cycle 1              E =  59.0  SD =  8.3  (likely 50.7 to 67.3)
execute regression cycle     E =  17.0  SD =  2.3  (likely 14.7 to 19.3)
total expected effort        E = 111.0 person-hours

Notice that each estimate E is larger than the most likely value m. The pessimistic figure pulls it up, because in testing, as in most work, things tend to take longer rather than shorter than hoped: a blocked environment or a defect-heavy build costs far more time than a smooth one saves. A plan that used m alone would be optimistic by construction.

Worked example: a test plan for ExamReg release 2.0

Below is a complete plan in the IEEE 829-1998 outline, the form the paired practical asks for. It is short because ExamReg is small; a bank's plan under the same headings runs to fifty pages.

1. Test plan identifier. STP-EXAMREG-2.0, version 1.1.

2. Introduction. This plan covers system testing and user acceptance testing of ExamReg release 2.0, the college's online examination registration portal, before the examination form opens for the November session. Objectives: show that fees are charged exactly as the fee rules say, that eligible students can register and ineligible ones cannot, and that the portal handles the last-day rush. References: ExamReg requirements v2.0, fee rules circular, test policy TP-01.

3. Test items. ExamReg web application build 2.0.x; the fee calculation service; the hall ticket generator; the payment gateway adapter (the gateway itself is the vendor's).

4. Features to be tested. Login and lockout; exam form, including backlog papers and eligibility; fee calculation, including late fees and concessions; payment and receipts; hall ticket generation; admin reports.

5. Features not to be tested. The payment gateway's internal processing (certified by the vendor; only our adapter is tested); the college's existing student database (read only, unchanged in this release); translations (English only in 2.0).

6. Approach. Risk-based. Specification-based techniques (equivalence partitioning, boundary values, decision tables for fee rules, state transitions for login); automated regression of fee and form tests on every build; a load test of 500 simultaneous users before release; exploratory sessions on the exam form. Independent testing by the college IT cell's test team.

7. Item pass/fail criteria. An item passes when all its high-priority tests pass and it has no open critical or major defect.

8. Suspension criteria and resumption requirements. Suspend if a build fails the smoke test or more than 25 per cent of scheduled tests are blocked. Resume on a build that passes the smoke test with the blocking cause fixed.

9. Test deliverables. This plan; test scenarios and cases; traceability matrix; test data; execution reports for each cycle; defect reports; load test report; test completion report.

10. Testing tasks. Requirement review; test design; environment set-up; cycle 1; defect fixing and retesting; regression cycle; load test; user acceptance testing; completion report. Test design depends on the approved requirements; execution depends on build delivery.

11. Environmental needs. Test server TEST-2 matching production; a controllable server clock; the payment gateway's sandbox; six test student accounts with prepared papers; Chrome, Firefox and Edge on desktop and on Android.

12. Responsibilities. Test lead: plan, monitor, report. Two testers: design and execution. Developers: fixes and unit tests. IT cell: environments. Exam cell: acceptance testing and answers to requirement questions.

13. Staffing and training needs. Two testers and a test lead for four weeks; one tester trained on the load testing tool.

14. Schedule. Test design weeks 1 and 2; environment ready end of week 1; cycle 1 week 3; regression and load test week 4; acceptance testing and sign-off at the end of week 4, one week before the form opens.

15. Risks and contingencies. Late build: reduce cycle 1 scope to high-priority tests. Gateway sandbox unavailable: test the adapter against a simulator and retest when the sandbox returns. Requirement changes from the exam cell: frozen after week 1 except for defects.

16. Approvals. Test lead; development lead; head of the exam cell; principal. Names, signatures and dates.

What it does not mean

A test plan is not a list of test cases. It plans the testing; test cases are designed later, in test design, and live in their own documents.

IEEE 829-1998 is not the current standard. It was superseded by IEEE 829-2008 and then by ISO/IEC/IEEE 29119-3; its sixteen headings survive because they make a good checklist.

"Features not to be tested" is not a weakness. It records a decision and its reason, so the people who carry the risk can accept or reject it.

A plan is not written once and frozen. It is revised when the project changes, and test control acts on it throughout.

Quick revision

• Test plan: objectives, means and schedule for testing a test item or items (ISO/IEC/IEEE 29119-2); scope, approach, resources and schedule (IEEE 1012-2024).
• Hierarchy: test policy (organisation) above test strategy (project, level or type) above the test plan; the test approach is chosen within the strategy.
• IEEE 829-1998's 16 headings: identifier; introduction; test items; features to be tested; features not to be tested; approach; item pass/fail criteria; suspension and resumption; deliverables; testing tasks; environmental needs; responsibilities; staffing and training; schedule; risks and contingencies; approvals.
• ISTQB v4.0.1 content: context, assumptions and constraints, stakeholders, communication, risk register, test approach, budget and schedule.
• Entry and exit criteria per level; in agile, Definition of Ready and Definition of Done.
• Estimation: ratios, extrapolation, Wideband Delphi, three-point (E = (a + 4m + b) / 6, SD = (b - a) / 6).

Test yourself

1. Define a test plan and state its purposes. A document describing the test objectives and the means, resources and schedule for achieving them. It documents how the objectives will be met, helps ensure the test activities meet their criteria, communicates with the team and stakeholders, and shows that testing follows the test policy and strategy.

2. List the sixteen headings of the IEEE 829-1998 test plan. Test plan identifier; introduction; test items; features to be tested; features not to be tested; approach; item pass/fail criteria; suspension criteria and resumption requirements; test deliverables; testing tasks; environmental needs; responsibilities; staffing and training needs; schedule; risks and contingencies; approvals.

3. Distinguish test policy, test strategy and test plan. The policy is an executive document stating the purpose, goals and principles of testing across the organisation. The strategy describes the approach to testing for a particular project, level or type. The plan sets the objectives, means, resources and schedule for testing specific items.

4. Give typical entry and exit criteria for system testing. Entry: the build has passed its smoke test, the environment and test data are ready, and the test cases are approved. Exit: all high-priority tests executed and passed, requirements coverage complete, and no open critical defect.

5. Using three-point estimation, estimate a task with a = 6, m = 9 and b = 18 person-hours. E = (6 + 4 × 9 + 18) / 6 = 10 person-hours, with SD = (18 - 6) / 6 = 2, so the task is likely to take between 8 and 12 person-hours. This is the worked example in the ISTQB syllabus.

6. Why should a test plan say which features will not be tested? Because excluding a feature is a risk decision, and recording it with its reason lets the stakeholders who carry the risk accept or reject it before testing, not discover it after release.

Two points about the set are worth making in an answer. First, the specification is split three ways on purpose: the design says what to cover, the case says with what values, the procedure says in what steps. Keeping them apart lets one test case be reused in several designs and one procedure run many cases. Second, the reporting documents follow execution in time: the transmittal report comes before it, the log during it, the incident reports as problems appear, and the summary after.

IEEE 829-2008 and ISO/IEC/IEEE 29119-3

The 2008 revision renamed and extended the set around test levels: a Master Test Plan over all levels and a Level Test Plan for each; Level Test Design, Level Test Case and Level Test Procedure documents; a Level Test Log; an Anomaly Report; a Level Interim Test Status Report; a Level Test Report; and a Master Test Report. The name anomaly report was chosen deliberately, as Wikipedia's account of the standard explains, because a discrepancy between expected and actual results can have causes other than a fault in the system, such as a wrong expected result.

IEEE 829-2008 has in turn been superseded by ISO/IEC/IEEE 29119-3, whose current edition, the 2021 one, is the standard SEVOCAB draws its test documentation terms from. The names map as follows.

The newer standard adds documents for things 1998 left implicit: the status report during testing (Chapter Ten, on test reporting), the readiness of the environment and the data (Chapter Nine, on test execution), and the traceability matrix.

Must every document be produced?

No. Wikipedia's account of IEEE 829-2008 notes that the standard "specified the format of these documents, but did not stipulate whether they must all be produced". Which documents a project writes is decided in its test plan, and the answer depends on context: a medical device project may produce every one, formally reviewed and signed; a three-person team building ExamReg may keep test cases and logs in a test management tool and write only a plan and a completion report as documents.

Agile teams go further. The Agile Manifesto values "Working software over comprehensive documentation", and it adds that "while there is value in the items on the right, we value the items on the left more". In agile testing the same information is still kept, but lighter: acceptance criteria on user stories instead of design specifications, automated tests that are themselves the test cases, and the continuous integration server's history instead of a hand-written log.

Traceability across the documents

The documents are only as useful as the links between them. A requirement should lead forward to its test cases, their procedures, their log entries and any incidents; and an incident should lead back to the requirement it affects. The program below records those links for one ExamReg requirement across two execution cycles, then follows them in both directions.

requirement = {"FEE-1": "no late fee on or before the last date"}
test_cases = {"TC-FEE-01": {"requirement": "FEE-1", "procedure": "TP-FEE"}}
log = [   # test log entries: cycle, test case, build, result, incident raised
    {"cycle": 1, "case": "TC-FEE-01", "build": "2.0.1", "result": "fail", "incident": "IR-014"},
    {"cycle": 2, "case": "TC-FEE-01", "build": "2.0.2", "result": "pass", "incident": None},
]
incidents = {"IR-014": {"case": "TC-FEE-01", "status": "closed after retest on 2.0.2"}}

print("forward, from the requirement:")
for rid, text in requirement.items():
    print(f"  {rid}: {text}")
    for case, info in test_cases.items():
        if info["requirement"] == rid:
            print(f"    test case {case}, run by procedure {info['procedure']}")
            for entry in log:
                if entry["case"] == case:
                    note = f", incident {entry['incident']}" if entry["incident"] else ""
                    print(f"      cycle {entry['cycle']} on build {entry['build']}: {entry['result']}{note}")

print("backward, from the incident:")
case = incidents["IR-014"]["case"]
rid = test_cases[case]["requirement"]
print(f"  IR-014 ({incidents['IR-014']['status']}) <- {case} <- {rid}: {requirement[rid]}")

forward, from the requirement:
  FEE-1: no late fee on or before the last date
    test case TC-FEE-01, run by procedure TP-FEE
      cycle 1 on build 2.0.1: fail, incident IR-014
      cycle 2 on build 2.0.2: pass
backward, from the incident:
  IR-014 (closed after retest on 2.0.2) <- TC-FEE-01 <- FEE-1: no late fee on or before the last date

Forward, the chain answers "was this requirement tested, and what happened?": it failed on build 2.0.1, an incident was raised, and it passed on 2.0.2. Backward, it answers "which rule does this incident break?" in one step. Without the links in the documents, both questions need a person's memory.

What it does not mean

IEEE 829 is not the current standard. The 1998 version was superseded in 2008, and 829-2008 by ISO/IEC/IEEE 29119-3. Its eight documents are still taught because they divide the work cleanly.

Not every document must be written. The standards define formats; the test plan decides which documents a project produces.

A test case specification is not a test design specification. The design says which features are covered and how they pass; the case gives the actual values and expected outputs.

Less documentation in agile does not mean less information. The same facts are kept in lighter forms: acceptance criteria, automated tests, tool histories.

Quick revision

• IEEE 829-1998: eight documents in three groups.
• Planning: test plan.
• Specification: test design specification (what to cover, feature pass/fail criteria); test case specification (input values and expected outputs); test procedure specification (steps to run cases).
• Reporting: test item transmittal report (items handed over); test log (what happened); test incident report (events to investigate); test summary report (summary and evaluation).
• IEEE 829-2008: master and level test plans, level design, case, procedure, log, anomaly report, interim status report, level and master test reports.
• ISO/IEC/IEEE 29119-3 supersedes IEEE 829: test status report, test completion report, test execution log, incident report, readiness reports, test traceability matrix.
• Not every document is mandatory; agile keeps the same information more lightly.

Test yourself

1. Name the eight documents of IEEE 829-1998 in their groups. Planning: the test plan. Specification: the test design specification, the test case specification and the test procedure specification. Reporting: the test item transmittal report, the test log, the test incident report and the test summary report.

2. Distinguish the test design, test case and test procedure specifications. The design specification says which features are covered, which cases and procedures are needed, and the pass/fail criteria per feature. The case specification gives the actual input values and expected outputs of each case. The procedure specification lists the steps to run the cases, in order.

3. What is a test item transmittal report, and when is it used? A document identifying the items being handed over for testing, used when a separate development group delivers builds to a test group, or when a formal start of test execution is wanted.

4. Which standard replaced IEEE 829, and what do the log and summary report become there? ISO/IEC/IEEE 29119-3, now in its 2021 edition. The test log becomes the test execution log and the test summary report becomes the test completion report; the incident report keeps its name, and a test status report is added for progress during testing.

5. Why did IEEE 829-2008 call it an anomaly report rather than a fault report? Because a difference between expected and actual results can arise from causes other than a fault in the system, such as a wrong expected result or a test run incorrectly.

6. Are all the documents mandatory for every project? No. The standards specify the format of each document, not whether it must be produced; the test plan decides, according to context, and agile teams keep the same information in lighter forms.

The arrow from maintenance back to requirements in the figure is the part students forget. A system that is used gets changed, and every change runs through requirements, design, code and test again. For most successful software, far more of its life is spent in that loop than in the first build.

The families of models

The ISTQB syllabus groups the models into three families, and adds the agile methods that sit alongside them.

The standard vocabulary defines the sequential model's defining property precisely. ISO/IEC/IEEE 24765:2017 describes the waterfall model as one in which the phases "are performed in that order, possibly with overlap but with little or no iteration", and incremental development as a technique in which requirements, design, implementation and testing "occur in an overlapping, iterative (rather than sequential) manner, resulting in incremental completion of the overall software product."

Why the choice of model matters to a tester

This is a testing paper, and the SDLC appears on its syllabus because the model decides how testing is done. The ISTQB syllabus lists what the choice of SDLC affects: the "Scope and timing of test activities", the "Level of detail of test documentation", the "Choice of test techniques and test approach", the "Extent of test automation" and the "Role and responsibilities of a tester".

In practice the differences are large:

• In a sequential model, testers review requirements and design tests early, but "dynamic testing cannot be performed early in the SDLC", because the code arrives late (ISTQB).
• In iterative and incremental models each iteration delivers something that runs, so "both static testing and dynamic testing may be performed at all test levels" in every iteration, and frequent delivery "requires fast feedback and extensive regression testing".
• In agile development, change is expected throughout, so "lightweight work product documentation and extensive test automation to make regression testing easier are favored", and much manual testing uses experience-based techniques.

Chapter Eighteen, on the role of testing in each phase, takes the phases one by one from the tester's side.

Worked example: ExamReg release 2.0, phase by phase

Choosing a model

No model is right for every project, which is the SDLC form of the principle that testing is context dependent. The usual considerations, stated as practice:

What it does not mean

The SDLC is not a single fixed sequence. It is a set of kinds of work; the model decides the order and the repetition.

The life cycle does not end at deployment. Maintenance is usually the longest part of a product's life, and every change passes through the phases again.

Testing is not only one phase of the SDLC. It appears as a phase in the table, but test activities run alongside every phase (principle 3, early testing, and Chapter Eighteen).

"Agile" does not mean without a life cycle. Agile teams still do requirements, design, coding, testing and deployment, in short cycles instead of long phases.

Quick revision

• Life cycle: evolution of a system from conception through retirement (ISO/IEC/IEEE 12207:2026).
• Life cycle model: a framework of processes and activities organised into stages, a common reference for everyone (ISO/IEC/IEEE 12207:2026).
• Generic phases: feasibility, requirements, design, implementation, testing, deployment, maintenance; maintenance loops back.
• Families (ISTQB): sequential (waterfall, V-model), iterative (spiral, prototyping), incremental (Unified Process), plus agile methods (Scrum, Kanban, XP, TDD, BDD, ATDD).
• The SDLC decides the scope and timing of testing, documentation detail, techniques, automation and the tester's role.
• Choose by requirement stability, risk, customer availability and the evidence required.

Test yourself

1. Define the software development life cycle and a life cycle model. The life cycle is the evolution of a software system from conception through retirement. A life cycle model is a framework of processes and activities organised into stages, acting as a common reference for communication and understanding, such as the waterfall or spiral model.

2. List the phases of the SDLC with the work product of each. Feasibility (feasibility report), requirements (requirements specification), design (design documents), implementation (source code and unit tests), testing (test reports and fixed defects), deployment (the released system and user guides), maintenance (fixes, changes and new releases).

3. Name the three families of SDLC models with an example of each. Sequential, such as the waterfall model and the V-model; iterative, such as the spiral model and prototyping; incremental, such as the Unified Process. Agile methods such as Scrum combine iteration and increments.

4. How does the choice of SDLC model affect testing? Give three effects. It sets the scope and timing of test activities, the level of detail of test documentation, the choice of techniques and approach, the extent of automation, and the tester's role. For example, in a sequential model dynamic testing comes late, while in agile development automated regression testing runs every iteration.

5. Why is maintenance drawn as a loop back to requirements? Because a system in use keeps changing, and each change must again be specified, designed, coded and tested; for most software this loop makes up most of its life.

He does not throw the sequence away: "However, I believe the illustrated approach to be fundamentally sound." Instead he adds five features to reduce the risk, and every one of them is still good advice.

Royce on testing, fifty years early

Step 4 is worth reading in full, because it anticipates half of this syllabus. Royce writes: "Without question the biggest user of project resources, whether it be manpower, computer time, or management judgment, is the test phase. It is the phase of greatest risk in terms of dollars and schedule." He then recommends four things:

1. Independent testers. Many parts of testing are best done by specialists who did not contribute to the design; if only the designer can test the design, the documentation has failed. This is the independent test group of Chapter Thirty-One, on the strategic approach to testing.
2. Visual inspection by a second person. Most errors are obvious and can be spotted by a second party scanning the analysis and the code, "dropped minus signs, missing factors of two, jumps to wrong addresses". This is the review and inspection of Chapters Twenty-Eight and Twenty-Nine, six years before Fagan published inspections.
3. Every logic path, at least once. "Test every logic path in the computer program at least once with some kind of numerical check." This is path and branch coverage (Chapters Fifty-Nine to Sixty and Seventy-Two).
4. The computer last. Only after the simple errors are removed should the software go to formal checkout on the machine.

The paper that gave the world the sequential model also argued for early reviews, independent testing, coverage and a pilot version. The model that took its name kept the staircase and dropped the advice.

The waterfall model as it is taught

The textbook waterfall keeps Royce's staircase in its simplest form. Its properties follow directly from the sequence.

Where testing sits in the waterfall

In the pure waterfall, testing is a phase near the end: the code is complete, then it is tested. That is exactly the placement Royce warned about. The ISTQB syllabus describes the consequence for any sequential model: "in the initial phases testers typically participate in requirement reviews, test analysis, and test design. The executable code is usually created in the later phases, so typically dynamic testing cannot be performed early in the SDLC."

The sentence contains the remedy. Even in a strict waterfall, testers need not wait: they can review the requirements and designs as they are written (static testing) and design their tests while the code is being built. The V-model, the next chapter, is essentially the waterfall with that remedy drawn into it.

Worked example: ExamReg in a pure waterfall

Suppose ExamReg release 2.0 were run as a pure waterfall over sixteen weeks: requirements weeks 1 to 3, design 4 to 6, coding 7 to 12, testing 13 to 15, deployment week 16. Coding overruns by two weeks, as coding often does. The deadline cannot move, because the examination form opens on a fixed date. Testing shrinks from three weeks to one.

In that one week, testing finds that the portal slows to a crawl above 300 simultaneous users, far short of the 500 expected on the last day. The cause is the design of the fee service, fixed in week 5. This is Royce's Figure 4 exactly: a real behaviour, "experienced as distinguished from analyzed", met for the first time in the test phase, and the fix is a redesign, not a patch. The team now chooses between releasing a portal that will fail on the last day and missing the date.

Every remedy in this book is a way of moving that discovery earlier: a design review in week 5 asking how the fee service handles 500 users; a small load test of a pilot version, Royce's "do it twice"; or an iterative model in which a working fee service exists by week 6.

What it does not mean

Royce did not recommend the simple waterfall. He called it "risky" and said it "invites failure", then added five steps to fix it.

The word "waterfall" is not in Royce's paper. It does not occur anywhere in the text; the name was attached to his staircase later.

Waterfall does not forbid going back. The standard definition allows "overlap" and "little" iteration, and Royce drew iteration between successive steps; what the model resists is going back more than a step.

Sequential does not mean testers wait until coding ends. Reviews and test design run alongside the early phases even in a waterfall.

Quick revision

• Waterfall model: phases performed in order, possibly with overlap, "with little or no iteration" (ISO/IEC/IEEE 24765:2017); the classic sequential model.
• Royce 1970 (IEEE WESCON): seven steps: system requirements, software requirements, analysis, program design, coding, testing, operations.
• Royce: "risky and invites failure", because testing is the first time real behaviour is met; up to 100 per cent overrun.
• His five fixes: program design comes first; document the design; do it twice; plan, control and monitor testing; involve the customer.
• His testing advice: independent specialists, visual inspection by a second person, every logic path tested at least once, the computer last.
• Strengths: simple, documented, good for stable requirements and contracts. Weaknesses: late working software, late defects, poor with change, testing squeezed.

Test yourself

1. Explain the waterfall model with its phases. A sequential model in which each phase is completed before the next begins: in Royce's form, system requirements, software requirements, analysis, program design, coding, testing and operations. Each phase ends in a document that becomes the input to the next, with little or no iteration.

2. What did Royce say was wrong with the simple sequence? That it is risky and invites failure, because testing at the end is the first time timing, storage and input/output are experienced rather than analysed. A failure then usually demands a redesign that can break the requirements, sending the project back to the start with up to a 100 per cent overrun.

3. State Royce's five recommended steps. Program design comes first; document the design; do it twice (build a pilot version); plan, control and monitor testing; involve the customer.

4. Give three advantages and three disadvantages of the waterfall model. Advantages: simple to manage, with progress visible by phase; documented phase outputs useful for contracts and audits; efficient when requirements are stable. Disadvantages: working software only at the end; requirement and design defects found late and expensively; poor at absorbing change, with testing squeezed when earlier phases overrun.

5. Where does testing sit in the waterfall, and how can testers still start early? Dynamic testing is a late phase, after coding. Testers can still start early by reviewing the requirements and designs as they are written and by designing their tests during the earlier phases.

Each phase and its test level

The right-hand column comes from the current ISTQB syllabus's own descriptions of the levels: component testing "focuses on testing components in isolation"; component integration testing "focuses on testing the interfaces and interactions between components"; system testing "focuses on the overall behavior and capabilities of an entire system or product"; and acceptance testing "focuses on validation and on demonstrating readiness for deployment, which means that the system fulfills the user's business needs." The syllabus adds a fifth level, system integration testing, which tests "the interfaces of the system under test and other systems and external services"; in a V it sits beside system testing. Test levels have their own chapter (Chapter Thirty-Two).

The relationship between phases and testing

This is the part of the syllabus MU words as "phases and their relationship to testing", and the V states the relationship in four rules, all of which the current ISTQB syllabus lists as good practice whatever model is used:

1. Every development activity has a matching test activity. "For every software development activity, there is a corresponding test activity, so that all development activities are subject to quality control."
2. Each test level has its own objectives. Different levels "have specific and different test objectives, which allows for testing to be appropriately comprehensive while avoiding redundancy". Unit tests do not re-test what system tests cover, and system tests do not repeat unit tests.
3. Test design starts in the matching phase. "Test analysis and design for a given test level begins during the corresponding development phase of the SDLC, so that testing can adhere to the principle of early testing".
4. Testers review drafts early. "Testers are involved in reviewing work products as soon as drafts of these work products are available".

The third rule is the one that pays. Writing an acceptance test forces the question how will we know this requirement is met? while the requirement can still be changed at the cost of a sentence. A requirement nobody can write a test for is usually a requirement nobody can build to either.

Worked example: ExamReg release 2.0 on a V

Look at the integration row. The paise-for-rupees interface defect of Chapter Two, on errors, faults and failures, could be caught by an integration test designed in week 5, from the architecture document, before either module was coded. In a pure waterfall nobody would have written that test until week 13.

Strengths and weaknesses

The V-model is common in safety-critical and regulated industries, such as vehicles, medical devices and aviation, where every requirement must be shown to have been verified. There it often sits inside a larger iterative plan: a V for each increment.

The W-model, briefly

Some writers add a second V inside the first, drawn as a W: alongside each development phase on the left is a review of that phase's work product. The idea is already in rule 4 above: static testing of the requirements, the specification and the designs, as soon as drafts exist, before any dynamic test runs. The W is the V with static testing drawn in.

What it does not mean

The V-model is not the waterfall with testing at the end. Its test execution is at the end, but its test design is at the start, phase by phase.

The V does not mean each level is tested only once. Defects found at a level are fixed and retested, and regression testing follows every change.

Unit testing does not check the requirements. Each level checks the work product opposite it; only acceptance testing checks the users' needs directly.

The V-model is not obsolete. The current ISTQB syllabus still lists it, and regulated industries use it, often within iterative development.

Quick revision

• V-model: a sequential model pairing each development phase with a test level; test design starts in the matching phase, so it "integrates the test process throughout the development process" (ISTQB 2018).
• Pairs: user requirements with acceptance testing; system specification with system testing; architectural design with integration testing; detailed design with component (unit) testing; coding at the point.
• Left arm down, right arm up; the arrows across are tests designed early from each work product.
• Good practice for any model (ISTQB v4.0.1): a test activity for every development activity; distinct objectives per level; test design during the matching phase; early review of drafts.
• Strengths: early test design, clear levels, traceability. Weaknesses: still sequential, late working software, costly change.
• W-model: a review beside every development phase.

Test yourself

1. Explain the V-model with a diagram. Draw development phases down the left (user requirements, system specification, architectural design, detailed design), coding at the point, and test levels up the right (component, integration, system, acceptance testing), with each level opposite the phase whose work product is its test basis. Tests for each level are designed during the matching phase and executed on the way up.

2. Which test level is paired with the architectural design, and why? Integration testing, because the architecture defines the components and their interfaces, and integration testing checks exactly those interfaces and interactions.

3. How does the V-model improve on the waterfall model? It integrates testing throughout development: tests for each level are designed as soon as the matching work product exists, so defects in requirements and designs are found early instead of in a single late test phase.

4. State two strengths and two weaknesses of the V-model. Strengths: testing planned and designed from the start; clear objectives and traceability for each level. Weaknesses: still sequential, so working software comes late; assumes stable early requirements, so changes are costly.

5. What does "phases and their relationship to testing" mean in the V-model? That every development phase has a corresponding test level whose test basis is that phase's work product, and whose test design begins during that phase.

There are two kinds. A throwaway prototype is built quickly to learn something, usually what users want from a screen, and then discarded. An evolutionary prototype is built carefully and grown into the product. For ExamReg, a clickable mock-up of the exam form shown to twenty students before any code is written is a throwaway prototype: it costs a day and settles which fields confuse them. Royce's step 3, "do it twice", from the waterfall chapter, is a prototype in all but name.

The spiral model

Barry Boehm, then chief scientist of TRW's Defense Systems Group, published "A Spiral Model of Software Development and Enhancement" in the magazine Computer in May 1988. It opens on the debate of its day ("The waterfall model is dead." "No, it isn't, but it should be.") and states its own central idea plainly: "The major distinguishing feature of the spiral model is that it creates a risk-driven approach to the software process rather than a primarily document-driven or code-driven process."

Figure 16.1 Boehm's spiral model, simplified: each cycle passes through the four quadrants, and cost grows with the radius

Boehm reads his own figure this way: "The radial dimension in Figure 2 represents the cumulative cost incurred in accomplishing the steps to date; the angular dimension represents the progress made in completing each cycle of the spiral."

Each cycle goes through four stages, one per quadrant:

1. Determine objectives, alternatives and constraints. For the part of the product being worked on: its objectives (performance, functionality, ability to change), the alternative ways of building it (design A, design B, reuse, buy), and the constraints (cost, schedule, interfaces).
2. Evaluate alternatives; identify and resolve risks. Evaluating the alternatives against the objectives exposes uncertainties, and those are the risks. They are resolved by whatever is cheapest and convincing: prototyping, simulation, benchmarking, reference checking, user questionnaires, analytic modelling.
3. Develop and verify the next-level product. What is built next depends on the risk that remains. If performance or user-interface risks dominate, the next step is another, more detailed prototype. If those are settled and the risks are in program development, the next step follows the familiar waterfall stages, each "followed by a validation step".
4. Plan the next phases. Each cycle ends, in Boehm's account, with a review by the people concerned with the product, covering everything produced in the cycle and the plans for the next, so that all parties commit to the next step.

The cycle then repeats, further out on the spiral, until the product is complete. Boehm adds a small Round 0 in his worked example, a feasibility study before the first full cycle.

Why "risk-driven" matters to a tester

The spiral's defining idea is that risk decides what to do next. Boehm extends that to testing directly: risk considerations "can determine the amount of time and effort that should be devoted to such other project activities as" quality assurance, formal verification and testing. That is risk-based testing, which Chapter Eleven's test plan used as its approach, stated as a property of the whole process.

Testing also appears in every quadrant. Quadrant 2 uses prototypes, benchmarks and simulations, which are experiments, often run by testers. Quadrant 3 ends each level of the product with verification and validation. And quadrant 4's review is a static test of everything the cycle produced.

Testing in iterative and incremental models generally

The ISTQB syllabus states the consequence for any model of this family: "it is assumed that each iteration delivers a working prototype or product increment. This implies that in each iteration both static testing and dynamic testing may be performed at all test levels. Frequent delivery of increments requires fast feedback and extensive regression testing."

The last sentence is the one to remember. Every new increment is added to software that already works, and each one can break what came before. By the third ExamReg release the regression suite includes the tests of releases one and two, and it runs on every build. This is why iterative development and test automation grew up together, and why Chapter Thirty-Nine, on regression testing and continuous integration, belongs to the same story.

Worked example: ExamReg release 2.0 as a spiral

Compare it with the pure waterfall story of Chapter Fourteen. There, the load problem was discovered in the last week, with no time left. Here it was the first risk attacked, in cycle 2, while choosing between two designs cost nothing but a benchmark.

Strengths and weaknesses of the spiral model

The last row is Boehm's own honesty: his third conclusion was that the model "is not yet as fully elaborated as the more established models", needing more work on contracting, milestones, reviews, scheduling and status monitoring before it would suit every situation.

What it does not mean

Iterative does not mean unplanned. Each iteration is planned; the spiral plans every cycle explicitly.

Incremental is not the same as iterative. Incremental adds pieces; iterative refines the whole. Most real projects do both.

The spiral is not a fixed number of loops. The number of cycles, and what each builds, is decided by the risks that remain.

A prototype is not the product. A throwaway prototype is discarded once it has answered its question; building on it by accident is a common and costly mistake.

Quick revision

• Iterative development: repeated planning, developing and testing to refine the same product. Incremental development: overlapping stages delivering the product in increments.
• Increment: a tested, deliverable version providing new or modified capabilities (ISO/IEC/IEEE 24748-8:2026).
• Prototyping: a preliminary version for user feedback, feasibility or timing; throwaway or evolutionary.
• Spiral model (Boehm, Computer, May 1988): risk-driven; radius is cumulative cost, angle is progress.
• Four quadrants: determine objectives, alternatives and constraints; evaluate alternatives and resolve risks; develop and verify the next-level product; plan the next phases, with a review and commitment.
• Testing: static and dynamic testing in every iteration; fast feedback and extensive regression testing (ISTQB).

Test yourself

1. Distinguish iterative from incremental development, with an example. Iterative development repeats the cycle to refine the same product, such as a rough version of the whole portal improved in later rounds. Incremental development delivers the product in pieces, each adding capability, such as login and the exam form first, then fees and payment, then hall tickets.

2. Explain the spiral model with a diagram. Draw two axes and a spiral growing outward from the centre, clockwise, through four quadrants: determine objectives, alternatives and constraints; evaluate alternatives and identify and resolve risks; develop and verify the next-level product; plan the next phases. The radius shows cumulative cost and the angle progress within a cycle; each cycle ends with a review and commitment.

3. Why is the spiral model called risk-driven? Because at each cycle the remaining risks decide what to do next: a prototype, a simulation, a specification or waterfall-style development, and even how much effort goes into testing and verification.

4. What is a prototype? Distinguish throwaway from evolutionary prototypes. A preliminary version of a system used for feedback, feasibility or investigation. A throwaway prototype is built quickly to answer a question and then discarded; an evolutionary prototype is built carefully and grown into the product.

5. How does testing change in iterative and incremental models? Every iteration delivers something that runs, so static and dynamic testing happen in every iteration and at all levels, and frequent delivery demands fast feedback and extensive, usually automated, regression testing.

The Scrum Team is "one Scrum Master, one Product Owner, and Developers", with no sub-teams or hierarchies, "focused on one objective at a time, the Product Goal." The guide speaks of three accountabilities rather than roles. The Product Owner is accountable for the value of the product and orders the Product Backlog. The Developers create the Increment, and "Developers" includes everyone who builds it: programmers, testers, designers. The Scrum Master is accountable for the team's effectiveness with Scrum. There is no separate tester accountability: testing is part of creating a usable Increment, and the Developers are accountable for, among other things, "Instilling quality by adhering to a Definition of Done".

The events. The Sprint is the container for all the others: Sprints are "fixed length events of one month or less". Sprint Planning decides the Sprint Goal and the work to reach it. The Daily Scrum is "a 15-minute event for the Developers" to inspect progress toward the Sprint Goal. The Sprint Review inspects the Increment with stakeholders. The Sprint Retrospective plans improvements to how the team works.

The artifacts and their commitments. The Product Backlog (committed to the Product Goal), the Sprint Backlog (committed to the Sprint Goal) and the Increment, committed to the Definition of Done.

The Definition of Done: where testing lives in Scrum

The Scrum Guide defines it precisely: "The Definition of Done is a formal description of the state of the Increment when it meets the quality measures required for the product." It is also absolute: "If a Product Backlog item does not meet the Definition of Done, it cannot be released or even presented at the Sprint Review." The ISTQB syllabus draws the connection to testing: in agile development, exit criteria are called the Definition of Done, and the entry criteria a user story must meet before work begins are the Definition of Ready.

A Definition of Done for ExamReg's team might read: code reviewed; unit tests written and passing; acceptance tests for the story automated and passing; no open defect of major severity or above; regression suite green on the build server; exam cell has seen it working. Every line is a test or a review. In Scrum, "done" means "tested".

Testing in agile development

The ISTQB syllabus summarises how agile changes testing: agile "assumes that change may occur throughout the project. Therefore, lightweight work product documentation and extensive test automation to make regression testing easier are favored in agile projects. Also, most of the manual testing tends to be done using experience-based test techniques ... that do not require extensive prior test analysis and design."

Three practices, which the syllabus groups as approaches in which tests direct development, make the tests come first:

A BDD scenario for ExamReg's late fee reads like this, and a tool can run it as a test once each step is bound to code:

Given a student has completed the examination form
And today is 3 days after the last date
When the student opens the fee page
Then the late fee shown is Rs 100

TDD's cycle is worked through in full in Chapter Thirty-Six, on unit testing best practices.

DevOps and continuous integration

The ISTQB syllabus describes DevOps as "an organizational approach aiming to create synergy by getting development (including testing) and operations to work together to achieve a set of common goals." Its technical core is continuous integration (CI), a "technique that continually merges artifacts, including source code updates from all developers on a team, into a shared mainline to build and test the developed system" (IEEE 2675-2021), and continuous delivery (CD), which keeps the system always ready to release.

For testing, the syllabus lists the benefits: fast feedback on code quality; CI "promotes shift left in testing ... by encouraging developers to submit high quality code accompanied by component tests and static analysis"; stable automated test environments; more visibility of non-functional qualities such as performance; less repetitive manual testing; and a smaller regression risk because automated regression tests run at scale. It lists the costs too: the pipeline must be built, the CI and CD tools maintained, and "Test automation requires additional resources and may be difficult to establish and maintain." And it adds a warning worth repeating: even with this much automation, manual testing from the user's perspective will still be needed. Continuous integration, and Jenkins as the tool the practical uses for it, has its own chapter (Chapter Thirty-Nine).

Shift left, and retrospectives

Shift left is the syllabus's name for the principle of early testing applied deliberately: reviewing specifications from a tester's point of view, writing test cases before code, using CI with automated component tests, running static analysis before dynamic testing, and starting non-functional testing at component level where possible. The syllabus adds the balancing sentence: shift left "does not mean that testing later in the SDLC should be neglected."

Retrospectives, held at the end of each iteration, ask what went well, what did not, and how to improve. The syllabus lists their benefits for testing, including more effective testing, better testware, better requirements and better cooperation between development and testing. They are the agile form of the lessons learned in test completion (Chapter Five, on the basic test process).

The test pyramid

When tests run every day, their cost and speed matter. The test pyramid is the model most teams use to balance them. Martin Fowler's account of it records that most people know it "due to Mike Cohn, when he described it in his 2009 book Succeeding with Agile", and that Cohn had drawn it in conversation with Lisa Crispin in 2003 and 2004.

Figure 17.1 The test pyramid (Cohn): many small, fast tests at the base and few broad, slow ones at the top

The ISTQB syllabus explains the shape: "The higher the layer, the lower the test granularity, the lower the test isolation ... and the higher the test execution time." Tests at the base are small, isolated and fast, so many are needed; tests at the top are end-to-end and slow, so few are used. Cohn's original three layers are "unit tests", "service tests" and "UI tests". Fowler adds the reason not to invert it: tests that run end to end through the user interface are "brittle, expensive to write, and time consuming to run."

The testing quadrants

A second agile model, the testing quadrants, defined by Brian Marick, sorts tests along two lines: whether they face the business or the technology, and whether they support the team (guide development) or critique the product. The ISTQB syllabus fills in the four quadrants:

The quadrants are a checklist: a team whose tests all sit in Q1 has fast unit tests and no idea whether users can use the product.

Worked example: one ExamReg Sprint

The team runs two-week Sprints. In Sprint 4 the Sprint Goal is: students with a fee concession can submit the form and pay only the backlog and late fees.

Compare the waterfall of Chapter Fourteen. There, the first working fee page existed in week 13 of 16. Here, a tested increment exists every two weeks.

What it does not mean

Agile does not mean no documentation or no planning. The manifesto values the items on the right; it values those on the left more. Scrum plans every Sprint.

Agile does not mean no testers. Scrum folds testing into the Developers' accountability; people who specialise in testing are Developers.

Automation does not replace all manual testing. The syllabus says user-perspective manual testing will still be needed; exploratory and usability testing sit in Q3.

Shift left does not mean skipping later testing. System and acceptance testing still happen; they just find fewer defects.

Quick revision

• Agile development: iterative, incremental, frequent inspection and adaptation, cross-functional collaboration.
• Manifesto values: individuals and interactions, working software, customer collaboration, responding to change, over processes and tools, comprehensive documentation, contract negotiation, following a plan.
• Scrum (Guide, November 2020): Scrum Team of Product Owner, Scrum Master and Developers; Sprint (one month or less), Sprint Planning, Daily Scrum (15 minutes), Sprint Review, Sprint Retrospective; Product Backlog, Sprint Backlog, Increment; Definition of Done = exit criteria; Definition of Ready = entry criteria.
• TDD, ATDD, BDD: tests first; BDD in Given/When/Then.
• DevOps: development and operations together; CI and CD; fast feedback; manual user-perspective testing still needed.
• Shift left: test earlier, without neglecting later testing. Retrospectives: continuous improvement.
• Test pyramid (Cohn): many unit, some service, few UI tests. Quadrants (Marick): business or technology facing, supporting the team or critiquing the product.

Test yourself

1. State the four values of the Agile Manifesto. Individuals and interactions over processes and tools; working software over comprehensive documentation; customer collaboration over contract negotiation; responding to change over following a plan. The items on the right still have value; those on the left are valued more.

2. Describe the Scrum framework. A Scrum Team of one Product Owner, one Scrum Master and Developers works in Sprints of one month or less. Each Sprint has Sprint Planning, Daily Scrums of 15 minutes, a Sprint Review and a Sprint Retrospective, and produces an Increment that meets the Definition of Done, drawn from the Product Backlog through the Sprint Backlog.

3. What is the Definition of Done, and how does it relate to testing? A formal description of the state of the Increment when it meets the quality measures required for the product. It works as the exit criteria for a backlog item: an item that does not meet it cannot be released or even shown at the Sprint Review, so it normally lists the reviews and tests that must pass.

4. Distinguish TDD, ATDD and BDD. TDD writes unit tests before the code and directs coding through them; ATDD derives tests from acceptance criteria before the feature is developed; BDD writes the expected behaviour in simple Given/When/Then language that is turned into executable tests.

5. Explain the test pyramid. A model with many small, fast, isolated unit tests at the base, fewer service or API tests in the middle, and few slow, broad UI end-to-end tests at the top; higher layers have lower granularity and isolation and longer execution time.

6. Give two benefits and two challenges of DevOps for testing. Benefits: fast feedback on code quality, and a smaller regression risk because automated regression tests run on every change. Challenges: the delivery pipeline and its CI and CD tools must be built and maintained, and test automation needs resources and can be hard to maintain.

Design: testing the plan before the building

What a tester does:

• Reviews the architecture and detailed design against the requirements: does every requirement have a place in the design? Can the fee service really take 500 requests at once?
• Designs integration tests from the interfaces between components, as the V-model pairs them.
• Designs system tests and the test environment, including test data, from the specification.
• Reviews the design for testability: can the clock be set for late-fee tests? Can the payment gateway be replaced by a simulator in the test environment?

Implementation: the developer's testing

What happens:

• Unit (component) testing by developers, often written first under test-driven development (Chapter Thirty-Three onwards covers unit testing).
• Static analysis by tools that read code without running it, finding unused variables, unreachable code and suspicious patterns (Chapter Twenty-Seven, on the kinds of V&V).
• Code reviews and inspections by other developers (Chapters Twenty-Eight to Thirty).
• Continuous integration, which builds and runs the tests on every change (Chapter Thirty-Nine).

The testing phase: integration, system and acceptance

Here the test levels of the V-model execute: integration testing of the interfaces, system testing of the whole against its specification, system integration testing against external systems such as the payment gateway, and acceptance testing by the users. Module 1's last row takes each of these in turn, from Chapter Thirty-One, on the strategic approach to testing.

Deployment: the last checks before users arrive

What a tester does:

• Installation or deployment testing: the system installs correctly in production, with the right configuration.
• A smoke test in production: the main functions respond after deployment.
• Operational acceptance testing: backups, recovery, monitoring and the other things the IT team will need to run the system (a form of acceptance testing, Chapter Forty-Two).

Maintenance: testing never stops

A system in use changes, and the ISTQB syllabus names the kinds of maintenance: "corrective, adaptive to changes in the environment or improve performance or maintainability". It also names the triggers for maintenance testing: modifications (planned enhancements, corrective changes, hot fixes); upgrades or migrations of the operational environment, including data conversion; and retirement, which may need testing of data archiving and of restoring archived data.

Testing a change "includes both evaluating the success of the implementation of the change and the checking for possible regressions in parts of the system that remain unchanged (which is usually most of the system)". The size of that effort depends, in the syllabus's words, on "The degree of risk of the change", "The size of the existing system" and "The size of the change". The tool for judging it is impact analysis: the "identification of all system and software products that a change request affects" (ISO/IEC/IEEE 24765).

The phases side by side

Worked example: testing the requirements before anything is built

The cheapest test in this whole book runs on sentences. A reviewer reading ExamReg's draft requirements looks first for words that no test can check, because a requirement nobody can test is a requirement nobody can prove was met. The program below does a crude version of that first pass: it flags any requirement containing a word from a list of vague terms.

VAGUE = ["fast", "quickly", "user-friendly", "easy", "appropriate", "as needed",
         "etc", "approximately", "reasonable", "adequate", "efficient", "normally"]

requirements = {
    "R1": "A form submitted 1 to 7 days after the last date pays a late fee of Rs 100.",
    "R2": "The fee page should load quickly even when many students use the portal.",
    "R3": "The hall ticket shows the seat number, subjects, dates etc.",
    "R4": "The portal locks an account for 30 minutes after 3 wrong passwords in a row.",
    "R5": "Error messages must be user-friendly and appropriate.",
    "R6": "The fee page appears within 2 seconds for 500 simultaneous users.",
}
for rid, text in requirements.items():
    words = text.lower().replace(".", " ").replace(",", " ").split()
    found = [v for v in VAGUE if (v in words if " " not in v else v in text.lower())]
    verdict = "testable as written" if not found else "VAGUE: " + ", ".join(found)
    print(f"{rid}: {verdict}")

R1: testable as written
R2: VAGUE: quickly
R3: VAGUE: etc
R4: testable as written
R5: VAGUE: user-friendly, appropriate
R6: testable as written

Three of six requirements come back vague, and each flag is a question for the exam cell before design starts. How quickly, for how many students? Which fields does a hall ticket show, exactly, since "etc." cannot be tested? What does a user-friendly error message say? R6 is what R2 becomes once someone answers the first question, and it can be tested with a load test.

The program is deliberately crude, and its limits are the lesson. A word list catches vague words; it cannot catch a requirement that is precise and wrong, like the fee table that forgot negative days in Chapter One, on what software testing is. That needs a person reading the requirement and asking what happens at every edge, which is why requirement reviews are done by people, with tools only to help.

What it does not mean

The testing phase is not the only place testing happens. It is where dynamic testing of the integrated system happens; testing activities run in every phase.

Early testing does not replace late testing. Shift left moves effort earlier, and system and acceptance testing still run; they simply find fewer defects.

Maintenance testing is not only retesting the fix. It is also regression testing of the unchanged system, usually the larger part, sized by impact analysis.

Testability is not only a property of code. A requirement is testable or not before a line of code exists.

Quick revision

• ISTQB good practices: a test activity for every development activity; distinct objectives per level; test design during the matching phase; testers review drafts early.
• Requirements: review for ambiguity, gaps, testability ("extent to which an objective and feasible test can be designed"); acceptance criteria; traceability; risks.
• Design: review designs; design integration and system tests; design the environment; check design testability.
• Implementation: unit testing, static analysis, code reviews, continuous integration.
• Testing: integration, system, system integration, acceptance testing.
• Deployment: installation, smoke, operational acceptance testing.
• Maintenance: triggers are modifications, upgrades or migrations, retirement; test the change and regressions; scope by risk and size; impact analysis.
• Fixing after delivery is often 100 times dearer than in requirements and design (Boehm and Basili 2001).

Test yourself

1. Explain the role of testing in each phase of the SDLC. Requirements: review for ambiguity, gaps and testability, write acceptance criteria. Design: review designs and design integration and system tests. Implementation: unit tests, static analysis and code reviews. Testing: integration, system and acceptance testing. Deployment: installation and smoke testing. Maintenance: confirmation and regression testing guided by impact analysis.

2. What is testability of a requirement? Give a testable and an untestable example. The extent to which an objective and feasible test can be designed to show the requirement is met. Untestable: the fee page should load quickly. Testable: the fee page appears within 2 seconds for 500 simultaneous users.

3. What triggers maintenance testing? Modifications such as planned enhancements, corrective changes and hot fixes; upgrades or migrations of the operational environment, including data conversion; and retirement, which may require testing of data archiving and restoration.

4. What decides how much maintenance testing is needed? The degree of risk of the change, the size of the existing system and the size of the change, judged with impact analysis of what the change affects.

5. Why is testing in the requirements phase the most valuable? Because a defect in a requirement is copied into the design, code and tests built on it, and fixing it after delivery can cost around a hundred times more than fixing it while the requirement is being written.

Notice how many definitions begin with effort. For the revision and transition factors, quality is measured by how much work a change costs, which is exactly the cost the Air Force was trying to control.

From factors to criteria to metrics

A factor is how a user sees quality, and it cannot be measured directly. McCall's model therefore works in three layers.

1. Factors: the user's view (reliability).
2. Criteria: attributes of the software that produce the factor. The report gives reliability's as "error tolerance, consistency, accuracy, and simplicity", and it defines each criterion, for example error tolerance as "Those attributes of the software that provide continuity of operation under nonnominal conditions."
3. Metrics: measurements of each criterion, taken from the code, the documents or the test results.

The report defines twenty-three criteria, and many serve more than one factor. The program below records, from the report's Table 4.1-1, which factors each criterion supports, then counts them both ways.

related = {   # McCall 1977, Table 4.1-1: criterion -> the factors it supports
    "traceability": ["correctness"],
    "completeness": ["correctness"],
    "consistency": ["correctness", "reliability", "maintainability"],
    "accuracy": ["reliability"],
    "error tolerance": ["reliability"],
    "simplicity": ["reliability", "maintainability", "testability"],
    "modularity": ["maintainability", "flexibility", "testability",
                   "portability", "reusability", "interoperability"],
    "generality": ["flexibility", "reusability"],
    "expandability": ["flexibility"],
    "instrumentation": ["testability"],
    "self-descriptiveness": ["flexibility", "maintainability", "testability",
                             "portability", "reusability"],
    "execution efficiency": ["efficiency"],
    "storage efficiency": ["efficiency"],
    "access control": ["integrity"],
    "access audit": ["integrity"],
    "operability": ["usability"],
    "training": ["usability"],
    "communicativeness": ["usability"],
    "software system independence": ["portability", "reusability"],
    "machine independence": ["portability", "reusability"],
    "communications commonality": ["interoperability"],
    "data commonality": ["interoperability"],
    "conciseness": ["maintainability"],
}
print(len(related), "criteria")
factors = {}
for criterion, fs in related.items():
    for f in fs:
        factors.setdefault(f, []).append(criterion)
for f in sorted(factors, key=lambda f: -len(factors[f])):
    print(f"{f:<17}{len(factors[f])} criteria: {', '.join(factors[f])}")
print()
widest = sorted(related, key=lambda c: -len(related[c]))[:3]
for c in widest:
    print(f"'{c}' supports {len(related[c])} factors")

23 criteria
maintainability  5 criteria: consistency, simplicity, modularity, self-descriptiveness, conciseness
reusability      5 criteria: modularity, generality, self-descriptiveness, software system independence, machine independence
reliability      4 criteria: consistency, accuracy, error tolerance, simplicity
testability      4 criteria: simplicity, modularity, instrumentation, self-descriptiveness
flexibility      4 criteria: modularity, generality, expandability, self-descriptiveness
portability      4 criteria: modularity, self-descriptiveness, software system independence, machine independence
correctness      3 criteria: traceability, completeness, consistency
interoperability 3 criteria: modularity, communications commonality, data commonality
usability        3 criteria: operability, training, communicativeness
efficiency       2 criteria: execution efficiency, storage efficiency
integrity        2 criteria: access control, access audit

'modularity' supports 6 factors
'self-descriptiveness' supports 5 factors
'consistency' supports 3 factors

Two facts stand out from the counts. Modularity supports six of the eleven factors and self-descriptiveness five: a program built from independent, well-explained modules is easier to fix, change, test, move and reuse, all at once. That is the strongest argument in the report for good design. And testability appears as a factor in its own right, built from simplicity, modularity, instrumentation and self-descriptiveness: a design decision, made long before testing, decides how hard testing will be.

Factors pull against each other

The report is honest that the factors cannot all be maximised together. Its section 4.2 lists the trade-offs generally found, and almost all of them involve efficiency:

The practical consequence for a project is that its quality requirements must rank the factors. ExamReg must be correct and secure above all, because it handles fees and student records; it can afford to be a little less efficient than a hand-optimised program in exchange for being maintainable, because the fee rules change every session.

Worked example: ranking McCall's factors for ExamReg

The ranking is not a table for its own sake: it decides what the testing concentrates on, which is the subject of Chapter Twenty-One, on how quality factors shape testing.

What it does not mean

McCall's model is not the current standard. It dates from 1977; the international product quality model today is ISO/IEC 25010:2023, the next chapter. Its factors survive there under other names.

Factors are not measured directly. They are refined into criteria and then metrics, which are what is actually measured.

More of every factor is not always better. The factors trade against each other, and a project chooses its balance.

Testability is not a property of the test team. In McCall's model it is a property of the software, built from simplicity, modularity, instrumentation and self-descriptiveness.

Quick revision

• McCall, Richards and Walters, "Factors in Software Quality", RADC-TR-77-369, US Air Force, November 1977.
• Three activities, eleven factors: operation (correctness, reliability, efficiency, integrity, usability); revision (maintainability, flexibility, testability); transition (portability, reusability, interoperability).
• Questions: Does it do what I want? Can I fix it? Will I be able to use it on another machine?
• Layers: factors (user view), criteria (23 software attributes), metrics (measures).
• Modularity supports six factors and self-descriptiveness five.
• Trade-offs: efficiency against most factors; integrity against flexibility, reusability and interoperability.

Test yourself

1. List McCall's quality factors under their three product activities. Product operation: correctness, reliability, efficiency, integrity, usability. Product revision: maintainability, flexibility, testability. Product transition: portability, reusability, interoperability.

2. Define reliability and maintainability as McCall did. Reliability is the extent to which a program can be expected to perform its intended function with required precision. Maintainability is the effort required to locate and fix an error in an operational program.

3. What are criteria in McCall's model? Give the criteria of reliability. Criteria are attributes of the software that produce a factor and can be measured through metrics. Reliability's are error tolerance, consistency, accuracy and simplicity.

4. Why can the factors not all be maximised together? Give two trade-offs. Because improving one often costs another. Integrity against efficiency: access control adds code and processing time. Portability against efficiency: optimised, direct code ties a program to one system.

5. Which criterion supports the most factors, and what does that suggest? Modularity, which supports six factors: maintainability, flexibility, testability, portability, reusability and interoperability. It suggests that a modular design raises many qualities at once.

A few of the newer subcharacteristics deserve a sentence each, because they are the ones older books lack:

• Faultlessness (reliability) replaces the old "maturity": the "capability of a product to perform specified functions without fault under normal operation".
• Resistance (security) is new: the "capability of a product to sustain operations while under attack from a malicious actor".
• Inclusivity (interaction capability) is new: the "capability of a product to be utilized by people of various backgrounds".
• Scalability (flexibility) is new: the "capability of a product to handle growing or shrinking workloads or to adapt its capacity to handle variability".
• Fail safe (safety) is the "capability of a product to automatically place itself in a safe operating mode, or to revert to a safe condition in the event of a failure". Therac-25, in Chapter Three on why software must be tested, is what its absence looks like.

McCall's factors in the 2023 model

Nearly every McCall factor survives, sometimes as a characteristic and sometimes as a subcharacteristic. The program below records where each lands, and then asks the question in reverse: which 2023 characteristics have no McCall ancestor at all?

mccall_to_2023 = {        # McCall 1977 factor -> ISO/IEC 25010:2023 characteristic (subcharacteristic)
    "correctness":      ("functional suitability", "functional correctness"),
    "reliability":      ("reliability", "faultlessness"),
    "efficiency":       ("performance efficiency", "time behaviour, resource utilization"),
    "integrity":        ("security", "confidentiality, integrity"),
    "usability":        ("interaction capability", "operability, learnability"),
    "maintainability":  ("maintainability", "analysability, modifiability"),
    "flexibility":      ("maintainability", "modifiability"),
    "testability":      ("maintainability", "testability"),
    "portability":      ("flexibility", "adaptability, installability"),
    "reusability":      ("maintainability", "reusability"),
    "interoperability": ("compatibility", "interoperability"),
}
iso_2023 = ["functional suitability", "performance efficiency", "compatibility",
            "interaction capability", "reliability", "security", "maintainability",
            "flexibility", "safety"]

for factor, (characteristic, sub) in mccall_to_2023.items():
    print(f"{factor:<17} -> {characteristic} ({sub})")
used = {c for c, _ in mccall_to_2023.values()}
print("2023 characteristics with no McCall factor:", [c for c in iso_2023 if c not in used])
landing = {}
for factor, (c, _) in mccall_to_2023.items():
    landing.setdefault(c, []).append(factor)
print("maintainability now holds:", landing["maintainability"])

correctness       -> functional suitability (functional correctness)
reliability       -> reliability (faultlessness)
efficiency        -> performance efficiency (time behaviour, resource utilization)
integrity         -> security (confidentiality, integrity)
usability         -> interaction capability (operability, learnability)
maintainability   -> maintainability (analysability, modifiability)
flexibility       -> maintainability (modifiability)
testability       -> maintainability (testability)
portability       -> flexibility (adaptability, installability)
reusability       -> maintainability (reusability)
interoperability  -> compatibility (interoperability)
2023 characteristics with no McCall factor: ['safety']
maintainability now holds: ['maintainability', 'flexibility', 'testability', 'reusability']

Three things come out of the mapping. Safety is the one characteristic with no ancestor in McCall, which is the 2023 edition's most important addition. Maintainability absorbs four of McCall's factors: his maintainability, flexibility, testability and reusability are all, in 2023, about how effectively a product can be modified. And there is a trap in the names: McCall's flexibility (the effort to modify a program) now sits under ISO's maintainability, while ISO's flexibility is what used to be called portability. The same word, two meanings, fifty years apart.

Quality in use: the other model

ISO/IEC 25010:2011 contained a second model, quality in use, which describes the effect of the product on its users in a real context: effectiveness, efficiency, satisfaction, freedom from risk and context coverage. The current SQuaRE series places the quality-in-use model in its own standard, ISO/IEC 25019:2023, whose vocabulary SEVOCAB also carries. The distinction is worth knowing for an examination: product quality is a property of the software, measurable in testing; quality in use is the result of using it, measured with real users in real contexts.

Worked example: ExamReg's quality requirements in 2023 terms

The last row is as important as the others. A quality model is a checklist of what to consider, not a list of what every product must have; "not applicable, because" is a valid and useful entry.

What it does not mean

ISO/IEC 9126 is not current. It was replaced in 2011; a modern answer names ISO/IEC 25010:2023 and may mention 9126 as its predecessor.

The 2023 model does not drop usability and portability. It renames them, as interaction capability and flexibility, and adds safety.

McCall's flexibility is not ISO's flexibility. McCall's is about modifying a program; ISO's is about adapting it to new requirements, contexts or environments.

Every product need not score on every characteristic. The model is used to decide which characteristics matter, and "not applicable" is a legitimate answer.

Quick revision

• ISO/IEC 9126 (1991, 2001): six characteristics: functionality, reliability, usability, efficiency, maintainability, portability.
• ISO/IEC 25010:2011: eight; adds compatibility and security; functionality became functional suitability, efficiency became performance efficiency.
• ISO/IEC 25010:2023 (current, November 2023): nine: functional suitability, performance efficiency, compatibility, interaction capability (was usability), reliability, security, maintainability, flexibility (was portability), safety (new).
• New subcharacteristics include faultlessness, resistance, inclusivity, user engagement, self-descriptiveness, scalability and the five of safety.
• McCall's factors map into the 2023 model; only safety has no McCall ancestor; maintainability absorbs four McCall factors.
• Quality in use (effect on users) is a separate model, now ISO/IEC 25019:2023.

Test yourself

1. Name the nine quality characteristics of ISO/IEC 25010:2023. Functional suitability, performance efficiency, compatibility, interaction capability, reliability, security, maintainability, flexibility and safety.

2. What changed between ISO/IEC 25010:2011 and 2023? Usability was renamed interaction capability, portability was renamed flexibility, and safety was added as a ninth characteristic, with new subcharacteristics such as faultlessness, resistance, inclusivity and scalability.

3. List the six characteristics of ISO/IEC 9126 and say what replaced it. Functionality, reliability, usability, efficiency, maintainability and portability. It was replaced by ISO/IEC 25010:2011, which has since been revised as ISO/IEC 25010:2023.

4. Define security and give its subcharacteristics in the 2023 model. The capability of a product to protect information and data so that persons or other products have the degree of data access appropriate to their authorisation. Its subcharacteristics are confidentiality, integrity, non-repudiation, accountability, authenticity and resistance.

5. Which McCall factor has a namesake with a different meaning in ISO/IEC 25010:2023? Flexibility. McCall's flexibility is the effort to modify an operational program, which now falls under maintainability; ISO's flexibility is the capability to be adapted to new requirements, contexts or environments, which was formerly portability.

6. Distinguish product quality from quality in use. Product quality is a property of the software itself, described by the nine characteristics and measured largely in testing. Quality in use is the effect of the product on its users in a real context, such as their effectiveness, satisfaction and freedom from risk.

Chapter Forty-Four takes the system-level types among these (recovery, security, stress, performance and deployment testing) in detail, and Chapters Forty-Five and Forty-Six the load and cross-browser testing that the practical sets.

Testability: the factor that governs all the others

One characteristic is different from the rest, because it is about testing itself. ISO/IEC 25010:2023 defines testability as the "capability of a product to enable an objective and feasible test to be designed and performed to determine whether a requirement is met". It sits under maintainability, and McCall made it a factor of its own in 1977.

A product with poor testability makes every other test expensive or impossible. If ExamReg's late-fee code reads the real system clock and nothing else, no test can check the fee for "16 days late" without waiting sixteen days. If the fee service can only talk to the real payment gateway, every test costs real money. Testability is designed in: a clock that tests can set, a gateway that tests can replace with a simulator, logs that show what happened. Testers who review designs (Chapter Eighteen, on the role of testing in each phase) ask for these things, because they cannot be added cheaply later.

Quality requirements must be measurable

A quality factor becomes testable only when its requirement is stated with a number and a condition. The same rule met in the last two chapters bears repeating here, because non-functional requirements break it most often:

Deciding where the effort goes: product risk

A project cannot test every characteristic deeply, so it tests hardest where failure would hurt most. The ISTQB syllabus calls the approach product risk analysis, and connects it directly to the quality model: product risks "are related to the product quality characteristics (e.g., described in the ISO 25010 quality model)". A risk has two attributes, likelihood and impact, and together they express the risk level. In the quantitative approach, the syllabus says, "the risk level is calculated as the multiplication of risk likelihood and risk impact."

The results decide, in the syllabus's list, the test scope, the test levels and types, the techniques and coverage, the effort for each task, and the order of testing, "to find the critical defects as early as possible".

Worked example: sharing ExamReg's test effort by risk

The ExamReg test lead scores each relevant characteristic from 1 to 5 for likelihood of failure and for impact if it fails. The scores are judgements, written down so they can be argued with. The program multiplies them into risk levels, as the syllabus describes, and shares 400 hours of test effort in proportion.

EFFORT_HOURS = 400
risks = {                          # characteristic: (likelihood 1-5, impact 1-5)
    "functional suitability": (4, 5),   # many fee and eligibility rules; wrong fee harms students
    "performance efficiency": (4, 4),   # a last-day rush never tested before
    "security":               (3, 5),   # student data and payments
    "interaction capability": (3, 3),   # first-year students, once a semester
    "reliability":            (2, 4),   # a crash during the window is serious
    "compatibility":          (3, 2),   # gateway adapter, several browsers
    "maintainability":        (2, 2),   # fee table changes each session
    "flexibility":            (1, 2),   # one server
}
levels = {c: l * i for c, (l, i) in risks.items()}
total = sum(levels.values())
print(f"{'characteristic':<24}{'L':>3}{'I':>3}{'level':>7}{'share':>8}{'hours':>7}")
for c in sorted(levels, key=lambda c: -levels[c]):
    l, i = risks[c]
    share = levels[c] / total
    print(f"{c:<24}{l:>3}{i:>3}{levels[c]:>7}{share:>8.1%}{round(EFFORT_HOURS * share):>7}")
print(f"{'total':<24}{'':>6}{total:>7}")

characteristic            L  I  level   share  hours
functional suitability    4  5     20   25.0%    100
performance efficiency    4  4     16   20.0%     80
security                  3  5     15   18.8%     75
interaction capability    3  3      9   11.2%     45
reliability               2  4      8   10.0%     40
compatibility             3  2      6    7.5%     30
maintainability           2  2      4    5.0%     20
flexibility               1  2      2    2.5%     10
total                              80

The table turns a quality model into a test plan. Functional suitability gets the most effort, but performance efficiency and security together get more than a third of the total, far more than a team that thought only of functional testing would have given them. Flexibility gets a few hours of installation testing. And because the scores are written down, the exam cell can challenge them: if they believe a crash during the window is catastrophic, they can raise reliability's impact to 5 and rerun the arithmetic.

What it does not mean

Non-functional testing is not optional extra testing. It is testing of eight of the nine characteristics, and its failures are often the most visible.

A risk score is not an objective measurement. It is a structured judgement; its value is that it is written down, argued over and revisited.

Testability is not the testers' responsibility alone. It is designed into the product by architects and developers, and testers ask for it in reviews.

Each characteristic does not need its own separate test phase. Many non-functional tests reuse functional tests with an extra check, and several run at every level.

Quick revision

• Quality factors decide the test types, the testability needed, the priorities and the timing of testing.
• Functional testing checks functional suitability (completeness, correctness, appropriateness); non-functional testing checks "how well the system behaves" against the other characteristics.
• Pairs: performance efficiency with performance, load, stress and volume testing; compatibility with interoperability and cross-browser testing; interaction capability with usability and accessibility testing; reliability with reliability and recovery testing; security with security testing; maintainability with static analysis and maintainability testing; flexibility with portability and installation testing; safety with safety analysis.
• Testability: the capability of a product to enable an objective and feasible test to be designed and performed (ISO/IEC 25010:2023); designed in, not added later.
• Product risk: likelihood and impact; quantitatively, risk level = likelihood × impact (ISTQB); used to set scope, types, techniques, effort and order.
• Worked example: 400 hours shared by risk level; functional suitability first, performance efficiency and security next.

Test yourself

1. How do software quality factors affect testing? They decide which test types are needed, one or more for each characteristic that matters; they require quality requirements to be measurable and the product to be testable; they set priorities through product risk analysis; and they push some non-functional testing early, because late non-functional defects are the most dangerous.

2. Distinguish functional from non-functional testing, with one example of each. Functional testing checks what the system does against its functional requirements, for example that a form 3 days late is charged Rs 100. Non-functional testing checks how well it behaves, for example that the fee page appears within 2 seconds for 500 simultaneous users.

3. Which test types check performance efficiency and security? Performance efficiency: performance, load, stress and volume testing. Security: security testing, including vulnerability scanning and penetration testing.

4. What is testability, and how is it achieved? The capability of a product to enable an objective and feasible test to be designed and performed to show a requirement is met. It is achieved in design, for example by letting tests set the clock, replace external services with simulators, and read logs of what happened.

5. How is a product risk level calculated, and what is it used for? In the quantitative approach, as risk likelihood multiplied by risk impact. It is used to decide the test scope, the levels, types and techniques, the effort for each area, and the order of testing.

The word "implied" in the last line matters as much as any. A student never writes down the portal must not show my marks to someone else, but it is an expectation that is generally implied, and a portal that breaks it has poor quality however well it meets its written specification. The standards' word for fulfilling a requirement is conformity, and its opposite, a nonconformity, is the "non-fulfillment of a requirement".

Fitness for use, and conformance to requirements

ASQ, the American Society for Quality, records in its glossary that quality is "A subjective term for which each person or sector has its own definition", and gives two technical meanings: "the characteristics of a product or service that bear on its ability to satisfy stated or implied needs" and "a product or service free of deficiencies". It then names the two classic short definitions: "According to Joseph Juran, quality means 'fitness for use'; according to Philip Crosby, it means 'conformance to requirements.'"

The last row connects this chapter to the rest of the syllabus. Checking conformance to requirements is verification; checking fitness for use is validation. Chapter Four's seventh principle, the absence-of-defects fallacy, is exactly the gap between the two: software can conform perfectly and still not be fit for use. Verification and validation have their own chapter (Chapter Twenty-Six).

Garvin's five approaches to quality

Garvin's article identifies "five major approaches to the definition of quality": "(1) the transcendent approach of philosophy; (2) the product-based approach of economics; (3) the user-based approach of economics, marketing, and operations management; and (4) the manufacturing-based and (5) value-based approaches of operations management." Each is a way a real person judges software.

The practical lesson is that a quality plan has to satisfy several approaches at once, because a project has stakeholders who use each of them. ExamReg's tests serve the manufacturing-based view; its usability sessions serve the user-based view; its budget serves the value-based view. A project that pleases only one of its judges has not delivered quality in the eyes of the others.

Quality is not grade

One distinction clears up a common confusion. SEVOCAB defines grade as a "scalar, qualitative measure or ranking to distinguish the relative fitness for use of similar items". A budget smartphone and a premium one are different grades; either can be of high or low quality. A low-grade product is not a failure; a low-quality product is.

For software, grade is the set of features and capabilities chosen (a basic ExamReg without online payment is a lower grade than one with it); quality is how well whichever grade was chosen fulfils its requirements. A basic ExamReg that charges every fee correctly and never crashes has higher quality than a feature-rich one that overcharges students.

Satisfaction: quality as the customer experiences it

The standards also name the outcome that quality is for. Customer satisfaction is the "state of fulfillment in which the needs of a customer are met or exceeded for the customer's expected experiences as assessed by the customer at the moment of evaluation" (SEVOCAB, from ISO/IEC/IEEE 24765). Three phrases in it are worth an examiner's attention: the needs are met or exceeded; it is assessed by the customer, not by the supplier; and it is judged at the moment of evaluation, so it can change. A portal that satisfied students in May can disappoint them in November, when their expectations have risen or a competitor has shown them something better.

Worked example: is ExamReg release 2.0 of good quality?

A single-word answer, yes, would hide everything useful in the table. The honest answer is that release 2.0 has high conformance, meets the implied needs that were checked, is fit for use with one improvement, and is good value at its grade.

What it does not mean

Quality does not mean luxury or expense. It is the degree to which requirements are fulfilled; a simple, cheap product can have excellent quality.

Quality is not only conformance to the written specification. Implied needs count, and a product can conform to a wrong specification.

Quality is not only the user's opinion either. Satisfaction is assessed by the customer, but conformance is measured objectively; both matter.

Quality is not a yes or no. ISO's definition begins with "degree".

Quick revision

• Quality (ISO 9000:2026, via ISO): "degree to which a set of inherent characteristics ... of an object ... fulfils requirements".
• Requirement: a need or expectation that is stated, generally implied, or obligatory. Conformity: fulfilment of a requirement; nonconformity: non-fulfilment.
• Juran: fitness for use (validation). Crosby: conformance to requirements (verification).
• Garvin (MIT Sloan Management Review, Fall 1984): five approaches: transcendent, product-based, user-based, manufacturing-based, value-based.
• Grade ranks similar items by capability; it is not quality.
• Customer satisfaction: needs met or exceeded, as assessed by the customer at the moment of evaluation.

Test yourself

1. Define quality as ISO 9000 does, and explain its key terms. Quality is the degree to which a set of inherent characteristics of an object fulfils requirements. Degree makes it a scale; inherent characteristics belong to the object itself, not assigned ones like price; an object can be a product, service, process or organisation; and requirements are needs or expectations that are stated, generally implied or obligatory.

2. Distinguish Juran's and Crosby's definitions of quality. Juran defined quality as fitness for use: whether the product serves the user's actual purpose. Crosby defined it as conformance to requirements: whether the product meets what was specified. The first corresponds to validation, the second to verification.

3. List Garvin's five approaches to quality with an example of each. Transcendent (recognised on sight), product-based (the amount of desirable attributes, such as features), user-based (whatever satisfies the user), manufacturing-based (conformance to specification) and value-based (performance at an acceptable price).

4. Distinguish quality from grade. Grade ranks similar items by their capabilities or features; quality is how well an item of any grade fulfils its requirements. A low-grade product can have high quality, and a high-grade one low quality.

5. Why does the word "implied" in the definition of requirement matter to a tester? Because users expect things they never write down, such as privacy of their data, and a product that breaks an implied need has poor quality even if it meets its written specification; testers must therefore test implied needs too.

That does not mean software quality stays fixed. Lyu names the two ways it falls: "software may experience reliability decrease due to abrupt changes of its operational usage or incorrect modifications to the software." Both happen to ExamReg every session.

• A change in use. The code that handled a few dozen forms a day in the first week meets 300 on the last date. Nothing in the program has changed, but it is being used as it never was before, and a fault that no ordinary day reached is reached.
• A change in the code. The exam cell revises the fee table, a developer edits the fee function, and the edit breaks a case that used to work. Every correction is itself a change to a design, and can bring a new fault with it. Guarding against that is the job of regression testing (Chapter Thirty-Nine).

The same passage names the good news. Software "generally enjoys reliability growth during testing and operation", because each fault found and removed is gone for good from every copy.

Chapter Ninety, on software reliability, returns to this contrast with the failure curves of hardware and software.

Brooks: four properties that make software quality hard

In "No Silver Bullet", a paper first given at the IFIP World Computing Conference in 1986 and later reprinted in his book The Mythical Man-Month, Frederick Brooks separated the difficulties of software into accidental ones, which better tools can remove, and essential ones, which belong to software itself. He named "the inherent properties of this irreducible essence of modern software systems: complexity, conformity, changeability, and invisibility." Each is a reason quality is hard to achieve in software development, and each makes a demand on testing.

Complexity. "Software entities are more complex for their size than perhaps any other human construct, because no two parts are alike (at least above the statement level)." Brooks traced unreliability straight to it: "From the complexity comes the difficulty of enumerating, much less understanding, all the possible states of the program, and from that comes the unreliability." For a tester this is why exhaustive testing is impossible, the second of the seven principles in Chapter Four, and why test design techniques exist: to choose, from far more states than anyone could try, the few tests most likely to find faults.

Conformity. A physicist can hope that nature obeys a few simple laws. Software must instead fit interfaces that other people designed for their own reasons, and Brooks observed that "much complexity comes from conformation to other interfaces; this cannot be simplified out by any redesign of the software alone." ExamReg must fit the paper codes it is given, the exam cell's fee rules, the payment gateway's protocol and the browsers students use. Its developers can simplify none of them, and each is a place where the program can be right by its own logic and wrong against the thing it must fit. Integration testing and compatibility testing exist for exactly these seams.

Changeability. "All successful software gets changed." Software, Brooks wrote, "is pure thought-stuff, infinitely malleable", and "The pressures for extended function come chiefly from users who like the basic function and invent new uses for it." A program that will certainly change has to be built to be changed, which is why maintainability and flexibility are quality characteristics in their own right in the quality model today (Chapter Twenty), and why every change needs its regression tests.

Invisibility. "Software is invisible and unvisualizable." A building has a floor plan, on which "Contradictions become obvious, omissions can be caught"; a program's structure, when anyone tries to draw it, turns out to be "not one, but several, general directed graphs, superimposed one upon another." Nobody can look at a program and see whether it is good, the way an inspector can look at a weld. Its quality has to be made visible by other means: documents that can be reviewed, models, the measures of Module 2, and tests that turn behaviour into evidence.

Three views of quality: internal, external and in use

The last difference is where quality is seen from. The standards distinguish three views, and each is judged by different people with different evidence.

Internal quality is the "totality of attributes of a product that determine its ability to satisfy stated and implied needs when used under specified conditions" (ISO/IEC/IEEE 24765). It is the quality of the code, design and documents themselves, judged without running the program: by reviews, by static analysis, and by internal measures, each a "measure of the product itself, either direct or indirect", such as the size of a module or its complexity.

External quality is the "extent to which a product satisfies stated and implied needs when used under specified conditions". It is judged by running the software, as testers do. An external measure is an "indirect measure of a product derived from measures of the behavior of the system of which it is a part", such as the number of failures in a test cycle or the response time under load.

Quality in use is, in ISO/IEC 25019:2023, the "extent to which the system or product, when it is used in a specified context of use, satisfies or exceeds" what its stakeholders need "to achieve specified beneficial goals or outcomes". The context of use is the "combination of users, goals and tasks, resources, and environment" (ISO TR 25060:2023). Quality in use is judged by the people who use the software, for their own purposes, in their own conditions. ISO's own page describes the 2023 standard as "a quality-in-use model composed of three characteristics". They are beneficialness, the "extent of benefit resulting from the use of a product, system, or service"; freedom from risk, the "extent to which a product or system mitigates the potential risk to economic status, human life, health, society, financial values, enterprise activities, or the environment"; and acceptability.

Figure 23.1 Three views of software quality: internal, external and in use

The older product quality standard, ISO/IEC 9126, linked the three views in the chain the figure shows. Internal quality influences external quality: well-structured, well-reviewed code is more likely to behave well when it runs. External quality influences quality in use: software that behaves well under test is more likely to serve its users. Read the other way, each depends on the one before it: quality in use cannot be had without external quality, nor external quality without internal quality. The influence is real, but it is not a guarantee, and the gaps between the views are where some of the most instructive failures live. The worked example below is one of them.

Who judges software quality

Each of these people can be satisfied while another is not. The developers can be proud of clean code that the students find confusing; the students can be happy with a form that the IT cell must restart every night. This is Garvin's point from Chapter Twenty-Two, on what quality means, seen from inside a software project: a quality plan has to ask every one of them.

Worked example: one test result, two very different qualities in use

Two builds of ExamReg's fee function each contain one defect. Build A has lost the on-time case, Chapter One's defect from the chapter on what software testing is: it charges a form submitted on or before the last date as if it were late. Build B refuses a form submitted on the fifteenth day late, which the rule still accepts. The portal records any on-time form as 0 days late, and the test team runs one test for each value from 0 to 20 days late; both builds fail exactly one test. By that external measure, they are equally good.

The exam cell also has last session's record of how many forms arrived on each day. The counts below are this book's illustration, not a real college's data. The program weighs each build's failures by how many students would actually have met them.

def rule(days_late):                     # the exam cell's rule; 0 means on or before the last date
    if days_late < 0 or days_late > 15:
        raise ValueError("form not accepted")
    if days_late == 0:
        return 0
    if days_late <= 7:
        return 100
    return 500

def build_a(days_late):                  # defect A: Chapter One's, the on-time case lost
    if days_late < 0 or days_late > 15:
        raise ValueError("form not accepted")
    if days_late <= 7:
        return 100
    return 500

def build_b(days_late):                  # defect B: the fifteenth day refused
    if days_late < 0 or days_late >= 15:
        raise ValueError("form not accepted")
    if days_late == 0:
        return 0
    if days_late <= 7:
        return 100
    return 500

def outcome(fee, days_late):
    try:
        return fee(days_late)
    except ValueError:
        return "refused"

days = list(range(0, 21))                # one test for each value of days late, 0 to 20
forms = [720,                            # last session's forms on or before the last date
         50, 30, 25, 20, 15, 12, 28,     # 1 to 7 days late
         18, 10, 8, 7, 6, 5, 9, 14,      # 8 to 15 days late
         8, 6, 4, 3, 2]                  # 16 to 20 days late (refused, as the rule says)
total = sum(forms)
print("tests:", len(days), " forms in the session:", total)
for name, build in [("build A", build_a), ("build B", build_b)]:
    wrong = [d for d in days if outcome(build, d) != outcome(rule, d)]
    hit = sum(n for d, n in zip(days, forms) if d in wrong)
    print(f"{name}: wrong on day {', '.join(map(str, wrong))};"
          f" tests failed {len(wrong)} of {len(days)} ({len(wrong) / len(days):.1%});"
          f" students hit {hit} of {total} ({hit / total:.1%})")

tests: 21  forms in the session: 1000
build A: wrong on day 0; tests failed 1 of 21 (4.8%); students hit 720 of 1000 (72.0%)
build B: wrong on day 15; tests failed 1 of 21 (4.8%); students hit 14 of 1000 (1.4%)

By the test results the two builds are identical: each fails 1 of its 21 tests, about 4.8 per cent. In use they are nothing alike. Build A's defect meets every student who submitted on time, which is most of them: it would have charged 720 of the 1,000 students a late fee they did not owe, 72 per cent of all users. Build B's defect would have met 14 students, 1.4 per cent.

But a count of students is not the whole of quality in use either. Build A's harm is Rs 100 taken wrongly, which can be refunded. Build B's harm is a refused form: under the rule, those 14 students cannot sit the examination unless somebody notices in time. Quality in use includes freedom from risk, and a defect that meets few users can still do the most damage. A tester who reports only 1 of 21 tests failed for each build has told the exam cell almost nothing it needs to know. The defect report has to say who is hit and how badly, which is what severity and priority in a defect report (Chapter Seventy-Six) exist to record.

The general lesson is about test design. A test count treats every input as equally important; users do not. The on-time case deserves more tests, and more careful ones, than the eleventh day late, because that is where the students are.

When is software good enough?

No release is free of defects; testing can show that defects are present but never that they are absent, the first of the seven principles in Chapter Four. Every release is therefore a decision that the software is good enough. The NIST study of 2002 called this the central difficulty: "The major problem for the software industry is deciding when a firm should stop testing". It reported that commercial developers decide with a combination of rules of thumb: a sufficient percentage of test cases passing, statistics from a code coverage tool, counts and trends of defects by severity, beta testing by real users, and the number of new problem reports falling below a threshold. NIST described these as nonanalytical: none of them calculates the risk that remains.

The three views make the decision more honest. Internal quality asks whether any serious review finding is still open. External quality asks whether the exit criteria are met: the planned tests run, the pass rate reached, no critical defect open. Quality in use asks whether the software has been tried in its real context of use, with its real users and its busiest day. The worked example shows why the third question cannot be skipped. A release rule of at least 95 per cent of tests passed would let build A through, with 20 of 21 tests passed, about 95.2 per cent, and it would still overcharge 720 students.

The cost of getting it wrong

Chapter Three, on why software must be tested, showed what poor quality has cost when software fails in the field, from a lost rocket to a national estimate of $59.5 billion a year. ExamReg's scale is smaller but the pattern is the same. Released, build A's defect would have taken 720 × 100 = 72,000 rupees from students who owed nothing, and the college would then pay again: in refunds, corrected receipts, apologies and an emergency release made under pressure. Found in a review of the code, the same defect would have cost one changed line and a retest. Chapter One Hundred One, on the cost of quality, turns this into the four kinds of quality cost and shows how to count them.

What it does not mean

Software not wearing out does not mean its quality is permanent. Its reliability falls when its use changes or when a change to the code brings in a new fault.

Internal quality is not a private concern of developers. External quality and quality in use are built on it, and it decides how costly every later change will be.

Passing tests is not the same as quality in use. Tests measure external quality over the inputs someone chose; users meet the inputs their context produces.

Good enough is not an excuse for poor quality. It is a decision, made with evidence, that the remaining risk is acceptable to the people who will carry it.

Quick revision

• Software quality: "capability of a software product to satisfy stated and implied needs when used under specified conditions" (ISO/IEC 25000:2014); "degree to which a software product meets established requirements" (IEEE 730-2014).
• Software faults are design faults, and every copy is identical, so quality must be built in during development, not inspected in at the end.
• Software does not wear out (Lyu 1996), but its reliability falls with a change in use or a faulty change to the code, and grows as faults are found and removed.
• Brooks (1986), the essential properties of software: complexity, conformity, changeability, invisibility.
• Internal quality: the product itself, judged without running it. External quality: the software running, judged by tests. Quality in use: in a context of use (users, goals and tasks, resources, environment); ISO/IEC 25019:2023 has three characteristics: beneficialness, freedom from risk, acceptability.
• Internal quality influences external quality, which influences quality in use (the chain of ISO/IEC 9126).
• Worked example: two builds each failed 1 of 21 tests; in use one would have hit 720 of 1,000 students, the other 14 students with a worse harm.
• Deciding when software is good enough is, in NIST's words of 2002, the major problem; decide with all three views.

Test yourself

1. What is software quality? Give two standard definitions. ISO/IEC 25000:2014 defines it as the capability of a software product to satisfy stated and implied needs when used under specified conditions. IEEE 730-2014 defines it more narrowly as the degree to which a software product meets established requirements.

2. Why is the quality of software different from the quality of a manufactured product? Software's faults are design faults, not physical ones, and every copy is identical, so quality cannot be controlled by inspecting copies; it must be built in during development. Software does not wear out, but its reliability falls when its use changes or when a change to the code introduces a fault.

3. Explain Brooks's four essential properties of software and their effect on testing. Complexity: software has more states than can be listed, so exhaustive testing is impossible and tests must be designed. Conformity: it must fit interfaces designed by others, which calls for integration and compatibility testing. Changeability: all successful software gets changed, which calls for maintainable design and regression testing. Invisibility: its structure cannot be seen, so quality must be made visible through reviews, measures and tests.

4. Distinguish internal quality, external quality and quality in use, with an example of each for an online registration portal. Internal quality is the quality of the product itself, judged without running it, for example the complexity of the fee code found in a review. External quality is the quality of the running software, judged by testing, for example the number of failures in a test cycle. Quality in use is the quality experienced by real users in their context of use, for example whether students register correctly and without harm on the last date.

5. Two builds each fail one of 21 tests. Why might one be far worse than the other? Because the test count weighs every input equally, while real users do not arrive equally. A defect in the commonest case, such as an on-time form, can hit hundreds of users while a defect on a rare day hits a few, and a defect that hits few can still do more harm, such as refusing a registration. Quality in use depends on the context of use and on the severity of the harm.

A note for readers who meet older copies. The release notes of version 4.0.1 record that, in this section, the word QC was replaced with testing, "because the section compares QA with testing, not with QC". An older copy may therefore call QC product-oriented and corrective where version 4.0.1 says it of testing. The two versions agree on the substance: the product-oriented, corrective side of quality work is quality control, and testing is its largest part.

Quality assurance: confidence from the process

Quality assurance is defined by what it gives: confidence. ISO/IEC/IEEE 12207:2026 calls it a "process that is focused on providing confidence that quality requirements are fulfilled"; ISO/IEC/IEEE 15288:2023 places it as the "part of quality management focused on providing confidence that quality requirements are fulfilled". The word itself has a standard meaning: assurance is "grounds for justified confidence that a claim has been or will be achieved" (ISO/IEC/IEEE 15026-1:2025). QA's product, in other words, is evidence that the organisation's way of working can be trusted to produce what was promised.

The ISTQB syllabus gives its method: "QA is a process-oriented, preventive approach that focuses on the implementation and improvement of processes. It works on the basis that if a good process is followed correctly, then it will generate a good product." And its scope: "QA applies to both the development and testing processes, and is the responsibility of everyone on a project."

Its typical activities act on the way work is done:

• Defining processes and standards: how requirements are written, how code is reviewed, what a test plan must contain, when a change may be released.
• Training people to follow them.
• Audits: an audit is an "independent examination of a work product or set of work products to assess compliance with specifications, standards, contractual agreements, or other criteria" (ISO/IEC/IEEE 12207:2026). A QA audit asks whether the process was followed, and produces findings about the process.
• Measuring and improving the process, using the data that quality control produces, which is the subject of the worked example below.

Prevention and detection

The simplest way to hold the two apart is by what they do to a defect. Quality assurance tries to stop a defect from being made; quality control tries to find the defects that were made anyway.

ExamReg shows the difference on a single kind of defect. Suppose the fee form keeps accepting impossible input, such as a negative number of backlog papers. Quality control finds each instance: a tester enters -1, the form accepts it, a defect report is written, the developer fixes that field, and the test is added to the regression suite. Next release, a different form has the same kind of defect, and the cycle repeats. Quality assurance asks why the same kind keeps appearing, and changes the process: a shared validation component that every form must use, a rule in the coding standard, and an item on the code review checklist. The defect stops being made.

Neither replaces the other. Prevention is never perfect, so detection is always needed; and detection alone is expensive, because each defect is found only after it has been built, and a process that keeps making the same mistake keeps paying for it.

The same results, used twice

The ISTQB syllabus names the point where the two meet: "Test results are used by QA and testing. In testing they are used to fix defects, while in QA they provide feedback on how well the development and test processes are performing." The program below reads the ExamReg release 2.0 defect data, fixed once for this whole book, first as quality control reads it and then as quality assurance does.

found_by = {"requirements review": 16, "design review": 24, "code review": 34,
            "unit testing": 44, "integration testing": 32, "system testing": 28,
            "acceptance testing": 10, "after release": 12}
by_type = {"input validation": 58, "logic and computation": 44, "interface": 30,
           "user interface": 24, "data and database": 18, "documentation": 12,
           "performance": 8, "security": 6}
total = sum(found_by.values())
assert total == sum(by_type.values()) == 200          # ExamReg release 2.0

# Quality control reads the data for the PRODUCT: what is wrong, and was it caught in time?
escaped = found_by["after release"]
print(f"QC: {total} defects in release 2.0; {total - escaped} found before release;"
      f" {escaped} found by students after it")

# Quality assurance reads the same data for the PROCESS: where do defects come from,
# and which activities catch them?
reviews = sum(n for activity, n in found_by.items() if activity.endswith("review"))
tests = sum(n for activity, n in found_by.items() if activity.endswith("testing"))
print(f"QA: caught by reviews {reviews} ({reviews / total:.0%}), by tests {tests}"
      f" ({tests / total:.0%}), by students {escaped} ({escaped / total:.0%})")
commonest = max(by_type, key=by_type.get)
print(f"QA: the commonest kind is {commonest}: {by_type[commonest]} of {total}"
      f" ({by_type[commonest] / total:.0%})")

QC: 200 defects in release 2.0; 188 found before release; 12 found by students after it
QA: caught by reviews 74 (37%), by tests 114 (57%), by students 12 (6%)
QA: the commonest kind is input validation: 58 of 200 (29%)

Quality control's reading is about this release. There were 200 defects; 188 were caught before release and 12 reached students. Every one must be fixed and its fix confirmed, and the 12 in production come first, because students are meeting them now.

Quality assurance's reading is about the next release. Reviews caught 74 defects, 37 per cent, before any code ran, which says the review process is earning its time. But 12 escaped, 6 per cent of the total, and the commonest kind of defect, input validation, is 58 of 200, 29 per cent: nearly three in every ten defects are the same kind of mistake. That is a process signal, not a product one. The response is the one in the last section: a shared validation component, a coding rule and a review checklist item, followed by a check in release 2.1 that the count has fallen. Chapter Eighty, on using defect data to improve the process, takes this analysis much further.

A review is quality control; checking that reviews happen is quality assurance

One pair of activities causes more confusion than any other, because both involve reading documents.

• A code review of the fee function reads the product and finds its defects. It is quality control, of a static kind.
• An audit at the end of a sprint, which checks whether every change to the fee rules actually went through a code review as the process requires, reads records of the process. It is quality assurance.

The rule that settles every such case is to ask what the activity examines. If it examines the product, it is quality control; if it examines or changes the way the product is made, it is quality assurance.

QA and QC side by side

Worked contrast: the ExamReg project's quality activities

What it does not mean

Quality assurance is not another name for testing. Testing is quality control; QA is about the process, and a tester's job title does not change that.

Quality control is not only testing. Reviews and inspections of documents and code examine the product too.

Quality assurance does not make quality control unnecessary. A good process lowers the number of defects made; it never lowers it to zero.

Quality assurance is not one department's job alone. In the syllabus's words, it "is the responsibility of everyone on a project", even when a separate SQA group leads it.

Quick revision

• QA: "process that is focused on providing confidence that quality requirements are fulfilled" (ISO/IEC/IEEE 12207:2026); "part of quality management" (ISO/IEC/IEEE 15288:2023). Process-oriented, preventive.
• QC: "tasks to evaluate the quality of services as delivered or the quality of developed or manufactured products" (ISO/IEC/IEEE 12207:2026). Product-oriented, corrective.
• Assurance: grounds for justified confidence that a claim has been or will be achieved.
• Testing is "a major form of quality control"; so are reviews and inspections of work products (ISTQB v4.0.1, section 1.2.2).
• QA prevents defects; QC detects them. Both are needed.
• The same test results serve both: QC uses them to fix defects, QA as feedback on the process.
• A review of the product is QC; an audit of whether reviews happen is QA.
• ExamReg release 2.0: 188 of 200 defects caught before release, 12 after; reviews caught 74 (37 per cent); input validation is 58 of 200 (29 per cent), a process signal.

Test yourself

1. Define quality assurance and quality control as the standards do. Quality assurance is a process focused on providing confidence that quality requirements are fulfilled (ISO/IEC/IEEE 12207:2026), and part of quality management (ISO/IEC/IEEE 15288:2023). Quality control is the tasks that evaluate the quality of delivered services or of developed or manufactured products (ISO/IEC/IEEE 12207:2026).

2. Distinguish QA from QC under five headings. Focus: QA the process, QC the product. Aim: QA prevents defects and gives confidence, QC finds defects to be corrected. Approach: QA preventive, QC corrective. Timing: QA throughout, from before the product exists; QC once a work product exists. Activities: QA defines, audits and improves processes; QC tests, reviews and inspects work products.

3. Is testing quality assurance or quality control? Explain. Quality control. It examines the product by running it and finds defects so that they can be corrected; the ISTQB syllabus calls it product-oriented and corrective, and a major form of quality control, while QA is process-oriented and preventive.

4. How can the same test results be used for both QA and QC? QC uses them to fix defects in the product. QA uses them as feedback on the process: for example, finding that 58 of ExamReg's 200 defects were input validation defects shows a weakness in how code is written and reviewed, which QA corrects with a shared validation component, a coding rule and a checklist item.

5. Classify with reasons: a code review of a module, and an audit of whether code reviews were held. The code review is quality control, because it examines the product for defects. The audit is quality assurance, because it examines whether the process was followed.

The standard an organisation's quality management system can be certified against is ISO 9001, whose requirements, in ISO's own words, "define how to establish, implement, maintain, and continually improve a quality management system (QMS)." Its current edition, ISO 9001:2026, and the family around it are the subject of Chapters Ninety-Four and Ninety-Five.

Software quality assurance

For software, IEEE 730 applies the same ideas. It defines software quality management in words that mirror the general definition: "coordinated activities to direct and control an organization with regard to software quality". Within it sits SQA, defined as the "set of activities that define and assess the adequacy of software processes to provide evidence that establishes confidence that the software processes are appropriate for and produce software products of suitable quality for their intended purposes."

Every phrase of that definition does work, and together they make a complete short answer.

1. Define and assess. SQA both sets the processes (what the project will do, and to what standard) and checks them.
2. The adequacy of software processes. Its object is the process, which places SQA on the quality assurance side of Chapter Twenty-Four's distinction, not the quality control side.
3. Evidence that establishes confidence. Its output is evidence: audit findings, records, measurements, that let the customer and management trust the work.
4. Appropriate for, and produce, software products of suitable quality. It asks two questions: is the process fit for this project, and does it actually yield good products? So SQA looks at products too, as evidence about the process.
5. For their intended purposes. Quality is judged against use, which is fitness for use from Chapter Twenty-Two, on what quality means.

A note on currency. The definitions above are from IEEE 730-2014, the edition whose wording the public software engineering vocabulary (SEVOCAB) still carries. IEEE's own page lists that edition as inactive since 27 March 2025 and superseded by IEEE 730-2026, published on 21 August 2026, which establishes "Requirements for initiating, planning, controlling, and executing the software quality assurance (SQA) processes of a software development or maintenance project". The 2026 text is not freely available; a student quoting the definitions above should name them as IEEE 730-2014's.

Assure, not ensure

IEEE 730-2014 draws a distinction that explains what an SQA group can and cannot do. To assure is "to promise or state with certainty by one person to another person or group"; to ensure is "to make certain that things occur or events take place". An SQA group assures: it gives the customer and management evidence they can rely on. It does not, by itself, ensure quality; the developers who write the code and the testers who check it do that. An SQA group that finds a process being skipped cannot fix the product. What it can do is make the gap visible to the people who can.

Independence: why SQA stands apart from development

Evidence is only worth something if it is objective. A reviewer who reports to the project manager, days before a deadline, is under pressure to find that everything is fine. IEEE 730-2014 therefore defines independence of SQA as the "situation in which SQA is free from technical, managerial, and financial influences, intentional or unintentional", and names three kinds:

A small organisation may not manage all three, and the definitions show what is lost when one is missing: an auditor who wrote the code, reports to the project manager or depends on the project's budget is judging their own side. The ISTQB syllabus's reminder from the last chapter still holds as well: quality assurance "is the responsibility of everyone on a project", even where an independent group leads it.

The four terms compared

Worked example: does ExamReg's software house meet its quality objectives?

Quality management becomes real only when its objectives are measurable and someone checks them. Suppose the software house that builds ExamReg set three quality objectives for the year. The targets are this book's illustration; the measured values come from ExamReg release 2.0's defect data, used throughout the book.

release = {"defects": 200, "critical": 8, "found by reviews": 74, "found after release": 12}

objectives = [            # (objective, measured value in per cent, "at most" or "at least", target)
    ("defects found by reviews", 100 * release["found by reviews"] / release["defects"], "at least", 35),
    ("defects found after release", 100 * release["found after release"] / release["defects"], "at most", 5),
    ("critical defects among all defects", 100 * release["critical"] / release["defects"], "at most", 5),
]
for name, actual, sense, target in objectives:
    met = actual <= target if sense == "at most" else actual >= target
    goal = f"{sense} {target}%"
    print(f"{name:<36}{actual:>5.1f}%   target {goal:<14}{'met' if met else 'NOT MET'}")

defects found by reviews             37.0%   target at least 35%  met
defects found after release           6.0%   target at most 5%    NOT MET
critical defects among all defects    4.0%   target at most 5%    met

Two objectives are met and one is not: 12 of 200 defects, 6 per cent, reached students, against a target of at most 5 per cent. Each of the four terms has a different part to play in what happens next.

• Quality control fixes the 12 defects in the product and confirms each fix.
• Software quality assurance audits the test process for release 2.0 and reports, with evidence, where it let the 12 through: which were missed by a planned test, and which had no test planned at all.
• Quality management takes the missed objective to its management review, decides the improvement (a change to the test process, more time for it, or both), funds it, and sets the objective again for release 2.1.
• Quality assurance, in the broad sense, is the confidence the whole cycle produces: evidence that when an objective is missed, the organisation notices and acts.

Chapter Eighty, on using defect data to improve the process, shows how the analysis of the 12 would be done.

What it does not mean

Quality management is not a department. It is the whole organisation's direction and control of quality, and it starts with top management.

SQA is not testing. It defines and assesses the processes; testing is quality control of the product.

SQA assures; it does not ensure. It supplies objective evidence. The quality itself is made by those who build and check the software.

An SQA group is not the only one responsible for quality. Its independence makes its evidence trustworthy; it does not take the responsibility away from anyone else.

Quick revision

• QM: "coordinated activities to direct and control an organization with regard to quality" (ISO/IEC/IEEE 12207:2026): policy and objectives, then planning, control, assurance and improvement.
• Juran's trilogy: quality planning, quality control, quality improvement.
• QA: "part of quality management" giving confidence (ISO/IEC/IEEE 15288:2023). QC: evaluates the products.
• SQA (IEEE 730-2014): activities that define and assess the adequacy of software processes to provide evidence that establishes confidence that they produce software of suitable quality. IEEE 730-2026, published 21 August 2026, now replaces the 2014 edition.
• Assure (give evidence to others) is not ensure (make it happen).
• SQA independence: technical, managerial, financial.
• Nesting: QM contains QA and QC; SQA is QA for software.
• Worked example: reviews 37 per cent (target at least 35, met); after release 6 per cent (target at most 5, not met); critical 4 per cent (target at most 5, met).

Test yourself

1. Define quality management and name its parts. Quality management is the coordinated activities to direct and control an organization with regard to quality. It sets a quality policy and quality objectives, and achieves them through quality planning, quality control, quality assurance and quality improvement.

2. Define software quality assurance and explain the definition. SQA is the set of activities that define and assess the adequacy of software processes to provide evidence that establishes confidence that the processes are appropriate for, and produce, software of suitable quality for its intended purposes (IEEE 730-2014). It sets and checks processes, its object is the process, its output is evidence, and it judges the process both by its fitness and by the products it yields.

3. Distinguish QM, QA, QC and SQA. QM is the whole: direction and control of quality, from policy to improvement, led by top management. QA is the part of QM that gives confidence that processes will meet quality requirements; it is preventive. QC is the part that evaluates products and finds their defects; it is corrective. SQA is QA applied to the processes of software projects, ideally by a group independent of development.

4. What are the three kinds of SQA independence, and why do they matter? Technical independence (SQA staff are not involved in developing the system), managerial independence (SQA reports to an organisation separate from development and project management) and financial independence (the SQA budget is controlled outside the development organisation). They matter because SQA's value is objective evidence, which pressure from the project would weaken.

5. What is the difference between assuring and ensuring quality? To assure is to state with certainty to others, backed by evidence; to ensure is to make certain that things happen. SQA assures quality by supplying evidence; the developers and testers who build and check the software ensure it.

The general vocabulary combines the two into one definition of V&V: the "process of determining whether the requirements for a system or component are complete and correct, the products of each development phase fulfill the requirements or conditions imposed by the previous phase, and the final system or component complies with specified requirements" (ISO/IEC/IEEE 24765).

Verification and validation side by side

Two cautions about the table. First, validation is not only an end-of-project activity. CMMI says validation "is performed early (concept/exploration phases) and incrementally throughout the product lifecycle", and the cheapest validation of all is showing users the requirements, or a prototype, before anything is built. Second, the two use similar methods. In CMMI's words, "Validation activities use approaches similar to verification (e.g., test, analysis, inspection, demonstration, simulation). Often, the end users and other relevant stakeholders are involved in the validation activities." What makes an activity verification or validation is what it checks against, not the technique.

The V-model (Chapter Fifteen) draws the same line: its lower test levels verify each phase's product against the phase opposite, and acceptance testing at the top validates the system against the users' needs.

Worked example: built right, but not the right thing

ExamReg gives some students a fee concession. As the Scrum sprint in Chapter Seventeen set it out, a concession waives the form fee only: a concession student who submits late still pays the late fee. Suppose, for this example, that the analyst who wrote the specification misread the exam cell's note and wrote that concession students "pay no fees", late fees included. That is the book's invented mistake, but it is an ordinary kind of requirements defect: a rule written down slightly wrong.

The program checks three builds on 42 cases: every value of days late from 0 (the portal's value for any form on or before the last date) to 20, with and without a concession. Verification compares each build with the specification it was coded from; validation compares it with what the exam cell actually needs.

def late_fee(days_late):                 # the late-fee rule; 0 means on or before the last date
    if days_late < 0 or days_late > 15:
        return "refused"
    if days_late == 0:
        return 0
    return 100 if days_late <= 7 else 500

def need(days_late, concession):         # what the exam cell actually applies: a concession
    return late_fee(days_late)           # waives the form fee only; late fees apply to all

def spec_v1(days_late, concession):      # the specification as the analyst wrote it:
    fee = late_fee(days_late)            # concession students pay no fees, late ones included
    return 0 if concession and fee != "refused" else fee

spec_v2 = need                           # the specification after validation corrected it

def build_1(days_late, concession):      # coded faithfully from spec v1
    if days_late < 0 or days_late > 15:
        return "refused"
    if concession or days_late == 0:
        return 0
    return 100 if days_late <= 7 else 500

def build_2(days_late, concession):      # coded from spec v1, with Chapter One's on-time defect
    if days_late < 0 or days_late > 15:
        return "refused"
    if concession:
        return 0
    return 100 if days_late <= 7 else 500

def build_3(days_late, concession):      # coded from spec v2
    if days_late < 0 or days_late > 15:
        return "refused"
    if days_late == 0:
        return 0
    return 100 if days_late <= 7 else 500

cases = [(d, c) for c in (False, True) for d in range(0, 21)]
print(len(cases), "cases: days late from 0 to 20, with and without a concession")
for name, build, label, spec in [("build 1", build_1, "spec v1", spec_v1),
                                 ("build 2", build_2, "spec v1", spec_v1),
                                 ("build 3", build_3, "spec v2", spec_v2)]:
    verified = sum(build(d, c) == spec(d, c) for d, c in cases)
    validated = sum(build(d, c) == need(d, c) for d, c in cases)
    print(f"{name}, coded from {label}: verification {verified} of {len(cases)},"
          f" validation {validated} of {len(cases)}")

42 cases: days late from 0 to 20, with and without a concession
build 1, coded from spec v1: verification 42 of 42, validation 27 of 42
build 2, coded from spec v1: verification 41 of 42, validation 26 of 42
build 3, coded from spec v2: verification 42 of 42, validation 42 of 42

Build 1 is the case to remember. It passes verification perfectly, 42 of 42: every test written from the specification agrees with it, because the developer did exactly what the specification said. It fails validation on 15 cases, every concession student from 1 to 15 days late, whom it lets off a late fee the exam cell requires. No amount of verification could have found this, because verification's yardstick, the specification, was itself wrong. This is Chapter Four's seventh principle, the absence-of-defects fallacy, on a small scale: a product that conforms perfectly can still be the wrong product.

Build 2 shows that verification still earns its keep. It departs from its own specification in one case, an on-time form charged Rs 100 (the defect of Chapter One, on what software testing is), and verification finds that one case, cheaply and early, in a unit test. Validation fails 16 cases: the same one, and the 15 inherited from the specification.

Build 3 is what both together produce. Validation, ideally a review of the specification with the exam cell before any code was written, corrected the specification; verification then confirmed that the code matches the corrected specification. It passes 42 of 42 on both.

Why both matter

The significance of V&V in software development follows from the example.

1. Verification alone builds the wrong thing well. Every error in the requirements passes straight through it, as build 1 did.
2. Validation alone finds defects late and vaguely. Users see that an outcome is wrong, not which phase went wrong, and much of what matters (the security of a password store, the accuracy of an audit log) is invisible in ordinary use.
3. Both produce objective evidence. The definitions insist on "objective evidence", which is what a customer, an auditor or a regulator can rely on, where an assurance of it works on my machine is not.
4. Checking each phase before the next builds on it keeps defects cheap. Verification at every phase stops a design defect from being coded; early validation stops a requirements defect from being designed.
5. The effort can match the risk. IEEE 1012-2024 specifies V&V requirements "for different integrity levels": the more harm a failure could do, the more V&V the software receives.

Independent verification and validation (IV&V)

For software whose failure would be serious, V&V is often handed to people who did not build it. IEEE 1012-2024 defines independent verification and validation (IV&V) as "verification and validation performed by an organization that is technically, managerially, and financially independent of the development organization." The three kinds of independence are the same three that Chapter Twenty-Five gave for software quality assurance: IV&V staff did not build the system; they report outside the development and project management chain; and their budget cannot be cut by the project they are checking.

Independence brings two things that an internal team struggles to supply. The first is freedom from pressure: an IV&V team does not have to meet the development schedule it is judging. The second is a different set of assumptions. The analyst who misread the concession note in the worked example would read the specification the same way on every review; a reader from outside, who has never seen the note before, has a better chance of asking why concession students should pay no late fee at all.

IV&V costs money, so it is kept for software whose failure would cost far more than the checking, such as software that controls a medical device or an aircraft: in IEEE 1012's terms, software at a high integrity level. A college portal would not usually have a full IV&V effort, but ExamReg's college could reasonably ask an outside auditor to validate its fee calculations before the first session goes live, which is IV&V in miniature.

What it does not mean

Verification is not only testing. Reviews, inspections, walkthroughs and static analysis verify too, without running anything; the next chapter, Chapter Twenty-Seven, sorts the static and dynamic kinds of V&V.

Validation is not only acceptance testing at the end. It starts with the requirements and continues throughout.

Passing verification does not prove the product is right. It proves the product matches its specification, which may itself be wrong.

Independent does not mean detached. An IV&V team works throughout the project and reports its findings promptly to the developers as well as to management.

Quick revision

• Verification: "confirmation, through the provision of objective evidence, that specified requirements have been fulfilled" (ISO/IEC/IEEE 12207:2026). CMMI: "you built it right".
• Validation: "confirmation, through the provision of objective evidence, that the requirements for a specific intended use or application have been fulfilled". CMMI: "you built the right thing".
• IEEE 1012-2024: verification checks each phase's products against the conditions set at its start; validation shows the product solves the right problem and meets intended use and user needs.
• Verification checks against the specification; validation against the users' needs. The difference is the yardstick, not the technique.
• Validation starts early (requirements, prototypes) and continues to acceptance testing.
• Worked example: build 1 verified 42 of 42 but validated 27 of 42, because the specification was wrong.
• IV&V: V&V by an organisation technically, managerially and financially independent of development (IEEE 1012-2024); used at high integrity levels.

Test yourself

1. Define verification and validation, and state the difference in one sentence. Verification is confirmation, through objective evidence, that specified requirements have been fulfilled; validation is confirmation, through objective evidence, that the requirements for a specific intended use have been fulfilled. Verification asks whether we built the product right, validation whether we built the right product.

2. Can software pass verification and fail validation? Give an example. Yes. If the specification is wrong, code that matches it perfectly passes verification and fails validation: an ExamReg build coded from a specification that wrongly excused concession students from late fees agreed with its specification in every case but charged the wrong fee in 15 cases the exam cell cared about.

3. Give three methods of verification and three of validation. Verification: reviews and inspections of work products, static analysis, and unit, integration and system testing against the specification. Validation: requirements reviews and prototypes with users, acceptance testing, and alpha and beta testing in the users' own setting.

4. Explain the significance of V&V in software development. Verification stops defects passing from one phase to the next while they are cheap to fix; validation makes sure the requirements, and so the product, fit the real need. Both produce objective evidence for the customer and regulators, and the effort can be scaled to the software's integrity level.

5. What is IV&V and when is it used? Independent verification and validation: V&V performed by an organisation that is technically, managerially and financially independent of the developers (IEEE 1012-2024). It is used where failure would be serious, for example in software that controls medical devices or aircraft, because independence removes schedule pressure and brings fresh assumptions.

Reviews have the next three chapters to themselves: the review process and its types (Chapter Twenty-Eight), inspection (Chapter Twenty-Nine) and the walkthrough (Chapter Thirty).

Static analysis deserves a word more here, because it is the static mechanism a machine does. The ISTQB syllabus notes that it "can identify problems prior to dynamic testing while often requiring less effort, since no test cases are required", and that it "is often incorporated into CI frameworks", running on every change. It has one requirement: "for static analysis, work products need a structure against which they can be checked (e.g., models, code or text with a formal syntax)." A tool can analyse code; it cannot yet read a requirements document the way a reviewer does.

The dynamic mechanisms

The ISTQB syllabus names the same set when it places testing among the forms of quality control: "Testing is a major form of quality control, while others include formal methods (model checking and proof of correctness), simulation and prototyping."

Worked example: a static analyser finds two defects without running anything

A static analysis tool is a program that reads other programs. The one below is small, about forty lines, but it does what real analysers do. It reads the source of three versions of ExamReg's fee function, never calls any of them, and follows each function's if-statements in order, keeping track of which values of days_late can still reach each line. When a branch can never be reached, or some values run off the end of the function with no return, it reports it.

import ast

SOURCE = '''
def fee_a(days_late):
    if days_late < 0:
        raise ValueError("days late cannot be negative")
    if days_late > 15:
        raise ValueError("form not accepted more than 15 days late")
    if days_late <= 7:
        return 100
    if days_late <= 0:
        return 0
    return 500

def fee_b(days_late):
    if days_late < 0:
        raise ValueError("days late cannot be negative")
    if days_late <= 0:
        return 0
    if days_late <= 7:
        return 100
    if days_late < 15:
        return 500

def fee_c(days_late):
    if days_late < 0:
        raise ValueError("days late cannot be negative")
    if days_late > 15:
        raise ValueError("form not accepted more than 15 days late")
    if days_late <= 0:
        return 0
    if days_late <= 7:
        return 100
    return 500
'''

INF = float("inf")
TRUE_FOR = {ast.Lt: lambda k: (-INF, k - 1), ast.LtE: lambda k: (-INF, k),
            ast.Gt: lambda k: (k + 1, INF), ast.GtE: lambda k: (k, INF)}
FALSE_FOR = {ast.Lt: lambda k: (k, INF), ast.LtE: lambda k: (k + 1, INF),
             ast.Gt: lambda k: (-INF, k), ast.GtE: lambda k: (-INF, k - 1)}

def meet(a, b):                          # the whole numbers in both ranges
    return (max(a[0], b[0]), min(a[1], b[1]))

def empty(r):
    return r[0] > r[1]

def shown(r):
    return f"{r[0]} upward" if r[1] == INF else f"{r[0]} to {r[1]}"

def analyse(fn):
    """Walk fn's statements in order, tracking which values of its argument can still reach
    each one. Nothing is run: the program reads the source's structure."""
    arg = fn.args.args[0].arg
    live = (-INF, INF)
    findings = []
    for stmt in fn.body:
        if isinstance(stmt, ast.If) and isinstance(stmt.test, ast.Compare):
            test = stmt.test
            op, k = type(test.ops[0]), test.comparators[0]
            if isinstance(test.left, ast.Name) and test.left.id == arg and isinstance(k, ast.Constant):
                if empty(meet(live, TRUE_FOR[op](k.value))):
                    findings.append(f"line {stmt.lineno}: 'if {ast.unparse(test)}' can never be true here")
                if isinstance(stmt.body[-1], (ast.Return, ast.Raise)):
                    live = meet(live, FALSE_FOR[op](k.value))
                continue
        if isinstance(stmt, (ast.Return, ast.Raise)):
            live = (1, 0)                # nothing gets past a return or a raise
    if not empty(live):
        findings.append(f"{arg} from {shown(live)} reaches the end and returns None")
    return findings or ["no finding"]

for fn in ast.parse(SOURCE).body:
    for finding in analyse(fn):
        print(f"{fn.name}: {finding}")

fee_a: line 9: 'if days_late <= 0' can never be true here
fee_b: days_late from 15 upward reaches the end and returns None
fee_c: no finding

Each finding points straight at a real defect.

• fee_a checks days_late <= 7 before days_late <= 0. Every on-time value is at most 7, so the on-time branch can never run, and an on-time form is charged Rs 100. It is Chapter One's defect in another shape, from the chapter on what software testing is, and here no test case found it: the analyser worked it out from the structure alone.
• fee_b has no final return and no check for forms more than 15 days late. For 15 days and above it falls off the end and returns None: a form 15 days late gets no fee at all instead of Rs 500, and a form 30 days late is never refused.
• fee_c, the corrected rule, gets no finding.

The limits of the tool are just as instructive. It knows nothing about the fee rule: if fee_c said return 50 instead of return 100, the analyser would report nothing, because the structure would be perfect and only the value wrong. Finding that needs a test with an expected result, which is dynamic testing, or a reviewer who knows the rule, which is a different static mechanism. This is the ISTQB syllabus's point exactly: static and dynamic testing "complement each other", and "there are some defect types that can only be found by either static or dynamic testing."

Static against dynamic

The ISTQB syllabus lists the differences, and they make a complete answer to a common examination question.

Some defects are found far more easily by one side. The syllabus lists, among those easier or cheaper to find statically, defects in requirements such as "inconsistencies, ambiguities, contradictions, omissions, inaccuracies, duplications", design defects, coding defects such as "variables with undefined values, undeclared variables, unreachable or duplicated code, excessive code complexity", deviations from standards, and "Incorrect interface specifications (e.g., mismatched number, type or order of parameters)". Dynamic mechanisms own everything that only shows when the software runs in its environment: timing, load, memory, and the behaviour of the whole system with its real data.

Where each mechanism sits in the life cycle

The pattern is the one the ISTQB syllabus gives as the value of static testing: it "can detect defects in the earliest phases of the SDLC, fulfilling the principle of early testing". Static mechanisms dominate the left of the table, where nothing runs yet; dynamic mechanisms take over as the product becomes executable; and both continue to the end.

What it does not mean

Static does not mean verification, nor dynamic validation. A review of the requirements with the users is static validation; a unit test against the design is dynamic verification.

Static analysis does not replace testing. It finds structural problems without knowing what the right answer is; only a test or a knowledgeable reviewer can say that a fee is wrong.

A static finding is not always a defect. Tools report suspicious patterns, and some are harmless; a person judges each one.

Formal proof is not proof that the software is right for its users. It proves the program meets its formal specification, which can itself be wrong, as the last chapter showed.

Quick revision

• Static testing: a test item "examined against a set of quality or other criteria without the test item being executed"; dynamic testing: evaluated "by executing it" (ISO/IEC/IEEE 29119-2).
• Static mechanisms: reviews (informal, walkthrough, technical, inspection), audits, desk checking, static analysis, formal proof.
• Dynamic mechanisms: testing, dynamic analysis, simulation, prototyping.
• Static finds defects directly, works on any work product, starts earliest; dynamic finds failures, needs something executable, and measures run-time qualities such as performance.
• Static analysis needs a formal structure (code, models); it "can identify problems prior to dynamic testing while often requiring less effort".
• Static and dynamic both serve verification and validation.
• Worked example: an ast-based analyser found an unreachable on-time branch and a fall-off-the-end without running the code, and could not have found a wrong fee value.

Test yourself

1. Distinguish static from dynamic V&V mechanisms, with examples of each. Static mechanisms examine a work product without executing it: reviews, inspections, walkthroughs, audits, desk checking, static analysis and formal proof. Dynamic mechanisms evaluate it by executing it: testing at every level, dynamic analysis, simulation and prototyping.

2. What is static analysis, and what can it find? The process of evaluating a system or component based on its form, structure, content or documentation, usually by a tool and without running it. It finds structural defects such as unreachable code, undefined variables, standards violations, some security weaknesses and excessive complexity, and it needs a work product with a formal structure, such as code.

3. Give three differences between static and dynamic testing. Static testing finds defects directly, while dynamic testing causes failures that must then be analysed; static testing can examine non-executable work products such as requirements, while dynamic testing needs executable software; and static testing reaches rarely executed code as easily as any other, while dynamic testing reaches only what its tests drive.

4. Which V&V mechanisms can be used in the requirements phase? Static ones such as reviews and inspections of the requirements and audits against standards, and dynamic ones that do not need the final product, such as prototypes shown to users and simulations.

5. Why is static analysis not enough on its own? Because it checks structure, not meaning: it cannot know what the right output is. A function that returns the wrong fee through perfectly structured code passes static analysis and is caught only by a test with an expected result or a reviewer who knows the rule.

One word in the process needs care. An anomaly is "anything observed in the documentation or operation of a system that deviates from expectations based on previously verified system, software, or hardware products or reference documents" (IEEE 1012-2024). A reviewer logs anomalies; only the analysis decides which of them are defects. A question that turns out to have a good answer is an anomaly, but not a defect.

Review techniques: how a reviewer reads

ISO/IEC 20246 names several techniques for the individual review. A reviewer can use more than one.

The roles

The ISTQB syllabus names six principal roles, and one person may hold several.

The four review types

The syllabus observes that review types range "from informal reviews to formal reviews", and ISO/IEC 20246 defines the two ends: an informal review is a "form of review that does not follow a defined process and has no formal documented output", and a formal review is a "form of review that follows a defined process with formal documented output". Four types are in common use.

• Informal review. "Informal reviews do not follow a defined process and do not require a formal documented output. The main objective is detecting anomalies." A colleague reading a draft and scribbling in the margin is an informal review, and so is much of pair programming.
• Walkthrough. "A walkthrough, which is led by the author, can serve many objectives", from "evaluating quality and building confidence in the work product" to "educating reviewers, gaining consensus, generating new ideas" and "detecting anomalies". ISO/IEC 20246 defines it as a "formal review in which an author leads members of the review through a work product, and the participants ask questions and make comments about possible issues." Chapter Thirty is about the walkthrough.
• Technical review. "A technical review is performed by technically qualified reviewers and led by a moderator." Its objectives are "to gain consensus and make decisions regarding a technical problem", and also to detect anomalies and evaluate quality. ISO/IEC 20246 defines it as a "formal peer review of a work product by a team of technically qualified personnel that examines the suitability of the work product for its intended use and identifies discrepancies from specifications and standards".
• Inspection. "As inspections are the most formal type of review, they follow the complete generic process". "The main objective is to find the maximum number of anomalies", and "Metrics are collected and used to improve the SDLC, including the inspection process. In inspections, the author cannot act as the review leader or scribe." Chapter Twenty-Nine is about the inspection.

The level of formality is a choice. The syllabus says it depends on "the SDLC being followed, the maturity of the development process, the criticality and complexity of the work product being reviewed, legal or regulatory requirements, and the need for an audit trail", and that "The same work product can be reviewed with different review types, e.g., first an informal one and later a more formal one."

The formal technical review of Module 2 (Chapter Ninety-Seven) returns to reviews from the quality assurance side, as an SQA activity with its own guidelines; this chapter's process and types are the foundation it builds on.

Worked example: reviewing ExamReg's fee requirement

Here is the requirement as the analyst first wrote it.

Planning and initiation. The review leader chooses a technical review, because the fee rules decide money. The scope is R-FEE-3 and the exam cell's note it came from; the focus is boundaries and exceptions; the exit criterion is that no defect of major severity remains open. Each reviewer receives the requirement, the note and the team's requirements checklist.

Individual review. Each reviewer reads with a different technique. The tester uses the checklist and drafts test cases as they read; the exam cell clerk reads in role, as a student would meet the rule; the developer reads for anything that cannot be coded unambiguously. Their logs, merged by the scribe:

Communication and analysis. The meeting, run by the moderator, decides each item's status.

Fixing and reporting. The analyst rewrites R-FEE-3; the scribe's record goes to the project; the exit criterion is met once the major defect, item 4, is corrected and checked.

Four defects in one short requirement, found in a two-hour meeting. Item 1 is the on-time defect that Chapter One, on what software testing is, found with a failing test after the code was written; item 4 is the wrong specification that Chapter Twenty-Six showed no amount of verification could catch; and items 1 to 3 are the boundaries that the static analyser of Chapter Twenty-Seven found in code. Found here, each costs a sentence to fix.

What makes reviews succeed

The ISTQB syllabus lists the success factors, and every one of them can be seen in the example.

• "Defining clear objectives and measurable exit criteria. Evaluation of participants should never be an objective"
• "Choosing the appropriate review type to achieve the given objectives"
• "Performing reviews on small chunks, so that reviewers do not lose concentration"
• Feedback to stakeholders and authors, adequate time to prepare, and support from management
• "Making reviews part of the organization's culture, to promote learning and process improvement"
• Adequate training for all participants, and facilitated meetings

The first deserves emphasis. A review examines the work, never the person. If authors fear that their anomalies will be counted against them, they stop bringing work to review early, and the cheapest defect-finding activity in the project is lost.

What it does not mean

A review is not an inspection by another name. Inspection is one type, the most formal; the informal review, the walkthrough and the technical review are reviews too.

Every anomaly is not a defect. Anomalies are analysed; some are questions with good answers, and some are recommendations.

A review is not an assessment of the author. Evaluating participants "should never be an objective".

Reviews are not only for code. Any work product that can be read, from a requirement to a test plan, can be reviewed.

Quick revision

• Review: a process or meeting in which work products are presented to stakeholders for comment or approval.
• Process (ISO/IEC 20246, ISTQB v4.0.1): planning, review initiation, individual review, communication and analysis, fixing and reporting.
• Anomaly: a deviation from expectations; analysis decides whether it is a defect.
• Techniques: ad hoc, checklist-based, scenario-based, role-based, perspective-based.
• Roles: manager, author, moderator (facilitator), scribe (recorder), reviewer, review leader.
• Types: informal review (no defined process), walkthrough (led by the author), technical review (led by a moderator, technical consensus), inspection (most formal, maximum anomalies, metrics, author never leader or scribe).
• Formality depends on the SDLC, process maturity, criticality, regulation and the need for an audit trail.
• Success: clear objectives and exit criteria, never evaluating participants, small chunks, time, training, management support.

Test yourself

1. Describe the activities of the review process. Planning defines the scope, focus, exit criteria, effort and timeframes. Review initiation makes sure everyone has the work product and understands their role. In individual review each reviewer applies review techniques and logs anomalies, recommendations and questions. In communication and analysis the anomalies are discussed, usually in a meeting, and each gets a status, owner and action. In fixing and reporting, defects are reported and fixed, the exit criteria are checked, and the results are reported.

2. Name the roles in a review and their responsibilities. The manager decides what is reviewed and provides resources; the author creates and fixes the work product; the moderator runs the meeting and keeps it safe and on time; the scribe records anomalies and decisions; the reviewers perform the review; and the review leader takes overall responsibility and organises it.

3. Compare the four types of review. An informal review follows no defined process and needs no documented output, and aims to detect anomalies. A walkthrough is led by the author and serves many objectives, including education and consensus. A technical review is led by a moderator with technically qualified reviewers, to reach decisions on technical problems. An inspection is the most formal, follows the complete process, aims to find the most anomalies, collects metrics, and never lets the author lead or record.

4. Why is every anomaly found in a review not a defect? Because an anomaly is only a deviation from what a reviewer expected; on analysis it may turn out to be a question with a good answer, a recommendation, or a misunderstanding by the reviewer. The meeting decides which anomalies are defects.

5. Give four factors for a successful review. Clear objectives and measurable exit criteria, with evaluation of participants never an objective; the right review type for the objectives and work product; reviewing in small chunks with adequate preparation time; and management support with training for all participants.

K.NCSS is a thousand non-commentary source statements, roughly a thousand lines of code without comments. Each operation has its own rules.

1. Overview. "The designer first describes the overall area being addressed and then the specific area he has designed in detail", and the design documents are handed out. A code inspection needs no overview, because the same people inspected the design.
2. Preparation. Participants "literally do their homework to try to understand the design, its intent and logic." They also study the ranked distributions of error types found by recent inspections, and checklists of clues, so that they look where errors are most likely.
3. Inspection. A reader chosen by the moderator, usually the coder, paraphrases the design or code. "Every piece of logic is covered at least once, and every branch is taken at least once." Each error found is noted, its type classified and its severity (major or minor) recorded, and the reading moves on. No one designs solutions at the meeting: "The inspection is not intended to redesign, evaluate alternate design solutions, or to find solutions to errors; it is intended just to find errors!" Within one day the moderator writes the inspection report.
4. Rework. "All errors or problems noted in the inspection report are resolved by the designer or coder/implementor."
5. Follow-up. The moderator checks that every issue is resolved. "If more than five percent of the material has been reworked, the team should reconvene and carry out a 100 percent reinspection."

Two practical rules come from Fagan's experience. The meeting's error detection falls off after two hours, so "it is advisable to schedule inspection sessions of no more than two hours at a time. Two two-hour sessions per day are acceptable." And the time for inspections and rework "must be scheduled and managed with the same attention as other important project activities", because under pressure inspections are the first thing a project drops.

Fagan also defined what the meeting is looking for: "an error is defined as any condition that causes malfunction or that precludes the attainment of expected or previously specified results."

The roles

"The inspection team is best served when its members play their particular roles", Fagan wrote, and he named four.

The reader is not a fifth person: it is a job the moderator gives, usually to the coder. When one person has done two jobs, the roles are refilled from outside: if the same person designed and coded the work, they take the designer's role and "a coder from some related or similar program will perform the role of the coder." On size, "Four people constitute a good-sized inspection team", and the team "should not be artificially increased over four" unless the code touches several interfaces whose owners should be present.

The modern syllabus keeps the essential rule in a sentence: "In inspections, the author cannot act as the review leader or scribe." Fagan's moderator is today's review leader and moderator; his hand-written notes are today's scribe's log.

Looking where the errors are: checklists and error types

Fagan observed that finding errors has to be taught: "it is one thing to direct people to find errors in design or code. It is quite another problem for them to find errors." His answer had two parts. Inspectors study the ranked distribution of error types from recent inspections, so they concentrate on the most common and costly kinds; and they use checklists of clues for each type. His Figure 5 shows part of the design checklist for logic, with questions such as "Are All Constants Defined?" and "Are All Increment Counts Properly Initialized (0 or 1)?" Every error found is also classified as missing, wrong or extra.

The program below recomputes the error distributions from Fagan's Figures 3 and 4, read row by row off the page, and adds the arithmetic of his Tables 1 and 3.

import math

# Fagan 1976, Table 1: the Aetna application, errors found per K.NCSS
by_inspection, by_test, after = 38, 8, 0
total = by_inspection + by_test + after
print(f"Table 1: inspections found {by_inspection} of {total} errors per K.NCSS,"
      f" {by_inspection / total:.1%}")

# Figures 3 and 4: errors by type as (missing, wrong, extra), read off the page
design = {"logic": (126, 57, 24), "prologue/prose": (44, 38, 7), "CB usage": (18, 17, 1),
          "other": (15, 10, 10), "more detail": (24, 6, 2), "interconnect calls": (18, 9, 0),
          "test and branch": (12, 7, 2), "CB definition": (16, 2, 0),
          "maintainability": (8, 5, 3), "return code/msg": (5, 7, 2),
          "interconnect reqts": (4, 5, 2), "performance": (1, 2, 3),
          "register usage": (1, 2, 0), "higher level docu": (1, 0, 1), "FPFS": (1, 0, 0),
          "mod attributes": (1, 0, 0), "pass data areas": (0, 1, 0)}
code = {"logic": (33, 49, 10), "design error": (31, 32, 14), "prologue/prose": (25, 24, 3),
        "CB usage": (3, 21, 1), "code comments": (5, 17, 1), "interconnect calls": (7, 9, 3),
        "maintainability": (5, 7, 2), "PL/S or BAL use": (4, 9, 1), "performance": (3, 2, 5),
        "F1": (0, 8, 0), "test and branch": (2, 5, 0), "register usage": (4, 2, 0),
        "storage usage": (1, 0, 0)}
for name, table in [("Design (Figure 3)", design), ("Code (Figure 4)", code)]:
    errors = sum(sum(row) for row in table.values())
    missing, wrong, extra = (sum(row[i] for row in table.values()) for i in range(3))
    print(f"{name}: {errors} errors; missing {missing / errors:.0%},"
          f" wrong {wrong / errors:.0%}, extra {extra / errors:.0%}")
    ranked = sorted(table, key=lambda t: -sum(table[t]))[:3]
    print("  most frequent:", ", ".join(f"{t} {sum(table[t])} ({sum(table[t]) / errors:.1%})"
                                         for t in ranked))

# Table 3: rates of progress for systems programming, per person, for 1,000 lines
people, lines = 4, 1000
for name, overview, preparation, meeting, rework in [("design I1", 500, 100, 130, 20),
                                                     ("code I2", None, 125, 150, 16)]:
    hours = {"overview": people * lines / overview if overview else 0,
             "preparation": people * lines / preparation,
             "inspection": people * lines / meeting,
             "rework": rework}
    sessions = math.ceil(lines / meeting / 2)            # no session longer than two hours
    print(f"{name}: " + ", ".join(f"{k} {v:.1f}" for k, v in hours.items() if v)
          + f"; total {sum(hours.values()):.1f} people-hours; {sessions} two-hour meetings")

Table 1: inspections found 38 of 46 errors per K.NCSS, 82.6%
Design (Figure 3): 520 errors; missing 57%, wrong 32%, extra 11%
  most frequent: logic 207 (39.8%), prologue/prose 89 (17.1%), CB usage 36 (6.9%)
Code (Figure 4): 348 errors; missing 35%, wrong 53%, extra 11%
  most frequent: logic 92 (26.4%), design error 77 (22.1%), prologue/prose 52 (14.9%)
design I1: overview 8.0, preparation 40.0, inspection 30.8, rework 20.0; total 98.8 people-hours; 4 two-hour meetings
code I2: preparation 32.0, inspection 26.7, rework 16.0; total 74.7 people-hours; 4 two-hour meetings

The recomputed totals match what Fagan printed: 520 design errors split 57, 32 and 11 per cent, logic at 39.8 per cent, and 348 code errors with logic at 26.4 per cent. Three lessons come out of the numbers.

• Design inspections find mostly what is missing; code inspections mostly what is wrong. More than half the design errors were missing items, while more than half the code errors were wrong ones. A design checklist should therefore ask what has been left out?, and a code checklist what is incorrect?
• Logic dominates both, and documentation is close behind. Prologue and prose errors, in the comments and descriptions, are the second commonest design error and the third commonest code error; Fagan inspected them because the next programmer relies on them.
• The rates turn into a plan. Inspecting 1,000 lines of design with four people costs about 99 people-hours by Table 3's rates, and the meeting alone needs four sessions of at most two hours. Fagan's own estimate for the whole process, "overview through follow-up", was "about 90 to 100 people-hours for systems programming". Code, needing no overview and read faster, costs about 75.

On ExamReg, the same arithmetic tells the test lead what a code inspection of the fee module will cost before the first meeting is booked, which is exactly what Fagan meant by managing inspections "with the same attention as other important project activities".

What Fagan reported

Fagan gave results from two settings, and warned in the paper that they "cannot be considered representative of every situation".

A systems programming study at IBM. A piece of an operating system component, designed by three programmers and coded by 13, went through I1 and I2 inspections for the first time. The net saving "translated into a 23 percent increase in the productivity of the coding operation alone." A control sample, taken once the inspections were routine, differed by only 0.9 per cent, so the gain was not a novelty effect. The net savings were 94 programmer hours per K.NCSS from I1 and 51 from I2, while a third inspection after unit test, I3, cost 20 hours more than it saved, and "As a consequence, I3 is no longer in effect." In testing after unit test, the inspected sample had "38 percent less errors" than a comparable piece built with walk-throughs.

An application at Aetna Life and Casualty. A COBOL program of 4,439 non-commentary statements in eight modules, written by two programmers with inspections as the only change to their process, was estimated to need 62 programmer days and took 46.5, including inspection meetings: "The resulting saving in programmer resources was 25 percent." Table 1 records where its errors were found: 38 per K.NCSS by the design and code inspections, 8 by unit and preparation for acceptance testing, and none in acceptance testing or in six months of use. That is Fagan's error detection efficiency of 82 per cent (38 ÷ 46 is 0.826, and the table prints 82), where

error detection efficiency = errors found by an inspection ÷ total errors in the product before inspection.

The reason inspections pay, in Fagan's words, is where they find errors: rework at the early levels "is 10 to 100 times less expensive than if it is done in the last half of the process." Chapter Ninety-Six, on why reviews pay, returns to that cost.

Inspection results are not for appraising people

Fagan was emphatic that inspection data belongs to the programmer: the results "should not under any circumstances be used for programmer performance appraisal." The reason is practical. An inspection works only if authors bring work early and errors are reported freely; the moment error counts are used against the people who made them, both stop. It is the same rule as the ISTQB success factor in Chapter Twenty-Eight, on software reviews: evaluation of participants should never be an objective.

What makes an inspection formal

What it does not mean

An inspection is not a meeting to fix the code. Solutions are noted if obvious and worked out afterwards; the meeting only finds errors.

An inspection does not replace testing. In Fagan's Aetna data inspections found 82 per cent of the errors; testing found the rest.

The moderator is not the author's manager or the author. The moderator should come from an unrelated project, and the author cannot lead or record.

Fagan's rates are not universal. They were measured for systems programming and are, by his note, conservative; application code went four to six times faster.

Quick revision

• Inspection (ISO/IEC 20246): a formal review that uses defined team roles and measurement to improve the review process.
• Fagan (IBM Systems Journal, 1976): inspection points I0, I1 (design complete), I2 (code), IT1, IT2 (test plan, test cases), PI (publications).
• Operations: overview (communication, education), preparation (education), inspection (find errors), rework, follow-up.
• Roles: moderator ("the key person", trained, from an unrelated project), designer, coder/implementor, tester; a reader paraphrases; four is a good size.
• Rules: at most two hours per session; report within one day; no solution hunting; reinspect if more than 5 per cent reworked; never use results to appraise programmers.
• Error detection efficiency = errors found by the inspection ÷ total errors before inspection; 82 per cent at Aetna.
• Results: coding productivity up 23 per cent; 38 per cent fewer errors than walk-throughs; Aetna 25 per cent fewer programmer days.
• Design errors are mostly missing (57 per cent); code errors mostly wrong (53 per cent); logic tops both.

Test yourself

1. Describe the five operations of a Fagan inspection. Overview: the designer presents the design to the whole team and hands out the documents. Preparation: each participant studies the material, the common error types and the checklists. Inspection: a reader paraphrases the work, every piece of logic and every branch is covered, and errors are noted and classified without seeking solutions; the moderator reports within a day. Rework: the designer or coder resolves every error. Follow-up: the moderator verifies the rework and calls a full reinspection if more than 5 per cent was reworked.

2. What are the roles in an inspection, and why must the moderator be trained and independent? Moderator, designer, coder/implementor and tester, with a reader chosen by the moderator. The moderator runs the whole process and the meeting, so needs training in leading it; coming from an unrelated project preserves objectivity, and the author may never lead or record.

3. Define error detection efficiency and compute it for Fagan's Aetna data. It is the errors found by an inspection divided by the total errors in the product before inspection. At Aetna, inspections found 38 errors per K.NCSS out of 46 found in total, about 82 per cent.

4. What results did Fagan report for inspections? In a systems programming study, a 23 per cent increase in coding productivity and 38 per cent fewer errors than a walk-through sample; in an application at Aetna, 25 per cent fewer programmer days than estimated, 82 per cent of errors found by inspection, and no errors in acceptance testing or six months of use.

5. What makes an inspection more formal than other reviews? Entry and exit criteria, a trained moderator who is not the author, defined roles, required preparation with checklists, a meeting with the single objective of finding errors, classification and written records of every error, verified rework, and measurement used to improve the process.

• teaching: a new team member learns the fee module fastest by having its author walk through it;
• early drafts: an author who wants reactions before finishing a design;
• consensus: a team choosing between two approaches, where discussing alternatives is the point;
• stakeholders who are not programmers: an analyst walking the exam cell through the requirements, which is validation in the sense of Chapter Twenty-Six;
• work of moderate risk, where the cost of a full inspection is not justified.

Roles in a walkthrough

A walkthrough needs fewer roles than an inspection, and they are less fixed.

Leading a code walkthrough: tracing test cases by hand

A code walkthrough can drift into a line-by-line reading that nobody remembers. A more effective way for the author to lead is to trace test cases through the code by hand: participants bring a few test cases, each with its expected result, and the author "executes" the code on paper for each one, saying aloud what every line does to every variable. The participants follow, and a wrong value shows up at the line where it happens.

Here is the author of ExamReg's fee function leading such a walkthrough. The fee rules are the book's fixed ones: a form fee of Rs 800, Rs 150 for each backlog paper, the late fee of Chapter One, on what software testing is, forms more than 15 days late refused, a negative number of days refused as invalid input, and a concession that waives the form fee only, as the Scrum team of Chapter Seventeen established.

FORM_FEE, BACKLOG_FEE = 800, 150

def total_fee(days_late, backlog_papers, concession):
    if days_late < 0 or days_late > 15:
        raise ValueError("form not accepted")
    fee = FORM_FEE + BACKLOG_FEE * backlog_papers
    if concession:
        fee = 0                                  # the concession
    if days_late <= 0:
        late = 0
    elif days_late <= 7:
        late = 100
    else:
        late = 500
    return fee + late

print("author's own check before the walkthrough, TC1:", total_fee(0, 0, False))

The author ran one case before the meeting, and it passed:

author's own check before the walkthrough, TC1: 800

The participants bring three test cases:

The author traces each one aloud, and the scribe writes the trace on the board.

TC3 stops the room. At the concession line the trace sets fee to 0, wiping out the Rs 300 for two backlog papers along with the form fee. A participant asks what the concession is supposed to waive; the answer, the form fee only, is the fix: subtract the form fee instead of zeroing the total. The failing case was never run: the defect was found on paper, at the line that caused it. After the walkthrough the author makes the change, and the three test cases are run on both versions to confirm what the trace showed.

FORM_FEE, BACKLOG_FEE = 800, 150

def total_fee(days_late, backlog_papers, concession):         # as walked through
    if days_late < 0 or days_late > 15:
        raise ValueError("form not accepted")
    fee = FORM_FEE + BACKLOG_FEE * backlog_papers
    if concession:
        fee = 0                                                 # the concession
    if days_late <= 0:
        late = 0
    elif days_late <= 7:
        late = 100
    else:
        late = 500
    return fee + late

def total_fee_fixed(days_late, backlog_papers, concession):   # after the walkthrough
    if days_late < 0 or days_late > 15:
        raise ValueError("form not accepted")
    fee = FORM_FEE + BACKLOG_FEE * backlog_papers
    if concession:
        fee -= FORM_FEE                                         # waives the form fee only
    if days_late <= 0:
        late = 0
    elif days_late <= 7:
        late = 100
    else:
        late = 500
    return fee + late

cases = [("TC1", (0, 0, False), 800), ("TC2", (10, 1, False), 1450), ("TC3", (3, 2, True), 400)]
for version in (total_fee, total_fee_fixed):
    print(version.__name__)
    for name, args, expected in cases:
        actual = version(*args)
        mark = "" if actual == expected else "  <- the walkthrough's finding"
        print(f"  {name} {args}: expected {expected}, got {actual}{mark}")

total_fee
  TC1 (0, 0, False): expected 800, got 800
  TC2 (10, 1, False): expected 1450, got 1450
  TC3 (3, 2, True): expected 400, got 100  <- the walkthrough's finding
total_fee_fixed
  TC1 (0, 0, False): expected 800, got 800
  TC2 (10, 1, False): expected 1450, got 1450
  TC3 (3, 2, True): expected 400, got 400

The run agrees with the hand trace line for line: the version walked through returns 100 for TC3, and the fixed one 400. Notice also what this walkthrough did and did not do. It found a real defect and it taught the room how the concession works. But it did not use a checklist, did not classify the defect, did not measure anything, and nobody is assigned to verify the fix except the author. Those are exactly the things an inspection adds.

Walkthrough against inspection

Fagan, who defined the inspection, compared the two in his 1976 paper. His starting point was that walk-throughs "are practiced in many different ways in different places, with varying regularity and thoroughness. This inconsistency causes the results of walk-throughs to vary widely and to be nonrepeatable. Inspections, however, having an established process and a formal procedure, tend to vary less and produce more repeatable results."

His Table 4 sets the two processes side by side:

His note under the table makes the key point: "Note the separation of objectives in the inspection process." An inspection meeting does one thing; a walk-through meeting does three at once.

His Table 5 compares their key properties:

And in one comparison of equivalent pieces of an operating system, the inspected sample had "38 percent less errors" in later testing than the walk-through sample, as Chapter Twenty-Nine, on inspection, reported.

Putting Fagan's comparison together with the modern syllabus gives the table a 5-mark answer needs.

Choosing between them

The choice follows from the ISTQB syllabus's factors from Chapter Twenty-Eight, on reviews: the criticality of the work product, the objectives, and the need for an audit trail. On ExamReg, the new help pages would go to a walkthrough with the exam cell, because the aim is to check that students will understand them; the fee and payment code would go to an inspection, because a missed error there costs students money. Many teams use both on the same work product: a walkthrough while the design is taking shape, and an inspection once it is complete.

What it does not mean

A walkthrough is not a weak inspection. It has different objectives, and for teaching and consensus it does better than an inspection would.

A walkthrough does not mean no preparation is ever done. Preparation is optional, not forbidden, and participants who prepare find more.

Tracing by hand is not the same as testing. It checks the code against a few expected results in people's heads; the cases still need to be run, as they were after this walkthrough.

Being led by the author is not a flaw. It is what makes a walkthrough good at teaching; it is also why a walkthrough should not be the only review of critical work.

Quick revision

• Walkthrough: an author leads members of the review through a work product; participants ask questions and comment (ISO/IEC 20246). Led by the author; preparation optional (ISTQB).
• Objectives (ISTQB v4.0.1): evaluate quality and build confidence, educate reviewers, gain consensus, generate ideas, motivate authors, detect anomalies.
• Tracing test cases by hand is an effective way to lead a code walkthrough; the ExamReg trace found a concession that wiped out backlog fees.
• Fagan's Table 4: the inspection has five operations with separate objectives; the walk-through has preparation and one meeting with mixed objectives.
• Fagan's Table 5: inspections have moderator training, defined roles, checklists, error-type data, follow-up, feedback and process analysis; walk-throughs do not.
• Walk-through results "vary widely" and are "nonrepeatable"; inspections are more repeatable; the inspected sample had 38 per cent fewer errors.
• Use walkthroughs for teaching, drafts, consensus and stakeholders; inspections for critical work.

Test yourself

1. What is a walkthrough, and who leads it? A review in which the author leads members of the review through a work product while the participants ask questions and comment on possible issues. The author leads it, presenting the work in the order they choose.

2. What are the objectives of a walkthrough? Evaluating quality and building confidence in the work product, educating reviewers, gaining consensus, generating new ideas, motivating and enabling authors to improve, and detecting anomalies.

3. Differentiate between a walkthrough and an inspection. A walkthrough is led by the author, an inspection by a trained moderator who is never the author. A walkthrough aims at understanding and consensus as well as problems, and may discuss solutions; an inspection's meeting aims only at finding errors. A walkthrough is a meeting with optional preparation; an inspection has overview, preparation, inspection, rework and follow-up. An inspection uses defined roles, checklists, error-type data and metrics, and verifies fixes; a walkthrough usually does not, so its results are less repeatable.

4. How can an author lead a code walkthrough effectively? By tracing test cases through the code by hand: participants bring test cases with expected results, and the author executes the code on paper, saying what each line does to each variable, so that a wrong value appears at the line that causes it.

5. When would you choose a walkthrough rather than an inspection? When the aim is teaching, gathering reactions to an early draft, reaching consensus, or checking a work product with non-technical stakeholders, and the work is not so critical that the cost of a full inspection is justified.

There are two reasons for the order. The first is localisation. When a unit test fails, the fault is in that unit; when a system test fails, it could be in any unit or in any interface between them, and finding it is a longer piece of debugging. Testing the small pieces first means that by the time the large tests run, most faults that could be found in one place already have been. The second is the division of objectives. Each level looks for what it is best placed to find: a unit test for a wrong calculation, an integration test for two units that disagree about a parameter, a system test for a behaviour that only appears when everything runs together, an acceptance test for a requirement that was wrong all along. The syllabus notes that in sequential development "the exit criteria of one level are part of the entry criteria for the next level", so the levels form a chain of checkpoints.

Reviews come before all of them. A strategy that starts testing only when code exists has already given up the cheapest defects, as Chapters Twenty-Eight to Thirty on reviews, inspection and the walkthrough showed.

Worked example: why one level is never enough

ExamReg release 2.0's defect data, used throughout this book, records which activity found each of its 200 defects. Reviews found 74 before any code ran; the remaining 126 were in the code when unit testing began. The program follows those 126 through the four test levels, assuming, as a simplification, that no fix brought in a new defect.

found = {"requirements review": 16, "design review": 24, "code review": 34,
         "unit testing": 44, "integration testing": 32, "system testing": 28,
         "acceptance testing": 10}
after_release = 12
total = sum(found.values()) + after_release                     # 200, ExamReg release 2.0
remaining = total - sum(found[r] for r in ("requirements review", "design review", "code review"))
at_start = remaining
print(f"defects in the code when unit testing began: {remaining} of {total}")
for level in ("unit testing", "integration testing", "system testing", "acceptance testing"):
    caught = found[level]
    print(f"{level:<20} caught {caught:>2} of the {remaining:>3} still there ({caught / remaining:.1%})")
    remaining -= caught
print(f"{'students':<20} met the last {remaining}")
print(f"the four test levels together: {at_start - remaining} of {at_start}"
      f" ({(at_start - remaining) / at_start:.1%})")

defects in the code when unit testing began: 126 of 200
unit testing         caught 44 of the 126 still there (34.9%)
integration testing  caught 32 of the  82 still there (39.0%)
system testing       caught 28 of the  50 still there (56.0%)
acceptance testing   caught 10 of the  22 still there (45.5%)
students             met the last 12
the four test levels together: 114 of 126 (90.5%)

No level caught even three in five of the defects still present when it began: unit testing caught about 35 per cent, integration testing 39, system testing 56 and acceptance testing about 45. Each was far from enough on its own. Together they caught 114 of the 126, about 90.5 per cent, and with the reviews in front of them the whole strategy stopped 188 of 200 defects before students saw them. That is the argument for a strategy of several levels in one table: each level is a filter with holes, and the holes in different filters are in different places. Chapter Seventy-Nine, on defect metrics, gives these ratios their standard names.

Who tests: levels of independence

A strategy also decides who does the testing at each level. The ISTQB syllabus describes four degrees of independence: work products "can be tested by their author (no independence), by the author's peers from the same team (some independence), by testers from outside the author's team but within the organization (high independence), or by testers from outside the organization (very high independence)."

Neither extreme is right on its own. Independence helps "due to differences between the author's and the tester's cognitive biases", but "Independence is not, however, a replacement for familiarity, e.g., developers can efficiently find many defects in their own code." So the syllabus recommends mixing them: "For most projects, it is usually best to carry out testing with multiple levels of independence (e.g., developers performing component testing and component integration testing, test team performing system and system integration testing, and business representatives performing acceptance testing)." That is exactly the pattern in ExamReg's strategy above.

Kinds of test strategy

The 2018 Foundation syllabus, which the current version replaced, named seven common kinds of test strategy. The current syllabus no longer lists them, but the names are still in wide use and describe real choices.

The same syllabus's advice still holds: "An appropriate test strategy is often created by combining several of these types of test strategies", and its example is the very pair ExamReg uses, risk-based testing combined with exploratory testing.

When is testing complete?

Exhaustive testing is impossible, the second of the seven principles in Chapter Four, so testing never finishes by running out of tests. It finishes when agreed conditions are met. ISO/IEC/IEEE 29119-2 calls them completion criteria, "conditions under which the testing activities are considered complete", and Chapter Eleven, on writing a test plan, showed the typical exit criteria for each level: planned tests run, coverage reached, no open defect of critical severity, and so on. Three further points belong to strategy.

1. Completion is decided per level and then for the release. Each level's exit criteria are part of the next level's entry criteria; the release decision is made on the evidence of all of them.
2. Completion is a decision about risk. As Chapter Twenty-Three, on quality in software development, reported from the NIST study, deciding when to stop testing is the industry's hardest question, and the usual rules of thumb do not measure the risk that remains. The honest form of the decision is: the exit criteria are met, the remaining defects and risks are known, and the people who will carry them accept them.
3. Completion can be predicted. Reliability models fit the record of failures found during testing and forecast forward; in Lyu's words, one purpose is "to predict the extra time needed to test the software to achieve a specified objective". Chapter Ninety-Two, on measuring software reliability, computes such a prediction.

What it does not mean

A strategy is not a list of tests. It is the set of decisions that determines which tests are written, by whom, and when they are enough.

Testing from small to large does not mean the levels never overlap. In iterative development all of them run in every iteration, and the syllabus notes that "Test levels may overlap in time."

Independent testing does not mean developers stop testing. The recommended mix has developers testing components and their integration.

Testing is not complete when the schedule runs out. It is complete when the agreed criteria are met, or when the remaining risk is knowingly accepted.

Quick revision

• Test strategy: "part of the test plan that describes the approach to testing for a specific project, test level, or test type" (ISO/IEC/IEEE 29119-2:2021).
• A strategy decides the levels, the people, the techniques, the types, regression and completion.
• Test from the small to the large: component, component integration, system, system integration, acceptance, after reviews; smaller levels localise faults and each level has its own objectives.
• ExamReg: unit 35, integration 39, system 56, acceptance 45 per cent of the defects still present; together 90.5 per cent; with reviews, 188 of 200.
• Independence: author (none), peers (some), testers inside the organisation (high), outside it (very high); best to mix levels.
• Kinds of strategy (ISTQB 2018): analytical, model-based, methodical, process-compliant, directed, regression-averse, reactive; usually combined.
• Completion criteria: "conditions under which the testing activities are considered complete"; per level, then a risk-based release decision.

Test yourself

1. What is a test strategy, and what does it decide? The part of the test plan that describes the approach to testing for a project, test level or test type. It decides which test levels are used and in what order, who tests at each level, the techniques and test types, how regressions are prevented, and the criteria for completion.

2. Why does a testing strategy proceed from unit testing to system testing? Because a failure in a small test points to a small place, so faults are cheapest to locate there, and because each level has objectives it is best placed to meet: calculations in units, disagreements at interfaces in integration, whole-system behaviour in system testing, and fitness for use in acceptance testing.

3. Explain the levels of independence in testing, with their benefits and drawbacks. Testing may be done by the author, by peers, by testers outside the team, or by testers outside the organisation. Independent testers see different defects and challenge assumptions, but may become isolated, adversarial or a bottleneck, and developers may feel less responsible for quality; so a mix of levels is usually best.

4. Name four kinds of test strategy. Analytical (for example risk-based), model-based, methodical (for example checklist-based), process-compliant, directed, regression-averse and reactive (for example exploratory); any four of these.

5. How does a project decide that testing is complete? By completion criteria agreed in advance: exit criteria for each level such as tests run, coverage reached and no critical defect open, followed by a release decision in which the remaining risks are known and accepted by those who carry them. Reliability models can also predict how much more testing a target needs.

The four test types

Where a level is a place, a type is a purpose. The ISTQB syllabus names four.

• Functional testing "evaluates the functions that a component or system should perform". Its objective is checking "the functional completeness, functional correctness and functional appropriateness". It asks what the system does.
• Non-functional testing "evaluates attributes other than functional characteristics of a component or system", which the syllabus calls testing "how well the system behaves": performance efficiency, compatibility, usability, reliability, security, maintainability, portability and safety, the characteristics of ISO/IEC 25010 from Chapter Twenty, on the quality model today.
• Black-box testing "is specification-based and derives tests from documentation not related to the internal structure of the test object". ISO/IEC/IEEE 29119-1 calls the same thing specification-based testing, in which "the principal test basis is the external inputs and outputs of the test item".
• White-box testing "is structure-based and derives tests from the system's implementation or internal structure (e.g., code, architecture, work flows, and data flows)". Its objective is "to cover the underlying structure by the tests to an acceptable level". The standard's name is structure-based testing.

Notice that the four are not one list of alternatives. Functional and non-functional divide testing by what is being evaluated; black-box and white-box divide it by where the tests come from. A single test can be functional and black-box (a fee case from the fee table) or non-functional and white-box (a timing test aimed at one slow loop). Chapter Fifty-Two, on black-box and white-box testing, starts Module 2 with the techniques of each family.

The grid: every level, every type

The syllabus says the four types "can be applied to all test levels, although the focus will be different at each level." On ExamReg the grid looks like this.

Confirmation testing and regression testing

Two more kinds of testing are defined not by level or type but by what triggers them: a change. The syllabus says that after a change, "Testing should then also include confirmation testing and regression testing."

Confirmation testing, which ISO/IEC/IEEE 29119-2 calls retesting, is "testing performed to check that modifications made to correct a fault have successfully removed the fault". In the syllabus's version it "confirms that an original defect has been successfully fixed", by executing the tests that failed because of it and, where needed, new tests covering the change.

Regression testing is "testing performed following modifications to a test item or to its operational environment, to identify whether failures in unmodified parts of the test item occur" (ISO/IEC/IEEE 29119-1:2022). The syllabus stresses its scope: adverse consequences "could affect the same component where the change was made, other components in the same system, or even other connected systems", so "It is advisable first to perform an impact analysis to recognize the extent of the regression testing."

Worked example: a fix that passes, and a regression it causes

ExamReg's test team reports that a form submitted exactly 15 days late is refused, although the rule says it pays Rs 500. The developer fixes the first condition and, while the file is open, "tidies" another line. The program runs the confirmation test on the fix, then reruns the rest of the suite as a regression test.

def late_fee_before(days_late):          # the build tested: refuses a form 15 days late
    if days_late < 0 or days_late >= 15:
        return "refused"
    if days_late <= 0:
        return 0
    if days_late <= 7:
        return 100
    return 500

def late_fee_after(days_late):           # the developer's fix, with a tidy-up on the side
    if days_late < 0 or days_late > 15:
        return "refused"
    if days_late <= 0:
        return 0
    if days_late <= 8:                   # "tidied" while the file was open
        return 100
    return 500

suite = {"TC-01": (-1, "refused"), "TC-02": (0, 0), "TC-03": (1, 100), "TC-04": (7, 100),
         "TC-05": (8, 500), "TC-06": (15, 500), "TC-07": (16, "refused")}
failed_before = [t for t, (d, e) in suite.items() if late_fee_before(d) != e]
print("failed on the build tested:", ", ".join(failed_before))

for t in failed_before:                                   # confirmation testing
    d, e = suite[t]
    got = late_fee_after(d)
    print(f"confirmation {t} ({d} days late): expected {e}, got {got}:",
          "pass" if got == e else "FAIL")

rest = [t for t in suite if t not in failed_before]      # regression testing
broken = [t for t in rest if late_fee_after(suite[t][0]) != suite[t][1]]
print(f"regression: {len(rest)} tests rerun, {len(broken)} failed")
for t in broken:
    d, e = suite[t]
    print(f"  {t} ({d} days late): expected {e}, got {late_fee_after(d)}")

failed on the build tested: TC-06
confirmation TC-06 (15 days late): expected 500, got 500: pass
regression: 6 tests rerun, 1 failed
  TC-05 (8 days late): expected 500, got 100

The confirmation test passes: the reported failure is gone. A team that stopped there would ship a new defect, because the "tidy-up" moved day 8 into the Rs 100 band. Only the regression run, rerunning tests that had passed before and whose code nobody meant to change, catches it. This is why the syllabus calls regression suites "a strong candidate for automation": they are rerun after every change, they grow with every release, and the failures they catch are exactly the ones nobody is looking for. Chapter Thirty-Nine, on regression testing, smoke testing and continuous integration, runs them automatically on every build.

Where each kind of testing is taught in this book

The rest of Module 1 goes through the levels, and Module 2 through the techniques, in this order:

• Static testing: the kinds of V&V and reviews, inspection and the walkthrough, Chapters Twenty-Seven to Thirty.
• Unit testing: its purpose, techniques, frameworks and best practices, Chapters Thirty-Three to Thirty-Six.
• Integration testing: why units fail together, the integration approaches, regression and smoke testing with continuous integration, and the challenges of integration, Chapters Thirty-Seven to Forty.
• Validation and acceptance testing: validation testing and acceptance testing with alpha and beta, Chapters Forty-One and Forty-Two.
• System testing: system testing end to end, the system test types, load testing and cross-browser testing, Chapters Forty-Three to Forty-Six.
• Debugging, which follows a failure at any level, Chapter Forty-Seven.
• Test automation: automation, browser drivers, data-driven testing and the page object model, Chapters Forty-Eight to Fifty-One.
• Black-box, white-box and experience-based techniques in Module 2, from black-box and white-box testing in Chapter Fifty-Two to checklist-based testing in Chapter Sixty-Four.

What it does not mean

A test level is not a test type. "System testing" says where; "performance testing" says what; a system-level performance test is both.

White-box testing is not confined to unit testing. Structure can be covered at every level: code at component level, call paths at integration level, workflows at system level.

Confirmation testing is not regression testing. The first checks the fix; the second checks everything the fix might have disturbed.

Regression testing is not only for code changes. A change to the operating system, the database or a connected system can break unchanged code, and the definition includes changes to the "operational environment".

Quick revision

• Test level: a test stage with its own objectives and risks. Five: component, component integration, system, system integration, acceptance.
• Levels differ in test object, objectives, test basis, defects and failures, and approach and responsibilities.
• Test type: testing focused on specific quality characteristics. Four: functional (what), non-functional (how well), black-box (from the specification), white-box (from the structure). All four apply at every level.
• Confirmation testing (retesting): checks the fix removed the fault. Regression testing: checks unmodified parts still work after a change, scoped by impact analysis, and automated.
• Worked example: the fix for day 15 passed its confirmation test; regression caught the new day-8 defect.

Test yourself

1. Name the five test levels and give the test object of each. Component testing (a single unit in isolation), component integration testing (units combined and their interfaces), system testing (the whole system), system integration testing (the system's interfaces with external systems) and acceptance testing (the system in its intended use).

2. Name the four test types and state what each evaluates. Functional testing evaluates what the system does; non-functional testing how well it behaves (performance, usability, security and so on); black-box testing derives tests from the specification, not the internal structure; white-box testing derives them from the internal structure, aiming to cover it.

3. Can a test type be applied at more than one level? Give an example. Yes; all four types apply at all levels. Performance, a non-functional type, can be tested at component level (the fee function's speed) and at system level (500 students on the last date).

4. Distinguish confirmation testing from regression testing. Confirmation testing reruns the tests that failed, and new tests for the change, to check that a defect has been fixed. Regression testing reruns tests that passed before, chosen by impact analysis, to check that the change has not broken anything else in the component, the system, connected systems or the environment.

5. Why is regression testing a strong candidate for automation? Because the same tests are rerun after every change, the suite grows with every release, and running it by hand each time would be slow and error-prone; automated in continuous integration, it runs on every build.

The standards stress one more thing: unit tests are normally written by the developer who wrote the unit. The ISTQB syllabus says component testing "is normally performed by developers in their development environments", and ISO/IEC TR 7052 allows "the developer or an independent tester". It is the one level where the author is expected to test their own work, because no one else knows the unit's inside as well, and the tests are needed long before a test team sees the code.

What a unit test checks

A useful checklist for testing one unit has five parts.

Each aspect has a chapter of its own later. Boundaries become boundary value analysis (Chapter Fifty-Five); paths become branch testing and, made exact, basis path testing (Chapter Seventy-Two); interfaces between units become integration testing (Chapter Thirty-Seven).

Worked example: unit testing the fee calculator

Here is ExamReg's fee calculator, with the fee rules fixed for this book: a form fee of Rs 800, Rs 150 for each backlog paper, the late-fee bands, forms more than 15 days late or with a negative number of days refused, and a concession that waives the form fee only. The checks below take the five aspects in turn. Each is a line of ordinary Python that calls the unit with chosen inputs and compares the answer with the rule; Chapter Thirty-Five, on writing unit tests with a framework, shows the same checks as a proper test suite.

FORM_FEE, BACKLOG_FEE = 800, 150

def total_fee(days_late, backlog_papers, concession):      # the unit under test
    if days_late < 0 or days_late > 15:
        raise ValueError("form not accepted")
    fee = FORM_FEE + BACKLOG_FEE * backlog_papers
    if concession:
        fee -= FORM_FEE                                      # a concession waives the form fee
    if days_late == 0:
        late = 0
    elif days_late <= 7:
        late = 100
    else:
        late = 500
    return fee + late

def refused(*args):
    try:
        total_fee(*args)
    except ValueError:
        return True
    return False

checks = {
    "interface: returns a whole number of rupees": isinstance(total_fee(0, 0, False), int),
    "local data: form fee plus two backlog papers": total_fee(0, 2, False) == 1100,
    "boundary: 7 days late pays Rs 100 late fee": total_fee(7, 0, False) == 900,
    "boundary: 8 days late pays Rs 500 late fee": total_fee(8, 0, False) == 1300,
    "boundary: 15 days late is accepted": total_fee(15, 0, False) == 1300,
    "boundary: 16 days late is refused": refused(16, 0, False),
    "path: no concession, on time": total_fee(0, 0, False) == 800,
    "path: concession, 3 days late, 2 backlog papers": total_fee(3, 2, True) == 400,
    "error handling: negative days refused": refused(-1, 0, False),
    "error handling: negative backlog papers refused": refused(0, -2, False),
}
for name, passed in checks.items():
    print("pass" if passed else "FAIL", " ", name)
print(f"{sum(checks.values())} of {len(checks)} checks passed;",
      "total_fee(0, -2, False) returned", total_fee(0, -2, False))

pass   interface: returns a whole number of rupees
pass   local data: form fee plus two backlog papers
pass   boundary: 7 days late pays Rs 100 late fee
pass   boundary: 8 days late pays Rs 500 late fee
pass   boundary: 15 days late is accepted
pass   boundary: 16 days late is refused
pass   path: no concession, on time
pass   path: concession, 3 days late, 2 backlog papers
pass   error handling: negative days refused
FAIL   error handling: negative backlog papers refused
9 of 10 checks passed; total_fee(0, -2, False) returned 500

Nine checks pass, and the one that fails is exactly the kind a unit test exists to catch. The calculator validates days late but never validates backlog papers, so a student record corrupted to hold -2 backlog papers produces a bill of Rs 500: the form fee of Rs 800 less Rs 300 for "negative" papers. No end-to-end system test is likely to try that input, because the form's own screen would never offer it; but a unit can be called by any other code, including a data import or a future screen, and it has to defend itself. The fix is one line at the top of the unit, refusing a negative or non-whole number of backlog papers as it already refuses bad days, and the failing check becomes the confirmation test for the fix.

Three things about the example are true of unit testing in general. The tests ran in a fraction of a second, so they can run every time the code is saved. The failure named the unit and the input exactly, so there was nothing to debug but one condition. And the checks stay with the code, so the next person to change total_fee inherits a record of what it must do.

What it does not mean

Unit testing is not debugging. The test shows that the unit fails for -2 backlog papers; finding and fixing the missing check is debugging, a separate activity (Chapter Forty-Seven).

Passing unit tests do not show that the units work together. The fee calculator can pass all its tests while the payment adapter misreads the total it returns; that is integration testing's job.

A unit is not always one function. It is the smallest separately testable element, which may be a class or a small module.

Unit tests are not a chore to add at the end. Written as the unit is written, or before it as the next chapters show, they cost least and find most.

Quick revision

• Unit: a separately testable element of a software component; software unit: an atomic-level component that can be tested standalone.
• Unit test: testing of individual routines and modules, usually by the developer, in the development environment.
• Purposes: find defects where they are made and early; enable safe change; improve design and testability.
• Late fixes are "often 100 times more expensive" than early ones, "more like 5:1" for small, noncritical systems (Boehm and Basili 2001).
• Checklist: interface, local data, boundaries, paths, error handling.
• Worked example: nine of ten checks passed; the unit accepted -2 backlog papers and billed Rs 500.

Test yourself

1. What is a unit, and what is unit testing? A unit is the smallest separately testable element of a program, such as a function or a class. Unit testing is the testing of individual units in isolation, normally by the developer in the development environment, to find defects in each unit's own logic.

2. State four purposes of unit testing. To find defects where they are made, in a small known place; to find them early, when fixing is cheapest; to provide tests that can be rerun after every change as regression tests; and to expose design problems, since a unit that is hard to test alone is usually doing too much.

3. What aspects of a unit should its tests check? Its interface (what it accepts and returns), its local data (its own variables), its boundaries (every edge and either side of it), its paths (every branch at least once), and its error handling (bad input refused cleanly).

4. Why are defects found by unit tests cheaper to fix than those found later? Because a failing unit test points to one small unit and one input, so diagnosis is quick, and because the defect has not yet been built upon by other code or shipped to users; Boehm and Basili found fixes after delivery often cost 100 times more than early fixes.

5. Who usually writes unit tests, and why? The developer who wrote the unit, because they know its inside best and the tests are needed as soon as the unit exists, long before an independent test team sees the code.

The standard vocabulary has an older, looser entry for the last: a mock object is one of the "temporary dummy objects created to aid testing until the real objects become available" (ISO/IEC/IEEE 24765). In everyday speech the words are used loosely; as Fowler puts it, "all sorts of words are used: stub, mock, fake, dummy". Meszaros's names are worth learning because each says what the double does.

State verification and behaviour verification

The doubles fall into two groups by how a test uses them to decide pass or fail. Fowler draws the distinction exactly.

• State verification "means that we determine whether the exercised method worked correctly by examining the state of the SUT and its collaborators after the method was exercised." (SUT is the system under test.) The test checks results: the amount returned, the messages the spy recorded.
• Behaviour verification means "we instead check to see if the order made the correct calls on the warehouse", in his example; in ours, whether the fee service made the right call on the gateway. The test checks interactions.

"Of these kinds of doubles, only mocks insist upon behavior verification. The other doubles can, and usually do, use state verification."

Worked example: one unit, five doubles

Python's standard library includes unittest.mock, which, in its documentation's words, "allows you to replace parts of your system under test with mock objects and make assertions about how they have been used." The program uses it for the stub, the mock and the dummy, and writes the fake and the spy by hand, so that what each double is stays visible. The fee rules are the book's fixed ones; 10 days late with one backlog paper is Rs 800 + Rs 150 + Rs 500.

from datetime import date
from unittest.mock import Mock, sentinel

FORM_FEE, BACKLOG_FEE = 800, 150

def total_fee(days_late, backlog_papers, concession):
    if days_late < 0 or days_late > 15:
        raise ValueError("form not accepted")
    if not isinstance(backlog_papers, int) or backlog_papers < 0:
        raise ValueError("backlog papers must be a whole number, 0 or more")
    fee = FORM_FEE + BACKLOG_FEE * backlog_papers - (FORM_FEE if concession else 0)
    late = 0 if days_late == 0 else (100 if days_late <= 7 else 500)
    return fee + late

class FeeService:                                     # the unit under test
    def __init__(self, students, clock, gateway, mailer, audit_log):
        self.students, self.clock = students, clock
        self.gateway, self.mailer, self.audit_log = gateway, mailer, audit_log

    def pay(self, roll_no, last_date):
        student = self.students.find(roll_no)
        days_late = max(0, (self.clock.today() - last_date).days)
        amount = total_fee(days_late, student["backlog"], student["concession"])
        receipt = self.gateway.charge(roll_no, amount)
        self.mailer.send(student["email"], f"Paid Rs {amount}; receipt {receipt}")
        return amount

    def refund(self, roll_no, amount):                # not exercised by this test
        self.audit_log.write(f"refund {roll_no} {amount}")
        return self.gateway.refund(roll_no, amount)

class FakeStudentStore:                               # FAKE: really works, but only in memory
    def __init__(self):
        self.rows = {}
    def add(self, roll_no, **row):
        self.rows[roll_no] = row
    def find(self, roll_no):
        return self.rows[roll_no]

class MailerSpy:                                      # SPY: records how it was called
    def __init__(self):
        self.sent = []
    def send(self, to, text):
        self.sent.append((to, text))

# the DRIVER: set up the doubles, call the unit, check what happened
students = FakeStudentStore()
students.add("2026CS014", backlog=1, concession=False, email="s014@college.example")
clock = Mock()                                        # STUB: a canned answer for today()
clock.today.return_value = date(2026, 11, 20)
gateway = Mock()                                      # MOCK: the calls it receives are verified
gateway.charge.return_value = "R-1042"
mailer = MailerSpy()
service = FeeService(students, clock, gateway, mailer, audit_log=sentinel.audit_log)  # DUMMY

amount = service.pay("2026CS014", last_date=date(2026, 11, 10))
print("amount returned:", amount)                                 # state verification
gateway.charge.assert_called_once_with("2026CS014", 1450)         # behaviour verification
print("gateway charged:", gateway.charge.call_args)
print("mails sent:", len(mailer.sent), "to", mailer.sent[0][0])
print("receipt text:", mailer.sent[0][1])

class CarelessFeeService(FeeService):                 # a defect: forgets the backlog paper
    def pay(self, roll_no, last_date):
        student = self.students.find(roll_no)
        days_late = max(0, (self.clock.today() - last_date).days)
        amount = total_fee(days_late, 0, student["concession"])
        self.gateway.charge(roll_no, amount)
        return amount

gateway = Mock()
CarelessFeeService(students, clock, gateway, MailerSpy(), sentinel.audit_log).pay(
    "2026CS014", last_date=date(2026, 11, 10))
try:
    gateway.charge.assert_called_once_with("2026CS014", 1450)
except AssertionError as problem:
    print("the mock rejects the careless version:", str(problem).splitlines()[0])
    print("  actually called:", gateway.charge.call_args)

amount returned: 1450
gateway charged: call('2026CS014', 1450)
mails sent: 1 to s014@college.example
receipt text: Paid Rs 1450; receipt R-1042
the mock rejects the careless version: expected call not found.
  actually called: call('2026CS014', 1300)

Each double did one job, and together they made a unit test out of something that would otherwise need a database, a calendar, a bank and a mail server.

• The fake store answered the lookup with a real, working record, held in memory.
• The stub clock gave the same "today" on every run, so the test is repeatable: the form is 10 days late whenever the test runs, not only on 20 November.
• The mock gateway was never charged real money, and the test verified the call it received: exactly once, for student 2026CS014, for Rs 1,450. That is behaviour verification.
• The spy mailer kept the receipt instead of sending it, so the test could read its address and text afterwards. That is state verification on a double.
• The dummy filled a parameter the test never needed. sentinel supplies a unique object that fits the purpose: if the code ever did use it, the test would fail loudly.

The second half of the program shows what behaviour verification is for. A careless version of pay forgets the backlog paper and charges Rs 1,300. The mock's check, assert_called_once_with, reports "expected call not found" and shows the call it actually received, pointing straight at the missing Rs 150.

Choosing doubles well

Doubles are powerful, and Fowler's essay warns about the cost of using them too freely. A test built on mocks checks how a unit talks to its collaborators, so "Mockist tests are thus more coupled to the implementation of a method. Changing the nature of calls to collaborators usually cause a mockist test to break." A few rules of thumb follow.

• Use the real thing when it is cheap and deterministic. total_fee is a plain function; the test above calls the real one rather than doubling it.
• Double what is slow, costly, uncontrollable or unfinished: payment, email, the clock, a network service, a unit not yet written.
• Mock only the calls that matter to the requirement. The gateway charge matters; how many times the student store is read does not, so it is a fake, not a mock.
• Make time and randomness injectable, as the clock is here. A unit that reads the system clock directly cannot be tested for "10 days late" without waiting ten days; this is the testability point of Chapter Twenty-One, on how quality factors shape testing.

What it does not mean

A stub is not a mock. A stub supplies answers; a mock also checks the calls it receives. The title of Fowler's essay says it: mocks aren't stubs.

Test doubles do not prove the real collaborators work. The mock gateway accepts any call the test allows; whether the real gateway does is a question for integration testing (Chapter Thirty-Seven).

A driver is not only for procedural code. In a test framework, each test method is a driver; the framework runs them all.

More doubles do not make a better test. Each one couples the test to the unit's design, so use the fewest that make the test fast and repeatable.

Quick revision

• Test driver: invokes the module under test, provides inputs, monitors execution and reports results. Stub: a skeletal implementation standing in for a module the unit calls. Together, the test harness.
• Test double (Meszaros, via Fowler): any pretend object used in place of a real one for testing.
• Dummy: passed but never used. Fake: works, with a shortcut (in-memory database). Stub: canned answers. Spy: a stub that records how it was called. Mock: pre-programmed with expectations of the calls it should receive.
• State verification: check the state after the call. Behaviour verification: check the calls made; only mocks insist on it.
• unittest.mock: Mock, return_value, assert_called_once_with, call_args, sentinel.
• Mock-heavy tests are coupled to the implementation: double only what is slow, costly, uncontrollable or unfinished.

Test yourself

1. What are a test driver and a stub? Why are they needed in unit testing? A driver is a module that calls the unit under test with test inputs, monitors it and reports results; a stub is a simplified stand-in for a module that the unit calls. They are needed because a unit cannot run alone: its caller and its collaborators may not exist yet, or may be slow, costly or uncontrollable.

2. Name and explain the five kinds of test double. A dummy is passed but never used, only filling a parameter list. A fake has a working but simplified implementation, such as an in-memory database. A stub gives canned answers to the calls made in the test. A spy is a stub that also records how it was called. A mock is pre-programmed with expectations of the calls it should receive, and the test checks them.

3. Distinguish state verification from behaviour verification. State verification checks whether the unit worked by examining the state of the unit and its collaborators after the call, such as the amount returned. Behaviour verification checks whether the unit made the correct calls on its collaborators, such as charging the gateway once with the right amount; only mocks insist on it.