Test Design Techniques: The Three Families

Family two: white-box, or structure-based

ISO/IEC/IEEE 29119-1 defines structure-based testing as "dynamic testing in which the tests are derived from an examination of the structure of the test item". The box is now transparent: the tester reads the code, finds its statements, decisions and paths, and designs tests to exercise them.

This family has a mirror-image limitation. The syllabus: "As the test cases are dependent on how the software is designed, they can only be created after the design or implementation of the test object." And if the code changes, the tests may need to change with it. The white-box techniques MU names are statement testing (Chapter Fifty-Nine) and branch testing (Chapter Sixty), with structural testing introduced as a whole in Chapter Fifty-Eight.

Family three: experience-based

ISO/IEC/IEEE 29119-4:2021 defines experience-based testing as a "class of test case design techniques based on using the experience of testers to generate test cases". A tester who has seen date fields fail on 29 February, and forms fail when a user double-clicks Submit, tries those things first.

The syllabus is candid about both sides: the effectiveness of these techniques "depends heavily on the tester's skills", and yet they "can detect defects that may be missed using the black-box test techniques and white-box test techniques. Hence, experience-based test techniques are complementary". MU names three: error guessing (Chapter Sixty-Two), exploratory testing (Chapter Sixty-Three) and checklist-based testing (Chapter Sixty-Four).

The three families compared

The "misses" row is the reason all three are used. A black-box tester cannot see a stray line of code that does something nobody asked for. A white-box tester cannot see a requirement the programmer forgot, because there is no code for it to cover. And both follow rules, which is exactly where an experienced tester's instinct for the unexpected earns its place.

Worked example: three families, three defects

Below, ExamReg's late-fee function contains three planted defects of three different kinds. The oracle is the fee table, with one clarification the exam cell gave after the review of the requirement in Chapter One, on what software testing is: a negative number of days, or anything that is not a whole number of days, must be refused cleanly. Each family's tests are chosen the way that family chooses them, and the program reports what each finds.

def late_fee(days_late):
    if days_late > 15:
        raise ValueError("form not accepted more than 15 days late")
    if days_late <= 0:                  # defect C: the input is never validated
        return 0
    if days_late == 13:                 # defect B: a forgotten debugging shortcut
        return 0
    if days_late <= 6:                  # defect A: the rule says 7, not 6
        return 100
    return 500

def expected(days):                     # the oracle: the fee table and the exam cell's ruling
    if not isinstance(days, int) or days < 0 or days > 15:
        return "refused"
    return 0 if days == 0 else (100 if days <= 7 else 500)

def actual(days):
    try:
        return late_fee(days)
    except ValueError:
        return "refused"
    except Exception as problem:        # anything else is a crash, not a clean refusal
        return "crash: " + type(problem).__name__

families = {
    "black-box, from the fee table":    [0, 1, 7, 8, 15, 16],
    "white-box, one test per branch":   [16, 0, 13, 3, 10],
    "experience-based, likely mistakes": [-1, "7", 7.5],
}
for family, inputs in families.items():
    found = [f"{d!r}: expected {expected(d)}, got {actual(d)}"
             for d in inputs if actual(d) != expected(d)]
    print(f"{family}: {len(inputs)} tests, {len(found)} failure(s)")
    for line in found:
        print("   ", line)

black-box, from the fee table: 6 tests, 1 failure(s)
    7: expected 100, got 500
white-box, one test per branch: 5 tests, 1 failure(s)
    13: expected 500, got 0
experience-based, likely mistakes: 3 tests, 3 failure(s)
    -1: expected refused, got 0
    '7': expected refused, got crash: TypeError
    7.5: expected refused, got 500

Each family found what the others could not. The black-box tests were chosen at the edges of the fee table, so 7 days, the boundary the programmer got wrong, was among them; but 13 is an ordinary value in the middle of a band, and no specification-based technique would single it out. The white-box tests were chosen to make every branch of the code run at least once, so the strange == 13 branch was forced to execute and gave itself away; but no branch exists for "exactly 7", so branch testing had no reason to try it. And the experience-based tests tried what users actually do wrong: a negative number, a number typed as text, a fraction. All three failed, and all three are the same defect showing three faces: the function never validates its input, so it charges nothing for minus one day, crashes on text, and charges Rs 500 for seven and a half days. The fee table never mentioned any of these inputs, so no specification-based test would have tried them.

The lesson is the one the ISTQB syllabus gives: the families are complementary. A test plan that uses only one of them has chosen, in advance, which kinds of defect it will not find.

The wider catalogue

MU's syllabus names nine techniques. The software testing standard ISO/IEC/IEEE 29119-4:2021 defines more, and a student meeting their names elsewhere should know which family each belongs to.

Choosing a technique

No technique is best everywhere, which is Chapter Four's sixth principle (testing is context dependent) applied to design. A few rules of thumb from practice:

• Input fields with ranges (marks, days, amounts): equivalence partitioning with boundary value analysis.
• Business rules combining several conditions (fee waivers, eligibility): decision tables.
• Behaviour that depends on history (login attempts, an order's status): state transition testing.
• Safety-critical or complex code: white-box coverage targets, set in the test plan.
• New, poorly specified or rushed features: exploratory testing, guided by checklists.
• Every time: some error guessing, because it costs little and finds what rules do not.

What it does not mean

Black-box does not mean careless or blind. It means the tests are derived from the specification without looking at the code; the design is as systematic as any white-box technique.

White-box testing is not only for developers. Anyone who can read the code can use it, though in practice developers apply it most, at unit level.

Experience-based testing is not random clicking. It is disciplined by charters, checklists and notes, as Module 2 shows; and random testing is a separate, specification-based technique in ISO/IEC/IEEE 29119-4.

100 per cent coverage does not mean no defects remain. It means every coverage item of that technique was exercised. In the worked example, 100 per cent branch coverage missed defect A.

Quick revision

• Test design technique: a "procedure used to create or select a test model, identify test coverage items, and derive corresponding test cases" (ISO/IEC/IEEE 29119-2).
• Test coverage: the percentage of specified coverage items exercised by the tests.
• Black-box (specification-based): from the specification; tests survive code changes; misses code nobody specified. MU: equivalence partitioning, boundary value analysis, decision tables, state transitions.
• White-box (structure-based): from the code; needs the code; misses behaviour never coded. MU: statement and branch testing.
• Experience-based: from the tester's experience; complementary to both. MU: error guessing, exploratory, checklist-based.
• Worked example: each family found exactly one kind of planted defect the others missed.

Test yourself

1. What is a test design technique, and why are techniques used? A procedure for identifying test coverage items and deriving test cases from a test basis. They are used because exhaustive testing is impossible, and they produce a small but sufficient set of tests systematically, so the result is repeatable and its coverage can be measured.

2. Name the three families of techniques and give MU's examples of each. Black-box or specification-based: equivalence partitioning, boundary value analysis, decision table testing and state transition testing. White-box or structure-based: statement testing and branch testing. Experience-based: error guessing, exploratory testing and checklist-based testing.

3. Why can white-box tests not be written at the start of a project? Because they are derived from the internal structure of the code or detailed design, which does not exist until design or implementation has been done.

4. Give one kind of defect each family is good at finding and one it misses. Black-box finds wrong or missing behaviour against the specification but misses code nobody specified; white-box finds untested or unexpected code but misses required behaviour that was never coded; experience-based finds unusual real-world inputs but misses whatever the tester has never encountered.

5. In the worked example, which family found the forgotten debugging shortcut, and why could the others not? White-box branch testing, because it had to execute every branch, including the one for 13 days. Black-box tests had no reason to choose 13, an ordinary value in the middle of a band, and experience-based tests looked at invalid inputs instead.

6. Does 100 per cent branch coverage guarantee a correct program? No. It guarantees each branch ran at least once. In the worked example the branch tests achieved full branch coverage and still missed the wrong boundary at 7 days, because no branch exists for a value the code handles wrongly within a range.