Equivalence Partitioning

Worked problem 1: MU's own grade table

MU prints the rule in the circular that carries this syllabus, as its table of letter grades and grade points:

Notice how the table is written: 80.0 to < 90.0, not 80 to 90. Every value belongs to exactly one row, which is the syllabus's rule that partitions "must not overlap". The table is a partition of the input already, and it gives the tester nine valid partitions, one of them not a number at all: a student who was absent has no percentage, and the input is the mark Ab.

The table does not say what to do with values no student can have, so the tester adds the invalid partitions the specification leaves implicit and records the decision: a percentage below 0, a percentage above 100, and an input that is not a number. Below 40.0 is read as 0 to below 40.0, because nothing below 0 is a percentage. That gives twelve partitions and twelve tests, one value from the middle of each.

The function under test is printed so the program can import it; the tester works from MU's table.

def grade(percent):
    """Letter grade and grade point for a percentage of marks, or for "Ab" (absent)."""
    if percent == "Ab":
        return ("Ab", 0)
    if not isinstance(percent, (int, float)) or percent < 0:
        raise ValueError("not a percentage of marks")
    if percent >= 90.0:
        return ("O", 10)
    if percent > 80.0:
        return ("A+", 9)
    if percent >= 70.0:
        return ("A", 8)
    if percent >= 60.0:
        return ("B+", 7)
    if percent >= 50.0:
        return ("C", 5)
    if percent >= 40.0:
        return ("P", 4)
    return ("F", 0)

from mu_grade import grade

# (partition, valid?, one value from inside it, expected result from MU's table)
partitions = [("90.0 to 100",     "valid",   95,        ("O", 10)),
              ("80.0 to < 90.0",  "valid",   85,        ("A+", 9)),
              ("70.0 to < 80.0",  "valid",   75,        ("A", 8)),
              ("60.0 to < 70.0",  "valid",   65,        ("B+", 7)),
              ("55.0 to < 60.0",  "valid",   57.5,      ("B", 6)),
              ("50.0 to < 55.0",  "valid",   52.5,      ("C", 5)),
              ("40.0 to < 50.0",  "valid",   45,        ("P", 4)),
              ("0 to < 40.0",     "valid",   20,        ("F", 0)),
              ("absent",          "valid",   "Ab",      ("Ab", 0)),
              ("below 0",         "invalid", -5,        "refused"),
              ("above 100",       "invalid", 120,       "refused"),
              ("not a number",    "invalid", "seventy", "refused")]

def run(value):
    try:
        letter, point = grade(value)
        return f"{letter} ({point})"
    except ValueError:
        return "refused"

failed = 0
for name, kind, value, expected in partitions:
    want = expected if expected == "refused" else f"{expected[0]} ({expected[1]})"
    got = run(value)
    failed += got != want
    print(f"{name:<16} {kind:<8} {value!r:<10} expected {want:<8} got {got:<8}"
          f" {'pass' if got == want else 'FAIL'}")
print(f"partition coverage: {len(partitions)} of {len(partitions)} partitions, {failed} failed")

90.0 to 100      valid    95         expected O (10)   got O (10)   pass
80.0 to < 90.0   valid    85         expected A+ (9)   got A+ (9)   pass
70.0 to < 80.0   valid    75         expected A (8)    got A (8)    pass
60.0 to < 70.0   valid    65         expected B+ (7)   got B+ (7)   pass
55.0 to < 60.0   valid    57.5       expected B (6)    got C (5)    FAIL
50.0 to < 55.0   valid    52.5       expected C (5)    got C (5)    pass
40.0 to < 50.0   valid    45         expected P (4)    got P (4)    pass
0 to < 40.0      valid    20         expected F (0)    got F (0)    pass
absent           valid    'Ab'       expected Ab (0)   got Ab (0)   pass
below 0          invalid  -5         expected refused  got refused  pass
above 100        invalid  120        expected refused  got O (10)   FAIL
not a number     invalid  'seventy'  expected refused  got refused  pass
partition coverage: 12 of 12 partitions, 2 failed

Twelve tests, 100 per cent partition coverage, and two failures, each of a kind equivalence partitioning is built to catch.

A missing partition. A student with 57.5 per cent should get B (Above Average), grade point 6, and gets C. The code has no branch for the B band at all: its test for C begins at 50.0, so every value from 55.0 to below 60.0 falls into C. One representative value from the B partition was enough to show it, and any other value from the partition would have shown the same.

A missing rejection. A percentage of 120 should be refused and is given O. The code checks for values below 0 and never for values above 100. The invalid partition exists only because the tester wrote it down; a test set built from the valid rows of MU's table alone would never have tried it.

The code has a third defect, which none of these twelve tests touches. Every value tested sat in the middle of its partition, where, if the partitions are right, one value is as good as another. The places where that assumption fails are the edges, and Chapter Fifty-Five, on boundary value analysis, tests them.

Coverage

The ISTQB syllabus defines the measure: "Coverage is measured as the number of partitions exercised by at least one test case, divided by the total number of identified partitions, and is expressed as a percentage." To reach 100 per cent, "test cases must exercise all identified partitions (including invalid partitions)". The twelve tests above reached 12 of 12.

Two words in that definition deserve attention. Identified: coverage is measured against the partitions the tester found, so a partition nobody identified (had the tester not thought of percentages above 100) lowers nothing and is never tested. And including invalid partitions: a suite that tests only valid values cannot reach 100 per cent.

Several inputs: Each Choice coverage

Most functions take more than one input, and each input has its own partitions. ExamReg's fee rule has three inputs:

That is ten partitions in all. Every combination of them would be 5 × 3 × 2 = 30 tests; working through combinations in a disciplined way is what decision tables do, in Chapter Fifty-Six. Equivalence partitioning asks for less. The ISTQB syllabus calls the simplest criterion Each Choice coverage (after Ammann and Offutt), and it "requires test cases to exercise each partition from each set of partitions at least once". Since one test takes one value for each input, it covers one partition of each at once, and the smallest set that achieves Each Choice coverage has as many tests as the input with the most partitions: here five.

One invalid value per test

The smallest set has a trap in it, and the program below springs it. The version under test checks for negative backlog papers but has forgotten to refuse a number that is not whole.

def total_fee(days_late, backlog_papers, concession):        # the version under test
    if days_late < 0 or days_late > 15:
        raise ValueError("form not accepted")
    if backlog_papers < 0:
        raise ValueError("backlog papers cannot be negative")
    fee = (0 if concession else 800) + 150 * backlog_papers
    late = 0 if days_late == 0 else (100 if days_late <= 7 else 500)
    return fee + late

# the partitions of each input: (name, valid?, the test that a value belongs to it)
partitions = {
    "days late": [("negative", False, lambda d: d < 0),     ("0", True, lambda d: d == 0),
                  ("1 to 7", True, lambda d: 1 <= d <= 7),  ("8 to 15", True, lambda d: 8 <= d <= 15),
                  ("over 15", False, lambda d: d > 15)],
    "backlog papers": [("negative whole", False, lambda b: b == int(b) and b < 0),
                       ("whole, 0 or more", True, lambda b: b == int(b) and b >= 0),
                       ("not whole", False, lambda b: b != int(b))],
    "concession": [("yes", True, lambda c: c is True), ("no", True, lambda c: c is False)],
}

def check(name, tests):
    covered, combined, lines = set(), 0, []
    for args, expected in tests:
        invalid = 0
        for (inp, parts), value in zip(partitions.items(), args):
            for part, valid, contains in parts:
                if contains(value):
                    covered.add((inp, part))
                    invalid += not valid
        combined += invalid > 1
        try:
            actual = total_fee(*args)
        except ValueError:
            actual = "refused"
        lines.append(f"   {str(args):<18} expected {str(expected):<8} got {str(actual):<8}"
                     f" {'pass' if actual == expected else 'FAIL'}")
    total = sum(len(parts) for parts in partitions.values())
    print(f"{name}: {len(tests)} tests, Each Choice coverage {len(covered)} of {total},"
          f" {combined} test(s) with more than one invalid value")
    print("\n".join(lines))

check("smallest set", [((-2, -1, True), "refused"), ((20, 1.5, False), "refused"),
                       ((0, 2, True), 300), ((4, 0, False), 900), ((10, 1, True), 650)])
check("one invalid value per test",
      [((-2, 0, False), "refused"), ((20, 0, False), "refused"), ((0, -1, False), "refused"),
       ((0, 1.5, False), "refused"), ((0, 2, True), 300), ((4, 0, False), 900),
       ((10, 1, True), 650)])

smallest set: 5 tests, Each Choice coverage 10 of 10, 2 test(s) with more than one invalid value
   (-2, -1, True)     expected refused  got refused  pass
   (20, 1.5, False)   expected refused  got refused  pass
   (0, 2, True)       expected 300      got 300      pass
   (4, 0, False)      expected 900      got 900      pass
   (10, 1, True)      expected 650      got 650      pass
one invalid value per test: 7 tests, Each Choice coverage 10 of 10, 0 test(s) with more than one invalid value
   (-2, 0, False)     expected refused  got refused  pass
   (20, 0, False)     expected refused  got refused  pass
   (0, -1, False)     expected refused  got refused  pass
   (0, 1.5, False)    expected refused  got 1025.0   FAIL
   (0, 2, True)       expected 300      got 300      pass
   (4, 0, False)      expected 900      got 900      pass
   (10, 1, True)      expected 650      got 650      pass

Both sets reach Each Choice coverage of 10 of 10. The smallest set passes completely. The second finds the defect: one and a half backlog papers is charged Rs 1025.0 instead of being refused.

The smallest set had a test for the not whole partition, (20, 1.5, False), and it passed, because the form was refused for being 20 days late before the backlog papers were ever examined. The second invalid value never had a chance to fail. This is fault masking, a "condition in which one fault prevents the detection of another" (ISO/IEC/IEEE 24765). The 2018 edition of the ISTQB syllabus gives the rule for partitions: invalid partitions "should be tested individually, i.e., not combined with other invalid equivalence partitions, to ensure that failures are not masked." The current syllabus says the same of state transitions: "Testing only one invalid transition in a single test case helps to avoid defect masking". The price is two extra tests, and it buys a test that can actually fail.

The procedure, in five steps

1. List the inputs and outputs the specification names, including the ones that are not numbers (the Ab mark, the concession).
2. Partition each into valid and invalid partitions, from the specification's own groupings, and add the invalid partitions it leaves implicit; check that no two partitions overlap and none is empty.
3. Choose one value from each partition, usually from its middle, away from the edges that the next chapter tests.
4. Combine the values into test cases: with several inputs, reach Each Choice coverage, and put only one invalid value in any test.
5. Write each expected result from the specification, run, and measure coverage as partitions exercised over partitions identified.

Strengths and limits

Strengths. Few tests replace many: twelve tests for a grade function that accepts a thousand valid percentages. Every partition of the specification is tried, so a missing rule, like the B band, shows itself. Invalid partitions make rejection part of the test design instead of an afterthought. And coverage is a clear number.

Limits. The technique rests on the partitions being right. If the code splits a partition the specification treats as one (a special case at some value inside it), one representative will usually miss it. It tests the middles and says nothing about the edges, which is where the next chapter looks. And Each Choice coverage ignores combinations of partitions, which is where decision tables look.

What it does not mean

Equivalent does not mean equal. The values of a partition are equivalent for testing, because the specification treats them alike; they still produce different outputs (every percentage from 80.0 to below 90.0 is A+, and every backlog count gives a different fee).

One value per partition is not a law. It is the minimum. A tester may choose more, and should, where a partition is large or the risk is high.

Invalid partitions are not optional. Coverage counts them, and the defects they find (the grade given for 120 per cent) are defects a user will meet.

Each Choice coverage is not combination coverage. It exercises every partition, not every combination of partitions.

Quick revision

• Equivalence partition: a "class of inputs or outputs that are expected to be treated similarly by the test item" (ISO/IEC/IEEE 29119-4); partitions must not overlap and must be non-empty (ISTQB).
• One value per partition, because a defect shown by one value should be shown by any other in the same partition.
• Valid partitions are processed, invalid ones rejected; conventions vary, so write them down.
• Coverage = partitions exercised ÷ partitions identified, including the invalid ones.
• MU's grade table (page 107 of its circular): 9 valid partitions (8 bands and Ab) plus 3 invalid; 12 tests found the missing B band (57.5 gave C) and the missing check above 100 (120 gave O).
• Each Choice coverage: every partition of every input at least once; the smallest set has as many tests as the input with most partitions.
• One invalid value per test, or one fault masks another: (20, 1.5, False) passed, (0, 1.5, False) failed.

Test yourself

1. What is equivalence partitioning? On what assumption does it rest? A black-box test design technique that divides the inputs and outputs of a test item into partitions of values the specification treats alike, and tests at least one representative value from each. It assumes that if one value of a partition reveals a defect, any other value of the same partition would too, so one test per partition is sufficient.

2. Distinguish valid from invalid partitions, with an example of each from MU's grade table. A valid partition holds values the program should process, such as percentages from 80.0 to below 90.0, which give A+; an invalid partition holds values it should reject, such as a percentage above 100 or an input that is not a number.

3. How is equivalence partition coverage measured? As the number of partitions exercised by at least one test divided by the total number of identified partitions, including invalid ones, expressed as a percentage.

4. Derive equivalence partitions and one test value for each for MU's grade table. Valid: 90.0 to 100 (95, O), 80.0 to below 90.0 (85, A+), 70.0 to below 80.0 (75, A), 60.0 to below 70.0 (65, B+), 55.0 to below 60.0 (57.5, B), 50.0 to below 55.0 (52.5, C), 40.0 to below 50.0 (45, P), 0 to below 40.0 (20, F) and absent (Ab). Invalid: below 0 (-5), above 100 (120), not a number (seventy). Twelve tests.

5. What is Each Choice coverage, and how many tests does it need for ExamReg's fee rule? Coverage in which every partition of every input is exercised by at least one test, without regard to combinations. The fee rule's inputs have 5, 3 and 2 partitions, so at least 5 tests are needed; with invalid partitions tested one at a time, 7.

6. Why should invalid partitions be tested one at a time? Because when a test holds two invalid values, the program may reject it for the first and never examine the second, so a defect in handling the second is masked. In the worked example, a test that was both 20 days late and had 1.5 backlog papers passed, while a test with only the 1.5 found the defect.