Boundary Value Analysis

Finding the boundary values, and the question of precision

For whole numbers, a boundary value's neighbour is one step away: the neighbour of 7 days is 8 days. For a continuous quantity, one step is the precision the values are recorded in, and the specification has to say what it is.

MU's table prints percentages to one decimal place (80.0 to < 90.0) and grade point averages to two (8.00 to < 9.00). If the percentages really are recorded to one decimal place, the value just below 80.0 is 79.9. If the portal computed them to two, it would be 79.99, and a test at 79.9 would leave 79.91 to 79.99 untested. A tester who cannot find the precision in the specification asks, and writes the answer down. This chapter takes the table at its word: one decimal place, a step of 0.1.

With the step settled, each partition of MU's table has a lowest and a highest value: F is 0.0 to 39.9, P is 40.0 to 49.9, and so on up to O at 90.0 to 100.0. Between every pair of adjacent partitions there are two boundary values, the highest of one and the lowest of the next, and those pairs are exactly the 2-value coverage items. At the two ends of the valid range the invalid partitions supply the neighbours: -0.1 below 0.0, and 100.1 above 100.0.

Worked example 1: the grade table's third defect

The same grade function that Chapter Fifty-Four's equivalence partitioning tests ran against, unchanged:

def grade(percent):
    """Letter grade and grade point for a percentage of marks, or for "Ab" (absent)."""
    if percent == "Ab":
        return ("Ab", 0)
    if not isinstance(percent, (int, float)) or percent < 0:
        raise ValueError("not a percentage of marks")
    if percent >= 90.0:
        return ("O", 10)
    if percent > 80.0:
        return ("A+", 9)
    if percent >= 70.0:
        return ("A", 8)
    if percent >= 60.0:
        return ("B+", 7)
    if percent >= 50.0:
        return ("C", 5)
    if percent >= 40.0:
        return ("P", 4)
    return ("F", 0)

The program writes MU's table as ordered partitions, derives the boundary values from them instead of typing them, and judges each result against the table.

from mu_grade import grade

# MU's ordered partitions at the table's precision of 0.1: (lowest, highest, expected result)
partitions = [(None, -0.1, "refused"),                     # invalid: below 0
              (0.0, 39.9, "F (0)"), (40.0, 49.9, "P (4)"), (50.0, 54.9, "C (5)"),
              (55.0, 59.9, "B (6)"), (60.0, 69.9, "B+ (7)"), (70.0, 79.9, "A (8)"),
              (80.0, 89.9, "A+ (9)"), (90.0, 100.0, "O (10)"),
              (100.1, None, "refused")]                    # invalid: above 100

def expected(value):
    for low, high, result in partitions:
        if (low is None or value >= low) and (high is None or value <= high):
            return result

def run(value):
    try:
        letter, point = grade(value)
        return f"{letter} ({point})"
    except ValueError:
        return "refused"

# 2-value BVA: every partition's minimum and maximum (open ends have none)
values = sorted({v for low, high, _ in partitions for v in (low, high) if v is not None})
print(f"2-value boundary values ({len(values)}):", " ".join(f"{v:.1f}" for v in values))
failed = [v for v in values if run(v) != expected(v)]
for v in failed:
    print(f"   {v:.1f}: expected {expected(v)}, got {run(v)}")
print(f"{len(failed)} of {len(values)} failed")

# 3-value BVA: each boundary value and both its neighbours, one step of 0.1 away
three = sorted({round(v + step, 1) for v in values for step in (-0.1, 0, 0.1)})
failed3 = [v for v in three if run(v) != expected(v)]
print(f"3-value: {len(three)} values, {len(failed3)} failed:", " ".join(f"{v:.1f}" for v in failed3))

2-value boundary values (18): -0.1 0.0 39.9 40.0 49.9 50.0 54.9 55.0 59.9 60.0 69.9 70.0 79.9 80.0 89.9 90.0 100.0 100.1
   55.0: expected B (6), got C (5)
   59.9: expected B (6), got C (5)
   80.0: expected A+ (9), got A (8)
   100.1: expected refused, got O (10)
4 of 18 failed
3-value: 36 values, 7 failed: 55.0 55.1 59.8 59.9 80.0 100.1 100.2

Eighteen boundary values, and four failures that between them show all three of the function's defects.

• 80.0 gets A instead of A+. This is the third defect. The code says percent > 80.0 where MU's table says 80.0 and above, so the boundary sits one step too high and a student with exactly 80.0 per cent is given a grade point of 8 instead of 9. Chapter Fifty-Four's equivalence partitioning test at 85 could not see it; the test at 80.0 is the only kind that can. It is the ISTQB syllabus's first typical defect, a boundary "misplaced to positions above or below their intended positions".
• 55.0 and 59.9 get C instead of B. The missing B band shows itself again, now at both of its edges.
• 100.1 gets O instead of being refused. The upper limit of the valid range is not in the code at all, the syllabus's other typical defect, a boundary "omitted altogether".

The 3-value run tests 36 values and fails 7, but the three extra failures (55.1, 59.8 and 100.2) sit inside the same faulty partitions and reveal no new defect. Here 3-value BVA doubled the tests and found nothing that 2-value had missed. The next example shows when the third value earns its place.

Worked example 2: two values or three

ExamReg's late fee has integer days, so the step is 1. Its ordered partitions are: negative (invalid), exactly 0, 1 to 7, 8 to 15, and more than 15 (invalid). The version under test is the one Chapter Fifty-Three's specification-based tests ran against, which charges Rs 500 at exactly 7 days. The program derives the 2-value and 3-value test sets from the partitions and runs both, then runs one value from each partition, as equivalence partitioning would, for comparison.

It then reproduces the ISTQB syllabus's own example of the difference between the two versions. A decision meant to accept every x up to and including 10 has been written as a test for x equal to 10, and the syllabus observes that "no test data derived from the 2-value BVA (x = 10, x = 11) can detect the defect. However, x = 9, derived from the 3-value BVA, is likely to detect it."

def late_fee(days_late):                     # the version under test, from Chapter 53
    if days_late < 0 or days_late > 15:
        raise ValueError("form not accepted")
    if days_late == 0:
        return 0
    elif days_late < 7:
        return 100
    return 500

def specified(days_late):                    # the oracle, from ExamReg's fee rule
    if not 0 <= days_late <= 15:
        return "refused"
    return 0 if days_late == 0 else 100 if days_late <= 7 else 500

def run(days_late):
    try:
        return late_fee(days_late)
    except ValueError:
        return "refused"

# ordered partitions of days late (whole days): (lowest, highest); None is an open end
partitions = [(None, -1), (0, 0), (1, 7), (8, 15), (16, None)]
two = sorted({v for p in partitions for v in p if v is not None})
three = sorted({v + d for v in two for d in (-1, 0, 1)})

def report(name, values, got, want):
    bad = [f"{v} (expected {want(v)}, got {got(v)})" for v in values if got(v) != want(v)]
    print(f"{name} {values}: {len(bad)} failed" + (": " + "; ".join(bad) if bad else ""))

report("late fee, 2-value BVA", two, run, specified)
report("late fee, 3-value BVA", three, run, specified)
report("late fee, one value per partition", [-5, 0, 4, 10, 20], run, specified)

def accepted(x):                             # specified: accept x <= 10
    return x == 10                           # ISTQB's example of a defect: = where <= was meant

report("ISTQB's example, 2-value", [10, 11], accepted, lambda x: x <= 10)
report("ISTQB's example, 3-value", [9, 10, 11], accepted, lambda x: x <= 10)

late fee, 2-value BVA [-1, 0, 1, 7, 8, 15, 16]: 1 failed: 7 (expected 100, got 500)
late fee, 3-value BVA [-2, -1, 0, 1, 2, 6, 7, 8, 9, 14, 15, 16, 17]: 1 failed: 7 (expected 100, got 500)
late fee, one value per partition [-5, 0, 4, 10, 20]: 0 failed
ISTQB's example, 2-value [10, 11]: 0 failed
ISTQB's example, 3-value [9, 10, 11]: 1 failed: 9 (expected True, got False)

Read the first three lines together. The 2-value set, seven tests, puts a test on 7 and on 8 by rule, and the test at 7 fails: the defect that Chapter Fifty-Three's two hundred random tests, judged by the specification, found only because six of them happened to land on 7, found here the first time, with a test designed to find it. One value from the middle of each partition (4 for the Rs 100 band, 10 for the Rs 500 band) passes everything. The 3-value set, thirteen tests, finds the same single defect and nothing more.

The last two lines are the syllabus's example, run. A comparison written as x == 10 agrees with x <= 10 at 10 (both true) and at 11 (both false), so the two 2-value tests pass. Only a value on the valid side away from the boundary, 9, exposes it, and only 3-value BVA includes 9. The difference is not in where the boundary is but in what the comparison does next to it. "3-value BVA is more rigorous than 2-value BVA as it may detect defects overlooked by 2-value BVA", at the price of more tests.

Why did the late fee not need the third value? Its partition 1 to 7 has two boundaries, and 2-value BVA tests both ends, 1 and 7. A defect of the == 7 kind there would fail at 1. The partition of values up to 10 in the syllabus's example has no lower end to test, which is exactly where the third value matters.

Coverage

The ISTQB syllabus measures each version against its own coverage items. For 2-value BVA, "Coverage is measured as the number of boundary values that were exercised, divided by the total number of identified boundary values"; for 3-value BVA, "the number of boundary values and their neighbors exercised, divided by the total number of identified boundary values and their neighbors". In the worked examples, full 2-value coverage took 18 tests for the grade table and 7 for the late fee; full 3-value coverage took 36 and 13.

The procedure

1. Start from the ordered partitions equivalence partitioning found (Chapter Fifty-Four, on equivalence partitioning).
2. Settle the step: 1 for whole numbers; for anything continuous, the precision the specification states, or the answer to the question the tester asks.
3. List the boundary values: the lowest and highest value of every partition, including the edges of the invalid partitions that bound the valid range.
4. For 3-value BVA, add each boundary value's neighbours one step either side.
5. Write each expected result from the specification, run, and measure coverage.

Boundaries that are not written as numbers

The 24765 definition speaks of input, internal and output values, and boundaries hide in places a table does not show.

• Output boundaries. The largest fee ExamReg can ever charge is set by the largest number of backlog papers a student may have. The fee rule does not give one, so BVA cannot even list the boundary. Asking for it is itself a finding: a limit the specification never stated.
• Time boundaries. On or before the last date has an edge in time: a form submitted at the last minute of the last date must be charged nothing, and one a minute later must be charged the Rs 100 fee. The ISTQB syllabus includes "time-related values" among the things partitions, and so boundaries, can be found for.
• Size boundaries. A name field that holds 60 characters has boundaries at 60 and 61 characters, and at 0.

Strengths and limits

Strengths. BVA aims at the place where defects in comparisons actually show, and it finds them with a test designed to find them rather than by luck. It adds only a few tests to equivalence partitioning, and its coverage is measurable. And listing boundaries forces questions a specification often leaves open: the precision, the largest value, the last minute.

Limits. It needs ordered partitions. It inherits every mistake in the partitions: a partition nobody identified has no boundaries to test. It tests one input at a time, and a defect that needs a boundary value of one input and a particular value of another is a combination, which is the subject of the next chapter.

What it does not mean

BVA does not replace equivalence partitioning. It builds on the same partitions and tests their edges; the middles are still equivalence partitioning's.

A boundary is not always printed as a number. On or before the last date is a boundary in time, and a field's length is a boundary in size.

3-value BVA is not always better value. It may find more; in both worked examples on ExamReg and MU's table it found nothing that 2-value had missed, at up to twice the cost.

Full BVA coverage does not prove the partitions were right. It proves only that their edges, as identified, were tried.

Quick revision

• BVA (ISO/IEC/IEEE 29119-1): exercising the boundaries of equivalence partitions; only ordered partitions; boundary values are each partition's minimum and maximum (ISTQB).
• Why: "developers are more likely to make errors with these boundary values"; defects where boundaries are "misplaced" or "omitted altogether".
• 2-value: the boundary value and its closest neighbour in the adjacent partition. 3-value: the boundary value and both neighbours. Coverage = items exercised ÷ items identified.
• Precision: the step between a boundary and its neighbour must be known; MU's table prints 0.1 for percentages.
• MU's grade table: 18 boundary values found all three defects, including 80.0 graded A instead of A+, which the middles had missed.
• Late fee: 2-value tests 7 and 8 and finds the defect at 7 that one value per partition misses; 3-value (13 tests) finds nothing more there.
• ISTQB's example: x <= 10 written as x == 10 passes 10 and 11 and fails only at 9, a 3-value test.

Test yourself

1. What is boundary value analysis, and why is it used? A black-box technique that tests the boundary values of ordered equivalence partitions, each partition's minimum and maximum and their neighbours. It is used because programmers most often go wrong at the edges, writing the wrong comparison or putting a limit one step out, and such defects give right answers everywhere except at or next to the boundary.

2. Distinguish 2-value and 3-value BVA. In 2-value BVA each boundary value gives two coverage items, the value and its closest neighbour in the adjacent partition; in 3-value BVA it gives three, the value and both its neighbours. 3-value BVA needs more tests and can find defects 2-value misses, such as a comparison that is right at the boundary but wrong next to it.

3. Derive the 2-value boundary values for MU's grade table, taking percentages to one decimal place. -0.1 and 0.0; 39.9 and 40.0; 49.9 and 50.0; 54.9 and 55.0; 59.9 and 60.0; 69.9 and 70.0; 79.9 and 80.0; 89.9 and 90.0; 100.0 and 100.1. Eighteen values.

4. Derive the 2-value and 3-value test values for ExamReg's late fee (whole days; refused if negative or over 15; Rs 0 at 0 days, Rs 100 for 1 to 7, Rs 500 for 8 to 15). 2-value: -1, 0, 1, 7, 8, 15, 16. 3-value: -2, -1, 0, 1, 2, 6, 7, 8, 9, 14, 15, 16, 17.

5. Why can BVA not be applied to every input? Because it needs an order: a boundary is the smallest or largest value of a partition, with neighbours on either side. Inputs whose partitions are unordered, such as a yes or no concession or the Ab mark for an absent student, have no boundaries and are tested by equivalence partitioning alone.

6. A comparison x <= 10 has been written as x == 10. Which test values find the defect, and why does 2-value BVA miss it? 2-value BVA tests 10 and 11; the faulty comparison is true at 10 and false at 11, just as the correct one is, so both tests pass. 3-value BVA adds 9, where the correct comparison is true and the faulty one false, so the test at 9 fails.