Error Guessing

The list maps onto the syllabus's six kinds: most rows are input, the wrong units are interfaces, and the date row is computation. It will be wrong for some other application and incomplete for this one, which is exactly why a list is kept and extended rather than written once.

Worked example: a fault attack on the calendar

ExamReg counts the days a form is late: calendar days after the last date, and 0 for any form on or before it. Here is the developer's version, and a list of attacks aimed at the ways date arithmetic is known to go wrong. The expected results come from Python's own datetime.date, which knows the calendar and is independent of the code under test.

from datetime import date

def days_late(last_date, submitted):              # the version under test
    """Calendar days after the last date; 0 for a form on or before it."""
    days = (submitted.month - last_date.month) * 30 + (submitted.day - last_date.day)
    return max(days, 0)

def oracle(last_date, submitted):                 # the calendar, from Python's own date type
    return max((submitted - last_date).days, 0)

# the attack list: where date arithmetic is known to go wrong
attacks = [("on the last date itself",           date(2026, 10, 15), date(2026, 10, 15)),
           ("the day after",                     date(2026, 10, 15), date(2026, 10, 16)),
           ("before the last date",              date(2026, 10, 15), date(2026, 10, 1)),
           ("across the end of a 31-day month",  date(2026, 10, 31), date(2026, 11, 1)),
           ("across February, leap year",        date(2028, 2, 28),  date(2028, 3, 1)),
           ("across February, ordinary year",    date(2027, 2, 28),  date(2027, 3, 1)),
           ("across the end of the year",        date(2026, 12, 28), date(2027, 1, 2)),
           ("a year late",                       date(2026, 10, 15), date(2027, 10, 15))]

failed = 0
for name, last, sent in attacks:
    want, got = oracle(last, sent), days_late(last, sent)
    failed += want != got
    print(f"{name:<34} expected {want:>3}, got {got:>3}  {'pass' if want == got else 'FAIL'}")
print(f"{len(attacks)} attacks, {failed} failed")

on the last date itself            expected   0, got   0  pass
the day after                      expected   1, got   1  pass
before the last date               expected   0, got   0  pass
across the end of a 31-day month   expected   1, got   0  FAIL
across February, leap year         expected   2, got   3  FAIL
across February, ordinary year     expected   1, got   3  FAIL
across the end of the year         expected   5, got   0  FAIL
a year late                        expected 365, got   0  FAIL
8 attacks, 5 failed

The first three attacks are the ordinary cases, and they pass: a test designed from the fee rule alone, using dates in the middle of one month, would pass too. The other five fail, and every one of them is a failure date code is famous for, because the developer treated every month as 30 days long and ignored the year.

• The end of a 31-day month. A form submitted on 1 November for a last date of 31 October is 1 day late; the code says 0. Every 31st of a month is a free day.
• February, in both kinds of year. Across the end of February the code counts 3 days where there are 2 in a leap year and 1 in an ordinary one, so a student who was 1 day late is charged as if 3 days late.
• The end of the year. A last date of 28 December and a form on 2 January is 5 days late. The month difference is negative (January minus December), the total is negative, and the code returns 0: the form counts as on time.
• A year late. The same thing, at its worst: a form a whole year late counts as on time, and is accepted.

None of these came from a specification. The fee rule says calendar days after the last date and nothing about months or years. They came from knowing how programmers get dates wrong, and each took one line to try.

Building and keeping the list

The ISTQB syllabus says where lists come from: they "can be built based on experience, defect and failure data, or from common knowledge about why software fails". Three habits keep one useful.

• Feed it from the defect records. Every defect that escaped the formal tests is a candidate entry, described as a kind: not the name D'Souza was refused but names with punctuation. ExamReg's release 2.0 defect records, which later chapters analyse, put input validation first among the defect types, 58 of 200, and a list should weight its attacks the same way.
• Keep the entries specific enough to act on. Check the dates is a reminder; a last date of 31 October and a form on 1 November is a test.
• Share it. A list written down turns one tester's experience into the whole team's, which is also the idea behind the checklists of Chapter Sixty-Four, on checklist-based testing.

Strengths and limits

Strengths. It is fast and cheap: each attack is a single test with an obvious target. It finds real defects early, often the ones users meet first, because the list is made of failures that really happen. And it needs no detailed specification.

Limits. It is only as good as the list and the tester's knowledge: a failure nobody has met or imagined is on no list. It gives no coverage measure beyond the list itself. And it is not systematic about the specification: it complements the formal techniques, which test every partition and boundary the specification defines, and does not replace them.

What it does not mean

Error guessing is not random testing. Random testing (Chapter Fifty-Three, on specification-based testing) chooses inputs by chance; error guessing chooses them by knowledge, aimed at a named kind of failure.

The guess is not the test. An attack names a kind of failure; the test still needs exact inputs and an expected result, here from the calendar.

A list is not finished. New kinds of failure are added as they are found, and entries that stop finding anything can be retired.

It is not only for input fields. Output formats, computations, interfaces between components and stored data all have their typical failures.

Quick revision

• Error guessing (ISO/IEC/IEEE 29119-1): tests from "the tester's knowledge of past failures, or general knowledge of failure modes".
• Knowledge (ISTQB): how the application worked in the past; the errors developers tend to make; failures in similar applications.
• Six kinds (ISTQB): input, output, logic, computation, interfaces, data.
• Fault attacks: a list of possible errors, defects and failures, and tests designed to expose each.
• Starter list: empty, zero, negative, not whole, edges, special characters, duplicates, wrong units, dates.
• Worked example: 8 date attacks on days_late, 5 failed: a 31-day month end, February in both kinds of year, the year end (5 days late counted as 0) and a year late (counted as 0).
• Lists come from experience, defect and failure data, and common knowledge; keep them specific, shared and up to date.

Test yourself

1. What is error guessing? On what does the tester's guess depend? A test design technique in which tests are derived from the tester's knowledge of past failures and of how software typically fails. The knowledge comes from how the application has worked in the past, the kinds of errors its developers tend to make, and the failures of similar applications.

2. What is a fault attack? A systematic way of doing error guessing: the tester creates or acquires a list of possible errors, defects and failures, and designs tests that will expose each, cause the failure or identify the defect behind it.

3. List six categories of errors that an error guessing list can be organised by, with an example of each. Input, such as correct input not accepted; output, such as a wrong format; logic, such as a missing case; computation, such as a wrong operand; interfaces, such as a parameter mismatch; data, such as incorrect initialisation.

4. Write an attack list for a function that counts days between two dates. The same date; the next day; a date before the first; across the end of a 30-day and a 31-day month; across February in a leap year and in an ordinary year; across the end of a year; a whole year apart; and, where the dates can be far in the future, a century year that is not a leap year.

5. In the worked example, why did the ordinary tests pass while the attacks failed? Because the ordinary tests used dates within one month, where treating every month as 30 days and ignoring the year makes no difference. The defect shows only when the dates cross the end of a month of another length, the end of February, or the end of a year, and those are exactly the cases the attack list aims at.

6. What are the strengths and weaknesses of error guessing? It is quick and cheap, needs no detailed specification, and finds the defects users are likely to meet. It depends entirely on the tester's knowledge and the list, gives no coverage measure beyond the list, and is not systematic, so it complements the formal techniques rather than replacing them.