What a Test Case Is, and What Makes a Good One

Two fields are filled in only when the test runs: the actual result and the status. Everything else is written before, which is what makes the test repeatable. The postcondition is easily forgotten and quietly important: it says what state the test leaves behind, so the next test does not start from a surprise.

The expected result, and where it comes from

The part of a test case that decides pass or fail has a name. ISO/IEC TR 29119-11:2020 defines a test oracle as a "source of information for determining whether a test has passed or failed". For ExamReg's fee test the oracle is the fee table in the requirements. Oracles come from several places, and a tester should know which one each test relies on:

• The specification, as with the fee table.
• An independent calculation, as when Chapter Two, on errors, faults and failures, used Python's own calendar.isleap to judge a leap-year function.
• A previous version of the software, trusted for the features that did not change.
• A comparable product, such as a published calculator for the same rule.
• A person's judgement, for things like readability or layout, where no document can say what "right" is.

The same technical report names the difficulty that sits under all of this, the test oracle problem: the "challenge of determining whether a test has passed or failed for a given set of test inputs and state". When nobody can say what the right answer is, the test cannot be judged, however carefully it was run.

From test condition to test suite

A test case sits in a hierarchy, and the practical's documents use every level of it.

Test scenarios and test cases

In industry, and in the practical of this paper, a test scenario is usually written as one line saying what situation will be tested, and several test cases are then written saying exactly how, each with its own data and expected result. A scenario is quick to write and quick to review; it lets a business analyst check the testers have thought of the right situations before anyone writes a single step.

What makes a good test case

These are principles of practice, each followed because of what goes wrong without it.

1. It is traceable. It names the requirement or risk it checks. Without that link, a failure cannot be reported against a rule and coverage cannot be measured (Chapter Five, on the basic test process).
2. It has one objective. A case that checks the fee, the receipt and the email at once gives one "fail" for three possible causes, and the defect report cannot say which.
3. Its preconditions are explicit. Most "cannot reproduce" arguments between testers and developers are really two different starting states.
4. Its expected result is exact and decided in advance. "The fee is calculated correctly" is not an expected result; "Rs 100" is.
5. It is repeatable and independent. It gives the same result every time on the same build, and does not depend on another test having run first unless its preconditions say so.
6. It includes the unwelcome inputs. Invalid values, empty fields, the maximum, the moment just after a deadline. Most defects live in the cases a developer did not think of; a suite of only valid, typical inputs finds few of them.
7. It is not redundant. Each case should be able to find something the others cannot. Ten cases that all enter 3 days late add effort and no evidence. Module 2's techniques are, in effect, methods for choosing non-redundant cases.
8. Another tester could run it. Clear steps and data, no private knowledge.
9. It is prioritised. ISTQB v4.0.1 section 5.1.5 describes prioritising test cases, most commonly by risk, so that if time runs out the most important ones have already run.

Worked example: a table of test cases, executed

The six late-fee cases of scenario TS-01 are written below as data, one record per case, with the same fields as the table above. A short program then does what a tester does: runs each case, records the actual result, and sets the status. The function under test is the corrected version from Chapter One.

def late_fee(days_late):
    if days_late > 15:
        raise ValueError("form not accepted more than 15 days late")
    if days_late <= 0:
        return 0
    if days_late <= 7:
        return 100
    return 500

test_cases = [   # id, requirement, input (days late), expected result
    ("TC-FEE-01", "FEE-1", 0,  0),
    ("TC-FEE-02", "FEE-2", 1,  100),
    ("TC-FEE-03", "FEE-2", 7,  100),
    ("TC-FEE-04", "FEE-3", 8,  500),
    ("TC-FEE-05", "FEE-3", 15, 500),
    ("TC-FEE-06", "FEE-4", 16, "refused"),
]
print(f"{'ID':<10}{'req':<7}{'input':>6}  {'expected':<9}{'actual':<9}status")
for case_id, requirement, days, expected in test_cases:
    try:
        actual = late_fee(days)
    except ValueError:
        actual = "refused"
    status = "pass" if actual == expected else "FAIL"
    print(f"{case_id:<10}{requirement:<7}{days:>6}  {str(expected):<9}{str(actual):<9}{status}")

ID        req     input  expected actual   status
TC-FEE-01 FEE-1       0  0        0        pass
TC-FEE-02 FEE-2       1  100      100      pass
TC-FEE-03 FEE-2       7  100      100      pass
TC-FEE-04 FEE-3       8  500      500      pass
TC-FEE-05 FEE-3      15  500      500      pass
TC-FEE-06 FEE-4      16  refused  refused  pass

Notice what the records contain and what the program adds. Each record was written before the run: identifier, requirement, input and expected result. The program supplies only the actual result and the status, exactly the two fields left blank in the written test case. Notice also that the test for 16 days has an expected result that is not a number: a refusal is a result too, and a test case that forgot to say what should happen after 15 days would have been incomplete.

What it does not mean

A test scenario is not a test case. A scenario says what situation to test; it cannot be executed or judged until test cases with data and expected results are derived from it.

A test case is not a test script or a procedure. The case says what to check; the procedure says in what order and with what setup to run cases; the script is the document or program that carries the procedure.

More test cases do not mean better testing. Ten redundant cases are worth less than three well-chosen ones; what counts is what the cases can find.

Expected results do not come from running the software. An expected result copied from the program's own output tests nothing: it certifies whatever the program already does, including its defects.

Quick revision

• Test case: "set of preconditions, inputs and expected results, developed to drive the execution of a test item to meet test objectives" (ISO/IEC/IEEE 29119-2:2021).
• Fields: ID, title, requirement traced, priority, preconditions, test data, steps, expected result, actual result, status, postconditions.
• Test oracle: the source that decides pass or fail: specification, independent calculation, previous version, comparable product, human judgement. The oracle problem is not knowing the right answer.
• Hierarchy: test condition, scenario, case, procedure, script, suite.
• Scenario says what situation; case says exactly how, with data and expected result; several cases per scenario.
• A good test case: traceable, one objective, explicit preconditions, exact expected result in advance, repeatable, includes invalid inputs, non-redundant, runnable by others, prioritised.

Test yourself

1. Define a test case and name its essential parts. A set of preconditions, inputs and expected results developed to drive the execution of a test item to meet test objectives. In practice it also carries an identifier, the requirement it traces to, steps, a priority, postconditions, and, after execution, the actual result and a status.

2. Differentiate a test scenario from a test case, with an example. A scenario is a one-line situation to be tested, such as a student submitting the form late and paying the late fee; it cannot be executed as it stands. A test case derived from it is executable and exact, such as a form submitted 7 days late with an expected late fee of Rs 100.

3. What is a test oracle? Give three kinds. The source of information that decides whether a test passed or failed. Examples: the specification, an independent calculation such as a trusted library function, a previous version of the software, a comparable product, or a person's judgement.

4. Why must the expected result be decided before the test is run? Because the test is a comparison, and a result decided after seeing the output simply accepts whatever the program did, defects included.

5. State any five principles of good test case design. Any five of: traceable to a requirement; one objective per case; explicit preconditions; exact expected result decided in advance; repeatable and independent; includes invalid and unexpected inputs; not redundant; clear enough for another tester; prioritised by risk.

6. In what order are these produced during testing: test case, test suite, test condition, test scenario, test procedure? Test condition first (in test analysis), then the test scenario and the test cases derived from it (in test design), then the test procedure that puts cases in execution order with their setup, and the test suite that groups them (both in test implementation).