State Transition Testing

Five states are enough to describe it: ready, 1 failure, 2 failures, locked and logged in. There are four events: a correct password, a wrong one, 30 minutes passing, and log out.

Figure 57.1 ExamReg's login rule: the ten valid transitions, each labelled event / action

The diagram shows the valid transitions only, labelled in the syllabus's syntax, event then action: wrong / lock from 2 failures to locked. Two of them return to the state they left (correct and wrong while locked); a transition may do that, and the definition allows for it ("or the same state").

Guard conditions are the other half of the syntax. The same rule could be drawn with a single state, logged out, and a count of failures, where a wrong password is two transitions with guards: wrong [failures < 2] / add a failure, which stays logged out, and wrong [failures = 2] / lock, which goes to locked. Both models describe the same behaviour. The five-state version makes every step visible, which is why it is used here; the guarded version stays small when the limit is ten attempts instead of three.

The state table, and the transitions that must be refused

The diagram cannot show what happens when an event arrives in a state that has no arrow for it. The state table can. Its "rows represent states, and its columns represent events", its cells hold the target state, and, in the syllabus's words, "In contrast to the state diagram, the state table explicitly shows invalid transitions, which are represented by empty cells."

The table is written in a file the programs share:

# ExamReg's login rule as a state table: (state, event) -> (next state, action).
# A (state, event) pair with no entry is an INVALID transition: the event is refused
# and the state does not change.
STATES = ["ready", "1 failure", "2 failures", "locked", "logged in"]
EVENTS = ["correct", "wrong", "30 minutes", "log out"]
TABLE = {
    ("ready", "correct"):      ("logged in", "welcome"),
    ("ready", "wrong"):        ("1 failure", "try again"),
    ("1 failure", "correct"):  ("logged in", "welcome"),
    ("1 failure", "wrong"):    ("2 failures", "try again"),
    ("2 failures", "correct"): ("logged in", "welcome"),
    ("2 failures", "wrong"):   ("locked", "locked for 30 minutes"),
    ("locked", "correct"):     ("locked", "account locked"),
    ("locked", "wrong"):       ("locked", "account locked"),
    ("locked", "30 minutes"):  ("ready", "unlocked"),
    ("logged in", "log out"):  ("ready", "goodbye"),
}

def expected_states(events, start="ready"):
    """Walk the table: the state after each event, and the (state, event) pairs used."""
    state, states, pairs = start, [], []
    for e in events:
        pairs.append((state, e))
        state = TABLE.get((state, e), (state, None))[0]
        states.append(state)
    return states, pairs

from login_rule import STATES, EVENTS, TABLE

print((f"{'state':<12}" + "".join(f"{e:<13}" for e in EVENTS)).rstrip())
for s in STATES:
    row = f"{s:<12}" + "".join(f"{TABLE.get((s, e), ('',))[0]:<13}" for e in EVENTS)
    print(row.rstrip())
cells = len(STATES) * len(EVENTS)
print(f"{cells} cells: {len(TABLE)} valid transitions, {cells - len(TABLE)} invalid (blank)")

state       correct      wrong        30 minutes   log out
ready       logged in    1 failure
1 failure   logged in    2 failures
2 failures  logged in    locked
locked      locked       locked       ready
logged in                                          ready
20 cells: 10 valid transitions, 10 invalid (blank)

Five states and four events make twenty cells: ten valid transitions and ten blanks. Each blank is a question the specification answers with refuse, and change nothing: logging out when nobody is logged in, entering a password when already logged in, the 30 minutes passing when the account is not locked. A blank cell is not a cell nobody thought about. Filling in the table is how a tester makes sure somebody did, and a blank the specification does not account for is a finding in itself.

Three levels of coverage

The ISTQB syllabus discusses three criteria.

• All states coverage: "the coverage items are the states", measured as "the number of exercised states divided by the total number of states".
• Valid transitions coverage, "also called 0-switch coverage": "the coverage items are single valid transitions", and every valid transition must be exercised.
• All transitions coverage: "the coverage items are all the transitions shown in a state table", so the tests "must exercise all the valid transitions and attempt to execute invalid transitions". And, as with invalid partitions: "Testing only one invalid transition in a single test case helps to avoid defect masking".

The syllabus ranks them: "All states coverage is weaker than valid transitions coverage, because it can typically be achieved without exercising all the transitions. Valid transitions coverage is the most widely used coverage criterion. Achieving full valid transitions coverage guarantees full all states coverage. Achieving full all transitions coverage guarantees both full all states coverage and full valid transitions coverage and should be a minimum requirement for mission and safety-critical software."

Worked example: the login rule, tested three ways

The version under test keeps a count of failures and two flags. It has two defects. The program runs three test sets against it, one designed for each criterion, walks the state table to work out the expected state after every event, and stops a test at the first state that differs.

• All states: one test, wrong, wrong, wrong, 30 minutes, correct, which visits all five states.
• Valid transitions: four tests that between them take every one of the ten arrows.
• All transitions: those four, plus one test for each of the ten blank cells: a short sequence that reaches the state, then the one invalid event.

from login_rule import STATES, EVENTS, TABLE, expected_states

class Login:                                            # the version under test
    def __init__(self):
        self.failures, self.locked, self.logged_in = 0, False, False

    def state(self):
        if self.logged_in:
            return "logged in"
        if self.locked:
            return "locked"
        return ["ready", "1 failure", "2 failures"][self.failures]

    def event(self, e):
        if e == "correct" and not self.logged_in:
            self.logged_in, self.failures, self.locked = True, 0, False
        elif e == "wrong" and not self.logged_in and not self.locked:
            self.failures += 1
            if self.failures == 3:
                self.locked, self.failures = True, 0
        elif e == "30 minutes" and self.locked:
            self.locked = False
        elif e == "log out":
            self.logged_in, self.locked = False, False
        # anything else is refused, and nothing changes

def run(name, tests):
    states, valid, invalid, failures = set(["ready"]), set(), set(), []
    for events in tests:
        want, pairs = expected_states(events)
        states.update(want)
        valid.update(p for p in pairs if p in TABLE)
        invalid.update(p for p in pairs if p not in TABLE)
        login = Login()
        for i, (e, w) in enumerate(zip(events, want)):
            login.event(e)
            if login.state() != w:
                failures.append(f"{events}: after '{e}' in state '{pairs[i][0]}',"
                                f" expected '{w}', got '{login.state()}'")
                break
    print(f"{name}, {len(tests)} test(s): states {len(states)} of {len(STATES)},"
          f" valid transitions {len(valid)} of {len(TABLE)},"
          f" invalid {len(invalid)} of {len(STATES) * len(EVENTS) - len(TABLE)}; {len(failures)} failed")
    for f in failures:
        print("   FAIL", f)

all_states = [["wrong", "wrong", "wrong", "30 minutes", "correct"]]
valid_transitions = [["correct", "log out"], ["wrong", "correct"], ["wrong", "wrong", "correct"],
                     ["wrong", "wrong", "wrong", "wrong", "correct", "30 minutes"]]
reach = {"ready": [], "1 failure": ["wrong"], "2 failures": ["wrong", "wrong"],
         "locked": ["wrong", "wrong", "wrong"], "logged in": ["correct"]}
invalid_tests = [reach[s] + [e] for s in STATES for e in EVENTS if (s, e) not in TABLE]

run("all states", all_states)
run("valid transitions", valid_transitions)
run("all transitions", valid_transitions + invalid_tests)

all states, 1 test(s): states 5 of 5, valid transitions 5 of 10, invalid 0 of 10; 0 failed
valid transitions, 4 test(s): states 5 of 5, valid transitions 10 of 10, invalid 0 of 10; 1 failed
   FAIL ['wrong', 'wrong', 'wrong', 'wrong', 'correct', '30 minutes']: after 'correct' in state 'locked', expected 'locked', got 'logged in'
all transitions, 14 test(s): states 5 of 5, valid transitions 10 of 10, invalid 10 of 10; 2 failed
   FAIL ['wrong', 'wrong', 'wrong', 'wrong', 'correct', '30 minutes']: after 'correct' in state 'locked', expected 'locked', got 'logged in'
   FAIL ['wrong', 'wrong', 'wrong', 'log out']: after 'log out' in state 'locked', expected 'locked', got 'ready'

All states coverage: 5 of 5 states, and nothing found. One test visited every state and passed. It used only 5 of the 10 valid transitions, and the ones it skipped are where both defects are. This is the syllabus's point that all states coverage "can typically be achieved without exercising all the transitions".

Valid transitions coverage: 10 of 10, and the first defect. The fourth test locks the account, tries a wrong password (refused, correctly), and then the right one, and the student is logged in. The code checks the lock when a password is wrong and forgets it when the password is right. A lock exists to stop someone guessing; with this defect the guesser simply carries on through the 30 minutes, every wrong guess refused without delay and the first right one let in, so the lock slows nobody down. Only a test that takes the arrow correct out of locked could find it.

All transitions coverage: 10 of 10 valid and 10 of 10 invalid, and the second defect. Logging out is not possible while the account is locked, since nobody is logged in, and the specification says the event must be refused. The code's log-out branch clears the lock as well as the login, so a request to log out, which any browser can send, unlocks the account at once. No valid transition goes near it; only the attempt at an invalid one found it. The same test set that found it tried each invalid transition in its own test, so neither defect could hide the other.

The pattern is the syllabus's ranking made concrete: each stronger criterion found everything the weaker one did, and more. For a login lock, a security control, the syllabus's advice that all transitions coverage "should be a minimum requirement for mission and safety-critical software" is the right standard.

The procedure

1. Identify the states from the specification: the distinct situations in which the system behaves differently. Name each for what it means (locked, not state 4).
2. Identify the events, and the actions and guard conditions that go with them.
3. Draw the state diagram of the valid transitions, then build the state table of every state and event, and decide each blank: refused, or a missing requirement.
4. Choose a coverage criterion by risk: all states at the least, valid transitions as the usual choice, all transitions for anything safety- or security-critical.
5. Write test cases as event sequences from the start state, each with the expected state (and action) after every event, putting at most one invalid event in each test.
6. Run, check the state after every event, and measure coverage over states, valid transitions and invalid transitions.

What it does not mean

A state is not a screen. 1 failure and 2 failures show the same login page; they are different states because the next wrong password does different things.

A test case is not one event. It is a sequence from a known start, and the state must be checked after each event, not only at the end: a defect in the middle of a sequence can be undone by the events after it.

The invalid transitions are not optional. They are half of the table here, and the more dangerous of the two defects was behind one of them.

Full valid transitions coverage does not test every path. It takes each arrow at least once; longer combinations of arrows are a stronger criterion still.

Quick revision

• State (ISO/IEC/IEEE 29148), transition "to another state or the same state" (ISO/IEC 11411), state diagram (ISO/IEC/IEEE 24765); state transition testing exercises "transitions in a state model" (ISO/IEC/IEEE 29119-4).
• Label syntax: "event [guard condition] / action" (ISTQB). A state table shows the invalid transitions as empty cells.
• A test case is a sequence of events from a start state, checked after every event.
• Coverage: all states is weaker than valid transitions (0-switch, the most used), which is weaker than all transitions (valid ones exercised, invalid ones attempted, one per test).
• ExamReg's login: 5 states, 4 events, 20 cells, 10 valid, 10 invalid.
• Worked example: all states (1 test) found nothing; valid transitions (4 tests) found the correct password accepted while locked; all transitions (14 tests) also found that logging out while locked unlocks the account.

Test yourself

1. What is state transition testing, and when is it used? A black-box technique that models the system as states, events and transitions, and derives tests as sequences of events that exercise the model's states and transitions. It is used when the system's response to an input depends on its history, such as logins, orders, workflows and devices with modes.

2. Draw the state diagram and state table for ExamReg's login lockout. States: ready, 1 failure, 2 failures, locked, logged in. Transitions: a wrong password moves ready to 1 failure, 1 failure to 2 failures, and 2 failures to locked; a correct password moves ready, 1 failure or 2 failures to logged in; while locked, correct and wrong both stay locked and are refused; 30 minutes moves locked to ready; log out moves logged in to ready. In the table the other ten cells are blank: invalid transitions, refused with no change.

3. Distinguish the three coverage criteria of state transition testing. All states coverage requires every state to be visited; valid transitions (0-switch) coverage requires every valid transition to be exercised; all transitions coverage requires every valid transition to be exercised and every invalid transition to be attempted. Each is stronger than the one before, and full valid transitions coverage guarantees full all states coverage.

4. Why should each invalid transition be tested in a separate test case? Because one defect can mask another: once a test has gone wrong at one invalid event, the state it is in is no longer the one expected, and the next invalid event is being tried in the wrong state, so its result means nothing. One invalid transition per test keeps each result interpretable.

5. What is a guard condition? Give an example. A condition that must be true for an event to cause a particular transition, written in square brackets after the event. For example, with a count of failures, wrong [failures = 2] / lock moves the account to locked, while wrong [failures < 2] / add a failure leaves it logged out.

6. In the worked example, why did all states coverage miss both defects? Because it can be reached with a single test that visits every state without taking every transition: the test never tried a correct password while locked, and never tried to log out while locked, which are the two places where the code was wrong.