System Testing: The Whole System, End to End

The third and fourth items are the defects no other level can reach: each part may do its job and the flow between them may still be wrong.

End-to-end testing

System testing's characteristic test is the end-to-end scenario: one complete task, from the user's first action to the system's last effect, run through every part it touches. A scenario is a "step-by-step description of a series of events that occur concurrently or sequentially" (ISO/IEC/IEEE 24765). For ExamReg the central scenario is a registration: log in, pass the eligibility check, choose papers, see the fee, pay, receive the hall ticket. The scenario succeeds only if every step works and every hand-over between steps carries the right thing.

Good end-to-end scenarios are chosen, not just the happy path. For each use case, a system test set covers:

• the main flow that most users follow;
• the alternative flows (a concession, backlog papers, a late form);
• the exception flows (an ineligible student, a form refused as too late, a payment declined);
• and, for the next chapter, the non-functional conditions under which each must still work.

Worked example: five students, end to end

The program is ExamReg reduced to its five steps and joined together as one system, with the payment provider's sandbox standing in for the real gateway; the sandbox declines one student's card, as test sandboxes let a tester arrange. Five scenarios cover the main flow, one alternative flow (a concession student, late, with backlog papers) and three exception flows, each with the end state the requirements expect: the form's result, the payment, and whether a hall ticket was issued.

FORM_FEE, BACKLOG_FEE = 800, 150

def total_fee(days_late, backlog_papers, concession):
    if days_late < 0 or days_late > 15:
        raise ValueError("form not accepted")
    fee = FORM_FEE + BACKLOG_FEE * backlog_papers - (FORM_FEE if concession else 0)
    return fee + (0 if days_late == 0 else (100 if days_late <= 7 else 500))

class SandboxGateway:                         # the provider's test sandbox: declines one card
    def charge(self, roll_no, rupees):
        return roll_no != "2026CS040"

class ExamReg:                                # the whole system, every module integrated
    def __init__(self, students, gateway):
        self.students, self.gateway = students, gateway
        self.forms, self.hall_tickets, self.next_seat = {}, {}, 101

    def submit_form(self, roll_no, backlog_papers, days_late):
        student = self.students[roll_no]
        if not student["term_granted"]:
            return "not eligible"
        try:
            fee = total_fee(days_late, backlog_papers, student["concession"])
        except ValueError:
            return "refused"
        self.forms[roll_no] = {"fee": fee, "status": "submitted"}
        return fee

    def pay(self, roll_no):
        form = self.forms[roll_no]
        form["status"] = "paid" if self.gateway.charge(roll_no, form["fee"]) else "declined"
        return form["status"]

    def issue_hall_ticket(self, roll_no):
        if roll_no in self.forms:             # a seat for every student with a form
            self.hall_tickets[roll_no] = self.next_seat
            self.next_seat += 1
        return self.hall_tickets.get(roll_no)

students = {"2026CS014": {"term_granted": True, "concession": False},
            "2026CS027": {"term_granted": True, "concession": True},
            "2026CS031": {"term_granted": False, "concession": False},
            "2026CS040": {"term_granted": True, "concession": False},
            "2026CS052": {"term_granted": True, "concession": False}}
scenarios = [  # (student, backlog papers, days late, expected: form result, payment, hall ticket?)
    ("2026CS014", 0, 0, (800, "paid", True)),
    ("2026CS027", 2, 3, (400, "paid", True)),
    ("2026CS031", 0, 0, ("not eligible", None, False)),
    ("2026CS040", 1, 10, (1450, "declined", False)),
    ("2026CS052", 0, 16, ("refused", None, False)),
]

def describe(form, payment, ticket):
    return (f"form {form}, payment {payment or 'not attempted'},"
            f" hall ticket {'issued' if ticket else 'none'}")

system = ExamReg(students, SandboxGateway())
for roll_no, backlog, days, expected in scenarios:
    result = system.submit_form(roll_no, backlog, days)
    payment = system.pay(roll_no) if isinstance(result, int) else None
    ticket = system.issue_hall_ticket(roll_no) is not None
    actual = (result, payment, ticket)
    print(f"{roll_no}: {describe(*actual)}: {'pass' if actual == expected else 'FAIL'}")
    if actual != expected:
        print(f"           expected {describe(*expected)}")

2026CS014: form 800, payment paid, hall ticket issued: pass
2026CS027: form 400, payment paid, hall ticket issued: pass
2026CS031: form not eligible, payment not attempted, hall ticket none: pass
2026CS040: form 1450, payment declined, hall ticket issued: FAIL
           expected form 1450, payment declined, hall ticket none
2026CS052: form refused, payment not attempted, hall ticket none: pass

Four scenarios pass, and the one that fails is the one a unit test would never run. Student 2026CS040's card is declined, correctly, and the form's status says so; but the hall-ticket module issues a seat anyway, because it checks that a form exists, not that it has been paid for. Every module did its own job: the fee was right, the gateway's answer was recorded, the seat was allocated. The defect is in the flow between payment and hall ticket, "Incorrect control and/or data flows within the system", in the syllabus's words, and a student who never paid would sit the examination. It needed a scenario that went all the way through with an exception in the middle, which is exactly what end-to-end system testing designs for.

The test environment

A test environment is the "environment containing facilities, hardware, software, firmware, and procedures needed to conduct a test" (ISO/IEC/IEEE 29119-2:2021). For system testing it matters more than at any earlier level, because the system is tested with all its parts and dependencies, and "The test environment should ideally correspond to the final target or production environment" (ISTQB 2018). For ExamReg that means the same web server, database version and configuration as the college's live portal; the payment provider's sandbox in place of the live gateway; realistic student records, with personal data replaced; and a clock that can be set, so that the last date and the late bands can be tested on any day. Chapter Forty, on the challenges of integration testing, gave the reasons for each.

System integration testing

The syllabus names a companion level: system integration testing "focuses on testing the interfaces of the system under test and other systems and external services", and "requires suitable test environments preferably similar to the operational environment." ExamReg's external partners are the payment gateway and the email service that sends receipts. System integration tests check the conversations with them: a payment approved, declined, timed out; a receipt sent, bounced, delayed. The worked example's declined card is a system integration condition that led straight to a system defect, which is why the two levels are usually planned together.

Who does system testing

The syllabus says that "System testing is typically carried out by independent testers who rely heavily on specifications", and warns of the cost of poor specifications at this level: "Defects in specifications (e.g., missing user stories, incorrectly stated business requirements, etc.) can lead to a lack of understanding of, or disagreements about, expected system behavior." Its remedy is the one this book has repeated since Chapter Eighteen, on the role of testing in each phase: "Early involvement of testers in user story refinement or static testing activities, such as reviews, helps to reduce the incidence of such situations."

What it does not mean

System testing is not the sum of the unit tests. It tests flows across the whole system, which no unit test sees; the hall-ticket defect passed every unit test.

End to end does not mean only the happy path. The most revealing scenarios contain an exception partway through.

System testing is not acceptance testing. Testers do it against the specification to find defects; users do acceptance testing to decide whether to accept.

A test environment is not the production environment. It should resemble production closely, and every difference is a risk to be named.

Quick revision

• System testing: testing a complete, integrated system against its specified requirements (ISO/IEC 29110-5-1-2:2025); end-to-end functional tasks plus non-functional quality characteristics (ISTQB).
• Objectives: reduce risk, verify functional and non-functional behaviour, validate completeness, build confidence, find defects, stop them escaping; informs release decisions.
• Test basis: requirements, risk reports, use cases, user stories, behaviour and state models, manuals.
• Typical defects: incorrect calculations, wrong behaviour, incorrect control and data flows, failed end-to-end tasks, environment failures, behaviour unlike the manuals.
• End-to-end scenarios: main, alternative and exception flows through every part.
• Worked example: a declined payment still received a hall ticket; only an end-to-end exception scenario showed it.
• Test environment as close to production as possible; system integration testing of external interfaces; usually done by independent testers.

Test yourself

1. What is system testing, and what is it based on? Testing of a complete, integrated system to evaluate its compliance with its specified requirements, both functional and non-functional. It is based on requirement specifications, use cases and user stories, risk analyses, models of system behaviour, and system and user manuals.

2. List four typical defects that system testing finds. Incorrect calculations; incorrect control or data flows within the system, such as a hall ticket issued for an unpaid form; failure to carry out end-to-end tasks completely; and failure of the system to work in its environment or as its manuals describe.

3. What is an end-to-end scenario, and how should scenarios be chosen? A complete task from the user's first action to the system's final effect, run through every part it touches. Scenarios should cover the main flow, the alternative flows and the exception flows of each use case, since the flows with an exception partway through are the most revealing.

4. Why must the system test environment resemble production? Because the whole system is tested with all its parts and dependencies, and any difference in software versions, configuration, data or external services is a risk that behaviour in testing will differ from behaviour in use.

5. Distinguish system testing from system integration testing. System testing tests the behaviour and qualities of the whole system against its specification. System integration testing tests the interfaces between the system and other systems and external services, such as a payment gateway or an email service.