Writing Unit Tests With a Framework

Two outcomes other than pass and fail are worth knowing. A skipped test is deliberately not run, with a reason recorded. An error (as distinct from a failure) means the test itself crashed with an unexpected exception, rather than an assertion finding a wrong answer.

Worked example: ExamReg's fee service under unittest

The unit is the fee service of Chapter Thirty-Four, on drivers, stubs and test doubles, now in its own module, feeservice.py, as it would be in the project.

FORM_FEE, BACKLOG_FEE = 800, 150

def total_fee(days_late, backlog_papers, concession):
    if days_late < 0 or days_late > 15:
        raise ValueError("form not accepted")
    if not isinstance(backlog_papers, int) or backlog_papers < 0:
        raise ValueError("backlog papers must be a whole number, 0 or more")
    fee = FORM_FEE + BACKLOG_FEE * backlog_papers - (FORM_FEE if concession else 0)
    late = 0 if days_late == 0 else (100 if days_late <= 7 else 500)
    return fee + late

class FeeService:
    def __init__(self, students, clock, gateway, mailer):
        self.students, self.clock, self.gateway, self.mailer = students, clock, gateway, mailer

    def pay(self, roll_no, last_date):
        student = self.students.find(roll_no)
        days_late = max(0, (self.clock.today() - last_date).days)
        amount = total_fee(days_late, student["backlog"], student["concession"])
        receipt = self.gateway.charge(roll_no, amount)
        self.mailer.send(student["email"], f"Paid Rs {amount}; receipt {receipt}")
        return amount

The test module imports it. The fixture, setUp, builds a fresh fake student store, stub clock, mock gateway and mock mailer before every test, so no test can be disturbed by another. Five test cases check the fee rules and the gateway call; one is the exam cell's new requirement, that the receipt must show the late fee separately; and one is skipped because the hall ticket PDF does not exist yet. The runner's report is printed without its timing line, which differs on every run.

import io
import unittest
from datetime import date
from unittest.mock import Mock
from feeservice import FeeService

class FakeStudentStore:
    def __init__(self, rows):
        self.rows = rows
    def find(self, roll_no):
        return self.rows[roll_no]

class FeeServiceTests(unittest.TestCase):
    def setUp(self):                                # the fixture, built afresh for every test
        self.students = FakeStudentStore({
            "2026CS014": {"backlog": 1, "concession": False, "email": "s014@college.example"},
            "2026CS027": {"backlog": 2, "concession": True, "email": "s027@college.example"}})
        self.clock = Mock()
        self.clock.today.return_value = date(2026, 11, 20)
        self.gateway = Mock()
        self.gateway.charge.return_value = "R-1042"
        self.mailer = Mock()
        self.service = FeeService(self.students, self.clock, self.gateway, self.mailer)

    def test_late_student_pays_form_backlog_and_late_fee(self):
        self.assertEqual(self.service.pay("2026CS014", date(2026, 11, 10)), 1450)

    def test_concession_waives_only_the_form_fee(self):
        self.assertEqual(self.service.pay("2026CS027", date(2026, 11, 17)), 400)

    def test_gateway_is_charged_once_with_the_total(self):
        self.service.pay("2026CS014", date(2026, 11, 10))
        self.gateway.charge.assert_called_once_with("2026CS014", 1450)

    def test_sixteen_days_late_is_refused(self):
        with self.assertRaises(ValueError):
            self.service.pay("2026CS014", date(2026, 11, 4))

    def test_receipt_shows_the_late_fee_separately(self):     # the exam cell's new requirement
        self.service.pay("2026CS014", date(2026, 11, 10))
        to, text = self.mailer.send.call_args.args
        self.assertIn("late fee Rs 500", text)

    @unittest.skip("hall ticket PDF not built yet")
    def test_hall_ticket_link_in_receipt(self):
        pass

def run(suite):
    """Run a suite and print the runner's report, less its timing line."""
    report = io.StringIO()
    unittest.TextTestRunner(stream=report, verbosity=1).run(suite)
    lines = report.getvalue().splitlines()
    print(lines[0])                                      # one mark per test: . F E s
    for line in lines:
        if line.startswith(("FAIL:", "ERROR:", "AssertionError")):
            print(line)
    print(lines[-1])                                     # OK, or FAILED with the counts

print("whole class:")
run(unittest.defaultTestLoader.loadTestsFromTestCase(FeeServiceTests))
print("smoke group, in the order given:")
run(unittest.TestSuite([FeeServiceTests("test_late_student_pays_form_backlog_and_late_fee"),
                        FeeServiceTests("test_gateway_is_charged_once_with_the_total")]))

whole class:
..s.F.
FAIL: test_receipt_shows_the_late_fee_separately (__main__.FeeServiceTests.test_receipt_shows_the_late_fee_separately)
AssertionError: 'late fee Rs 500' not found in 'Paid Rs 1450; receipt R-1042'
FAILED (failures=1, skipped=1)
smoke group, in the order given:
..
OK

Read the runner's first line as a row of marks, one per test in the order the loader found them, which for unittest is alphabetical: a dot for a pass, F for a failure, E for an error and s for a skip. Four tests passed, one was skipped, and one failed; and the failure did not stop the tests after it, which is the first thing a framework buys. The failure report names the test and says exactly what was wrong: the receipt reads Paid Rs 1450; receipt R-1042 and never mentions the late fee. The test is right and the code is behind the requirement; the test will stay red until the receipt is changed, which is exactly the signal it exists to give.

The second run is a suite built by hand: two chosen tests, run in the order given. That is how unittest does what TestNG calls groups and priorities, a named subset of tests run in a chosen order, and it is how a team would build a quick smoke suite to run on every build.

TestNG, the practical's framework

The practical asks students to "Create and execute automated test cases using TestNG annotations. Group test cases, define priorities, and generate HTML test reports." TestNG does with annotations what unittest does with method names and subclasses. Here is how the first test above looks in TestNG; it is an excerpt, not a whole program, and it has not been compiled here.

import org.testng.annotations.*;
import static org.testng.AssertJUnit.*;

public class FeeServiceTest {
    private FeeService service;

    @BeforeMethod
    public void setUp() { service = new FeeService(students, clock, gateway, mailer); }

    @Test(groups = {"smoke"}, priority = 1)
    public void lateStudentPaysFormBacklogAndLateFee() {
        assertEquals(1450, service.pay("2026CS014", lastDate));
    }
}

The pieces map one to one onto what the Python version did.

• Annotations mark each method's job. The documentation defines them precisely: @BeforeMethod means "The annotated method will be run before each test method", like unittest's setUp; @BeforeClass "will be run before the first test method in the current class is invoked"; @BeforeSuite "will be run before all tests in this suite have run". As TestNG's documentation says, "it's the annotations that tell TestNG what they are", so test methods can have any names.
• Groups. "A test method can belong to one or several groups", named in @Test(groups = {...}). A run can then include or exclude whole groups, which is how a smoke group or a slow group is run on its own.
• Priority. The priority attribute is "The priority for this test method. Lower priorities will be scheduled first." It orders tests where the order matters; the best practice of the next chapter is to write tests that do not depend on order at all.
• Reports. "The results of the test run are created in a file called index.html in the directory specified when launching SuiteRunner", by default test-output, and that page links to the rest of the HTML report. unittest's own runner, as its documentation says, may report through "a graphical interface, a textual interface, or return a special value"; the one in the standard library is the text runner used above.

Groups are chosen for a run in the suite file, testng.xml:

<suite name="ExamReg">
  <test name="Smoke">
    <groups><run><include name="smoke"/></run></groups>
    <classes><class name="examreg.FeeServiceTest"/></classes>
  </test>
</suite>

What it does not mean

A framework does not decide what to test. It runs the checks it is given; choosing them is test design, the subject of Module 2.

A skipped test is not a passed test. It records that something is not being checked, with the reason, so it can be found and finished.

A failure and an error are not the same. A failure means an assertion found a wrong answer; an error means the test crashed, which may be a fault in the code or in the test.

Groups and priorities are not a substitute for independent tests. A test that passes only when another runs first is a defect in the test suite.

Quick revision

• xUnit concepts (unittest documentation): test fixture (preparation and cleanup), test case ("the individual unit of testing"), test suite (a collection of cases and suites), test runner (runs tests and reports); plus assertions.
• unittest: subclass TestCase; methods named test...; setUp/tearDown; assertEqual, assertIn, assertRaises; @unittest.skip; TestSuite; TextTestRunner.
• Runner marks: . pass, F failure, E error, s skip; a failure does not stop the other tests.
• TestNG: annotations (@Test, @BeforeMethod, @AfterMethod, @BeforeClass, @BeforeSuite, @BeforeTest, @BeforeGroups); groups; priority (lower runs first); HTML report in test-output/index.html; groups chosen in testng.xml.
• Worked example: 6 tests; 4 passed, 1 skipped, 1 failed (the receipt lacks the late fee); a two-test smoke suite passed.

Test yourself

1. Explain the terms test fixture, test case, test suite and test runner. A test fixture is the preparation needed to perform one or more tests, and the cleanup after them. A test case is the individual unit of testing, checking a specific response to a particular set of inputs. A test suite is a collection of test cases, suites or both, run together. A test runner orchestrates the running of tests and reports the outcome.

2. What does a unit testing framework provide that a hand-written loop of checks does not? Independent tests, so that one failure or crash does not stop the rest; shared setup and cleanup through fixtures; grouping into suites and selective running; standard assertions with clear failure messages; and a runner with a consistent report that a CI server can read.

3. Name four TestNG annotations and state when each method runs. @BeforeSuite runs before all tests in the suite; @BeforeClass before the first test method in the class; @BeforeMethod before each test method; @AfterMethod after each test method. @Test marks a test method.

4. What are groups and priority in TestNG? Groups are named sets that a test method can belong to, one or several, so that a run can include or exclude whole groups such as a smoke group. Priority is an attribute of @Test that orders test methods, with lower priorities scheduled first.

5. Distinguish a failure from an error in a test run. A failure means an assertion was false: the code gave a wrong answer. An error means the test raised an unexpected exception and did not finish, which may be a fault in the code or in the test itself.