Test Levels and Test Types

The four test types

Where a level is a place, a type is a purpose. The ISTQB syllabus names four.

• Functional testing "evaluates the functions that a component or system should perform". Its objective is checking "the functional completeness, functional correctness and functional appropriateness". It asks what the system does.
• Non-functional testing "evaluates attributes other than functional characteristics of a component or system", which the syllabus calls testing "how well the system behaves": performance efficiency, compatibility, usability, reliability, security, maintainability, portability and safety, the characteristics of ISO/IEC 25010 from Chapter Twenty, on the quality model today.
• Black-box testing "is specification-based and derives tests from documentation not related to the internal structure of the test object". ISO/IEC/IEEE 29119-1 calls the same thing specification-based testing, in which "the principal test basis is the external inputs and outputs of the test item".
• White-box testing "is structure-based and derives tests from the system's implementation or internal structure (e.g., code, architecture, work flows, and data flows)". Its objective is "to cover the underlying structure by the tests to an acceptable level". The standard's name is structure-based testing.

Notice that the four are not one list of alternatives. Functional and non-functional divide testing by what is being evaluated; black-box and white-box divide it by where the tests come from. A single test can be functional and black-box (a fee case from the fee table) or non-functional and white-box (a timing test aimed at one slow loop). Chapter Fifty-Two, on black-box and white-box testing, starts Module 2 with the techniques of each family.

The grid: every level, every type

The syllabus says the four types "can be applied to all test levels, although the focus will be different at each level." On ExamReg the grid looks like this.

Confirmation testing and regression testing

Two more kinds of testing are defined not by level or type but by what triggers them: a change. The syllabus says that after a change, "Testing should then also include confirmation testing and regression testing."

Confirmation testing, which ISO/IEC/IEEE 29119-2 calls retesting, is "testing performed to check that modifications made to correct a fault have successfully removed the fault". In the syllabus's version it "confirms that an original defect has been successfully fixed", by executing the tests that failed because of it and, where needed, new tests covering the change.

Regression testing is "testing performed following modifications to a test item or to its operational environment, to identify whether failures in unmodified parts of the test item occur" (ISO/IEC/IEEE 29119-1:2022). The syllabus stresses its scope: adverse consequences "could affect the same component where the change was made, other components in the same system, or even other connected systems", so "It is advisable first to perform an impact analysis to recognize the extent of the regression testing."

Worked example: a fix that passes, and a regression it causes

ExamReg's test team reports that a form submitted exactly 15 days late is refused, although the rule says it pays Rs 500. The developer fixes the first condition and, while the file is open, "tidies" another line. The program runs the confirmation test on the fix, then reruns the rest of the suite as a regression test.

def late_fee_before(days_late):          # the build tested: refuses a form 15 days late
    if days_late < 0 or days_late >= 15:
        return "refused"
    if days_late <= 0:
        return 0
    if days_late <= 7:
        return 100
    return 500

def late_fee_after(days_late):           # the developer's fix, with a tidy-up on the side
    if days_late < 0 or days_late > 15:
        return "refused"
    if days_late <= 0:
        return 0
    if days_late <= 8:                   # "tidied" while the file was open
        return 100
    return 500

suite = {"TC-01": (-1, "refused"), "TC-02": (0, 0), "TC-03": (1, 100), "TC-04": (7, 100),
         "TC-05": (8, 500), "TC-06": (15, 500), "TC-07": (16, "refused")}
failed_before = [t for t, (d, e) in suite.items() if late_fee_before(d) != e]
print("failed on the build tested:", ", ".join(failed_before))

for t in failed_before:                                   # confirmation testing
    d, e = suite[t]
    got = late_fee_after(d)
    print(f"confirmation {t} ({d} days late): expected {e}, got {got}:",
          "pass" if got == e else "FAIL")

rest = [t for t in suite if t not in failed_before]      # regression testing
broken = [t for t in rest if late_fee_after(suite[t][0]) != suite[t][1]]
print(f"regression: {len(rest)} tests rerun, {len(broken)} failed")
for t in broken:
    d, e = suite[t]
    print(f"  {t} ({d} days late): expected {e}, got {late_fee_after(d)}")

failed on the build tested: TC-06
confirmation TC-06 (15 days late): expected 500, got 500: pass
regression: 6 tests rerun, 1 failed
  TC-05 (8 days late): expected 500, got 100

The confirmation test passes: the reported failure is gone. A team that stopped there would ship a new defect, because the "tidy-up" moved day 8 into the Rs 100 band. Only the regression run, rerunning tests that had passed before and whose code nobody meant to change, catches it. This is why the syllabus calls regression suites "a strong candidate for automation": they are rerun after every change, they grow with every release, and the failures they catch are exactly the ones nobody is looking for. Chapter Thirty-Nine, on regression testing, smoke testing and continuous integration, runs them automatically on every build.

Where each kind of testing is taught in this book

The rest of Module 1 goes through the levels, and Module 2 through the techniques, in this order:

• Static testing: the kinds of V&V and reviews, inspection and the walkthrough, Chapters Twenty-Seven to Thirty.
• Unit testing: its purpose, techniques, frameworks and best practices, Chapters Thirty-Three to Thirty-Six.
• Integration testing: why units fail together, the integration approaches, regression and smoke testing with continuous integration, and the challenges of integration, Chapters Thirty-Seven to Forty.
• Validation and acceptance testing: validation testing and acceptance testing with alpha and beta, Chapters Forty-One and Forty-Two.
• System testing: system testing end to end, the system test types, load testing and cross-browser testing, Chapters Forty-Three to Forty-Six.
• Debugging, which follows a failure at any level, Chapter Forty-Seven.
• Test automation: automation, browser drivers, data-driven testing and the page object model, Chapters Forty-Eight to Fifty-One.
• Black-box, white-box and experience-based techniques in Module 2, from black-box and white-box testing in Chapter Fifty-Two to checklist-based testing in Chapter Sixty-Four.

What it does not mean

A test level is not a test type. "System testing" says where; "performance testing" says what; a system-level performance test is both.

White-box testing is not confined to unit testing. Structure can be covered at every level: code at component level, call paths at integration level, workflows at system level.

Confirmation testing is not regression testing. The first checks the fix; the second checks everything the fix might have disturbed.

Regression testing is not only for code changes. A change to the operating system, the database or a connected system can break unchanged code, and the definition includes changes to the "operational environment".

Quick revision

• Test level: a test stage with its own objectives and risks. Five: component, component integration, system, system integration, acceptance.
• Levels differ in test object, objectives, test basis, defects and failures, and approach and responsibilities.
• Test type: testing focused on specific quality characteristics. Four: functional (what), non-functional (how well), black-box (from the specification), white-box (from the structure). All four apply at every level.
• Confirmation testing (retesting): checks the fix removed the fault. Regression testing: checks unmodified parts still work after a change, scoped by impact analysis, and automated.
• Worked example: the fix for day 15 passed its confirmation test; regression caught the new day-8 defect.

Test yourself

1. Name the five test levels and give the test object of each. Component testing (a single unit in isolation), component integration testing (units combined and their interfaces), system testing (the whole system), system integration testing (the system's interfaces with external systems) and acceptance testing (the system in its intended use).

2. Name the four test types and state what each evaluates. Functional testing evaluates what the system does; non-functional testing how well it behaves (performance, usability, security and so on); black-box testing derives tests from the specification, not the internal structure; white-box testing derives them from the internal structure, aiming to cover it.

3. Can a test type be applied at more than one level? Give an example. Yes; all four types apply at all levels. Performance, a non-functional type, can be tested at component level (the fee function's speed) and at system level (500 students on the last date).

4. Distinguish confirmation testing from regression testing. Confirmation testing reruns the tests that failed, and new tests for the change, to check that a defect has been fixed. Regression testing reruns tests that passed before, chosen by impact analysis, to check that the change has not broken anything else in the component, the system, connected systems or the environment.

5. Why is regression testing a strong candidate for automation? Because the same tests are rerun after every change, the suite grows with every release, and running it by hand each time would be slow and error-prone; automated in continuous integration, it runs on every build.