The Software Testing Life Cycle, Phase by Phase

Phase 3: test case development. Test scenarios and test cases are written, with test data and expected results, and reviewed. The requirement traceability matrix is updated so every requirement maps to its test cases. Automated test scripts are written here too.

Phase 4: test environment set-up. The hardware, software, network, browsers, databases, test accounts and permissions are prepared, and the environment is checked, usually with a short smoke test that confirms the build installs and its main functions respond. This phase often runs in parallel with phase 3, and it is often run by a separate team.

Phase 5: test execution. The test cases are run as scheduled, results are compared with expected results and logged, failures are reported as defects with severity and priority, fixed defects are retested, and regression tests are run after fixes. Execution usually takes several test cycles, each on a new build.

Phase 6: test cycle closure. The team checks the exit criteria, prepares the test summary (or completion) report, makes sure every defect is closed or knowingly deferred, archives the testware, returns the environment, and holds a meeting on lessons learned.

Entry criteria, exit criteria and deliverables

The phases are made controllable by their criteria. ISO/IEC/IEEE 24765:2017 defines entry criteria as "states of being that have to be present before an effort can begin successfully" and exit criteria as those that "have to be present before an effort can end successfully". The table gives typical ones; a real test plan states them for its own project.

The six phases against the standard process

Read the table and one difference stands out. The STLC puts requirement analysis before planning, while ISTQB puts planning first and analysis after. In practice both happen together: a plan cannot be made without knowing what the requirements ask for, and the analysis cannot be complete without knowing the plan's scope. The last row matters too. Monitoring and control are not a phase of the STLC, but they run through all six, exactly as in the basic test process of Chapter Five.

Worked example: does execution exit?

ExamReg release 2.0 has 240 planned test cases, 40 of them critical, 80 high priority and 120 medium. The test plan's exit criteria for the execution phase are: every critical test has been run; at least 90 per cent of all planned tests have been run; at least 95 per cent of the tests run have passed; and no critical defect is open. After the first execution cycle the figures are as below, and the program checks each criterion.

planned = {"critical": 40, "high": 80, "medium": 120}
run     = {"critical": 40, "high": 78, "medium": 101}
passed  = {"critical": 38, "high": 72, "medium": 96}
open_defects = {"critical": 1, "major": 3, "minor": 7}

total_planned = sum(planned.values())
total_run = sum(run.values())
total_passed = sum(passed.values())
criteria = [
    ("every critical test run", run["critical"] == planned["critical"]),
    (f"at least 90% of planned tests run ({total_run} of {total_planned}, "
     f"{100 * total_run / total_planned:.1f}%)", total_run >= 0.90 * total_planned),
    (f"at least 95% of tests run passed ({total_passed} of {total_run}, "
     f"{100 * total_passed / total_run:.1f}%)", total_passed >= 0.95 * total_run),
    ("no critical defect open", open_defects["critical"] == 0),
]
for text, met in criteria:
    print(("met     " if met else "NOT MET ") + text)
print("execution phase may close:", all(met for _, met in criteria))

met     every critical test run
met     at least 90% of planned tests run (219 of 240, 91.2%)
NOT MET at least 95% of tests run passed (206 of 219, 94.1%)
NOT MET no critical defect open
execution phase may close: False

The phase does not close. Two criteria are met and two are not: the pass rate is just under the threshold, and one critical defect is still open. Test control now acts: the critical defect is fixed first, the failed tests are rerun on the next build, and a second cycle begins. Nobody argues about whether testing is finished, because the plan said in advance what finished means. That is the whole value of exit criteria.

The STLC inside the SDLC

The STLC is not a separate life cycle running beside development; it runs inside the software development life cycle (SDLC), which Module 1's second row covers from Chapter Thirteen. Its first two phases can begin as soon as requirements exist, long before code, which is principle 3 (early testing) in practice. In the V-model (Chapter Fifteen), each development phase on the left of the V has its test level on the right, and a complete STLC, from analysis to closure, can be run for each test level.

In agile development the six phases compress into each short iteration. Requirement analysis becomes refining a user story and its acceptance criteria, planning becomes the sprint's test tasks, case development and execution happen within the sprint, and closure becomes the sprint review and retrospective. The phases are still there; they are just two weeks long.

What it does not mean

The STLC is not an international standard. It is industry's phase view of the standard test process; ISO/IEC/IEEE 29119-2 and ISTQB define the process it describes.

The STLC does not make a product defect-free. Some descriptions give its objective as a defect-free product; no process can deliver that (principle 1, Chapter Four). Its objective is to find defects and give evidence about quality in a planned, controlled way.

The phases are not strictly sequential. Environment set-up usually overlaps case development, and execution runs in cycles with fixes between them.

The STLC phases are not test levels. Unit, integration, system and acceptance testing are levels (Chapter Thirty-Two); the STLC can be followed within any of them.

Quick revision

• STLC: the phases testing goes through in a project; industry's view of the test process, not an international standard.
• Six phases: requirement analysis, test planning, test case development, test environment set-up, test execution, test cycle closure.
• Each has entry criteria (to begin), exit criteria (to end) and deliverables.
• Deliverables: review comments; test plan; test cases, data, traceability matrix; ready environment and smoke test result; logs, defect reports, execution reports; summary report and lessons learned.
• Maps onto ISTQB's activities and ISO/IEC/IEEE 29119-2's processes; monitoring and control run throughout.
• Worked example: 219 of 240 tests run (91.25 per cent), 206 of them passed (about 94.1 per cent), one critical defect open, so execution does not exit.
• In agile, all six phases happen inside each iteration.

Test yourself

1. What is the software testing life cycle? Name its phases. It is the sequence of phases through which testing proceeds in a project: requirement analysis, test planning, test case development, test environment set-up, test execution and test cycle closure, each with entry criteria, exit criteria and deliverables.

2. Define entry and exit criteria, with one example of each for test execution. Entry criteria are the conditions that must hold before a phase begins, such as approved test cases and an environment that has passed its smoke test. Exit criteria are those that must hold before it ends, such as every critical test run and no critical defect open.

3. What are the deliverables of test cycle closure? The test summary or completion report, confirmation that defects are closed or knowingly deferred, archived testware, the environment returned, and documented lessons learned.

4. How does the STLC relate to the SDLC? It runs inside the SDLC, as its testing thread. Its first phases can begin as soon as requirements exist, before any code, and in the V-model a full testing cycle can be run for each test level.

5. In the worked example, why could the execution phase not close? Two of its four exit criteria were not met: only 94.1 per cent of the tests run had passed against a threshold of 95 per cent, and one critical defect was still open.

6. Is it correct to say the objective of the STLC is a defect-free product? Justify. No. Testing can show the presence of defects but never their absence (principle 1), so no process can guarantee a defect-free product. The STLC's objective is to find defects and give evidence of quality in a planned and controlled way.