The Test Documents, From Design to Completion

Two points about the set are worth making in an answer. First, the specification is split three ways on purpose: the design says what to cover, the case says with what values, the procedure says in what steps. Keeping them apart lets one test case be reused in several designs and one procedure run many cases. Second, the reporting documents follow execution in time: the transmittal report comes before it, the log during it, the incident reports as problems appear, and the summary after.

IEEE 829-2008 and ISO/IEC/IEEE 29119-3

The 2008 revision renamed and extended the set around test levels: a Master Test Plan over all levels and a Level Test Plan for each; Level Test Design, Level Test Case and Level Test Procedure documents; a Level Test Log; an Anomaly Report; a Level Interim Test Status Report; a Level Test Report; and a Master Test Report. The name anomaly report was chosen deliberately, as Wikipedia's account of the standard explains, because a discrepancy between expected and actual results can have causes other than a fault in the system, such as a wrong expected result.

IEEE 829-2008 has in turn been superseded by ISO/IEC/IEEE 29119-3, whose current edition, the 2021 one, is the standard SEVOCAB draws its test documentation terms from. The names map as follows.

The newer standard adds documents for things 1998 left implicit: the status report during testing (Chapter Ten, on test reporting), the readiness of the environment and the data (Chapter Nine, on test execution), and the traceability matrix.

Must every document be produced?

No. Wikipedia's account of IEEE 829-2008 notes that the standard "specified the format of these documents, but did not stipulate whether they must all be produced". Which documents a project writes is decided in its test plan, and the answer depends on context: a medical device project may produce every one, formally reviewed and signed; a three-person team building ExamReg may keep test cases and logs in a test management tool and write only a plan and a completion report as documents.

Agile teams go further. The Agile Manifesto values "Working software over comprehensive documentation", and it adds that "while there is value in the items on the right, we value the items on the left more". In agile testing the same information is still kept, but lighter: acceptance criteria on user stories instead of design specifications, automated tests that are themselves the test cases, and the continuous integration server's history instead of a hand-written log.

Traceability across the documents

The documents are only as useful as the links between them. A requirement should lead forward to its test cases, their procedures, their log entries and any incidents; and an incident should lead back to the requirement it affects. The program below records those links for one ExamReg requirement across two execution cycles, then follows them in both directions.

requirement = {"FEE-1": "no late fee on or before the last date"}
test_cases = {"TC-FEE-01": {"requirement": "FEE-1", "procedure": "TP-FEE"}}
log = [   # test log entries: cycle, test case, build, result, incident raised
    {"cycle": 1, "case": "TC-FEE-01", "build": "2.0.1", "result": "fail", "incident": "IR-014"},
    {"cycle": 2, "case": "TC-FEE-01", "build": "2.0.2", "result": "pass", "incident": None},
]
incidents = {"IR-014": {"case": "TC-FEE-01", "status": "closed after retest on 2.0.2"}}

print("forward, from the requirement:")
for rid, text in requirement.items():
    print(f"  {rid}: {text}")
    for case, info in test_cases.items():
        if info["requirement"] == rid:
            print(f"    test case {case}, run by procedure {info['procedure']}")
            for entry in log:
                if entry["case"] == case:
                    note = f", incident {entry['incident']}" if entry["incident"] else ""
                    print(f"      cycle {entry['cycle']} on build {entry['build']}: {entry['result']}{note}")

print("backward, from the incident:")
case = incidents["IR-014"]["case"]
rid = test_cases[case]["requirement"]
print(f"  IR-014 ({incidents['IR-014']['status']}) <- {case} <- {rid}: {requirement[rid]}")

forward, from the requirement:
  FEE-1: no late fee on or before the last date
    test case TC-FEE-01, run by procedure TP-FEE
      cycle 1 on build 2.0.1: fail, incident IR-014
      cycle 2 on build 2.0.2: pass
backward, from the incident:
  IR-014 (closed after retest on 2.0.2) <- TC-FEE-01 <- FEE-1: no late fee on or before the last date

Forward, the chain answers "was this requirement tested, and what happened?": it failed on build 2.0.1, an incident was raised, and it passed on 2.0.2. Backward, it answers "which rule does this incident break?" in one step. Without the links in the documents, both questions need a person's memory.

What it does not mean

IEEE 829 is not the current standard. The 1998 version was superseded in 2008, and 829-2008 by ISO/IEC/IEEE 29119-3. Its eight documents are still taught because they divide the work cleanly.

Not every document must be written. The standards define formats; the test plan decides which documents a project produces.

A test case specification is not a test design specification. The design says which features are covered and how they pass; the case gives the actual values and expected outputs.

Less documentation in agile does not mean less information. The same facts are kept in lighter forms: acceptance criteria, automated tests, tool histories.

Quick revision

• IEEE 829-1998: eight documents in three groups.
• Planning: test plan.
• Specification: test design specification (what to cover, feature pass/fail criteria); test case specification (input values and expected outputs); test procedure specification (steps to run cases).
• Reporting: test item transmittal report (items handed over); test log (what happened); test incident report (events to investigate); test summary report (summary and evaluation).
• IEEE 829-2008: master and level test plans, level design, case, procedure, log, anomaly report, interim status report, level and master test reports.
• ISO/IEC/IEEE 29119-3 supersedes IEEE 829: test status report, test completion report, test execution log, incident report, readiness reports, test traceability matrix.
• Not every document is mandatory; agile keeps the same information more lightly.

Test yourself

1. Name the eight documents of IEEE 829-1998 in their groups. Planning: the test plan. Specification: the test design specification, the test case specification and the test procedure specification. Reporting: the test item transmittal report, the test log, the test incident report and the test summary report.

2. Distinguish the test design, test case and test procedure specifications. The design specification says which features are covered, which cases and procedures are needed, and the pass/fail criteria per feature. The case specification gives the actual input values and expected outputs of each case. The procedure specification lists the steps to run the cases, in order.

3. What is a test item transmittal report, and when is it used? A document identifying the items being handed over for testing, used when a separate development group delivers builds to a test group, or when a formal start of test execution is wanted.

4. Which standard replaced IEEE 829, and what do the log and summary report become there? ISO/IEC/IEEE 29119-3, now in its 2021 edition. The test log becomes the test execution log and the test summary report becomes the test completion report; the incident report keeps its name, and a test status report is added for progress during testing.

5. Why did IEEE 829-2008 call it an anomaly report rather than a fault report? Because a difference between expected and actual results can arise from causes other than a fault in the system, such as a wrong expected result or a test run incorrectly.

6. Are all the documents mandatory for every project? No. The standards specify the format of each document, not whether it must be produced; the test plan decides, according to context, and agile teams keep the same information in lighter forms.