Software Reviews: The Process, the Roles and the Types

One word in the process needs care. An anomaly is "anything observed in the documentation or operation of a system that deviates from expectations based on previously verified system, software, or hardware products or reference documents" (IEEE 1012-2024). A reviewer logs anomalies; only the analysis decides which of them are defects. A question that turns out to have a good answer is an anomaly, but not a defect.

Review techniques: how a reviewer reads

ISO/IEC 20246 names several techniques for the individual review. A reviewer can use more than one.

The roles

The ISTQB syllabus names six principal roles, and one person may hold several.

The four review types

The syllabus observes that review types range "from informal reviews to formal reviews", and ISO/IEC 20246 defines the two ends: an informal review is a "form of review that does not follow a defined process and has no formal documented output", and a formal review is a "form of review that follows a defined process with formal documented output". Four types are in common use.

• Informal review. "Informal reviews do not follow a defined process and do not require a formal documented output. The main objective is detecting anomalies." A colleague reading a draft and scribbling in the margin is an informal review, and so is much of pair programming.
• Walkthrough. "A walkthrough, which is led by the author, can serve many objectives", from "evaluating quality and building confidence in the work product" to "educating reviewers, gaining consensus, generating new ideas" and "detecting anomalies". ISO/IEC 20246 defines it as a "formal review in which an author leads members of the review through a work product, and the participants ask questions and make comments about possible issues." Chapter Thirty is about the walkthrough.
• Technical review. "A technical review is performed by technically qualified reviewers and led by a moderator." Its objectives are "to gain consensus and make decisions regarding a technical problem", and also to detect anomalies and evaluate quality. ISO/IEC 20246 defines it as a "formal peer review of a work product by a team of technically qualified personnel that examines the suitability of the work product for its intended use and identifies discrepancies from specifications and standards".
• Inspection. "As inspections are the most formal type of review, they follow the complete generic process". "The main objective is to find the maximum number of anomalies", and "Metrics are collected and used to improve the SDLC, including the inspection process. In inspections, the author cannot act as the review leader or scribe." Chapter Twenty-Nine is about the inspection.

The level of formality is a choice. The syllabus says it depends on "the SDLC being followed, the maturity of the development process, the criticality and complexity of the work product being reviewed, legal or regulatory requirements, and the need for an audit trail", and that "The same work product can be reviewed with different review types, e.g., first an informal one and later a more formal one."

The formal technical review of Module 2 (Chapter Ninety-Seven) returns to reviews from the quality assurance side, as an SQA activity with its own guidelines; this chapter's process and types are the foundation it builds on.

Worked example: reviewing ExamReg's fee requirement

Here is the requirement as the analyst first wrote it.

Planning and initiation. The review leader chooses a technical review, because the fee rules decide money. The scope is R-FEE-3 and the exam cell's note it came from; the focus is boundaries and exceptions; the exit criterion is that no defect of major severity remains open. Each reviewer receives the requirement, the note and the team's requirements checklist.

Individual review. Each reviewer reads with a different technique. The tester uses the checklist and drafts test cases as they read; the exam cell clerk reads in role, as a student would meet the rule; the developer reads for anything that cannot be coded unambiguously. Their logs, merged by the scribe:

Communication and analysis. The meeting, run by the moderator, decides each item's status.

Fixing and reporting. The analyst rewrites R-FEE-3; the scribe's record goes to the project; the exit criterion is met once the major defect, item 4, is corrected and checked.

Four defects in one short requirement, found in a two-hour meeting. Item 1 is the on-time defect that Chapter One, on what software testing is, found with a failing test after the code was written; item 4 is the wrong specification that Chapter Twenty-Six showed no amount of verification could catch; and items 1 to 3 are the boundaries that the static analyser of Chapter Twenty-Seven found in code. Found here, each costs a sentence to fix.

What makes reviews succeed

The ISTQB syllabus lists the success factors, and every one of them can be seen in the example.

• "Defining clear objectives and measurable exit criteria. Evaluation of participants should never be an objective"
• "Choosing the appropriate review type to achieve the given objectives"
• "Performing reviews on small chunks, so that reviewers do not lose concentration"
• Feedback to stakeholders and authors, adequate time to prepare, and support from management
• "Making reviews part of the organization's culture, to promote learning and process improvement"
• Adequate training for all participants, and facilitated meetings

The first deserves emphasis. A review examines the work, never the person. If authors fear that their anomalies will be counted against them, they stop bringing work to review early, and the cheapest defect-finding activity in the project is lost.

What it does not mean

A review is not an inspection by another name. Inspection is one type, the most formal; the informal review, the walkthrough and the technical review are reviews too.

Every anomaly is not a defect. Anomalies are analysed; some are questions with good answers, and some are recommendations.

A review is not an assessment of the author. Evaluating participants "should never be an objective".

Reviews are not only for code. Any work product that can be read, from a requirement to a test plan, can be reviewed.

Quick revision

• Review: a process or meeting in which work products are presented to stakeholders for comment or approval.
• Process (ISO/IEC 20246, ISTQB v4.0.1): planning, review initiation, individual review, communication and analysis, fixing and reporting.
• Anomaly: a deviation from expectations; analysis decides whether it is a defect.
• Techniques: ad hoc, checklist-based, scenario-based, role-based, perspective-based.
• Roles: manager, author, moderator (facilitator), scribe (recorder), reviewer, review leader.
• Types: informal review (no defined process), walkthrough (led by the author), technical review (led by a moderator, technical consensus), inspection (most formal, maximum anomalies, metrics, author never leader or scribe).
• Formality depends on the SDLC, process maturity, criticality, regulation and the need for an audit trail.
• Success: clear objectives and exit criteria, never evaluating participants, small chunks, time, training, management support.

Test yourself

1. Describe the activities of the review process. Planning defines the scope, focus, exit criteria, effort and timeframes. Review initiation makes sure everyone has the work product and understands their role. In individual review each reviewer applies review techniques and logs anomalies, recommendations and questions. In communication and analysis the anomalies are discussed, usually in a meeting, and each gets a status, owner and action. In fixing and reporting, defects are reported and fixed, the exit criteria are checked, and the results are reported.

2. Name the roles in a review and their responsibilities. The manager decides what is reviewed and provides resources; the author creates and fixes the work product; the moderator runs the meeting and keeps it safe and on time; the scribe records anomalies and decisions; the reviewers perform the review; and the review leader takes overall responsibility and organises it.

3. Compare the four types of review. An informal review follows no defined process and needs no documented output, and aims to detect anomalies. A walkthrough is led by the author and serves many objectives, including education and consensus. A technical review is led by a moderator with technically qualified reviewers, to reach decisions on technical problems. An inspection is the most formal, follows the complete process, aims to find the most anomalies, collects metrics, and never lets the author lead or record.

4. Why is every anomaly found in a review not a defect? Because an anomaly is only a deviation from what a reviewer expected; on analysis it may turn out to be a question with a good answer, a recommendation, or a misunderstanding by the reviewer. The meeting decides which anomalies are defects.

5. Give four factors for a successful review. Clear objectives and measurable exit criteria, with evaluation of participants never an objective; the right review type for the objectives and work product; reviewing in small chunks with adequate preparation time; and management support with training for all participants.