Formal Technical Reviews

1. The type of review: an inspection, a structured walkthrough or another method, chosen for the product and the risk.
2. The data to be collected during the review, decided in advance so that every review records the same things.
3. Entry and exit criteria: when the product is ready to be reviewed, and when the review is finished.
4. Criteria for requiring another review, such as the amount of rework.
5. Checklists "to ensure that work products are reviewed consistently", covering items such as "Rules of construction", "Design guidelines", "Completeness", "Correctness", "Maintainability" and "Common defect types".
6. A schedule, including when materials will be available.
7. Checking the entry criteria before the product is distributed.
8. Distributing the product early, "early enough to enable them to adequately prepare for the peer review."
9. Assigning roles. CMMI's examples are "Leader", "Reader", "Recorder" and "Author".
10. Individual preparation: each reviewer reviews the product before the meeting.

The roles have the duties Chapter Twenty-Nine, on inspection, gave Fagan's moderator, reader, recorder and author: the leader plans and runs the review, the reader leads the team through the product, the recorder writes down every issue, and the author answers questions and later fixes the product.

The review meeting

CMMI's practice "Conduct peer reviews of selected work products and identify issues resulting from these reviews" is the meeting. Its subpractices are: perform the assigned roles; "Identify and document defects and other issues in the work product"; "Record results of the peer review, including action items"; collect the review data; communicate issues to the people concerned; hold an additional review if needed; and "Ensure that the exit criteria for the peer review are satisfied."

Three rules keep the meeting productive. It finds and records issues; it does not solve them, which is the author's work afterwards (Fagan, in Chapter Twenty-Nine, on inspection). It is kept short, since Fagan found detection falls off after two hours. And its subject is the product: "The focus of the peer review should be on the work product in review, not on the person who produced it." When issues arise, "they should be communicated to the primary developer of the work product for correction."

The issues list and the review report

A formal review is defined by its documented output. Two documents come out of it.

• The issues list records each issue: an identifier, where in the product it is, a description, its severity (for example major, minor, or a question to be answered), and, after the meeting, its resolution. ISO/IEC 20246's definition is deliberately broad: an issue is any "observation that deviates from expectations", so a question the author must answer is an issue as much as a defect is.
• The review report records the review as a whole: the product and its size, the team and their roles, the preparation and meeting times, the number and kinds of issues, the action items, and the decision against the exit criteria (accept, accept once the issues are fixed and checked, or review again).

The report's data are CMMI's third practice: "Analyze data about the preparation, conduct, and results of the peer reviews." CMMI lists typical data: "product name, product size, composition of the peer review team, type of peer review, preparation time per reviewer, length of the review meeting, number of defects found, type and origin of defect". And it warns how they must not be used: "Examples of the inappropriate use of peer review data include using data to evaluate the performance of people and using data for attribution."

Worked example: analysing a review of the hall ticket module

In release 2.1 the hall ticket module was restructured, as the waiver on NC-04 required (Chapter Eighty-Six, on SQA activities), and given a formal technical review. Four people took part: a leader, a reader and a recorder, all reviewers, and the author. The program analyses the review's data the way CMMI asks: preparation against the expected rate, the meeting against its limit, the issues by severity, and the rework against the criterion for reviewing again. The expected rate and the limits are Fagan's, as Chapter Twenty-Nine, on inspection, gave them; treating a rate more than twice the expected one as not prepared is this book's rule.

# release 2.1's formal technical review of the hall ticket module (FINDINGS 5.8)
lines = 420
preparation = {"leader": 3.2, "reader": 3.5, "recorder": 1.0}          # hours, each alone
meeting_hours = 2.5
issues = [("I-1", "major"), ("I-2", "major"), ("I-3", "major"), ("I-4", "minor"), ("I-5", "minor"),
          ("I-6", "minor"), ("I-7", "minor"), ("I-8", "minor"), ("I-9", "question")]
reworked = 38

# the guidelines used to analyse the review (Fagan, as Chapter 29 quoted him)
EXPECTED_PREP = 125          # lines per hour for code preparation
MAX_SESSION = 2              # hours: no session longer than two hours
REINSPECT_OVER = 0.05        # reinspect if more than 5 per cent of the material was reworked

for role, hours in preparation.items():
    rate = lines / hours
    note = "   <- over twice the expected rate" if rate > 2 * EXPECTED_PREP else ""
    print(f"{role:<9} prepared {hours:.1f} h: {rate:>4.0f} lines per hour{note}")
verdict = "over the limit" if meeting_hours > MAX_SESSION else "within it"
print(f"meeting {meeting_hours} h: {verdict}")
for severity in ("major", "minor", "question"):
    print(f"{severity:<9}{sum(s == severity for _, s in issues):>2}")
share = reworked / lines
print(f"reworked {reworked} of {lines} lines, {share:.1%}:",
      "another review required" if share > REINSPECT_OVER else "moderator follow-up only")

leader    prepared 3.2 h:  131 lines per hour
reader    prepared 3.5 h:  120 lines per hour
recorder  prepared 1.0 h:  420 lines per hour   <- over twice the expected rate
meeting 2.5 h: over the limit
major     3
minor     5
question  1
reworked 38 of 420 lines, 9.0%: another review required

What the review found. Nine issues: three major (the hall ticket shown before the fee is confirmed, a status code the fee module never sends, an expired session treated as a paid one), five minor, and one question about the exam cell's rule for detained students, which goes to the exam cell as an action item.

What the data say about the review itself. The leader and reader prepared at 131 and 120 lines an hour, close to the expected 125. The recorder spent one hour on 420 lines, 420 an hour: at that speed the product was skimmed, not studied, and the recorder's contribution to finding issues was probably small. The meeting ran 2.5 hours, beyond the two hours after which Fagan found detection falls off. Neither observation is about blaming the recorder or the leader; both are process data, and the SQA group raises them as improvements to how reviews are scheduled (preparation time booked in advance, the module split into two sessions).

The decision. The rework changed 38 of the 420 lines, 9.0 per cent, above the 5 per cent at which Fagan's rule calls for the material to be reviewed again. The report's decision is therefore a second review after rework, not a follow-up check by the leader alone.

Guidelines for formal technical reviews

What it does not mean

A formal technical review is not a management review. CMMI says so directly: peer reviews "are structured and are not management reviews". Managers learn the outcome from the report.

Formal does not mean long. It means a defined process with documented output; a well-run technical review of a small product can take an hour.

The review does not fix the product. It finds and records issues; the author fixes them afterwards, and the leader or a second review checks the fixes.

Review data are not performance data. Using them to rate the author or the reviewers is, in CMMI's words, an inappropriate use, and it would stop people reporting honestly.

Quick revision

• Technical review (ISO/IEC 20246:2017): a formal peer review by technically qualified people of a product's suitability for its intended use and its discrepancies from specifications and standards. Formal review: a defined process with formal documented output.
• Objective (CMMI): identify defects for removal and recommend other changes needed.
• Prepare (CMMI SP 2.1): type; data to collect; entry and exit criteria; criteria for another review; checklists; schedule; entry check; early distribution; roles (leader, reader, recorder, author); individual preparation.
• Conduct (SP 2.2): roles performed; issues identified and documented; results and action items recorded; data collected; issues communicated; another review if needed; exit criteria met.
• Outputs: the issues list and the review report. Analyse (SP 2.3): preparation, conduct and results data; never to evaluate people.
• Guidelines (CMMI): sufficient preparation; managed and controlled conduct; consistent and sufficient data; action items recorded; focus on the product, not the person.
• Worked example: 9 issues (3 major); the recorder prepared at 420 lines an hour against 125 expected; a 2.5-hour meeting; 9.0 per cent reworked, so a second review.

Test yourself

1. What is a formal technical review, and what are its objectives? A formal peer review of a work product by technically qualified people, following a defined process with documented output, that examines the product's suitability for its intended use and identifies discrepancies from its specifications and standards. Its objectives are to identify defects for removal and to recommend other changes that are needed.

2. Describe how a formal technical review is prepared. The type of review is chosen; the data to be collected, entry and exit criteria and criteria for another review are set; checklists are prepared; the review is scheduled; the product is checked against the entry criteria and distributed early; roles (leader, reader, recorder, author) are assigned; and each reviewer studies the product before the meeting.

3. What happens in the review meeting, and what does it produce? The participants perform their roles, the reader leads through the product, and the issues found are identified and recorded, not solved. The review produces an issues list, with each issue's location, description and severity, and a review report recording the product, the team, the times, the issues, the action items and the decision against the exit criteria.

4. State the guidelines for conducting formal technical reviews. Review the product, not the producer; prepare sufficiently, with the product distributed in time; manage and control the conduct, with roles and a time limit; find issues rather than solving them; use checklists; record consistent data and every action item; set entry, exit and re-review criteria; and never use review data to judge people.

5. What data should a review record, and how may they not be used? The product and its size, the team, the type of review, each reviewer's preparation time, the length of the meeting, and the number, types and origins of the defects found. They must not be used to evaluate people's performance or to attribute blame.

6. In the worked example, why was a second review required, and what else did the data show? Because the rework changed 9.0 per cent of the module, above the 5 per cent at which the material is reviewed again. The data also showed that one reviewer prepared at 420 lines an hour, more than three times the expected rate, and that the meeting ran beyond two hours: both are process improvements for how reviews are scheduled, not faults of the people.