SQA Activities: What the SQA Group Does

Criteria. Every evaluation works from criteria stated in advance. CMMI's subpractice lists what they settle: what will be evaluated, when or how often, how the evaluation will be conducted, and who must be involved. For ExamReg these are checklists, one for each process, built from the project's own standards.

The activities

Taking part in planning

SQA does not start with the first audit. CMMI says that "Quality assurance should begin in the early phases of a project to establish plans, processes, standards, and procedures that will add value to the project", and that those who do quality assurance take part in setting them up "to ensure that they fit project needs and that they will be usable for performing quality assurance evaluations." The processes and work products to be evaluated are chosen at this stage too. The result is the software quality assurance plan, the subject of Chapter Eighty-Seven.

Evaluating processes

The SQA group checks that the processes the project performs match their descriptions, standards and procedures: that a requirements review was held as the procedure says, with the people it names; that builds are tagged in version control; that defects are retested before they are closed. CMMI lists ways to do it, from formal to informal:

• "Formal audits by organizationally separate quality assurance organizations"
• "Peer reviews, which can be performed at various levels of formality"
• "In-depth review of work at the place it is performed (i.e., desk audits)"
• "Distributed review and comment of work products"
• Process checks built into the processes themselves, such as a fail-safe that stops a process done incorrectly (CMMI's example is Poka-Yoke).

An audit, in CMMI's glossary, is "An objective examination of a work product or set of work products against specific criteria (e.g., requirements)", and the term covers process compliance audits as well as configuration audits.

Evaluating work products

The SQA group also checks selected work products (a requirements specification, a test plan, a release package) against the standards and procedures that apply to them, choosing them by documented sampling criteria when not everything can be checked. CMMI's examples of when to evaluate them: "Before delivery to the customer", "During delivery to the customer", "Incrementally, when it is appropriate", "During unit testing", "During integration" and "When demonstrating an increment".

Handling noncompliance

A noncompliance issue is a problem "identified in evaluations that reflect a lack of adherence to applicable standards, process descriptions, or procedures." Every one is identified and recorded. CMMI then sets the order of resolution. First, resolve it in the project, with the people concerned. Its examples of ways to do that are "Fixing the noncompliance", "Changing the process descriptions, standards, or procedures that were violated" and "Obtaining a waiver to cover the noncompliance". Only if the project cannot resolve it, escalate: "When noncompliance issues cannot be resolved in the project, use established escalation mechanisms to ensure that the appropriate level of management can resolve the issue." Either way, "Track noncompliance issues to resolution."

The second way deserves attention. A noncompliance does not always mean the people were wrong: sometimes the process description is, and the right resolution is to change it.

Keeping records and reporting

The group keeps "records of quality assurance activities" in enough detail that "status and results are known": evaluation logs, quality assurance reports, the status of corrective actions and reports of quality trends. It makes sure the people concerned hear the results in time, and it periodically reviews open noncompliance issues and trends with the manager designated to act on them.

Learning

Every evaluation also looks for "lessons learned that could improve processes", and the group analyses its noncompliance issues "to see if there are quality trends that can be identified and addressed". This is where SQA feeds the improvement loop of Chapter Eighty, on using defect data to improve the process.

Public standards phrase the same work as tasks. NASA's software assurance standard, for example, pairs each engineering requirement with assurance tasks that begin "Confirm that" or "Assess": to "Confirm that all plans, including security plans, are in place and have expected content", or to "Assess plans for compliance". The verbs are the point. The assurance function does not write the plans or the code; it confirms and assesses them.

Worked example: release 2.0's SQA audits

ExamReg's software house has a small SQA group, independent of the project team and reporting to the head of delivery. During release 2.0 it held eight audits, one for each process in the SQA plan, each against a checklist. The program reads the audit log and produces the measures the group reported to management.

from statistics import mean, median

# release 2.0's SQA audits, in the order held (FINDINGS 5.4): (audit, items checked)
audits = [("requirements review", 12), ("design review", 10), ("coding standard", 15),
          ("unit testing", 12), ("configuration management", 10), ("defect tracking", 10),
          ("test reporting", 8), ("release readiness", 12)]
# each noncompliance: (id, audit, how it was resolved, days from report to resolution)
issues = [("NC-01", "requirements review", "fixed", 5), ("NC-02", "requirements review", "fixed", 3),
          ("NC-03", "design review", "fixed", 1), ("NC-04", "coding standard", "waiver", 7),
          ("NC-05", "coding standard", "fixed", 4), ("NC-06", "coding standard", "fixed", 2),
          ("NC-07", "unit testing", "fixed", 6), ("NC-08", "unit testing", "process changed", 10),
          ("NC-09", "configuration management", "fixed", 1),
          ("NC-10", "configuration management", "escalated", 21),
          ("NC-11", "defect tracking", "fixed", 3), ("NC-12", "test reporting", "fixed", 2),
          ("NC-13", "release readiness", "fixed", 1)]

print(f"{'audit':<26}{'items':>6}{'noncompliant':>14}{'compliance':>12}")
for name, items in audits:
    nc = sum(1 for _, audit, _, _ in issues if audit == name)
    print(f"{name:<26}{items:>6}{nc:>14}{(items - nc) / items:>12.0%}")
total = sum(items for _, items in audits)
print(f"{'all audits':<26}{total:>6}{len(issues):>14}{(total - len(issues)) / total:>12.1%}")

ways = {"fixed": "fixed in the project", "process changed": "process description changed",
        "waiver": "waiver granted", "escalated": "escalated to management"}
for how, label in ways.items():
    ids = [i for i, _, resolved, _ in issues if resolved == how]
    print(f"{label:<28}{len(ids):>3}  {' '.join(ids)}")
days = [d for *_, d in issues]
print(f"days to resolve: mean {mean(days):.1f}, median {median(days)}, longest {max(days)}")
print("over 14 days, for the manager's review:", " ".join(i for i, *_, d in issues if d > 14))

audit                      items  noncompliant  compliance
requirements review           12             2         83%
design review                 10             1         90%
coding standard               15             3         80%
unit testing                  12             2         83%
configuration management      10             2         80%
defect tracking               10             1         90%
test reporting                 8             1         88%
release readiness             12             1         92%
all audits                    89            13       85.4%
fixed in the project         10  NC-01 NC-02 NC-03 NC-05 NC-06 NC-07 NC-09 NC-11 NC-12 NC-13
process description changed   1  NC-08
waiver granted                1  NC-04
escalated to management       1  NC-10
days to resolve: mean 5.1, median 3, longest 21
over 14 days, for the manager's review: NC-10

Compliance. Of 89 checklist items, 13 were found out of compliance, 85.4 per cent compliance overall. The coding standard and configuration management audits were the weakest at 80 per cent, and the release readiness audit the best at 92. The numbers are only as meaningful as the checklists: an audit of 8 items and one of 15 are not the same test, and a trend is read across releases, audit by audit, not across different audits.

Resolution. Ten noncompliances were fixed in the project, most within a few days, as CMMI expects. Three show the other paths:

• NC-08, process description changed. The unit testing standard asked for a coverage report for every module on every build, and nobody read them. The right resolution was to change the standard, to one report per release, not to force the team to produce reports nobody used.
• NC-04, waiver granted. The coding standard sets a limit of 10 on cyclomatic complexity, and hall_ticket_status measured 15 (Chapter Seventy, on cyclomatic complexity). Splitting it days before release would have risked new defects, so a waiver was granted with a condition: split it in release 2.1. A waiver is a recorded decision, not an oversight.
• NC-10, escalated to management. The database scripts were kept outside version control. The project lead declined to move them before release; the SQA group could not resolve it in the project, so it used its independent reporting channel and escalated to the head of delivery, who ordered the move. It took 21 days, the only issue over the 14-day threshold at which the group reviews open issues with the manager.

What the measures are for. The mean time to resolve, 5.1 days, is pulled up by the one escalation; the median, 3 days, describes the typical issue better, as Chapter Sixty-Eight, on quality, process and test metrics, showed for change times. And none of these numbers measures ExamReg's quality directly. They measure whether the project did what its plan said, which is exactly what process and product quality assurance is for; the product's quality is measured by the defect metrics of Chapter Seventy-Nine.

What it does not mean

The SQA group is not the test team. Testing checks the product against its requirements; SQA checks that the planned processes, testing among them, were followed.

An audit is not a hunt for culprits. Its findings are about adherence to processes, and one of CMMI's own resolutions is to change the process when it is the thing that is wrong.

Escalation is not the first step. Noncompliance is resolved in the project wherever possible; escalation is for what the project cannot resolve.

A waiver is not an exception quietly made. It is a decision recorded with its reason and its conditions, and tracked like any other issue.

High compliance is not high quality. A project can follow a weak process faithfully; compliance measures adherence, and the process itself must be judged by its results.

Quick revision

• PPQA purpose (CMMI): "to provide staff and management with objective insight into processes and associated work products".
• Against verification: PPQA ensures "planned processes are implemented"; verification that "specified requirements are satisfied".
• Objectivity: "by both independence and the use of criteria"; an independent group, or QA embedded in the process with training, separation from the work product's authors, and an independent reporting channel.
• Activities: take part in planning; objectively evaluate processes (SP 1.1) and work products (SP 1.2); communicate and resolve noncompliance issues (SP 2.1): fix, change the process description, waiver, or escalate; establish records (SP 2.2); analyse trends and lessons learned.
• Evaluation methods: formal audits, peer reviews, desk audits, distributed review, built-in process checks.
• Worked example: 8 audits, 89 items, 13 noncompliances (85.4 per cent compliance); 10 fixed, 1 process changed, 1 waiver, 1 escalated (21 days); mean 5.1 days, median 3.

Test yourself

1. What is the purpose of software quality assurance as CMMI states it, and how does it differ from verification? To provide staff and management with objective insight into processes and associated work products. SQA ensures that the planned processes are implemented; verification ensures that specified requirements are satisfied. Both may examine the same work product, from different perspectives.

2. List the activities of an SQA group. Taking part in establishing the project's plans, processes, standards and procedures; objectively evaluating performed processes against their descriptions, standards and procedures; objectively evaluating selected work products; identifying, documenting and communicating noncompliance issues and ensuring they are resolved, escalating those the project cannot resolve; keeping records of quality assurance activities; and analysing trends and lessons learned to improve processes.

3. How is objectivity achieved in SQA evaluations? By independence and by the use of criteria: evaluations are done against stated criteria by people who did not produce the work product, traditionally a quality assurance group independent of the project. Where QA is embedded in the process, its people are trained, separate from the work's authors, and have an independent reporting channel for escalation.

4. What is a noncompliance issue, and how is it resolved? A problem found in an evaluation that reflects a lack of adherence to applicable standards, process descriptions or procedures. It is resolved in the project if possible, by fixing the noncompliance, changing the process description that was violated, or obtaining a waiver; if the project cannot resolve it, it is escalated to the designated level of management; in every case it is tracked to resolution.

5. Give four ways of performing objective evaluations. Formal audits by an organisationally separate quality assurance group; peer reviews at various levels of formality; desk audits, reviewing work where it is performed; distributed review and comment of work products; and process checks built into the processes themselves.

6. In the worked example, why was NC-10 escalated, and what do the resolution times show? The project lead declined to put the database scripts under version control before release, so the SQA group could not resolve the issue in the project and escalated it to the head of delivery, who ordered it; it took 21 days. Most issues were resolved in a few days: the median was 3 days, while the mean of 5.1 days was pulled up by the escalation.