The Challenges in Software Quality Assurance

Changing requirements

Requirements change, and modern methods treat that as normal: the Agile Manifesto's principles say "Welcome changing requirements, even late in development." Each change is also a challenge to assurance. A requirement that changes after the code is written reopens the design, the code, the tests written against the old version and every result that depended on them. ExamReg met this in miniature when the exam cell ruled that a negative number of days must be refused: the ruling arrived after the fee code was written (Chapter Eighty, on using defect data to improve the process, traced a class of defects back to it).

The answers are traceability and discipline, not resistance to change. If every requirement is linked to its design, code and tests, a change shows at once which tests must be revised and rerun (Chapter Seventy-Seven, on tracking defects to closure, used the same links to see which requirements still had open defects). Changes go through a change control board that weighs their cost; and regression testing (Chapter Thirty-Nine, on regression and smoke testing) confirms that what was not meant to change did not.

Schedule pressure

Release dates rarely move, and the activities at the end of a project, system and acceptance testing, absorb every delay before them. Schedule pressure also causes defects in the first place: the ISTQB syllabus lists time pressure first among the reasons "Humans make errors". The program below measures what squeezing the test phase would have cost release 2.0. It takes the ten test weeks of Chapter Seventy-Seven, on tracking defects to closure, and asks, for each week the testing might have been stopped at, how many defects would have gone into the release: those that testing found only in the later weeks, and those found but still open.

from itertools import accumulate

# release 2.0's ten test weeks (FINDINGS 5.2.5): defects found and defects closed each week
found = [10, 16, 18, 17, 14, 12, 10, 8, 5, 4]
closed = [4, 10, 14, 16, 16, 15, 13, 11, 8, 5]
found_by, closed_by = list(accumulate(found)), list(accumulate(closed))

print(f"{'testing stops':>13} {'found':>9} {'found only in':>15} {'found but':>11} {'left in the':>12}")
print(f"{'after week':>13} {'so far':>9} {'later weeks':>15} {'still open':>11} {'release':>12}")
for week in range(4, 11):
    later = found_by[-1] - found_by[week - 1]
    still_open = found_by[week - 1] - closed_by[week - 1]
    print(f"{week:>13} {found_by[week - 1]:>9} {later:>15} {still_open:>11} {later + still_open:>12}")

testing stops     found   found only in   found but  left in the
   after week    so far     later weeks  still open      release
            4        61              53          17           70
            5        75              39          15           54
            6        87              27          12           39
            7        97              17           9           26
            8       105               9           6           15
            9       110               4           3            7
           10       114               0           2            2

With the full ten weeks, release 2.0 shipped with 2 known, accepted defects. Had the release date cut testing to six weeks, the 27 defects that testing found only in weeks 7 to 10 would have stayed in the product unfound, and the 12 open at the end of week 6 would have shipped with them: 39 defects instead of 2. Cutting to four weeks would have left 70. Every week counts: dropping only the last week would have shipped 7 defects instead of 2, and each earlier week dropped adds more, from 8 to 16.

Two cautions keep the reading honest. Some defects found late may have been introduced by fixes made during testing, and would not have existed had testing stopped earlier; the data cannot separate them, so the counts are an upper bound. And the 12 defects that students found after release are in none of these columns: they escaped the full ten weeks, and no length of this testing would have caught them. What the table does show is the challenge: when the schedule decides the length of testing, it also decides how many defects are shipped, and SQA's task is to make that trade visible to the people who set the date, with numbers like these.

Measuring quality

Quality is hard to assure partly because it is hard to measure. It is not one quantity: ISO/IEC 25010:2023 describes product quality with nine characteristics (Chapter Twenty, on the quality model today), and a release can improve in one and decline in another. The counts themselves depend on definitions. Chapter Seventy-Three, on what a defect is, gave four honest answers to how many defects did release 2.0 have?: 250, 227, 209 or 200, depending on what was counted. And some measures invite misuse: Chapter Sixty-Five, on what a software metric is, showed that averaging severity codes can reverse a comparison, because the codes are ordinal.

The challenge for SQA is to measure without misleading: to define each measure before collecting it, as the goal-question-metric approach of Chapter Sixty-Six asks, to use measures that fit the scale of the data, and to read any single number, such as a defect count or a pass rate, beside the others.

People and culture

Assurance is done by people to the work of other people, and that makes it delicate. The ISTQB syllabus is plain about it: "Testers are often the bearers of bad news. It is a common human trait to blame the bearer of bad news." Test results "may be perceived as criticism of the product and of its author", and "Confirmation bias can make it difficult to accept information that disagrees with currently held beliefs." An SQA group that audits a project's process meets the same reaction more strongly, because its findings go to management.

Three answers recur in the sources. Communicate constructively: the syllabus asks that "information about defects and failures should be communicated in a constructive way." Remove fear: Deming's eighth point, "Drive out fear", because people who fear blame hide defects instead of reporting them. And share the responsibility: in the whole team approach "everyone is responsible for quality", so a defect report is information for the team, not an accusation. Independence, which SQA needs for its findings to be trusted, has to be combined with this; Chapter Twenty-Five, on quality management and SQA, described its three kinds.

Third-party and reused code

Every product contains code its team did not write: at the least a language's libraries and an operating system, and usually frameworks, services and components carried over from earlier systems. Brooks's conformity applies to all of it, and assurance has less to work with, since the team may not see the code's history, its tests or even its source.

Two cases in this book show the two sides of the risk. Ariane 5's inertial reference system reused software from Ariane 4, where it had worked; on Ariane 5's faster trajectory it failed, and the Inquiry Board found that the reused software had not been checked against the new rocket's flight conditions (Chapter Three, on why software must be tested). And a library can stop being maintained while applications still depend on it: Apache's page for Log4j 1 records that it reached end of life on 5 August 2015, and that "Since Log4j 1 is no longer maintained none of the issues listed will be fixed" (Chapter Nine, on test execution).

The assurance answers follow from the challenge. Know what the product contains: an inventory of every third-party component and its version. Know each component's support status, and plan to replace what is no longer maintained. Re-verify reused code against the new system's requirements and conditions, not the old one's. And test at the seams, where the product meets what it did not build.

The challenges at a glance

What it does not mean

The challenges are not reasons to give up assurance. They explain why assurance gives justified confidence, never certainty, and why it must be planned rather than improvised.

Changing requirements are not a defect of the customer. They are normal; the challenge is to change the product under control, with traceability and regression testing.

A shorter test phase does not save its cost. In release 2.0's data, every week cut leaves defects in the product, to be found later by users, when fixing them costs more (Chapter Ninety-Six, on why reviews pay, gives the evidence on the cost of a late defect).

Third-party code is not someone else's quality problem. Once shipped in the product, its defects are the product's defects.

Quick revision

• Brooks's essential difficulties for assurance: complexity (confidence never complete), conformity (depends on others' interfaces), changeability (assurance is per version), invisibility (evidence, not inspection).
• Changing requirements: "Welcome changing requirements, even late in development" (Agile Manifesto); answers: traceability, change control, regression testing.
• Schedule pressure: time pressure causes errors (ISTQB); cutting release 2.0's testing from ten weeks to six would have shipped 39 defects instead of 2; to four, 70.
• Measurement: nine characteristics (ISO/IEC 25010:2023); 250, 227, 209 or 200 defects by definition; ordinal data.
• People and culture: testers are "the bearers of bad news"; confirmation bias; constructive communication; drive out fear; the whole team approach.
• Third-party and reused code: Ariane 5's reused software; Log4j 1's end of life (2015); inventory, support status, re-verification, testing at the seams.

Test yourself

1. How do Brooks's four essential difficulties make software quality assurance hard? Complexity means no set of checks covers every state, so confidence is never complete; conformity means quality depends on interfaces and rules the team did not design; changeability means any assurance holds for one version only and a change can undo it; invisibility means software cannot be inspected by looking, so assurance must rest on evidence such as reviews, records, measures and test results.

2. Why are changing requirements a challenge for SQA, and how is it met? Each change after work has been done reopens the design, code, tests and results that depended on the old requirement. It is met with traceability from requirements to design, code and tests, so the impact of a change is known; with change control that weighs each change; and with regression testing to confirm that nothing else changed.

3. How does schedule pressure affect software quality? Use release 2.0's data. Testing at the end absorbs earlier delays, and people make more errors under time pressure. Had release 2.0's testing been cut from ten weeks to six, the 27 defects found only in weeks 7 to 10 and the 12 still open at the end of week 6 would have shipped, 39 defects instead of 2.

4. Why is software quality hard to measure? Quality has many characteristics, nine in ISO/IEC 25010:2023, which can move in different directions; even simple counts depend on definitions, as release 2.0's defects numbered 250, 227, 209 or 200 by four definitions; and some data, such as severity codes, is ordinal and cannot be averaged meaningfully.

5. What people and cultural challenges does SQA face, and how are they addressed? Testers and SQA staff bring bad news, which people tend to blame on the bearer; results can be felt as criticism, and confirmation bias resists unwelcome information. They are addressed by communicating defects constructively, removing fear of blame so that people report problems, and sharing responsibility for quality across the whole team.

6. What is the challenge of third-party and reused code, and what can SQA do about it? The team did not write it and may not see its tests or source, it may stop being maintained, and it may be used in conditions it was not built for, as Ariane 5's reused software was. SQA keeps an inventory of components and versions, tracks their support status, re-verifies reused code against the new system's requirements and tests the interfaces where the product meets it.