Background Issues in Software Quality Assurance

# figures quoted at the 1968 NATO conference (the report, section 2), carried ten years forward
computers = 10_000                          # installed in Europe in 1968 (Helms)
for rate in (0.25, 0.50):                   # "from 25 per cent to 50 per cent per year"
    print(f"computers growing {rate:.0%} a year: {computers * (1 + rate) ** 10:,.0f} by 1978")

# d'Agapeyeff: fewer than 50 programmers at a manufacturer in 1958, 1,000 to 2,000 in 1968
for now in (1_000, 2_000):
    factor = now / 50
    rate = factor ** (1 / 10) - 1
    print(f"50 to {now:,} programmers in ten years: {factor:.0f} times, {rate:.1%} a year;"
          f" at that rate, {now * factor:,.0f} by 1978")

computers growing 25% a year: 93,132 by 1978
computers growing 50% a year: 576,650 by 1978
50 to 1,000 programmers in ten years: 20 times, 34.9% a year; at that rate, 20,000 by 1978
50 to 2,000 programmers in ten years: 40 times, 44.6% a year; at that rate, 80,000 by 1978

Ten years at Helms's rates would take Europe's computers from 10,000 to somewhere between about 93,000 and 577,000. D'Agapeyeff's figures mean the programming staff of one manufacturer grew at least 20 to 40 times in a decade, between 35 and 45 per cent a year, since the starting figure was "less than 50"; another decade at those rates would need 20,000 to 80,000 people. No discipline, testing included, could train people and invent methods that fast, and the report says the growth "was viewed with more alarm than pride."

Failure. The alarm was about what failures could do. David and Fraser wrote: "Particularly alarming is the seemingly unavoidable fallibility of large software, since a malfunction in an advanced hardware-software system can be a matter of life and death". Dijkstra put the risk in one line: "the massive dissemination of error-loaded software is frightening." Kinslow described IBM's OS/360 and TSS/360 as "straight-through, start-to-finish, no-test-development, revolutions", and added: "I have never seen an engineer build a bridge of unprecedented span, with brand new materials, for a kind of traffic never seen before".

The software crisis. Some participants called the situation a software crisis, and the name itself was argued over. Kolence did not like "the use of the word 'crisis'"; for him the problem was that "certain classes of systems are placing demands on us which are beyond our capabilities", and "It is large systems that are encountering great difficulties." Hastings, running large installations, found users "reasonably satisfied". Dijkstra welcomed the debate itself: "the admission of shortcomings is the primary condition for improvement."

Quality cannot be added afterwards

One remark at the conference states the principle this whole module rests on. Discussing whether design could be separated from production, Dijkstra said: "I am convinced that the quality of the product can never be established afterwards. Whether the correctness of a piece of software can be guaranteed or not depends greatly on the structure of the thing made." It is Deming's third point, to cease dependence on inspection (Chapter Eighty-Two, on Shewhart, Deming and Juran), arrived at independently for software. Testing at the end can find defects; it cannot put quality in. That is why quality assurance is about the process that makes the software, from its first requirement.

1968: quality assurance as a separate function

The conference's working papers include R.W. Bemer's Checklist for planning software system production, and its ninth section is headed Quality Assurance. Its questions, written as a manager's checklist, contain the core of SQA as later standards define it.

The ratio of one in ten is Bemer's question, not a rule any later standard sets; its point is that assurance needs people of its own, planned from the start.

Military and government origins

Much of the early formal work on software quality was paid for by governments that bought large software systems. McCall, Richards and Walters's Factors in Software Quality, the model of Chapter Nineteen, on McCall's quality factors, was a 1977 technical report (RADC-TR-77-369) written by General Electric for the Rome Air Development Center of the U.S. Air Force Systems Command. The U.S. Department of Defense set standards for how its contractors developed software: Boehm noted in 1988 that its standard on software management, DoD-Std-2167, "requires that developers produce and use risk management plans", and the SEI's 1992 measurement reports cite DOD-STD-2167A, Military Standard, Defense System Software Development (1988). Space agencies keep standards of their own: NASA's software assurance standard, NASA-STD-8739.8B, was approved in September 2022.

The pattern explains a feature of SQA that surprises students: its vocabulary of plans, audits, reviews and records comes from buyers who could not inspect software themselves and needed evidence that it had been built properly. That is what assurance means: "grounds for justified confidence that a claim has been or will be achieved" (ISO/IEC/IEEE 15026-1:2025, Chapter Twenty-Four).

The IEEE and ISO standards

The civilian standards followed. IEEE 730, IEEE Standard for Software Quality Assurance Processes, establishes requirements "for initiating, planning, controlling, and executing" the SQA processes of a software development or maintenance project (IEEE SA's page). Its 2014 edition superseded 730-2002, and in 2026 it was itself superseded by IEEE 730-2026, published on 21 August 2026 and "harmonized with the software life cycle processes of ISO/IEC/IEEE 12207:2017". The definition of SQA that SEVOCAB still carries is the 2014 edition's: the "set of activities that define and assess the adequacy of software processes to provide evidence that establishes confidence that the software processes are appropriate for and produce software products of suitable quality for their intended purposes". NASA's standard adopts the same definition, and adds: "A key attribute of software assurance is the objectivity of the software assurance function with respect to the project."

Two further standards frame the rest of this module. ISO/IEC/IEEE 12207, now in its 2026 edition, places quality assurance among the life cycle processes (Chapter Twenty-Four gave its definition), and ISO 9001, now ISO 9001:2026, is the quality management standard an organisation can be certified against (Chapters Ninety-Four and Ninety-Five, on the ISO 9000 family and ISO 9001). A textbook written before 2026 will cite older editions of all of these; the definitions this book quotes are the current ones, and it says where an older one is used.

Who is responsible for quality?

The history gives two answers, and both are right.

Everyone. Quality cannot be added afterwards, so it is made by everyone whose work goes into the product. The ISTQB syllabus says that QA "is the responsibility of everyone on a project", and Deming's fourteenth point says "The transformation is everybody's job."

And an assurance function that is objective. Bemer wanted an agency that reports independently and speaks for the user; IEEE 730-2014 defines SQA independence as freedom "from technical, managerial, and financial influences"; NASA makes objectivity "a key attribute". The assurance function does not replace the others' responsibility. It checks that the process is being followed and says so to people who can act. NASA's standard also names the condition for any of it to work: "Project and SMA Management support of the software assurance function is essential".

On ExamReg's project the division looks like this. It is the book's own, and Chapter Eighty-Six, on SQA activities, describes the SQA group's side in full.

What it does not mean

The software crisis was not the whole field failing. Participants at the conference itself disagreed; Kolence placed the difficulty in large systems, and Buxton said most computers worked tolerably well. The concern was about scale and criticality.

Quality assurance is not the SQA group's job alone. The group gives independent assurance; the quality itself is made by everyone on the project.

Independence is not isolation. An SQA function reports separately so that its findings reach people who can act; it still works with the project throughout.

Standards are not the history's end. Each edition replaces the one before; IEEE 730 alone has had editions in 2002, 2014 and 2026.

Quick revision

• NATO 1968 (Garmisch, October): "software engineering" chosen as provocative; growth of 25 to 50 per cent a year in computers (Helms) and 20 to 40 times in programmers in a decade (d'Agapeyeff); failures "a matter of life and death"; the software crisis debated.
• Dijkstra: "the quality of the product can never be established afterwards"; "the admission of shortcomings is the primary condition for improvement".
• Bemer's checklist, section 9: QA distinct from QC; independent, speaking for the user; usefulness as well as the specification; tests built concurrently; one in ten; the test library rerun on each modification.
• Military and government: McCall's factors for the U.S. Air Force (1977); DOD-STD-2167A (1988); NASA-STD-8739.8B (2022).
• IEEE 730: SQA processes; 2002, 2014, now 2026; SQA definition (2014 edition, through SEVOCAB).
• Responsibility: everyone on the project makes quality; an objective SQA function assures it; management support is essential.
• Program: the conference's figures carried forward: 93,132 to 576,650 computers; 20,000 to 80,000 programmers per manufacturer by 1978 at the same rates.

Test yourself

1. What were the main background issues that gave rise to software quality assurance? The rapid growth of software in size and number, faster than people and methods could keep up; the seriousness of failures in large and critical systems; the conviction that quality cannot be added after the product is built; and buyers, especially governments, who needed evidence that software had been built properly. The 1968 NATO conference brought these together and named software engineering.

2. Why was the phrase software engineering chosen for the 1968 conference? It was deliberately provocative: it implied that software manufacture should be based on the theoretical foundations and practical disciplines traditional in the established branches of engineering, which at the time it was not.

3. What was the software crisis? Did everyone at the conference agree there was one? The name some participants gave to the gap between what large software systems were expected to do and what could be built reliably, on time and within cost. They did not agree: Kolence disliked the word and placed the difficulty in large systems; Hastings found users reasonably satisfied; others, such as David and Fraser, stressed that failures could be a matter of life and death.

4. What did Bemer's 1968 checklist say about quality assurance? That QA is distinct from the continuous quality control of production; that it should be done by an independently reporting agency representing the user; that the product should be tested for usefulness as well as against its specification; that QA tests should be built concurrently with the software; that there should be at least one QA person for every ten in production; and that the test library should be rerun on every modification.

5. Who is responsible for software quality? Everyone on the project: analysts, developers, testers, managers and the customer each make part of it, since quality cannot be added afterwards. An SQA function, objective and independent of the project, gives assurance that the process is followed, and management must support it.

6. Name a military and an IEEE standard in the history of software quality. DOD-STD-2167A, Military Standard, Defense System Software Development (1988), and IEEE 730, the IEEE Standard for Software Quality Assurance Processes, whose current edition is IEEE 730-2026.