What a Defect Is

Not every reported problem is a defect

Florac's report for the SEI's Software Metrics Definition Working Group is plain about it: "Problems are unsatisfactory encounters with the software; consequently, some of the problems reported are not software failures or software defects, and may not even be software product problems." It sorts every problem into one of three subtypes, and each subtype into values:

• Software defects, by the work product that contains them: a requirements defect, "A mistake made in the definition or specification of the customer needs for a software product"; a design defect; a code defect; a document defect; a test case defect, where "A mistake in the test case causes the software product to give an unexpected result"; and a defect in another work product, such as a test tool or a configuration library.
• Other problems, with no evidence of a software defect: a hardware problem, an operating system problem, a user mistake, an operations mistake, or a new requirement or enhancement outside the agreed requirements.
• Undetermined: not repeatable or cause unknown, or not yet evaluated.

A separate attribute, uniqueness, marks a report as original or duplicate; a duplicate is one where "The problem or defect has been previously discovered."

Two consequences follow for a tester. A failed test is not automatically a product defect: the test case itself may be wrong, which is Florac's test case defect and the false positive of Chapter Two, on errors, faults and failures. And a defect need not be in code: requirements, designs and documents are work products, and their defects count.

The attributes a defect carries

Florac's framework asks "who, what, why, when, where, and how" of every problem, and turns the answers into attributes with defined values: "In developing the attribute values, we are careful to ensure they are mutually exclusive; that is, any given problem may have one, and only one, value for each attribute." The main ones, filled in for DR-311, the major defect ExamReg's acceptance testing found (Chapter Forty-Two, on acceptance testing):

Two of the attributes look alike and are not. Criticality "is a measure of the disruption a problem gives users when they encounter it", and is normally given by whoever reported it. Urgency "is the degree of importance that the evaluation, resolution, and closure of a problem is given by the organization charged with executing the problem management process", assigned by the team that fixes it. They are the severity and the priority that Chapter Seventy-Six, on writing a defect report, keeps apart.

Worked example: how many defects did release 2.0 have?

Release 2.0 of ExamReg drew 250 problem reports, from reviewers, testers and, after release, students. Each was investigated and classified by problem type and uniqueness. The program counts them under Florac's subtypes, and then answers one simple question under four definitions.

# ExamReg release 2.0's problem reports, classified by Florac's attributes (FINDINGS 5.2.3):
# (problem type, uniqueness) -> number of reports
reports = {("requirements defect", "original"): 28, ("design defect", "original"): 40,
           ("code defect", "original"): 120, ("operational document defect", "original"): 12,
           ("test case defect", "original"): 9, ("code defect", "duplicate"): 18,
           ("user mistake", "original"): 7, ("operations mistake", "original"): 3,
           ("new requirement/enhancement", "original"): 6,
           ("not repeatable/cause unknown", "value not identified"): 5,
           ("value not identified", "value not identified"): 2}

PRODUCT = {"requirements defect", "design defect", "code defect", "operational document defect"}
SOFTWARE = PRODUCT | {"test case defect", "other work product defect"}
UNDETERMINED = {"not repeatable/cause unknown", "value not identified"}

def subtype(problem_type):
    return ("software defect" if problem_type in SOFTWARE
            else "undetermined" if problem_type in UNDETERMINED else "other problem")

def count(rule):
    return sum(n for key, n in reports.items() if rule(*key))

print("by Florac's subtypes:")
for s in ("software defect", "other problem", "undetermined"):
    print(f"   {s:<16} {count(lambda t, u: subtype(t) == s):>4}")

print("'how many defects did release 2.0 have?', by four definitions:")
definitions = {
    "every problem report": lambda t, u: True,
    "software defects, originals and duplicates": lambda t, u: t in SOFTWARE,
    "original software defects (product and tests)": lambda t, u: t in SOFTWARE and u == "original",
    "original defects in the product itself": lambda t, u: t in PRODUCT and u == "original",
}
for name, rule in definitions.items():
    print(f"   {name:<47} {count(rule):>4}")

print("the product's defects by the work product they were in:")
for t in sorted(PRODUCT, key=lambda t: -reports[(t, "original")]):
    print(f"   {t:<30} {reports[(t, 'original')]:>4}")

by Florac's subtypes:
   software defect   227
   other problem      16
   undetermined        7
'how many defects did release 2.0 have?', by four definitions:
   every problem report                             250
   software defects, originals and duplicates       227
   original software defects (product and tests)    209
   original defects in the product itself           200
the product's defects by the work product they were in:
   code defect                     120
   design defect                    40
   requirements defect              28
   operational document defect      12

Four honest answers, from 250 down to 200, to the same question. A report that says release 2.0 had 250 defects counts reports, including duplicates, users' mistakes and enhancement requests; one that says 200 counts original defects in the product, which is the figure this book has used since Chapter Twenty-Four, on quality control and quality assurance, and the one every defect density in Module 2 divides. The difference is not a matter of accuracy but of definition, exactly the problem Park found with lines of code (Chapter Sixty-Seven, on size metrics). Florac's report exists to fix it the same way: with attributes whose values are exclusive, and a checklist that states which values a count includes.

The last lines show where the product's defects were: 120 in code, but 80 in requirements, designs and documents together. Two defects in five were not in the code at all, which is why reviews of requirements and designs (Chapters Ninety-Six to Ninety-Eight) are part of finding defects.

What it does not mean

A problem report is not a defect. It is a claim that something is wrong, which investigation confirms or rejects.

A failed test is not always a product defect. The test case may be wrong, and Florac counts that as a test case defect.

A defect is not only in code. Requirements, designs, documents and tests are work products with defects of their own.

Criticality is not urgency. One is the disruption to users, reported by the originator; the other is the order of work, set by the team.

Quick revision

• Defect: "imperfection or deficiency in a work product where that work product does not meet its requirements or specifications and needs to be either repaired or replaced" (ISO/IEC 23531).
• Anomaly (IEEE 1012-2024), incident and problem (ISO/IEC/IEEE 12207:2026), incident report (ISO/IEC/IEEE 29119-2): the stages before a defect is confirmed.
• Florac (SEI 1992): problems are "unsatisfactory encounters with the software"; subtypes software defect (requirements, design, code, document, test case, other work product), other problem (hardware, operating system, user mistake, operations mistake, enhancement), undetermined; uniqueness: original or duplicate.
• Attributes, one value each: identification, finding activity, finding mode, criticality, status, type, uniqueness, urgency, environment, timing, originator, defects found in, changes made to.
• Criticality (disruption to users) is not urgency (order of work).
• Worked example: 250 reports; 227 software defects, 16 other problems, 7 undetermined; "how many defects" is 250, 227, 209 or 200 by definition; 80 of the 200 were outside code.

Test yourself

1. Define a defect. How does it differ from an anomaly and an incident? A defect is an imperfection or deficiency in a work product that does not meet its requirements or specifications and needs to be repaired or replaced. An anomaly is anything observed that deviates from expectations, and an incident is the unexpected event recorded; both are what is noticed and reported, before investigation shows whether a defect is behind them.

2. Why are some problem reports not defects? Give four examples. Because a problem is any unsatisfactory encounter with the software, and investigation may find the cause elsewhere. Examples: a duplicate of a defect already reported; a user's mistake in using the software; an operations mistake such as a wrongly set server clock; a request for a new feature outside the requirements; a problem that cannot be repeated.

3. In which work products can defects be found? Give an example of each from ExamReg. Requirements (a fee rule that does not say what happens on the last date itself), design (a hall ticket generated in a way too slow for a basic phone), code (a late fee charged at the wrong boundary), documents (a user guide that gives the wrong last date), and test cases (a test that expects the wrong fee).

4. List the attributes a defect record carries. Identification of the product; finding activity; finding mode; criticality; problem status; problem type; uniqueness; urgency; environment; timing (reported, discovered, corrected); originator; the work product the defect was found in; the work products changed to correct it; related changes; projected availability and release of the fix.

5. Distinguish criticality from urgency. Criticality measures how much the problem disrupts users, and is normally given by whoever reported it. Urgency is the priority the problem management organisation gives to evaluating, fixing and closing it, and decides the order of work.

6. Why can two reports of "the number of defects" for the same release differ, and how is that prevented? Because they may count different things: all problem reports, all software defects including duplicates and test case defects, or only original defects in the product. It is prevented by recording defects with attributes whose values are mutually exclusive and by stating, as a checklist, which values the count includes.