How Quality Factors Shape Testing

Chapter Forty-Four takes the system-level types among these (recovery, security, stress, performance and deployment testing) in detail, and Chapters Forty-Five and Forty-Six the load and cross-browser testing that the practical sets.

Testability: the factor that governs all the others

One characteristic is different from the rest, because it is about testing itself. ISO/IEC 25010:2023 defines testability as the "capability of a product to enable an objective and feasible test to be designed and performed to determine whether a requirement is met". It sits under maintainability, and McCall made it a factor of its own in 1977.

A product with poor testability makes every other test expensive or impossible. If ExamReg's late-fee code reads the real system clock and nothing else, no test can check the fee for "16 days late" without waiting sixteen days. If the fee service can only talk to the real payment gateway, every test costs real money. Testability is designed in: a clock that tests can set, a gateway that tests can replace with a simulator, logs that show what happened. Testers who review designs (Chapter Eighteen, on the role of testing in each phase) ask for these things, because they cannot be added cheaply later.

Quality requirements must be measurable

A quality factor becomes testable only when its requirement is stated with a number and a condition. The same rule met in the last two chapters bears repeating here, because non-functional requirements break it most often:

Deciding where the effort goes: product risk

A project cannot test every characteristic deeply, so it tests hardest where failure would hurt most. The ISTQB syllabus calls the approach product risk analysis, and connects it directly to the quality model: product risks "are related to the product quality characteristics (e.g., described in the ISO 25010 quality model)". A risk has two attributes, likelihood and impact, and together they express the risk level. In the quantitative approach, the syllabus says, "the risk level is calculated as the multiplication of risk likelihood and risk impact."

The results decide, in the syllabus's list, the test scope, the test levels and types, the techniques and coverage, the effort for each task, and the order of testing, "to find the critical defects as early as possible".

Worked example: sharing ExamReg's test effort by risk

The ExamReg test lead scores each relevant characteristic from 1 to 5 for likelihood of failure and for impact if it fails. The scores are judgements, written down so they can be argued with. The program multiplies them into risk levels, as the syllabus describes, and shares 400 hours of test effort in proportion.

EFFORT_HOURS = 400
risks = {                          # characteristic: (likelihood 1-5, impact 1-5)
    "functional suitability": (4, 5),   # many fee and eligibility rules; wrong fee harms students
    "performance efficiency": (4, 4),   # a last-day rush never tested before
    "security":               (3, 5),   # student data and payments
    "interaction capability": (3, 3),   # first-year students, once a semester
    "reliability":            (2, 4),   # a crash during the window is serious
    "compatibility":          (3, 2),   # gateway adapter, several browsers
    "maintainability":        (2, 2),   # fee table changes each session
    "flexibility":            (1, 2),   # one server
}
levels = {c: l * i for c, (l, i) in risks.items()}
total = sum(levels.values())
print(f"{'characteristic':<24}{'L':>3}{'I':>3}{'level':>7}{'share':>8}{'hours':>7}")
for c in sorted(levels, key=lambda c: -levels[c]):
    l, i = risks[c]
    share = levels[c] / total
    print(f"{c:<24}{l:>3}{i:>3}{levels[c]:>7}{share:>8.1%}{round(EFFORT_HOURS * share):>7}")
print(f"{'total':<24}{'':>6}{total:>7}")

characteristic            L  I  level   share  hours
functional suitability    4  5     20   25.0%    100
performance efficiency    4  4     16   20.0%     80
security                  3  5     15   18.8%     75
interaction capability    3  3      9   11.2%     45
reliability               2  4      8   10.0%     40
compatibility             3  2      6    7.5%     30
maintainability           2  2      4    5.0%     20
flexibility               1  2      2    2.5%     10
total                              80

The table turns a quality model into a test plan. Functional suitability gets the most effort, but performance efficiency and security together get more than a third of the total, far more than a team that thought only of functional testing would have given them. Flexibility gets a few hours of installation testing. And because the scores are written down, the exam cell can challenge them: if they believe a crash during the window is catastrophic, they can raise reliability's impact to 5 and rerun the arithmetic.

What it does not mean

Non-functional testing is not optional extra testing. It is testing of eight of the nine characteristics, and its failures are often the most visible.

A risk score is not an objective measurement. It is a structured judgement; its value is that it is written down, argued over and revisited.

Testability is not the testers' responsibility alone. It is designed into the product by architects and developers, and testers ask for it in reviews.

Each characteristic does not need its own separate test phase. Many non-functional tests reuse functional tests with an extra check, and several run at every level.

Quick revision

• Quality factors decide the test types, the testability needed, the priorities and the timing of testing.
• Functional testing checks functional suitability (completeness, correctness, appropriateness); non-functional testing checks "how well the system behaves" against the other characteristics.
• Pairs: performance efficiency with performance, load, stress and volume testing; compatibility with interoperability and cross-browser testing; interaction capability with usability and accessibility testing; reliability with reliability and recovery testing; security with security testing; maintainability with static analysis and maintainability testing; flexibility with portability and installation testing; safety with safety analysis.
• Testability: the capability of a product to enable an objective and feasible test to be designed and performed (ISO/IEC 25010:2023); designed in, not added later.
• Product risk: likelihood and impact; quantitatively, risk level = likelihood × impact (ISTQB); used to set scope, types, techniques, effort and order.
• Worked example: 400 hours shared by risk level; functional suitability first, performance efficiency and security next.

Test yourself

1. How do software quality factors affect testing? They decide which test types are needed, one or more for each characteristic that matters; they require quality requirements to be measurable and the product to be testable; they set priorities through product risk analysis; and they push some non-functional testing early, because late non-functional defects are the most dangerous.

2. Distinguish functional from non-functional testing, with one example of each. Functional testing checks what the system does against its functional requirements, for example that a form 3 days late is charged Rs 100. Non-functional testing checks how well it behaves, for example that the fee page appears within 2 seconds for 500 simultaneous users.

3. Which test types check performance efficiency and security? Performance efficiency: performance, load, stress and volume testing. Security: security testing, including vulnerability scanning and penetration testing.

4. What is testability, and how is it achieved? The capability of a product to enable an objective and feasible test to be designed and performed to show a requirement is met. It is achieved in design, for example by letting tests set the clock, replace external services with simulators, and read logs of what happened.

5. How is a product risk level calculated, and what is it used for? In the quantitative approach, as risk likelihood multiplied by risk impact. It is used to decide the test scope, the levels, types and techniques, the effort for each area, and the order of testing.