Acceptance Testing: Alpha, Beta and User Acceptance

Contractual and regulatory acceptance testing. "Contractual acceptance testing is performed against a contract's acceptance criteria for producing custom-developed software. Acceptance criteria should be defined when the parties agree to the contract." ExamReg was built for the college by a software house, so the contract's criteria decide whether the college accepts it. "Regulatory acceptance testing is performed against any regulations that must be adhered to, such as government, legal, or safety regulations", sometimes with results witnessed or audited by the regulator.

Alpha and beta testing. These, the syllabus explains, "are typically used by developers of commercial off-the-shelf (COTS) software who want to get feedback from potential or existing users, customers, and/or operators before the software product is put on the market."

• "Alpha testing is performed at the developing organization's site, not by the development team, but by potential or existing customers, and/or operators or an independent test team."
• "Beta testing is performed by potential or existing customers, and/or operators at their own locations. Beta testing may come after alpha testing, or may occur without any preceding alpha testing having occurred."

Beta testing's special value is its environment. One of its objectives, in the syllabus, is "the detection of defects related to the conditions and environment(s) in which the system will be used, especially when those conditions and environment(s) are difficult to replicate by the development team." ExamReg's equivalent is a pilot: one department's students register for real, on their own phones and home connections, a week before the portal opens to the whole college. Those phones and networks are exactly what the development team cannot reproduce, and the 4.2-second hall ticket found in the validation testing of Chapter Forty-One is the kind of finding a beta produces.

The forms side by side

Acceptance criteria, and writing them first

Acceptance is only a decision if its criteria are known in advance. Acceptance criteria are the "criteria that a system or component is required to satisfy to be accepted by a user, customer, or other authorized entity" (ISO/IEC 33202:2024), and for contractual acceptance the syllabus insists they "should be defined when the parties agree to the contract."

Acceptance test-driven development (ATDD) takes the idea to its end. In the current ISTQB syllabus, ATDD "Derives tests from acceptance criteria as part of the system design process", and "Tests are written before the part of the application is developed to satisfy the tests." The acceptance tests then exist before the code, and the final acceptance run is the last of many.

Worked example: the exam cell's sign-off

The contract between the college and the software house lists ExamReg's acceptance criteria. After UAT with students and clerks and OAT with the IT cell, the exam cell checks each criterion against the results, and signs off only if all are met. The program applies the criteria to release 2.0's results, including the two defects still open.

uat = {"critical": (8, 8), "other": (11, 12)}           # scenarios passed, run: students and clerks
oat = {"backup and restore": "pass", "upgrade from release 1.9": "pass",
       "add and remove clerk accounts": "pass", "restore minutes": 40}
open_defects = [("DR-311", "major", "hall ticket takes 4.2 s on a basic phone", "waiver W-07"),
                ("DR-318", "minor", "hall ticket font small when printed", None)]

checks = [
    ("UAT: every critical scenario passes", uat["critical"][0] == uat["critical"][1]),
    ("UAT: at least 90% of other scenarios pass", uat["other"][0] / uat["other"][1] >= 0.90),
    ("OAT: backup, restore, upgrade, accounts pass",
     all(v == "pass" for v in oat.values() if isinstance(v, str))),
    ("OAT: a full restore within 60 minutes", oat["restore minutes"] <= 60),
    ("no critical defect open", not any(sev == "critical" for _, sev, _, _ in open_defects)),
    ("every open major defect has a waiver", all(w for _, sev, _, w in open_defects if sev == "major")),
]
for text, met in checks:
    print(f"{'met    ' if met else 'NOT MET'}  {text}")
if all(met for _, met in checks):
    carried = [f"{d} ({sev}, {w or 'to fix in 2.1'})" for d, sev, _, w in open_defects]
    print("sign-off: ACCEPTED, carrying", "; ".join(carried))
else:
    print("sign-off: NOT ACCEPTED")

met      UAT: every critical scenario passes
met      UAT: at least 90% of other scenarios pass
met      OAT: backup, restore, upgrade, accounts pass
met      OAT: a full restore within 60 minutes
met      no critical defect open
met      every open major defect has a waiver
sign-off: ACCEPTED, carrying DR-311 (major, waiver W-07); DR-318 (minor, to fix in 2.1)

Every criterion is met, so the exam cell signs off, and the sign-off is honest about what it carries: the slow hall ticket, under the waiver the exam cell granted in validation, and a minor printing defect, both to be fixed in release 2.1. Two things about the example matter more than its numbers. The criteria were written into the contract before the testing, so nobody could argue them into shape afterwards. And acceptance did not require zero defects; it required that no critical defect be open and that every major one be knowingly accepted by the people who carry its risk.

Sign-off

The standards give the final step its formal meaning. Acceptance is the "action by an authorized representative of the acquirer by which the acquirer assumes ownership of products as partial or complete performance of an agreement" (ISO/IEC/IEEE 24748-5:2017). An older definition in the general vocabulary describes the classic contractual case: an acceptance test is a "test of a system or functional unit usually performed by the purchaser on his premises after installation with the participation of the vendor to help ensure that the contractual requirements are met" (ISO/IEC 2382). A sign-off records who accepted, what was accepted (the exact release), against which criteria, and with which known deviations and waivers.

What it does not mean

Acceptance testing is not a hunt for defects. Its purpose is confidence and a decision; many defects found at this stage mean the system arrived too early.

Alpha testing is not the developers testing their own system. It happens at the developer's site but is done by customers, operators or an independent team.

Beta testing is not unplanned. Its participants, duration, feedback channel and exit criteria are planned like any other test.

Acceptance does not mean the system is perfect. It means the owners have accepted it, with its known deviations recorded.

Quick revision

• Acceptance testing: formal testing to let a user, customer or other authorised entity decide whether to accept a system (IEEE 1012-2024).
• Objectives: confidence in the whole system, validation that it is complete and works as expected, verification of functional and non-functional behaviour; finding defects is often not an objective.
• UAT: fitness for use by intended users. OAT: operability (backup and restore, install and upgrade, disaster recovery, user management, maintenance). Contractual: against the contract's acceptance criteria. Regulatory: against regulations.
• Alpha: at the developer's site, not by the developers. Beta: by users at their own locations, in real conditions.
• Acceptance criteria agreed in advance; ATDD writes the acceptance tests first.
• Acceptance: the acquirer assumes ownership; sign-off records the release, criteria and carried deviations.

Test yourself

1. What is acceptance testing, and how does it differ from system testing? Formal testing that lets the users, customer or other authorised body decide whether to accept the system. System testing is done by the testers to find defects against the specification; acceptance testing is done by or for the owners to build confidence and make the acceptance decision, and finding many defects there is itself a warning.

2. Describe four forms of acceptance testing. User acceptance testing checks that the intended users can do their work with the system. Operational acceptance testing checks that operators can run it: backup and restore, installation and upgrade, disaster recovery, user management. Contractual acceptance testing checks the contract's acceptance criteria. Regulatory acceptance testing checks compliance with applicable regulations, sometimes witnessed by the regulator.

3. Distinguish alpha testing from beta testing. Alpha testing is done at the developing organisation's site by potential or existing customers, operators or an independent test team, not by the developers. Beta testing is done by customers or operators at their own locations, in real conditions, and can find defects that depend on environments the developers cannot reproduce.

4. Why should acceptance criteria be agreed before testing? So that acceptance is a decision against known conditions rather than an argument afterwards; for contractual acceptance they should be agreed when the contract is made, and in ATDD they become tests before the code is written.

5. What does a sign-off record? That an authorised representative of the acquirer has accepted the product, which exact release was accepted, against which acceptance criteria, and with which known deviations and waivers carried into use.