Experience-Based Testing

import re

def valid_name(name):                           # the version under test
    """ExamReg: may this name be printed on the hall ticket?"""
    return re.fullmatch(r"[A-Za-z ]{1,40}", name) is not None

# from the specification: "the student's name as on the admission record, at most 40 characters"
formal = [("", False), ("A", True), ("A" * 40, True), ("A" * 41, False), ("Priya Sharma", True)]

# from experience of names on real forms; "ask" marks a question the specification must answer
experience = [("Priya D'Souza", True),           # an apostrophe
              ("S. R. Iyer", True),              # initials with dots
              ("Anne-Marie Fernandes", True),    # a hyphen
              ("   ", False),                    # nothing but spaces
              (" Priya Sharma", "ask"),          # a leading space: store it, or trim it?
              ("अमित पाटील", "ask")]              # Devanagari: is the admission record in English only?

for name, tests in [("formal (length partitions and boundaries)", formal),
                    ("experience-based", experience)]:
    failed = [(n, want) for n, want in tests if want != "ask" and valid_name(n) != want]
    asked = [n for n, want in tests if want == "ask"]
    print(f"{name}: {len(tests)} tests, {len(failed)} failed, {len(asked)} questions")
    for n, want in failed:
        print(f"   FAIL {n!r}: expected {'accepted' if want else 'refused'},"
              f" got {'accepted' if valid_name(n) else 'refused'}")
    for n in asked:
        print(f"   ASK  {n!r}: the code says {'accepted' if valid_name(n) else 'refused'};"
              f" the specification does not say")

formal (length partitions and boundaries): 5 tests, 0 failed, 0 questions
experience-based: 6 tests, 4 failed, 2 questions
   FAIL "Priya D'Souza": expected accepted, got refused
   FAIL 'S. R. Iyer': expected accepted, got refused
   FAIL 'Anne-Marie Fernandes': expected accepted, got refused
   FAIL '   ': expected refused, got accepted
   ASK  ' Priya Sharma': the code says accepted; the specification does not say
   ASK  'अमित पाटील': the code says refused; the specification does not say

The formal tests are right and they pass: the length rule is implemented correctly. The experience-based tests find four defects the formal ones could not have been aimed at. Three real names are refused, because the developer's pattern allows only letters and spaces; a name made of nothing but spaces is accepted, because the pattern allows spaces anywhere. And two tests end not in a verdict but in a question for the person who owns the requirement: should a leading space be trimmed or stored, and may a name be printed in Devanagari at all? The specification cannot answer either. Asking is part of the result.

That is the family's value in one example. The formal techniques test what the specification says, and here it said almost nothing. The tester's knowledge filled the gap, found defects, and improved the specification.

When experience-based testing is the right tool

The ISTQB syllabus names the situations in which exploratory testing, the most flexible of the three, is useful: "when there are few or inadequate specifications or there is significant time pressure on the testing", and "to complement other more formal test techniques". James Bach's list of where exploratory testing fits is longer; among its entries:

• "You need to provide rapid feedback on a new product or feature."
• "You need to learn the product quickly."
• "You have already tested using scripts, and seek to diversify the testing."
• "You want to find the single most important bug in the shortest time."
• "You want to investigate and isolate a particular defect."

And in general, he writes, it is called for "in any situation where it's not obvious what the next test should be, or when you want to go beyond the obvious tests."

The worked example was the first of the syllabus's situations: an inadequate specification. It would have been worth doing even with a good one, as the complement the syllabus describes.

The three techniques compared

All three share the family's defining trait: the tests come from knowledge, not from a document. The ISTQB syllabus's description of checklists states the trade-off that runs through all of them: "If the checklists are high-level, some variability in the actual testing is likely to occur, resulting in potentially greater coverage but less repeatability."

Strengths and limits

Strengths. The family finds defects no specification describes, because it works from how software fails in practice. It works where specifications are thin or absent. It is quick to start: a skilled tester can find important defects in the first hour, before any formal test is designed. And it improves the specification, because a knowledgeable tester's tests raise questions that the specification's author did not think to answer.

Limits. Its effectiveness "depends heavily on the tester's skills": a new tester has little experience to draw on. Its coverage is hard to measure, because there is no model of the software to count items against. Its tests may be hard to repeat. And it finds only the kinds of failure somebody has met or imagined. That is why it complements the formal techniques instead of replacing them.

What it does not mean

Experience-based testing is not unplanned testing. Each technique has its structure: a list of attacks, a charter and a session report, a checklist.

It is not only for senior testers. Lists and checklists carry one tester's experience to another, and a session charter gives a less experienced tester a mission.

It is not a substitute for the formal techniques. In the worked example the formal tests were still needed: they proved the length rule right, which the experience-based tests did not examine.

A question is not a failure. When the specification does not say what should happen, the right result of the test is a question to the requirement's owner.

Quick revision

• Experience-based testing (ISO/IEC/IEEE 29119-4): test cases generated from "the experience of testers".
• Depends "heavily on the tester's skills"; "complementary" to black-box and white-box techniques (ISTQB).
• Knowledge comes from how the application has worked, the errors developers tend to make, and failures in similar applications (ISTQB).
• Useful with "few or inadequate specifications" or "significant time pressure", and as a complement (ISTQB).
• MU's three: error guessing, exploratory testing, checklist-based testing.
• Worked example: 5 formal tests of the name rule passed; 6 experience-based tests found 4 defects (three real names refused, a blank name accepted) and raised 2 questions.

Test yourself

1. What is experience-based testing? How does it differ from the other two families? A family of test design techniques in which test cases are derived from the knowledge and experience of testers, developers and users about how software is used and how it fails. Black-box techniques derive tests from the specification and white-box techniques from the code; experience-based techniques from knowledge that neither document holds.

2. On what knowledge does experience-based testing draw? On how the application has worked in the past, the kinds of errors developers tend to make and the defects they cause, the failures seen in similar applications, the expected use of the software and its environment, and where defects have clustered before.

3. When is experience-based testing most useful? When specifications are few or inadequate, when there is significant time pressure, when rapid feedback or rapid learning is needed, and always as a complement to the formal techniques to go beyond the obvious tests.

4. Name the three experience-based techniques in MU's syllabus and what guides each. Error guessing, guided by a list of likely errors, defects and failures; exploratory testing, guided by a charter and by what each test reveals; checklist-based testing, guided by a checklist of test conditions.

5. What are the limitations of experience-based testing? It depends heavily on the tester's skill and knowledge; its coverage is hard to measure; its tests can be hard to repeat; and it finds only kinds of failure that someone has met or can imagine.

6. In the worked example, why could the formal techniques not find the defects in valid_name? Because the specification gave them only one rule to work from, the length of at most 40 characters, which was implemented correctly. The defects lay in which characters a real name may contain, which the specification did not state and only the tester's knowledge of real names supplied.