Regression Testing, Smoke Testing and Continuous Integration

• It "should exercise the entire system from end to end. It does not have to be exhaustive, but it should be capable of exposing major problems."
• It "should be thorough enough that if the build passes, you can assume that it is stable enough to be tested more thoroughly."
• "The smoke test must evolve as the system evolves."
• And without it the practice is worthless: "The daily build has little value without the smoke test."

A build that fails the smoke test is broken, and McConnell's rule is that "fixing it becomes top priority."

Smoke testing and regression testing compared

Continuous integration

McConnell's daily build is the ancestor of continuous integration (CI), which shortens the cycle from a day to every change. Martin Fowler's article defines the practice: "each member of a team merges their changes into a codebase together with their colleagues changes at least daily. Each of these integrations is verified by an automated build (including test) to detect integration errors as quickly as possible."

His article sets out the practices that make it work. Among them:

• Put everything in a version controlled mainline, and automate the build.
• Make the build self-testing: the automated build runs the tests.
• Everyone pushes commits to the mainline every day, and every push to mainline triggers a build.
• Fix broken builds immediately. He quotes Kent Beck: "nobody has a higher priority task than fixing the build", and gives the usual remedy: "Usually the best way to fix the build is to revert the latest commit from the mainline, taking the system back to the last-known good build."
• Keep the build fast, because "The whole point of Continuous Integration is to provide rapid feedback."
• Everyone can see what's happening.

The 2018 ISTQB syllabus connects this back to integration testing: CI "has become common practice", and "Such continuous integration often includes automated regression testing, ideally at multiple test levels."

Worked example: ExamReg's build history

The program plays the part of a CI server for six commits to ExamReg's late-fee code. Every commit is built and smoke tested first; only if the smoke test passes is the regression suite, every value of days late from 0 to 16, run against the rule. The last good build is remembered, which is what a build history is for.

def rule(days):                                       # the late-fee rule every build must keep
    if days < 0 or days > 15:
        return "refused"
    return 0 if days == 0 else (100 if days <= 7 else 500)

def late_fee_41(days):                                # commit 41: the fee bands
    if days < 0 or days > 15:
        return "refused"
    return 0 if days == 0 else (100 if days <= 7 else 500)

def late_fee_43(days):                                # commit 43: a "tidy-up" of the bands
    if days < 0 or days > 15:
        return "refused"
    return 0 if days == 0 else (100 if days <= 8 else 500)

def late_fee_45(days):                                # commit 45: a constant renamed, one use missed
    if days < 0 or days > 15:
        return "refused"
    return 0 if days == 0 else (LATE_BAND_ONE if days <= 7 else 500)

history = [(41, "fee bands", late_fee_41), (42, "concession screen", late_fee_41),
           (43, "tidy fee bands", late_fee_43), (44, "revert 43", late_fee_41),
           (45, "rename fee constant", late_fee_45), (46, "finish the rename", late_fee_41)]

def smoke(fee):                                       # a few end-to-end cases, run first
    try:
        return fee(0) == 0 and fee(3) == 100
    except Exception:
        return False

def regression(fee):                                  # every value of days late, 0 to 16
    return [d for d in range(0, 17) if fee(d) != rule(d)]

last_good = None
for build, message, fee in history:
    if not smoke(fee):
        print(f"#{build} {message:<20} smoke FAIL   BROKEN: fix or revert before anything else")
        continue
    failed = regression(fee)
    if failed:
        days = ", ".join(map(str, failed))
        print(f"#{build} {message:<20} smoke pass   regression FAIL on day {days}:"
              f" look at what changed after #{last_good}")
    else:
        print(f"#{build} {message:<20} smoke pass   regression 17 of 17   good")
        last_good = build

#41 fee bands            smoke pass   regression 17 of 17   good
#42 concession screen    smoke pass   regression 17 of 17   good
#43 tidy fee bands       smoke pass   regression FAIL on day 8: look at what changed after #42
#44 revert 43            smoke pass   regression 17 of 17   good
#45 rename fee constant  smoke FAIL   BROKEN: fix or revert before anything else
#46 finish the rename    smoke pass   regression 17 of 17   good

Each line is what a team would see in its build history.

• Builds 41 and 42 pass both stages. Commit 42 changed a different screen, and the regression suite proves it did not disturb the fees.
• Build 43 passes the smoke test, because the portal works and the common cases are right, but the regression suite catches the "tidied" band: 8 days late now costs Rs 100. The history says where to look: build 42 was good, so the defect is in commit 43. McConnell's Day 17 and Day 18, shrunk to one commit.
• Build 44 reverts commit 43, Fowler's standard remedy, and the mainline is green again while the tidy-up is redone properly.
• Build 45 fails the smoke test: a renamed constant was missed in one place, and the fee page crashes for any late form. The build is broken, and nothing else should be tested or merged until it is fixed.
• Build 46 finishes the rename and is green.

Notice what the two kinds of test caught. The smoke test caught the crash in seconds but would never have noticed the day-8 band; the regression suite caught the band but would have been wasted on a build that could not run. Each is there for what the other misses.

Jenkins: continuous integration in the practical

The practical asks students to "Configure Jenkins to execute Selenium automation scripts automatically. Generate build reports and analyze test execution history." Jenkins's documentation describes the parts.

• Pipeline: "a suite of plugins which supports implementing and integrating continuous delivery pipelines into Jenkins"; a Pipeline's code "defines your entire build process, which typically includes stages for building an application, testing it and then delivering it."
• Jenkinsfile: the Pipeline "is written into a text file (called a Jenkinsfile) which in turn can be committed to a project's source control repository", so the build process is "versioned and reviewed like any other code."
• Stage: "a conceptually distinct subset of tasks performed through the entire Pipeline (e.g. 'Build', 'Test' and 'Deploy' stages)".
• Step: "A single task", such as sh, which runs a shell command, or junit, a step "for aggregating test reports".

A Jenkinsfile for the build history above, in the declarative syntax the documentation's example uses, looks like this. The make targets are placeholders for whatever commands run the project's build, smoke tests and regression tests; for the practical they would start the Selenium suite.

pipeline {
    agent any
    stages {
        stage('Build') {
            steps { sh 'make' }
        }
        stage('Smoke test') {
            steps { sh 'make smoke' }
        }
        stage('Regression tests') {
            steps {
                sh 'make check'
                junit 'reports/**/*.xml'
            }
        }
    }
}

Each push runs the stages in order; a failing stage stops the run and marks the build, and the junit step turns the test results into reports Jenkins can show. The list of builds, with each one's result, is the build history: the same record the worked example printed, which is how a team reads "look at what changed after #42" off a screen.

What it does not mean

A smoke test is not a quick version of the whole suite. It is a deliberately small end-to-end check of whether the build is fit to test, and it grows only as the system grows.

Regression testing is not retesting the fix. That is confirmation testing; regression testing checks everything else the change might have disturbed.

Continuous integration is not a tool. It is a practice of integrating and testing every change; Jenkins is one tool that automates it.

A red build is not a disaster. It is the system working: a defect found minutes after it was made, with the change that caused it already known.

Quick revision

• Regression testing: after a change, check that unmodified parts still work; grows with every release, so it is automated.
• Smoke test (McConnell 1996): a simple end-to-end check of whether the build "smokes"; passing means it is stable enough to test more thoroughly; failing means the build is broken and fixing it is top priority.
• Daily build benefits: less integration risk, less risk of low quality, easier defect diagnosis (Day 17 good, Day 18 broken), better morale.
• Continuous integration (Fowler; IEEE 2675-2021): everyone merges at least daily, every push triggers an automated self-testing build, broken builds fixed immediately (often by revert), builds kept fast, results visible.
• Jenkins: Pipeline, Jenkinsfile in source control, stages, steps (sh, junit), build history.
• Worked example: #43 passed smoke but failed regression on day 8; #45 failed smoke; the last good build points at the culprit commit.

Test yourself

1. Distinguish smoke testing from regression testing. A smoke test is a small, quick, end-to-end check run first on every build to decide whether the build works well enough to test further; failing it means the build is broken. Regression testing runs a large and growing suite, after the smoke test passes, to check that changes have not broken anything that used to work.

2. What is the daily build and smoke test, and what are its benefits? Every day the whole program is compiled, linked and combined, then given a smoke test. It minimises integration risk, reduces the risk of low quality by keeping the system in a known good state, makes defects easy to diagnose because a failure must come from the day's changes, and improves morale.

3. Define continuous integration and list four of its practices. A practice in which every team member merges changes into a shared mainline at least daily, and each integration is verified by an automated build with tests. Practices: keep everything in version-controlled mainline; automate the build and make it self-testing; trigger a build on every push; fix broken builds immediately; keep the build fast; let everyone see the results.

4. How does a build history help locate a defect? It records the result of every build. If build 42 passed and build 43 failed, the defect came in with commit 43, so the search is narrowed to one change, which can be reverted while it is fixed.

5. What are a Jenkins Pipeline, a Jenkinsfile, a stage and a step? A Pipeline is Jenkins's model of the whole build, test and delivery process; a Jenkinsfile is the text file, kept in source control, in which the Pipeline is written; a stage is a distinct group of tasks such as Build or Test; and a step is a single task within a stage, such as running a shell command or collecting test reports.