Practical 17: Continuous Integration with Jenkins

The lab's Jenkins skips the wizard and has the JUnit plugin installed already. It also has no users, so the commands below need no password. On your Jenkins, with a user, add -auth yourname:yourtoken to each jenkins-cli.jar command and --user yourname:yourtoken to each curl, with an API token made on your user's Security page; Jenkins's documentation describes both.

A second suite file

smoke.xml runs only the smoke group of Practical 16's tests, for a quick check; testng.xml runs everything:

<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Smoke">
  <test name="Smoke">
    <groups>
      <run>
        <include name="smoke"/>
      </run>
    </groups>
    <classes>
      <class name="tests.PortalTest"/>
    </classes>
  </test>
</suite>

Creating the job

In Jenkins: New Item, name it stqa-tests, choose Freestyle project, and click OK. Then fill in the form:

On Windows the build step is Execute Windows batch command, and the parameter is written %SUITE%: Jenkins's help says each parameter becomes an environment variable, ${PARAMETER_NAME}, or %PARAMETER_NAME% on Windows. The custom workspace is your Eclipse project's folder.

Figure 20.1 General: the description, and SUITE with its two choices. The word Edited beside Advanced shows a setting in there has been changed: the custom workspace.

Figure 20.2 Under the schedule, Jenkins shows when the job would last have run and would next run, here 7:31 and 7:46 PM: fifteen minutes apart, at the minutes it chose for this job's name.

Figure 20.3 The post-build action that reads Surefire's XML results after every build.

Save. The job's page now offers Build with Parameters instead of Build Now, which Jenkins's help says happens to every parameterized job.

The same job as a file

Jenkins keeps every job as a file named config.xml. The form above saves the settings below, among many defaults it fills in itself. The lab creates the job from this file with Jenkins's command-line tool, which does what the form does, and which Jenkins itself hands out at /jnlpJars/jenkins-cli.jar:

<?xml version="1.1" encoding="UTF-8"?>
<project>
  <description>Runs the Practice Portal's Selenium tests with Maven and TestNG.</description>
  <properties>
    <hudson.model.ParametersDefinitionProperty>
      <parameterDefinitions>
        <hudson.model.ChoiceParameterDefinition>
          <name>SUITE</name>
          <description>The TestNG suite file to run</description>
          <choices>
            <string>testng.xml</string>
            <string>smoke.xml</string>
          </choices>
        </hudson.model.ChoiceParameterDefinition>
      </parameterDefinitions>
    </hudson.model.ParametersDefinitionProperty>
  </properties>
  <scm class="hudson.scm.NullSCM"/>
  <triggers>
    <hudson.triggers.TimerTrigger>
      <spec>H/15 * * * *</spec>
    </hudson.triggers.TimerTrigger>
  </triggers>
  <customWorkspace>/home/student/stqa-practicals</customWorkspace>
  <builders>
    <hudson.tasks.Shell>
      <command>mvn test -Dsurefire.suiteXmlFiles=$SUITE -Dmaven.test.failure.ignore=true</command>
    </hudson.tasks.Shell>
  </builders>
  <publishers>
    <hudson.tasks.junit.JUnitResultArchiver>
      <testResults>target/surefire-reports/TEST-*.xml</testResults>
    </hudson.tasks.junit.JUnitResultArchiver>
  </publishers>
</project>

$ curl -s -o jenkins-cli.jar http://localhost:8090/jnlpJars/jenkins-cli.jar
$ java -jar jenkins-cli.jar -s http://localhost:8090/ -http create-job stqa-tests < stqa-tests.xml

Both commands print nothing when they work. -http is needed because Jenkins's command-line tool has used WebSocket by default since Jenkins 2.391, and its documentation says HTTP must then be asked for; WebSocket also needs the Jenkins URL set, which the wizard does on your Jenkins and the lab's skipped.

Build 1: the smoke suite

In the browser: Build with Parameters, choose smoke.xml, Build, then open the build and its Console Output. The lab starts the same build from the command line, which waits for it (-s) and prints its console output (-v):

$ java -jar jenkins-cli.jar -s http://localhost:8090/ -http build stqa-tests -p SUITE=smoke.xml -s -v
Started stqa-tests #1
Started from command line by anonymous
Running as SYSTEM
Building in workspace /home/student/stqa-practicals
[stqa-practicals] $ /bin/sh -xe /tmp/jenkins1300623275929803492.sh
+ mvn test -Dsurefire.suiteXmlFiles=smoke.xml -Dmaven.test.failure.ignore=true
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------< stqa:stqa-practicals >------------------------
[INFO] Building stqa-practicals 1.0
[INFO]   from pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- resources:3.3.1:resources (default-resources) @ stqa-practicals ---
[INFO] skip non existing resourceDirectory /home/student/stqa-practicals/src/main/resources
[INFO]
[INFO] --- compiler:3.16.0:compile (default-compile) @ stqa-practicals ---
[INFO] Recompiling the module because of changed source code.
[INFO] Compiling 5 source files with javac [debug release 17] to target/classes
[INFO]
[INFO] --- resources:3.3.1:testResources (default-testResources) @ stqa-practicals ---
[INFO] skip non existing resourceDirectory /home/student/stqa-practicals/src/test/resources
[INFO]
[INFO] --- compiler:3.16.0:testCompile (default-testCompile) @ stqa-practicals ---
[INFO] Recompiling the module because of changed dependency.
[INFO] Compiling 6 source files with javac [debug release 17] to target/test-classes
[INFO]
[INFO] --- surefire:3.5.6:test (default-test) @ stqa-practicals ---
[INFO] Using configured provider org.apache.maven.surefire.junit4.JUnit4Provider
[INFO] Using configured provider org.apache.maven.surefire.testng.TestNGProvider
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 0, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running TestSuite
Sep 30, 2026 7:29:35 PM org.openqa.selenium.devtools.CdpVersionFinder findNearestMatch
WARNING: Unable to find an exact match for CDP version 154, returning the closest version; found: 153; Please update to a Selenium version that supports CDP version 154
Sep 30, 2026 7:29:47 PM org.openqa.selenium.devtools.CdpVersionFinder findNearestMatch
WARNING: Unable to find an exact match for CDP version 154, returning the closest version; found: 153; Please update to a Selenium version that supports CDP version 154
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 53.02 s -- in TestSuite
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:04 min
[INFO] Finished at: 2026-09-30T19:29:50+05:30
[INFO] ------------------------------------------------------------------------
Recording test results
[Checks API] No suitable checks publisher found.
Finished: SUCCESS
Completed stqa-tests #1 : SUCCESS

The console output, part by part:

• "Started from command line by anonymous" is the cause; a build from the browser says "Started by user" and your name. "Running as SYSTEM" and "Building in workspace" are Jenkins stating who runs the build and where: the custom workspace, the project folder.
• The line starting + mvn test is the build step, with $SUITE already replaced by the value chosen, smoke.xml. Jenkins runs the step as a small shell script, whose name it prints first.
• The Maven run is Practical 16's: the smoke suite's two tests passed. The CDP warning appears once per browser, as before.
• "Recording test results" is the post-build action reading target/surefire-reports/TEST-*.xml. "No suitable checks publisher found" is the JUnit plugin saying it has no code-hosting site to report to, which is right for this job.
• "Finished: SUCCESS".

Build 2: the whole suite

This time the command does not print the console output (no -v), only how the build ended:

$ java -jar jenkins-cli.jar -s http://localhost:8090/ -http build stqa-tests -p SUITE=testng.xml -s
Started stqa-tests #2
Completed stqa-tests #2 : UNSTABLE

UNSTABLE: Maven built the project and ran all six tests, and one failed, the Marks sort of Practicals 12 and 16. The command-line tool also exits with a non-zero code when a build does not succeed.

Build 3: started by the schedule

H/15 means every fifteen minutes. The five fields are minute, hour, day of the month, month and day of the week, and * means every value. H is Jenkins's own addition: its help says it stands for a hash of the job's name, so that a hundred jobs scheduled "every fifteen minutes" do not all start at the same second, yet each job's minutes stay fixed. For stqa-tests Jenkins chose minutes 1, 16, 31 and 46, as the preview under the schedule showed.

Nobody waits a quarter of an hour in a practical, so the lab changed the schedule to , every minute, waited until build #3 appeared, and put H/15 * straight back. In the browser that is: Configure, change the Schedule, Save, wait a minute, change it back.

$ sed -i 's|H/15 \* \* \* \*|* * * * *|' stqa-tests.xml
$ java -jar jenkins-cli.jar -s http://localhost:8090/ -http update-job stqa-tests < stqa-tests.xml
$ until curl -sf http://localhost:8090/job/stqa-tests/3/api/json > /dev/null; do sleep 5; done
$ sed -i 's|<spec>\* \* \* \* \*</spec>|<spec>H/15 * * * *</spec>|' stqa-tests.xml
$ java -jar jenkins-cli.jar -s http://localhost:8090/ -http update-job stqa-tests < stqa-tests.xml
$ until curl -s http://localhost:8090/job/stqa-tests/3/api/json | grep -q '"building":false'; do sleep 5; done
$ curl -sg "http://localhost:8090/job/stqa-tests/3/api/json?tree=result,actions[causes[shortDescription]]&pretty=true"
{
  "_class" : "hudson.model.FreeStyleBuild",
  "actions" : [
    {
      "_class" : "hudson.model.CauseAction",
      "causes" : [
        {
          "_class" : "hudson.triggers.TimerTrigger$TimerTriggerCause",
          "shortDescription" : "Started by timer"
        }
      ]
    },
    {
      "_class" : "hudson.model.ParametersAction"
    },
    {
      "_class" : "hudson.tasks.junit.TestResultAction"
    },
    {

    },
    {
      "_class" : "org.jenkinsci.plugins.displayurlapi.actions.RunDisplayAction"
    }
  ],
  "result" : "UNSTABLE"
}

The until lines wait: the first until build #3 exists, the second until it has finished. The last command asks Jenkins's remote API, the JSON behind every page, for the build's result and its cause: "Started by timer". Nobody started it. It ran testng.xml, because Jenkins's help says a build started automatically uses each parameter's default, and a choice parameter's default is the value on its first line.

The history

The job's page draws the history for you:

Figure 20.4 The job page after three builds: #1 green, #2 and #3 unstable. The Test Result Trend chart shows 2 passed in #1, and 5 passed with 1 failed in #2 and #3.

The same history, as data from the remote API, is what the analysis below reads:

$ curl -sg "http://localhost:8090/job/stqa-tests/api/json?tree=builds[number,result]&pretty=true"
{
  "_class" : "hudson.model.FreeStyleProject",
  "builds" : [
    {
      "_class" : "hudson.model.FreeStyleBuild",
      "number" : 3,
      "result" : "UNSTABLE"
    },
    {
      "_class" : "hudson.model.FreeStyleBuild",
      "number" : 2,
      "result" : "UNSTABLE"
    },
    {
      "_class" : "hudson.model.FreeStyleBuild",
      "number" : 1,
      "result" : "SUCCESS"
    }
  ]
}
$ curl -sg "http://localhost:8090/job/stqa-tests/3/testReport/api/json?tree=passCount,failCount,skipCount,suites[cases[className,name,status,age]]&pretty=true"
{
  "_class" : "hudson.tasks.junit.TestResult",
  "failCount" : 1,
  "passCount" : 5,
  "skipCount" : 0,
  "suites" : [
    {
      "cases" : [
        {
          "age" : 0,
          "className" : "tests.PortalTest",
          "name" : "loginPageOpens",
          "status" : "PASSED"
        },
        {
          "age" : 0,
          "className" : "tests.PortalTest",
          "name" : "validLogin",
          "status" : "PASSED"
        },
        {
          "age" : 0,
          "className" : "tests.PortalTest",
          "name" : "validLogin",
          "status" : "PASSED"
        },
        {
          "age" : 0,
          "className" : "tests.PortalTest",
          "name" : "wrongPasswordIsRefused",
          "status" : "PASSED"
        },
        {
          "age" : 0,
          "className" : "tests.PortalTest",
          "name" : "filterByPune",
          "status" : "PASSED"
        },
        {
          "age" : 2,
          "className" : "tests.PortalTest",
          "name" : "marksSortLowestFirst",
          "status" : "FAILED"
        }
      ]
    }
  ]
}

Build #3's test report, in the browser, and the history of its one failing test:

Figure 20.5 Build #3's test result: six tests, one failed. Age 2 means marksSortLowestFirst has failed in two builds in a row.

Figure 20.6 The test's own history page: it failed in both builds that ran it, #2 and #3.

Analysing the test execution history

• Build #1 ran the smoke suite: two tests, both passed.
• Build #2 ran the whole suite for the first time: six test runs, five passed, and marksSortLowestFirst failed. It did not fail in #1 because #1 never ran it.
• Build #3, started by the timer, failed the same test with the same result. The report gives it age 2, and its history page shows it failing in both builds that ran it.

What the pattern means: a test that fails in every build since it was first run, in the same way, is a steady defect, here the Marks column sorting as text, found in Practical 12. A flaky test looks different: it passes and fails in turn with nothing changed. A test that passed for many builds and fails from one build onwards points to a change made just before that build, which is the question continuous integration exists to answer quickly.

Observations

Result

Jenkins was configured with a parameterized Freestyle job that runs the Selenium suite through Maven in the project's folder, publishes the JUnit results after every build, and starts by itself on a schedule. Three builds were run, the third started by the timer. Jenkins's reports, the console output, the test result pages and the Test Result Trend, showed the smoke suite passing and the whole suite failing one test in every build that ran it; the history identified the failure as a steady defect, not a flaky test.

Where marks are lost

Leaving out the maven.test.failure.ignore setting. A failing test then turns the build red, FAILURE, the same as a build that cannot compile.

The wrong folder for the results. The Test report XMLs pattern is relative to the workspace; without the custom workspace, Jenkins looks in its own empty folder and finds nothing.

left in place. A build every minute buries the useful history. Use H/15 or a daily H H .

$SUITE in a Windows batch step. Windows writes it %SUITE%.

Reading only the latest build. The point of Jenkins is the history: say when a failure started and whether it is steady.

A screenshot-free journal. The job's form, a console output, a test result page and the trend chart are the build reports MU asks for.

For the journal

Aim; the words table; how Jenkins was installed and started; the job's settings table; the console output of a build; the results of the three builds; the trend chart and a test's history; the analysis; the observations table; the result.

Quick revision

• CI: every change built and tested automatically. Jenkins runs jobs and keeps the history.
• Freestyle job: parameters, triggers, build steps, post-build actions.
• Build step: mvn test with two settings, each written -Dname=value: surefire.suiteXmlFiles set to $SUITE, and maven.test.failure.ignore set to true.
• Post-build: Publish JUnit test result report, target/surefire-reports/TEST-*.xml.
• Results: SUCCESS, UNSTABLE (tests failed), FAILURE (a step failed).
• Schedule: minute, hour, day of month, month, day of week; H is a hash of the job name.
• An automatic build uses each parameter's default: a choice's first line.
• History: a failure's age, its history page and the trend chart show whether it is steady or flaky.

Questions you must be able to answer

1. What is continuous integration? Building and testing a project automatically every time it changes, so that a change that breaks something is found at once.

2. Why does the build step include -Dmaven.test.failure.ignore=true? So that a failing test does not stop Maven with an error; Jenkins then records the results and marks the build UNSTABLE instead of FAILURE.

3. What does H/15 mean? Every fifteen minutes, at minutes chosen from a hash of the job's name: minutes 1, 16, 31 and 46 for this job.

4. Build #3 ran testng.xml though nobody chose it. Why? It was started by the timer, and an automatic build uses each parameter's default; a choice parameter's default is its first choice, testng.xml.

5. What is the difference between UNSTABLE and FAILURE? UNSTABLE: the build ran and at least one test failed. FAILURE: a build step itself failed, for example Maven could not compile.

6. marksSortLowestFirst has age 2 in build #3. What does that tell you? It has failed in two builds in a row, every build that ran it, so it is a steady failure, a real defect, not a flaky test.