Agile, Scrum and DevOps: Testing in Short Cycles

The Scrum Team is "one Scrum Master, one Product Owner, and Developers", with no sub-teams or hierarchies, "focused on one objective at a time, the Product Goal." The guide speaks of three accountabilities rather than roles. The Product Owner is accountable for the value of the product and orders the Product Backlog. The Developers create the Increment, and "Developers" includes everyone who builds it: programmers, testers, designers. The Scrum Master is accountable for the team's effectiveness with Scrum. There is no separate tester accountability: testing is part of creating a usable Increment, and the Developers are accountable for, among other things, "Instilling quality by adhering to a Definition of Done".

The events. The Sprint is the container for all the others: Sprints are "fixed length events of one month or less". Sprint Planning decides the Sprint Goal and the work to reach it. The Daily Scrum is "a 15-minute event for the Developers" to inspect progress toward the Sprint Goal. The Sprint Review inspects the Increment with stakeholders. The Sprint Retrospective plans improvements to how the team works.

The artifacts and their commitments. The Product Backlog (committed to the Product Goal), the Sprint Backlog (committed to the Sprint Goal) and the Increment, committed to the Definition of Done.

The Definition of Done: where testing lives in Scrum

The Scrum Guide defines it precisely: "The Definition of Done is a formal description of the state of the Increment when it meets the quality measures required for the product." It is also absolute: "If a Product Backlog item does not meet the Definition of Done, it cannot be released or even presented at the Sprint Review." The ISTQB syllabus draws the connection to testing: in agile development, exit criteria are called the Definition of Done, and the entry criteria a user story must meet before work begins are the Definition of Ready.

A Definition of Done for ExamReg's team might read: code reviewed; unit tests written and passing; acceptance tests for the story automated and passing; no open defect of major severity or above; regression suite green on the build server; exam cell has seen it working. Every line is a test or a review. In Scrum, "done" means "tested".

Testing in agile development

The ISTQB syllabus summarises how agile changes testing: agile "assumes that change may occur throughout the project. Therefore, lightweight work product documentation and extensive test automation to make regression testing easier are favored in agile projects. Also, most of the manual testing tends to be done using experience-based test techniques ... that do not require extensive prior test analysis and design."

Three practices, which the syllabus groups as approaches in which tests direct development, make the tests come first:

A BDD scenario for ExamReg's late fee reads like this, and a tool can run it as a test once each step is bound to code:

Given a student has completed the examination form
And today is 3 days after the last date
When the student opens the fee page
Then the late fee shown is Rs 100

TDD's cycle is worked through in full in Chapter Thirty-Six, on unit testing best practices.

DevOps and continuous integration

The ISTQB syllabus describes DevOps as "an organizational approach aiming to create synergy by getting development (including testing) and operations to work together to achieve a set of common goals." Its technical core is continuous integration (CI), a "technique that continually merges artifacts, including source code updates from all developers on a team, into a shared mainline to build and test the developed system" (IEEE 2675-2021), and continuous delivery (CD), which keeps the system always ready to release.

For testing, the syllabus lists the benefits: fast feedback on code quality; CI "promotes shift left in testing ... by encouraging developers to submit high quality code accompanied by component tests and static analysis"; stable automated test environments; more visibility of non-functional qualities such as performance; less repetitive manual testing; and a smaller regression risk because automated regression tests run at scale. It lists the costs too: the pipeline must be built, the CI and CD tools maintained, and "Test automation requires additional resources and may be difficult to establish and maintain." And it adds a warning worth repeating: even with this much automation, manual testing from the user's perspective will still be needed. Continuous integration, and Jenkins as the tool the practical uses for it, has its own chapter (Chapter Thirty-Nine).

Shift left, and retrospectives

Shift left is the syllabus's name for the principle of early testing applied deliberately: reviewing specifications from a tester's point of view, writing test cases before code, using CI with automated component tests, running static analysis before dynamic testing, and starting non-functional testing at component level where possible. The syllabus adds the balancing sentence: shift left "does not mean that testing later in the SDLC should be neglected."

Retrospectives, held at the end of each iteration, ask what went well, what did not, and how to improve. The syllabus lists their benefits for testing, including more effective testing, better testware, better requirements and better cooperation between development and testing. They are the agile form of the lessons learned in test completion (Chapter Five, on the basic test process).

The test pyramid

When tests run every day, their cost and speed matter. The test pyramid is the model most teams use to balance them. Martin Fowler's account of it records that most people know it "due to Mike Cohn, when he described it in his 2009 book Succeeding with Agile", and that Cohn had drawn it in conversation with Lisa Crispin in 2003 and 2004.

Figure 17.1 The test pyramid (Cohn): many small, fast tests at the base and few broad, slow ones at the top

The ISTQB syllabus explains the shape: "The higher the layer, the lower the test granularity, the lower the test isolation ... and the higher the test execution time." Tests at the base are small, isolated and fast, so many are needed; tests at the top are end-to-end and slow, so few are used. Cohn's original three layers are "unit tests", "service tests" and "UI tests". Fowler adds the reason not to invert it: tests that run end to end through the user interface are "brittle, expensive to write, and time consuming to run."

The testing quadrants

A second agile model, the testing quadrants, defined by Brian Marick, sorts tests along two lines: whether they face the business or the technology, and whether they support the team (guide development) or critique the product. The ISTQB syllabus fills in the four quadrants:

The quadrants are a checklist: a team whose tests all sit in Q1 has fast unit tests and no idea whether users can use the product.

Worked example: one ExamReg Sprint

The team runs two-week Sprints. In Sprint 4 the Sprint Goal is: students with a fee concession can submit the form and pay only the backlog and late fees.

Compare the waterfall of Chapter Fourteen. There, the first working fee page existed in week 13 of 16. Here, a tested increment exists every two weeks.

What it does not mean

Agile does not mean no documentation or no planning. The manifesto values the items on the right; it values those on the left more. Scrum plans every Sprint.

Agile does not mean no testers. Scrum folds testing into the Developers' accountability; people who specialise in testing are Developers.

Automation does not replace all manual testing. The syllabus says user-perspective manual testing will still be needed; exploratory and usability testing sit in Q3.

Shift left does not mean skipping later testing. System and acceptance testing still happen; they just find fewer defects.

Quick revision

• Agile development: iterative, incremental, frequent inspection and adaptation, cross-functional collaboration.
• Manifesto values: individuals and interactions, working software, customer collaboration, responding to change, over processes and tools, comprehensive documentation, contract negotiation, following a plan.
• Scrum (Guide, November 2020): Scrum Team of Product Owner, Scrum Master and Developers; Sprint (one month or less), Sprint Planning, Daily Scrum (15 minutes), Sprint Review, Sprint Retrospective; Product Backlog, Sprint Backlog, Increment; Definition of Done = exit criteria; Definition of Ready = entry criteria.
• TDD, ATDD, BDD: tests first; BDD in Given/When/Then.
• DevOps: development and operations together; CI and CD; fast feedback; manual user-perspective testing still needed.
• Shift left: test earlier, without neglecting later testing. Retrospectives: continuous improvement.
• Test pyramid (Cohn): many unit, some service, few UI tests. Quadrants (Marick): business or technology facing, supporting the team or critiquing the product.

Test yourself

1. State the four values of the Agile Manifesto. Individuals and interactions over processes and tools; working software over comprehensive documentation; customer collaboration over contract negotiation; responding to change over following a plan. The items on the right still have value; those on the left are valued more.

2. Describe the Scrum framework. A Scrum Team of one Product Owner, one Scrum Master and Developers works in Sprints of one month or less. Each Sprint has Sprint Planning, Daily Scrums of 15 minutes, a Sprint Review and a Sprint Retrospective, and produces an Increment that meets the Definition of Done, drawn from the Product Backlog through the Sprint Backlog.

3. What is the Definition of Done, and how does it relate to testing? A formal description of the state of the Increment when it meets the quality measures required for the product. It works as the exit criteria for a backlog item: an item that does not meet it cannot be released or even shown at the Sprint Review, so it normally lists the reviews and tests that must pass.

4. Distinguish TDD, ATDD and BDD. TDD writes unit tests before the code and directs coding through them; ATDD derives tests from acceptance criteria before the feature is developed; BDD writes the expected behaviour in simple Given/When/Then language that is turned into executable tests.

5. Explain the test pyramid. A model with many small, fast, isolated unit tests at the base, fewer service or API tests in the middle, and few slow, broad UI end-to-end tests at the top; higher layers have lower granularity and isolation and longer execution time.

6. Give two benefits and two challenges of DevOps for testing. Benefits: fast feedback on code quality, and a smaller regression risk because automated regression tests run on every change. Challenges: the delivery pipeline and its CI and CD tools must be built and maintained, and test automation needs resources and can be hard to maintain.