System Testing and Acceptance

• Reflow (SC 1.4.10). At 320 pixels every page's content is exactly 320 wide: nothing has to be scrolled sideways, on any page, at either width.
• Labels (SC 3.3.2). Every field on every page has one, the owner's 49 included: those are the price, stock and availability controls of each menu item, built by the page's script, which is exactly where a label is easiest to forget.
• Contrast (SC 1.4.3). The lowest anywhere is 6.78 to 1, half as much again as the 4.5 the requirement asks for. It is the grey hint text, which is the text an author is most tempted to make too light.
• Taps (NFR-7). From the menu page, a one-item order is two taps, the item's plus button and Place order, because the first open slot is already chosen; choosing another slot makes four. The limit is four.
• The counter's single tap (NFR-7). Every action on the counter's screen is one button: Start preparing, Mark ready, Collected and paid, Not collected. Nothing there is typed.

The two targets under 24 pixels

A measurement that flags something is not a failure until the standard says so. SC 2.5.8 asks for 24 by 24 CSS pixels except in five cases, two of which apply here:

• "See the menu", 100 by 18 pixels, on the orders page. It is a link inside the sentence "No orders today yet. See the menu.", whose line height is 24 pixels. The criterion's Inline exception covers a target "in a sentence or its size is otherwise constrained by the line-height of non-target text".
• The owner's checkboxes, 13 by 13 pixels. The stylesheet deliberately leaves checkboxes as the browser draws them, which is the User Agent Control exception, "the size of the target is determined by the user agent and is not modified by the author". And in practice the target is larger anyway: each box sits inside a label measuring 92 by 44 pixels, and tapping the label toggles the box.

The honest way to record this is not to ignore the two numbers, and not to change the design to chase them, but to write both in the test report with the exception each relies on, so that a reader can check the reasoning. The rest of the page's targets have room to spare: the stepper's plus and minus are 44 by 44, Place order is 324 by 45, and the header's links are exactly 24 tall.

The keyboard (SC 2.1.1)

Every action must work from a keyboard, which for a page built this way means: every clickable thing is a real button or a real link. The pages' scripts attach every click handler to a button element, and nothing sets a tabindex. So the Tab key reaches every control in the order they appear, Enter and Space activate them, and the focus ring of Chapter 40's stylesheet shows where you are.

Testing the browsers (NFR-9)

Four browsers, and the Android app, each doing the same short list: sign in, order, see the order, and the counter's screen moving it on. What differs between browsers is rarely the logic; it is layout, dates, and whichever CSS feature is newest. The worked team ran the list on Chrome and Firefox on Windows, Safari on the MacBook, Edge on a lab machine, and the APK on two phones (Chapter 59), and recorded each as a line in the report.

Acceptance testing

Everything above is the team testing its own work. Acceptance testing is the users deciding, and it is a different thing: not "does it do what the SRS says" but "will we use this".

The worked team's session, at the canteen after the break on Friday 9 October, with Lata Pawar and Ganesh More, and the head cook watching:

What it found, which is what acceptance testing is for:

• Ganesh asked for the order number to be bigger on the counter's list: at arm's length on a tablet, he was reading it twice. It was changed the same afternoon, and now draws at 24 pixels against the student name's 18.
• The owner wanted the stock page to open on today's items only, which it already did; she had been looking at the menu page. A wording change, not a code change: "Menu and today's stock" became the heading.
• The head cook asked whether he could have the kitchen list on paper. Out of scope, recorded in the report's limitations as the first thing a next version should do (Chapter 73).

The result: accepted. The owner agreed to run it for the rest of the term, which is the sentence that matters, and it goes in the report.

Two rules for a session like this. Watch, do not help: the moment you reach over and tap it for them, you have learned nothing. And write down what they say in their words, not your summary of it: "I have to read the number twice" is a finding, "minor UI issue" is not.

Do this for your project

1. Walk your SRS, requirement by requirement, on the deployed system, doing each thing as its user would.
2. Measure the requirements that have numbers, and write the numbers down.
3. When a measurement flags something, read the standard before you call it a defect or change your design.
4. Test the browsers and devices your users have, not only yours.
5. Put the system in front of its real users, let them use it unaided, and watch.
6. Record every finding in the user's own words, and what you did about it.
7. Get the acceptance in writing, even if it is one line in an email.

Mistakes that cost marks

System testing by the person who wrote the code, on their own laptop, in the browser they develop in.

"It works on my phone", with no width, no browser and no number.

Ignoring a measurement that fails, or changing the design to chase a number without reading the criterion's exceptions.

Acceptance testing with the team driving, so the users watch a demonstration and agree to a system nobody tried.

No record of what the users said, so the report's "the client was satisfied" is the team's word for it.

Finding out on the examination day that the college's Edge draws one page differently.

Quick revision

• System testing: the whole thing, deployed, against every requirement, as its user would.
• What only it can reach: NFR-7 (320 px, taps), NFR-8 (labels, contrast, targets, keyboard), NFR-9 (browsers), NFR-12 (machines).
• Measure, then read the standard: SC 2.5.8 has five exceptions, of which Inline and User Agent Control applied here.
• Measured: no sideways scroll at 320; every field labelled; lowest contrast 6.78 to 1; an order in 2 taps; the counter's actions one tap each.
• Acceptance testing is the users deciding: watch, do not help; record their words; get the acceptance written down.

Questions you must be able to answer

1. What does system testing add to unit and integration testing? It tests the whole system, deployed as it will be used, against every requirement, including the ones no automated test can reach: how the pages behave at a phone's width, whether a person can operate them, whether they work in other browsers, and whether the system installs elsewhere.

2. How were NFR-7 and NFR-8 measured rather than judged? Each page was opened in a browser at 320 and 390 CSS pixels, signed in as its user, and the page's own measurements were read: the width of its content against the window, whether every field has a label, the contrast of each text colour against its background by WCAG's formula, and the size of every target.

3. Two targets measured less than 24 by 24 pixels. Why is neither a failure? Because SC 2.5.8 lists exceptions. The link measuring 100 by 18 is inside a sentence, which the Inline exception covers. The 13 by 13 checkbox is drawn by the browser and not styled by the author, which the User Agent Control exception covers; and its label, which also toggles it, measures 92 by 44.

4. What is the difference between system testing and acceptance testing? System testing is the team checking the system against the specification. Acceptance testing is the users deciding whether they will use it, doing their own work with it, and its outcome is a decision rather than a pass or fail against a document.

5. Why should the team not help during an acceptance session? Because the thing being tested is whether the users can use the system without the team. Every time a developer reaches over and taps, a finding is lost, and the same difficulty will appear when nobody is there to help.

6. What did acceptance testing find in the worked project that no test had? That the order number on the counter's list was too small to read at arm's length on a tablet during service, and that a heading misled the owner about where to set the stock. Both came from watching real people do their own work.