Cross-Browser and Compatibility Testing

Pairwise testing

Pairwise testing rests on an assumption: that a compatibility defect is usually triggered by one value (a browser that mishandles a control) or by two values together (a browser that misbehaves only on small screens), and seldom needs three specific values at once. Combinatorial testing is the "class of specification-based test design techniques based on exercising combinations of parameter-value (P-V) pairs" (ISO/IEC/IEEE 29119-1:2022), and its commonest form, pairwise testing, chooses configurations so that every pair of values appears together in at least one of them.

Worked example: from 36 configurations to 9

The program lists every configuration and every pair of values the matrix contains, then chooses configurations greedily: at each step, the one that covers the most pairs not yet covered, until none is left. It then compares the run time of the regression suite on one machine against a Grid, for the practical's comparison of execution time.

from itertools import combinations, product

params = {"browser": ["Chrome", "Firefox", "Edge"],
          "platform": ["Windows 11", "Ubuntu", "Android"],
          "screen": ["phone-sized", "desktop-sized"],
          "network": ["fast", "slow"]}
values = list(params.values())
every_config = list(product(*values))

def pairs_in(config):                        # every pair of parameter values one config tests
    return {((i, config[i]), (j, config[j])) for i, j in combinations(range(len(config)), 2)}

all_pairs = set().union(*(pairs_in(c) for c in every_config))
chosen, uncovered = [], set(all_pairs)
while uncovered:                             # greedy: take the config that covers most new pairs
    best = max(every_config, key=lambda c: len(pairs_in(c) & uncovered))
    chosen.append(best)
    uncovered -= pairs_in(best)

print(f"every combination: {len(every_config)} configurations; pairs of values: {len(all_pairs)}")
print(f"pairwise selection: {len(chosen)} configurations cover every pair")
for n, config in enumerate(chosen, 1):
    print(f"  {n}. " + ", ".join(config))

suite_seconds = {"Chrome": 142, "Firefox": 168, "Edge": 151}   # one run of the regression suite
print(f"suite on one machine, one browser after another: {sum(suite_seconds.values())} s;"
      f" on a Grid, one node per browser: {max(suite_seconds.values())} s")

every combination: 36 configurations; pairs of values: 37
pairwise selection: 9 configurations cover every pair
  1. Chrome, Windows 11, phone-sized, fast
  2. Chrome, Ubuntu, desktop-sized, slow
  3. Firefox, Android, phone-sized, slow
  4. Edge, Android, desktop-sized, fast
  5. Firefox, Windows 11, desktop-sized, fast
  6. Edge, Windows 11, phone-sized, slow
  7. Firefox, Ubuntu, phone-sized, fast
  8. Chrome, Android, phone-sized, fast
  9. Edge, Ubuntu, phone-sized, fast
suite on one machine, one browser after another: 461 s; on a Grid, one node per browser: 168 s

Nine configurations cover all 37 pairs of values, a quarter of the 36. Nine is also the least any selection could manage: browser and platform alone make 3 × 3 = 9 pairs, and each configuration contains only one of them. Every browser meets every platform, every screen size and every network speed; every platform meets both screens and both networks. If Firefox breaks only on phone-sized screens, configuration 3 or 7 will show it.

The honest limit is the other side of the same fact: a defect that needs three particular values together, say Edge on Ubuntu on a slow network, may not be in the nine. Pairwise testing is a bet on the assumption above, and it is made knowingly. Where a particular triple is known to be risky, it is added by hand.

The last line answers the practical's second question. Run one browser after another on one machine, the suite takes 142 + 168 + 151 = 461 seconds; run on a Grid with a node for each browser, the three runs go in parallel and the whole takes as long as the slowest, 168 seconds.

Selenium Grid

Selenium Grid runs WebDriver tests on remote machines, so that one test suite can drive many browsers on many platforms from a single entry point. Its documentation describes three ways to deploy it.

• Standalone: "Standalone combines all Grid components seamlessly into one", in a single process on a single machine, and "is also the easiest mode to spin up a Selenium Grid." Its listed uses include running quick suites before pushing code and a simple Grid inside a CI tool such as Jenkins.
• Hub and Node: described as "the most used role because it allows to" combine different machines in one Grid, including "Machines with different operating systems and/or browser versions", to have "a single entry point to run WebDriver tests in different environments", and to scale capacity without tearing the Grid down. The Hub receives the tests; each Node is a machine with browsers, which on startup "will detect the available drivers that it can use".
• Distributed: "each component is started separately, and ideally on different machines."

Inside, the documentation says, "Grid is composed by six different components", and a Hub is made of five of them, "Router, Distributor, Session Map, New Session Queue, and Event Bus":

A test asks the Grid for a browser by its capabilities, for example Firefox on a particular platform; the Distributor finds a Node that has them and starts the session there. That is how the practical's instruction to "Execute automation scripts across multiple browsers (Chrome, Firefox, Edge) using Selenium Grid" is carried out: one suite, three requests for different capabilities, three Nodes working at once.

Recording compatibility issues

A compatibility failure is only useful if it says where it happened. Each defect report from cross-browser testing should record the browser and its version, the platform and its version, the screen size and the network, alongside the steps and the expected and actual results; without them, a developer on a different machine will often be unable to reproduce the failure. Chapter Seventy-Six, on writing a defect report, sets out the full report.

What it does not mean

Cross-browser testing is not running the suite once in the developer's favourite browser. It is the same tests on every supported combination, chosen deliberately.

Pairwise selection does not guarantee every defect is found. It guarantees that every pair of values is tried; defects needing three specific values may slip through.

Selenium Grid does not write tests. It runs existing WebDriver tests on remote browsers; the tests themselves are the subject of Chapter Forty-Nine, on driving a browser.

Compatibility is not only about browsers. It includes co-existence with other products in the same environment and interoperability with the systems a product exchanges data with.

Quick revision

• Compatibility (ISO/IEC 25010:2023): exchanging information with other products, or working while sharing an environment; two sub-characteristics, co-existence and interoperability.
• Compatibility testing (ISO/IEC/IEEE 29119-1:2022); cross-browser testing runs the same tests on each supported browser and platform.
• A matrix of browser × platform × screen × network gave 36 configurations.
• Pairwise testing covers every pair of values; greedy selection reached 9, the minimum (3 browsers × 3 platforms); triples may be missed.
• Selenium Grid: Standalone, Hub and Node, Distributed; components Router, Distributor, Session Map, New Session Queue, Event Bus, Node; capabilities choose the Node.
• Execution time: 461 s one browser after another, 168 s in parallel on a Grid.

Test yourself

1. What is compatibility testing? Distinguish co-existence from interoperability. Testing that measures how well a product functions alongside other products in a shared environment and exchanges information with other systems. Co-existence is working efficiently while sharing an environment and its resources without harming other products; interoperability is exchanging information with other products and using the information exchanged.

2. Why cannot every browser and platform combination be tested, and what technique reduces them? Because the combinations multiply: three browsers, three platforms, two screen sizes and two network speeds already give 36 configurations, each needing the whole suite. Pairwise testing reduces them by choosing configurations so that every pair of values appears together at least once, here nine configurations.

3. What is the weakness of pairwise selection? It guarantees coverage of every pair of values, not of every combination of three or more, so a defect that appears only with three particular values together may not be in the selection; known risky combinations are added by hand.

4. Describe the Hub and Node arrangement of Selenium Grid. A Hub is the single entry point that receives WebDriver test requests; it contains the Router, Distributor, Session Map, New Session Queue and Event Bus. Nodes are machines with browsers and drivers that register with the Hub; the Distributor assigns each new session to a Node whose capabilities match the browser and platform the test asked for.

5. Why does Selenium Grid shorten the time for cross-browser testing? Because the suite runs on several Nodes in parallel instead of on one machine in turn, so the total time is that of the slowest browser's run rather than the sum of all of them: 168 seconds instead of 461 in the worked example.