Sampling

Cluster sampling. The population is divided into groups, usually geographical, and whole clusters are selected at random, every unit within a selected cluster being studied. Its merit is cost: a field team works in a few villages instead of travelling to scattered households. Its cost is precision, since units within a cluster resemble one another.

Multi-stage sampling. Sampling in stages, which is how every large Indian survey is actually done: districts, then villages within selected districts, then households within selected villages, then a person within the household.

Non-probability sampling

The chance of selection is unknown, so no margin of error can be calculated and no statistical generalisation is justified. These designs are legitimate for exploratory and qualitative work and are illegitimate when used to support statements about a population.

Convenience or accidental sampling: whoever is available. The weakest design, and the commonest in student work.

Purposive or judgment sampling: units chosen deliberately because the researcher judges them informative. Appropriate for case studies and for expert respondents.

Quota sampling: interviewers are told to fill quotas matching the population's composition, choosing whom they like within each quota. It looks like stratified sampling and is not, because the choice within the quota is the interviewer's, and interviewers select the approachable.

Snowball sampling: each respondent introduces the next. The only practicable design for hidden or hard-to-reach populations, such as migrant workers without addresses or people engaged in stigmatised occupations, and it is biased towards the well-connected within that population.

Sampling error and bias, which are not the same thing

Sampling error is the difference between the sample estimate and the true population value arising from the fact that a part rather than the whole was studied. It is random, it can be calculated for a probability sample, and it falls as the sample size rises.

Bias is a systematic distortion arising from the way the sample was selected or the data collected. It is not random, it cannot be calculated, and it does not fall as the sample size rises.

This is the most important proposition in the chapter. Increasing the size of a biased sample increases the confidence with which a wrong answer is stated. A survey of ten thousand respondents drawn only from those with telephones is worse, not better, than a survey of five hundred drawn properly, because its precision is real and its accuracy is not.

Sources of bias: a defective sampling frame, which in Indian conditions routinely omits migrants, the homeless and recent settlements; non-response, since those who decline differ from those who answer; substituting an available respondent for a selected one, which destroys the design; and self-selection, where respondents volunteer.

How large should a sample be?

The honest answer has four parts, and giving all four is what makes it a good answer.

1. It depends on the variability of the population. A homogeneous population needs fewer units.
2. It depends on the precision required, and precision improves with the square root of the sample size, so quadrupling the sample halves the error. This is why very large samples are so expensive for so little gain.
3. It depends on how the results will be broken down. A sample adequate for a national estimate is inadequate to say anything about one district, because the sub-sample is what supports the sub-estimate.
4. It depends almost not at all on the size of the population. This is the counter-intuitive point worth stating: a well-drawn sample of the same size gives similar precision for a district and for a country. The fraction sampled is not what matters; the number is.

A worked example

Population: all persons in judicial custody awaiting trial in the state's prisons on a stated date. Defining it by a date matters, because the population changes daily.

Frame: the prison registers. Their defects must be checked first, since a frame that omits a category omits it from the findings.

Design: stratified multi-stage. Stratify prisons by type, since a central prison differs from a district one; select prisons within each stratum; then draw a random sample of prisoners within each selected prison from the register.

Why not convenience sampling: interviewing whoever the prison authorities produce would be a convenience sample selected by the very body whose conduct is in question, and no number of interviews would repair it.

Disproportionate stratification: women prisoners are a small fraction and must be over-sampled if anything is to be said about them separately, with the sample re-weighted at the analysis stage.

Size: driven by the breakdowns wanted. If the report is to say anything about each of five offence categories separately, each category needs enough units, and that requirement, not the total prison population, determines the number.

The bias that would destroy it: interviewing only those willing to speak, only in the presence of officials, or only those the administration selects. Any one of these makes the study worthless whatever its size, and saying so in the report is the difference between research and advocacy.

Quick revision

1. Population or universe, sampling frame, sampling unit, sample size.
2. Why sample: cost, time, feasibility, speed, and sometimes greater accuracy, since a small sample can be studied with better trained staff than a full enumeration.
3. Probability: simple random, systematic (every kth after a random start; beware periodicity), stratified (proportionate and disproportionate), cluster, multi-stage. Only these support a calculated margin of error.
4. Non-probability: convenience, purposive, quota (not stratified, because the interviewer chooses within the quota), snowball (for hidden populations).
5. Sampling error is random, calculable and falls with size. Bias is systematic, incalculable and does NOT fall with size. A larger biased sample states a wrong answer more confidently.
6. Sources of bias: defective frame, non-response, substitution of respondents, self-selection.
7. Sample size depends on variability, the precision required (which improves with the square root of size), the breakdowns wanted, and almost not at all on the size of the population.

Test yourself

1. Define sampling and its associated terms. Sampling is the selection of a portion of a population for study in such a way that the characteristics of the whole may be inferred from it. The population or universe is the entire set of units about which conclusions are to be drawn; the sampling frame is the list from which the sample is actually drawn, and may be incomplete; the sampling unit is the element selected; and the sample size is the number of units selected.

2. Distinguish probability from non-probability sampling and name the types of each. In probability sampling every unit has a known and non-zero chance of selection, which is what permits the accuracy of the estimate to be calculated and generalisation to the population to be justified: its forms are simple random, systematic, stratified, cluster and multi-stage. In non-probability sampling the chance of selection is unknown, so no margin of error can be computed: its forms are convenience, purposive, quota and snowball, and they are proper for exploratory and qualitative work but not for statements about a population.

3. Distinguish sampling error from bias, and state why the difference matters. Sampling error is the random difference between a sample estimate and the true value arising from studying a part rather than the whole; it can be calculated for a probability sample and it falls as the sample grows. Bias is a systematic distortion produced by the way the sample was drawn or the data collected; it cannot be calculated and it does not fall as the sample grows. The difference matters because increasing the size of a biased sample increases the confidence with which a wrong answer is stated.

4. Why is quota sampling not the same as stratified sampling? Because although both ensure that the sample's composition matches the population's on stated characteristics, stratified sampling selects units within each stratum at random, while quota sampling leaves the choice within each quota to the interviewer. Interviewers select those who are approachable, available and willing, so the units within each quota are systematically unrepresentative, and no margin of error can be calculated for the result.

5. What determines the size of a sample? The variability of the population, since a homogeneous population needs fewer units; the precision required, which improves only with the square root of the sample size, so that quadrupling the sample halves the error; and the breakdowns the report will contain, since any sub-estimate is supported only by its own sub-sample. It depends almost not at all on the size of the population, so a well-drawn sample of a given size yields similar precision for a district and for a country.