The Survey
Chapter -Five
Syllabus topic 5, "Research Methods"
Pages 382 to 384 of 543
In one line
A survey collects the same information from many units so that it can be counted, and everything that makes a survey good or bad is decided before anybody is asked anything.
In the wording a student can write in an exam: the survey method collects standardised information from a defined population, or from a sample of it, by questionnaire, schedule, interview or observation, so that the results can be tabulated, compared and generalised to the population within stated limits; it is the principal method for establishing how widespread something is, and its validity depends on the definition of the population, the method of sampling, the design of the instrument and the treatment of non-response.
What a survey can and cannot answer
It answers how many, what proportion, how often and whether two things go together.
It does not answer why, except so far as it asked, and it can only ask what was thought of in advance. That is chapter 820's limitation and it is why interviews usually precede a survey.
And it does not answer what happens in a sequence, which is the case study's work, chapter 980.
The four decisions that determine its worth
One: the population. Who exactly is the survey about? Every household in three named settlements; every accused produced for remand in one court in six weeks; every college affiliated to one University. A survey whose population is undefined cannot generalise to anything.
Two: the sample. All of the population, or some of it chosen how. Chapters 1000 to 1020.
Three: the instrument. Questionnaire or schedule, and how the questions are drafted, chapter 970.
Four: non-response. Who did not answer, how many, and whether they differ from those who did.
All four are decided before collection begins, and none can be repaired afterwards. That is why the design chapters of Module IV matter.
Non-response, which deserves its own section
It is the most under-taught idea in student surveys and the one that most often invalidates them.
The problem. If a fifth of those approached refuse, the results describe the four fifths who agreed, and those four fifths may differ systematically. People who refuse are often busier, poorer, less confident or more suspicious of officialdom, and in a survey about legal awareness they are quite likely to be the least aware.
So a high refusal rate does not merely reduce the sample; it biases it, and in a predictable direction.
What to do. Record every approach and every refusal. Report the response rate. Where possible, note anything observable about those who refused, such as the settlement or the time of day. And treat a low response rate as a limitation to be stated, chapter 1280, not as an embarrassment to be hidden.
The Survey
The kinds of survey
Census, covering the whole population. Feasible where the population is small, such as the twelve colleges affiliated to one University, or eighty-four entries in a clinic register.
Sample survey, covering a part. The ordinary form.
Cross-sectional, taken at one point in time. Almost all student surveys.
Longitudinal, repeated over time on the same or a comparable population, which is how change is measured. Chapter 660's closing suggestion, reading the clinic register a year later, is a longitudinal design in miniature.
And by instrument: mailed or distributed questionnaire, schedule administered in person, telephone, and observation-based.
Conducting one
Define and record everything before starting, as above.
Pre-test the instrument, chapter 690.
Train anybody who helps. Two investigators asking a question differently produce two datasets.
Keep to the sampling plan. The single commonest destruction of a survey is substituting an easier respondent for the chosen one.
Record the conditions. Dates, places, times of day, weather if it affects who is at home, and anything unusual.
Keep the completed instruments.
And tabulate as you go if the survey is long, because errors found on day three can be corrected and errors found at the end cannot.
Its merits
It establishes magnitude, which nothing else does.
It is comparable. Two settlements, two courts, two years.
It supports generalisation within limits, if the sampling was sound.
It is efficient per unit of information.
And it is checkable, because another researcher with the same instrument and the same population can repeat it.
Its demerits
It measures only what was anticipated.
It depends entirely on the four decisions, and a fault in any one is not visible in the result, chapter 870.
It records what people say, unless the instrument is observational.
Non-response biases it, as above.
It is superficial by design, which is not a criticism but a consequence.
And it can be very precise about something that does not matter, which is why the question comes first.
A worked example
A census rather than a sample, which a student should prefer whenever the population is small enough.
The population. Every law college affiliated to one University, as listed by the University itself. Suppose there are eleven.
Why a census. Eleven is small enough to take entirely, which removes sampling error and every objection about representativeness in one stroke.
The instrument. A comparison sheet per college: for papers 21 to 24, what Schedule II prescribes against what the college's published examination pattern provides, chapter 500.
The source. Published documents, so there is no respondent, no consent problem and no non-response in the ordinary sense, though a college that publishes nothing is a non-response and must be recorded as one.
The Survey
The result. A table of eleven colleges against four papers, and it is a complete description of that University.
Its limits, stated. One University; published patterns, which may differ from practice; and one year.
And what it cost. A week, and no money at all.
That is the survey a student should do first, and it is very rarely done.
Quick revision
A survey collects standardised information from a defined population or a sample of it, so that results can be tabulated, compared and generalised within stated limits.
Four decisions, all made before collection and none repairable afterwards: the population; the sample; the instrument; and the treatment of non-response.
Non-response biases rather than merely shrinks, because those who refuse often differ systematically and, in a survey about awareness, are likely to be the least aware. Record every approach and refusal and report the response rate.
Kinds: census and sample; cross-sectional and longitudinal; and by instrument.
Merits: establishes magnitude, comparable, generalisable within limits, efficient, checkable. Demerits: measures only what was anticipated; a fault in the four decisions is invisible in the result; records what people say; is biased by non-response; and is superficial by design.
Prefer a census where the population is small, which removes sampling error entirely.
Test yourself
1. Name the four decisions that determine a survey's worth. The definition of the population; the method of sampling; the design of the instrument; and the treatment of non-response. All are made before collection begins and none can be repaired afterwards.
2. Why does non-response bias a survey rather than merely reduce it? Because those who refuse often differ systematically from those who agree, being busier, poorer, less confident or more suspicious of officialdom. In a survey about legal awareness they are likely to be among the least aware, so their absence pushes the result in a predictable direction.
3. Distinguish a census from a sample survey and say when a student should prefer a census. A census covers the whole population and a sample survey covers a part of it. A student should prefer a census whenever the population is small enough to take entirely, such as the colleges affiliated to one University or the entries in one clinic register, because it removes sampling error and every objection about representativeness.
4. Give two things a survey cannot do. It cannot answer why, except so far as it thought to ask, since it can only measure what was anticipated; and it cannot describe a sequence, which is the work of a case study.
The rest of this subject
These notes are cut from the University's printed syllabus. Open the syllabus itself, or the past papers, for the same subject.