Research Methods: Documentary, Empirical and Survey

Its steps

Formulating the problem; framing hypotheses; deciding the design; selecting the field and the units; collecting data by observation, measurement or interview; analysing; and reporting. This is the sequence in [The Stages of Data Collection].

Its merits

Data collected for the research question itself; the researcher controls the quality; behaviour is studied as it occurs rather than as it is reported; and it can reach what people do not or cannot say about themselves.

Its limitations

Expensive and slow; limited in coverage, since one worker can be in one place; the observer's presence alters what is observed; and it cannot reach the past.

3. The survey method

What it is

Information collected from a sample of a defined population by putting the same questions in the same way to everyone, and generalised to the population. It is the workhorse of modern social research and the method behind the Census, the National Sample Survey and the National Family Health Survey.

Its steps

1. Define the problem and the objectives.
2. Define the population to which the findings are to apply. Getting this wrong is fatal and irreparable later.
3. Design the sample, which is [Sampling].
4. Construct the instrument, a questionnaire or a schedule, from [The Questionnaire and the Schedule].
5. Pre-test or pilot it on a small group. This step is skipped by beginners and it is where most defects are found.
6. Train the field staff, so that the questions are put identically.
7. Collect the data.
8. Edit, code and tabulate.
9. Analyse and interpret.
10. Report, including the method and the response rate.

Its types

By coverage: a census survey covers every unit in the population; a sample survey covers a part.

By time: a cross-sectional survey takes one measurement at one time; a longitudinal survey follows the same subject over time, either as a panel, following the same individuals, or as a trend, repeating the same questions on fresh samples.

By mode: face to face, postal, telephonic and, increasingly, online.

Its merits

1. Generalisation. A properly drawn sample supports statements about a whole population, which no other method does.
2. Large coverage at reasonable cost.
3. Standardisation, which permits comparison between respondents, between places and over time.
4. Quantification, which makes statistical analysis possible.
5. Speed, relative to the coverage obtained.

Its limitations

1. It measures what people say, not what they do, and the two differ, particularly on income, on caste practice, on domestic violence and on anything socially disapproved.
2. Non-response, and it is not random: those who do not answer differ systematically from those who do, so a low response rate can destroy the study.
3. Question wording effects. Small changes in wording change answers, which is why the instrument must be piloted.
4. It is superficial. It obtains breadth and forfeits depth, which is exactly the trade-off against the case study.
5. It presumes a settled question. A survey can only ask what the designer already thought to ask, which is why exploratory work must come first.
6. Sampling frames are often defective in Indian conditions: lists are out of date, migrants and the homeless are missing, and those hardest to reach are frequently those the study most needs.

The three compared

They are complementary, and combining them is triangulation from [Social Research: Nature and Purpose]. A strong study of court delay would use documents, the court's own registers; empirical work, sitting in the courtroom and watching what actually happens on a listed date; and a survey of litigants and advocates. Where the three agree, the finding is secure; where they disagree, the disagreement is itself the finding.

A worked example

Documentary: examine the case files and cause lists for a year. They establish how many dates each matter took and what the recorded reason for each adjournment was.

Empirical: sit in the courtroom for thirty working days and record what actually occurs on each listed matter. This produces what the file does not: how long the court sat, how many matters were reached, why a matter was not reached, and what the recorded reason omits.

Survey: put a schedule to a sample of litigants: how far they travelled, what they spent, how many days of work they lost, and how many previous dates they had attended.

Note what each method alone would have missed. The file records a reason for adjournment which the courtroom shows to be a formality. The courtroom cannot show the cost to the litigant, who is a face in a corridor. The survey cannot establish what the register shows about the pattern across a year. The finding worth having exists only in the combination, and that is the practical argument for triangulation.

Quick revision

1. Documentary: existing personal and public records. Merits: reaches the past, cheap, unobtrusive, wide coverage, verifiable. Limits: made for another purpose, selective survival, authenticity, accuracy, changing recording practice, categories are the recorder's.
2. Thomas and Znaniecki's study of Polish migrants is the classic documentary study.
3. Empirical: first-hand data by direct contact. Merits: fit for the question, behaviour as it occurs. Limits: costly, narrow, observer effect, cannot reach the past.
4. Survey: standardised questions to a sample, generalised to a population. Steps: problem, population, sample, instrument, pilot, train, collect, edit and code, analyse, report.
5. Types: census and sample; cross-sectional and longitudinal, the latter as panel or trend; face to face, postal, telephonic, online.
6. Survey merits: generalisation, coverage, standardisation, quantification, speed. Limits: measures what people say; non-response is not random; wording effects; superficial; can ask only what was already thought of; defective sampling frames.
7. Triangulate: use all three and treat disagreement between them as a finding.

Test yourself

1. Describe the documentary method with its merits and limitations. The study of social phenomena through existing records, both personal, such as letters, diaries and life histories, and public, such as census reports, official statistics, judgments and archives. Its merits are that it is the only route to the past, that it is cheap and quick, that it is unobtrusive because the document was not created for the researcher, and that it offers coverage no fieldwork could match. Its limitations are that the material was produced for another purpose, that survival is selective and favours the records of the powerful and literate, that authenticity and accuracy must be established, and that changes in recording practice can be mistaken for changes in the thing recorded.

2. Set out the steps of a survey. Define the problem and objectives; define the population to which the findings are to apply; design the sample; construct the questionnaire or schedule; pilot it on a small group and revise it; train the field staff so that questions are put identically; collect the data; edit, code and tabulate; analyse and interpret; and report, including the method and the response rate.

3. What are the principal limitations of the survey method? That it records what people say rather than what they do, and the two diverge most on precisely the sensitive matters research most wants to reach; that non-response is not random, so those who decline differ systematically from those who answer; that small changes in the wording of a question change the answers; that it obtains breadth at the cost of depth; that it can only ask what the designer already thought to ask; and that sampling frames in Indian conditions are frequently defective, omitting migrants and the homeless.

4. Distinguish a panel from a trend survey. Both are longitudinal, taking measurements at more than one point in time. A panel survey follows the same individuals across successive rounds, so it can show how particular people changed and what changed for them. A trend survey repeats the same questions on fresh samples drawn from the same population, so it shows how the population changed in aggregate but cannot say whether the same individuals moved.

5. Why should the three methods be combined? Because each reaches what the others cannot, and their disagreement is informative. In a study of court delay, the file records a reason for each adjournment; sitting in the courtroom shows what actually happens on a listed date, which the file may record only formally; and a survey of litigants shows the cost in travel, expenditure and lost work, which neither the file nor the courtroom reveals. Where the three agree the finding is secure, and where they conflict the conflict is itself a finding about how records are made.