What Goes Wrong in a Research Design

Eight: conclusions that outrun the evidence

What it looks like. One taluk becoming a State; an association becoming a cause, chapter 1050.

Prevented by stating the scope and limitations as a design component, chapter 1280, and by putting the qualification in the same sentence as the figure, chapter 1060.

Nine: recommendations that follow from nothing

What it looks like. Awareness should be increased. More funds should be provided.

Prevented by the six requirements of chapter 1420, of which naming the power is the one that turns a wish into a proposal.

The one-page audit

A student can run this against their own design in twenty minutes, and should, before collecting anything.

Can I state my problem in one sentence, and say what would count as finishing?

For each research question, which part of the design answers it?

What claim will I make from my sample, and does the sampling method support it?

Have I drawn the empty tables, and can my instrument fill every cell?

For each hypothesis, what result would refute it?

Which permissions do I need, have I asked, and what is my fallback?

What will I do with the data, decided now?

What will my findings NOT establish?

And for each recommendation I expect to make, which body can act and under what power?

Nine questions. A design that answers all nine is not guaranteed to produce a good dissertation, and one that cannot answer them is guaranteed not to.

A worked example

One dissertation, audited against the nine defects at the design stage, before anything was collected.

The proposal. A study of legal aid awareness in Maharashtra, by interviewing people at the District Court, testing the hypothesis that awareness is low, and recommending that awareness be increased.

Defect one. Legal aid awareness in Maharashtra is a subject, not a problem. Narrowed: awareness among adult residents of three named settlements in one taluk.

Defect two. The question asks about Maharashtra and the design reaches one court's visitors. Corrected by matching the claim to the design.

Defect three. People at the District Court are not a sample of anybody except people at the District Court, and they are systematically the people who already reached the legal system. Corrected by constructing a frame of dwellings and taking every fifth.

Defect four. The instrument recorded awareness and nothing else, so no subsidiary question could be answered. Corrected by adding the classification variables and the source of awareness.

Defect five. Awareness is low is untestable. Corrected to fewer than half of adult respondents, with the thing defined by the words of a question on the schedule.

Defect six. No permission was needed for the settlements, which is why this design is better than one requiring court records; and the request for the clinic register was sent in week one.

Defect seven. The tables were drawn empty before collection.

Defect eight. The conclusion will be confined to the three settlements, and the sentence saying so was written at the design stage.

Defect nine. The recommendation names the District Legal Services Authority, the power in section 4 read with section 10, four named settlements and three named topics taken from the clinic register.

Nine defects, all found in twenty minutes, all before a single respondent was approached. That is what this chapter is for.

What this book has been arguing

Three things, and they are worth stating at the end because they are the connections MU's seven-line syllabus does not make.

First, that the four items in topic 4 are one process, chapter 460. The clinic teaches and serves; its register is a survey; the survey establishes a cause; and the cause is what a reform proposal needs. Module II is that sentence worked out.

Second, that Module I's failures are failures of measurement. The rules of legal education are unusually detailed and the practice does not follow them, chapter 520, and nobody has counted. Every question that module raises is answerable by the methods of Module III.

And third, that the reader is the person who can do it. A court cannot survey, chapter 750; an authority does not; and a student has access to a clinic register, an adopted village, their own college's compliance and a court they can sit in, chapter 740. MU requires a dissertation of 100 to 150 pages and an interdisciplinary project with a chapter of data analysis, and the subjects are lying about unexamined.

The syllabus prints seven lines. What it is asking for is a lawyer who can find out whether the law is working, and then say, to somebody who can act, what should be done about it.

Quick revision

Nine defects: a subject instead of a problem; a question the design cannot answer; a sample that does not support the claim; an instrument that omits a needed item; an untestable hypothesis or one written after the data; collection begun before permissions; analysis planned after the results; conclusions that outrun the evidence; and recommendations that follow from nothing.

All nine are visible in the design before any data is collected, and the second is the most damaging because it is discovered last.

The one-page audit: state the problem in a sentence; map each question to a part of the design; check the sample against the claim; draw the empty tables; write the refutation condition for each hypothesis; identify permissions and a fallback; plan the analysis now; state what the findings will not establish; and name the body and the power for each expected recommendation.

Test yourself

1. Name the nine defects. A subject chosen instead of a problem; a research question the design cannot answer; a sample that does not support the claim made from it; an instrument that omits an item the analysis needs; a hypothesis that is untestable or formulated after the data; collection begun before permissions are obtained; analysis planned after the results are seen; conclusions that outrun the evidence; and recommendations that follow from nothing the study found.

2. Which defect is the most damaging, and why? A research question the design cannot answer, because it is usually discovered at the writing stage when the data has been collected and there is no time either to change the question or to gather what would have answered it.

3. Give the single check that prevents defects two and four. Drawing the empty tables before collection: the tables the dissertation will contain, with headings filled and cells blank. Then check that the instrument produces every number the tables need, that every table bears on a research question, and that every hypothesis has a table.

4. State the three connections this book has argued that the printed syllabus does not make. That the four items of topic 4 are one process, since the clinic serves and records, the record is a survey, and the survey supports reform; that the failures described in Module I are failures of measurement, since the rules are detailed, the practice does not follow them and nobody has counted; and that the student is the person able to measure it, because a court cannot survey and an authority does not, while a student has access to clinic registers, an adopted village, their own college's compliance and a court they can sit in.