The Experimental Design
Chapter -Nine
Syllabus topic 7, "Preparation of the Research Design"
Pages 465 to 467 of 543
In one line
An experiment changes one thing deliberately and holds the rest constant, so that any difference in the result can be attributed to the change, and that is the only design that establishes causation directly.
In the wording a student can write in an exam: the experimental research design tests a causal hypothesis by manipulating an independent variable and observing its effect on a dependent variable while controlling other influences, usually by comparing an experimental group which receives the treatment with a control group which does not; its distinguishing feature is manipulation, which neither the descriptive nor the diagnostic design employs, and it is rarely available in legal research because the law cannot ordinarily be applied to one group and withheld from another.
The elements
The independent variable, which the researcher changes: the treatment.
The dependent variable, which is measured to see whether it changed.
The control, being everything else held constant so that it cannot explain the difference.
The experimental group, which receives the treatment, and the control group, which does not.
And randomisation, assigning units to the groups by chance so that the groups do not differ systematically to begin with. This is what makes an experiment powerful, and it is usually the element that cannot be had in legal research.
Why law rarely permits it
Because withholding the law from a control group is not permissible. A study of whether informing an accused of the right to legal services improves outcomes cannot inform half of them and deliberately not inform the other half. The duty exists for all of them, chapter 630.
Because the units are people in real proceedings, whose interests cannot be subordinated to a research design, chapter 1400.
Because the legal system does not hold anything else constant. Two courts differ in every respect at once.
And because the effects are slow. A change whose consequence appears over years cannot be studied in a semester.
What is available instead
Four designs that approach the experiment without manipulating anybody's legal position, and a student should know them because they are the realistic options.
The before-and-after study. Measure, wait for a change that somebody else makes, measure again. If a District Authority establishes a clinic in a settlement, applications from that settlement before and after are a natural comparison.
Its weakness. Anything else that changed in the interval is an alternative explanation, so the design needs a reason to think nothing else did.
The comparison of comparable units. Two similar taluks, one with a clinic and one without. Not an experiment, because the assignment was not random, but a real comparison if the two are genuinely alike.
The Experimental Design
Its weakness. They are never alike, and the reason one got a clinic may be the reason it differs.
The natural experiment. Where something outside the researcher's control assigns units in a way that resembles randomness. A change of law applying from a fixed date creates a before group and an after group.
And the simulation or classroom experiment, which is genuinely experimental and is available. Two groups of students taught the same clinical topic by different methods, chapter 330, and assessed on the same task, is a legitimate experimental design in legal education research, because nobody's legal rights are affected.
That last one is worth naming, because it means the one area of this syllabus where an experiment is actually available to a student is the study of legal education itself, which is Module I.
Its merits and demerits
Merits. It is the only design that establishes causation directly. Its logic is transparent. And its result is difficult to dispute if the design was sound.
Demerits. It is rarely available in law. It is artificial, so what holds in a controlled setting may not hold in a court. It raises acute ethical questions wherever the treatment matters, chapter 1400. Small numbers are common and reduce the power to detect an effect, chapter 1160. And it can only test one or two variables at a time, while legal outcomes are produced by many.
A worked example, of the kind a student could actually do
The question. Does teaching the drafting component of paper 21 by graded exercises with feedback produce better drafting than teaching it by lecture, chapter 330?
Why this is available. It concerns teaching method, not legal rights. Both groups receive teaching; neither is deprived of anything they are entitled to, and both are prepared for the same assessment.
The design. Two sections of the same year, assigned as the timetable already assigns them, which is not randomisation and must be stated. Section A receives the lecture course; section B receives the same content with fifteen graded drafting exercises as Schedule II prescribes, chapter 500. Both are assessed at the end on the same unseen drafting task, marked blind by a person who does not know which section a script came from.
The dependent variable. The mark on the unseen task.
Controls. Same content, same teacher, same time, same assessment, blind marking.
What could confound it. The sections may differ in ability to begin with, which is why a pre-test at the start is needed; and the students may talk to each other, which cannot be prevented and should be recorded.
And the ethical position. If the exercise method proves better, the other section is entitled to it afterwards, and saying so in advance is part of the design.
The Experimental Design
This is a real experiment, within a student's reach, on a question Module I of this book says nobody has answered.
Quick revision
The experimental design manipulates an independent variable and measures a dependent variable while controlling other influences, usually comparing an experimental group with a control group, ideally with randomisation. Manipulation is what distinguishes it from every other design.
Law rarely permits it because the law cannot be withheld from a control group, the units are people in real proceedings, nothing else is held constant, and the effects are slow.
Four available substitutes: the before-and-after study; the comparison of comparable units; the natural experiment, where something outside the researcher assigns the groups; and the classroom or simulation experiment, which is genuinely experimental and is available in legal education research.
Merits: the only design that establishes causation directly; transparent; hard to dispute. Demerits: rarely available; artificial; ethically acute; usually small; and limited to one or two variables.
Test yourself
1. What distinguishes an experimental design from a descriptive or diagnostic one? Manipulation. The researcher deliberately changes an independent variable and measures the effect on a dependent variable while controlling other influences, whereas descriptive and diagnostic designs observe without changing anything.
2. Give three reasons legal research rarely permits experiments. The law cannot ordinarily be applied to one group and withheld from a control group, since the duty or the entitlement exists for everybody; the units are people in real proceedings whose interests cannot be subordinated to a research design; and the legal system holds nothing else constant, so two courts differ in every respect at once.
3. Name the four available substitutes and the weakness of the first two. The before-and-after study, whose weakness is that anything else changing in the interval is an alternative explanation; the comparison of comparable units, whose weakness is that they are never truly alike and the reason one received the treatment may be the reason they differ; the natural experiment; and the classroom or simulation experiment.
4. Why is legal education itself the one area of this syllabus where an experiment is available? Because a study comparing two teaching methods affects nobody's legal rights. Both groups receive teaching and both are prepared for the same assessment, so a genuine experimental comparison can be made on a question about which nothing is currently known.
The rest of this subject
These notes are cut from the University's printed syllabus. Open the syllabus itself, or the past papers, for the same subject.