What Empirical Work in Law Cannot Do
Chapter -One
Syllabus topic 5, "Research Methods"
Pages 403 to 406 of 543
In one line
Data can show that a rule is not working and can never say what the rule ought to be, because no quantity of facts about the world entails a conclusion about what should be done.
In the wording a student can write in an exam: empirical research establishes what is the case, and it cannot by itself establish what ought to be the case, since a normative conclusion requires a value premise which no observation supplies; it also cannot establish causation from correlation alone, cannot generalise beyond its sample, cannot measure what it did not think to measure, and cannot settle the meaning of the concepts it uses.
Limit one: it cannot tell you what the law should be
The point in its plainest form. A survey may establish that most people do not know about free legal services. Nothing in that finding says whether the State ought to spend more on awareness, and nothing says it ought to spend it on this rather than on schools.
Why not. A conclusion about what ought to be done needs a premise about what matters, and observations are not premises of that kind. The move from what is to what ought to be requires a value, and the value comes from somewhere else: the Constitution, a statutory purpose, a moral argument or a political choice.
What follows for a dissertation. The recommendations chapter, chapter 1420, always rests on two things: a finding, and a standard. MU marks the suggestions, and a suggestion supported by a finding but no standard is a preference in the same way an unnamed criterion is, chapter 900.
And where a legal researcher's standard properly comes from. Article 39A; the four facets of access to justice in Anita Kushwaha, chapter 530; the declared purpose of the instrument itself, which is the strongest; or a stated moral or policy premise, openly stated as such.
Limit two: correlation is not causation
Two things varying together can arise three ways. One causes the other; the other causes the one; or a third thing causes both.
A legal example. Suppose districts with more legal services clinics show higher rates of legal aid applications. It may be that clinics produce applications; or that districts with more applications got more clinics; or that both follow from a more active District Judge.
What establishes causation. A design that rules the alternatives out: an experiment, which law rarely permits, chapter 1230; a comparison before and after a change; or a mechanism traced case by case, which is the case study's contribution, chapter 980.
And the discipline in writing. Use the words associated with, varied with or accompanied by, unless the design supports caused, and say what the design was.
What Empirical Work in Law Cannot Do
Limit three: it cannot generalise beyond its sample
Chapter 1000 and chapter 1020 between them settle this, and it needs restating because it is the temptation that survives everything else.
A study of one taluk is evidence about that taluk. A convenience sample is evidence about the people who agreed to speak.
The repair is not to weaken the finding but to state its scope, chapter 1280, and to say what would have to be done to extend it.
Limit four: it measures only what it thought to measure
A survey answers the questions it asked, chapter 990.
So a study can be perfectly executed and miss the thing that matters, because nobody put it on the schedule.
The defences. Qualitative work first, chapter 820. An open closing question, chapter 960. And an investigator's note after each interview recording anything observed that the answers did not capture, chapter 970.
Limit five: it cannot settle its own concepts
Chapter 830's point. Before counting whether an accused was informed, somebody must decide what counts as informed, and that decision is conceptual and is not settled by any amount of observation.
The consequence. Two studies of the same court can produce different figures without either being wrong, because they defined the thing differently.
The repair. State the operational definition, that is, the exact rule the researcher used to decide each case, in the methodology chapter, chapter 1310.
Two further limits particular to legal research
Access decides what can be studied, so the questions that get answered are the ones institutions permit. What happens inside a police station, a chamber or a disciplinary committee is largely unstudied for that reason, and the resulting picture of the legal system is systematically incomplete in one direction.
And measurement changes behaviour. A magistrate who knows an observer is recording whether the accused is informed may inform them, which is a real effect and is why chapter 960 lists it among observation's demerits. It is also, incidentally, an argument for the reform: if being watched produces compliance, a form in the file may produce it permanently.
What the limits do NOT mean
They do not mean empirical work is unreliable. Every limit above is also true of doctrinal work in its own way, and doctrinal work has the additional limit that it cannot see the world at all, chapter 850.
They do not mean findings should be hedged into uselessness. A finding stated with its scope is usable; a finding hedged until it says nothing is not.
And they do not mean a student should avoid the method. They mean the claims should match the design, which is the whole of chapter 790.
What Empirical Work in Law Cannot Do
A worked example
A study finds that in one court, over six weeks, the accused was informed of the right to free legal services in a small minority of remand productions.
What it establishes. That in this court, in this period, on this operational definition of informed, the proportion was what it was.
What it does not establish. That the same is true elsewhere; that the failure causes any particular outcome; that the magistrates are at fault rather than the assignment system; or that the practice ought to be changed.
How the dissertation gets from there to a recommendation. It adds a standard, and the standard is legal: Khatri holds that the duty exists and that the right is illusory unless the accused is informed, and Suk Das holds that a trial conducted without it is vitiated, chapter 630.
So the argument is: the law requires X, the study finds that X is not happening, therefore something should be done, and here is the cheapest thing that would do it.
Notice that the ought comes from the law and not from the data. That is this chapter in one sentence, and it is the sentence that makes a legal dissertation's recommendations defensible.
Quick revision
Five limits: it cannot say what the law ought to be, because a normative conclusion needs a value premise no observation supplies; correlation is not causation; it cannot generalise beyond its sample; it measures only what it thought to measure; and it cannot settle its own concepts, so the operational definition must be stated.
Two limits particular to law: access decides what can be studied, so the unstudied parts are systematically the closed ones; and measurement changes behaviour.
Where the ought comes from in a legal dissertation: Article 39A, a judicially stated standard such as Anita Kushwaha's four facets, the declared purpose of the instrument, or an openly stated policy premise.
The shape of a defensible recommendation: the law requires X; the study finds X is not happening; therefore something should be done; and here is the cheapest thing that would do it.
Test yourself
1. Why can no quantity of data establish what the law ought to be? Because a conclusion about what ought to be done requires a premise about what matters, and observations are not premises of that kind. The move from what is to what ought to be needs a value, which must come from the Constitution, a statutory purpose, a judicially stated standard or an openly stated policy premise.
2. Give the three ways two things can vary together, and say what a researcher should write instead of caused. One causes the other; the other causes the one; or a third factor causes both. Unless the design rules out the alternatives, a researcher should write that the two were associated with, varied with or were accompanied by each other, and should state what the design was.
What Empirical Work in Law Cannot Do
3. What is an operational definition, and why must it be stated? The exact rule the researcher used to decide whether each case counted, for example precisely what counted as an accused being informed. It must be stated because two studies of the same court can produce different figures without either being wrong, if they defined the thing differently.
4. Set out the shape of a defensible recommendation in a legal dissertation. That the law requires a particular thing, established doctrinally; that the study finds it is not happening, established empirically; that something should therefore be done, the ought coming from the law rather than from the data; and that here is the cheapest available measure that would do it, addressed to a body that can act.
The rest of this subject
These notes are cut from the University's printed syllabus. Open the syllabus itself, or the past papers, for the same subject.