Question Types: Objective, Problem and Open Book
Chapter Forty
Syllabus topic 3, "Examination reforms"
Pages 143 to 146 of 543
In one line
Three things can be changed inside the written paper without changing anything else: what kind of question is asked, whether the questions are about doctrine or about facts, and whether the candidate may bring materials in.
In the wording a student can write in an exam: objective type and multiple choice questions test a wide sample of the syllabus with reliable marking but reward recognition rather than reasoning; problem type or situational questions test application and issue identification, which is what a practitioner does, but are harder to set and to mark; and the open book examination removes the premium on memory and tests use of materials, but requires questions that cannot be answered by copying and is often harder for candidates than a closed book paper.
Objective and multiple choice questions
What they are. Short questions with a single correct answer, either supplied from options or written in a word or a line.
Merits. They sample widely, so a paper of sixty objective questions can cover the whole syllabus, which directly answers the sampling defect and kills question spotting. Marking is entirely reliable and can be done at scale. They are fast to answer, so more of the syllabus is reached per hour. They can test precise knowledge that an essay lets a candidate slide over, such as which provision applies and what a section number is. And a well built bank can be reused and analysed.
Demerits. They reward recognition rather than production, and recognising the right answer is easier than generating it. They cannot test argument, structure or judgment, which are what law is. Guessing contributes to the score. A good multiple choice question is very hard to write, and a bad one is ambiguous or tests trivia. And their direction effect is bad: students prepare by memorising lists, which is the opposite of what the rest of this syllabus wants.
Where they belong. As part of a paper rather than the whole of it, testing coverage and precision while other parts test reasoning. The All India Bar Examination is objective, which is a defensible choice for a certifying instrument that has to be marked reliably at national scale.
Problem type and situational questions
What they are. A set of facts and a question asking the candidate to advise, to decide or to state the legal position of the parties.
Merits. They test application, which certification for a profession ought to test. They test issue identification, which chapter 250 identifies as the skill most conspicuously missing. They cannot be answered from a printed guide, because the guide does not contain these facts. They discriminate well, because the range of quality in answers is wide. And they direct students towards the problem method of teaching, which closes the loop between topics 2 and 3.
Question Types: Objective, Problem and Open Book
Demerits. A good problem is expensive to write and must have a defensible answer. Marking is slower and needs a scheme, or it becomes as subjective as an essay. Candidates who have only been taught by exposition are disadvantaged in a way that reflects their teaching rather than their ability, which is a real fairness problem in the first year of a reform. And a problem with one obvious answer is a comprehension exercise.
Where they belong. In every paper, in some proportion. MU already sets them, which is the point chapter 250 makes.
The open book examination
What it is. The candidate may bring specified materials into the hall, usually the bare Acts and sometimes their own notes.
Merits. It removes the premium on memorising what can be looked up, which is most of what a lawyer looks up. It tests use rather than recall, which is closer to practice, where nobody recites a section from memory. It makes trivial questions impossible, so it forces the examiner to ask better ones. It reduces the value of the printed guide, since everyone has materials. And it reduces one motive for malpractice.
Demerits. It is harder to set, because every question must be one the materials do not answer directly. Candidates who rely on searching in the hall run out of time, so it punishes the unprepared more severely than a closed book paper, which surprises students who expect it to be easier. It advantages students with better materials unless the materials are supplied. Invigilation is more complex. And it does not by itself improve marking reliability.
A caution for an answer. An open book examination is not an easy examination, and a candidate who says it reduces the burden on students has misunderstood it. It reduces the burden on memory and increases the burden on understanding.
Choosing between them
Each answers a different defect. Objective questions answer narrow sampling and unreliable marking. Problem questions answer the failure to test application. Open book answers the premium on memory and the reliance on guides.
None answers all of them, and a paper that uses all three in parts is stronger than any one alone: a short objective section for coverage, an essay or two for argument, and a problem for application, with the bare Act permitted.
And the direction effect decides the value of each. Ask what a rational student will do to prepare for each type. For objective questions they will memorise lists; for problems they will practise applying rules to facts; for open book they will try to understand rather than to remember. Ranked by what preparation they induce, the order is the reverse of the order by ease of marking, which is the trade-off in one sentence.
Question Types: Objective, Problem and Open Book
A worked example
One topic, the entitlement to legal aid, examined three ways.
Objective. Under which section of the Legal Services Authorities Act 1987 are the criteria for giving legal services laid down? The answer is a number, marking is instant, and a candidate either knows it or does not.
Essay. Discuss the entitlement to free legal services under the Legal Services Authorities Act 1987. A prepared candidate reproduces an organised account, and a guide is sufficient preparation.
Problem. A woman in custody in Mumbai, whose annual income is below the prescribed limit, is produced before a magistrate and is not asked whether she wants a lawyer. She is convicted. Advise her on her entitlement and on the effect of the omission. This cannot be answered from a summary: it requires the entitlement provisions, the duty to inform and the consequence of failure, and it discriminates sharply between candidates.
Now ask which preparation each induces. A number, an organised summary, and the ability to reason about a person's position. That is the argument of this chapter in one comparison.
Distinctions
| Objective | Problem | Open book | |
|---|---|---|---|
| Answers which defect | Narrow sampling, unreliable marking | Failure to test application | Premium on memory, reliance on guides |
| Tests | Recognition and precision | Issue identification and application | Understanding and use of materials |
| Marking | Wholly reliable, fast | Slower, needs a scheme | As for the question type used |
| Setting cost | High per good question | High | Very high |
| Preparation it induces | Memorising lists | Practising application | Understanding |
Quick revision
Objective and multiple choice: wide sampling, reliable marking, fast, tests precision; but rewards recognition, cannot test argument, allows guessing, is hard to write well, and directs students to memorise lists. The All India Bar Examination is objective.
Problem type: tests application and issue identification, cannot be answered from a guide, discriminates well, and directs students towards the problem method; but is expensive to set, slower to mark, and unfair in the first year of a reform to students taught only by exposition.
Open book: removes the premium on memory, tests use, forces better questions, devalues the guide; but is very hard to set, punishes the unprepared more severely, advantages those with better materials, and is not an easier examination.
Use all three in parts, and judge each by the preparation it induces.
Test yourself
1. Give two merits and two demerits of objective type questions. Merits: they sample the whole syllabus, which ends question spotting, and marking is wholly reliable and fast at scale. Demerits: they reward recognition rather than production and cannot test argument or judgment, and their direction effect is poor because students prepare by memorising lists.
Question Types: Objective, Problem and Open Book
2. Why can a problem question not be answered from a printed guide? Because the guide contains statements of the law and not these facts. The candidate must identify which issues the facts raise, select the applicable rules and apply them, none of which a summary supplies.
3. Correct the statement that an open book examination is easier for students. It is not easier; it reallocates the burden. It removes the burden on memory and increases the burden on understanding, and a candidate who plans to search in the hall runs out of time, so it punishes the unprepared more severely than a closed book paper.
4. What single test should be applied to each question type when judging it? What preparation a rational student will undertake in order to do well at it, since that is what the examination will actually teach.
The rest of this subject
These notes are cut from the University's printed syllabus. Open the syllabus itself, or the past papers, for the same subject.