Testing a Hypothesis
Chapter -Two
Syllabus topic 6, "Identification of Research Problem and formulation of Hypothesis."
Pages 442 to 445 of 543
In one line
Testing a hypothesis means confronting it with evidence gathered in a way that could have gone against it, and reporting what happened whichever way it went.
In the wording a student can write in an exam: testing a hypothesis consists in collecting evidence capable of refuting it and determining whether the evidence supports or fails to support the proposition; in empirical work this involves comparing the observed result with what the hypothesis predicted and, where a statistical test is used, deciding whether the difference could reasonably have arisen by chance; in doctrinal work it involves testing the proposition against the authorities, including those which tell against it.
Testing in doctrinal work
The hypothesis. A proposition about what the law is or what it requires.
The evidence. The provisions and the decisions.
The test, and this is the part students omit. Not looking for authority that supports the proposition, which is advocacy, but looking for authority that would refute it. A proposition survives testing when the contrary authorities have been found, read and either distinguished or accepted as fatal.
A worked instance from this book. The proposition that the Bar Council of India cannot impose an examination as a condition of practice was supported by V. Sudeer for twenty-four years. Testing it required looking for later material, which produces Bonnie Foi and destroys it, chapter 950.
So doctrinal testing has a procedure: state the proposition; find the authority for it; find the authority against it; check that both are current; and state the position with the conflict shown rather than smoothed, chapter 840.
Testing in empirical work
The hypothesis predicts something about the data before the data exists.
Collection produces the number.
And the comparison is made. Did the observed proportion fall below the threshold, or not? Did the relationship run in the predicted direction, or not?
Three outcomes, and all three must be reported. The evidence supports the hypothesis; the evidence is against it; or the evidence is inconclusive, which is a real outcome and usually means the study was too small or the instrument too blunt.
And the reporting must be symmetrical. A dissertation that describes the supported hypothesis in three pages and the refuted one in a sentence has told the reader what the author wanted rather than what happened.
When a statistical test is needed, and when it is not
It is not needed for a descriptive hypothesis with a threshold. If the hypothesis was that the proportion is below one quarter and the observed proportion is well below it in a complete count, the hypothesis is supported and no test is required. A census, chapter 990, needs no inference at all, because there is no sample to generalise from.
Testing a Hypothesis
It is needed when a sample is used to say something about a population, and when the question is whether an observed difference is larger than chance would produce.
And a student should not use one they cannot explain, chapter 1040. Presenting the cross table and describing the pattern is honest and adequate.
The logic of significance, in plain words
The question a test asks. If there were really no difference in the population, how likely is it that a sample would show a difference as large as this one?
If that likelihood is very small, the researcher concludes that the assumption of no difference is hard to maintain, and rejects the null, chapter 1140.
If it is not small, the null is not rejected, which means the evidence was not strong enough to displace it and not that it has been shown true.
The level at which small is fixed is a convention chosen in advance, and choosing it after seeing the result is not permitted.
And significance is not importance. A difference can be statistically significant and too small to matter, particularly in a large sample; and a large, important difference in a small sample may not reach significance. A researcher must report the size of the difference as well as whether it was significant.
The two errors
Type I error: rejecting a null hypothesis that is in fact true. Concluding that there is a difference when there is not. The researcher has found something that is not there.
Type II error: failing to reject a null hypothesis that is in fact false. Concluding that the evidence does not show a difference when there really is one. The researcher has missed something that is there.
The trade-off. Making the test stricter reduces Type I and increases Type II, and the reverse.
And what reduces both. A larger sample, a better instrument and less measurement error.
The legal analogy that makes it memorable and should be used carefully. A criminal trial is designed to make one error much less likely than the other: convicting the innocent is treated as far worse than acquitting the guilty, so the standard of proof is set high. Research makes the same kind of choice about which error it can better afford, and in a study whose finding will support a reform proposal, wrongly reporting a problem that does not exist is usually the worse error.
What testing does not mean
It does not mean proving. MU uses the word proving, chapter 1120, and a candidate should use it and then explain that research supports or fails to support.
It does not mean the researcher decides. The design decides, and it was fixed before the data arrived.
Testing a Hypothesis
It does not permit changing the hypothesis after the result. Restating the hypothesis to fit what was found, without saying so, is the most serious form of dishonesty available in empirical work, because nothing in the finished document reveals it.
And it does not mean a single study settles anything. A finding is one study in one place, chapter 1050.
A worked example
H1, descriptive. In the study court over six weeks, the accused was informed in fewer than one quarter of remand productions.
The test. A complete count of every production in the period. Suppose 217 productions and information given in 31, which is about 14 per cent.
The conclusion. The hypothesis is supported. No statistical test is needed, because this is a complete count of the defined population and no inference to a wider population is being made. The dissertation says exactly that, and says that the finding does not extend to any other court.
H2, relational. The proportion informed falls as list length rises.
The test. Sittings grouped by size, and the proportion informed computed for each group.
Suppose the proportions are close. The honest report: the data does not show the expected pattern; the numbers in the largest group were small; and the hypothesis is not supported on this evidence, without concluding that there is no relationship.
Notice the last clause. Not rejecting is not proving the null, and writing that sentence is what distinguishes a careful researcher from a confident one.
Quick revision
Testing means confronting a hypothesis with evidence that could have gone against it, and reporting what happened either way.
Doctrinal testing: state the proposition; find authority for it; find authority against it; check both are current; state the position with the conflict shown.
Empirical testing: the hypothesis predicts, collection produces, and the comparison is made. Three outcomes, all reported symmetrically: supported, refuted, or inconclusive.
A statistical test is not needed for a complete count with a stated threshold; it is needed to generalise from a sample. Do not use one you cannot explain.
Significance: how likely a difference this large would be if there were none. Not rejecting is not proving the null. And significance is not importance, so report the size of the difference too.
Two errors: Type I, finding something that is not there; Type II, missing something that is. Stricter tests trade one for the other; a larger sample and a better instrument reduce both.
Never change the hypothesis after the result, since nothing in the finished document would reveal it.
Test yourself
1. What distinguishes testing a doctrinal proposition from arguing for it? Testing requires looking for the authority that would refute the proposition, not only for the authority that supports it. A proposition survives testing when the contrary authorities have been found, read and either distinguished or accepted as fatal.
Testing a Hypothesis
2. When is a statistical test unnecessary? When the study is a complete count of the defined population and the hypothesis stated a threshold, since there is no sample and no inference to a wider population. A test is needed only where a sample is used to say something about a population, or where the question is whether a difference exceeds what chance would produce.
3. Define the two errors and say what reduces both. A Type I error is rejecting a true null hypothesis, that is, finding a difference that is not there. A Type II error is failing to reject a false null hypothesis, that is, missing a difference that is there. Making a test stricter trades one for the other; a larger sample, a better instrument and less measurement error reduce both.
4. Why is changing the hypothesis after seeing the result the most serious form of dishonesty in empirical work? Because nothing in the finished document reveals it. A reader cannot tell that the proposition was written to fit the data, so the safeguards of falsifiability and of stating the expectation in advance are defeated without any visible trace.
The rest of this subject
These notes are cut from the University's printed syllabus. Open the syllabus itself, or the past papers, for the same subject.