Bug Tracking

The life of a bug

New when it is reported. Confirmed when someone else reproduces it, which is the step that catches a report that is really a misunderstanding. In progress when someone owns it. Fixed when there is a commit. Verified when the person who found it runs the case again and it passes. Closed only then.

Two states worth having, and using honestly: Won't fix, with the reason, for something real that the team decides not to do; and Not a bug, for behaviour that turns out to be correct. Both are better than a silently ignored issue.

Using GitHub Issues

MU makes GitHub mandatory (Chapter 1), and its issue tracker needs no setting up:

• The title is the symptom, not the suspected cause: "Sold-out message ends with stray numbers", not "showErrors is broken". The cause is often wrong, and the symptom is what everyone recognises.
• Labels carry the facts a filter needs: bug, severity: medium, area: frontend, and found: system testing.
• One issue, one fault. Two faults in one issue can never be half-closed.
• A template in .github/ISSUE_TEMPLATE gives everyone the same fields, so nothing is left out at midnight.
• The commit closes it: a message with "Closes #31" links the fix to the report and closes the issue when it is merged, which is also what a guide reads at the code review (Chapter 49).

The board at the end of the project is evidence for the examiner: issues opened across the weeks, each closed by a commit, is a history no downloaded project has.

The worked defect, from report to closed

Reported. On 7 October, running TC-12 by hand in a browser, Aditi saw the message above. She wrote issue 31 as printed in the table.

Confirmed. Rohan reproduced it in a minute, and found a second case at once: after five failed sign-ins the message read "Too many failed sign-ins. Try again later. 900". Same fault, different screen. He added the second case to the issue rather than opening another, because one fix would cure both.

The cause. The page helper put every entry of the error's details on the page. For a refusal of input those details are the messages for each field, which is exactly what should be shown; for a refusal of anything else they are facts for the program, the item's id and the stock left, or the seconds to wait.

The fix. One line, in the helper: only a refusal whose code is invalid_input carries per-field messages; every other refusal shows its message alone.

export function showErrors(form, err) {
  clearErrors(form);
  const fields = err.code === 'invalid_input' ? err.details : {};
  const unplaced = [];
  for (const [name, text] of Object.entries(fields)) {

Verified. Aditi ran TC-12 again, in the browser, and the message read "Only 2 Chicken Biryani left." Rohan ran the sign-in case. The issue was closed by the commit that made the change.

And a test was added, because a bug that has happened once can happen again:

    assert.equal(res.status, 409);
    assert.equal(res.body.error.message, 'Only 2 Veg Biryani left.');

That assertion is on the message, which is what the defect was about. The automated test that existed before checked the status and the stock, and would have passed for ever.

Why this one is worth studying

It is an ordinary defect, and it shows four things students meet:

1. It was not in the code that was changed. The helper had been right for every case anyone had tried; a new kind of error made it wrong.
2. The tests passed. All of them. Nothing was checking what a person reads.
3. It was found by a human running a written case, which is the argument for Chapter 54's kind of testing in one sentence.
4. The fix made the rule explicit, rather than special-casing the symptom: "only a refusal of input has per-field messages" is now a sentence in the code, with a comment saying why.

Do this for your project

1. Open an issue for every defect, however small, and one issue per fault.
2. Write the steps so that someone else can see it in a minute, with the values you used.
3. Always write both what happened and what should have happened.
4. Set severity and priority, and say in your test plan which severities may remain open.
5. Have someone else confirm it before anyone fixes it.
6. Close the issue with the commit that fixes it, and verify by running the case again.
7. Add a test that fails on the old code, so the bug cannot come back.

Mistakes that cost marks

"It doesn't work" as a bug report.

A cause in the title, which sends the next reader to the wrong file.

Several faults in one issue, so it stays open for weeks.

Fixed without a test, so the same bug returns in the report's screenshots.

Issues closed with no commit, so nobody can see what changed.

A tracker with nothing in it at the end of a semester's work, which says either that nothing was tested or that nothing was recorded.

Quick revision

• A report: number, title (the symptom), where, steps with values, what happened, what should happen, severity, who found it, environment, version.
• Severity is how bad; priority is when. The test plan says which severities may stay open.
• Life: new, confirmed, in progress, fixed, verified, closed; plus won't fix and not a bug, with reasons.
• GitHub: labels, one issue one fault, a template, and "Closes #31" in the commit.
• Every fix gets a test that fails on the old code.
• The worked defect: every detail of an error printed on the page; every test passed; found by a person running a written case.

Questions you must be able to answer

1. What must a bug report contain to be useful? A number, a title naming the symptom, where it happens, exact steps with the values used, what happened, what should have happened, the severity, who found it and how, and the environment and version. The two accounts, actual and expected, are what make it judgeable.

2. What is the difference between severity and priority? Severity is how serious the fault is in itself, from cosmetic to data loss. Priority is how soon it will be fixed, which depends on what else is happening: an ugly message the week of the examination may be fixed before a rare crash.

3. Why should someone other than the reporter confirm a defect? Because it catches reports that are really misunderstandings or environment problems, and because the second person often finds another case of the same fault, as happened here with the sign-in limit's message.

4. Why add a test when fixing a bug? Because the bug proves that nothing was checking that behaviour. A test that fails on the old code and passes on the new one makes the fix permanent; without it, the same fault can return in a later change.

5. The worked defect passed every automated test. What does that teach? That a suite checks only what it was told to check. These tests asserted the status, the code and the stock, and none of them looked at the sentence a student reads, which is why a person running a written case in a browser is a level of testing that cannot be skipped.

6. What does an examiner learn from your issue tracker? Whether the project was tested and managed over the semester: issues opened across the weeks, each with steps and a severity, confirmed by someone else, closed by a commit that names them, with tests added. It is evidence that cannot be produced at the end.