Evaluating a Model

And because two numbers are awkward to compare, they are combined as the F1 score, their harmonic mean:

The harmonic mean is used and not the ordinary average because it punishes an imbalance. Precision 1.0 with recall 0.0 has an ordinary average of 0.5 and an F1 of 0, which is the honest figure for a model that never flags anything.

The three classifiers, scored

# Evaluating a model, licensed by MU's own Course Outcome 3. Why accuracy lies
# on unbalanced data, the confusion matrix, precision, recall, F1, and k-fold
# cross validation, all computed.
def confusion(truth, pred):
    tp = sum(1 for t, p in zip(truth, pred) if t == 1 and p == 1)
    fn = sum(1 for t, p in zip(truth, pred) if t == 1 and p == 0)
    fp = sum(1 for t, p in zip(truth, pred) if t == 0 and p == 1)
    tn = sum(1 for t, p in zip(truth, pred) if t == 0 and p == 0)
    return tp, fn, fp, tn

def report(name, truth, pred):
    tp, fn, fp, tn = confusion(truth, pred)
    n = len(truth)
    acc = (tp + tn) / n
    prec = tp / (tp + fp) if tp + fp else 0.0
    rec = tp / (tp + fn) if tp + fn else 0.0
    f1 = 2 * prec * rec / (prec + rec) if prec + rec else 0.0
    spec = tn / (tn + fp) if tn + fp else 0.0
    print("   %s" % name)
    print("      confusion matrix          predicted spam   predicted not")
    print("         actually spam      %14d %15d" % (tp, fn))
    print("         actually not       %14d %15d" % (fp, tn))
    print("      accuracy  = (%d + %d)/%d          = %.4f" % (tp, tn, n, acc))
    print("      precision = %d/(%d + %d)            = %.4f" % (tp, tp, fp, prec))
    print("      recall    = %d/(%d + %d)            = %.4f" % (tp, tp, fn, rec))
    print("      specificity = %d/(%d + %d)          = %.4f" % (tn, tn, fp, spec))
    print("      F1 = 2*P*R/(P+R)                 = %.4f" % f1)
    return acc, prec, rec, f1

# 1000 messages, 30 of them spam: an UNBALANCED problem, which is the usual case.
truth = [1] * 30 + [0] * 970
print("1000 messages, %d of them spam. three classifiers, scored:" % sum(truth))
print()
lazy = [0] * 1000
report("A: mark EVERYTHING as not spam. it never catches anything.", truth, lazy)
print()
eager = [1] * 1000
report("B: mark EVERYTHING as spam. it never misses anything.", truth, eager)
print()
real = [1] * 24 + [0] * 6 + [1] * 12 + [0] * 958
report("C: a real filter: catches 24 of 30, and wrongly flags 12 good messages.",
       truth, real)
print()
print("classifier A has an accuracy of 0.9700 and is WORTHLESS. that single")
print("number is why MU's Course Outcome 3 says EVALUATE and not just implement.")
print()
print("PRECISION AND RECALL PULL AGAINST EACH OTHER. the same filter at three")
print("thresholds, from cautious to aggressive:")
print("   threshold | caught of 30 | good mail lost | precision | recall | F1")
for caught, lost in ((10, 1), (24, 12), (29, 90)):
    pred = [1] * caught + [0] * (30 - caught) + [1] * lost + [0] * (970 - lost)
    tp, fn, fp, tn = confusion(truth, pred)
    prec = tp / (tp + fp)
    rec = tp / (tp + fn)
    f1 = 2 * prec * rec / (prec + rec)
    label = "cautious" if caught == 10 else ("balanced" if caught == 24 else "aggressive")
    print("   %-9s | %12d | %14d | %9.4f | %6.4f | %.4f"
          % (label, caught, lost, prec, rec, f1))
print()
print("   cautious: high precision, poor recall. aggressive: the reverse.")
print("   which to prefer is a DECISION about cost, not a property of the model:")
print("   losing a wanted message is worse than seeing a spam one, so a mail")
print("   filter is tuned for precision, and a cancer screen for recall.")
print()
print("K-FOLD CROSS VALIDATION. 20 students, hours studied against pass, with")
print("three labels deliberately wrong. the learner is the best threshold on the")
print("TRAINING rows only; it is then scored on the fold held out.")
print()
# hours, passed. The honest rule is 'pass if hours >= 8'. Rows 5, 11 and 16 are
# WRONG on purpose, so no threshold can score 1.0 and the folds must disagree.
ROWS = [(1, 0), (2, 0), (3, 0), (4, 0), (5, 1), (6, 0), (7, 0), (8, 1),
        (9, 1), (10, 1), (11, 0), (12, 1), (13, 1), (14, 1), (15, 1),
        (16, 0), (17, 1), (18, 1), (19, 1), (20, 1)]

def fit_threshold(train):
    """The threshold with the fewest training mistakes: predict pass if >= t."""
    best = None
    for t in range(1, 22):
        wrong = sum(1 for h, y in train if (1 if h >= t else 0) != y)
        if best is None or wrong < best[0]:
            best = (wrong, t)
    return best[1]

def score(t, rows):
    return sum(1 for h, y in rows if (1 if h >= t else 0) == y) / len(rows)

k = 5
size = len(ROWS) // k
scores = []
print("   fold | tested on hours     | threshold learnt | accuracy on the fold")
for i in range(k):
    test = ROWS[i * size:(i + 1) * size]
    train = ROWS[:i * size] + ROWS[(i + 1) * size:]
    t = fit_threshold(train)
    acc = score(t, test)
    scores.append(acc)
    print("   %4d | %-19s | pass if >= %-5d | %.4f"
          % (i + 1, " ".join(str(h) for h, _ in test), t, acc))
mean = sum(scores) / k
var = sum((s - mean) ** 2 for s in scores) / k
print("   mean accuracy %.4f, spread %.4f" % (mean, var ** 0.5))
print()
print("   every row is tested EXACTLY ONCE, and every row is trained on k-1")
print("   times. the five folds scored %s." % ", ".join("%.2f" % s for s in scores))
print("   a single train/test split is one of those five numbers, chosen by")
print("   accident: it would have reported anything from %.2f to %.2f. the mean"
      % (min(scores), max(scores)))
print("   of five is far more stable, and the SPREAD says how much a single")
print("   number should be trusted.")
print()
print("   for comparison, the threshold fitted on ALL twenty rows is pass if")
t_all = fit_threshold(ROWS)
print("   hours >= %d, and it scores %.4f on those same twenty rows. that number"
      % (t_all, score(t_all, ROWS)))
print("   is training accuracy and means nothing: the model has seen every row.")

Reading the scores

The three classifiers say everything the section needs.

A has the second-highest accuracy in the table and an F1 of zero. Precision and recall are both 0, because it never produces a true positive, and F1 collapses to 0 while accuracy stays at 0.9700. That is the single most useful thing in this chapter: F1 exposes what accuracy hides.

And B is the opposite trap. It has perfect recall: it never misses a spam message, because it flags everything. A perfect recall on its own is worthless, and so is a perfect precision, which is why the pair is always quoted together or through F1.

C is the only real model. Note that its accuracy, 0.9820, is only 1.2 points above the useless A, while its F1 is 0.7273 against 0.0000. On an unbalanced problem the accuracy scale is compressed into almost nothing, and a difference that matters enormously looks small.

The trade-off, measured

Precision and recall are not independent quantities that happen to differ. They pull against each other, and the run shows it on one filter at three settings.

Moving the threshold moves both numbers in opposite directions. Flag less and you are right more often about what you flag, and you miss more. Flag more and you catch nearly everything at the cost of ninety wanted letters.

There is no best row. Which to choose depends on which mistake costs more, and that is not something the data can decide:

• A mail filter is tuned for precision. Losing a wanted letter is far worse than seeing an advertisement.
• A cancer screen is tuned for recall. A false alarm costs a second test; a miss costs a life.
• A search engine's first page is tuned for precision; a lawyer's document discovery is tuned for recall.

F1 picked the balanced row here, but F1 weights the two equally, which is an assumption and usually the wrong one. When one error genuinely costs more, say so and use the measure that reflects it.

Cross validation, and why one split is not enough

A single train and test split gives one number, and that number depends on which rows happened to be held out. k-fold cross validation removes the accident: split the data into k equal folds, train on k - 1 of them and test on the one left out, k times, then average.

Every row is tested exactly once and trained on k - 1 times, which is the property to state in an answer.

The run does it properly: the threshold is fitted on the training rows of each fold alone and then scored on the rows held out.

Mean 0.8000, spread 0.1871.

Read fold 2, which is the most instructive line in the chapter. With the four rows from 5 to 8 hours held out, the training data contains no example between 4 and 9 hours, so every threshold in that gap makes exactly the same training mistakes. The thresholds are tied, the tie is broken arbitrarily, pass if >= 5 is chosen, and on the held-out fold it scores 0.5000 where pass if >= 8 would have scored 0.7500.

Two lessons, both worth marks. A tie on the training data is a real event, and what breaks it is the implementation, not the learning. And the held-out fold is exactly where such a choice shows up, which is the argument for cross validation in a single example.

The spread is the point of the whole method. The five folds scored 1.00, 0.50, 0.75, 0.75 and 1.00. A single split would have reported any one of those, so a paper quoting "accuracy 1.0000" and a paper quoting "accuracy 0.5000" could describe the same model on the same data. The mean of five is stable; the spread of 0.1871 is the honest statement of how little a single figure should be trusted.

And the last line of the run: the threshold fitted on all twenty rows scores 0.8500 on those same twenty rows. That number is training accuracy and means nothing, because the model has seen every row it is being judged on. Overfitting and Underfitting made the same point; here it is one line of output.

Conventional choices: k = 5 or k = 10. Leave-one-out is k = n, the most thorough and the most expensive. Stratified folds keep each class in its original proportion in every fold, which matters on exactly the unbalanced data this chapter began with.

Distinctions

What it does not mean

A high accuracy is not a good model. 0.9700 here is what a program with no model scores.

Precision and recall are not two views of the same quantity. They have different denominators and move in opposite directions as the threshold moves.

A perfect recall is not success. Flag everything and recall is 1.0000.

F1 is not the ordinary average. It is the harmonic mean, so one low number drags it down.

F1 is not neutral. It weights the two errors equally, which is itself a choice, and usually not the right one.

Cross validation does not improve the model. It estimates the model's performance more reliably. Nothing is trained better by being cross validated.

Training accuracy is not a score. The model has seen those rows.

Quick revision

• Accuracy lies on unbalanced data. 30 spam in 1000: marking everything not spam gives 0.9700 and an F1 of 0.0000. Always read accuracy against the largest class's share.
• The confusion matrix: TP, FN (a miss), FP (a false alarm), TN. The two mistakes have different costs, and choosing between them is a decision outside the model.
• precision = TP/(TP+FP), by what the model said. recall = TP/(TP+FN), by what the truth is. specificity = TN/(TN+FP). F1 = 2PR/(P+R), the harmonic mean, which punishes imbalance.
• Measured trade-off: cautious 0.9091 / 0.3333, balanced 0.6667 / 0.8000, aggressive 0.2437 / 0.9667. Mail filters favour precision, cancer screens recall.
• k-fold cross validation: k folds, train on k - 1, test on 1, k times, average. Every row tested once, trained on k - 1 times.
• Measured: folds 1.00, 0.50, 0.75, 0.75, 1.00, mean 0.8000, spread 0.1871. A single split would have reported anything in that range.
• Fold 2 learnt >= 5 instead of >= 8 because its training rows had a gap and the thresholds tied; the arbitrary tie-break cost it 0.25 on the held-out fold.
• The threshold fitted on all 20 rows scores 0.8500 on those 20 rows, which is training accuracy and means nothing.
• k = 5 or 10 conventionally; k = n is leave-one-out; stratified folds preserve class proportions.

Test yourself

1. A spam filter marks every one of 1000 messages as not spam, 30 of which are spam. Give its accuracy and its F1, and say what the figures show. Accuracy is 970/1000 = 0.9700. It never produces a true positive, so precision and recall are both 0 and F1 is 0.0000. A high accuracy on unbalanced data measures how common the majority class is, not how good the model is, so accuracy must always be read against the share of the largest class.

2. Draw the confusion matrix and name its four cells. Rows are the truth, columns the prediction. Actually positive and predicted positive is a true positive; actually positive and predicted negative is a false negative, a miss; actually negative and predicted positive is a false positive, a false alarm; actually negative and predicted negative is a true negative.

3. Define precision and recall, and state the difference in their denominators. Precision is TP/(TP+FP), the fraction of the items the model flagged that should have been flagged. Recall is TP/(TP+FN), the fraction of the items that should have been flagged that were. Precision divides by what the model said; recall divides by what the truth is.

4. Why is F1 the harmonic mean rather than the ordinary average? Because the harmonic mean punishes an imbalance between the two. A model with precision 1.0 and recall 0.0 has an ordinary average of 0.5, which flatters it, and an F1 of 0, which is the honest figure.

5. A model can be tuned for precision or for recall. Give one application of each and justify. A mail filter is tuned for precision, because a wanted letter in the junk folder costs far more than one advertisement in the inbox. A cancer screen is tuned for recall, because a false alarm costs a second test while a miss can cost a life. The choice is about the cost of the two errors and cannot be made from the data.

6. Describe k-fold cross validation and state what it gives that a single split does not. Divide the data into k equal folds; for each fold in turn, train on the other k-1 and test on it; average the k scores. Every row is tested exactly once and trained on k-1 times. It gives a mean that does not depend on which rows were held out, and a spread that says how much a single figure should be trusted: the folds in this chapter ran from 0.50 to 1.00 around a mean of 0.8000.

7. In this chapter's cross validation, one fold learnt a different threshold and scored 0.5000. Explain. Its four held-out rows were the only training examples between 4 and 9 hours, so with them removed every threshold in that gap made exactly the same training mistakes. The thresholds were tied, the tie was broken arbitrarily in favour of the lowest, and on the held-out rows that choice scored 0.5000 where the better threshold would have scored 0.7500. The training data could not distinguish the two; only the held-out fold could.