Using an AI Library Responsibly

SVC(kernel='rbf') is a decision, not a default. The Kernel Trick showed what a kernel does and what it costs. Choosing one without being able to say what shape of boundary it allows is choosing at random.

RandomForestClassifier gives up interpretability. Transparency and Explainability measured the trade: one readable tree at 0.6657 against an unreadable forest at 0.7450. That is a real gain and it is a real loss, and the loss is invisible until someone asks why a person was refused.

The shape of a scikit-learn program

X, y = features, labels                 # X: rows by columns, y: one label per row
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.2, stratify=y, random_state=0)
scaler.fit(Xtr); Xtr = scaler.transform(Xtr); Xte = scaler.transform(Xte)
model = SomeClassifier(**chosen_hyperparameters)
model.fit(Xtr, ytr)                     # the learning
print(classification_report(yte, model.predict(Xte)))
print(confusion_matrix(yte, model.predict(Xte)))

Four things in that form carry the whole responsibility of the chapter, and every one of them is a line a beginner deletes.

stratify=y. Without it, an unbalanced problem can put almost none of the rare class in the test set, and Evaluating a Model showed what unbalanced data does to an accuracy figure.

random_state=0. Without a fixed seed the split changes every run, so the score changes every run, and a comparison between two models measures the split rather than the models.

scaler.fit(Xtr) and not scaler.fit(X). Fitting the scaler on all the data lets the test set's mean and spread into the training, which is data leakage: the score improves and the model does not. The rule is absolute: anything fitted must be fitted on the training data alone.

classification_report and confusion_matrix rather than score. Evaluating a Model measured a classifier at 0.9700 accuracy that was worthless. A single number is not a result.

The shape of a TensorFlow or Keras program

model = Sequential([Dense(32, activation='relu'), Dense(n_classes, activation='softmax')])
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])
history = model.fit(Xtr, ytr, epochs=50, validation_split=0.2, verbose=0)
# the two curves in history are the point: training loss falling while
# validation loss rises IS overfitting, seen as it happens
loss, acc = model.evaluate(Xte, yte)

The comment is the content. The validation curve turning upward while the training curve keeps falling is the picture Overfitting and Underfitting drew, and it is the one thing a practical write-up should contain and usually does not.

And the honest note about epochs=50: it is a guess. Early stopping, halting when the validation loss stops improving, is the principled version, and it is one argument to the same call.

The shape of a call to a hosted model

key = os.environ["OPENAI_API_KEY"]      # NEVER written in the source file
reply = client.responses.create(model=..., input=prompt)
# everything in `prompt` has left the machine. treat it as published.

Three rules, and the first two are absolute.

The key is a secret. It goes in an environment variable or a secrets manager, never in the source and never in a repository. A key in a public repository is found by automated scanners in minutes and billed to its owner.

The prompt leaves the machine. Anything sent is disclosed to a third party: a classmate's marks, a patient record, a friend's message, an unpublished manuscript. Treat the prompt as published, because for the purposes of the person whose data it was, it is.

The reply is unverified. Hallucination in Generative AI measured why: the model scores how ordinary a sequence of words looks and a false sentence of ordinary phrases scores well. Every specific claim, number, citation and date in a reply must be checked against a source before it is used.

The checks before a result is believed

Ten, in order, and this list is the answer to "how would you validate your practical's result".

Check 8 is the one that quietly ruins student projects. Trying twenty models and reporting the best test score reports the maximum of twenty noisy numbers. Hyperparameters are chosen on a validation set or by cross validation inside the training data, and the test set is touched once, at the end.

Responsibilities that are not about accuracy

Distinctions

What it does not mean

A library call is not a black box. Every one in the table is an algorithm in this book.

A default is not a choice. A kernel, a depth, a learning rate and a number of epochs are all decisions.

A high score is not a result. Compare it with the baseline first.

Fitting a scaler on all the data is not harmless. It is leakage and it inflates every number afterwards.

Choosing a model on the test set is not evaluation. It makes the test set training data.

An API key in a file is not private. It is found by scanners within minutes of being published.

A prompt is not private either. Treat it as published.

A model's reply is not a source. Check every specific claim against one.

Quick revision

• Every library call is a chapter of this book. LinearRegression, SGDClassifier, Ridge, DecisionTreeClassifier, RandomForestClassifier, AdaBoostClassifier, GaussianNB, KNeighborsClassifier, SVC, KMeans, AgglomerativeClustering, cross_val_score, StandardScaler, Sequential with Dense.
• The four lines that carry the responsibility: stratify, random_state, fit the scaler on the training data only, and report a confusion matrix, not a score.
• Data leakage: anything fitted must be fitted on the training data alone.
• In Keras, the training and validation curves are the result: validation rising while training falls is overfitting. Early stopping replaces a guessed epoch count.
• Hosted models: the key is a secret, the prompt leaves the machine and should be treated as published, and the reply is unverified.
• The ten checks: baseline, genuinely held-out test data, duplicates, a feature encoding the answer, the confusion matrix, per-group scores, the spread across folds, hyperparameters not chosen on the test set, survival of a different seed, and not reporting the training score.
• Choosing among twenty models by test score reports the maximum of twenty noisy numbers. Tune on a validation set; touch the test set once.
• Beyond accuracy: licence, data provenance, personal data under the DPDP Act, 2023, attribution, reproducibility (versions and seed), cost, and scope.

Test yourself

1. A practical uses RandomForestClassifier. What is it doing, in the terms of this book? Growing many decision trees, each on a bootstrap sample of the rows drawn with replacement and each restricted at every split to a random subset of the features, then combining them by majority vote. The random feature subset makes the trees less alike so that their errors cancel, at the cost of each tree being individually worse and of the whole model no longer being readable as a set of rules.

2. Why must a scaler be fitted on the training data only? Because fitting it on all the data uses the test set's mean and spread to transform the training data, so information from the test set enters the model. That is data leakage: the reported score improves while the model does not, and the improvement disappears on genuinely new data.

3. Name four things in a scikit-learn pipeline a beginner omits and say why each matters. Stratifying the split, without which an unbalanced problem may place almost none of the rare class in the test set; fixing the random seed, without which the score changes every run and comparisons measure the split rather than the model; fitting transformations on the training data alone, without which there is leakage; and reporting a confusion matrix and per-class report rather than a single accuracy, since an accuracy of 0.97 was shown earlier to belong to a worthless classifier.

4. What should a practical write-up show from a Keras training run? The training and validation curves together. Training loss falling while validation loss rises is overfitting, visible as it happens, and it is the one piece of evidence that shows whether the chosen number of epochs was right. Early stopping on the validation loss replaces the guess.

5. Give three rules for calling a hosted model. The API key is a secret and belongs in an environment variable or secrets manager, never in source or a repository, since published keys are found by automated scanners within minutes. Everything in the prompt leaves the machine and should be treated as published, so another person's data must not be sent without their consent. And the reply is unverified text produced by a process that scores how ordinary a word sequence looks, so every specific claim, number, citation and date must be checked against a source.

6. Why is reporting the best test score among twenty tried models misleading? Because it reports the maximum of twenty noisy measurements, which is biased upward, and because each comparison used the test set to make a choice, so the test set has become part of the training process. Hyperparameters must be selected on a validation set or by cross validation within the training data, and the test set used once, at the end.

7. List the checks you would make before believing a classification result. Compare it with the baseline given by the largest class or the simplest rule; confirm the test data was genuinely held out and that nothing was fitted on it; look for duplicate rows across the split; check that no feature records the answer or something recorded after it; read the confusion matrix and decide which error costs more; compute the scores separately for each group; report the spread across folds and not only the mean; confirm the hyperparameters were not chosen on the test set; repeat with a different seed; and make sure the number quoted is not the training score.