Healthcare ML: a perfect score can be a broken split

A model scores 100% on its test set. Then the evaluation changes from unseen visits to unseen patients, and accuracy falls to 50%. The model has not changed. The question has.

That is the whole trick in the small Python experiment below. It does not train a diagnostic system or use patient data. It builds an intentionally weak lookup model, then gives it a test split that makes memorization look impressive. The failure is useful precisely because the code is too simple to hide behind a neural network.

For a developer entering healthcare ML, this is a better starting point than choosing an image-classification backbone. First establish what the test set allows a score to mean.

A visit is not an independent patient

The fixture has 40 invented patients, each with two visits. Each patient's binary label stays the same across both visits. Half the patients have label 0 and half label 1. These labels are arbitrary; there is no disease, treatment or biological relationship in the data.

The baseline remembers the label attached to each patient identifier. For any identifier it has not seen, it predicts 0:

def memorizer_accuracy(train, test):
    if not train or not test:
        raise ValueError("both splits need rows")
    labels = {row.patient: row.label for row in train}
    # A lookup baseline, not a clinical model: unseen patients get class 0.
    return sum(labels.get(row.patient, 0) == row.label for row in test) / len(test)

Put the first visit of every patient in training and the second in testing. Every test identifier now has an answer in the lookup table. The baseline gets 100% without learning anything that generalizes to another patient.

Next, train on both visits from patients 0–19 and test on both visits from patients 20–39. None of the test identifiers exists in the lookup table. The baseline always predicts 0 and gets exactly half the labels right.

Download evaluation.py and test_evaluation.py, then run python3 -B evaluation.py. The first two lines are:

row split accuracy: 100%
patient split accuracy: 50%

Those are properties of a deliberately constructed fixture. They are not estimates of how much leakage affects a real dataset. Real leakage can be subtler: multiple images from one examination, recordings from the same device, duplicate notes, or features recorded after the intended prediction time.

Removing the explicit identifier is not a sufficient repair. Other features can carry patient or site identity. The split must reflect the claim you want to make.

Choose the evaluation unit before fitting anything

If the proposed use is prediction for new patients, keep a patient's records together. A split by patient does not establish performance at a new hospital, with another scanner or after a change in coding practice. Those are additional generalization questions, requiring corresponding evaluation data.

For predicting a later visit for an already known patient, patient overlap may be part of the intended setting. Then the important boundary is time: information unavailable at the prediction timestamp must stay unavailable during evaluation. There is no universally correct split independent of intended use.

The example includes an assert_disjoint_patients guard. Call it where a patient-independent split is required, before feature fitting or model selection. In a larger Python pipeline, GroupKFold supports non-overlapping groups across folds. Grouping is a mechanism, not evidence that all remaining dependencies have been removed.

The IMDRF's 2025 good machine learning practice principles explicitly address independence between training and test data, including patient, site and acquisition dependencies. They also call for evaluation data representative of the intended population. That gives a reason to document the split rather than treating it as a random seed buried in a notebook.

A clean split still leaves a denominator problem

Suppose, purely for arithmetic, that a classifier has 90% sensitivity and 95% specificity. What fraction of its positive predictions are correct?

You cannot answer without the prevalence in the population being evaluated. The second part of the script computes expected counts for 10,000 people. It assumes the same sensitivity and specificity at both prevalence levels; that assumption is not evidence that a real classifier would retain its performance.

For the first row, 100 people have the condition under the hypothetical input. Ninety are detected. Of the 9,900 without it, 5% generate a false positive: 495. Positive predictive value is therefore 90 / (90 + 495).

This calculation does not select a clinical threshold. It shows why a sensitivity figure cannot stand in for the composition of an alert queue. The script reports an undefined value when there are no positive predictions rather than returning a misleading zero.

Keep the evidence attached to the score

Seven tests passed on Python 3.11.5 with python3 -B -m unittest -v test_evaluation.py. They detect patient overlap, reproduce both split scores, check the count arithmetic against explicit expected values, and cover perfect classification, empty positive predictions and invalid inputs. No trained clinical model, real health records or treatment outcomes were evaluated.

For an actual model report, retain the prediction time, evaluation unit, cohort definition, outcome definition and split rules alongside the score. Fit preprocessing on training data. Keep a final evaluation set out of repeated tuning. Report counts and uncertainty for relevant subgroups rather than assuming an overall average describes everyone.

A stronger model is worth investigating once this evaluation can reject a weak one. Until then, a perfect score may only show that the test set remembered the patients too.