Predicting Exam Outcomes in Ruby: Split Before You Scale

A model can see information from the evaluation set before its first training step. Fitting a scaler on the complete dataset is enough: the held-out rows help choose the means and standard deviations used for training.

This Rumale experiment keeps that boundary visible. It predicts a binary pass outcome from two synthetic features, fits preprocessing only on the training cohort, evaluates an untouched holdout cohort and reports the confusion counts. The fixture contains no real students. Its scores test the Ruby pipeline; they provide no evidence that it can predict a school's outcomes or improve them.

Define the target before choosing the estimator

The target is passed, encoded as zero or one. Attendance percentage and practice hours are the two input columns. An exam mark such as 80 would be a different target: treating arbitrary numeric marks as classifier labels does not make the resulting output a probability of passing.

Each CSV row explicitly belongs to train or holdout. Twelve rows train the model; six rows evaluate it. The separation is fixed before any scaler or classifier is fitted. The synthetic cohorts are supplied in students.csv, so the split is reproducible and inspectable.

A real study would need a defensible collection and split design: for example, features available at the intended prediction time and later cohorts held out from fitting. Rows from the same student or information recorded after the exam could defeat a nominal train/test split. This fixture is too small and artificial to evaluate those educational questions.

Parse a number, or reject the row

The loader uses Float(...), then checks finiteness and declared ranges. Attendance must lie between zero and one hundred; practice hours between zero and forty in this fixture's schema. Missing cells, unknown strings, infinities and out-of-range values fail explicitly. Turning an empty string into zero would assert a measurement that was never present.

The allowed ranges are input assumptions, not learned bounds. The fixture also validates binary labels and known cohort names. Its tests include malformed rows so a future parser change cannot silently broaden the accepted data without showing a difference.

The tested stack is Ruby 3.3.2, Rumale 0.28.1 and its pinned native dependencies. The downloadable Gemfile, lockfile and README describe reproduction. The version pin matters because online API documentation may describe a newer release.

The scaler belongs to the fitted model

Rumale's StandardScaler API separates fitting from transformation. The experiment calls fit on training features, transforms those features for the classifier and later transforms held-out features using the same stored statistics.

def self.fit(rows)
  raise ArgumentError,'fit accepts training cohort only' unless rows.all? { |r| r['cohort']=='train' }
  x=Numo::DFloat.asarray(rows.map { |r| features(r) })
  y=Numo::Int32.asarray(rows.map { |r| Integer(r.fetch('passed')) })
  raise ArgumentError,'both classes required' unless y.to_a.uniq.sort == [0,1]
  scaler=Rumale::Preprocessing::StandardScaler.new
  scaler.fit(x)
  raise ArgumentError,'constant or invalid training feature' unless scaler.std_vec.to_a.all? { |v| v.finite? && v>0 }
  classifier=Rumale::LinearModel::LogisticRegression.new(reg_param: 0.1,max_iter: 1000,tol: 1e-8)
  classifier.fit(scaler.transform(x),y)
  [scaler,classifier]
end

The implementation rejects constant training features because this pinned scaler divides by its estimated standard deviation. Dividing by zero is not a meaningful normalization policy. A larger pipeline might drop constant columns, but that decision would need to be saved with the feature schema and applied consistently at inference.

The recorded training means are approximately 69.1667 attendance percentage and 5.5833 practice hours. The tests calculate those values from training rows alone, call prediction on the holdout and verify that the means remain unchanged. The fit function also rejects a mixed cohort, making this particular leakage mistake harder to introduce accidentally.

A probability needs its class label

The classifier is Rumale's logistic regression with an explicit regularization setting. predict_proba returns one column per class. The code finds class 1 in classifier.classes before selecting a column, rather than assuming that whichever column looks plausible means “pass.” The LogisticRegression documentation describes those outputs; the installed 0.28.1 source was checked as well.

A threshold turns those values into binary decisions. Changing it changes which mistakes the application makes. In the recorded fixture run:

At threshold 0.5, four of six predictions are correct—the same count as always predicting pass. Raising the threshold to 0.8 improves this tiny fixture's count by one. That is not a validated threshold-selection procedure: choosing a threshold after inspecting the holdout uses the evaluation data for tuning. In a real experiment, select it on a separate validation set and evaluate the locked rule afterward.

The baseline also makes the error tradeoff visible. Always predicting pass has no false negatives here, while incorrectly predicting pass for both failing rows. A single accuracy number hides that behavior. Neither this table nor a probability close to one establishes calibration on future students.

Run the failure cases as well as the prediction

Download the implementation and tests beside the CSV. Once the pinned dependencies are installed:

ruby test_example.rb

The suite passes five tests and nineteen assertions. It checks training-only scaler statistics, unchanged preprocessing during prediction, class labels, threshold behavior, rejected inputs, constant features and single-class training data. The example prints the actual confusion counts and probabilities from the supplied fixture.

Before replacing the fixture with real records, define what action a prediction is meant to support and what evidence would justify it. Keep student data access and evaluation design deliberate; avoid treating an experimental score as an assessment of a person's potential. The concrete result here is a reproducible Rumale pipeline whose preprocessing boundary and error counts can be inspected. Whether a real dataset supports useful predictions remains a separate empirical question.