Model Evaluation, Bias-Variance and Tuning

Splits and cross-validation, metric formulas, bias-variance diagnosis, search strategies and imbalanced classes.

Diagnose train and test error#

Symptom	Cause	Fix
High train error, high test error	High bias	More features, complex model, less regularization
Low train error, high test error	High variance	More data, regularization, simpler model
Train error falls, test error rises	Overfitting in progress	Early stopping
Both curves flat and high	Bias. More data will not help	Change the model
Large gap, validation still falling	Variance that data can fix	Collect more data
99% accuracy, zero positives caught	Class imbalance	F1, AUPRC or MCC
Validation score too optimistic	Scaler fit before the split	Preprocess inside a `Pipeline`
Best value at the search edge	Range too narrow	Widen the search space

scores = cross_val_score(model, X, y, cv=StratifiedKFold(5), scoring="f1")

param_dist = {

"learning_rate": loguniform(1e-4, 0.5),log scale

"max_depth": randint(2, 10),

"n_estimators": randint(50, 500),

}

search = RandomizedSearchCV(GradientBoostingRegressor(), param_dist,

n_iter=100, cv=5, scoring="neg_mean_squared_error", n_jobs=-1)

search.fit(X_train, y_train)

search.best_params_best settings

outer_scores = cross_val_score(inner_cv, X, y, cv=5, scoring="neg_mean_squared_error")

Command	What it does
Grid search	Every combination. Cost grows exponentially
Random search	Samples distributions. More unique values per budget
Bayesian optimization	Surrogate plus acquisition. 2-5x fewer evaluations
Hyperband	Many configs on small budgets. 10-50x faster
`MedianPruner`	Stops a trial worse than the median at that step
`GridSearchCV`	Exhaustive grid with cross-validation
`RandomizedSearchCV`	Random search with CV. `n_jobs=-1` uses all cores
`trial.suggest_float`	Optuna sample. Add `log=True` for lr

Setting	Default	Note
Mild, under 70/30	class weights	Try them first
Moderate, 70/30 to 95/5	SMOTE + class weights	Or SMOTE with threshold tuning
Severe, over 95/5	combine multiple strategies	SMOTE, class weights and threshold
Under 1000 samples	Oversample or SMOTE, avoid undersampling	Small data
1000-10000 samples	SMOTE + threshold tuning	Medium data
Over 10000 samples	Undersampling OK, or class weights	Large data

Setting	Default	Note
K folds	5	Or 10
Decision threshold	0.5	Lower it for rare positives
Random search trials	60	95% chance to land within 5% of the optimum
Hyperband start	81	Configs at 1 epoch. Keep the top third
Nested CV fits	675	5 outer x 5 inner x 27 grid points
Bootstrap fits	200	`n_bootstrap=200` in the decomposition
SMOTE `k`	5	`n_synthetic=100`
Positive class weight	10.0	1000 samples, 50 positive

Why stratified folds?: Each fold keeps the class ratio of the full dataset.
Why use the test set once?: Each look and adjust turns it into a second validation set.
Why random over grid?: Only 1-2 of 6 hyperparameters usually matter.
Bagging or boosting?: Bagging for high variance. Boosting for high bias.
Class weights or oversampling?: Weights equal oversampling in expectation, with no duplicates.
Where is the worst capacity?: At the interpolation threshold, where parameters are near samples.
How do I report a final score?: Use nested CV or a test set you never touch while tuning.

Split first, then preprocess.
Tune the learning rate first. Search it on a log scale.
Use early stopping in place of tuning n_estimators.
Spend 60% of the tuning budget on the top 2 hyperparameters.
Start with library defaults. Coarse search 20-50 trials, then fine search 50-100.
Sweep thresholds from 0.0 to 1.0 on validation data. Keep the best F1.

Task	Code
Stratified folds	`stratified_kfold_split(y, k=5, seed=42)`
ROC area	`auc_roc(y_true, y_scores)`
Synthetic minority rows	`smote(X_minority, k=5, n_synthetic=100)`
Balanced class weights	`compute_class_weights(y)`
Best cutoff by F1	`find_optimal_threshold(y_true, y_probs)`