Cheat sheets

Classical Machine Learning

Fit, validate, tune and fix regression, classification and clustering models with scikit-learn.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Defaults and numbers
  • Troubleshooting
  • Questions and tips

Source

Fit a model#

TaskCode
L2 regularized regressionRidge(alpha=1.0)
Classifier for rare positivesLogisticRegression(class_weight="balanced")
Random forestRandomForestClassifier(n_estimators=100, random_state=42)
Kernel SVMSVC(kernel="rbf", C=1.0, gamma="scale")
Linear SVM for large dataLinearSVC(C=1.0, max_iter=10000)
k nearest neighboursKNeighborsClassifier(n_neighbors=5, metric="euclidean")
K-Means clustersKMeans(n_clusters=3, random_state=42).fit(data)
Density clusters and noiseDBSCAN(eps=1.5, min_samples=5).fit(data)

From Lesson 02.02, Lesson 02.17, Lesson 02.04, Lesson 02.05, Lesson 02.06, Lesson 02.07

Metrics#

MetricFormula and use
Accuracy(TP + TN) / (TP + TN + FP + FN). Misleading on imbalanced classes
PrecisionTP / (TP + FP). Use when false positives are costly
RecallTP / (TP + FN). Use when false negatives are costly
F12 \cdot P \cdot R / (P + R). Harmonic mean of both
AUC-ROC0.5 is random, 1.0 is perfect. Threshold independent
R-squared1 - SS_{\text{res}} / SS_{\text{tot}}. Below 0 is worse than the mean

From Lesson 02.09, Lesson 02.17

Key formulas#

MSE
\text{MSE} = \frac{1}{n} \sum (y_{\text{predicted}} - y_{\text{actual}})^2
Normal equation
w = (X^T X)^{-1} X^T y. Cost is O(n^3) in features.
Ridge
\text{Cost} = \text{MSE} + \lambda \sum w_i^2
Log loss
-\frac{1}{n} \sum \left( y \log(p) + (1-y) \log(1-p) \right), p = \sigma(wx + b)
Gini
\text{Gini}(S) = 1 - \sum_k p_k^2. 0 is a pure node.
Soft margin SVM
\min \frac{1}{2} \lVert w \rVert^2 + C \sum_i \max(0, 1 - y_i (w^T x_i + b))
RBF kernel
K(x, z) = \exp(-\gamma \lVert x - z \rVert^2)
Error split
\text{Bias}^2 + \text{Variance} + \text{Irreducible Noise}

From Lesson 02.02, Lesson 02.03, Lesson 02.04, Lesson 02.05, Lesson 02.10

Leak-free preprocessing#

Leaky: the scaler sees the test rows
X_scaled = scaler.fit_transform(X)
Correct: fit on train, reuse on test
X_train_scaled = scaler.fit_transform(X_train)learn mean and std
X_test_scaled = scaler.transform(X_test)train statistics only
Best: one Pipeline object for train and serve
pipe = Pipeline([
("scaler", StandardScaler()),
("model", LogisticRegression()),
])
pipe.fit(X_train, y_train)fit_transform, then fit
predictions = pipe.predict(X_test)transform, then predict
scores = cross_val_score(full_pipe, df, y, cv=5, scoring="accuracy")refit per fold

From Lesson 02.13, 02.13 code/pipeline.py

Validate without leakage#

1Splithold out a test set. Touch it exactly once, atthe end.2Foldssplit the rest into $K$ folds. Stratify forclass ratios.3Fitthe scaler calls fit_transform on the $K-1$training folds.4Trainthe model calls fit on the scaled trainingfolds.5Scorethe scaler calls transform on the validationfold, then the model predicts.6Averagerepeat for each fold and average the $K$ scores.7Reportscore the final model on the test set.

From Lesson 02.09, Lesson 02.13

Questions#

Why not MSE for logistic regression?
With a sigmoid it is non-convex. Log loss is convex.
Why scale before an SVM?
The margin depends on \lVert w \rVert.
Why does L1 select features?
It pushes some weights exactly to zero.
Will more data fix high bias?
No. Change the model.
Why class weights over oversampling?
Same in expectation, no duplicate rows.

From Lesson 02.03, Lesson 02.05, Lesson 02.10, Lesson 02.17, Lesson 02.09

Tips#

  • Sweep the learning rate first. Fix the rest at defaults.
  • Search learning rate and regularization on a log scale.
  • Set n_estimators high and let early stopping decide.
  • Use nested cross-validation for the final estimate.
  • Add a was_this_missing column before you impute.
  • Pick the threshold on the validation set, not the test set.

From Lesson 02.12, Lesson 02.09, Lesson 02.08, Lesson 02.17

ColumnTransformer pipeline#

selectselectDataFramenumeric and categoricalNumeric columnsage, income, scoreCategorical columnscity, planSimpleImputermedian, then StandardScalerSimpleImputermost_frequent, then OneHotEncoderColumnTransformerjoins both outputsModelGradientBoostingClassifier, predictselectselectDataFramenumericandcategoricalNumericcolumnsage,income,scoreCategoricalcolumnscity,planSimpleImputermedian,thenStandardScalerSimpleImputermost_frequent,thenOneHotEncoderColumnTransformerjoinsbothoutputsModelGradientBoostingClassifier,predict

From Lesson 02.13, 02.13 code/pipeline.py

Tune hyperparameters#

param_dist = {
"learning_rate": loguniform(1e-4, 0.5),log scale for rates
"max_depth": randint(2, 10),integer range
"n_estimators": randint(50, 500),
}
search = RandomizedSearchCV(
GradientBoostingRegressor(), param_dist,
n_iter=100, cv=5,100 samples, 5 folds each
scoring="neg_mean_squared_error",negated, higher is better
random_state=42, n_jobs=-1,all CPU cores
)
search.fit(X_train, y_train)
print(search.best_params_, -search.best_score_)negate back to MSE

From Lesson 02.12

Troubleshooting#

SymptomCauseFix
High train and test errorhigh biasmore features, complex model, less regularization
Low train error, high test errorhigh variancemore data, regularization, simpler model
Test error rises, train error fallsoverfitting in progressearly stopping
CV score higher than productionscaler fit on full dataPipeline with cross_val_score
Crash on a new categoryencoder never saw the valueOneHotEncoder(handle_unknown="ignore")
99% accuracy, no positives foundclass imbalanceclass weights, threshold tuning
SVM or KNN scores lowunscaled featuresStandardScaler() first
Best value at the search edgerange too narrowwiden the search space

From Lesson 02.10, Lesson 02.13, Lesson 02.17, Lesson 02.05, Lesson 02.12, Lesson 02.06

Starting values#

SettingDefaultNote
Decision threshold0.5lower it for rare positives
CV folds K510 is also standard
Features per split, forestsqrt(n_features)n_features / 3 for regression
Trees in a forest100n_estimators in the lesson code
Boosting learning_rate0.1typical 0.01 to 0.3
Boosting max_depth3lesson GradientBoosting class
Naive Bayes alpha1.0Laplace smoothing
Out of bag share36.8%free validation set

From Lesson 02.17, Lesson 02.09, Lesson 02.11, Lesson 02.04, Lesson 02.14

Abbreviations#

AUC-ROC
Area Under the Receiver Operating Characteristic curve.
SMOTE
synthetic minority samples by k-NN interpolation.
GMM
Gaussian Mixture Model. Soft clusters by Expectation-Maximization.
RFE
Recursive Feature Elimination. Train, rank, prune, repeat.

From Lesson 02.09, Lesson 02.17, Lesson 02.07, Lesson 02.18