Classical Models Compared

Linear and logistic regression, trees and forests, SVM, KNN and Naive Bayes side by side, with when to use each.

Models side by side#

Model	Trains with	Boundary, prediction cost and memory
Linear regression	Gradient descent	A hyperplane. Normal equation is $inline$ in features
Logistic regression	binary cross-entropy	Always linear. Boundary where p = 0.5
Decision tree	Gini impurity	Rectangular regions from greedy splits. High variance
Random forest	Bagging	Majority vote or average of many trees
SVM	Hinge loss	Maximum margin. Stores only the support vectors
KNN	Lazy	No training. $inline$ per query, stores all data
Naive Bayes	Count frequencies	Linear. Prediction is one matrix multiply

Model	Scale features	Wins on
Linear regression	yes	Baselines for every regression task
Trees and forests	no	Tabular data, mixed types, under 10k rows
SVM	yes	Small or high-dimensional sparse data
KNN	yes	Small data, up to about 20-50 features
Multinomial NB	no	Text with few training samples
Gaussian NB	no	Continuous features near a bell curve

Setting	Default	Note
Tree `max_depth`	None	Grows until every leaf is pure
Tree `min_samples_split`	2	A pre-pruning control
Tree `criterion`	"gini"	Gini and entropy split almost the same
Forest `n_trees`	100	scikit-learn uses `n_estimators=100`
Forest `max_features`	"sqrt"	Regression uses n_features/3
KNN `k`	5	Start near sqrt(N). Odd K for binary
SVM `rbf_kernel` gamma	0.5	`polynomial_kernel` degree 3
GaussianNB `var_smoothing`	1e-9	Stops division by zero variance

Symptom	Cause	Fix
Logistic fails on circles	The boundary is always linear	Add polynomial features
SVM accuracy is poor	Unscaled features distort the margin	`StandardScaler` in a `Pipeline`
`SVC` too slow	Dual form is $inline$ to $inline$	`LinearSVC` primal form
KNN degrades with many features	Curse of dimensionality	PCA or UMAP first
One tree changes with the data	A single tree is high variance	Use a random forest
MDI ranks a noise feature high	Bias to high cardinality	Permutation importance
NB gives probability 0	Unseen word in a class	Laplace smoothing, alpha 1
MultinomialNB rejects features	Negative feature values	GaussianNB or shift values

Use linear regression as the baseline for every regression task.
Use logistic regression for yes-or-no answers with probabilities and a linear boundary.
Use a random forest for flat tables. It needs almost no hyperparameter tuning.
Use an SVM for small data or sparse text with TF-IDF features.
Use KNN when the data changes often and you want zero training time.
Use TF-IDF with MultinomialNB for text with fewer than 10,000 training samples.
Use BernoulliNB for short text. Set binary=True in CountVectorizer.

clf = Pipeline([

("scaler", StandardScaler()),SVM and KNN need it

("svm", SVC(kernel="rbf", C=1.0, gamma="scale")),dual form, kernels

])

clf.fit(X_train, y_train)

clf.score(X_test, y_test)accuracy

clf['svm'].n_support_support vectors

("svm", LinearSVC(C=1.0, max_iter=10000))large data, primal

("knn", KNeighborsClassifier(n_neighbors=5, metric="euclidean"))

mnb = MultinomialNB(alpha=1.0)counts, no scaling

mnb.fit(X_train_counts, y_train)

Command	What it does
Large `C`	Narrow margin, fewer misclassifications. Overfits
Small `C`	Wide margin, more violations. Underfits
`K = 1`	Zero training error. High variance
`K = N`	Predicts the majority class. Maximum bias
Max depth	Stops the tree early. Pre-pruning
NB `alpha`	Higher alpha gives stronger smoothing
Ridge $inline$	Higher gives smaller weights
SVR $inline$	Wider tube, fewer support vectors

Why not MSE for logistic regression?: With a sigmoid, MSE is non-convex. Binary cross-entropy is convex.
Why does NB work with a wrong assumption?: Classification needs correct rankings, not correct probabilities.
Why does a forest beat one tree?: Averaging decorrelated trees reduces variance without more bias.
Why do only support vectors count?: Points beyond the margin have zero hinge loss.
NB or logistic regression?: Start with NB. Switch when data is large and NB plateaus.
Normal equation or gradient descent?: Normal equation for small data. Gradient descent for large data.
Do I trust NB probabilities?: No. Use CalibratedClassifierCV with cv=5.