Feature Engineering and Pipelines

Impute, scale, encode, select and ensemble inside one leak-free pipeline. Python from the lessons.

One pipeline for mixed columns#

numeric_pipe = Pipeline([

("impute", SimpleImputer(strategy="median")),

("scale", StandardScaler()),

])

categorical_pipe = Pipeline([

("impute", SimpleImputer(strategy="most_frequent")),

("encode", OneHotEncoder(handle_unknown="ignore")),unseen city gives zeros

])

preprocessor = ColumnTransformer([

("num", numeric_pipe, ["age", "income", "score"]),

("cat", categorical_pipe, ["city", "gender", "plan"]),

])

full_pipeline = Pipeline([("preprocess", preprocessor),

("model", GradientBoostingClassifier())])

Task	Code
Scale to [0, 1]	`min_max_scale(values)`
Mean 0, std 1	`standardize(values)`
Compress right skew	`log_transform(values)` uses log(v + 1)
Equal-width bins	`bin_values(values, n_bins=5)`
Squares and pairs	`polynomial_features(row, degree=2)`
Smoothed target mean	`target_encode(feature_values, target_values, smoothing=10)`
Flag holes first	`add_missing_indicator(values)`
Text weights	`tfidf(documents)` returns vectors, vocab

Task	Code
Drop near-constant columns	`variance_threshold(X, threshold=0.01)`
Rank by MI	`mutual_information(X, y, n_bins=10)`
Prune by weight	`rfe(X, y, n_features_to_select=5)`
Zero weights with L1	`l1_feature_selection(X, y, alpha=0.1)`
Split-gain importance	`tree_importance(X, y, n_trees=50)`
sklearn RFE	`RFE(LogisticRegression(), n_features_to_select=10)`
sklearn L1	`SelectFromModel(Lasso(alpha=0.01))`
sklearn trees	`rf.feature_importances_`

Command	What it does
Bagging / Random Forest	Cuts variance. Noisy data, many features
AdaBoost	Cuts bias. Clean data. Hurt by outliers
Gradient Boosting	Cuts bias. Tabular data. Easy to overfit
XGBoost / LightGBM	Both. Start here for production tabular ML
Stacking	Both. Last 1-2% accuracy. Use CV meta-features
Soft voting	Average probabilities of diverse models

weights = np.full(n, 1 / n)AdaBoost starts uniform

err = np.clip(err, 1e-10, 1 - 1e-10)weighted stump error

alpha = 0.5 * np.log((1 - err) / err)low error, high vote

weights *= np.exp(-alpha * y * pred)misses gain weight

weights /= weights.sum()

self.initial_pred = np.mean(y)gradient boosting start

residuals = y - current_predsquared error gradient

current_pred += self.lr * updateshrink each tree

Setting	Default	Note
Target encoding `smoothing`	10	Pulls rare categories to the global mean
`remove_correlated` threshold	0.9	Drops the second column of a pair
`AdaBoostScratch` `n_estimators`	50	Decision stumps
`GradientBoostingScratch`	100	learning_rate=0.1, max_depth=3
`BaggingClassifier`	20	n_estimators=20, max_depth=5
Learning rate range	0.01 to 0.3	Smaller needs more trees
Bootstrap unique rows	63.2%	36.8% out-of-bag for validation
Forest split features	`sqrt(n_features)`	`n_features / 3` for regression

Symptom	Cause	Fix
Test score too high	Scaler fit before the split	Use Pipeline with `cross_val_score`
Train and serve outputs differ	Transforms outside the pipeline	One Pipeline object for both
Crash on a new city	Encoder never saw the value	`OneHotEncoder(handle_unknown="ignore")`
Breaks on schema change	Hardcoded column names	Column name lists from config
Random ID ranks high	Tree importance cardinality bias	Check with permutation importance
Target encoding overfits	Encoded on test rows	Fit on train data only, smooth
Stacking leaks	Meta-features from fitted data	Generate them with cross-validation

Why scale features?: K-Means, KNN and SVM use distance. Scaling makes all features count equally.
Why add a missing indicator?: The fact that data is missing can itself be informative.
Why MI and not correlation?: MI catches quadratic or periodic links. Correlation can read zero.
Why soft voting?: It uses confidence information. It is usually better than hard voting.
What makes a run reproducible?: Fixed seeds, pinned dependencies, versioned data and config files.
Which model first on tabular data?: LightGBM or XGBoost with default parameters.