ML System Design, Drift and Feature Stores

Frame the task, price serving, beat baselines, roll out safely, catch drift and training-serving skew.

One definition, two readers#

Task	Code
Cheapest plan in budget	`choose_serving(workload, costs)`
Lift over the best baseline	`compare_to_baselines(..., min_lift=0.02)`
Score a design doc	`review_design_doc(doc, ready_score=0.85)`
Numeric drift, decile bins	`psi(reference, current, bins=10)`
Largest CDF gap	`ks_statistic(sample_a, sample_b)`
Categorical drift	`chi_square_test(reference_counts, current_counts)`
As-of join with TTL	`point_in_time_join(labels, rows, ttl_days)`
Compare logged features	`detect_skew(training, serving, features)`

refresh_hours = workload.max_staleness_hours - costs.batch_job_hours

if refresh_hours <= 0:job longer than the limit

return ServingPlan("batch", False, ...)

runs_per_day = 24.0 / refresh_hours

predictions = workload.entities * runs_per_day

dollars = (predictions / 1e6 * costs.batch_dollars_per_million

+ workload.requests_per_day / 1e6 * costs.lookup_dollars_per_million)

replicas = max(costs.min_replicas, math.ceil(workload.peak_qps / costs.replica_qps))

dollars = replicas * costs.replica_dollars_per_hour * 24.0online pays for idle

feasible = costs.online_latency_ms <= workload.latency_budget_ms

Covariate drift: P(x) changes. Input monitors see it.
Prior drift: P(y) changes. Prediction and outcome monitors see it.
Concept drift: P(y | x) changes. Only outcome monitors see it.
PSI: $inline$ over bins.
KS D: largest gap between two empirical CDFs. 0 for identical, 1 for no overlap.
Cramér's V: square root of chi2 / grand_total for a two-row table.
Little's law: in progress = throughput times latency. 400 rps at 50 ms is 20.
Lift: model score minus the strongest baseline score.

Command	What it does
`python3 code/main.py`	Runs each lesson demo with the standard library
`python3 code/main.py my_design_doc.json`	Reviews your own design document
`stats.ks_2samp(reference_amounts, current_amounts)`	KS statistic and p-value in scipy
`stats.chi2_contingency(table)`	Chi-square statistic, p-value and dof
`DummyClassifier(strategy="most_frequent")`	Majority-class baseline
`feast apply`	Registers feature definitions in Feast
`feast materialize-incremental <end time>`	Copies latest values to the online store
`store.get_historical_features(entity_df=...)`	Point-in-time training features

Setting	Default	Note
`family_alpha`	0.01	Bonferroni, 0.005 for each of two tests
`psi_warn`	0.10	PSI below 0.1 is stable
`psi_drift`	0.25	Above 0.25 is a significant shift
`ks_drift`	0.10	D must pass and p below alpha
`min_cramers_v`	0.10	Effect size for categories
`patience`	2	Consecutive drift windows before an alert
`window_size`	500	step=250, label_delay=500
PSI share clip	0.0001	Stops an infinite log

Symptom	Cause	Fix
Model service times out	No plan for the failure	Use the heuristic baseline
Nightly batch job fails	Scores go missing	Serve last good scores, then the rule
Score distribution jumps	Wrong decisions at scale	Pause actions and alert a person
Inputs drift, accuracy stable	Covariate drift	Often none. Watch the outcome
Accuracy drops, inputs quiet	Concept drift	Collect new labels, then retrain
Offline 0.948, live worse	Naive join reads future rows	Point-in-time join
Serving default -1, training 365	Two code paths	One registry definition
60 false alarms per day	50 features hourly at 0.05	Bonferroni, effect size, persistence

When not to use ML?: A short rule works, no labels, nobody acts, or one error is unacceptable.
Why not AUC?: It averages over thresholds you never use. Match the decision instead.
How to split data?: Train 60-80%, validation 10-20%, test 10-20%. Hold out the newest period.
How to break a feedback loop?: Keep a random holdout that never gets the intervention.
How to find the skew cause?: Compare online read, fresh compute and second code path.
After retraining?: Test on recent labels, shadow and canary, then reset the reference window.