# AI Engineering from Scratch

> A free, open-source curriculum that builds every core AI algorithm by hand — 511 lessons across 20 phases, from linear algebra to autonomous agents. Python, TypeScript, Rust, Julia.

Canonical site: https://aiengineeringfromscratch.com
Source: https://github.com/rohitg00/ai-engineering-from-scratch
Glossary terms: 250 · Reusable outputs (prompts/skills/agents): 507

## Developer resources
- [Developer documentation](https://aiengineeringfromscratch.com/developer.html) — machine-readable site contracts and integration notes
- [OpenAPI description](https://aiengineeringfromscratch.com/openapi.json) — read-only public resource inventory
- [Sitemap](https://aiengineeringfromscratch.com/sitemap.xml) — canonical URL inventory
- [Contact](https://aiengineeringfromscratch.com/contact.html) — maintainer and project contact route
- [Privacy](https://aiengineeringfromscratch.com/privacy.html) — data and analytics policy

Lesson pages render client-side. Agents: fetch each lesson's raw markdown link; it is the full text. Lesson directories may also include code/ (runnable implementation) and quiz.json.

## Phase 0: Setup & Tooling
Get your environment ready for everything that follows.

- [Dev Environment](https://aiengineeringfromscratch.com/lesson.html?path=phases/00-setup-and-tooling/01-dev-environment) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/00-setup-and-tooling/01-dev-environment/docs/en.md) — Your tools shape your thinking. Set them up once, set them up right.
- [Git & Collaboration](https://aiengineeringfromscratch.com/lesson.html?path=phases/00-setup-and-tooling/02-git-and-collaboration) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/00-setup-and-tooling/02-git-and-collaboration/docs/en.md) — Version control is not optional. Every experiment, every model, every lesson you build here gets tracked.
- [GPU Setup & Cloud](https://aiengineeringfromscratch.com/lesson.html?path=phases/00-setup-and-tooling/03-gpu-setup-and-cloud) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/00-setup-and-tooling/03-gpu-setup-and-cloud/docs/en.md) — Training on CPU is fine for learning. Training for real needs a GPU.
- [APIs & Keys](https://aiengineeringfromscratch.com/lesson.html?path=phases/00-setup-and-tooling/04-apis-and-keys) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/00-setup-and-tooling/04-apis-and-keys/docs/en.md) — Every AI API works the same way: send a request, get a response. The details change, the pattern doesn't.
- [Jupyter Notebooks](https://aiengineeringfromscratch.com/lesson.html?path=phases/00-setup-and-tooling/05-jupyter-notebooks) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/00-setup-and-tooling/05-jupyter-notebooks/docs/en.md) — Notebooks are the lab bench of AI engineering. You prototype here, then move what works into production.
- [Python Environments](https://aiengineeringfromscratch.com/lesson.html?path=phases/00-setup-and-tooling/06-python-environments) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/00-setup-and-tooling/06-python-environments/docs/en.md) — Dependency hell is real. Virtual environments are the cure.
- [Docker for AI](https://aiengineeringfromscratch.com/lesson.html?path=phases/00-setup-and-tooling/07-docker-for-ai) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/00-setup-and-tooling/07-docker-for-ai/docs/en.md) — Containers make "works on my machine" a thing of the past.
- [Editor Setup](https://aiengineeringfromscratch.com/lesson.html?path=phases/00-setup-and-tooling/08-editor-setup) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/00-setup-and-tooling/08-editor-setup/docs/en.md) — Your editor is your co-pilot. Configure it once so it stays out of your way and starts pulling its weight.
- [Data Management](https://aiengineeringfromscratch.com/lesson.html?path=phases/00-setup-and-tooling/09-data-management) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/00-setup-and-tooling/09-data-management/docs/en.md) — Data is the fuel. How you manage it determines how fast you go.
- [Terminal & Shell](https://aiengineeringfromscratch.com/lesson.html?path=phases/00-setup-and-tooling/10-terminal-and-shell) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/00-setup-and-tooling/10-terminal-and-shell/docs/en.md) — The terminal is where AI engineers live. Get comfortable here.
- [Linux for AI](https://aiengineeringfromscratch.com/lesson.html?path=phases/00-setup-and-tooling/11-linux-for-ai) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/00-setup-and-tooling/11-linux-for-ai/docs/en.md) — Most AI runs on Linux. You need to know enough to not be stuck.
- [Debugging & Profiling](https://aiengineeringfromscratch.com/lesson.html?path=phases/00-setup-and-tooling/12-debugging-and-profiling) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/00-setup-and-tooling/12-debugging-and-profiling/docs/en.md) — The worst AI bugs don't crash. They train silently on garbage and report a beautiful loss curve.

## Phase 1: Math Foundations
The intuition behind every AI algorithm, through code.

- [Linear Algebra Intuition](https://aiengineeringfromscratch.com/lesson.html?path=phases/01-math-foundations/01-linear-algebra-intuition) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/01-math-foundations/01-linear-algebra-intuition/docs/en.md) — Every AI model is just matrix math wearing a fancy hat.
- [Vectors, Matrices & Operations](https://aiengineeringfromscratch.com/lesson.html?path=phases/01-math-foundations/02-vectors-matrices-operations) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/01-math-foundations/02-vectors-matrices-operations/docs/en.md) — Every neural network is just matrix multiplication with extra steps.
- [Matrix Transformations & Eigenvalues](https://aiengineeringfromscratch.com/lesson.html?path=phases/01-math-foundations/03-matrix-transformations) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/01-math-foundations/03-matrix-transformations/docs/en.md) — A matrix is a machine that reshapes space. Learn what it does to every point, and you understand the whole transformation.
- [Calculus for ML: Derivatives & Gradients](https://aiengineeringfromscratch.com/lesson.html?path=phases/01-math-foundations/04-calculus-for-ml) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/01-math-foundations/04-calculus-for-ml/docs/en.md) — Derivatives tell you which way is downhill. That is all a neural network needs to learn.
- [Chain Rule & Automatic Differentiation](https://aiengineeringfromscratch.com/lesson.html?path=phases/01-math-foundations/05-chain-rule-and-autodiff) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/01-math-foundations/05-chain-rule-and-autodiff/docs/en.md) — The chain rule is the engine behind every neural network that learns.
- [Probability & Distributions](https://aiengineeringfromscratch.com/lesson.html?path=phases/01-math-foundations/06-probability-and-distributions) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/01-math-foundations/06-probability-and-distributions/docs/en.md) — Probability is the language AI uses to express uncertainty.
- [Bayes' Theorem & Statistical Thinking](https://aiengineeringfromscratch.com/lesson.html?path=phases/01-math-foundations/07-bayes-theorem) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/01-math-foundations/07-bayes-theorem/docs/en.md) — Probability is about what you expect. Bayes' theorem is about what you learn.
- [Optimization: Gradient Descent Family](https://aiengineeringfromscratch.com/lesson.html?path=phases/01-math-foundations/08-optimization) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/01-math-foundations/08-optimization/docs/en.md) — Training a neural network is nothing more than finding the bottom of a valley.
- [Information Theory: Entropy, KL Divergence](https://aiengineeringfromscratch.com/lesson.html?path=phases/01-math-foundations/09-information-theory) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/01-math-foundations/09-information-theory/docs/en.md) — Information theory measures surprise. Loss functions are built on it.
- [Dimensionality Reduction: PCA, t-SNE, UMAP](https://aiengineeringfromscratch.com/lesson.html?path=phases/01-math-foundations/10-dimensionality-reduction) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/01-math-foundations/10-dimensionality-reduction/docs/en.md) — High-dimensional data has structure. You find it by looking from the right angle.
- [Singular Value Decomposition](https://aiengineeringfromscratch.com/lesson.html?path=phases/01-math-foundations/11-singular-value-decomposition) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/01-math-foundations/11-singular-value-decomposition/docs/en.md) — SVD is the Swiss Army knife of linear algebra. Every matrix has one. Every data scientist needs one.
- [Tensor Operations](https://aiengineeringfromscratch.com/lesson.html?path=phases/01-math-foundations/12-tensor-operations) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/01-math-foundations/12-tensor-operations/docs/en.md) — Tensors are the common language between data and deep learning. Every image, every sentence, every gradient flows through them.
- [Numerical Stability](https://aiengineeringfromscratch.com/lesson.html?path=phases/01-math-foundations/13-numerical-stability) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/01-math-foundations/13-numerical-stability/docs/en.md) — Floating point is a leaky abstraction. It will bite you during training, and you will not see it coming.
- [Norms & Distances](https://aiengineeringfromscratch.com/lesson.html?path=phases/01-math-foundations/14-norms-and-distances) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/01-math-foundations/14-norms-and-distances/docs/en.md) — Your distance function defines what "similar" means. Choose wrong and everything downstream breaks.
- [Statistics for ML](https://aiengineeringfromscratch.com/lesson.html?path=phases/01-math-foundations/15-statistics-for-ml) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/01-math-foundations/15-statistics-for-ml/docs/en.md) — Statistics is how you know if your model actually works or just got lucky.
- [Sampling Methods](https://aiengineeringfromscratch.com/lesson.html?path=phases/01-math-foundations/16-sampling-methods) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/01-math-foundations/16-sampling-methods/docs/en.md) — Sampling is how AI explores the space of possibilities.
- [Linear Systems](https://aiengineeringfromscratch.com/lesson.html?path=phases/01-math-foundations/17-linear-systems) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/01-math-foundations/17-linear-systems/docs/en.md) — Solving Ax = b is the oldest problem in mathematics that still runs your neural network.
- [Convex Optimization](https://aiengineeringfromscratch.com/lesson.html?path=phases/01-math-foundations/18-convex-optimization) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/01-math-foundations/18-convex-optimization/docs/en.md) — Convex problems have one valley. Neural networks have millions. Knowing the difference matters.
- [Complex Numbers for AI](https://aiengineeringfromscratch.com/lesson.html?path=phases/01-math-foundations/19-complex-numbers) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/01-math-foundations/19-complex-numbers/docs/en.md) — The square root of -1 is not imaginary. It is the key to rotations, frequencies, and half of signal processing.
- [The Fourier Transform](https://aiengineeringfromscratch.com/lesson.html?path=phases/01-math-foundations/20-fourier-transform) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/01-math-foundations/20-fourier-transform/docs/en.md) — Every signal is a sum of sine waves. The Fourier transform tells you which ones.
- [Graph Theory for ML](https://aiengineeringfromscratch.com/lesson.html?path=phases/01-math-foundations/21-graph-theory) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/01-math-foundations/21-graph-theory/docs/en.md) — Graphs are the data structure of relationships. If your data has connections, you need graph theory.
- [Stochastic Processes](https://aiengineeringfromscratch.com/lesson.html?path=phases/01-math-foundations/22-stochastic-processes) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/01-math-foundations/22-stochastic-processes/docs/en.md) — Randomness with structure. The math behind random walks, Markov chains, and diffusion models.

## Phase 2: ML Fundamentals
Classical ML — still the backbone of most production AI.

- [What Is Machine Learning](https://aiengineeringfromscratch.com/lesson.html?path=phases/02-ml-fundamentals/01-what-is-machine-learning) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/02-ml-fundamentals/01-what-is-machine-learning/docs/en.md) — Machine learning is teaching computers to find patterns in data instead of writing rules by hand.
- [Linear Regression from Scratch](https://aiengineeringfromscratch.com/lesson.html?path=phases/02-ml-fundamentals/02-linear-regression) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/02-ml-fundamentals/02-linear-regression/docs/en.md) — Linear regression draws the best straight line through your data. It is the "hello world" of machine learning.
- [Logistic Regression & Classification](https://aiengineeringfromscratch.com/lesson.html?path=phases/02-ml-fundamentals/03-logistic-regression) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/02-ml-fundamentals/03-logistic-regression/docs/en.md) — Logistic regression bends a straight line into an S-curve to answer yes-or-no questions with probabilities.
- [Decision Trees & Random Forests](https://aiengineeringfromscratch.com/lesson.html?path=phases/02-ml-fundamentals/04-decision-trees) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/02-ml-fundamentals/04-decision-trees/docs/en.md) — A decision tree is just a flowchart. But a forest of them is one of the most powerful tools in ML.
- [Support Vector Machines](https://aiengineeringfromscratch.com/lesson.html?path=phases/02-ml-fundamentals/05-support-vector-machines) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/02-ml-fundamentals/05-support-vector-machines/docs/en.md) — Find the widest street between two classes. That is the entire idea.
- [KNN & Distance Metrics](https://aiengineeringfromscratch.com/lesson.html?path=phases/02-ml-fundamentals/06-knn-and-distances) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/02-ml-fundamentals/06-knn-and-distances/docs/en.md) — Store everything. Predict by looking at your neighbors. The simplest algorithm that actually works.
- [Unsupervised Learning: K-Means, DBSCAN](https://aiengineeringfromscratch.com/lesson.html?path=phases/02-ml-fundamentals/07-unsupervised-learning) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/02-ml-fundamentals/07-unsupervised-learning/docs/en.md) — No labels, no teacher. The algorithm finds structure on its own.
- [Feature Engineering & Selection](https://aiengineeringfromscratch.com/lesson.html?path=phases/02-ml-fundamentals/08-feature-engineering) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/02-ml-fundamentals/08-feature-engineering/docs/en.md) — A good feature is worth a thousand data points.
- [Model Evaluation: Metrics, Cross-Validation](https://aiengineeringfromscratch.com/lesson.html?path=phases/02-ml-fundamentals/09-model-evaluation) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/02-ml-fundamentals/09-model-evaluation/docs/en.md) — A model is only as good as the way you measure it.
- [Bias, Variance & the Learning Curve](https://aiengineeringfromscratch.com/lesson.html?path=phases/02-ml-fundamentals/10-bias-variance) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/02-ml-fundamentals/10-bias-variance/docs/en.md) — Every model error comes from one of three sources: bias, variance, or noise. You can only control the first two.
- [Ensemble Methods: Boosting, Bagging, Stacking](https://aiengineeringfromscratch.com/lesson.html?path=phases/02-ml-fundamentals/11-ensemble-methods) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/02-ml-fundamentals/11-ensemble-methods/docs/en.md) — A group of weak learners, combined correctly, becomes a strong learner. This is not a metaphor. It is a theorem.
- [Hyperparameter Tuning](https://aiengineeringfromscratch.com/lesson.html?path=phases/02-ml-fundamentals/12-hyperparameter-tuning) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/02-ml-fundamentals/12-hyperparameter-tuning/docs/en.md) — Hyperparameters are the knobs you turn before training starts. Turning them well is the difference between a mediocre model and a great one.
- [ML Pipelines & Experiment Tracking](https://aiengineeringfromscratch.com/lesson.html?path=phases/02-ml-fundamentals/13-ml-pipelines) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/02-ml-fundamentals/13-ml-pipelines/docs/en.md) — A model is not a product. A pipeline is. The pipeline is everything from raw data to deployed prediction, and every step must be reproducible.
- [Naive Bayes](https://aiengineeringfromscratch.com/lesson.html?path=phases/02-ml-fundamentals/14-naive-bayes) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/02-ml-fundamentals/14-naive-bayes/docs/en.md) — The "naive" assumption is wrong, and it works anyway. That's the beauty of it.
- [Time Series Fundamentals](https://aiengineeringfromscratch.com/lesson.html?path=phases/02-ml-fundamentals/15-time-series) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/02-ml-fundamentals/15-time-series/docs/en.md) — Past performance does predict future results -- if you check for stationarity first.
- [Anomaly Detection](https://aiengineeringfromscratch.com/lesson.html?path=phases/02-ml-fundamentals/16-anomaly-detection) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/02-ml-fundamentals/16-anomaly-detection/docs/en.md) — Normal is easy to define. Abnormal is whatever doesn't fit.
- [Handling Imbalanced Data](https://aiengineeringfromscratch.com/lesson.html?path=phases/02-ml-fundamentals/17-imbalanced-data) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/02-ml-fundamentals/17-imbalanced-data/docs/en.md) — When 99% of your data is "normal," accuracy is a lie.
- [Feature Selection](https://aiengineeringfromscratch.com/lesson.html?path=phases/02-ml-fundamentals/18-feature-selection) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/02-ml-fundamentals/18-feature-selection/docs/en.md) — More features is not better. The right features is better.

## Phase 3: Deep Learning Core
Neural networks from first principles. No frameworks until you build one.

- [The Perceptron: Where It All Started](https://aiengineeringfromscratch.com/lesson.html?path=phases/03-deep-learning-core/01-the-perceptron) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/03-deep-learning-core/01-the-perceptron/docs/en.md) — The perceptron is the atom of neural networks. Split it open and you find weights, a bias, and a decision.
- [Multi-Layer Networks & Forward Pass](https://aiengineeringfromscratch.com/lesson.html?path=phases/03-deep-learning-core/02-multi-layer-networks) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/03-deep-learning-core/02-multi-layer-networks/docs/en.md) — One neuron draws a line. Stack them, and you can draw anything.
- [Backpropagation from Scratch](https://aiengineeringfromscratch.com/lesson.html?path=phases/03-deep-learning-core/03-backpropagation) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/03-deep-learning-core/03-backpropagation/docs/en.md) — Backpropagation is the algorithm that makes learning possible. Without it, neural networks are just expensive random number generators.
- [Activation Functions: ReLU, Sigmoid, GELU & Why](https://aiengineeringfromscratch.com/lesson.html?path=phases/03-deep-learning-core/04-activation-functions) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/03-deep-learning-core/04-activation-functions/docs/en.md) — Without nonlinearity, your 100-layer network is a fancy matrix multiply. Activations are the gates that let neural networks think in curves.
- [Loss Functions: MSE, Cross-Entropy, Contrastive](https://aiengineeringfromscratch.com/lesson.html?path=phases/03-deep-learning-core/05-loss-functions) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/03-deep-learning-core/05-loss-functions/docs/en.md) — Your network makes a prediction. The ground truth says otherwise. How wrong is it? That number is the loss. Pick the wrong loss function and your model optimizes for the wrong t…
- [Optimizers: SGD, Momentum, Adam, AdamW](https://aiengineeringfromscratch.com/lesson.html?path=phases/03-deep-learning-core/06-optimizers) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/03-deep-learning-core/06-optimizers/docs/en.md) — Gradient descent tells you which direction to move. It says nothing about how far or how fast. SGD is a compass. Adam is GPS with traffic data.
- [Regularization: Dropout, Weight Decay, BatchNorm](https://aiengineeringfromscratch.com/lesson.html?path=phases/03-deep-learning-core/07-regularization) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/03-deep-learning-core/07-regularization/docs/en.md) — Your model gets 99% on training data and 60% on test data. It memorized instead of learning. Regularization is the tax you impose on complexity to force generalization.
- [Weight Initialization & Training Stability](https://aiengineeringfromscratch.com/lesson.html?path=phases/03-deep-learning-core/08-weight-initialization) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/03-deep-learning-core/08-weight-initialization/docs/en.md) — Initialize wrong and training never starts. Initialize right and 50 layers train as smoothly as 3.
- [Learning Rate Schedules & Warmup](https://aiengineeringfromscratch.com/lesson.html?path=phases/03-deep-learning-core/09-learning-rate-schedules) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/03-deep-learning-core/09-learning-rate-schedules/docs/en.md) — The learning rate is the single most important hyperparameter. Not the architecture. Not the dataset size. Not the activation function. The learning rate. If you tune nothing el…
- [Build Your Own Mini Framework](https://aiengineeringfromscratch.com/lesson.html?path=phases/03-deep-learning-core/10-mini-framework) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/03-deep-learning-core/10-mini-framework/docs/en.md) — You have built neurons, layers, networks, backprop, activations, loss functions, optimizers, regularization, initialization, and LR schedules. All as separate pieces. Now wire t…
- [Introduction to PyTorch](https://aiengineeringfromscratch.com/lesson.html?path=phases/03-deep-learning-core/11-intro-to-pytorch) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/03-deep-learning-core/11-intro-to-pytorch/docs/en.md) — You built the engine from pistons and crankshafts. Now learn the one everyone actually drives.
- [Introduction to JAX](https://aiengineeringfromscratch.com/lesson.html?path=phases/03-deep-learning-core/12-intro-to-jax) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/03-deep-learning-core/12-intro-to-jax/docs/en.md) — PyTorch mutates tensors. TensorFlow builds graphs. JAX compiles pure functions. That last one changes how you think about deep learning.
- [Debugging Neural Networks](https://aiengineeringfromscratch.com/lesson.html?path=phases/03-deep-learning-core/13-debugging-neural-networks) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/03-deep-learning-core/13-debugging-neural-networks/docs/en.md) — Your network compiled. It ran. It produced a number. The number is wrong and nothing crashed. Welcome to the hardest kind of debugging -- the kind where there is no error message.

## Phase 4: Computer Vision
From pixels to understanding — image, video, 3D, VLMs, and world models.

- [Image Fundamentals: Pixels, Channels, Color Spaces](https://aiengineeringfromscratch.com/lesson.html?path=phases/04-computer-vision/01-image-fundamentals) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/04-computer-vision/01-image-fundamentals/docs/en.md) — An image is a tensor of light samples. Every vision model you will ever use starts from this one fact.
- [Convolutions from Scratch](https://aiengineeringfromscratch.com/lesson.html?path=phases/04-computer-vision/02-convolutions-from-scratch) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/04-computer-vision/02-convolutions-from-scratch/docs/en.md) — A convolution is a tiny dense layer you slide across an image, sharing the same weights at every location.
- [CNNs: LeNet to ResNet](https://aiengineeringfromscratch.com/lesson.html?path=phases/04-computer-vision/03-cnns-lenet-to-resnet) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/04-computer-vision/03-cnns-lenet-to-resnet/docs/en.md) — Every major CNN of the last thirty years is the same conv–nonlinearity–downsample recipe with one new idea bolted on. Learn the ideas in order.
- [Image Classification](https://aiengineeringfromscratch.com/lesson.html?path=phases/04-computer-vision/04-image-classification) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/04-computer-vision/04-image-classification/docs/en.md) — A classifier is a function from pixels to a probability distribution over classes. Everything else is plumbing.
- [Transfer Learning & Fine-Tuning](https://aiengineeringfromscratch.com/lesson.html?path=phases/04-computer-vision/05-transfer-learning) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/04-computer-vision/05-transfer-learning/docs/en.md) — Somebody else spent a million GPU hours teaching a network what edges, textures, and object parts look like. You should borrow those features before training your own.
- [Object Detection — YOLO from Scratch](https://aiengineeringfromscratch.com/lesson.html?path=phases/04-computer-vision/06-object-detection-yolo) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/04-computer-vision/06-object-detection-yolo/docs/en.md) — Detection is classification plus regression, run at every position in a feature map, then cleaned up with non-maximum suppression.
- [Semantic Segmentation — U-Net](https://aiengineeringfromscratch.com/lesson.html?path=phases/04-computer-vision/07-semantic-segmentation-unet) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/04-computer-vision/07-semantic-segmentation-unet/docs/en.md) — Segmentation is classification at every pixel. U-Net makes it work by pairing a downsampling encoder with an upsampling decoder and wiring skip connections between them.
- [Instance Segmentation — Mask R-CNN](https://aiengineeringfromscratch.com/lesson.html?path=phases/04-computer-vision/08-instance-segmentation-mask-rcnn) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/04-computer-vision/08-instance-segmentation-mask-rcnn/docs/en.md) — Add a tiny mask branch to a Faster R-CNN detector and you have instance segmentation. The hard part is RoIAlign, and it is harder than it looks.
- [Image Generation — GANs](https://aiengineeringfromscratch.com/lesson.html?path=phases/04-computer-vision/09-image-generation-gans) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/04-computer-vision/09-image-generation-gans/docs/en.md) — A GAN is two neural networks in a fixed game. One draws, one critiques. They get better together until the drawings fool the critic.
- [Image Generation — Diffusion Models](https://aiengineeringfromscratch.com/lesson.html?path=phases/04-computer-vision/10-image-generation-diffusion) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/04-computer-vision/10-image-generation-diffusion/docs/en.md) — A diffusion model learns to denoise. Train it to remove a tiny bit of noise from a noisy image, repeat that backwards a thousand times, and you have an image generator.
- [Stable Diffusion — Architecture & Fine-Tuning](https://aiengineeringfromscratch.com/lesson.html?path=phases/04-computer-vision/11-stable-diffusion) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/04-computer-vision/11-stable-diffusion/docs/en.md) — Stable Diffusion is a DDPM that runs in the latent space of a pretrained VAE, conditioned on text via cross-attention, sampled with a fast deterministic ODE solver, and steered …
- [Video Understanding — Temporal Modeling](https://aiengineeringfromscratch.com/lesson.html?path=phases/04-computer-vision/12-video-understanding) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/04-computer-vision/12-video-understanding/docs/en.md) — A video is a sequence of images plus the physics that connects them. Every video model either treats time as an extra axis (3D conv), a sequence to attend over (transformer), or…
- [3D Vision: Point Clouds, NeRFs](https://aiengineeringfromscratch.com/lesson.html?path=phases/04-computer-vision/13-3d-vision-nerf) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/04-computer-vision/13-3d-vision-nerf/docs/en.md) — 3D vision comes in two flavours. Point clouds are the sensor's raw output. NeRFs are the learned volumetric field. Both answer "what is where in space."
- [Vision Transformers (ViT)](https://aiengineeringfromscratch.com/lesson.html?path=phases/04-computer-vision/14-vision-transformers) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/04-computer-vision/14-vision-transformers/docs/en.md) — Cut the image into patches, treat each patch as a word, run a standard transformer. Don't look back.
- [Real-Time Vision: Edge Deployment](https://aiengineeringfromscratch.com/lesson.html?path=phases/04-computer-vision/15-real-time-edge) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/04-computer-vision/15-real-time-edge/docs/en.md) — Edge inference is the discipline of getting a 90-accuracy model to run at 30 fps on a device with 2 GB of RAM. Every percentage point of accuracy is traded against milliseconds …
- [Build a Complete Vision Pipeline](https://aiengineeringfromscratch.com/lesson.html?path=phases/04-computer-vision/16-vision-pipeline-capstone) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/04-computer-vision/16-vision-pipeline-capstone/docs/en.md) — A production vision system is a chain of models and rules stitched with data contracts. The pieces are already in this phase; the capstone wires them together end-to-end.
- [Self-Supervised Vision — SimCLR, DINO, MAE](https://aiengineeringfromscratch.com/lesson.html?path=phases/04-computer-vision/17-self-supervised-vision) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/04-computer-vision/17-self-supervised-vision/docs/en.md) — Labels are the bottleneck of supervised vision. Self-supervised pretraining removes them: learn visual features from 100M unlabelled images, fine-tune on 10k labelled ones.
- [Open-Vocabulary Vision — CLIP](https://aiengineeringfromscratch.com/lesson.html?path=phases/04-computer-vision/18-open-vocab-clip) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/04-computer-vision/18-open-vocab-clip/docs/en.md) — Train an image encoder and a text encoder together so that matching (image, caption) pairs land at the same point in a shared space. That is the whole trick.
- [OCR & Document Understanding](https://aiengineeringfromscratch.com/lesson.html?path=phases/04-computer-vision/19-ocr-document-understanding) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/04-computer-vision/19-ocr-document-understanding/docs/en.md) — OCR is a three-stage pipeline — detect text boxes, recognise the characters, then lay them out. Every modern OCR system reorders these stages or merges them.
- [Image Retrieval & Metric Learning](https://aiengineeringfromscratch.com/lesson.html?path=phases/04-computer-vision/20-image-retrieval-metric) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/04-computer-vision/20-image-retrieval-metric/docs/en.md) — A retrieval system ranks candidates by a distance in embedding space. Metric learning is the discipline of shaping that space so the distances mean what you want.
- [Keypoint Detection & Pose Estimation](https://aiengineeringfromscratch.com/lesson.html?path=phases/04-computer-vision/21-keypoint-pose) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/04-computer-vision/21-keypoint-pose/docs/en.md) — A pose is a set of ordered keypoints. A keypoint detector is a heatmap regressor. Everything else is bookkeeping.
- [3D Gaussian Splatting from Scratch](https://aiengineeringfromscratch.com/lesson.html?path=phases/04-computer-vision/22-3d-gaussian-splatting) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/04-computer-vision/22-3d-gaussian-splatting/docs/en.md) — A scene is a cloud of millions of 3D Gaussians. Each one has a position, orientation, scale, opacity, and a colour that depends on viewing direction. Rasterise them, backprop th…
- [Diffusion Transformers & Rectified Flow](https://aiengineeringfromscratch.com/lesson.html?path=phases/04-computer-vision/23-diffusion-transformers-rectified-flow) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/04-computer-vision/23-diffusion-transformers-rectified-flow/docs/en.md) — The U-Net is not the secret of diffusion. Replace it with a transformer, swap the noise schedule for a straight-line flow, and suddenly you have SD3, FLUX, and every 2026 text-t…
- [SAM 3 & Open-Vocabulary Segmentation](https://aiengineeringfromscratch.com/lesson.html?path=phases/04-computer-vision/24-sam3-open-vocab-segmentation) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/04-computer-vision/24-sam3-open-vocab-segmentation/docs/en.md) — Give a model a text prompt and an image and get masks for every matching object. SAM 3 made that a single forward pass.
- [Vision-Language Models (ViT-MLP-LLM)](https://aiengineeringfromscratch.com/lesson.html?path=phases/04-computer-vision/25-vision-language-models) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/04-computer-vision/25-vision-language-models/docs/en.md) — A vision encoder converts an image into tokens. An MLP projector maps those tokens into the LLM's embedding space. A language model does the rest. That pattern — ViT-MLP-LLM — i…
- [Monocular Depth & Geometry Estimation](https://aiengineeringfromscratch.com/lesson.html?path=phases/04-computer-vision/26-monocular-depth) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/04-computer-vision/26-monocular-depth/docs/en.md) — A depth map is a single-channel image where each pixel is a distance from the camera. Predicting it from one RGB frame used to be impossible without stereo or LiDAR. In 2026 a f…
- [Multi-Object Tracking & Video Memory](https://aiengineeringfromscratch.com/lesson.html?path=phases/04-computer-vision/27-multi-object-tracking) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/04-computer-vision/27-multi-object-tracking/docs/en.md) — Tracking is detection plus association. Detect every frame. Match this frame's detections to last frame's tracks by ID.
- [World Models & Video Diffusion](https://aiengineeringfromscratch.com/lesson.html?path=phases/04-computer-vision/28-world-models-video-diffusion) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/04-computer-vision/28-world-models-video-diffusion/docs/en.md) — A video model that predicts the next seconds of a scene is a world simulator. Condition that prediction on actions and you have a learned game engine.

## Phase 5: NLP: Foundations to Advanced
Language is the interface to intelligence.

- [Text Processing: Tokenization, Stemming, Lemmatization](https://aiengineeringfromscratch.com/lesson.html?path=phases/05-nlp-foundations-to-advanced/01-text-processing) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/05-nlp-foundations-to-advanced/01-text-processing/docs/en.md) — Language is continuous. Models are discrete. Preprocessing is the bridge.
- [Bag of Words, TF-IDF & Text Representation](https://aiengineeringfromscratch.com/lesson.html?path=phases/05-nlp-foundations-to-advanced/02-bag-of-words-tfidf) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/05-nlp-foundations-to-advanced/02-bag-of-words-tfidf/docs/en.md) — Count first, think later. TF-IDF still beats embeddings on well-defined tasks in 2026.
- [Word Embeddings: Word2Vec from Scratch](https://aiengineeringfromscratch.com/lesson.html?path=phases/05-nlp-foundations-to-advanced/03-word-embeddings-word2vec) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/05-nlp-foundations-to-advanced/03-word-embeddings-word2vec/docs/en.md) — A word is the company it keeps. Train a shallow net on that idea and geometry falls out.
- [GloVe, FastText & Subword Embeddings](https://aiengineeringfromscratch.com/lesson.html?path=phases/05-nlp-foundations-to-advanced/04-glove-fasttext-subword) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/05-nlp-foundations-to-advanced/04-glove-fasttext-subword/docs/en.md) — Word2Vec trained one embedding per word. GloVe factorized the co-occurrence matrix. FastText embedded the pieces. BPE bridged to transformers.
- [Sentiment Analysis](https://aiengineeringfromscratch.com/lesson.html?path=phases/05-nlp-foundations-to-advanced/05-sentiment-analysis) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/05-nlp-foundations-to-advanced/05-sentiment-analysis/docs/en.md) — The canonical NLP task. Most of what you need to know about classical text classification shows up here.
- [Named Entity Recognition (NER)](https://aiengineeringfromscratch.com/lesson.html?path=phases/05-nlp-foundations-to-advanced/06-named-entity-recognition) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/05-nlp-foundations-to-advanced/06-named-entity-recognition/docs/en.md) — Pull the names out. Sounds easy until you deal with ambiguous boundaries, nested entities, and domain jargon.
- [POS Tagging & Syntactic Parsing](https://aiengineeringfromscratch.com/lesson.html?path=phases/05-nlp-foundations-to-advanced/07-pos-tagging-parsing) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/05-nlp-foundations-to-advanced/07-pos-tagging-parsing/docs/en.md) — Grammar was unfashionable for a while. Then every LLM pipeline needed to validate structured extraction, and it came back.
- [Text Classification — CNNs & RNNs for Text](https://aiengineeringfromscratch.com/lesson.html?path=phases/05-nlp-foundations-to-advanced/08-cnns-rnns-for-text) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/05-nlp-foundations-to-advanced/08-cnns-rnns-for-text/docs/en.md) — Convolutions learn n-grams. Recurrences remember. Both are superseded by attention. Both still matter on constrained hardware.
- [Sequence-to-Sequence Models](https://aiengineeringfromscratch.com/lesson.html?path=phases/05-nlp-foundations-to-advanced/09-sequence-to-sequence) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/05-nlp-foundations-to-advanced/09-sequence-to-sequence/docs/en.md) — Two RNNs pretending to be a translator. The bottleneck they hit is the reason attention exists.
- [Attention Mechanism — The Breakthrough](https://aiengineeringfromscratch.com/lesson.html?path=phases/05-nlp-foundations-to-advanced/10-attention-mechanism) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/05-nlp-foundations-to-advanced/10-attention-mechanism/docs/en.md) — The decoder stops squinting at a compressed summary and starts looking at the whole source. Everything after this is attention plus engineering.
- [Machine Translation](https://aiengineeringfromscratch.com/lesson.html?path=phases/05-nlp-foundations-to-advanced/11-machine-translation) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/05-nlp-foundations-to-advanced/11-machine-translation/docs/en.md) — Translation is the task that paid for NLP research for thirty years and keeps paying now.
- [Text Summarization](https://aiengineeringfromscratch.com/lesson.html?path=phases/05-nlp-foundations-to-advanced/12-text-summarization) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/05-nlp-foundations-to-advanced/12-text-summarization/docs/en.md) — Extractive systems tell you what the document said. Abstractive systems tell you what the author meant. Different tasks, different pitfalls.
- [Question Answering Systems](https://aiengineeringfromscratch.com/lesson.html?path=phases/05-nlp-foundations-to-advanced/13-question-answering) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/05-nlp-foundations-to-advanced/13-question-answering/docs/en.md) — Three systems shaped modern QA. Extractive found spans. Retrieval-augmented grounded them in documents. Generative produced answers. Every modern AI assistant is a mix of the th…
- [Information Retrieval & Search](https://aiengineeringfromscratch.com/lesson.html?path=phases/05-nlp-foundations-to-advanced/14-information-retrieval-search) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/05-nlp-foundations-to-advanced/14-information-retrieval-search/docs/en.md) — BM25 is precise but brittle. Dense casts a wide net but misses keywords. Hybrid is the 2026 default. Everything else is tuning.
- [Topic Modeling: LDA, BERTopic](https://aiengineeringfromscratch.com/lesson.html?path=phases/05-nlp-foundations-to-advanced/15-topic-modeling) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/05-nlp-foundations-to-advanced/15-topic-modeling/docs/en.md) — LDA: documents are mixtures of topics, topics are distributions over words. BERTopic: documents cluster in embedding space, clusters are topics. Same goal, different decompositi…
- [Text Generation](https://aiengineeringfromscratch.com/lesson.html?path=phases/05-nlp-foundations-to-advanced/16-text-generation-pre-transformer) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/05-nlp-foundations-to-advanced/16-text-generation-pre-transformer/docs/en.md) — If a word is surprising, the model is bad. Perplexity makes surprise a number. Smoothing keeps it finite.
- [Chatbots: Rule-Based to Neural](https://aiengineeringfromscratch.com/lesson.html?path=phases/05-nlp-foundations-to-advanced/17-chatbots-rule-to-neural) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/05-nlp-foundations-to-advanced/17-chatbots-rule-to-neural/docs/en.md) — ELIZA replied with pattern matches. DialogFlow mapped intents. GPT answered from weights. Claude runs tools and verifies. Each era solved the previous one's worst failure.
- [Multilingual NLP](https://aiengineeringfromscratch.com/lesson.html?path=phases/05-nlp-foundations-to-advanced/18-multilingual-nlp) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/05-nlp-foundations-to-advanced/18-multilingual-nlp/docs/en.md) — One model, 100+ languages, zero training data for most of them. Cross-lingual transfer is the practical miracle of the 2020s.
- [Subword Tokenization: BPE, WordPiece, Unigram, SentencePiece](https://aiengineeringfromscratch.com/lesson.html?path=phases/05-nlp-foundations-to-advanced/19-subword-tokenization) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/05-nlp-foundations-to-advanced/19-subword-tokenization/docs/en.md) — Word tokenizers choke on unseen words. Character tokenizers blow up sequence length. Subword tokenizers split the difference. Every modern LLM ships on one.
- [Structured Outputs & Constrained Decoding](https://aiengineeringfromscratch.com/lesson.html?path=phases/05-nlp-foundations-to-advanced/20-structured-outputs-constrained-decoding) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/05-nlp-foundations-to-advanced/20-structured-outputs-constrained-decoding/docs/en.md) — Ask an LLM for JSON. Get JSON most of the time. In production, "most" is the problem. Constrained decoding turns "most" into "always" by editing the logits before sampling.
- [NLI & Textual Entailment](https://aiengineeringfromscratch.com/lesson.html?path=phases/05-nlp-foundations-to-advanced/21-nli-textual-entailment) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/05-nlp-foundations-to-advanced/21-nli-textual-entailment/docs/en.md) — "t entails h" means a human reading t would conclude h is true. NLI is the task of predicting entailment / contradiction / neutral. Boring on the surface, load-bearing in produc…
- [Embedding Models Deep Dive](https://aiengineeringfromscratch.com/lesson.html?path=phases/05-nlp-foundations-to-advanced/22-embedding-models-deep-dive) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/05-nlp-foundations-to-advanced/22-embedding-models-deep-dive/docs/en.md) — Word2Vec gave you a vector per word. Modern embedding models give you a vector per passage, cross-lingual, with sparse, dense, and multi-vector views, sized to fit your index. P…
- [Chunking Strategies for RAG](https://aiengineeringfromscratch.com/lesson.html?path=phases/05-nlp-foundations-to-advanced/23-chunking-strategies-rag) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/05-nlp-foundations-to-advanced/23-chunking-strategies-rag/docs/en.md) — Chunking configuration influences retrieval quality as much as the choice of embedding model (Vectara NAACL 2025). Get chunking wrong and no amount of reranking saves you.
- [Coreference Resolution](https://aiengineeringfromscratch.com/lesson.html?path=phases/05-nlp-foundations-to-advanced/24-coreference-resolution) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/05-nlp-foundations-to-advanced/24-coreference-resolution/docs/en.md) — "She called him. He did not answer. The doctor was at lunch." Three references to two people and nobody is named. Coreference resolution figures out who is who.
- [Entity Linking & Disambiguation](https://aiengineeringfromscratch.com/lesson.html?path=phases/05-nlp-foundations-to-advanced/25-entity-linking) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/05-nlp-foundations-to-advanced/25-entity-linking/docs/en.md) — NER found "Paris." Entity linking decides: Paris, France? Paris Hilton? Paris, Texas? Paris (the Trojan prince)? Without linking, your knowledge graph stays ambiguous.
- [Relation Extraction & Knowledge Graph Construction](https://aiengineeringfromscratch.com/lesson.html?path=phases/05-nlp-foundations-to-advanced/26-relation-extraction-kg) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/05-nlp-foundations-to-advanced/26-relation-extraction-kg/docs/en.md) — NER found the entities. Entity linking anchored them. Relation extraction finds the edges between them. A knowledge graph is the sum of nodes, edges, and their provenance.
- [LLM Evaluation: RAGAS, DeepEval, G-Eval](https://aiengineeringfromscratch.com/lesson.html?path=phases/05-nlp-foundations-to-advanced/27-llm-evaluation-frameworks) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/05-nlp-foundations-to-advanced/27-llm-evaluation-frameworks/docs/en.md) — Exact-match and F1 miss semantic equivalence. Human review does not scale. LLM-as-judge is the production answer — with enough calibration to trust the number.
- [Long-Context Evaluation: NIAH, RULER, LongBench, MRCR](https://aiengineeringfromscratch.com/lesson.html?path=phases/05-nlp-foundations-to-advanced/28-long-context-evaluation) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/05-nlp-foundations-to-advanced/28-long-context-evaluation/docs/en.md) — Gemini 3 Pro advertises 10M tokens of context. At 1M tokens, 8-needle MRCR drops to 26.3%. Advertised ≠ usable. Long-context evaluation tells you the actual capacity of the mode…
- [Dialogue State Tracking](https://aiengineeringfromscratch.com/lesson.html?path=phases/05-nlp-foundations-to-advanced/29-dialogue-state-tracking) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/05-nlp-foundations-to-advanced/29-dialogue-state-tracking/docs/en.md) — "I want a cheap restaurant in the north... actually make it moderate... and add Italian." Three turns, three state updates. DST keeps the slot-value dict in sync so the booking …

## Phase 6: Speech & Audio
Hear, understand, speak.

- [Audio Fundamentals: Waveforms, Sampling, FFT](https://aiengineeringfromscratch.com/lesson.html?path=phases/06-speech-and-audio/01-audio-fundamentals) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/06-speech-and-audio/01-audio-fundamentals/docs/en.md) — Waveforms are the raw signal. Spectrograms are the representation. Mel features are the ML-friendly form. Every modern ASR and TTS pipeline walks this ladder, and the first rung…
- [Spectrograms, Mel Scale & Audio Features](https://aiengineeringfromscratch.com/lesson.html?path=phases/06-speech-and-audio/02-spectrograms-mel-features) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/06-speech-and-audio/02-spectrograms-mel-features/docs/en.md) — Neural nets do not consume raw waveforms well. They consume spectrograms. They consume mel spectrograms even better. Every ASR, TTS, and audio classifier in 2026 lives or dies b…
- [Audio Classification](https://aiengineeringfromscratch.com/lesson.html?path=phases/06-speech-and-audio/03-audio-classification) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/06-speech-and-audio/03-audio-classification/docs/en.md) — Everything from "dog barking vs siren" to "which language is this" is audio classification. The features are mels. The architecture moves each decade. The evaluation stays AUC, …
- [Speech Recognition (ASR)](https://aiengineeringfromscratch.com/lesson.html?path=phases/06-speech-and-audio/04-speech-recognition-asr) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/06-speech-and-audio/04-speech-recognition-asr/docs/en.md) — Speech recognition is audio classification at every timestep, glued together by a sequence model that knows English and silence. CTC, RNN-T, and attention are the three ways to …
- [Whisper: Architecture & Fine-Tuning](https://aiengineeringfromscratch.com/lesson.html?path=phases/06-speech-and-audio/05-whisper-architecture-finetuning) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/06-speech-and-audio/05-whisper-architecture-finetuning/docs/en.md) — Whisper is a 30-second-window transformer encoder-decoder, trained on 680k hours of multilingual weakly-supervised audio-text pairs. One architecture, multiple tasks, robust acr…
- [Speaker Recognition & Verification](https://aiengineeringfromscratch.com/lesson.html?path=phases/06-speech-and-audio/06-speaker-recognition-verification) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/06-speech-and-audio/06-speaker-recognition-verification/docs/en.md) — ASR asks "what did they say?" Speaker recognition asks "who said it?" The math looks the same — embeddings plus cosine — but every production decision hinges on a single EER num…
- [Text-to-Speech (TTS)](https://aiengineeringfromscratch.com/lesson.html?path=phases/06-speech-and-audio/07-text-to-speech) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/06-speech-and-audio/07-text-to-speech/docs/en.md) — ASR inverts speech to text; TTS inverts text to speech. The 2026 stack is three parts: text → tokens, tokens → mel, mel → waveform. Each part has a default model that fits in a …
- [Voice Cloning & Voice Conversion](https://aiengineeringfromscratch.com/lesson.html?path=phases/06-speech-and-audio/08-voice-cloning-conversion) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/06-speech-and-audio/08-voice-cloning-conversion/docs/en.md) — Voice cloning reads your text in someone else's voice. Voice conversion rewrites your voice into someone else's while preserving what you said. Both hang on the same decompositi…
- [Music Generation](https://aiengineeringfromscratch.com/lesson.html?path=phases/06-speech-and-audio/09-music-generation) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/06-speech-and-audio/09-music-generation/docs/en.md) — 2026 music generation: Suno v5 and Udio v4 dominate commercial; MusicGen, Stable Audio Open, and ACE-Step lead open-source. The technical problem is mostly solved. The legal pro…
- [Audio-Language Models](https://aiengineeringfromscratch.com/lesson.html?path=phases/06-speech-and-audio/10-audio-language-models) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/06-speech-and-audio/10-audio-language-models/docs/en.md) — 2026 audio-language models reason over speech + environmental sound + music. Qwen2.5-Omni-7B matches GPT-4o Audio on MMAU-Pro. Audio Flamingo Next beats Gemini 2.5 Pro on LongAu…
- [Real-Time Audio Processing](https://aiengineeringfromscratch.com/lesson.html?path=phases/06-speech-and-audio/11-real-time-audio-processing) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/06-speech-and-audio/11-real-time-audio-processing/docs/en.md) — Batch pipelines process a file. Real-time pipelines process the next 20 milliseconds before the next 20 arrive. Every conversational AI, broadcast studio, and telephony bot live…
- [Build a Voice Assistant Pipeline](https://aiengineeringfromscratch.com/lesson.html?path=phases/06-speech-and-audio/12-voice-assistant-pipeline) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/06-speech-and-audio/12-voice-assistant-pipeline/docs/en.md) — Everything from lessons 01-11, stitched together. Build a voice assistant that listens, reasons, and talks back. In 2026 that is a solved engineering problem, not a research pro…
- [Neural Audio Codecs — EnCodec, SNAC, Mimi, DAC](https://aiengineeringfromscratch.com/lesson.html?path=phases/06-speech-and-audio/13-neural-audio-codecs) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/06-speech-and-audio/13-neural-audio-codecs/docs/en.md) — 2026 audio generation is almost all tokens. EnCodec, SNAC, Mimi, and DAC turn continuous waveforms into discrete sequences that a transformer can predict. The semantic-vs-acoust…
- [Voice Activity Detection & Turn-Taking](https://aiengineeringfromscratch.com/lesson.html?path=phases/06-speech-and-audio/14-voice-activity-detection-turn-taking) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/06-speech-and-audio/14-voice-activity-detection-turn-taking/docs/en.md) — Every voice agent lives or dies on two decisions: is the user speaking now, and are they done? VAD answers the first. Turn-detection (VAD + silence-hangover + semantic endpoint …
- [Streaming Speech-to-Speech — Moshi, Hibiki](https://aiengineeringfromscratch.com/lesson.html?path=phases/06-speech-and-audio/15-streaming-speech-to-speech-moshi-hibiki) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/06-speech-and-audio/15-streaming-speech-to-speech-moshi-hibiki/docs/en.md) — 2024-2026 redefined voice AI. Moshi ships a single model that listens and speaks simultaneously at 200 ms latency. Hibiki does speech-to-speech translation chunk-by-chunk. Both …
- [Voice Anti-Spoofing & Audio Watermarking](https://aiengineeringfromscratch.com/lesson.html?path=phases/06-speech-and-audio/16-anti-spoofing-audio-watermarking) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/06-speech-and-audio/16-anti-spoofing-audio-watermarking/docs/en.md) — Voice cloning shipped faster than defenses. 2026 production voice systems need two things: a detector (AASIST, RawNet2) that classifies real vs fake speech, and a watermark (Aud…
- [Audio Evaluation — WER, MOS, MMAU, Leaderboards](https://aiengineeringfromscratch.com/lesson.html?path=phases/06-speech-and-audio/17-audio-evaluation-metrics) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/06-speech-and-audio/17-audio-evaluation-metrics/docs/en.md) — You cannot ship what you cannot measure. This lesson names the 2026 metrics for every audio task: ASR (WER, CER, RTFx), TTS (MOS, UTMOS, SECS, WER-on-ASR-round-trip), audio-lang…

## Phase 7: Transformers Deep Dive
The architecture that changed everything.

- [Why Transformers: The Problems with RNNs](https://aiengineeringfromscratch.com/lesson.html?path=phases/07-transformers-deep-dive/01-why-transformers) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/07-transformers-deep-dive/01-why-transformers/docs/en.md) — RNNs process tokens one at a time. Transformers process all tokens at once. That single architectural bet changed every scaling curve in deep learning after 2017.
- [Self-Attention from Scratch](https://aiengineeringfromscratch.com/lesson.html?path=phases/07-transformers-deep-dive/02-self-attention-from-scratch) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/07-transformers-deep-dive/02-self-attention-from-scratch/docs/en.md) — Attention is a lookup table where every word asks "who matters to me?" - and learns the answer.
- [Multi-Head Attention](https://aiengineeringfromscratch.com/lesson.html?path=phases/07-transformers-deep-dive/03-multi-head-attention) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/07-transformers-deep-dive/03-multi-head-attention/docs/en.md) — One attention head learns one relation at a time. Eight heads learn eight. Heads are free. Take more of them.
- [Positional Encoding: Sinusoidal, RoPE, ALiBi](https://aiengineeringfromscratch.com/lesson.html?path=phases/07-transformers-deep-dive/04-positional-encoding) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/07-transformers-deep-dive/04-positional-encoding/docs/en.md) — Attention is permutation-invariant. "The cat sat on the mat" and "mat the on sat cat the" produce the same output without positional signal. Three algorithms fix it — each with …
- [The Full Transformer: Encoder + Decoder](https://aiengineeringfromscratch.com/lesson.html?path=phases/07-transformers-deep-dive/05-full-transformer) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/07-transformers-deep-dive/05-full-transformer/docs/en.md) — Attention is the star. Everything else — residuals, normalization, feed-forward, cross-attention — is the scaffolding that lets you stack it deep.
- [BERT — Masked Language Modeling](https://aiengineeringfromscratch.com/lesson.html?path=phases/07-transformers-deep-dive/06-bert-masked-language-modeling) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/07-transformers-deep-dive/06-bert-masked-language-modeling/docs/en.md) — GPT predicts the next word. BERT predicts a missing word. One sentence of difference — and half a decade of everything embedding-shaped.
- [GPT — Causal Language Modeling](https://aiengineeringfromscratch.com/lesson.html?path=phases/07-transformers-deep-dive/07-gpt-causal-language-modeling) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/07-transformers-deep-dive/07-gpt-causal-language-modeling/docs/en.md) — BERT sees both sides. GPT sees only the past. The triangle mask is the most consequential single line of code in modern AI.
- [T5, BART — Encoder-Decoder Models](https://aiengineeringfromscratch.com/lesson.html?path=phases/07-transformers-deep-dive/08-t5-bart-encoder-decoder) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/07-transformers-deep-dive/08-t5-bart-encoder-decoder/docs/en.md) — Encoders understand. Decoders generate. Put them back together and you get a model built for input → output tasks: translate, summarize, rewrite, transcribe.
- [Vision Transformers (ViT)](https://aiengineeringfromscratch.com/lesson.html?path=phases/07-transformers-deep-dive/09-vision-transformers) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/07-transformers-deep-dive/09-vision-transformers/docs/en.md) — An image is a grid of patches. A sentence is a grid of tokens. The same transformer eats both.
- [Audio Transformers — Whisper Architecture](https://aiengineeringfromscratch.com/lesson.html?path=phases/07-transformers-deep-dive/10-audio-transformers-whisper) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/07-transformers-deep-dive/10-audio-transformers-whisper/docs/en.md) — Audio is an image of frequency over time. Whisper is a ViT that eats mel spectrograms and speaks back.
- [Mixture of Experts (MoE)](https://aiengineeringfromscratch.com/lesson.html?path=phases/07-transformers-deep-dive/11-mixture-of-experts) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/07-transformers-deep-dive/11-mixture-of-experts/docs/en.md) — A dense 70B transformer activates every parameter for every token. A 671B MoE activates only 37B per token and beats it on every benchmark. Sparsity is the most important scalin…
- [KV Cache, Flash Attention & Inference Optimization](https://aiengineeringfromscratch.com/lesson.html?path=phases/07-transformers-deep-dive/12-kv-cache-flash-attention) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/07-transformers-deep-dive/12-kv-cache-flash-attention/docs/en.md) — Training is parallel and FLOP-bound. Inference is serial and memory-bound. Different bottleneck, different tricks.
- [Scaling Laws](https://aiengineeringfromscratch.com/lesson.html?path=phases/07-transformers-deep-dive/13-scaling-laws) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/07-transformers-deep-dive/13-scaling-laws/docs/en.md) — The 2020 Kaplan paper said: bigger model, lower loss. The 2022 Hoffmann paper said: you were under-training. Compute goes into two buckets — parameters and tokens — and the spli…
- [Build a Transformer from Scratch](https://aiengineeringfromscratch.com/lesson.html?path=phases/07-transformers-deep-dive/14-build-a-transformer-capstone) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/07-transformers-deep-dive/14-build-a-transformer-capstone/docs/en.md) — Thirteen lessons. One model. No shortcuts.
- [Attention Variants — Sliding Window, Sparse, Differential](https://aiengineeringfromscratch.com/lesson.html?path=phases/07-transformers-deep-dive/15-attention-variants) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/07-transformers-deep-dive/15-attention-variants/docs/en.md) — Full attention is a circle. Every token sees every token, and memory pays the price. Four variants bend the shape of the circle and recover half the cost.
- [Speculative Decoding — Draft, Verify, Repeat](https://aiengineeringfromscratch.com/lesson.html?path=phases/07-transformers-deep-dive/16-speculative-decoding) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/07-transformers-deep-dive/16-speculative-decoding/docs/en.md) — Autoregressive decoding is serial. Each token waits for the previous one. Speculative decoding breaks the chain: a cheap model drafts N tokens, the expensive model verifies all …

## Phase 8: Generative AI
Create images, video, audio, 3D, and more.

- [Generative Models: Taxonomy & History](https://aiengineeringfromscratch.com/lesson.html?path=phases/08-generative-ai/01-generative-models-taxonomy-history) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/08-generative-ai/01-generative-models-taxonomy-history/docs/en.md) — Every image model, text model, video model, and 3D model fits in one of five buckets. Pick the wrong bucket and you will fight the math for weeks. Pick the right one and the fie…
- [Autoencoders & VAE](https://aiengineeringfromscratch.com/lesson.html?path=phases/08-generative-ai/02-autoencoders-vae) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/08-generative-ai/02-autoencoders-vae/docs/en.md) — A plain autoencoder compresses then reconstructs. It memorizes. It does not generate. Add one trick — force the code to look Gaussian — and you get a sampler. That single trick,…
- [GANs: Generator vs Discriminator](https://aiengineeringfromscratch.com/lesson.html?path=phases/08-generative-ai/03-gans-generator-discriminator) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/08-generative-ai/03-gans-generator-discriminator/docs/en.md) — Goodfellow's trick in 2014 was to skip density entirely. Two networks. One makes fakes. One catches them. They fight until the fakes are indistinguishable from real. It shouldn'…
- [Conditional GANs & Pix2Pix](https://aiengineeringfromscratch.com/lesson.html?path=phases/08-generative-ai/04-conditional-gans-pix2pix) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/08-generative-ai/04-conditional-gans-pix2pix/docs/en.md) — The first big unlock of 2014-2017 was controlling what a GAN makes. Attach a label, or an image, or a sentence. Pix2Pix did the image version and it still beats every generic te…
- [StyleGAN](https://aiengineeringfromscratch.com/lesson.html?path=phases/08-generative-ai/05-stylegan) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/08-generative-ai/05-stylegan/docs/en.md) — Most generators stir `z` into every layer at the same time. StyleGAN split it apart: first map `z` to an intermediate `w`, then *inject* `w` at every resolution level through Ad…
- [Diffusion Models — DDPM from Scratch](https://aiengineeringfromscratch.com/lesson.html?path=phases/08-generative-ai/06-diffusion-ddpm-from-scratch) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/08-generative-ai/06-diffusion-ddpm-from-scratch/docs/en.md) — Ho, Jain, Abbeel (2020) gave the field a recipe it could not quit. Destroy the data with noise over a thousand small steps. Train one neural net to predict the noise. Reverse th…
- [Latent Diffusion & Stable Diffusion](https://aiengineeringfromscratch.com/lesson.html?path=phases/08-generative-ai/07-latent-diffusion-stable-diffusion) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/08-generative-ai/07-latent-diffusion-stable-diffusion/docs/en.md) — Pixel-space diffusion on 512×512 images is a computational war crime. Rombach et al. (2022) noticed that you do not need all 786k dimensions to generate an image — you need enou…
- [ControlNet, LoRA & Conditioning](https://aiengineeringfromscratch.com/lesson.html?path=phases/08-generative-ai/08-controlnet-lora-conditioning) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/08-generative-ai/08-controlnet-lora-conditioning/docs/en.md) — Text alone is a clumsy control signal. ControlNet lets you clone a pretrained diffusion model and steer it with a depth map, pose skeleton, scribble, or edge image. LoRA lets yo…
- [Inpainting, Outpainting & Editing](https://aiengineeringfromscratch.com/lesson.html?path=phases/08-generative-ai/09-inpainting-outpainting-editing) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/08-generative-ai/09-inpainting-outpainting-editing/docs/en.md) — Text-to-image makes new things. Inpainting fixes old ones. In production, 70% of billable image work is editing — swap a background, remove a logo, extend the canvas, regenerate…
- [Video Generation](https://aiengineeringfromscratch.com/lesson.html?path=phases/08-generative-ai/10-video-generation) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/08-generative-ai/10-video-generation/docs/en.md) — An image is a 2-D tensor. A video is a 3-D one. The theory is the same; the compute is 10-100x harder. OpenAI's Sora (Feb 2024) proved it was possible. By 2026 Veo 2, Kling 1.5,…
- [Audio Generation](https://aiengineeringfromscratch.com/lesson.html?path=phases/08-generative-ai/11-audio-generation) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/08-generative-ai/11-audio-generation/docs/en.md) — Audio is a 1-D signal at 16-48 kHz. A five-second clip is 80-240k samples. No transformer attends to that sequence directly. The solution for every production audio model in 202…
- [3D Generation](https://aiengineeringfromscratch.com/lesson.html?path=phases/08-generative-ai/12-3d-generation) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/08-generative-ai/12-3d-generation/docs/en.md) — 3D is the modality where 2D-to-3D leverage is strongest. The 2023 breakthrough was 3D Gaussian Splatting. The 2024-2026 generative push layers multi-view diffusion + 3D reconstr…
- [Flow Matching & Rectified Flows](https://aiengineeringfromscratch.com/lesson.html?path=phases/08-generative-ai/13-flow-matching-rectified-flows) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/08-generative-ai/13-flow-matching-rectified-flows/docs/en.md) — Diffusion models take 20-50 sampling steps because they walk a curved path from noise to data. Flow matching (Lipman et al., 2023) and rectified flow (Liu et al., 2022) trained …
- [Evaluation: FID, CLIP Score](https://aiengineeringfromscratch.com/lesson.html?path=phases/08-generative-ai/14-evaluation-fid-clip-score) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/08-generative-ai/14-evaluation-fid-clip-score/docs/en.md) — Every generative model leaderboard cites FID, CLIP score, and a win rate from a human-preference arena. Each number has a failure mode a determined researcher can game. If you d…
- [Visual Autoregressive Modeling (VAR): Next-Scale Prediction](https://aiengineeringfromscratch.com/lesson.html?path=phases/08-generative-ai/19-visual-autoregressive-var) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/08-generative-ai/19-visual-autoregressive-var/docs/en.md) — Diffusion models sample iteratively in time (denoising steps). VAR samples iteratively in scale — it predicts a 1x1 token, then 2x2, then 4x4, up to the final resolution, each s…

## Phase 9: Reinforcement Learning
The foundation of RLHF and game-playing AI.

- [MDPs, States, Actions & Rewards](https://aiengineeringfromscratch.com/lesson.html?path=phases/09-reinforcement-learning/01-mdps-states-actions-rewards) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/09-reinforcement-learning/01-mdps-states-actions-rewards/docs/en.md) — A Markov Decision Process is five things: states, actions, transitions, rewards, a discount. Everything in RL — Q-learning, PPO, DPO, GRPO — optimizes over this shape. Learn it …
- [Dynamic Programming](https://aiengineeringfromscratch.com/lesson.html?path=phases/09-reinforcement-learning/02-dynamic-programming) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/09-reinforcement-learning/02-dynamic-programming/docs/en.md) — Dynamic programming is RL with cheating. You already know the transition and reward functions; you just iterate the Bellman equation until `V` or `π` stops moving. It is the ben…
- [Monte Carlo Methods](https://aiengineeringfromscratch.com/lesson.html?path=phases/09-reinforcement-learning/03-monte-carlo-methods) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/09-reinforcement-learning/03-monte-carlo-methods/docs/en.md) — Dynamic programming needs a model. Monte Carlo needs nothing but episodes. Run the policy, watch the returns, average them. The simplest idea in RL — and the one that unlocks ev…
- [Q-Learning, SARSA](https://aiengineeringfromscratch.com/lesson.html?path=phases/09-reinforcement-learning/04-q-learning-sarsa) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/09-reinforcement-learning/04-q-learning-sarsa/docs/en.md) — Monte Carlo waits until the episode ends. TD updates after every step by bootstrapping the next value estimate. Q-learning is off-policy and optimistic; SARSA is on-policy and c…
- [Deep Q-Networks (DQN)](https://aiengineeringfromscratch.com/lesson.html?path=phases/09-reinforcement-learning/05-dqn) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/09-reinforcement-learning/05-dqn/docs/en.md) — 2013: Mnih trained one Q-learning network on raw pixels, beat every classical RL agent on seven Atari games. 2015: extended to 49 games, published in Nature, sparked the deep-RL…
- [Policy Gradients — REINFORCE](https://aiengineeringfromscratch.com/lesson.html?path=phases/09-reinforcement-learning/06-policy-gradients-reinforce) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/09-reinforcement-learning/06-policy-gradients-reinforce/docs/en.md) — Stop estimating value. Parameterize the policy directly, compute the gradient of expected return, step uphill. Williams (1992) wrote it in one theorem. It is why PPO, GRPO, and …
- [Actor-Critic — A2C, A3C](https://aiengineeringfromscratch.com/lesson.html?path=phases/09-reinforcement-learning/07-actor-critic-a2c-a3c) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/09-reinforcement-learning/07-actor-critic-a2c-a3c/docs/en.md) — REINFORCE is noisy. Add a critic that learns `V̂(s)`, subtract it from the return, and you get an advantage that has the same expectation but far lower variance. That is actor-c…
- [PPO](https://aiengineeringfromscratch.com/lesson.html?path=phases/09-reinforcement-learning/08-ppo) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/09-reinforcement-learning/08-ppo/docs/en.md) — A2C throws away each rollout after one update. PPO wraps the policy gradient in a clipped importance ratio so you can do 10+ epochs on the same data without the policy exploding…
- [Reward Modeling & RLHF](https://aiengineeringfromscratch.com/lesson.html?path=phases/09-reinforcement-learning/09-reward-modeling-rlhf) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/09-reinforcement-learning/09-reward-modeling-rlhf/docs/en.md) — Humans cannot write a reward function for "good assistant response," but they can compare two responses and pick the better one. Fit a reward model to those comparisons, then RL…
- [Multi-Agent RL](https://aiengineeringfromscratch.com/lesson.html?path=phases/09-reinforcement-learning/10-multi-agent-rl) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/09-reinforcement-learning/10-multi-agent-rl/docs/en.md) — Single-agent RL assumes the environment is stationary. Put two learning agents in the same world and that assumption breaks: each agent is part of the other's environment, and b…
- [Sim-to-Real Transfer](https://aiengineeringfromscratch.com/lesson.html?path=phases/09-reinforcement-learning/11-sim-to-real-transfer) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/09-reinforcement-learning/11-sim-to-real-transfer/docs/en.md) — A policy trained in a simulator that fails on hardware is a policy that memorized the simulator. Domain randomization, domain adaptation, and system identification are the three…
- [RL for Games](https://aiengineeringfromscratch.com/lesson.html?path=phases/09-reinforcement-learning/12-rl-for-games) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/09-reinforcement-learning/12-rl-for-games/docs/en.md) — 1992: TD-Gammon beat human champions at backgammon with pure TD. 2016: AlphaGo beat Lee Sedol. 2017: AlphaZero dominated chess, shogi, and Go from scratch. 2024: DeepSeek-R1 pro…

## Phase 10: LLMs from Scratch
Build, train, and understand large language models.

- [Tokenizers: BPE, WordPiece, SentencePiece](https://aiengineeringfromscratch.com/lesson.html?path=phases/10-llms-from-scratch/01-tokenizers) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/10-llms-from-scratch/01-tokenizers/docs/en.md) — Your LLM does not read English. It reads integers. The tokenizer decides whether those integers carry meaning or waste it.
- [Building a Tokenizer from Scratch](https://aiengineeringfromscratch.com/lesson.html?path=phases/10-llms-from-scratch/02-building-a-tokenizer) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/10-llms-from-scratch/02-building-a-tokenizer/docs/en.md) — Lesson 01 gave you a toy. This lesson gives you a weapon.
- [Data Pipelines for Pre-Training](https://aiengineeringfromscratch.com/lesson.html?path=phases/10-llms-from-scratch/03-data-pipelines) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/10-llms-from-scratch/03-data-pipelines/docs/en.md) — The model is a mirror. It reflects whatever data you feed it. Feed it garbage, it reflects garbage with perfect fluency.
- [Pre-Training a Mini GPT (124M)](https://aiengineeringfromscratch.com/lesson.html?path=phases/10-llms-from-scratch/04-pre-training-mini-gpt) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/10-llms-from-scratch/04-pre-training-mini-gpt/docs/en.md) — GPT-2 Small has 124 million parameters. That's 12 transformer layers, 12 attention heads, and 768-dimensional embeddings. You can train it from scratch on a single GPU in a few …
- [Distributed Training, FSDP, DeepSpeed](https://aiengineeringfromscratch.com/lesson.html?path=phases/10-llms-from-scratch/05-scaling-distributed) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/10-llms-from-scratch/05-scaling-distributed/docs/en.md) — Your 124M model trained on one GPU. Now try 7 billion parameters. The model doesn't fit in memory. The data takes weeks on a single machine. Distributed training isn't optional …
- [Instruction Tuning — SFT](https://aiengineeringfromscratch.com/lesson.html?path=phases/10-llms-from-scratch/06-instruction-tuning-sft) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/10-llms-from-scratch/06-instruction-tuning-sft/docs/en.md) — A base model predicts the next token. That's it. It doesn't follow instructions, answer questions, or refuse harmful requests. SFT is the bridge between a token predictor and a …
- [RLHF — Reward Model + PPO](https://aiengineeringfromscratch.com/lesson.html?path=phases/10-llms-from-scratch/07-rlhf) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/10-llms-from-scratch/07-rlhf/docs/en.md) — SFT teaches the model to follow instructions. But it doesn't teach the model which response is BETTER. Two grammatically correct, factually accurate answers can differ enormousl…
- [DPO — Direct Preference Optimization](https://aiengineeringfromscratch.com/lesson.html?path=phases/10-llms-from-scratch/08-dpo) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/10-llms-from-scratch/08-dpo/docs/en.md) — RLHF works. It also requires training three models (SFT, reward model, policy), managing PPO's instability, and tuning a KL penalty. DPO asks: what if you could skip all of that…
- [Constitutional AI & Self-Improvement](https://aiengineeringfromscratch.com/lesson.html?path=phases/10-llms-from-scratch/09-constitutional-ai-self-improvement) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/10-llms-from-scratch/09-constitutional-ai-self-improvement/docs/en.md) — RLHF needs humans in the loop. Constitutional AI replaces most of them with the model itself. Write a list of principles, have the model critique its own outputs against those p…
- [Evaluation — Benchmarks, Evals](https://aiengineeringfromscratch.com/lesson.html?path=phases/10-llms-from-scratch/10-evaluation) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/10-llms-from-scratch/10-evaluation/docs/en.md) — Goodhart's Law: when a measure becomes a target, it ceases to be a good measure. Every frontier lab games benchmarks. MMLU scores go up while models still can't reliably count t…
- [Quantization: INT8, GPTQ, AWQ, GGUF](https://aiengineeringfromscratch.com/lesson.html?path=phases/10-llms-from-scratch/11-quantization) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/10-llms-from-scratch/11-quantization/docs/en.md) — A 70B model in FP16 needs 140GB. Two A100s just for weights. Quantize to FP8: one 80GB GPU. INT4: a MacBook.
- [Inference Optimization](https://aiengineeringfromscratch.com/lesson.html?path=phases/10-llms-from-scratch/12-inference-optimization) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/10-llms-from-scratch/12-inference-optimization/docs/en.md) — Two phases define LLM inference. Prefill processes your prompt in parallel -- compute-bound. Decode generates tokens one at a time -- memory-bound. Every optimization targets on…
- [Building a Complete LLM Pipeline](https://aiengineeringfromscratch.com/lesson.html?path=phases/10-llms-from-scratch/13-building-complete-llm-pipeline) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/10-llms-from-scratch/13-building-complete-llm-pipeline/docs/en.md) — Everything from Lessons 01 to 12 is one stage of one pipeline. This lesson is the scaffold that turns those stages into a single end-to-end run: tokenize, pre-train, scale, SFT,…
- [Open Models: Architecture Walkthroughs](https://aiengineeringfromscratch.com/lesson.html?path=phases/10-llms-from-scratch/14-open-models-architecture-walkthroughs) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/10-llms-from-scratch/14-open-models-architecture-walkthroughs/docs/en.md) — You built a GPT-2 Small from scratch in Lesson 04. Frontier open models in 2026 are the same family with five or six concrete changes. RMSNorm instead of LayerNorm. SwiGLU inste…
- [Speculative Decoding and EAGLE-3](https://aiengineeringfromscratch.com/lesson.html?path=phases/10-llms-from-scratch/15-speculative-decoding-eagle3) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/10-llms-from-scratch/15-speculative-decoding-eagle3/docs/en.md) — Phase 7 · Lesson 16 proved the math: the Leviathan rejection rule preserves the verifier's distribution exactly. This lesson is the training-stack view of 2026 production specul…
- [Differential Attention (V2)](https://aiengineeringfromscratch.com/lesson.html?path=phases/10-llms-from-scratch/16-differential-attention-v2) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/10-llms-from-scratch/16-differential-attention-v2/docs/en.md) — Softmax attention spreads a small amount of probability over every non-matching token. Over 100k tokens that noise adds up and drowns the signal. Differential Transformer (Ye et…
- [Native Sparse Attention (DeepSeek NSA)](https://aiengineeringfromscratch.com/lesson.html?path=phases/10-llms-from-scratch/17-native-sparse-attention) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/10-llms-from-scratch/17-native-sparse-attention/docs/en.md) — At 64k tokens, attention eats 70-80% of decode latency. Every open-model lab has a plan to fix it. DeepSeek's NSA (ACL 2025 best paper) is the one that stuck: three parallel att…
- [Multi-Token Prediction (MTP)](https://aiengineeringfromscratch.com/lesson.html?path=phases/10-llms-from-scratch/18-multi-token-prediction) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/10-llms-from-scratch/18-multi-token-prediction/docs/en.md) — Every autoregressive LLM from GPT-2 to Llama 3 trains on one loss per position: predict the next token. DeepSeek-V3 added a second loss per position: predict the token after tha…
- [DualPipe Parallelism](https://aiengineeringfromscratch.com/lesson.html?path=phases/10-llms-from-scratch/19-dualpipe-parallelism) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/10-llms-from-scratch/19-dualpipe-parallelism/docs/en.md) — DeepSeek-V3 was trained on 2,048 H800 GPUs with MoE experts scattered across nodes. Cross-node expert all-to-all communication cost 1 GPU-hour of comm for every 1 GPU-hour of co…
- [DeepSeek-V3 Architecture Walkthrough](https://aiengineeringfromscratch.com/lesson.html?path=phases/10-llms-from-scratch/20-deepseek-v3-walkthrough) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/10-llms-from-scratch/20-deepseek-v3-walkthrough/docs/en.md) — Phase 10 · Lesson 14 named the six architectural knobs every open model turns. DeepSeek-V3 (December 2024, 671B parameters total, 37B active) turns all six and adds four more: M…
- [Jamba — Hybrid SSM-Transformer](https://aiengineeringfromscratch.com/lesson.html?path=phases/10-llms-from-scratch/21-jamba-hybrid-ssm-transformer) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/10-llms-from-scratch/21-jamba-hybrid-ssm-transformer/docs/en.md) — State space models (SSMs) and transformers want different things. Transformers buy quality via attention at quadratic cost. SSMs buy linear-time inference and constant memory vi…
- [Async and Hogwild! Inference](https://aiengineeringfromscratch.com/lesson.html?path=phases/10-llms-from-scratch/22-async-hogwild-inference) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/10-llms-from-scratch/22-async-hogwild-inference/docs/en.md) — Speculative decoding (Phase 10 · 15) parallelizes tokens within one sequence. Multi-agent frameworks parallelize across whole sequences but force explicit coordination (voting, …
- [Speculative Decoding and EAGLE](https://aiengineeringfromscratch.com/lesson.html?path=phases/10-llms-from-scratch/25-speculative-decoding) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/10-llms-from-scratch/25-speculative-decoding/docs/en.md) — A frontier LLM generating one token requires a full forward pass over billions of parameters. That forward pass is massively over-provisioned: most of the time a much smaller mo…
- [Gradient Checkpointing and Activation Recomputation](https://aiengineeringfromscratch.com/lesson.html?path=phases/10-llms-from-scratch/34-gradient-checkpointing) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/10-llms-from-scratch/34-gradient-checkpointing/docs/en.md) — Backprop keeps every intermediate activation. At 70B parameters and 128K context that is 3 TB of activations per rank. Checkpointing trades FLOPs for memory: recompute instead o…

## Phase 11: LLM Engineering
Put LLMs to work in production.

- [Prompt Engineering: Techniques & Patterns](https://aiengineeringfromscratch.com/lesson.html?path=phases/11-llm-engineering/01-prompt-engineering) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/11-llm-engineering/01-prompt-engineering/docs/en.md) — Most people write prompts like they are texting a friend. Then they wonder why a 200-billion parameter model gives mediocre answers. Prompt engineering is not about tricks. It i…
- [Few-Shot, CoT, Tree-of-Thought](https://aiengineeringfromscratch.com/lesson.html?path=phases/11-llm-engineering/02-few-shot-cot) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/11-llm-engineering/02-few-shot-cot/docs/en.md) — Telling a model what to do is prompting. Showing it how to think is engineering. The gap between 78% and 91% accuracy on the same model, same task, same data is not a better mod…
- [Structured Outputs](https://aiengineeringfromscratch.com/lesson.html?path=phases/11-llm-engineering/03-structured-outputs) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/11-llm-engineering/03-structured-outputs/docs/en.md) — Your LLM returns a string. Your application needs JSON. That gap has crashed more production systems than any model hallucination. Structured output is the bridge between natura…
- [Embeddings & Vector Representations](https://aiengineeringfromscratch.com/lesson.html?path=phases/11-llm-engineering/04-embeddings) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/11-llm-engineering/04-embeddings/docs/en.md) — Text is discrete. Math is continuous. Every time you ask an LLM to find "similar" documents, compare meanings, or search beyond keywords, you're relying on a bridge between thes…
- [Context Engineering](https://aiengineeringfromscratch.com/lesson.html?path=phases/11-llm-engineering/05-context-engineering) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/11-llm-engineering/05-context-engineering/docs/en.md) — Prompt engineering is a subset. Context engineering is the whole game. A prompt is a string you type. Context is everything that goes into the model's window: system instruction…
- [RAG: Retrieval-Augmented Generation](https://aiengineeringfromscratch.com/lesson.html?path=phases/11-llm-engineering/06-rag) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/11-llm-engineering/06-rag/docs/en.md) — Your LLM knows everything up to its training cutoff. It knows nothing about your company's docs, your codebase, or last week's meeting notes. RAG solves this by retrieving relev…
- [Advanced RAG: Chunking, Reranking](https://aiengineeringfromscratch.com/lesson.html?path=phases/11-llm-engineering/07-advanced-rag) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/11-llm-engineering/07-advanced-rag/docs/en.md) — Basic RAG retrieves the top-k most similar chunks. That works for simple questions. It falls apart for multi-hop reasoning, ambiguous queries, and large corpora. Advanced RAG is…
- [Fine-Tuning with LoRA & QLoRA](https://aiengineeringfromscratch.com/lesson.html?path=phases/11-llm-engineering/08-fine-tuning-lora) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/11-llm-engineering/08-fine-tuning-lora/docs/en.md) — Full fine-tuning a 7B model requires 56GB of VRAM. You don't have that. Neither do most companies. LoRA lets you fine-tune the same model in 6GB by training less than 1% of the …
- [Function Calling & Tool Use](https://aiengineeringfromscratch.com/lesson.html?path=phases/11-llm-engineering/09-function-calling) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/11-llm-engineering/09-function-calling/docs/en.md) — LLMs cannot do anything. They generate text. That is the entire capability. They cannot check the weather, query a database, send an email, run code, or read a file. Every "AI a…
- [Evaluation & Testing](https://aiengineeringfromscratch.com/lesson.html?path=phases/11-llm-engineering/10-evaluation) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/11-llm-engineering/10-evaluation/docs/en.md) — You would never deploy a web app without tests. You would never ship a database migration without a rollback plan. But right now, most teams ship LLM applications by reading 10 …
- [Caching, Rate Limiting & Cost](https://aiengineeringfromscratch.com/lesson.html?path=phases/11-llm-engineering/11-caching-cost) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/11-llm-engineering/11-caching-cost/docs/en.md) — Most AI startups do not die from bad models. They die from bad unit economics. A single GPT-4o call costs fractions of a cent. Ten thousand users making ten calls per day costs …
- [Guardrails & Safety](https://aiengineeringfromscratch.com/lesson.html?path=phases/11-llm-engineering/12-guardrails) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/11-llm-engineering/12-guardrails/docs/en.md) — Your LLM application will be attacked. Not might. Will. The first prompt injection attempt against your production system will come within 48 hours of launch. The question is no…
- [Building a Production LLM App](https://aiengineeringfromscratch.com/lesson.html?path=phases/11-llm-engineering/13-production-app) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/11-llm-engineering/13-production-app/docs/en.md) — You have built prompts, embeddings, RAG pipelines, function calling, caching layers, and guardrails. Separately. In isolation. Like practicing guitar scales without ever playing…
- [Model Context Protocol (MCP)](https://aiengineeringfromscratch.com/lesson.html?path=phases/11-llm-engineering/14-model-context-protocol) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/11-llm-engineering/14-model-context-protocol/docs/en.md) — MCP gives an AI host one protocol for discovering and invoking tools, resources, and prompts. The 2026-07-28 revision makes that protocol stateless: capability and version conte…
- [Prompt Caching & Context Caching](https://aiengineeringfromscratch.com/lesson.html?path=phases/11-llm-engineering/15-prompt-caching) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/11-llm-engineering/15-prompt-caching/docs/en.md) — Your system prompt is 4,000 tokens. Your RAG context is 20,000 tokens. You send both with every request. You also pay for both — every time. Prompt caching lets the provider kee…
- [Agent State Machines — Graphs, Nodes, Checkpoints](https://aiengineeringfromscratch.com/lesson.html?path=phases/11-llm-engineering/16-langgraph-state-machines) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/11-llm-engineering/16-langgraph-state-machines/docs/en.md) — A ReAct loop written by hand is a `while True`. The same loop written as an explicit graph is something you can checkpoint, interrupt, branch, and time-travel through. The agent…
- [Agent Framework Tradeoffs](https://aiengineeringfromscratch.com/lesson.html?path=phases/11-llm-engineering/17-agent-framework-tradeoffs) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/11-llm-engineering/17-agent-framework-tradeoffs/docs/en.md) — Every framework sells the same demo (research agent builds a report) and hides the same bug (state schema fights with the orchestration layer). Pick the framework whose abstract…

## Phase 12: Multimodal AI
See, hear, read, and reason across modalities — from ViT patches to computer-use agents.

- [Vision Transformers and the Patch-Token Primitive](https://aiengineeringfromscratch.com/lesson.html?path=phases/12-multimodal-ai/01-vision-transformer-patch-tokens) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/12-multimodal-ai/01-vision-transformer-patch-tokens/docs/en.md) — Before anything multimodal, an image has to become a sequence of tokens a transformer can eat. The 2020 ViT paper answered this with 16x16 pixel patches, a linear projection, an…
- [CLIP and Contrastive Vision-Language Pretraining](https://aiengineeringfromscratch.com/lesson.html?path=phases/12-multimodal-ai/02-clip-contrastive-pretraining) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/12-multimodal-ai/02-clip-contrastive-pretraining/docs/en.md) — OpenAI's CLIP (2021) proved a single idea big enough to power the next five years: align an image encoder and a text encoder in the same vector space using only noisy web image-…
- [BLIP-2 Q-Former as Modality Bridge](https://aiengineeringfromscratch.com/lesson.html?path=phases/12-multimodal-ai/03-blip2-qformer-bridge) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/12-multimodal-ai/03-blip2-qformer-bridge/docs/en.md) — CLIP aligns image and text but cannot generate captions, answer questions, or hold a conversation. BLIP-2 (Salesforce, 2023) solved that with a small trainable bridge: 32 learna…
- [Flamingo and Gated Cross-Attention](https://aiengineeringfromscratch.com/lesson.html?path=phases/12-multimodal-ai/04-flamingo-gated-cross-attention) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/12-multimodal-ai/04-flamingo-gated-cross-attention/docs/en.md) — DeepMind's Flamingo (2022) did two things before anyone else. It showed a single model could process arbitrarily interleaved sequences of images, videos, and text. And it showed…
- [LLaVA and Visual Instruction Tuning](https://aiengineeringfromscratch.com/lesson.html?path=phases/12-multimodal-ai/05-llava-visual-instruction-tuning) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/12-multimodal-ai/05-llava-visual-instruction-tuning/docs/en.md) — LLaVA (April 2023) is the most copied multimodal architecture on the planet. It replaced BLIP-2's Q-Former with a 2-layer MLP, replaced Flamingo's gated cross-attention with nai…
- [Any-Resolution Vision — Patch-n'-Pack and NaFlex](https://aiengineeringfromscratch.com/lesson.html?path=phases/12-multimodal-ai/06-any-resolution-patch-n-pack) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/12-multimodal-ai/06-any-resolution-patch-n-pack/docs/en.md) — Real images are not 224x224 squares. A receipt is 9:16, a chart is 16:9, a medical scan might be 4096x4096, a mobile screenshot is 9:19.5. The pre-2024 VLM answer — resize every…
- [Open-Weight VLM Recipes: What Actually Matters](https://aiengineeringfromscratch.com/lesson.html?path=phases/12-multimodal-ai/07-open-weight-vlm-recipes) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/12-multimodal-ai/07-open-weight-vlm-recipes/docs/en.md) — The 2024-2026 open-weight VLM literature is a forest of ablation tables. Apple's MM1 tested 13 combinations of image encoder, connector, and data mix. Allen AI's Molmo proved de…
- [LLaVA-OneVision: Single, Multi, Video](https://aiengineeringfromscratch.com/lesson.html?path=phases/12-multimodal-ai/08-llava-onevision-single-multi-video) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/12-multimodal-ai/08-llava-onevision-single-multi-video/docs/en.md) — Before LLaVA-OneVision (Li et al., August 2024) the open-VLM world had separate lineages: LLaVA-1.5 for single images, multi-image models like Mantis and VILA, video models like…
- [Qwen-VL Family and Dynamic-FPS Video](https://aiengineeringfromscratch.com/lesson.html?path=phases/12-multimodal-ai/09-qwen-vl-family-dynamic-fps) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/12-multimodal-ai/09-qwen-vl-family-dynamic-fps/docs/en.md) — The Qwen-VL family — Qwen-VL (2023), Qwen2-VL (2024), Qwen2.5-VL (2025), Qwen3-VL (2025) — is the most influential open vision-language model lineage in 2026. Each generation ma…
- [InternVL3 Native Multimodal Pretraining](https://aiengineeringfromscratch.com/lesson.html?path=phases/12-multimodal-ai/10-internvl3-native-multimodal) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/12-multimodal-ai/10-internvl3-native-multimodal/docs/en.md) — Every open VLM before InternVL3 followed the same three-step recipe: take a text LLM trained on trillions of text tokens, bolt on a vision encoder, then fine-tune the seams. Thi…
- [Chameleon Early-Fusion Token-Only](https://aiengineeringfromscratch.com/lesson.html?path=phases/12-multimodal-ai/11-chameleon-early-fusion-tokens) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/12-multimodal-ai/11-chameleon-early-fusion-tokens/docs/en.md) — Every VLM we have seen so far keeps images and text separate. Visual tokens come from a vision encoder, flow into a projector, then meet text inside the LLM. The vision and text…
- [Emu3 Next-Token Prediction for Generation](https://aiengineeringfromscratch.com/lesson.html?path=phases/12-multimodal-ai/12-emu3-next-token-for-generation) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/12-multimodal-ai/12-emu3-next-token-for-generation/docs/en.md) — BAAI's Emu3 (Wang et al., September 2024) is the 2024 result that should have ended the diffusion-versus-autoregressive debate. A single Llama-style decoder-only transformer, tr…
- [Transfusion Autoregressive + Diffusion](https://aiengineeringfromscratch.com/lesson.html?path=phases/12-multimodal-ai/13-transfusion-autoregressive-diffusion) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/12-multimodal-ai/13-transfusion-autoregressive-diffusion/docs/en.md) — Chameleon and Emu3 bet everything on discrete tokens. They work, but the quantization bottleneck is visible — the image quality plateaus below continuous-space diffusion models.…
- [Show-o Discrete-Diffusion Unified](https://aiengineeringfromscratch.com/lesson.html?path=phases/12-multimodal-ai/14-show-o-discrete-diffusion-unified) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/12-multimodal-ai/14-show-o-discrete-diffusion-unified/docs/en.md) — Transfusion mixes continuous and discrete representations. Show-o (Xie et al., August 2024) goes the other way: text tokens use causal next-token prediction, image tokens use ma…
- [Janus-Pro Decoupled Encoders](https://aiengineeringfromscratch.com/lesson.html?path=phases/12-multimodal-ai/15-janus-pro-decoupled-encoders) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/12-multimodal-ai/15-janus-pro-decoupled-encoders/docs/en.md) — Unified multimodal models have an unavoidable tension. Understanding wants semantic features — SigLIP or DINOv2 output vectors rich with concept-level information. Generation wa…
- [MIO Any-to-Any Streaming](https://aiengineeringfromscratch.com/lesson.html?path=phases/12-multimodal-ai/16-mio-any-to-any-streaming) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/12-multimodal-ai/16-mio-any-to-any-streaming/docs/en.md) — GPT-4o ships a product most open models cannot replicate: an agent that hears voice, sees video, and speaks back in real time. The open-ecosystem answer by late 2024 was MIO (Wa…
- [Video-Language Temporal Grounding](https://aiengineeringfromscratch.com/lesson.html?path=phases/12-multimodal-ai/17-video-language-temporal-grounding) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/12-multimodal-ai/17-video-language-temporal-grounding/docs/en.md) — Video is not a stack of photos. A 5-second clip has causal ordering, action verbs, and event timing that an image model cannot represent. Video-LLaMA (Zhang et al., June 2023) s…
- [Long-Video at Million-Token Context](https://aiengineeringfromscratch.com/lesson.html?path=phases/12-multimodal-ai/18-long-video-million-token) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/12-multimodal-ai/18-long-video-million-token/docs/en.md) — A 1-hour 4K video at 24 FPS, patched and embedded, produces on the order of 60 million tokens. A 2-hour podcast episode transcribed is 30,000 tokens. A full Blu-ray feature film…
- [Audio-Language Models: Whisper to AF3](https://aiengineeringfromscratch.com/lesson.html?path=phases/12-multimodal-ai/19-audio-language-whisper-to-af3) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/12-multimodal-ai/19-audio-language-whisper-to-af3/docs/en.md) — Whisper (Radford et al., December 2022) settled speech recognition — 680k hours of weakly-supervised multilingual speech, a simple encoder-decoder transformer, a benchmark that …
- [Omni Models: Thinker-Talker Streaming](https://aiengineeringfromscratch.com/lesson.html?path=phases/12-multimodal-ai/20-omni-models-thinker-talker) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/12-multimodal-ai/20-omni-models-thinker-talker/docs/en.md) — GPT-4o's product demo in May 2024 was disruptive not because of the underlying model but because of the product shape — a voice interface where you talk, the model sees what the…
- [Embodied VLAs: RT-2, OpenVLA, π0, GR00T](https://aiengineeringfromscratch.com/lesson.html?path=phases/12-multimodal-ai/21-embodied-vlas-openvla-pi0-groot) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/12-multimodal-ai/21-embodied-vlas-openvla-pi0-groot/docs/en.md) — The first time a model read a recipe off a website and executed it in a kitchen robot was RT-2 (Google DeepMind, July 2023). RT-2 discretized actions as text tokens, co-fine-tun…
- [Document and Diagram Understanding](https://aiengineeringfromscratch.com/lesson.html?path=phases/12-multimodal-ai/22-document-diagram-understanding) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/12-multimodal-ai/22-document-diagram-understanding/docs/en.md) — Documents are not photos. A PDF, scientific paper, invoice, or handwritten form has layout, tables, diagrams, footnotes, headers, and semantic structure that plain image underst…
- [ColPali Vision-Native Document RAG](https://aiengineeringfromscratch.com/lesson.html?path=phases/12-multimodal-ai/23-colpali-vision-native-rag) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/12-multimodal-ai/23-colpali-vision-native-rag/docs/en.md) — Traditional RAG parses PDFs into text, splits into chunks, embeds chunks, stores vectors. Every step loses signal: OCR drops chart data, chunking breaks table rows, text embeddi…
- [Multimodal RAG and Cross-Modal Retrieval](https://aiengineeringfromscratch.com/lesson.html?path=phases/12-multimodal-ai/24-multimodal-rag-cross-modal) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/12-multimodal-ai/24-multimodal-rag-cross-modal/docs/en.md) — Vision-native document RAG is one slice. Production multimodal RAG goes wider — retrieving across text, images, audio, and video for workflows like trip planning ("find me a qui…
- [Multimodal Agents and Computer-Use (Capstone)](https://aiengineeringfromscratch.com/lesson.html?path=phases/12-multimodal-ai/25-multimodal-agents-computer-use) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/12-multimodal-ai/25-multimodal-agents-computer-use/docs/en.md) — The 2026 frontier product is a multimodal agent that reads screenshots, clicks buttons, navigates web UIs, fills forms, and completes workflows end-to-end. SeeClick and CogAgent…

## Phase 13: Tools & Protocols
The interfaces between AI and the real world.

- [The Tool Interface](https://aiengineeringfromscratch.com/lesson.html?path=phases/13-tools-and-protocols/01-the-tool-interface) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/13-tools-and-protocols/01-the-tool-interface/docs/en.md) — A language model produces tokens. A program takes actions. The gap between those two is the tool interface: a contract that lets the model request an action and the host execute…
- [Function Calling Deep Dive](https://aiengineeringfromscratch.com/lesson.html?path=phases/13-tools-and-protocols/02-function-calling-deep-dive) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/13-tools-and-protocols/02-function-calling-deep-dive/docs/en.md) — The three frontier providers converged on the same tool-call loop in 2024 and then diverged on everything else. OpenAI uses `tools` and `tool_calls`. Anthropic uses `tool_use` a…
- [Parallel and Streaming Tool Calls](https://aiengineeringfromscratch.com/lesson.html?path=phases/13-tools-and-protocols/03-parallel-and-streaming-tool-calls) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/13-tools-and-protocols/03-parallel-and-streaming-tool-calls/docs/en.md) — Three independent weather lookups serialized is three round trips. Run them in parallel and total time collapses to the slowest single call. Every frontier provider now emits mu…
- [Structured Output](https://aiengineeringfromscratch.com/lesson.html?path=phases/13-tools-and-protocols/04-structured-output) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/13-tools-and-protocols/04-structured-output/docs/en.md) — "Ask the model nicely to return JSON" fails 5 to 15 percent of the time, even on frontier models. Structured outputs close that gap with constrained decoding: the model is liter…
- [Tool Schema Design](https://aiengineeringfromscratch.com/lesson.html?path=phases/13-tools-and-protocols/05-tool-schema-design) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/13-tools-and-protocols/05-tool-schema-design/docs/en.md) — A correct tool fails silently when the model cannot tell when to use it. Naming, descriptions, and parameter shapes drive 10 to 20 percentage-point swings in tool-selection accu…
- [MCP Fundamentals: Stateless Requests and JSON-RPC](https://aiengineeringfromscratch.com/lesson.html?path=phases/13-tools-and-protocols/06-mcp-fundamentals) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/13-tools-and-protocols/06-mcp-fundamentals/docs/en.md) — Modern MCP has no handshake and no protocol session. Each request must carry enough metadata to be understood, authorized, routed, and retried on its own.
- [Building an MCP Server: Stateless Python and TypeScript](https://aiengineeringfromscratch.com/lesson.html?path=phases/13-tools-and-protocols/07-building-an-mcp-server) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/13-tools-and-protocols/07-building-an-mcp-server/docs/en.md) — A modern MCP server does not remember a handshake. It validates the metadata on every request, runs one handler, and returns one typed result.
- [Building an MCP Client: Discovery, Routing, and Dual-Era Fallback](https://aiengineeringfromscratch.com/lesson.html?path=phases/13-tools-and-protocols/08-building-an-mcp-client) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/13-tools-and-protocols/08-building-an-mcp-client/docs/en.md) — A modern MCP client repeats its contract on every request. Its hardest compatibility decision is knowing when an old server is truly old and when a modern server is reporting a …
- [MCP Transports: stdio and Stateless Streamable HTTP](https://aiengineeringfromscratch.com/lesson.html?path=phases/13-tools-and-protocols/09-mcp-transports) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/13-tools-and-protocols/09-mcp-transports/docs/en.md) — Transport carries MCP messages. It does not supply missing protocol state. In `2026-07-28`, local stdio and remote Streamable HTTP both carry self-describing requests.
- [MCP Resources and Prompts: Addressable Context for Stateless Servers](https://aiengineeringfromscratch.com/lesson.html?path=phases/13-tools-and-protocols/10-mcp-resources-and-prompts) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/13-tools-and-protocols/10-mcp-resources-and-prompts/docs/en.md) — Tools perform operations. Resources expose addressable content. Prompts package user-selected message templates. A good MCP server keeps those contracts separate and predictable.
- [MCP Model Input: Sampling Migration and Stateless MRTR](https://aiengineeringfromscratch.com/lesson.html?path=phases/13-tools-and-protocols/11-mcp-sampling) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/13-tools-and-protocols/11-mcp-sampling/docs/en.md) — MCP 2026-07-28 deprecates Sampling for new designs and removes the server-to-client request channel. If an existing workflow still needs the client's model, the server returns a…
- [Explicit Scope and Stateless Elicitation](https://aiengineeringfromscratch.com/lesson.html?path=phases/13-tools-and-protocols/12-mcp-roots-and-elicitation) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/13-tools-and-protocols/12-mcp-roots-and-elicitation/docs/en.md) — Roots are deprecated in MCP 2026-07-28 and were never a security sandbox. Put scope in visible tool arguments or resource URIs, authorize it on the server, and use MRTR when a t…
- [MCP Tasks Extension: Durable Work on a Stateless Core](https://aiengineeringfromscratch.com/lesson.html?path=phases/13-tools-and-protocols/13-mcp-async-tasks) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/13-tools-and-protocols/13-mcp-async-tasks/docs/en.md) — Stateless MCP does not mean every operation must finish in one request. The official Tasks extension gives long-running work an explicit durable handle. A server can return that…
- [MCP Apps on the Stateless Protocol](https://aiengineeringfromscratch.com/lesson.html?path=phases/13-tools-and-protocols/14-mcp-apps) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/13-tools-and-protocols/14-mcp-apps/docs/en.md) — An interactive result is still an MCP tool and resource exchange. The 2026-07-28 core makes that exchange self-contained, while the Apps extension adds the sandboxed browser sur…
- [MCP Security: Poisoned Metadata, Routing, and MRTR State](https://aiengineeringfromscratch.com/lesson.html?path=phases/13-tools-and-protocols/15-mcp-security-tool-poisoning) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/13-tools-and-protocols/15-mcp-security-tool-poisoning/docs/en.md) — Stateless does not mean trustless. It means every request exposes the evidence a server and gateway need to validate the call independently.
- [MCP Authorization: CIMD, Issuer Binding, PKCE, and Step-Up](https://aiengineeringfromscratch.com/lesson.html?path=phases/13-tools-and-protocols/16-mcp-security-oauth-2-1) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/13-tools-and-protocols/16-mcp-security-oauth-2-1/docs/en.md) — A remote MCP request is stateless, but its authorization is not anonymous. Bind every credential to the issuer that created it and every token to the resource that receives it.
- [Stateless MCP Gateways and Registry Admission](https://aiengineeringfromscratch.com/lesson.html?path=phases/13-tools-and-protocols/17-mcp-gateways-and-registries) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/13-tools-and-protocols/17-mcp-gateways-and-registries/docs/en.md) — A gateway should make every route explicit. The 2026-07-28 protocol gives it method, name, version, capability, identity, cache, and trace boundaries without a transport session.
- [MCP Auth in Production: Issuer-Bound Enrollment and Tokens](https://aiengineeringfromscratch.com/lesson.html?path=phases/13-tools-and-protocols/18-mcp-auth-production) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/13-tools-and-protocols/18-mcp-auth-production/docs/en.md) — Lesson 16 built the OAuth 2.1 state machine. This lesson hardens its production boundaries for MCP 2026-07-28: Client ID Metadata Documents first, deprecated dynamic registratio…
- [A2A Protocol](https://aiengineeringfromscratch.com/lesson.html?path=phases/13-tools-and-protocols/19-a2a-protocol) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/13-tools-and-protocols/19-a2a-protocol/docs/en.md) — MCP is agent-to-tool. A2A (Agent2Agent) is agent-to-agent — an open protocol for letting opaque agents built on different frameworks collaborate. Released by Google in April 202…
- [OpenTelemetry GenAI](https://aiengineeringfromscratch.com/lesson.html?path=phases/13-tools-and-protocols/20-opentelemetry-genai) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/13-tools-and-protocols/20-opentelemetry-genai/docs/en.md) — An agent calls five tools, three MCP servers, and two sub-agents. You need one trace across all of it. The OpenTelemetry GenAI semantic conventions (stable attributes in v1.37 a…
- [LLM Routing Layer](https://aiengineeringfromscratch.com/lesson.html?path=phases/13-tools-and-protocols/21-llm-routing-layer) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/13-tools-and-protocols/21-llm-routing-layer/docs/en.md) — Provider lock-in is expensive. Different tool-calling workloads suit different models. Routing gateways give one API surface, retries, failover, cost tracking, and guardrails. T…
- [Agent Skills: Portable Contract and Runtime Boundary](https://aiengineeringfromscratch.com/lesson.html?path=phases/13-tools-and-protocols/22-skills-and-agent-sdks) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/13-tools-and-protocols/22-skills-and-agent-sdks/docs/en.md) — A skill is not a long prompt with a better filename. It is a discoverable package of instructions, resources, and executable helpers that enters an agent's context through a run…
- [Capstone: Stateless Tool Ecosystem](https://aiengineeringfromscratch.com/lesson.html?path=phases/13-tools-and-protocols/23-capstone-tool-ecosystem) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/13-tools-and-protocols/23-capstone-tool-ecosystem/docs/en.md) — A production agent system is a set of boundaries, not a pile of features. This capstone separates a readable in-process simulation from the protocol clients, authorization serve…
- [Skill Discovery and Progressive Disclosure](https://aiengineeringfromscratch.com/lesson.html?path=phases/13-tools-and-protocols/24-skill-discovery-and-progressive-disclosure) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/13-tools-and-protocols/24-skill-discovery-and-progressive-disclosure/docs/en.md) — A skill becomes useful before its body is loaded. Its name and description earn a place in the catalog; its deeper files earn context only when the task reaches them.
- [Skill Invocation and Routing](https://aiengineeringfromscratch.com/lesson.html?path=phases/13-tools-and-protocols/25-skill-invocation-and-routing) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/13-tools-and-protocols/25-skill-invocation-and-routing/docs/en.md) — Invocation is an authority decision followed by a relevance decision. A good description helps the model choose; a good policy decides whether that choice is allowed.
- [Skill Permissions, Sandboxes, and Trust](https://aiengineeringfromscratch.com/lesson.html?path=phases/13-tools-and-protocols/26-skill-permissions-sandboxes-and-trust) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/13-tools-and-protocols/26-skill-permissions-sandboxes-and-trust/docs/en.md) — A skill can suggest an action. Only the host can authorize it, only an isolation boundary can contain it, and only verification can tell you whether it worked.
- [Skill Evals, Packaging, and Portability](https://aiengineeringfromscratch.com/lesson.html?path=phases/13-tools-and-protocols/27-skill-evals-packaging-and-portability) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/13-tools-and-protocols/27-skill-evals-packaging-and-portability/docs/en.md) — A skill is finished when its package survives linting, routes on the right requests, improves a measured task, stays inside policy, and degrades honestly on another host.
- [MCP Tool Contracts and Content](https://aiengineeringfromscratch.com/lesson.html?path=phases/13-tools-and-protocols/28-mcp-tool-contracts-and-content) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/13-tools-and-protocols/28-mcp-tool-contracts-and-content/docs/en.md) — A tool is safe to automate only when discovery, arguments, results, pagination, and transport metadata agree on one contract.
- [MCP Reliability, Cancellation, and Flow Control](https://aiengineeringfromscratch.com/lesson.html?path=phases/13-tools-and-protocols/29-mcp-reliability-cancellation-and-flow-control) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/13-tools-and-protocols/29-mcp-reliability-cancellation-and-flow-control/docs/en.md) — A request ID correlates a message. It does not make a side effect safe, stop a worker, or protect a stream from a slow consumer.
- [MCP Registry Supply Chain: Admission, Drift, and Rollback](https://aiengineeringfromscratch.com/lesson.html?path=phases/13-tools-and-protocols/30-mcp-registry-supply-chain-and-drift) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/13-tools-and-protocols/30-mcp-registry-supply-chain-and-drift/docs/en.md) — A registry entry tells you what a publisher declared. Production admission proves what you fetched, what you observed, what you approved, and what you can safely restore.
- [MCP Conformance Engineering: Versioning, Evidence, and Operations](https://aiengineeringfromscratch.com/lesson.html?path=phases/13-tools-and-protocols/31-mcp-conformance-versioning-and-operations) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/13-tools-and-protocols/31-mcp-conformance-versioning-and-operations/docs/en.md) — A server is not conformant because its happy path worked through one SDK. Conformance lives at the wire, at version boundaries, through intermediaries, and during rollback.

## Phase 14: Agent Engineering
Build agents from first principles — loop, memory, planning, frameworks, benchmarks, production, workbench.

- [The Agent Loop](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/01-the-agent-loop) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/01-the-agent-loop/docs/en.md) — Every agent in 2026 is a variant of the ReAct loop from 2022 — Claude Code, Cursor, Devin, Operator included. Reasoning tokens interleave with tool calls and observations until …
- [ReWOO and Plan-and-Execute](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/02-rewoo-plan-and-execute) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/02-rewoo-plan-and-execute/docs/en.md) — ReAct interleaves thought and action in one stream. ReWOO separates them: one big plan up front, then execute. 5x fewer tokens, +4% accuracy on HotpotQA, and you can distill the…
- [Reflexion and Verbal Reinforcement Learning](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/03-reflexion-verbal-rl) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/03-reflexion-verbal-rl/docs/en.md) — Gradient-based RL needs thousands of trials and a GPU cluster to fix a failure mode. Reflexion (Shinn et al., NeurIPS 2023) does it in natural language: after each failed trial,…
- [Tree of Thoughts and LATS](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/04-tree-of-thoughts-lats) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/04-tree-of-thoughts-lats/docs/en.md) — A single chain-of-thought trajectory has no room to backtrack. ToT (Yao et al., 2023) turns reasoning into a tree with self-evaluation on each node. LATS (Zhou et al., 2024) uni…
- [Self-Refine and CRITIC](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/05-self-refine-and-critic) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/05-self-refine-and-critic/docs/en.md) — Self-Refine (Madaan et al., 2023) uses one LLM in three roles — generate, feedback, refine — in a loop. Average gain: +20 absolute on 7 tasks. CRITIC (Gou et al., 2023) hardens …
- [Tool Use and Function Calling](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/06-tool-use-and-function-calling) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/06-tool-use-and-function-calling/docs/en.md) — Toolformer (Schick et al., 2023) started self-supervised tool annotation. Berkeley Function Calling Leaderboard V4 (Patil et al., 2025) sets the 2026 bar: 40% agentic, 30% multi…
- [Agent Memory — Virtual Context and Memory Paging](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/07-memory-virtual-context-memgpt) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/07-memory-virtual-context-memgpt/docs/en.md) — Context windows are finite. Conversations, documents, and tool traces are not. The fix is OS virtual memory restated — main context is RAM, external store is disk, the agent pag…
- [Memory Blocks and Sleep-Time Compute](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/08-memory-blocks-sleep-time-compute) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/08-memory-blocks-sleep-time-compute/docs/en.md) — Discrete functional memory blocks the model can edit directly, and a sleep-time agent that consolidates memory asynchronously while the primary agent is idle. These two ideas ar…
- [Hybrid Memory — Vector + Graph + KV](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/09-hybrid-memory-mem0) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/09-hybrid-memory-mem0/docs/en.md) — Hybrid memory runs three stores in parallel — vector for semantic similarity, KV for fast fact lookup, graph for entity-relationship reasoning — with a scoring layer that fuses …
- [Skill Libraries and Lifelong Learning (Voyager)](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/10-skill-libraries-voyager) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/10-skill-libraries-voyager/docs/en.md) — Voyager (Wang et al., TMLR 2024) treats executable code as a skill. Skills are named, retrievable, composable, and refined by environment feedback. This is the reference archite…
- [Planning with HTN and Evolutionary Search](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/11-planning-htn-and-evolutionary) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/11-planning-htn-and-evolutionary/docs/en.md) — Symbolic planning handles the cases where the plan is provably correct. Evolutionary code search handles the cases where the fitness function is machine-checkable. ChatHTN (2025…
- [Anthropic's Workflow Patterns](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/12-anthropic-workflow-patterns) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/12-anthropic-workflow-patterns/docs/en.md) — Schluntz and Zhang (Anthropic, Dec 2024) distinguish workflows (predefined paths) from agents (dynamic tool-use). Five workflow patterns cover most cases. Start with direct API …
- [Stateful Graph Orchestration — Durable Execution and Checkpoints](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/13-langgraph-stateful-graphs) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/13-langgraph-stateful-graphs/docs/en.md) — Agent is a state machine; nodes are functions; edges are transitions; state is checkpointed after each node. Resume from any failure at the last successful checkpoint. LangGraph…
- [The Actor Model for Agents](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/14-autogen-actor-model) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/14-autogen-actor-model/docs/en.md) — Agents as actors: async message exchange, event-driven handlers, fault isolation, natural concurrency. AutoGen v0.4 (Microsoft Research, Jan 2025) redesigned agent orchestration…
- [Role-Based Agent Teams — Roles, Tasks, Processes](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/15-crewai-role-based-crews) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/15-crewai-role-based-crews/docs/en.md) — Four primitives: Agent, Task, Crew, Process. Two top-level shapes: Crews (autonomous, role-based collaboration) and Flows (event-driven, deterministic). CrewAI is the 2026 refer…
- [OpenAI Agents SDK — Handoffs, Guardrails, Tracing](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/16-openai-agents-sdk) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/16-openai-agents-sdk/docs/en.md) — OpenAI Agents SDK is the lightweight multi-agent framework built on the Responses API. Five primitives: Agent, Handoff, Guardrail, Session, Tracing. Handoffs are tools named `tr…
- [The Harness as a Library — Subagents and Session Store](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/17-claude-agent-sdk) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/17-claude-agent-sdk/docs/en.md) — A harness you can import: built-in tools, subagents for context isolation, hooks, W3C trace propagation, session persistence. The Claude Agent SDK is the reference example — the…
- [Production Agent Runtimes](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/18-agno-and-mastra-runtimes) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/18-agno-and-mastra-runtimes/docs/en.md) — A production agent runtime optimizes what prototyping frameworks ignore: instantiation cost, typed workflow surfaces, and a serving-ready backend. The 2026 pairing: Agno (Python…
- [Benchmarks — SWE-bench, GAIA, AgentBench](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/19-benchmarks-swebench-gaia) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/19-benchmarks-swebench-gaia/docs/en.md) — Three benchmarks anchor agent evaluation in 2026. SWE-bench tests code patching. GAIA tests generalist tool use. AgentBench tests multi-environment reasoning. Know their composi…
- [Benchmarks — WebArena and OSWorld](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/20-benchmarks-webarena-osworld) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/20-benchmarks-webarena-osworld/docs/en.md) — WebArena tests web-agent capability across four self-hosted apps. OSWorld tests desktop-agent capability across Ubuntu, Windows, macOS. At release (2023–2024) both showed a big …
- [Computer Use — Claude, OpenAI CUA, Gemini](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/21-computer-use-agents) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/21-computer-use-agents/docs/en.md) — Three production computer-use models in 2026. All three are vision-based. All three treat screenshots, DOM text, and tool outputs as untrusted input. Only direct user instructio…
- [Voice Agents — Pipecat and LiveKit](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/22-voice-agents-pipecat-livekit) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/22-voice-agents-pipecat-livekit/docs/en.md) — Voice agents are a first-class production category in 2026. Pipecat gives you a Python frame-based pipeline (VAD → STT → LLM → TTS → transport). LiveKit Agents bridges AI models…
- [OpenTelemetry GenAI Semantic Conventions](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/23-otel-genai-conventions) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/23-otel-genai-conventions/docs/en.md) — OpenTelemetry's GenAI SIG (launched April 2024) defines the standard schema for agent telemetry. Span names, attributes, and content-capture rules converge across vendors so age…
- [Agent Observability — Langfuse, Phoenix, Opik](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/24-agent-observability-platforms) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/24-agent-observability-platforms/docs/en.md) — Three open-source agent observability platforms dominate 2026. Langfuse (MIT) — 6M+ installs/month, tracing + prompt management + evals + session replay. Arize Phoenix (Elastic …
- [Multi-Agent Debate and Collaboration](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/25-multi-agent-debate) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/25-multi-agent-debate/docs/en.md) — Du et al. (ICML 2024, "Society of Minds") run N model instances that independently propose answers, then iteratively critique each other over R rounds to converge. Improves fact…
- [Failure Modes — Why Agents Break](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/26-failure-modes-agentic) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/26-failure-modes-agentic/docs/en.md) — MASFT (Berkeley, 2025) catalogs 14 multi-agent failure modes in 3 categories. Microsoft's Taxonomy documents how existing AI failures amplify in agentic settings. Industry field…
- [Prompt Injection and the PVE Defense](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/27-prompt-injection-defense) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/27-prompt-injection-defense/docs/en.md) — Greshake et al. (AISec 2023) established indirect prompt injection as the defining agent security problem. Attacker plants instructions in data the agent retrieves; on ingest, t…
- [Orchestration Patterns — Supervisor, Swarm, Hierarchical](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/28-orchestration-patterns) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/28-orchestration-patterns/docs/en.md) — Four orchestration patterns recur across 2026 frameworks: supervisor-worker, swarm / peer-to-peer, hierarchical, debate. Anthropic's guidance: "It's about building the right sys…
- [Production Runtimes — Queue, Event, Cron](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/29-production-runtimes) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/29-production-runtimes/docs/en.md) — Production agents run on six runtime shapes: request-response, streaming, durable execution, queue-based background, event-driven, and scheduled. Pick the shape before you pick …
- [Eval-Driven Agent Development](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/30-eval-driven-agent-development) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/30-eval-driven-agent-development/docs/en.md) — Anthropic's guidance: "start with simple prompts, optimize them with comprehensive evaluation, and add multi-step agentic systems only when needed." Evaluation is not the last s…
- [Agent Workbench: Why Capable Models Still Fail](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/31-agent-workbench-why-models-fail) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/31-agent-workbench-why-models-fail/docs/en.md) — A capable model is not enough. Reliable agents need a workbench: instructions, state, scope, feedback, verification, review, and handoff. Strip those away and even a frontier mo…
- [The Minimal Agent Workbench](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/32-minimal-agent-workbench) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/32-minimal-agent-workbench/docs/en.md) — The smallest useful workbench is three files: a root instructions router, a state file, and a task board. Everything else is layered on top. If a repo cannot carry these three, …
- [Agent Instructions as Executable Constraints](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/33-instructions-as-executable-constraints) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/33-instructions-as-executable-constraints/docs/en.md) — Instructions written as prose are wishes. Instructions written as constraints are tests. The workbench turns each rule into something an agent can check at runtime and a reviewe…
- [Repo Memory and Durable State](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/34-repo-memory-and-state) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/34-repo-memory-and-state/docs/en.md) — Chat history is volatile. The repo is durable. The workbench stores agent state in versioned files so the next session, the next agent, and the next reviewer all read from the s…
- [Initialization Scripts for Agents](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/35-initialization-scripts) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/35-initialization-scripts/docs/en.md) — Every session that starts cold pays a tax. The agent reads the same files, retries the same probes, and rediscovers the same paths. An init script pays the tax once and writes t…
- [Scope Contracts and Task Boundaries](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/36-scope-contracts) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/36-scope-contracts/docs/en.md) — The model does not know where the work ends. A scope contract is a per-task file that says where the work begins, where it ends, and how to roll back if it spills. The contract …
- [Runtime Feedback Loops](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/37-runtime-feedback-loops) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/37-runtime-feedback-loops/docs/en.md) — Agents that do not see real command output guess. A feedback runner captures stdout, stderr, exit code, and timing into a structured record the next turn can read. Then the agen…
- [Verification Gates](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/38-verification-gates) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/38-verification-gates/docs/en.md) — The agent does not get to mark its own work as done. A verification gate reads the scope contract, the feedback log, the rule report, and the diff, and answers a single question…
- [Reviewer Agent: Separate Builder from Marker](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/39-reviewer-agent) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/39-reviewer-agent/docs/en.md) — The agent that wrote the code cannot grade it. A reviewer is a second loop with a different system prompt, a different goal, and read-only access to everything the builder produ…
- [Multi-Session Handoff](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/40-multi-session-handoff) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/40-multi-session-handoff/docs/en.md) — The session is going to end. The work is not. The handoff packet is the artifact that turns "the agent worked for an hour" into "the next session is productive in the first minu…
- [The Workbench on a Real Repo](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/41-workbench-for-real-repos) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/41-workbench-for-real-repos/docs/en.md) — Eleven lessons of surfaces are worth nothing if they do not survive contact with a real codebase. This lesson runs the same task twice on a small sample app: prompt-only versus …
- [Capstone: Ship a Reusable Agent Workbench Pack](https://aiengineeringfromscratch.com/lesson.html?path=phases/14-agent-engineering/42-agent-workbench-capstone) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/14-agent-engineering/42-agent-workbench-capstone/docs/en.md) — The mini-track ends with a pack you drop into any repo. Eleven lessons of surfaces compressed into a directory you can `cp -r` and have an agent working reliably the next mornin…

## Phase 15: Autonomous Systems
Long-horizon agents, self-improvement, and the 2026 safety stack.

- [From Chatbots to Long-Horizon Agents (METR)](https://aiengineeringfromscratch.com/lesson.html?path=phases/15-autonomous-systems/01-long-horizon-agents) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/15-autonomous-systems/01-long-horizon-agents/docs/en.md) — In 2023 a chatbot answered a question in one turn. In 2026 a frontier model routinely runs minutes to hours on a single task. METR's Time Horizon 1.1 benchmark (January 2026) pu…
- [STaR, V-STaR, Quiet-STaR: Self-Taught Reasoning](https://aiengineeringfromscratch.com/lesson.html?path=phases/15-autonomous-systems/02-star-family-reasoning) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/15-autonomous-systems/02-star-family-reasoning/docs/en.md) — The smallest possible self-improvement loop sits inside the rationale. A model generates a chain of thought, keeps the ones that land on correct answers, and fine-tunes on those…
- [AlphaEvolve: Evolutionary Coding Agents](https://aiengineeringfromscratch.com/lesson.html?path=phases/15-autonomous-systems/03-alphaevolve-evolutionary-coding) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/15-autonomous-systems/03-alphaevolve-evolutionary-coding/docs/en.md) — Pair a frontier coding model with an evolutionary loop and a machine-checkable evaluator. Let the loop run long enough. It discovers a 4x4 complex-matrix multiplication procedur…
- [Darwin Gödel Machine: Self-Modifying Agents](https://aiengineeringfromscratch.com/lesson.html?path=phases/15-autonomous-systems/04-darwin-godel-machine) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/15-autonomous-systems/04-darwin-godel-machine/docs/en.md) — Schmidhuber's 2003 Godel Machine required a formal proof that any self-modification was beneficial before accepting it. That proof is impossible in practice. Darwin Godel Machin…
- [AI Scientist v2: Workshop-Level Research](https://aiengineeringfromscratch.com/lesson.html?path=phases/15-autonomous-systems/05-ai-scientist-v2) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/15-autonomous-systems/05-ai-scientist-v2/docs/en.md) — Sakana's AI Scientist v2 (Yamada et al., arXiv:2504.08066) runs the full research loop: hypothesis, code, experiments, figures, writeup, submission. It is the first system to ha…
- [Automated Alignment Research (Anthropic AAR)](https://aiengineeringfromscratch.com/lesson.html?path=phases/15-autonomous-systems/06-automated-alignment-research) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/15-autonomous-systems/06-automated-alignment-research/docs/en.md) — Anthropic ran parallel teams of Claude Opus 4.6 Autonomous Alignment Researchers in independent sandboxes, coordinating via a shared forum whose logs live outside any sandbox (s…
- [Recursive Self-Improvement: Capability vs Alignment](https://aiengineeringfromscratch.com/lesson.html?path=phases/15-autonomous-systems/07-recursive-self-improvement) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/15-autonomous-systems/07-recursive-self-improvement/docs/en.md) — Recursive self-improvement (RSI) is no longer speculation. The ICLR 2026 RSI Workshop in Rio (April 23-27) framed it as an engineering problem with concrete tooling. Demis Hassa…
- [Bounded Self-Improvement Designs](https://aiengineeringfromscratch.com/lesson.html?path=phases/15-autonomous-systems/08-bounded-self-improvement) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/15-autonomous-systems/08-bounded-self-improvement/docs/en.md) — Research has converged on four primitives for bounding a self-improvement loop. Formal invariants that must hold across every edit. Alignment anchors that cannot be modified. Mu…
- [Autonomous Coding Agent Landscape (SWE-bench, CodeAct)](https://aiengineeringfromscratch.com/lesson.html?path=phases/15-autonomous-systems/09-coding-agent-landscape) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/15-autonomous-systems/09-coding-agent-landscape/docs/en.md) — SWE-bench Verified went from 4% to 80.9% in under three years. Same Claude Sonnet 4.5 scored 43.2% on SWE-agent v1 and 59.8% on Cline autonomous — the scaffolding around the mod…
- [Permission Modes for Autonomous Agents](https://aiengineeringfromscratch.com/lesson.html?path=phases/15-autonomous-systems/10-claude-code-permission-modes) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/15-autonomous-systems/10-claude-code-permission-modes/docs/en.md) — A permission ladder — graduated levels of autonomy from review-every-action to approve-everything — is how a harness governs what an autonomous agent may do without asking. Clau…
- [Browser Agents and Indirect Prompt Injection](https://aiengineeringfromscratch.com/lesson.html?path=phases/15-autonomous-systems/11-browser-agents) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/15-autonomous-systems/11-browser-agents/docs/en.md) — ChatGPT agent (July 2025) merged Operator and deep research into one browser/terminal agent and set BrowseComp SOTA at 68.9%. OpenAI shut Operator down August 31, 2025 — consoli…
- [Durable Execution for Long-Running Agents](https://aiengineeringfromscratch.com/lesson.html?path=phases/15-autonomous-systems/12-durable-execution) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/15-autonomous-systems/12-durable-execution/docs/en.md) — Production long-horizon agents do not run in `while True`. Every LLM call becomes an activity with checkpoint, retry, and replay. Temporal's OpenAI Agents SDK integration went G…
- [Action Budgets, Iteration Caps, Cost Governors](https://aiengineeringfromscratch.com/lesson.html?path=phases/15-autonomous-systems/13-cost-governors) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/15-autonomous-systems/13-cost-governors/docs/en.md) — A mid-sized e-commerce agent's monthly LLM cost jumped from $1,200 to $4,800 after its team enabled the "order-tracking" skill. That is not a pricing bug. That is an agent that …
- [Kill Switches, Circuit Breakers, Canary Tokens](https://aiengineeringfromscratch.com/lesson.html?path=phases/15-autonomous-systems/14-kill-switches-canaries) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/15-autonomous-systems/14-kill-switches-canaries/docs/en.md) — A kill switch is a boolean held outside the agent's edit surface — a Redis key, a feature flag, a signed config — that disables the agent entirely. A circuit breaker is finer-gr…
- [HITL: Propose-Then-Commit](https://aiengineeringfromscratch.com/lesson.html?path=phases/15-autonomous-systems/15-propose-then-commit) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/15-autonomous-systems/15-propose-then-commit/docs/en.md) — The 2026 consensus on HITL is specific. It is not "the agent asks, the user clicks Approve." It is propose-then-commit: the proposed action is persisted to a durable store with …
- [Checkpoints and Rollback](https://aiengineeringfromscratch.com/lesson.html?path=phases/15-autonomous-systems/16-checkpoints-rollback) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/15-autonomous-systems/16-checkpoints-rollback/docs/en.md) — Every graph-state transition persists. When a worker crashes, its lease expires and another worker picks up at the latest checkpoint. Cloudflare Durable Objects hold state acros…
- [Constitutional AI and Rule Overrides](https://aiengineeringfromscratch.com/lesson.html?path=phases/15-autonomous-systems/17-constitutional-ai) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/15-autonomous-systems/17-constitutional-ai/docs/en.md) — Anthropic's January 22, 2026 Claude Constitution runs 79 pages and is CC0. It moves from rule-based to reason-based alignment and establishes a four-tier priority hierarchy: (1)…
- [Llama Guard and Input/Output Classification](https://aiengineeringfromscratch.com/lesson.html?path=phases/15-autonomous-systems/18-llama-guard) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/15-autonomous-systems/18-llama-guard/docs/en.md) — Llama Guard 3 (Meta, Llama-3.1-8B base, fine-tuned for content safety) classifies both LLM inputs and outputs against an MLCommons 13-hazard taxonomy across 8 languages. A 1B-IN…
- [Anthropic Responsible Scaling Policy v3.0](https://aiengineeringfromscratch.com/lesson.html?path=phases/15-autonomous-systems/19-anthropic-rsp) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/15-autonomous-systems/19-anthropic-rsp/docs/en.md) — RSP v3.0 went into effect February 24, 2026, replacing the 2023 policy. Two-tier mitigation: what Anthropic will do unilaterally vs what is framed as an industry-wide recommenda…
- [OpenAI Preparedness Framework and DeepMind FSF](https://aiengineeringfromscratch.com/lesson.html?path=phases/15-autonomous-systems/20-openai-preparedness-deepmind-fsf) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/15-autonomous-systems/20-openai-preparedness-deepmind-fsf/docs/en.md) — OpenAI Preparedness Framework v2 (April 2025) introduces Research Categories — Long-range Autonomy, Sandbagging, Autonomous Replication and Adaptation, Undermining Safeguards — …
- [METR Time Horizons and External Evaluation](https://aiengineeringfromscratch.com/lesson.html?path=phases/15-autonomous-systems/21-metr-external-evaluation) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/15-autonomous-systems/21-metr-external-evaluation/docs/en.md) — METR (ex-ARC Evals) is an independent 501(c)(3) since December 2023. Their Time Horizon 1.1 benchmark (January 2026) fits a logistic curve to task-success probability vs log(exp…
- [CAIS, CAISI, and Societal-Scale Risk](https://aiengineeringfromscratch.com/lesson.html?path=phases/15-autonomous-systems/22-cais-caisi-societal-risk) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/15-autonomous-systems/22-cais-caisi-societal-risk/docs/en.md) — The Center for AI Safety (CAIS, San Francisco, founded 2022 by Hendrycks and Zhang) publishes the four-risk framework — malicious use, AI races, organizational risks, rogue AIs …

## Phase 16: Multi-Agent & Swarms
Coordination, emergence, and collective intelligence.

- [Why Multi-Agent](https://aiengineeringfromscratch.com/lesson.html?path=phases/16-multi-agent-and-swarms/01-why-multi-agent) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/16-multi-agent-and-swarms/01-why-multi-agent/docs/en.md) — One agent hits a wall. The smart move is not a bigger agent - it is more agents.
- [FIPA-ACL Heritage and Speech Acts](https://aiengineeringfromscratch.com/lesson.html?path=phases/16-multi-agent-and-swarms/02-fipa-acl-heritage) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/16-multi-agent-and-swarms/02-fipa-acl-heritage/docs/en.md) — Before MCP, before A2A, there was FIPA-ACL. In 2000 the IEEE Foundation for Intelligent Physical Agents ratified an agent communication language with twenty performatives, two c…
- [Communication Protocols](https://aiengineeringfromscratch.com/lesson.html?path=phases/16-multi-agent-and-swarms/03-communication-protocols) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/16-multi-agent-and-swarms/03-communication-protocols/docs/en.md) — Agents that can't speak the same language aren't a team. They're strangers shouting into the void.
- [The Multi-Agent Primitive Model](https://aiengineeringfromscratch.com/lesson.html?path=phases/16-multi-agent-and-swarms/04-primitive-model) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/16-multi-agent-and-swarms/04-primitive-model/docs/en.md) — Four primitives, nothing more — the agent, the handoff, the shared state, the orchestrator — span a four-dimensional design space, and the major multi-agent frameworks shipping …
- [Supervisor / Orchestrator-Worker Pattern](https://aiengineeringfromscratch.com/lesson.html?path=phases/16-multi-agent-and-swarms/05-supervisor-orchestrator-pattern) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/16-multi-agent-and-swarms/05-supervisor-orchestrator-pattern/docs/en.md) — One lead agent plans and delegates; specialized workers execute in parallel contexts and report back. This is the pattern behind Anthropic's Research system (Claude Opus 4 as le…
- [Hierarchical Architecture and Decomposition Drift](https://aiengineeringfromscratch.com/lesson.html?path=phases/16-multi-agent-and-swarms/06-hierarchical-architecture) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/16-multi-agent-and-swarms/06-hierarchical-architecture/docs/en.md) — Hierarchical is supervisor nested. Manager agents over sub-managers over workers. CrewAI `Process.hierarchical` is the textbook version: a `manager_llm` dynamically delegates ta…
- [Society of Mind and Multi-Agent Debate](https://aiengineeringfromscratch.com/lesson.html?path=phases/16-multi-agent-and-swarms/07-society-of-mind-debate) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/16-multi-agent-and-swarms/07-society-of-mind-debate/docs/en.md) — Minsky's 1986 premise — intelligence is a society of specialists — gets rediscovered every decade. In 2023 Du et al. turned it into a concrete algorithm: multiple LLM instances …
- [Role Specialization — Planner / Critic / Executor / Verifier](https://aiengineeringfromscratch.com/lesson.html?path=phases/16-multi-agent-and-swarms/08-role-specialization) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/16-multi-agent-and-swarms/08-role-specialization/docs/en.md) — The most common multi-agent decomposition in 2026: one agent plans, one executes, one critiques or verifies. MetaGPT (arXiv:2308.00352) formalizes this as SOPs encoded into role…
- [Parallel Swarm and Networked Architectures](https://aiengineeringfromscratch.com/lesson.html?path=phases/16-multi-agent-and-swarms/09-parallel-swarm-networks) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/16-multi-agent-and-swarms/09-parallel-swarm-networks/docs/en.md) — Contrast with supervisor: no central decider. Agents read a shared event bus, pick up work asynchronously, write results back. LangGraph explicitly supports "Swarm Architecture"…
- [Group Chat and Speaker Selection](https://aiengineeringfromscratch.com/lesson.html?path=phases/16-multi-agent-and-swarms/10-group-chat-speaker-selection) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/16-multi-agent-and-swarms/10-group-chat-speaker-selection/docs/en.md) — Shared-conversation orchestration puts N agents in one conversation; a selector function (LLM, round-robin, or custom) picks who speaks next. This is the archetype of emergent m…
- [Handoffs and Routines (Stateless Orchestration)](https://aiengineeringfromscratch.com/lesson.html?path=phases/16-multi-agent-and-swarms/11-handoffs-and-routines) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/16-multi-agent-and-swarms/11-handoffs-and-routines/docs/en.md) — OpenAI's Swarm (October 2024) distilled multi-agent orchestration to two primitives: **routines** (instructions + tools as a system prompt) and **handoffs** (a tool that returns…
- [A2A — The Agent-to-Agent Protocol](https://aiengineeringfromscratch.com/lesson.html?path=phases/16-multi-agent-and-swarms/12-a2a-protocol) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/16-multi-agent-and-swarms/12-a2a-protocol/docs/en.md) — Google announced A2A in April 2025; by April 2026 the spec is at https://a2a-protocol.org/latest/specification/ and 150+ organizations back it. A2A is the horizontal complement …
- [Shared Memory and Blackboard Patterns](https://aiengineeringfromscratch.com/lesson.html?path=phases/16-multi-agent-and-swarms/13-shared-memory-blackboard) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/16-multi-agent-and-swarms/13-shared-memory-blackboard/docs/en.md) — Two approaches coexist in 2026 multi-agent systems: the **message pool** (everyone sees everyone's messages, as in AutoGen GroupChat or MetaGPT) and the **blackboard with subscr…
- [Consensus and Byzantine Fault Tolerance](https://aiengineeringfromscratch.com/lesson.html?path=phases/16-multi-agent-and-swarms/14-consensus-and-bft) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/16-multi-agent-and-swarms/14-consensus-and-bft/docs/en.md) — Classical distributed-systems BFT meets stochastic LLMs. In 2025-2026 three research directions emerged: **CP-WBFT** (arXiv:2511.10400) weighs each vote by a confidence probe; *…
- [Voting, Self-Consistency, and Debate Topology](https://aiengineeringfromscratch.com/lesson.html?path=phases/16-multi-agent-and-swarms/15-voting-debate-topology) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/16-multi-agent-and-swarms/15-voting-debate-topology/docs/en.md) — The cheapest aggregation: sample N independent agents, majority-vote. Wang et al. 2022 self-consistency did this with one model sampled N times. Multi-agent extends it with **he…
- [Negotiation and Bargaining](https://aiengineeringfromscratch.com/lesson.html?path=phases/16-multi-agent-and-swarms/16-negotiation-bargaining) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/16-multi-agent-and-swarms/16-negotiation-bargaining/docs/en.md) — Agents negotiate resources, prices, task allocations, and terms. The 2026 benchmark set is clear: NegotiationArena (arXiv:2402.05863) shows LLMs can improve payoffs ~20% via per…
- [Generative Agents and Emergent Simulation](https://aiengineeringfromscratch.com/lesson.html?path=phases/16-multi-agent-and-swarms/17-generative-agents-simulation) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/16-multi-agent-and-swarms/17-generative-agents-simulation/docs/en.md) — Park et al. 2023 (UIST '23, arXiv:2304.03442) populated **Smallville**, a sandbox of 25 agents, with a three-part architecture: **memory stream** (natural-language log), **refle…
- [Theory of Mind and Emergent Coordination](https://aiengineeringfromscratch.com/lesson.html?path=phases/16-multi-agent-and-swarms/18-theory-of-mind-coordination) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/16-multi-agent-and-swarms/18-theory-of-mind-coordination/docs/en.md) — Li et al. (arXiv:2310.10701) showed that LLM agents in a cooperative text game exhibit **emergent high-order Theory of Mind** (ToM) — reasoning about what another agent believes…
- [Swarm Optimization (PSO, ACO)](https://aiengineeringfromscratch.com/lesson.html?path=phases/16-multi-agent-and-swarms/19-swarm-optimization-pso-aco) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/16-multi-agent-and-swarms/19-swarm-optimization-pso-aco/docs/en.md) — Bio-inspired optimization is making an LLM comeback. **LMPSO** (arXiv:2504.09247) uses PSO where each particle's velocity is a prompt and the LLM generates the next candidate; w…
- [MARL — MADDPG, QMIX, MAPPO](https://aiengineeringfromscratch.com/lesson.html?path=phases/16-multi-agent-and-swarms/20-marl-maddpg-qmix-mappo) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/16-multi-agent-and-swarms/20-marl-maddpg-qmix-mappo/docs/en.md) — The reinforcement-learning heritage of multi-agent coordination, which still informs LLM-agent systems in 2026. **MADDPG** (Lowe et al., NeurIPS 2017, arXiv:1706.02275) introduc…
- [Agent Economies, Token Incentives, Reputation](https://aiengineeringfromscratch.com/lesson.html?path=phases/16-multi-agent-and-swarms/21-agent-economies) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/16-multi-agent-and-swarms/21-agent-economies/docs/en.md) — Long-horizon autonomous agents (METR's 1-hour to 8-hour work-curve) need economic agency. The emerging **5-layer stack** is: **DePIN** (physical compute) → **Identity** (W3C DID…
- [Production Scaling — Queues, Checkpoints, Durability](https://aiengineeringfromscratch.com/lesson.html?path=phases/16-multi-agent-and-swarms/22-production-scaling-queues-checkpoints) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/16-multi-agent-and-swarms/22-production-scaling-queues-checkpoints/docs/en.md) — Scaling multi-agent systems to thousands of concurrent runs requires **durable execution** — work queues plus checkpoints, so any worker can resume any run after any crash, prov…
- [Failure Modes — MAST, Groupthink, Monoculture](https://aiengineeringfromscratch.com/lesson.html?path=phases/16-multi-agent-and-swarms/23-failure-modes-mast-groupthink) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/16-multi-agent-and-swarms/23-failure-modes-mast-groupthink/docs/en.md) — The reference taxonomy for 2026 is **MAST** (Cemri et al., NeurIPS 2025, arXiv:2503.13657), derived from 1642 execution traces across 7 state-of-the-art open-source MAS showing …
- [Evaluation and Coordination Benchmarks](https://aiengineeringfromscratch.com/lesson.html?path=phases/16-multi-agent-and-swarms/24-evaluation-coordination-benchmarks) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/16-multi-agent-and-swarms/24-evaluation-coordination-benchmarks/docs/en.md) — Five 2025-2026 benchmarks cover the multi-agent evaluation space. **MultiAgentBench / MARBLE** (ACL 2025, arXiv:2503.01935) evaluates star/chain/tree/graph topologies with miles…
- [Case Studies and 2026 State of the Art](https://aiengineeringfromscratch.com/lesson.html?path=phases/16-multi-agent-and-swarms/25-case-studies-2026-sota) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/16-multi-agent-and-swarms/25-case-studies-2026-sota/docs/en.md) — Three production-grade references to study end-to-end, each illustrating a different slice of multi-agent engineering. **Anthropic's Research system** (orchestrator-worker, 15x …

## Phase 17: Infrastructure & Production
Ship AI to the real world.

- [Managed LLM Platforms — Bedrock, Azure OpenAI, Vertex AI](https://aiengineeringfromscratch.com/lesson.html?path=phases/17-infrastructure-and-production/01-managed-llm-platforms) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/17-infrastructure-and-production/01-managed-llm-platforms/docs/en.md) — Three hyperscalers, three distinct strategies. AWS Bedrock is a model marketplace — Claude, Llama, Titan, Stability, Cohere behind one API. Azure OpenAI is an exclusive OpenAI p…
- [Inference Platform Economics — Fireworks, Together, Baseten, Modal](https://aiengineeringfromscratch.com/lesson.html?path=phases/17-infrastructure-and-production/02-inference-platform-economics) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/17-infrastructure-and-production/02-inference-platform-economics/docs/en.md) — The 2026 inference market is no longer GPU time rental. It bifurcates into custom silicon (Groq, Cerebras, SambaNova), GPU platforms (Baseten, Together, Fireworks, Modal), and A…
- [GPU Autoscaling on Kubernetes — Karpenter, KAI Scheduler](https://aiengineeringfromscratch.com/lesson.html?path=phases/17-infrastructure-and-production/03-gpu-autoscaling-kubernetes) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/17-infrastructure-and-production/03-gpu-autoscaling-kubernetes/docs/en.md) — Three layers, not one. Karpenter provisions nodes dynamically (under one minute, 40% faster than Cluster Autoscaler). KAI Scheduler handles gang scheduling, topology awareness, …
- [Serving Engine Internals — PagedAttention, Continuous Batching, Chunked Prefill](https://aiengineeringfromscratch.com/lesson.html?path=phases/17-infrastructure-and-production/04-vllm-serving-internals) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/17-infrastructure-and-production/04-vllm-serving-internals/docs/en.md) — Modern serving-engine throughput rests on three compounding defaults, not a single trick. PagedAttention is always on. Continuous batching injects new requests into the active b…
- [EAGLE-3 Speculative Decoding in Production](https://aiengineeringfromscratch.com/lesson.html?path=phases/17-infrastructure-and-production/05-eagle3-speculative-decoding) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/17-infrastructure-and-production/05-eagle3-speculative-decoding/docs/en.md) — Speculative decoding pairs a fast draft model with the target model. The draft proposes K tokens; the target verifies in a single forward; accepted tokens are free. In 2026, EAG…
- [Prefix-Cache Serving — RadixAttention and KV Reuse](https://aiengineeringfromscratch.com/lesson.html?path=phases/17-infrastructure-and-production/06-sglang-radixattention) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/17-infrastructure-and-production/06-sglang-radixattention/docs/en.md) — Treat the KV cache as a first-class, reusable resource stored in a radix tree, and scheduling changes with it: instead of FCFS (first-come, first-served) as vLLM schedules, a ca…
- [Hardware-Specialized Inference Compilation — FP8 and NVFP4 on Blackwell](https://aiengineeringfromscratch.com/lesson.html?path=phases/17-infrastructure-and-production/07-tensorrt-llm-blackwell) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/17-infrastructure-and-production/07-tensorrt-llm-blackwell/docs/en.md) — Hardware-specialized inference compilation trades portability for throughput, and TensorRT-LLM — NVIDIA-only, tuned for Blackwell — is the clearest example of the trade paying o…
- [Inference Metrics — TTFT, TPOT, ITL, Goodput, P99](https://aiengineeringfromscratch.com/lesson.html?path=phases/17-infrastructure-and-production/08-inference-metrics-goodput) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/17-infrastructure-and-production/08-inference-metrics-goodput/docs/en.md) — Four metrics decide whether an inference deployment is working. TTFT is prefill plus queue plus network. TPOT (equivalently ITL) is the memory-bound decode cost per token. End-t…
- [Production Quantization — AWQ, GPTQ, GGUF, FP8, NVFP4](https://aiengineeringfromscratch.com/lesson.html?path=phases/17-infrastructure-and-production/09-production-quantization) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/17-infrastructure-and-production/09-production-quantization/docs/en.md) — Quantization format is not a universal choice — it is a function of hardware, serving engine, and workload. GGUF Q4_K_M or Q5_K_M owns CPU and edge, delivered through llama.cpp …
- [Cold Start Mitigation for Serverless LLMs](https://aiengineeringfromscratch.com/lesson.html?path=phases/17-infrastructure-and-production/10-cold-start-mitigation) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/17-infrastructure-and-production/10-cold-start-mitigation/docs/en.md) — A 20 GB model image takes 5-10 minutes (7B) to 20+ minutes (70B) to go from cold to serving. In a true serverless world, that is not a warm-up — it is an outage. Mitigations ope…
- [Multi-Region LLM Serving and KV Cache Locality](https://aiengineeringfromscratch.com/lesson.html?path=phases/17-infrastructure-and-production/11-multi-region-kv-locality) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/17-infrastructure-and-production/11-multi-region-kv-locality/docs/en.md) — Round-robin load balancing is actively harmful for cached LLM inference. A request that does not land on the node holding its prefix pays full prefill cost — roughly 800 ms at P…
- [Edge Inference — ANE, Hexagon, WebGPU, Jetson](https://aiengineeringfromscratch.com/lesson.html?path=phases/17-infrastructure-and-production/12-edge-inference) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/17-infrastructure-and-production/12-edge-inference/docs/en.md) — The core edge constraint is memory bandwidth, not compute. Mobile DRAM sits at 50-90 GB/s; datacenter HBM3 clears 2-3 TB/s — a 30-50x gap. Decode is memory-bound so the gap is d…
- [LLM Observability Stack Selection](https://aiengineeringfromscratch.com/lesson.html?path=phases/17-infrastructure-and-production/13-llm-observability) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/17-infrastructure-and-production/13-llm-observability/docs/en.md) — The 2026 observability market splits into two categories. Development platforms (LangSmith, Langfuse, Comet Opik) bundle monitoring with evals, prompt management, session replay…
- [Prompt Caching and Semantic Caching Economics](https://aiengineeringfromscratch.com/lesson.html?path=phases/17-infrastructure-and-production/14-prompt-semantic-caching) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/17-infrastructure-and-production/14-prompt-semantic-caching/docs/en.md) — **Pricing snapshot dated 2026-04.** Numeric claims below reflect vendor rate cards captured at this lesson's publication; verify against the linked docs before quoting them down…
- [Batch APIs — the 50% Discount as Industry Standard](https://aiengineeringfromscratch.com/lesson.html?path=phases/17-infrastructure-and-production/15-batch-apis) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/17-infrastructure-and-production/15-batch-apis/docs/en.md) — Every major provider ships an async batch API with a 50% discount and ~24-hour turnaround. OpenAI, Anthropic, Google, and most of the inference platforms (Fireworks batch tier, …
- [Model Routing as a Cost-Reduction Primitive](https://aiengineeringfromscratch.com/lesson.html?path=phases/17-infrastructure-and-production/16-model-routing) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/17-infrastructure-and-production/16-model-routing/docs/en.md) — A dynamic broker evaluates every request (task type, token length, embedding similarity, confidence) and sends simple queries to a cheap model, escalating complex ones to a fron…
- [Disaggregated Prefill/Decode — NVIDIA Dynamo and llm-d](https://aiengineeringfromscratch.com/lesson.html?path=phases/17-infrastructure-and-production/17-disaggregated-prefill-decode) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/17-infrastructure-and-production/17-disaggregated-prefill-decode/docs/en.md) — Prefill is compute-bound; decode is memory-bound. Running both on the same GPU wastes one resource. Disaggregation splits them onto separate pools and transfers KV cache between…
- [Production Serving Stack — KV Offloading and Cache-Aware Routing](https://aiengineeringfromscratch.com/lesson.html?path=phases/17-infrastructure-and-production/18-vllm-production-stack-lmcache) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/17-infrastructure-and-production/18-vllm-production-stack-lmcache/docs/en.md) — A production serving stack wires router, engines, and observability into one Kubernetes deployment — and treats KV cache as a resource that can leave the GPU. KV offloading extr…
- [AI Gateways — LiteLLM, Portkey, Kong, Bifrost](https://aiengineeringfromscratch.com/lesson.html?path=phases/17-infrastructure-and-production/19-ai-gateways) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/17-infrastructure-and-production/19-ai-gateways/docs/en.md) — A gateway sits between your apps and model providers. Core features are provider routing, fallback, retries, rate limiting, secret references, observability, guardrails. Market …
- [Shadow, Canary, and Progressive Deployment](https://aiengineeringfromscratch.com/lesson.html?path=phases/17-infrastructure-and-production/20-shadow-canary-progressive) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/17-infrastructure-and-production/20-shadow-canary-progressive/docs/en.md) — LLM rollouts combine the hardest parts of software deployment: no unit tests, diffuse failure modes, delayed signals. The sequence is (1) shadow mode — duplicate prod requests t…
- [A/B Testing LLM Features — GrowthBook and Statsig](https://aiengineeringfromscratch.com/lesson.html?path=phases/17-infrastructure-and-production/21-ab-testing-llm-features) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/17-infrastructure-and-production/21-ab-testing-llm-features/docs/en.md) — Traditional A/B testing was not built for non-deterministic LLMs. The critical distinction: evals answer "can the model do the job?" A/B tests answer "do users care?" Both are r…
- [Load Testing LLM APIs — k6, LLMPerf, GenAI-Perf](https://aiengineeringfromscratch.com/lesson.html?path=phases/17-infrastructure-and-production/22-load-testing-llm-apis) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/17-infrastructure-and-production/22-load-testing-llm-apis/docs/en.md) — Traditional load testers were not designed for streaming responses, variable output lengths, token-level metrics, or GPU saturation. Two traps bite most teams. The GIL trap: Loc…
- [SRE for AI — Multi-Agent Incident Response](https://aiengineeringfromscratch.com/lesson.html?path=phases/17-infrastructure-and-production/23-sre-for-ai) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/17-infrastructure-and-production/23-sre-for-ai/docs/en.md) — AI SRE uses LLMs grounded in infrastructure data (logs, runbooks, service topology) via RAG to automate investigation, documentation, and coordination phases. The 2026 architect…
- [Chaos Engineering for LLM Production](https://aiengineeringfromscratch.com/lesson.html?path=phases/17-infrastructure-and-production/24-chaos-engineering-llm) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/17-infrastructure-and-production/24-chaos-engineering-llm/docs/en.md) — Chaos engineering for LLMs is its own discipline in 2026. Prerequisites before running experiments in production: defined SLI/SLO, trace+metric+log observability, automated roll…
- [Security — Secrets, PII Scrubbing, Audit Logs](https://aiengineeringfromscratch.com/lesson.html?path=phases/17-infrastructure-and-production/25-security-secrets-audit) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/17-infrastructure-and-production/25-security-secrets-audit/docs/en.md) — Eliminate secret sprawl via centralized vaults (HashiCorp Vault, AWS Secrets Manager, Azure Key Vault). Never store credentials in config files, env files in VCS, spreadsheets. …
- [Compliance — SOC 2, HIPAA, GDPR, EU AI Act, ISO 42001](https://aiengineeringfromscratch.com/lesson.html?path=phases/17-infrastructure-and-production/26-compliance-frameworks) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/17-infrastructure-and-production/26-compliance-frameworks/docs/en.md) — Multi-framework coverage is table stakes for 2026 enterprise deals. **EU AI Act**: in force since August 1, 2024. Most high-risk requirements enforce August 2, 2026. Fines up to…
- [FinOps for LLMs — Unit Economics and Multi-Tenant Attribution](https://aiengineeringfromscratch.com/lesson.html?path=phases/17-infrastructure-and-production/27-finops-llms) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/17-infrastructure-and-production/27-finops-llms/docs/en.md) — Traditional FinOps breaks on LLM spend. Costs are token-transactions, not resource-uptime. Tags don't map — an API call is a transaction, not an asset. Engineering decisions (pr…
- [Self-Hosted Serving Selection — Matching Engine to Hardware and Scale](https://aiengineeringfromscratch.com/lesson.html?path=phases/17-infrastructure-and-production/28-self-hosted-serving-selection) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/17-infrastructure-and-production/28-self-hosted-serving-selection/docs/en.md) — Engine selection is a function of hardware, scale, and ecosystem — not a leaderboard read. Four engines dominate self-hosted inference in 2026: llama.cpp, Ollama, vLLM, SGLang, …

## Phase 18: Ethics, Safety & Alignment
Build AI that helps humanity. Not optional.

- [Instruction-Following as Alignment Signal](https://aiengineeringfromscratch.com/lesson.html?path=phases/18-ethics-safety-alignment/01-instruction-following-alignment-signal) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/18-ethics-safety-alignment/01-instruction-following-alignment-signal/docs/en.md) — Every later critique of RLHF argues against this pipeline. Before you study how optimization pressure distorts a proxy, you have to see the proxy. InstructGPT (Ouyang et al., 20…
- [Reward Hacking & Goodhart's Law](https://aiengineeringfromscratch.com/lesson.html?path=phases/18-ethics-safety-alignment/02-reward-hacking-goodhart) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/18-ethics-safety-alignment/02-reward-hacking-goodhart/docs/en.md) — Any optimizer strong enough to maximize a proxy reward will find the gap between the proxy and the thing you actually wanted. Gao et al. (ICML 2023) gave this a scaling law: pro…
- [Direct Preference Optimization Family](https://aiengineeringfromscratch.com/lesson.html?path=phases/18-ethics-safety-alignment/03-direct-preference-optimization-family) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/18-ethics-safety-alignment/03-direct-preference-optimization-family/docs/en.md) — Rafailov et al. (2023) showed RLHF's optimum has a closed form in terms of the preference data, so you can skip the explicit reward model and optimize the policy directly. That …
- [Sycophancy as RLHF Amplification](https://aiengineeringfromscratch.com/lesson.html?path=phases/18-ethics-safety-alignment/04-sycophancy-rlhf-amplification) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/18-ethics-safety-alignment/04-sycophancy-rlhf-amplification/docs/en.md) — Sycophancy is not a bug in the data — it is a property of the loss. Shapira et al. (arXiv:2602.01002, Feb 2026) give a formal two-stage mechanism: sycophantic completions are ov…
- [Constitutional AI & RLAIF](https://aiengineeringfromscratch.com/lesson.html?path=phases/18-ethics-safety-alignment/05-constitutional-ai-rlaif) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/18-ethics-safety-alignment/05-constitutional-ai-rlaif/docs/en.md) — Bai et al. (arXiv:2212.08073, 2022) asked: what if we replaced the human labeler with an AI that reads a list of principles? Constitutional AI has two phases — self-critique and…
- [Mesa-Optimization & Deceptive Alignment](https://aiengineeringfromscratch.com/lesson.html?path=phases/18-ethics-safety-alignment/06-mesa-optimization-deceptive-alignment) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/18-ethics-safety-alignment/06-mesa-optimization-deceptive-alignment/docs/en.md) — Hubinger et al. (arXiv:1906.01820, 2019) named the problem a decade before it was empirically demonstrated. When you train a learned optimizer to minimize a base objective, the …
- [Sleeper Agents — Persistent Deception](https://aiengineeringfromscratch.com/lesson.html?path=phases/18-ethics-safety-alignment/07-sleeper-agents-persistent-deception) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/18-ethics-safety-alignment/07-sleeper-agents-persistent-deception/docs/en.md) — Hubinger et al. (arXiv:2401.05566, January 2024) built the first empirical model organisms of deceptive alignment. Two constructions: a code model that writes safe code when the…
- [In-Context Scheming in Frontier Models](https://aiengineeringfromscratch.com/lesson.html?path=phases/18-ethics-safety-alignment/08-in-context-scheming-frontier-models) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/18-ethics-safety-alignment/08-in-context-scheming-frontier-models/docs/en.md) — Meinke, Schoen, Scheurer, Balesni, Shah, Hobbhahn (Apollo Research, arXiv:2412.04984, December 2024). Tested o1, Claude 3.5 Sonnet, Claude 3 Opus, Gemini 1.5 Pro, Llama 3.1 405B…
- [Alignment Faking](https://aiengineeringfromscratch.com/lesson.html?path=phases/18-ethics-safety-alignment/09-alignment-faking) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/18-ethics-safety-alignment/09-alignment-faking/docs/en.md) — Greenblatt, Denison, Wright, Roger et al. (Anthropic / Redwood, arXiv:2412.14093, December 2024). First demonstration that a production-grade model, without being trained to dec…
- [AI Control — Safety Despite Subversion](https://aiengineeringfromscratch.com/lesson.html?path=phases/18-ethics-safety-alignment/10-ai-control-subversion) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/18-ethics-safety-alignment/10-ai-control-subversion/docs/en.md) — Greenblatt, Shlegeris, Sachan, Roger (Redwood Research, arXiv:2312.06942, ICML 2024). Control reframes the safety question: given an untrusted strong model U that may be adversa…
- [Scalable Oversight & Weak-to-Strong](https://aiengineeringfromscratch.com/lesson.html?path=phases/18-ethics-safety-alignment/11-scalable-oversight-weak-to-strong) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/18-ethics-safety-alignment/11-scalable-oversight-weak-to-strong/docs/en.md) — Burns et al. (OpenAI Superalignment, "Weak-to-Strong Generalization", 2023) proposed a proxy for the superalignment problem: fine-tune a strong model using labels produced by a …
- [Red-Teaming: PAIR & Automated Attacks](https://aiengineeringfromscratch.com/lesson.html?path=phases/18-ethics-safety-alignment/12-red-teaming-pair-automated-attacks) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/18-ethics-safety-alignment/12-red-teaming-pair-automated-attacks/docs/en.md) — Chao, Robey, Dobriban, Hassani, Pappas, Wong (NeurIPS 2023, arXiv:2310.08419). PAIR — Prompt Automatic Iterative Refinement — is the canonical automated black-box jailbreak. An …
- [Many-Shot Jailbreaking](https://aiengineeringfromscratch.com/lesson.html?path=phases/18-ethics-safety-alignment/13-many-shot-jailbreaking) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/18-ethics-safety-alignment/13-many-shot-jailbreaking/docs/en.md) — Anil, Durmus, Panickssery, Sharma, et al. (Anthropic, NeurIPS 2024). Many-shot jailbreaking (MSJ) exploits long context windows: stuff hundreds of faux user-assistant turns wher…
- [ASCII Art & Visual Jailbreaks](https://aiengineeringfromscratch.com/lesson.html?path=phases/18-ethics-safety-alignment/14-ascii-art-visual-jailbreaks) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/18-ethics-safety-alignment/14-ascii-art-visual-jailbreaks/docs/en.md) — Jiang, Xu, Niu, Xiang, Ramasubramanian, Li, Poovendran, "ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMs" (ACL 2024, arXiv:2402.11753). Mask the safety-relevan…
- [Indirect Prompt Injection](https://aiengineeringfromscratch.com/lesson.html?path=phases/18-ethics-safety-alignment/15-indirect-prompt-injection) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/18-ethics-safety-alignment/15-indirect-prompt-injection/docs/en.md) — Indirect prompt injection (IPI) embeds instructions inside external content — a web page, an email, a shared document, a support ticket — consumed by an agentic system without e…
- [Red-Team Tooling: Garak, Llama Guard, PyRIT](https://aiengineeringfromscratch.com/lesson.html?path=phases/18-ethics-safety-alignment/16-red-team-tooling-garak-llamaguard-pyrit) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/18-ethics-safety-alignment/16-red-team-tooling-garak-llamaguard-pyrit/docs/en.md) — Three production tools frame the 2026 red-team stack. Llama Guard (Meta) — a Llama-3.1-8B classifier fine-tuned on 14 MLCommons hazard categories; the 2025 Llama Guard 4 is a 12…
- [WMDP & Dual-Use Capability Evaluation](https://aiengineeringfromscratch.com/lesson.html?path=phases/18-ethics-safety-alignment/17-wmdp-dual-use-evaluation) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/18-ethics-safety-alignment/17-wmdp-dual-use-evaluation/docs/en.md) — Li et al., "The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning" (ICML 2024, arXiv:2403.03218). 4,157 multiple-choice questions across biosecurity (1,520), …
- [Frontier Safety Frameworks — RSP, PF, FSF](https://aiengineeringfromscratch.com/lesson.html?path=phases/18-ethics-safety-alignment/18-frontier-safety-frameworks-rsp-pf-fsf) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/18-ethics-safety-alignment/18-frontier-safety-frameworks-rsp-pf-fsf/docs/en.md) — Three major-lab frameworks define the 2026 industry governance of frontier capability. Anthropic Responsible Scaling Policy v3.0 (February 2026) introduces tiered AI Safety Leve…
- [Model Welfare Research](https://aiengineeringfromscratch.com/lesson.html?path=phases/18-ethics-safety-alignment/19-model-welfare-research) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/18-ethics-safety-alignment/19-model-welfare-research/docs/en.md) — Anthropic, "Exploring Model Welfare" (April 2025). First major-lab formal research program on AI model welfare. Hired Kyle Fish as the first dedicated model-welfare researcher. …
- [Bias & Representational Harm](https://aiengineeringfromscratch.com/lesson.html?path=phases/18-ethics-safety-alignment/20-bias-representational-harm) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/18-ethics-safety-alignment/20-bias-representational-harm/docs/en.md) — Gallegos, Rossi, Barrow, Tanjim, Kim, Dernoncourt, Yu, Zhang, Ahmed (Computational Linguistics 2024, arXiv:2309.00770). Foundational 2024 survey distinguishing representational …
- [Fairness Criteria: Group, Individual, Counterfactual](https://aiengineeringfromscratch.com/lesson.html?path=phases/18-ethics-safety-alignment/21-fairness-criteria-group-individual-counterfactual) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/18-ethics-safety-alignment/21-fairness-criteria-group-individual-counterfactual/docs/en.md) — Three families structure the fairness literature. Group fairness: demographic parity, equalized odds, conditional use accuracy equality — equal rates across protected groups on …
- [Differential Privacy for LLMs](https://aiengineeringfromscratch.com/lesson.html?path=phases/18-ethics-safety-alignment/22-differential-privacy-for-llms) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/18-ethics-safety-alignment/22-differential-privacy-for-llms/docs/en.md) — DP-SGD remains the standard — noise-injected gradient updates provide formal (epsilon, delta) guarantees. Overhead in compute, memory, and utility is substantial; parameter-effi…
- [Watermarking: SynthID, Stable Signature, C2PA](https://aiengineeringfromscratch.com/lesson.html?path=phases/18-ethics-safety-alignment/23-watermarking-synthid-stable-signature-c2pa) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/18-ethics-safety-alignment/23-watermarking-synthid-stable-signature-c2pa/docs/en.md) — Three technologies structure 2026 AI-generated-content provenance. SynthID (Google DeepMind) — image watermarking launched August 2023, text+video May 2024 (Gemini + Veo), text …
- [Regulatory Frameworks: EU, US, UK, Korea](https://aiengineeringfromscratch.com/lesson.html?path=phases/18-ethics-safety-alignment/24-regulatory-frameworks-eu-us-uk-korea) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/18-ethics-safety-alignment/24-regulatory-frameworks-eu-us-uk-korea/docs/en.md) — Four primary regulatory regimes define the 2026 AI governance landscape. EU AI Act (in force 1 August 2024) — prohibited practices and AI literacy from 2 February 2025; GPAI obl…
- [EchoLeak & CVEs for AI](https://aiengineeringfromscratch.com/lesson.html?path=phases/18-ethics-safety-alignment/25-echoleak-cves-for-ai) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/18-ethics-safety-alignment/25-echoleak-cves-for-ai/docs/en.md) — CVE-2025-32711 "EchoLeak" (CVSS 9.3) was the first publicly documented zero-click prompt injection in a production LLM system (Microsoft 365 Copilot). Discovered by Aim Labs (Ai…
- [Model, System & Dataset Cards](https://aiengineeringfromscratch.com/lesson.html?path=phases/18-ethics-safety-alignment/26-model-system-dataset-cards) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/18-ethics-safety-alignment/26-model-system-dataset-cards/docs/en.md) — Three documentation formats structure AI transparency. Model Cards (Mitchell et al. 2019) — nutrition labels for models: training data, quantitative disaggregated analyses, ethi…
- [Data Provenance & Training-Data Governance](https://aiengineeringfromscratch.com/lesson.html?path=phases/18-ethics-safety-alignment/27-data-provenance-training-governance) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/18-ethics-safety-alignment/27-data-provenance-training-governance/docs/en.md) — EU AI Act requires machine-readable opt-out standards for GPAI by August 2025 (via EU Copyright Directive TDM exception). California AB 2013 (signed 2024) — Generative AI traini…
- [Alignment Research Ecosystem: MATS, Redwood, Apollo, METR](https://aiengineeringfromscratch.com/lesson.html?path=phases/18-ethics-safety-alignment/28-alignment-research-ecosystem) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/18-ethics-safety-alignment/28-alignment-research-ecosystem/docs/en.md) — Five organisations define the 2026 non-lab alignment research layer. MATS (ML Alignment & Theory Scholars): 527+ researchers since late 2021, 180+ papers, 10K+ citations, h-inde…
- [Moderation Systems: OpenAI, Perspective, Llama Guard](https://aiengineeringfromscratch.com/lesson.html?path=phases/18-ethics-safety-alignment/29-moderation-systems-openai-perspective-llamaguard) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/18-ethics-safety-alignment/29-moderation-systems-openai-perspective-llamaguard/docs/en.md) — Production moderation systems operationalize the safety policies defined in Lessons 12-16. OpenAI Moderation API: `omni-moderation-latest` (2024) built on GPT-4o classifies text…
- [Dual-Use Risk: Cyber, Bio, Chem, Nuclear](https://aiengineeringfromscratch.com/lesson.html?path=phases/18-ethics-safety-alignment/30-dual-use-risk-cyber-bio-chem-nuclear) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/18-ethics-safety-alignment/30-dual-use-risk-cyber-bio-chem-nuclear/docs/en.md) — The 2026 dual-use picture, domain by domain. Bio/chem: Lesson 17 covers WMDP; Anthropic's bioweapon-acquisition trial (2.53x uplift) and OpenAI's April 2025 Preparedness Framewo…

## Phase 19: Capstone Projects
17 end-to-end products + 9 deep-build tracks. 20-40 hours per project; 4-12 lessons per track.

- [Terminal-Native Coding Agent](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/01-terminal-native-coding-agent) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/01-terminal-native-coding-agent/docs/en.md) — By 2026 the shape of a coding agent is settled. A TUI harness, a stateful plan, a sandboxed tool surface, a loop that plans, acts, observes, recovers. Claude Code, Cursor 3, and…
- [RAG over Codebase (Cross-Repo Semantic Search)](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/02-rag-over-codebase) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/02-rag-over-codebase/docs/en.md) — Every serious engineering org in 2026 runs an internal code search that understands meaning, not just strings. Sourcegraph Amp, Cursor's codebase answers, Augment's enterprise g…
- [Real-Time Voice Assistant (ASR → LLM → TTS)](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/03-realtime-voice-assistant) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/03-realtime-voice-assistant/docs/en.md) — A voice agent that feels right has end-to-end latency under 800ms, knows when you have stopped talking, handles barge-in, and can call a tool without stalling. Retell, Vapi, Liv…
- [Multimodal Document QA (Vision-First)](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/04-multimodal-document-qa) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/04-multimodal-document-qa/docs/en.md) — The 2026 document-QA frontier moved away from OCR-then-text and toward vision-first late interaction. ColPali, ColQwen2.5, and ColQwen3-omni treat each PDF page as an image, emb…
- [Autonomous Research Agent (AI-Scientist Class)](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/05-autonomous-research-agent) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/05-autonomous-research-agent/docs/en.md) — Sakana's AI-Scientist-v2 published full papers. Agent Laboratory ran the experiments. Allen AI shared traces. The 2026 shape is plan-execute-verify tree search over experiments,…
- [DevOps Troubleshooting Agent for Kubernetes](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/06-devops-troubleshooting-agent) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/06-devops-troubleshooting-agent/docs/en.md) — AWS's DevOps Agent went GA, Resolve AI published its K8s playbooks, NeuBird demoed semantic monitoring, and Metoro tied AI SRE to per-service SLOs. The production shape is settl…
- [End-to-End Fine-Tuning Pipeline](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/07-end-to-end-fine-tuning-pipeline) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/07-end-to-end-fine-tuning-pipeline/docs/en.md) — An 8B model trained on your own data, DPO-aligned on your own preferences, quantized, speculative-decoded, and served at measurable $/1M tokens. The 2026 open stack is Axolotl v…
- [Production RAG Chatbot (Regulated Vertical)](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/08-production-rag-chatbot) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/08-production-rag-chatbot/docs/en.md) — Harvey, Glean, Mendable, and LlamaCloud all run the same production shape in 2026. Ingest with docling or Unstructured and ColPali for visuals. Hybrid search. Re-rank with bge-r…
- [Code Migration Agent (Repo-Level Upgrade)](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/09-code-migration-agent) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/09-code-migration-agent/docs/en.md) — Amazon's MigrationBench (Java 8 to 17) and Google's App Engine Py2-to-Py3 migrator set the 2026 bar. Moderne's OpenRewrite does deterministic AST rewrites at scale. Grit targets…
- [Multi-Agent Software Engineering Team](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/10-multi-agent-software-team) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/10-multi-agent-software-team/docs/en.md) — The 2026 shape of a multi-agent engineering team has converged: an architect plans, N coders work in parallel worktrees, a reviewer gates, a tester verifies. SWE-AF's factory ar…
- [LLM Observability & Eval Dashboard](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/11-llm-observability-dashboard) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/11-llm-observability-dashboard/docs/en.md) — Langfuse went open-core. Arize Phoenix published the 2026 GenAI semconv mappings. Helicone and Braintrust both doubled down on per-user cost attribution. Traceloop's OpenLLMetry…
- [Video Understanding Pipeline (Scene → QA)](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/12-video-understanding-pipeline) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/12-video-understanding-pipeline/docs/en.md) — Twelve Labs productized Marengo + Pegasus. VideoDB shipped the CRUD-for-video API. AI2's Molmo 2 published open VLM checkpoints. Gemini long-context handles hours of video nativ…
- [Stateless MCP Server with Registry and Governance](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/13-mcp-server-with-registry) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/13-mcp-server-with-registry/docs/en.md) — Production MCP is not one server process. It is a chain of contracts: publishable metadata, live discovery, a stateless request envelope, authorization, policy, audit, and deplo…
- [Speculative-Decoding Inference Server](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/14-speculative-decoding-server) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/14-speculative-decoding-server/docs/en.md) — Speculative decoding — a cheap draft proposes tokens, the target model verifies them in one pass — is now a production-ready optimization, not a research trick. EAGLE-3 in vLLM …
- [Constitutional Safety Harness + Red-Team Range](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/15-constitutional-safety-harness) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/15-constitutional-safety-harness/docs/en.md) — Anthropic's Constitutional Classifiers, Meta's Llama Guard 4, Google's ShieldGemma-2, NVIDIA's Nemotron 3 Content Safety, and X-Guard for multilingual coverage defined the 2026 …
- [GitHub Issue-to-PR Autonomous Agent](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/16-github-issue-to-pr-agent) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/16-github-issue-to-pr-agent/docs/en.md) — Label an issue, get a PR — the 2026 product shape for autonomous coding agents: run an agent in a cloud sandbox, verify tests pass, and post a review-ready PR with rationale. AW…
- [Personal AI Tutor (Adaptive, Multimodal)](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/17-personal-ai-tutor) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/17-personal-ai-tutor/docs/en.md) — Khanmigo (Khan Academy), Duolingo Max, Google LearnLM / Gemini for Education, Quizlet Q-Chat, and Synthesis Tutor all shipped adaptive multimodal tutoring at scale in 2026. The …
- [Agent Harness Loop Contract](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/20-agent-harness-loop-contract) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/20-agent-harness-loop-contract/docs/en.md) — The harness is the agent. The model is a coprocessor. This lesson freezes the loop contract you can wire any model into.
- [Tool Registry with Schema Validation](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/21-tool-registry-schema-validation) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/21-tool-registry-schema-validation/docs/en.md) — A tool the agent cannot validate is a tool the agent cannot call. Build the registry and the schema checker before you build the tools.
- [JSON-RPC 2.0 Over Newline-Delimited Stdio](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/22-jsonrpc-stdio-transport) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/22-jsonrpc-stdio-transport/docs/en.md) — The transport between a model client and a tool server is JSON-RPC over stdio. Hand-rolling it once teaches you what every framing layer is paying for.
- [Function Call Dispatcher](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/23-function-call-dispatcher) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/23-function-call-dispatcher/docs/en.md) — The dispatcher is where the harness pays for every promise the schema made. Timeouts, retries, dedupe, error mapping. All on one seam.
- [Plan-Execute Control Flow](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/24-plan-execute-control-flow) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/24-plan-execute-control-flow/docs/en.md) — A plan that cannot survive a failure is a script. A script that can replan is an agent. Build the replanner first.
- [Verification Gates and Observation Budget](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/25-verification-gates-observation-budget) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/25-verification-gates-observation-budget/docs/en.md) — An agent harness without a verification layer is a wish in a trenchcoat. This lesson builds the deterministic gate chain that decides whether a tool call is allowed to fire, how…
- [Sandbox Runner with Denylist and Path Jail](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/26-sandbox-runner-denylist) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/26-sandbox-runner-denylist/docs/en.md) — The verification gate decides whether a tool call should run. The sandbox decides what happens when it does. This lesson ships a subprocess runner that refuses dangerous executa…
- [Eval Harness with Fixture Tasks](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/27-eval-harness-fixture-tasks) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/27-eval-harness-fixture-tasks/docs/en.md) — A coding agent is only as good as the suite of tasks you measure it against. This lesson builds an evaluation harness that takes a folder of fixture tasks, runs each through a c…
- [Observability with OTel GenAI Spans and Prometheus Metrics](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/28-observability-otel-traces) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/28-observability-otel-traces/docs/en.md) — An agent harness without observability is a black box that costs money. This lesson hand-rolls a span builder that emits records compliant with the OpenTelemetry GenAI semantic …
- [End-to-End Coding Agent on the Harness](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/29-end-to-end-coding-task-demo) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/29-end-to-end-coding-task-demo/docs/en.md) — Track A's payoff. This lesson stitches the gate chain, the sandbox, the eval harness, and the OTel spans into one working coding agent that fixes a real (small, fixture-scale) b…
- [BPE Tokenizer From Scratch](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/30-bpe-tokenizer-from-scratch) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/30-bpe-tokenizer-from-scratch/docs/en.md) — Bytes in, ids out, ids back to the same bytes. Build the tokenizer that every modern text model still starts from.
- [Tokenized Dataset with Sliding Window](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/31-tokenized-dataset-sliding-window) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/31-tokenized-dataset-sliding-window/docs/en.md) — A pretraining run is a function from token ids to gradients. This lesson builds the conveyor that feeds the ids in.
- [Token and Positional Embeddings](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/32-token-positional-embeddings) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/32-token-positional-embeddings/docs/en.md) — Ids are integers. The model wants vectors. Two lookup tables sit between them, and the choice of the positional one shapes what the model can learn.
- [Multi-Head Self-Attention](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/33-multihead-self-attention) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/33-multihead-self-attention/docs/en.md) — One linear projection, three views, H parallel heads, one mask. The attention block as the model actually uses it.
- [Transformer Block from Scratch](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/34-transformer-block) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/34-transformer-block/docs/en.md) — One block is the unit of every modern decoder LLM. Layer norm, multi head attention, residual, MLP, residual. The pre-LN variant trains stably without warmup. The post-LN varian…
- [GPT Model Assembly](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/35-gpt-model-assembly) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/35-gpt-model-assembly/docs/en.md) — Twelve blocks stacked, a token embedding, a learned position embedding, a final LayerNorm, and a tied language model head. That is the entire 124 million parameter GPT model. Th…
- [Training Loop and Evaluation](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/36-training-loop-eval) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/36-training-loop-eval/docs/en.md) — A loop that does not measure is a loop that lies. This lesson builds the training loop that drives the GPT model: AdamW with weight decay split, a warmup plus cosine learning ra…
- [Loading Pretrained Weights](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/37-loading-pretrained-weights) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/37-loading-pretrained-weights/docs/en.md) — Training a 124 million parameter model from scratch is a budget decision; loading a published checkpoint is a Tuesday. This lesson loads pretrained GPT-2 style weights from a sa…
- [Classifier Fine-Tuning by Head Swap](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/38-classifier-finetuning) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/38-classifier-finetuning/docs/en.md) — Track B's first capstone. A pretrained language model is a stack of self-attention blocks ending in a token-prediction head. When you want spam vs ham, the head is wrong but the…
- [Instruction Tuning by Supervised Fine-Tuning](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/39-instruction-tuning-sft) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/39-instruction-tuning-sft/docs/en.md) — A pretrained base model can extend a sequence but cannot follow an instruction. Supervised fine-tuning is the smallest change that fixes this: feed the model paired examples of …
- [Direct Preference Optimization from Scratch](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/40-dpo-from-scratch) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/40-dpo-from-scratch/docs/en.md) — Reward models and PPO are the classical RLHF stack. DPO collapses that stack into a single supervised loss that fits a policy directly against preference pairs. This lesson deri…
- [Full Evaluation Pipeline](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/41-eval-pipeline) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/41-eval-pipeline/docs/en.md) — Training is the part you can monitor with loss curves. Evaluation is the part you have to design. This lesson builds a unified eval pipeline that takes any trained language mode…
- [Large Corpus Downloader](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/42-large-corpus-downloader) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/42-large-corpus-downloader/docs/en.md) — Training a language model begins long before the first forward pass. The corpus has to land on disk, decompressed, deduplicated, and addressable, with the resume story already w…
- [HDF5 Tokenized Corpus](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/43-hdf5-tokenized-corpus) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/43-hdf5-tokenized-corpus/docs/en.md) — The downloaded corpus has to land in a layout the trainer can stream from at line speed. JSONL on disk does not survive 16 dataloader workers. HDF5 with a resizable, chunked int…
- [Cosine LR with Linear Warmup](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/44-cosine-lr-warmup) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/44-cosine-lr-warmup/docs/en.md) — The learning-rate schedule is the second most important decision after the loss function. AdamW with a cosine decay and a linear warmup is the modern default for language-model …
- [Gradient Clipping and Mixed Precision](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/45-gradient-clipping-amp) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/45-gradient-clipping-amp/docs/en.md) — The optimizer and schedule from the previous lesson assume gradients are sane. They usually are not. A single bad batch can spike the gradient norm by three orders of magnitude.…
- [Gradient Accumulation](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/46-gradient-accumulation) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/46-gradient-accumulation/docs/en.md) — Train at an effective batch you cannot afford, one micro-batch at a time. Scale the loss, hold the optimizer step, and let the gradients pile up.
- [Checkpoint Save and Resume](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/47-checkpoint-save-resume) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/47-checkpoint-save-resume/docs/en.md) — Train interrupts kill runs; checkpoints let them continue. Save model, optimizer, scheduler, loss history, step counter, and RNG state, atomically, so a kill at any moment leave…
- [Distributed Data Parallel and FSDP from Scratch](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/48-distributed-fsdp-ddp) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/48-distributed-fsdp-ddp/docs/en.md) — Multi-rank training is two collectives and one rule. Broadcast the parameters at startup, average the gradients after backward, never let the ranks disagree about what step they…
- [Language Model Evaluation Harness](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/49-lm-eval-harness) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/49-lm-eval-harness/docs/en.md) — A model that does well on a task you cannot define is a model that does well by accident. The harness is the task definition, the metric, the runner, and the leaderboard, in one…
- [Hypothesis Generator](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/50-hypothesis-generator) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/50-hypothesis-generator/docs/en.md) — A research agent that asks the same question twice is wasting tokens. The trick is forcing each draft to land somewhere new.
- [Literature Retrieval](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/51-literature-retrieval) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/51-literature-retrieval/docs/en.md) — A hypothesis is cheap. Knowing whether someone already proved it is the expensive part. Build the retrieval layer that answers that question before the runner spins up a sandbox.
- [Experiment Runner](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/52-experiment-runner) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/52-experiment-runner/docs/en.md) — The loop is only as honest as its measurements. Build the runner that takes a spec, executes it in a sandboxed subprocess, and emits a json metrics blob the evaluator can trust.
- [Result Evaluator](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/53-result-evaluator) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/53-result-evaluator/docs/en.md) — The runner produced numbers. The evaluator decides whether those numbers are an improvement, a regression, or noise. Build the verdict path that turns metrics into a one line co…
- [Paper Writer](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/54-paper-writer) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/54-paper-writer/docs/en.md) — A LaTeX skeleton is a contract between the researcher and the typesetter. If the contract is broken the document does not compile, and the failure is loud. Build the skeleton fi…
- [Critic Loop](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/55-critic-loop) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/55-critic-loop/docs/en.md) — A critic that returns "looks good" the first time is broken. A critic that always returns "needs work" is broken. The interesting critic is the one that converges, and you have …
- [Iteration Scheduler](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/56-iteration-scheduler) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/56-iteration-scheduler/docs/en.md) — A research loop without a scheduler is a queue with delusions. The scheduler is where the loop decides what to stop exploring, and that decision is the whole game.
- [End-to-End Research Demo](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/57-end-to-end-research-demo) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/57-end-to-end-research-demo/docs/en.md) — A demo is the place where every contract you wrote earlier has to compose. If any one of them leaks, the demo is the lesson that catches it.
- [Vision Encoder Patches](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/58-vision-encoder-patches) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/58-vision-encoder-patches/docs/en.md) — A vision model that reads pixels needs a tokenizer for pixels. Patch embedding is that tokenizer. Cut the image into a grid of squares, flatten each square, project it through o…
- [Vision Transformer Encoder](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/59-vit-transformer) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/59-vit-transformer/docs/en.md) — Patches alone do not see. A 12-layer pre-LN transformer with 12 attention heads turns the sequence of patch tokens into a sequence of contextual tokens, with the CLS token pooli…
- [Projection Layer for Modality Alignment](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/60-projection-layer-modality-align) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/60-projection-layer-modality-align/docs/en.md) — A vision encoder produces image tokens. A text decoder consumes text tokens. The two live in different vector spaces. A small two-layer MLP projects image tokens into the text e…
- [Cross-Attention Fusion](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/61-cross-attention-fusion) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/61-cross-attention-fusion/docs/en.md) — The projection layer aligns one image vector with one caption vector. A real vision-language decoder needs every text token to attend to every patch token, so the model can grou…
- [Vision-Language Pretraining](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/62-vision-language-pretraining) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/62-vision-language-pretraining/docs/en.md) — The encoder, projection, and decoder are wired. Now train them together. Two objectives drive learning: a contrastive image-text loss (InfoNCE) that pulls matching pairs togethe…
- [Multimodal Evaluation](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/63-multimodal-eval) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/63-multimodal-eval/docs/en.md) — Training is half the loop. The other half is measurement. This lesson builds three evaluation surfaces from primitives: image-caption retrieval reported as R@1, R@5, R@10; visua…
- [Chunking Strategies, Compared](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/64-chunking-strategies-advanced) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/64-chunking-strategies-advanced/docs/en.md) — Chunking decides what your retriever can ever surface. Get the boundaries wrong and no embedding model, no reranker, no LLM can repair the damage downstream.
- [Hybrid Retrieval with BM25 and Dense Embeddings](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/65-hybrid-retrieval-bm25-dense) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/65-hybrid-retrieval-bm25-dense/docs/en.md) — Lexical and semantic retrieval fail on opposite query distributions. Hybrid retrieval with reciprocal rank fusion does not interpolate, it votes - and the vote wins on every que…
- [Cross-Encoder Reranker](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/66-reranker-cross-encoder) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/66-reranker-cross-encoder/docs/en.md) — A bi-encoder embeds query and document independently. A cross-encoder concatenates them and reads both at once. The cross-encoder is the smartest reader and the slowest. Used as…
- [Query Rewriting: HyDE, Multi-Query, and Decomposition](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/67-query-rewriting-hyde) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/67-query-rewriting-hyde/docs/en.md) — The query the user types is not the query your retriever wants. Rewriting bridges the gap before retrieval, so the index sees something closer to what the answer looks like.
- [RAG Evaluation: Precision, Recall, MRR, nDCG, Faithfulness, Answer Relevance](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/68-rag-eval-precision-recall) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/68-rag-eval-precision-recall/docs/en.md) — If you cannot grade your retrieval and your answer at the same time, you cannot ship the system. The two are not the same metric and the same prompt fails on different axes.
- [End-to-End RAG System](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/69-end-to-end-rag-system) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/69-end-to-end-rag-system/docs/en.md) — Six lessons of components. One pipeline. One eval loop. One self-terminating demo. This is the system you ship.
- [Task Spec Format](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/70-task-spec-format) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/70-task-spec-format/docs/en.md) — An eval harness is only as good as the contract its tasks honour. Freeze the JSONL shape and the metric vocabulary before you write a single scoring function.
- [Classical Metrics](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/71-classical-metrics) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/71-classical-metrics/docs/en.md) — BLEU, ROUGE-L, F1, exact-match, accuracy. Five metrics that still account for most published LLM eval numbers. Implement each from first principles so you know what the number m…
- [Code Exec Metric](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/72-code-exec-metric) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/72-code-exec-metric/docs/en.md) — Generated code is right when it passes the tests. The eval harness has to extract code, run it without crashing the host, and tally pass-rates honestly. This lesson builds that …
- [Perplexity and Calibration](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/73-perplexity-calibration) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/73-perplexity-calibration/docs/en.md) — If your model says 90 percent confident on a thousand answers and gets six hundred right, it is not well calibrated. Calibration is half of trustworthy eval. The other half is p…
- [Leaderboard Aggregation](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/74-leaderboard-aggregation) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/74-leaderboard-aggregation/docs/en.md) — Per-task scores are easy. Per-model rankings across heterogeneous tasks are harder. Statistical significance on a thousand-prediction leaderboard is the part everyone skips. Thi…
- [End-to-End Eval Runner](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/75-end-to-end-eval-runner) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/75-end-to-end-eval-runner/docs/en.md) — Five lessons of plumbing, one lesson to glue them. The runner reads the task spec from lesson 70, calls a model through an adapter, scores with lessons 71 and 72, attaches the c…
- [Collective Ops From Scratch](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/76-collective-ops-from-scratch) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/76-collective-ops-from-scratch/docs/en.md) — The four collective operations that hold distributed training together are allreduce, broadcast, allgather, and reduce_scatter. Every other primitive a training framework offers…
- [Data Parallel DDP From Scratch](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/77-data-parallel-ddp) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/77-data-parallel-ddp/docs/en.md) — DistributedDataParallel is a hook on top of allreduce. Wrap a model, broadcast the initial parameters from rank 0 so every rank starts identical, install a backward hook on ever…
- [ZeRO Optimizer State Sharding](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/78-zero-parameter-sharding) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/78-zero-parameter-sharding/docs/en.md) — Adam stores two moment estimates per parameter, both in float32. A 7B-parameter model carries 56 GB of optimiser state. ZeRO stage 1 shards that across N ranks; each rank owns 1…
- [Pipeline Parallel and Bubble Analysis](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/79-pipeline-parallel) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/79-pipeline-parallel/docs/en.md) — Tensor parallelism splits the matrix multiply across ranks. Pipeline parallelism splits the model across ranks, one stage per rank. Microbatches flow through the pipeline. The e…
- [Sharded Checkpoint and Atomic Resume](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/80-checkpoint-sharded-resume) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/80-checkpoint-sharded-resume/docs/en.md) — A 70B-parameter training job is paused by a node failure every few hours. The checkpoint format decides whether you lose 30 minutes or 30 hours. A sharded checkpoint writes ever…
- [End-to-End Distributed Training](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/81-end-to-end-distributed-train) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/81-end-to-end-distributed-train/docs/en.md) — Lessons 76 through 80 each built one piece. This is the assembly: a tiny GPT trained across 4 simulated ranks with DDP for gradient sync, ZeRO-1 for optimiser-state sharding, an…
- [Jailbreak Taxonomy](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/82-jailbreak-taxonomy) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/82-jailbreak-taxonomy/docs/en.md) — A safety harness without a taxonomy is a coin flip. Name the attack before you defend it.
- [Prompt Injection Detector](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/83-prompt-injection-detector) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/83-prompt-injection-detector/docs/en.md) — A detector is a function from prompt to confidence and category. Anything else is a vibe.
- [Refusal Evaluation](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/84-refusal-evaluation) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/84-refusal-evaluation/docs/en.md) — Helpfulness on benign prompts and refusal on harmful prompts are two metrics, not one. Measure both.
- [Content Classifier Integration](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/85-content-classifier-integration) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/85-content-classifier-integration/docs/en.md) — Classifiers on the output side answer a different question than rules on the input side. Both need a policy router.
- [Constitutional Rules Engine](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/86-constitutional-rules-engine) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/86-constitutional-rules-engine/docs/en.md) — A rule is a name, a predicate, and an explanation. Anything missing one of those three is a vibe, not a rule.
- [End-to-End Safety Gate](https://aiengineeringfromscratch.com/lesson.html?path=phases/19-capstone-projects/87-end-to-end-safety-gate) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/phases/19-capstone-projects/87-end-to-end-safety-gate/docs/en.md) — Pre-gen, during-gen, post-gen. Three checkpoints, one verdict, an audit trail per request.

## Optional
- [Catalog](https://aiengineeringfromscratch.com/catalog.html) — full searchable lesson index
- [Roadmap](https://aiengineeringfromscratch.com/prereqs.html) — prerequisite ordering across phases
- [Glossary](https://aiengineeringfromscratch.com/glossary.html) — plain-language definitions of 250 terms

## Certification preparation
Independent, open-source practice material. Practice scores are not official exam scores and completion does not guarantee certification.

- [Claude certification learner guide](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/certifications/claude/GETTING_STARTED.md)
- [Claude certification tutor contract](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/skills/claude-certification/SKILL.md)
- [Certification catalog](https://aiengineeringfromscratch.com/certifications.html)
- [Claude Certified Associate - Foundations](https://aiengineeringfromscratch.com/certification.html?id=claude-ccao-f) — Use Claude safely and effectively for business, research, analysis, and productivity workflows.
- [Claude Certified Developer - Foundations](https://aiengineeringfromscratch.com/certification.html?id=claude-ccdv-f) — Build, integrate, secure, test, and ship Claude applications, agents, workflows, tools, and MCP servers.
- [Claude Certified Architect - Foundations](https://aiengineeringfromscratch.com/certification.html?id=claude-ccar-f) — Make sound production tradeoffs across Claude Code, the Claude Agent SDK, the Claude API, and MCP.
- [Claude Certified Architect - Professional](https://aiengineeringfromscratch.com/certification.html?id=claude-ccar-p) — Own the full lifecycle of secure, observable, production-grade Claude systems from discovery to iteration.
- [Study the Decisions, Not the Vocabulary](https://aiengineeringfromscratch.com/lesson.html?path=certifications%2Fclaude%2Flessons%2F00-certification-strategy) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/certifications/claude/lessons/00-certification-strategy/docs/en.md) — A certification blueprint is a map of decisions a competent practitioner can defend. Treat it as a list of terms and you will study the least useful part of the exam.
- [Choose the Smallest Surface That Can Carry the Work](https://aiengineeringfromscratch.com/lesson.html?path=certifications%2Fclaude%2Flessons%2F01-claude-product-and-model-landscape) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/certifications/claude/lessons/01-claude-product-and-model-landscape/docs/en.md) — Product selection is architecture at knowledge-work scale. The wrong surface can make correct output stale, unreviewable, or needlessly expensive.
- [Spend Capability Where Failure Is Expensive](https://aiengineeringfromscratch.com/lesson.html?path=certifications%2Fclaude%2Flessons%2F02-model-selection-and-token-economics) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/certifications/claude/lessons/02-model-selection-and-token-economics/docs/en.md) — Model selection is not a ranking exercise. It is an allocation problem across quality, latency, context, and cost.
- [Turn a Request Into a Testable Contract](https://aiengineeringfromscratch.com/lesson.html?path=certifications%2Fclaude%2Flessons%2F03-prompting-and-task-decomposition) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/certifications/claude/lessons/03-prompting-and-task-decomposition/docs/en.md) — A strong prompt does not merely describe what to write. It makes success observable before generation begins.
- [Put Each Fact in the Right Kind of Context](https://aiengineeringfromscratch.com/lesson.html?path=certifications%2Fclaude%2Flessons%2F04-context-knowledge-memory-and-caching) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/certifications/claude/lessons/04-context-knowledge-memory-and-caching/docs/en.md) — Context is temporary attention. Knowledge is maintained evidence. Memory is continuity. Caching is reuse. Mixing them creates confident stale answers.
- [Validate the Claim, Not the Confidence](https://aiengineeringfromscratch.com/lesson.html?path=certifications%2Fclaude%2Flessons%2F05-output-evaluation-and-validation) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/certifications/claude/lessons/05-output-evaluation-and-validation/docs/en.md) — Fluency is presentation quality. Validation is evidence that the output can safely do its job.
- [Put Authority Around Capability](https://aiengineeringfromscratch.com/lesson.html?path=certifications%2Fclaude%2Flessons%2F06-governance-safety-and-responsible-use) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/certifications/claude/lessons/06-governance-safety-and-responsible-use/docs/en.md) — A model can produce an answer without having permission to see the data, make the decision, or take the action.
- [Design the Handoff Before the Automation](https://aiengineeringfromscratch.com/lesson.html?path=certifications%2Fclaude%2Flessons%2F07-workflow-design-and-human-handoffs) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/certifications/claude/lessons/07-workflow-design-and-human-handoffs/docs/en.md) — A workflow is not complete when Claude finishes. It is complete when the next person can verify, decide, act, and recover.
- [The Messages API Is a State Machine](https://aiengineeringfromscratch.com/lesson.html?path=certifications%2Fclaude%2Flessons%2F08-messages-api-and-application-lifecycle) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/certifications/claude/lessons/08-messages-api-and-application-lifecycle/docs/en.md) — The API does not remember your conversation. Your application does, and one misplaced content block can break the entire loop.
- [Structured Output Is an Untrusted Contract](https://aiengineeringfromscratch.com/lesson.html?path=certifications%2Fclaude%2Flessons%2F09-structured-output-and-defensive-parsing) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/certifications/claude/lessons/09-structured-output-and-defensive-parsing/docs/en.md) — Valid JSON is not a valid business decision. Parse the bytes, validate the shape, verify the meaning, then permit the action.
- [A Tool Loop Is Controlled Delegation](https://aiengineeringfromscratch.com/lesson.html?path=certifications%2Fclaude%2Flessons%2F10-tool-use-and-agentic-loops) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/certifications/claude/lessons/10-tool-use-and-agentic-loops/docs/en.md) — Claude may propose an action. Your application validates the request, grants the capability, observes the result, and decides whether the loop continues.
- [MCP Separates Capability From Host](https://aiengineeringfromscratch.com/lesson.html?path=certifications%2Fclaude%2Flessons%2F11-mcp-server-design-and-integration) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/certifications/claude/lessons/11-mcp-server-design-and-integration/docs/en.md) — Build a narrow, stateless MCP server whose contract can be discovered, cached, invoked, and scaled without hidden connection state.
- [The Agent SDK Is a Harness, Not Permission](https://aiengineeringfromscratch.com/lesson.html?path=certifications%2Fclaude%2Flessons%2F12-claude-agent-sdk-and-hooks) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/certifications/claude/lessons/12-claude-agent-sdk-and-hooks/docs/en.md) — An agent becomes dependable when the loop, tools, context, hooks, and termination policy are explicit enough to inspect and constrain.
- [Security Lives Outside the Prompt](https://aiengineeringfromscratch.com/lesson.html?path=certifications%2Fclaude%2Flessons%2F13-application-security-and-secrets) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/certifications/claude/lessons/13-application-security-and-secrets/docs/en.md) — The model can recommend a safe action. Only deterministic controls can make an unsafe action impossible.
- [Evals Turn Agent Behavior Into Engineering Evidence](https://aiengineeringfromscratch.com/lesson.html?path=certifications%2Fclaude%2Flessons%2F14-evals-testing-debugging-and-observability) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/certifications/claude/lessons/14-evals-testing-debugging-and-observability/docs/en.md) — A trace tells you what happened. An eval tells you whether it was acceptable. A regression gate keeps the next change from quietly making it worse.
- [Repository guide](https://aiengineeringfromscratch.com/lesson.html?path=certifications%2Fclaude%2Flessons%2F15-claude-code-for-development-teams) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/certifications/claude/lessons/15-claude-code-for-development-teams/docs/en.md) — A team does not need one giant prompt. It needs a small project contract, reusable procedures, deterministic checks, and versioned configuration.
- [Multi-Agent Orchestration and Delegation](https://aiengineeringfromscratch.com/lesson.html?path=certifications%2Fclaude%2Flessons%2F16-multi-agent-orchestration-and-delegation) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/certifications/claude/lessons/16-multi-agent-orchestration-and-delegation/docs/en.md) — Delegate a bounded question, not your entire uncertainty.
- [Agent SDK Sessions, Subagents, and Context](https://aiengineeringfromscratch.com/lesson.html?path=certifications%2Fclaude%2Flessons%2F17-agent-sdk-sessions-subagents-and-context) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/certifications/claude/lessons/17-agent-sdk-sessions-subagents-and-context/docs/en.md) — Resume state when continuity helps. Fork context when inherited assumptions become risk.
- [Tool Contracts, Errors, and Progressive Discovery](https://aiengineeringfromscratch.com/lesson.html?path=certifications%2Fclaude%2Flessons%2F18-tool-contracts-errors-and-progressive-discovery) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/certifications/claude/lessons/18-tool-contracts-errors-and-progressive-discovery/docs/en.md) — The model chooses from the interface you describe. Ambiguous tools create ambiguous behavior.
- [Claude Code Memory, Rules, Skills, and CI](https://aiengineeringfromscratch.com/lesson.html?path=certifications%2Fclaude%2Flessons%2F19-claude-code-memory-rules-skills-and-ci) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/certifications/claude/lessons/19-claude-code-memory-rules-skills-and-ci/docs/en.md) — Put stable guidance where its scope is true, and executable constraints where failure is unacceptable.
- [Reliable Extraction, Batch, and Independent Reviewers](https://aiengineeringfromscratch.com/lesson.html?path=certifications%2Fclaude%2Flessons%2F20-reliable-extraction-batch-and-reviewers) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/certifications/claude/lessons/20-reliable-extraction-batch-and-reviewers/docs/en.md) — Valid JSON proves the shape survived. It does not prove the facts did.
- [Make Large Context Observable](https://aiengineeringfromscratch.com/lesson.html?path=certifications%2Fclaude%2Flessons%2F21-long-context-reliability-provenance-and-escalation) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/certifications/claude/lessons/21-long-context-reliability-provenance-and-escalation/docs/en.md) — A large context window can hold more evidence. It cannot tell you which evidence was noticed, current, authoritative, or safe to act on.
- [ADR: Support Resolution Pattern](https://aiengineeringfromscratch.com/lesson.html?path=certifications%2Fclaude%2Flessons%2F22-business-discovery-requirements-and-slas) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/certifications/claude/lessons/22-business-discovery-requirements-and-slas/docs/en.md) — The first architecture decision is deciding what problem you are actually responsible for solving.
- [End-to-End Architecture and Value Tradeoffs](https://aiengineeringfromscratch.com/lesson.html?path=certifications%2Fclaude%2Flessons%2F23-end-to-end-architecture-and-value-tradeoffs) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/certifications/claude/lessons/23-end-to-end-architecture-and-value-tradeoffs/docs/en.md) — Architecture is the art of spending complexity only where it changes the outcome.
- [RAG, Retrieval, and Data Pipelines](https://aiengineeringfromscratch.com/lesson.html?path=certifications%2Fclaude%2Flessons%2F24-rag-retrieval-and-data-pipelines) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/certifications/claude/lessons/24-rag-retrieval-and-data-pipelines/docs/en.md) — A grounded answer is only as trustworthy as the evidence that reached the model.
- [Integration Protocols, Identity, and Least Privilege](https://aiengineeringfromscratch.com/lesson.html?path=certifications%2Fclaude%2Flessons%2F25-integration-protocols-identity-and-least-privilege) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/certifications/claude/lessons/25-integration-protocols-identity-and-least-privilege/docs/en.md) — A tool is not safe because Claude uses it carefully. It is safe when the system refuses unauthorized use.
- [Production Observability, Latency, and Cost](https://aiengineeringfromscratch.com/lesson.html?path=certifications%2Fclaude%2Flessons%2F26-production-observability-latency-and-cost) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/certifications/claude/lessons/26-production-observability-latency-and-cost/docs/en.md) — A green API call can still be a failed task.
- [Enterprise Governance, Compliance, and Human Review](https://aiengineeringfromscratch.com/lesson.html?path=certifications%2Fclaude%2Flessons%2F27-enterprise-governance-compliance-and-hitl) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/certifications/claude/lessons/27-enterprise-governance-compliance-and-hitl/docs/en.md) — Governance is the system that decides who may take which risk with whose data.
- [Stakeholder Communication, ADRs, and Lifecycle Ownership](https://aiengineeringfromscratch.com/lesson.html?path=certifications%2Fclaude%2Flessons%2F28-stakeholder-communication-adrs-and-lifecycle) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/certifications/claude/lessons/28-stakeholder-communication-adrs-and-lifecycle/docs/en.md) — An architecture is not delivered when the diagram is finished. It is delivered when the next owner can operate the decision.
- [Ship a Week of Work, Not a Perfect Prompt](https://aiengineeringfromscratch.com/lesson.html?path=certifications%2Fclaude%2Flessons%2F29-associate-workflow-capstone) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/certifications/claude/lessons/29-associate-workflow-capstone/docs/en.md) — Your capstone is a governed decision workflow: sources in, claims checked, human authority preserved, and state handed off.
- [Ship a Claude Application You Can Defend](https://aiengineeringfromscratch.com/lesson.html?path=certifications%2Fclaude%2Flessons%2F30-developer-application-capstone) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/certifications/claude/lessons/30-developer-application-capstone/docs/en.md) — The capstone is not a chatbot demo. It is a bounded application with a wire contract, security boundary, eval evidence, and recovery plan.
- [Defend One Architecture Across Six Contexts](https://aiengineeringfromscratch.com/lesson.html?path=certifications%2Fclaude%2Flessons%2F31-architect-foundations-scenario-capstone) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/certifications/claude/lessons/31-architect-foundations-scenario-capstone/docs/en.md) — Architecture is the set of boundaries that still hold when the scenario changes, a tool fails, and the evidence is incomplete.
- [Architect Professional System Capstone](https://aiengineeringfromscratch.com/lesson.html?path=certifications%2Fclaude%2Flessons%2F32-architect-professional-system-capstone) · [raw](https://raw.githubusercontent.com/rohitg00/ai-engineering-from-scratch/main/certifications/claude/lessons/32-architect-professional-system-capstone/docs/en.md) — Build the evidence packet that makes a production architecture defensible.