Practice

1. Power iteration for the top eigenpair

   The top eigenvalue and eigenvector of a symmetric matrix by power iteration, with a tolerance, an iteration limit and a sign rule.

   eigenvaluespower-iterationlinear-algebrasvd

   ImplementMediumPhase 0125 min
2. Vectorized pairwise distances

   Euclidean distance and cosine similarity matrices from one matrix product, with a clip at 0 and a zero-vector rule.

   distancesvectorizationcosine-similaritynumerical-stability

   ImplementMediumPhase 0120 min
3. Linear regression by gradient descent

   Mean squared error, its gradients for weights and bias, and a batch gradient descent loop that records the cost after each update.

   linear-regressiongradient-descentloss-functionsgradients

   ImplementEasyPhase 0220 min
4. Train softmax regression on four blobs

   Train multinomial logistic regression with gradient descent to reach a held-out accuracy and log loss target on four Gaussian blobs.

   softmaxlogistic-regressiongradient-descentlog-loss

   TrainEasyPhase 0225 min
5. Logistic regression from scratch

   A stable sigmoid, the binary cross-entropy loss and its gradient, and a gradient descent loop that records the loss.

   logistic-regressionclassificationgradientsnumerical-stability

   ImplementMediumPhase 0225 min
6. Gini impurity and the best split

   Gini impurity and the best decision tree split over all features, with midpoint thresholds, weighted child impurity and a fixed tie rule.

   decision-treesgini-impurityclassificationsearch

   ImplementHardPhase 0235 min
7. K-means assignment and update

   The k-means assignment and update steps with a tie rule and an empty cluster rule, and a full run until the assignments stop changing.

   clusteringk-meansunsupervised-learningvectorization

   ImplementMediumPhase 0225 min
8. Measure bias and variance

   Fit polynomials of degree 1 to 12 on resampled training sets and measure bias squared, variance and test error at each degree.

   bias-varianceoverfittingmodel-selection

   LabMediumPhase 0230 min
9. Train a 2-4-1 network on XOR

   Train a small sigmoid network with full-batch gradient descent until it labels noisy XOR points with 95 percent held-out accuracy.

   mlpbackpropagationgradient-descentxor

   TrainEasyPhase 0330 min
10. Backpropagation through a two-layer MLP

    Forward pass with a cache and a backward pass that returns the gradient of every weight and bias of a two-layer sigmoid network.

    backpropagationgradientsmlpchain-rule

    ImplementHardPhase 0340 min
11. Fit a sine curve with a tanh network

    Train a 1-16-1 tanh network on noisy samples of sin(x) until its held-out mean squared error against the true curve is below 0.01.

    regressionmlptanhmse

    TrainMediumPhase 0335 min
12. Focal loss

    Binary and multi-class focal loss with gamma and alpha, from logits and without overflow, plus the binary gradient.

    loss-functionsfocal-lossclass-imbalancegradients

    ResearchEasyPhase 0320 min
13. Stable softmax and cross-entropy

    Batched softmax, mean cross-entropy and its gradient, without overflow for large logits.

    softmaxloss-functionsgradientsnumerical-stability

    ImplementEasyPhase 0320 min
14. SGD with momentum

    One momentum update with the lesson's velocity rule, and a loop that minimizes a quadratic with or without Nesterov lookahead.

    optimizersmomentumgradient-descent

    ImplementEasyPhase 0320 min
15. Adam and AdamW step

    One Adam update with bias correction and a step counter that starts at 1, plus AdamW decoupled weight decay, as a pure function of the state.

    optimizersadamweight-decay

    ImplementMediumPhase 0325 min
16. Inverted dropout

    Inverted dropout with a given random generator: training forward with an exact mask, backward with the same mask, identity at inference.

    dropoutregularizationgradients

    ImplementEasyPhase 0315 min
17. Batch normalization forward and backward

    Training and inference forward passes with running statistics, and the full backward pass for x, gamma and beta.

    batch-normnormalizationgradientsbackpropagation

    ImplementHardPhase 0345 min
18. Sweep the learning rate

    Sweep gradient descent learning rates on a seeded regression, find the best rate and the first rate that diverges, and compare with 2 over the top eigenvalue.

    learning-rategradient-descentoptimization

    LabEasyPhase 0320 min
19. Train a tanh network on two spirals

    Train a 2-24-24-1 tanh network with mini-batch Adam until it separates two noisy spirals with 95 percent held-out accuracy.

    mlpbackpropagationadammini-batch

    TrainMediumPhase 0345 min
20. CLIP symmetric contrastive loss

    Normalize image and text embeddings, build the temperature-scaled similarity matrix, and compute the symmetric InfoNCE loss and its gradient.

    clipcontrastive-learningloss-functionsgradients

    ResearchMediumPhase 0430 min
21. Beam search with EOS and length normalization

    Keep the best beam_width partial sequences over a step function, finish beams at EOS, and rank them with a length penalty.

    beam-searchdecodingseq2seqllm-inference

    ImplementHardPhase 0545 min
22. BM25 ranking

    Score tokenized documents against a query with BM25: the lesson idf, term frequency saturation with k1, length normalization with b, and a ranked list.

    bm25information-retrievalsparse-retrievalranking

    ImplementMediumPhase 0525 min
23. Train a character bigram language model

    Train a table of next-character logits with gradient descent until the held-out negative log-likelihood beats a unigram model.

    language-modelbigramcross-entropygradient-descent

    TrainMediumPhase 0540 min
24. Scaled dot-product attention with a mask

    Batched softmax(QK^T / sqrt(d_k)) V with an optional boolean mask, plus the causal mask that a decoder uses.

    attentiontransformersmaskingsoftmax

    ImplementMediumPhase 0725 min
25. Grouped-query attention

    GQA forward with H query heads and G shared key and value groups, plus mean pooling of a multi-head checkpoint into groups.

    attentiongqatransformerskv-cache

    ResearchMediumPhase 0730 min
26. Split heads, merge heads, multi-head attention

    Reshape (B, T, D) into H heads and back, then run batched multi-head self-attention with given projection matrices.

    attentionmulti-head-attentiontransformersreshape

    ImplementMediumPhase 0730 min
27. Sinusoidal positional encoding

    Build the sin and cos position table with interleaved columns and base 10000, and add it to a batch of token embeddings.

    positional-encodingtransformersembeddings

    ImplementEasyPhase 0715 min
28. Rotary position embedding

    Rotate adjacent coordinate pairs of queries and keys by position times base^(-2i/d), so attention scores depend only on the position gap.

    positional-encodingropetransformersattention

    ImplementMediumPhase 0725 min
29. LayerNorm and RMSNorm

    LayerNorm with gamma and beta and RMSNorm with a weight, both over the last axis, with eps inside the square root.

    layer-normrms-normnormalizationtransformers

    ImplementEasyPhase 0715 min
30. Sweep the sampling temperature

    Sweep the temperature on fixed logits, measure entropy and top 1 probability, find where entropy crosses 2 bits, and count the tokens top p keeps.

    samplingtemperatureentropytop-p

    LabEasyPhase 0720 min
31. Temperature, top-k and top-p sampling

    Scale logits by temperature, keep the top k or the top p mass, renormalize, and draw one token with a seeded rng.

    samplingdecodingsoftmaxllm-inference

    ImplementMediumPhase 0730 min
32. Top-k MoE gating and the Switch load-balancing loss

    Route tokens to the top k experts with renormalized softmax gates, and compute the Switch Transformer auxiliary loss E times sum f_i P_i.

    mixture-of-expertsroutingload-balancingtransformers

    ResearchMediumPhase 0730 min
33. Fixed-capacity KV cache

    Append keys and values to a preallocated cache, attend one query over the valid prefix, and reject an append past the capacity.

    kv-cacheattentioninferencetransformers

    ImplementMediumPhase 0725 min
34. Tiled attention with an online softmax

    FlashAttention in numpy: a running max and a running sum over key blocks give exact attention for any block size.

    attentionflash-attentionsoftmaxnumerical-stability

    ResearchHardPhase 0740 min
35. Speculative decoding: accept, reject and resample

    The min(1, p/q) acceptance rule and the residual distribution that keep speculative decoding exactly equal to the target model.

    speculative-decodingsamplinginferenceprobability

    ResearchHardPhase 0735 min
36. DDPM noise schedule, closed form noising and posterior

    Build the linear beta schedule and alpha bars, sample x_t from x_0 in closed form, and compute the mean and variance of q(x_{t-1} | x_t, x_0).

    diffusionddpmgenerative-modelsnoise-schedule

    ResearchMediumPhase 0830 min
37. Rectified flow loss and Euler sampler

    Interpolate along straight paths, regress the velocity x1 - x0 with the flow matching loss, and integrate a velocity field with Euler steps.

    rectified-flowflow-matchinggenerative-modelsode-solvers

    ResearchMediumPhase 0825 min
38. Q-learning and SARSA updates

    One tabular Q-learning or SARSA update with no bootstrap at a terminal step, and epsilon-greedy actions from a seeded rng with lowest-index ties.

    reinforcement-learningq-learningsarsaexploration

    ImplementEasyPhase 0915 min
39. Discounted returns with episode ends

    Reward-to-go returns for one episode and for a batch of joined episodes with done flags, plus return normalization.

    reinforcement-learningreturnspolicy-gradients

    ImplementEasyPhase 0915 min
40. Generalized advantage estimation

    TD errors and GAE advantages with gamma, lambda, done flags and a bootstrap value, plus the critic targets.

    reinforcement-learningadvantagesactor-criticppo

    ImplementMediumPhase 0925 min
41. PPO clipped surrogate loss

    The PPO loss to minimize from new and old log probabilities and advantages, the clip fraction, and the gradient with respect to the new log probabilities.

    reinforcement-learningppopolicy-gradientgradients

    ImplementMediumPhase 0925 min
42. GRPO group advantages and clipped objective

    Normalize rewards within each group, estimate the KL with k3, and compute the clipped GRPO objective averaged per answer over valid tokens.

    grporeinforcement-learningpolicy-gradientkl-divergence

    ResearchMediumPhase 0930 min
43. DPO loss and implicit rewards

    The direct preference optimization loss, its implicit rewards and its gradient, from policy and reference log probabilities.

    dpoalignmentloss-functionsgradients

    ResearchMediumPhase 1025 min
44. Symmetric int8 quantization

    Absmax int8 quantization with one scale per tensor and one scale per row, with dequantization and a zero-row rule.

    quantizationint8inference

    ImplementMediumPhase 1020 min
45. Sweep the quantization bit width

    Sweep symmetric absmax quantization from 2 to 8 bits on weights with outliers, compare one scale per tensor with one scale per row.

    quantizationinferenceoutliers

    LabMediumPhase 1030 min
46. Cosine similarity top-k search

    Normalize embedding rows, score queries against documents with cosine similarity, and return the top k documents with a fixed tie rule.

    embeddingscosine-similarityvector-searchretrieval

    ImplementEasyPhase 1115 min
47. LoRA linear layer

    Forward pass of a linear layer with a low rank adapter, merge of the adapter into the weight, and the trainable parameter count.

    lorafine-tuninglinear-algebra

    ImplementEasyPhase 1115 min
48. Read an SGLang sweep on an L4

    Use real SGLang runs on an NVIDIA L4 to find the saturation point, the SLO limit, the FP8 gain and the effect of a smaller prefill chunk.

    inferenceservinggoodputbenchmarking

    LabMediumPhase 1730 min
49. Parse and validate tool calls

    Parse tool calls from a raw model reply and validate their arguments against a JSON Schema subset with JSON pointer error paths.

    tool-callingjson-schemaparsingagents

    ImplementHardPhase 1950 min
50. BPE pair counts and merges

    Count weighted adjacent pairs, pick the best pair with a fixed tie rule, merge it without overlap and train a BPE merge list.

    tokenizationbpesubword

    ImplementMediumPhase 1930 min
51. Reciprocal rank fusion

    Merge several ranked lists of document ids into one ranking with reciprocal rank fusion, optional list weights and a fixed tie rule.

    rank-fusionhybrid-retrievalrankingretrieval

    ImplementEasyPhase 1912 min

No problem matches these filters.

1. Math foundationsMatrix operations, gradients and optimization written with numpy.0 / 3 solved
2. Classical MLThe models that come before deep learning, written from the equations.0 / 6 solved
3. Deep learning coreLosses, backpropagation, normalization and optimizers.0 / 13 solved
4. OptimizersGradient descent, momentum and Adam, one update rule at a time.0 / 4 solved
5. Transformers from scratchAttention, heads, positions and normalization: the parts of a transformer block.0 / 9 solved
6. LLM inferenceTokenizers, sampling, beam search, the KV cache and LoRA.0 / 11 solved
7. RL basicsReturns, advantages and the PPO objective.0 / 6 solved
8. RAG building blocksDense and sparse retrieval scores that a RAG system ranks with.0 / 3 solved
9. Agent building blocksThe parsing and validation code around an agent loop.0 / 1 solved
10. Papers to codeImplement the core method of a research paper and test it against the paper's equations.0 / 10 solved
11. Train tiny modelsTrain a small model in your browser to a target score inside a step budget.0 / 5 solved
12. Experiment labsChange one variable, measure the effect, and answer with numbers.0 / 5 solved