Positional Encoding: Sinusoidal, RoPE, ALiBi

Pick, build and check a position scheme. Sinusoidal, learned, RoPE, YaRN and ALiBi side by side, in Python.

Side by side#

Scheme	Acts on	Extrapolation	Training cost	Used by
Absolute sinusoidal	Embeddings, sum	poor	free	original transformer, early BERT
Learned absolute	Embeddings, lookup	none	tiny	GPT-2, GPT-3
RoPE	Rotates Q and K	good with scaling	free	Llama 2/3/4, Qwen 2/3, Mistral, DeepSeek-V3, Kimi
RoPE + YaRN	Rotates Q and K	excellent	fine-tune stage	Qwen2-1M, Llama 3.1 128K
ALiBi	Attention scores	excellent	free	BLOOM, MPT, Baichuan

def sinusoidal_pe(n, d, base=10000.0):add to embeddings

theta = pos / (base ** (2 * i / d))one frequency per pair i

pe[pos][2 * i] = math.sin(theta)even dim

pe[pos][2 * i + 1] = math.cos(theta)odd dim

def apply_rope(x, pos, base=10000.0):call on Q and on K

c = math.cos(theta)same theta as above

out[2 * i] = a * c - b * s2D rotation of pair i

out[2 * i + 1] = a * s + b * c

def alibi_slopes(n_heads):one slope per head

return [2 ** (-8 * (h + 1) / n_heads) for h in range(n_heads)]

row.append(-m * abs(i - j))bias for head slope m

row.append(float("-inf"))causal and j > i

Task	Code
Sinusoidal table, n positions	`sinusoidal_pe(n, d, base=10000.0)`
Rotate one Q or K vector	`apply_rope(x, pos, base=10000.0)`
ALiBi bias per head	`alibi_bias(n_heads, seq_len, causal=True)`
Learned table, one row per position	`LearnedPositionalEmbedding(max_context_length, d_model)`
Fixed table as a buffer	`SinusoidalPositionalEmbedding(max_context_length, d_model)`
Sum token and position vectors	`EmbeddingComposer(token_emb, learned_pos)`
Neighbour similarity diagnostic	`neighbour_cosine_curve(table, max_offset=8)`
Run the demo	`python3 code/main.py`

base: the RoPE frequency scaler. Increase it to extend context at inference.
NTK-aware: rescale base to base * (scale_factor)^(d/(d-2)) when extending from 4K to 16K+.
YaRN: per-dimension interpolation that preserves attention entropy. Llama 3.1 128K uses it.
LongRoPE: evolutionary search picks per-dimension scale factors. Phi-3-Long uses it.
Position interpolation: shrink positions by the extension factor. Fine-tune for 1–5B tokens.
Result: Llama 3 extended from 8K to 128K context with RoPE scaling.

Command	What it does
`rope_theta`	RoPE base frequency
`max_position_embeddings`	Training context length
`model.config.rope_scaling`	Scaling type and factor, for example yarn
`"position": "rope-yarn"`	Qwen 2.5 72B in the calculator
`"position": "rope-sliding"`	Gemma 2 9B, RoPE plus sliding layers
`"position": "learned"`	GPT-2 Small, 1024 positions
`sliding_window`	Last W tokens for sliding layers
`layer_types`	Which layers are full or sliding

Symptom	Cause	Fix
`seq_len <n> exceeds max_context_length <L>`	Input longer than the table	Raise `max_context_length`
`d_model must be even for sinusoidal`	Odd `d_model`	Use an even `d_model`
`token and positional embeddings must share d_model`	Two widths	Build both with one `d_model`
Output the same for shuffled tokens	No position signal	Add a position scheme
Learned model fails past trained length	The row does not exist	Use RoPE with scaling
Sinusoidal fails beyond `max_len`	No training at those positions	Use RoPE or ALiBi
Same gap, different RoPE scores	Rotation on Q or K only	Rotate both with `apply_rope`
Long context degrades with RoPE	Phase wrap at small base	Raise `base`, try YaRN

Use RoPE for a new decoder model. Every Llama, Mistral, Qwen, DeepSeek, and Gemma uses RoPE.
Use RoPE + YaRN when you extend context and can pay a fine-tune stage.
Use ALiBi when length extrapolation matters most and you want no embeddings.
Use sinusoidal when you want a parameter-free input table. It is historical for frontier models.
Use learned absolute only with a fixed maximum context, as in GPT-2.
Apply RoPE and ALiBi at the attention step, not at the input.

Why does attention need position?: It is order-blind. Shuffled rows of X give shuffled output rows.
Why is RoPE relative?: The dot product picks up a cos of the gap times theta on each pair.
Sum or concatenate?: Sum. It keeps D constant through the network.
Parameter cost?: Learned adds max_context_length * D. Sinusoidal adds zero.
Why did RoPE win?: No architecture change, relative position, and a base knob for long context.
Learned vs sinusoidal at init?: Learned neighbours look random. Sinusoidal decays smoothly.