Attention Variants Compared

Full, sliding window, strided, BigBird, differential and native sparse attention. Cost, KV cache, code, fixes.

Variants side by side#

Variant	Compute	KV cache	Quality vs full
Full attention	`O(N^2)`	`O(N)` per layer	baseline
SWA, window 1024	N times W	`O(W)` per layer	-0.1 ppl, good with global layers
Local + strided sparse	N times sqrt N	mixed	similar to SWA
BigBird, local + global + random	`O(N)` approx	mixed	matches full at 2× context
Native Sparse (DeepSeek-V3.2)	N times active fraction	`O(N)`	within 0.05 ppl
Differential	2 times N squared	`O(2N)`	-5 to -10% ppl

Full attention every layer up to ~16K context, or when retrieval quality is paramount.
SWA + global mix long context above 32K. The 2026 default above 32K.
Sparse block attention custom kernel, custom pattern. For retrieval and audio.
Differential attention attention sinks hurt, as in long-context RAG and needle-in-haystack.
DIFF from the start a new model that targets 64k-plus effective context.
NSA pre-training or continued training that targets 64k-plus context.
Not DIFF or NSA a pre-trained dense model, or context under 16k.

def swa_mask(n, window):

M = [[NEG_INF] * n for _ in range(n)]

for i in range(n):

lo = max(0, i - window + 1)window >= n gives full causal

for j in range(lo, i + 1):

M[i][j] = 0.00.0 attends, NEG_INF masks

return M

def strided_mask(n, window, stride):local window, then:

for j in range(0, i + 1, stride):

M[i][j] = 0.0every stride-th token back to 0

render(swa_mask(8, window=4), "sliding window (W=4)")

render(strided_mask(8, window=2, stride=3), "local (W=2) + strided (stride=3)")

from torch.nn.attention.flex_attention import flex_attention, create_block_mask

def swa_pattern(b, h, q_idx, kv_idx):

return (q_idx - kv_idx < 1024) & (q_idx >= kv_idx)window 1024, causal

mask = create_block_mask(swa_pattern, B=batch, H=heads, Q_LEN=n, KV_LEN=n)

out = flex_attention(q, k, v, block_mask=mask)compiles to a Triton kernel

Does Flash Attention cut compute?: No. It hides the activation memory. Every token still attends to every token.
Why is NSA trainable?: Gradients flow through the compressed-branch scores. top_k only picks blocks to load.
Why does DIFF help more at 128k?: The noise floor grows with context. At 4k it is small.
Is DIFF V2 GQA compatible?: Yes. V2 adds Q heads, not KV heads.
NSA keys at 1024 tokens, l 32, k 4, w 128?: 32 + 128 + 128 = 288 keys, 3.5x fewer.

SWA: sliding window attention. Each query attends to its last W tokens.
Receptive field: an L-layer SWA stack with window W sees up to L × W tokens.
Attention sink: rows sum to 1, so uninformative queries put weight on position 0.
NSA: Native Sparse Attention. Learned block sparsity, zero blocks skipped in the kernel.
DSA: Deepseek Sparse Attention, the successor of NSA.
HBM: High Bandwidth Memory. About 3 TB/s on H100.
SRAM: per-SM on-chip memory. About 30 TB/s.

Task	Code
SWA causal mask	`swa_mask(n, window)`
Local plus strided mask	`strided_mask(n, window, stride)`
Count attended cells	`count_nonmasked(M)`
KV cache bytes	`kv_cache_bytes(N, n_layers, n_heads_kv, d_head, dtype=2)`
Flash style softmax	`tiled_softmax_dot(q, Ks, Vs, tile=4)`
Signal to noise of a row	`snr(weights_row, signal_pos)`
DIFF V1 and V2 params	`compute_param_diff(hidden=4096, n_heads=32, kv_heads=8)`
NSA keys per query	`count_nsa(N, l, k, W)`

Symptom	Cause	Fix
SWA model fails long-range retrieval	No global layers	Interleave full layers, 5:1
Weight bleeds to token 0	Attention sink from softmax	Differential attention
Lost in the middle past 32k	Softmax noise floor grows with N	DIFF from the start, or LoRA on Q
DIFF V1 decode slower than baseline	Value cache loaded twice	DIFF V2
Late training unstable at 70B-plus	V1 per-head RMSNorm	V2 removes it
NSA on a dense pre-trained model	No retrofit	Continued training
No NSA gain under 16k	Three-branch overhead	Full attention
Inference-time sparsity gives small speedup	Model never routed through it	Natively trained sparsity