Cheat sheets

Edge Inference Targets

Pick an edge target and a quantization format. Bandwidth ceilings, formats, export steps and on-device measurement.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Comparisons
  • Troubleshooting
  • Questions and tips

Source

Four edge targets side by side#

FactorApple ANEQualcomm HexagonWebGPU / WebLLMNVIDIA Jetson
Peak38 TOPS, M4 / A1845 TOPS, X Elite / 8 Gen 4Browser GPU, no installOrin Nano Super, AGX Orin, Thor
MemoryUnified, no copySeparate memory domainBrowser penalty ~25%Memory-bound on Nano
AccessCore ML or MPSQNN SDK, AI HubWebGPU compute shadersvLLM, TensorRT Edge-LLM
FormatINT4 weights + FP16 activationsQNN INT8 / INT4Q4 MLC (q4f16_1)Q4 GGUF, TRT-LLM INT4, NVFP4
Observed8 tok/s estimate, A18, 8B Q47 tok/s, 8 Gen 3, 8B Q4~41 tok/s, M3 Max, 8B Q4~40 tok/s, AGX Orin, gpt-oss-20b
LimitNative ANE needs Core MLConvert from PyTorch or ONNX~70-75% mobile coverageNano fits 3B models

Use each target when#

Apple ANE
an iOS app. Convert with Core ML. INT4 weights plus FP16 activations.
Qualcomm Hexagon
Snapdragon phones and Windows ARM laptops. Convert with QNN or AI Hub.
WebGPU / WebLLM
a browser app with no install. Keep a server fallback on the same OpenAI-compatible API.
Jetson Orin Nano Super
Llama 3.2 3B or Phi-3 on 8GB.
Jetson AGX / Thor
larger models, NVFP4 plus FP8 KV, EAGLE-3.
GGUF Q4_K_M
CPU and Apple Silicon laptops.
GPTQ or AWQ INT4
GPU inference with CUDA or ROCm.

Decode ceiling#

def ceiling(target: Target, model_gb: float) -> float:
seconds_per_token = model_gb / target.bandwidth_gb_sall weights per token
return 1 / seconds_per_tokentok/s upper bound
model_gb = 4.7Llama 3.1 8B Q4
Target("Apple A18 (iPhone 16)", 60, 8, "phone with ANE")GB/s, observed tok/s
Target("Snapdragon 8 Gen 3", 77, 7, "mid/high Android")
Target("WebGPU on M3 Max", 400, 41, "browser penalty ~25%")
Target("Jetson AGX Orin", 205, 45, "edge-datacenter bridge")

Bandwidth math#

Rule
decode reads the full set of weights for every token.
7B Q4
3.5 GB. At 50 GB/s one token takes 70 ms, ~14 tok/s.
High-end mobile
90 GB/s gives ~25 tok/s.
HBM3 at 3 TB/s
1.2 ms per token, ceiling 830 tok/s.
Gap
30-50x between datacenter and mobile DRAM.
Long context
4 GB model plus 2 GB KV at 32K on an 8 GB phone is OOM.
Context on edge
keep 4K-8K unless you accept Q4 KV.

Quantization formats, Llama 3 70B#

FactorFP16FP8AWQ INT4GGUF Q4_K_M
Size140GB70GB35GB40GB
Perplexity3.123.144.184.25
MMLU79.5%79.3%78.1%77.9%
Tokens/sec38, A10055, A10075, A10028, CPU
Best forBaselineH100 deploysGPU inferenceCPU, Metal

Quantize and dequantize#

scale = max(abs(tensor)) / 127symmetric INT8
quantized = clamp(round(tensor / scale), -128, 127)
reconstructed = quantized * scaleerror at most scale / 2
quantized = round(tensor / scale) + zero_pointasymmetric, for ReLU outputs

Tasks#

TaskCode
Edge decode ceilings, lesson 12code/main.py
Measure p50, p95, p99 on the devicemeasure_latency(model, input_shape, device="cpu", warmup=10, iters=50)
Time GPU work correctlytorch.cuda.synchronize() before and after the timed block
Static INT8 PTQ configmodel.qconfig = tq.get_default_qconfig(backend)
Export to ONNXtorch.onnx.export(model, sample_input, path, opset_version=17)
GPTQ INT4GPTQConfig(bits=4, group_size=128)
GGUF Q4_K_Mllama.cpp/build/bin/llama-quantize llama-8b-f16.gguf llama-8b-q4km.gguf Q4_K_M
WebLLM weightsmlc_llm convert_weight plus a compiled .wasm

Vision deployment path#

1Pick a backboneunder 3M params, MobileNetV3-Small. 3-10M,EfficientNet-Lite-B0.2Measurewarm up 5-10 passes. Fix the input size toproduction resolution.3Quantisepost-training static INT8. Verify accuracy lossunder 1%.4ExportPyTorch to ONNX. dynamic_axes allows any batchsize.5RuntimeTensorRT on Jetson. Core ML on iOS. TFLite onAndroid.6Decidefrom a table of model, latency, memory andaccuracy measured on the device.

Troubleshooting#

SymptomCauseFix
8B Q4 at 3 tok/s on iPhone 16 ProMobile bandwidth gapSmaller model or fewer bits
OOM at long context on a phoneKV cache for 32K is 2 GBKeep context at 4K-8K or use Q4 KV
WebLLM rejects a GGUF fileGGUF is not supportedConvert to Q4 MLC (q4f16_1)
llama.cpp on iOS skips the ANEMetal backend uses MPSConvert with Core ML
No WebGPU on an old Android browserNeeds Chrome v121+Fall back to the server API
First latency numbers too highCold caches and JITWarm up with 5-10 dummy passes
Naive INT4 output is garbageRounding errors accumulateGPTQ or AWQ with calibration data
Perplexity delta above 2.0Something went wrongRecheck. Under 0.5 is excellent

Format glossary#

BF16
8 exponent bits, 7 mantissa bits. Same range as FP32.
FP8 E4M3
weights and activations for inference.
FP8 E5M2
gradients in training. More range.
INT8
256 values from -128 to 127. Needs a scale factor.
INT4
16 levels. Needs GPTQ or AWQ for quality.
GGUF
llama.cpp file with mixed precision layers.
PTQ
quantize after training. 128-1024 calibration examples.
QAT
fake quantization in training. Better at INT4 and INT2.

Q&A#

Why is compute secondary on edge?
Decode is memory-bound. Bandwidth sets the ceiling.
How fast is WebLLM against native?
About 70-80% of native on the same backend.
FLOPs or latency?
FLOPs for architecture search. On-device latency for deployment.
PTQ or QAT for vision?
PTQ static gives 95% of the benefit with 5% of the effort.
Is FP8 safe?
Yes on H100. 30-50% faster with under 0.1% quality loss.