Managed Platforms and Self-Hosting Compared

Pick a hyperscaler, an inference platform or a self-hosted engine. Price it, size it, cut cold starts.

Three hyperscalers side by side#

Factor	Bedrock	Azure OpenAI	Vertex AI
Strategy	Model marketplace	Exclusive OpenAI models	Gemini first
Catalog	Claude, Llama, Titan, Mistral, Cohere	GPT, o-series, DALL·E, Whisper	Gemini plus Model Garden
Median TTFT	~75 ms on-demand	~50 ms with PTUs	60 ms in `main.py`
Dedicated capacity	Provisioned Throughput, $21-$50/hr	PTU, up to 70% savings	Per Gemini SKU
Break-even	About half peak utilization	~40-60% sustained utilization	Less publicly advertised
Cost attribution	Application Inference Profiles	Scopes, tags, PTU reservations	Project per team, BigQuery export
Compliance	BAAs, VPC endpoints, guardrails	HIPAA, SOC 2, ISO 27001	HIPAA, GDPR, residency per region

Factor	Fireworks	Together	Baseten	Modal
Optimizes	Latency	Catalog breadth	Enterprise polish	Python-native DX
Billing	Per token	Per token	Per minute	Per second
Rate in `main.py`	$0.90/M output	$0.88/M output	$0.55/min	$0.48/min
Reserved floor	None	None	1440 min/day	60 min/day
Cold start in `main.py`	1.5 s	2.0 s	5.0 s	2.5 s
Key fact	Batch tier ~50% of serverless	200+ models	Truss, SOC 2 Type II	`@modal.function(gpu="A100")`

Bedrock: you need Claude, Llama and others behind one API and one IAM surface.
Bedrock: you need cost per team, product or feature with no post-processing.
Azure OpenAI: you need GPT models and TTFT P99 under 100 ms on PTUs.
Vertex AI: you need 1M-token Gemini context or multimodal input.
Two providers: every product-critical call runs on two hyperscalers.
Common pair: Claude from Bedrock, GPT from Azure OpenAI, one gateway.

Custom silicon: latency is critical. Groq is the pick for voice agents.
Fireworks: you need low latency and LoRA at the base model rate.
Together: you need new open models within days of release.
Baseten: you need a fine-tuned model, dedicated GPUs, SOC 2 and HIPAA.
Modal: your team writes Python and wants per-second billing.
Replicate: you serve image, video and audio models per prediction.
Anyscale: inference is one node in a larger Ray graph.
Self-host: data residency, custom fine-tune, cost at scale or no hosted model.

Phase	Raw	Pre-seeded image	Model streamer	GPU snapshot
node provision	50.0	50.0	50.0	0.5
image pull	180.0	0.0	180.0	0.0
weights to HBM	75.0	75.0	35.0	0.0
engine init	20.0	20.0	20.0	2.0
first forward	3.0	3.0	3.0	0.5
Tool	none	Bottlerocket, `EC2NodeClass`	Run:ai Model Streamer	Modal checkpoints
Gain	328s total	2-4 minutes saved	~2x weight-load speedup	10x faster restart

Task	Code
Cost and TTFT per hyperscaler	`simulate(tokens_in_per_day, tokens_out_per_day, sla_ttft_ms, use_ptu)`
Azure PTU break-even sweep	`break_even_demo()`
Cost of the two-provider minimum	`lock_in_cost()`
Cost per day for one vendor	`cost_per_day(v, tokens_per_day, predictions_per_day)`
Per-token vs per-minute sweep	`utilization_breakeven()`
Cold start for a mitigation stack	`total_for_stack({"pre_seeded", "streamer"})`
Warm pool break-even	`warm_pool_break_even(gpu_hourly=4.50, cold_seconds=328, sla_tolerated_drops_per_day=5)`
Engine for hardware, scale, workload	`pick_engine(hardware, scale, workload)`

Symptom	Cause	Fix
P99 TTFT misses 100 ms on-demand	Shared capacity competes with other traffic	Buy PTUs or Provisioned Throughput
PTU bill high at low traffic	You pay for the reservation when idle	Reserve only at ~40-60% sustained utilization
Bedrock bill up 4x, same traffic	No per-feature attribution	Route calls through Application Inference Profiles
No per-request cost on Azure	Tags come from resource groups	Application Insights custom metrics or gateway headers
Product down in a provider outage	Single-vendor lock-in	Two-provider minimum behind one gateway
First token takes 3-8 minutes	Raw 70B cold start after scale to zero	Stack pre-seeded image, streamer, snapshot
Snapshot restore fails on new node	Snapshots are per GPU topology	Re-checkpoint on the new SKU
New project picks TGI	Maintenance mode since Dec 11, 2025	Default to vLLM or SGLang

Why is Bedrock ~25 ms slower?: Shared on-demand capacity. Azure PTUs reserve GPUs for your tenant.
When does per-minute beat per-token?: The docs say above ~30% sustained utilization. main.py says ~60-70% under reserved-minute billing.
When is a warm pool mandatory?: Typically at TTFT P99 < 60s on a 70B+ model.
What does a warm pool cost?: Five products with one warm replica use 3,600 GPU-hours/month.
Why does live migration move tokens?: Input is KB. KV cache is GB. Recompute is cheaper.
Why not Ollama in production?: 15-30% slower than llama.cpp, 3x gap under prod load.

PTU: Provisioned Throughput Unit. Reserved inference GPUs, priced per hour.
BAA: Business Associate Agreement. Required for PHI. All three provide it.
Application Inference Profile: tagged Bedrock profile with cost per profile in CloudWatch.
Truss: Baseten manifest for dependencies, secrets and serving config.
FireAttention: Fireworks custom attention kernel.
RayTurbo: Anyscale proprietary inference engine on Ray.
min_workers: serverless keep-alive count. min_workers=1 is a warm pool.
GGUF: llama.cpp weight format with bundled K-quant variants.