Cheat sheets

GPUs, Cloud, Docker and API Keys

Check the GPU, size a model for VRAM, run CUDA in Docker, store API keys and fix API errors.

  • Tasks and commands
  • Annotated code
  • Diagrams
  • Defaults and numbers
  • Troubleshooting
  • Questions and tips

Source

Where to run#

SettingDefaultNote
Local NVIDIA GPU$0Install CUDA + cuDNN. Regular use, large datasets
Google Colab (free tier)$0Runtime > Change runtime type > T4 GPU
Cloud GPU (Lambda, RunPod, Vast.ai)$0.20-2.00/hrSSH + install. Serious training
CPUphases 1-3Most lessons run fine on CPU
fp16 rule of thumb2 bytes per parameterKeep a margin for activations

Check the GPU#

torch.cuda.is_available()False means no GPU
torch.version.cudaPyTorch CUDA version
torch.cuda.get_device_name(0)GPU name
props = torch.cuda.get_device_properties(0)
vram_gb = props.total_memory / 1e9VRAM in GB
params_fp16 = vram_gb * 1e9 / 2max parameters in fp16
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
torch.cuda.synchronize()before and after a GPU timing

Store and use an API key#

1Shellexport ANTHROPIC_API_KEY="sk-ant-...". Never putkeys in code.2.env fileANTHROPIC_API_KEY=sk-ant-.... Add .env to.gitignore.3SDKanthropic.Anthropic() reads the key from theenvironment.4Requestendpoint, key, and a body with model,max_tokens, messages.5ResponseJSON with content, usage and stop_reason.

First call with the SDK#

client = anthropic.Anthropic()
MODEL = os.environ.get("LLM_MODEL", "claude-sonnet-5")model id
response = client.messages.create(
model=MODEL,
max_tokens=256,
messages=[{"role": "user", "content": "<prompt>"}]
)
print(response.content[0].text)reply text
response.usage.input_tokens, response.usage.output_tokensbilled tokens

Raw HTTP call#

SettingDefaultNote
Endpointhttps://api.anthropic.com/v1/messagesmethod="POST"
x-api-keyANTHROPIC_API_KEYKey header
anthropic-version2023-06-01Version header
LLM_MODELclaude-sonnet-5Un-dated Sonnet alias
max_tokens256Lesson value
MOCK"1"TypeScript port then skips the network

Build and run the dev image#

docker build -t ai-dev -f phases/00-setup-and-tooling/07-docker-for-ai/code/Dockerfile .
docker run --rm -it --gpus all \
-v $(pwd):/workspace \your code
-v ~/models:/models \models survive rebuilds
-p 8888:8888 \Jupyter port
ai-dev jupyter notebook --ip=0.0.0.0 --port=8888 --no-browser --allow-root

Docker tasks#

TaskCode
Start the stackdocker compose up -d
Stop and delete the volumedocker compose down -v
Shell in a servicedocker compose exec ai-dev bash
Image sizesdocker images
Reclaim diskdocker system prune -a
GPU in a containerdocker exec -it <container_id> nvidia-smi
Follow logsdocker logs -f <container_id>

API errors#

SymptomCauseFix
401 UnauthorizedKey wrong or missingecho $ANTHROPIC_API_KEY | head -c 10
403 ForbiddenNo permission for endpoint or modelCheck the key permissions
429 Too Many RequestsRate limitedWait and retry, or reduce frequency
400 Bad RequestMalformed bodyCheck fields, model name, message format
500/502/503Server-side issueWait a minute and retry
TimeoutRequest took too longReduce max_tokens or use streaming
Connection refusedWrong base URL or networkcurl -I https://api.anthropic.com

Install Docker and GPU access#

CommandWhat it does
curl -fsSL https://get.docker.com | shInstall Docker on Ubuntu
sudo usermod -aG docker $USERDocker without sudo. Log out and back in
docker run hello-worldVerify the install
sudo apt-get install -y nvidia-container-toolkitGPU access for containers (Linux)
sudo nvidia-ctk runtime configure --runtime=dockerRegister the runtime, then sudo systemctl restart docker
docker run --rm --gpus all nvidia/cuda:12.4.1-base-ubuntu22.04 nvidia-smiTest the GPU in a container

Base images#

SettingDefaultNote
nvidia/cuda:12.4.1-devel-ubuntu22.04~4 GBnvcc for flash-attn, bitsandbytes
nvidia/cuda:12.4.1-runtime-ubuntu22.04~1.5 GBPre-built code. No compilers
pytorch/pytorch:2.6.0-cuda12.4-cudnn9-runtime~6 GBPyTorch already installed
python:3.12-slim~150 MBNo CUDA. CPU only

Compose stack#

volumeshttp://qdrant:6333volumestoreHost folders~/models, ~/datasets,repogpuai-devJupyter on 8888, nvidiaGPUstoreqdrantqdrant/qdrant:v1.12.5storeqdrant_data/qdrant/storagevolumeshttp://qdrant:6333volumestoreHost folders~/models, ~/datasets,repogpuai-devJupyter on 8888,nvidia GPUstoreqdrantqdrant/qdrant:v1.12.5storeqdrant_data/qdrant/storage

Docker fixes#

SymptomCauseFix
No matching distribution found for torch==2.6.0+cu124Mac picks the arm64 imageFROM --platform=linux/amd64
No GPU on a Mac or CPU boxNo CUDA thereDrop --gpus all and the NVIDIA deploy block
14 GB model gone after stopNo volume mount-v ~/models:/models
Server not reachable from hostBound to localhosthost="0.0.0.0" and -p 5000:5000

Tips#

  • Time the second GPU run. The first run includes start-up cost.
  • Run !nvidia-smi in Colab before you trust a benchmark.
  • Set up each API key when a lesson needs it.
  • Read the raw HTTP call when an SDK call fails.
  • Add new packages to the Dockerfile and rebuild. Unchanged layers come from cache.