Cheat sheets

Git, Shell and Linux for AI Work

Commit and branch, pipe and filter logs, keep runs alive in tmux, move files to a GPU box and fix Linux errors.

  • Concepts and terms
  • Tasks and commands
  • Annotated code
  • Diagrams
  • Troubleshooting
  • Questions and tips

Source

Where git moves your work#

git addgit commitgit pushgit fetchgit pullclientWorkingDirectoryyour filesqueueStaging Areanext commitstoreLocal RepocommitsstoreRemote (GitHub)your forkgit addgit commitgit pushgit fetchgit pullclientWorking Directoryyour filesqueueStaging Areanext commitstoreLocal RepocommitsstoreRemote (GitHub)your fork

Git tasks#

TaskCode
Set your identitygit config --global user.name "Your Name"
Save workgit add file.py then git commit -m "<message>"
Back it upgit push origin main
Branch for an experimentgit checkout -b experiment/new-optimizer
Merge it backgit checkout main then git merge experiment/new-optimizer
Work on your forkgit checkout -b my-progress then git push -u origin my-progress
History of one phasegit log --oneline -- phases/00-setup-and-tooling
Files of one commitgit show --stat <hash>

Pipes and redirects#

CommandWhat it does
>Write stdout to a file (overwrite)
>>Append stdout to a file
2>Write stderr to a file
2>&1Send stderr to the same place as stdout
|Send stdout as stdin to the next command
python train.py > train_full.log 2>&1Both streams to one file
python train.py 2>&1 | tee train.logLog and show the output

Filter training logs#

cat train.log | grep "loss" | wc -lcount loss lines
grep "loss:" train.log | awk '{print $NF}' > losses.txtlast field only
tail -f train.log | grep --line-buffered "ERROR"live errors
grep "final_accuracy" results/*.log | sort -t= -k2 -n -rbest run first
diff <(grep "accuracy" exp1.log) <(grep "accuracy" exp2.log)
find . -name "*.py" | xargs wc -l | tail -1project size
env | grep -i cudacheck before a run

tmux sessions#

TaskCode
Start a named sessiontmux new -s training
Split horizontallyCtrl+B then "
Split verticallyCtrl+B then %
Move between panesCtrl+B then arrow keys
Detach, run keeps goingCtrl+B then d
Reattachtmux attach -t training
List sessionstmux ls
Kill a sessiontmux kill-session -t training

Linux files, permissions, packages#

CommandWhat it does
mkdir -p a/b/cMake nested directories
rm -rf my-dir/Delete a directory. There is no undo
find . -name "*.ckpt" -size +1GCheckpoints larger than 1GB
grep -r "learning_rate" .Search all files here
chmod +x train.shMake a script executable
chmod 755 deploy.shOwner full, others read+execute
sudo apt update && sudo apt install -y build-essential git curl wget tmux htop unzip python3-venvFresh GPU box packages
sudo systemctl status nginxCheck a service. Also start, stop, restart, enable

Work on a remote GPU box#

ssh -i ~/.ssh/my_gpu_key user@gpu-box-ipconnect with a key
scp model.pt user@gpu-box-ip:~/models/copy to remote
scp user@gpu-box-ip:~/results/metrics.json ./copy from remote
rsync -avz --progress ./data/ user@remote:/data/changed bytes only
ssh -L 8888:localhost:8888 user@gpu-box-ipremote Jupyter on localhost:8888
~/.ssh/config: Host gpu, HostName, User, IdentityFile
ssh -G gpuprint the settings SSH uses
ssh gpuconnect by the short name

Troubleshooting#

SymptomCauseFix
"Permission denied"Permission bitschmod +x or sudo
Run dies when the laptop closesThe SSH session endsRun it inside tmux
sed -i 's/a/b/' file fails on macOSmacOS sed needs '' after -ised -i '' 's/a/b/' file
Bash script breaksWindows \r\n line endingsRun dos2unix
File not found on Linux onlyCase-sensitive filesystemModel.py and model.py differ
pbcopy not foundNot available over SSHUse cat or less
No CUDA in WSL2Linux driver in WSLInstall the Windows NVIDIA driver
Disk fullModels and datasetsdu -sh *, then clear caches

Course aliases#

Load them
source phases/00-setup-and-tooling/10-terminal-and-shell/code/shell_aliases.sh
gpu, gpumem, gpuprocs
GPU status, memory and processes as CSV.
ae, uvvenv
activate .venv, or make one with uv and activate it.
watchloss, taillog <pattern>
follow logs/*.log for a pattern.
trainenv <name>
tmux session with nvidia-smi and htop panes.
syncto <host> <remote_path>
rsync up. syncfrom pulls back.
bigmodels, memhogs
largest model files, top memory processes.

Questions#

&, nohup or tmux?
nohup and tmux survive a closed terminal. Only tmux lets you reattach.
rsync or scp?
rsync for anything large. It resumes and sends changed bytes.
Why fork the course repo?
Only maintainers can push. Your fork is origin.
Rebase or submodules?
The course does not need them.
Run as root?
No. Use sudo only when needed.

Tips#

  • Add *.pt, *.pth and *.safetensors to .gitignore.
  • Run sudo apt update before each install.
  • Use tmux for anything longer than a few minutes.
  • Double-check the path before rm -rf.
  • macOS defaults to ~/.zshrc. Most Linux servers use ~/.bashrc.

Run, monitor and free disk#

CommandWhat it does
watch -n1 nvidia-smiGPU use every second
nvidia-smi --query-compute-apps=pid,name,used_memory --format=csvProcesses on the GPU
htopF6 sorts, F5 tree, F9 kills, / searches
nohup python train.py > train.log 2>&1 &Background run that survives terminal close
df -hDisk use of all drives
du -sh ~/.cachepip and huggingface models land here
du -h --max-depth=1 / 2>/dev/null | sort -hr | head -20Biggest space hogs
pip cache purge and sudo apt cleanClear caches