Notebooks, Data Files and Profiling

Run notebooks safely, load and split datasets, keep big files out of git, profile time and memory, catch silent AI bugs.

Notebook troubleshooting#

Symptom	Cause	Fix
Works for me, breaks for others	Cells run out of order	Kernel > Restart & Run All before sharing
Clean notebook needs a deleted cell	Hidden state in the kernel	Restart the kernel regularly
RAM keeps growing	Variables persist across cells	`del variable_name` and `gc.collect()`
Kernel dies silently	UI hides the error	Read the terminal that runs Jupyter
`print(df)` shows text	Not the last expression	End the cell with `df`
Import error after pip install	Package in another Python	`!pip install package` in the notebook
Cell runs forever	Waits for `input()` or a loop	Kernel > Interrupt, or `I` twice
Colab files gone	Free session ended after 90 minutes of inactivity	`drive.mount('/content/drive')`

dataset = load_dataset("stanfordnlp/imdb", split="train")cached after first load

stream = load_dataset("wikimedia/wikipedia", "20231101.en", split="train", streaming=True)

dataset.to_parquet("imdb_train.parquet")also to_csv, to_json

split = dataset.train_test_split(test_size=0.2, seed=42)

train_val = split["train"].train_test_split(test_size=0.125, seed=42)

train_ds, val_ds, test_ds = train_val["train"], train_val["test"], split["test"]

make_splits(ds, train_ratio=0.8, val_ratio=0.1, seed=42)data_utils helper

model_dir = snapshot_download("sentence-transformers/all-MiniLM-L6-v2")

Task	Code
Stop only on a bad loss	`if loss.item() > 100 or torch.isnan(loss): breakpoint()`
Inspect in pdb	`p outputs.shape`, `p loss.item()`, then `c` or `q`
Profile functions	`python -m cProfile -s cumtime train.py`
Profile lines	`@profile`, then `kernprof -l -v train.py`
Memory by line	`python -m memory_profiler your_script.py`
Top allocations	`tracemalloc.take_snapshot().statistics("lineno")`
GPU memory	`torch.cuda.memory_allocated()`, `torch.cuda.memory_reserved()`
Training curves	`SummaryWriter("runs/experiment_1")`, then `tensorboard --logdir=runs`

Symptom	Cause	Fix
NaN after a few steps	Learning rate too high	Reduce by 10x. Add gradient clipping
NaN immediately	Log of zero or negative	`torch.log(x + 1e-8)`
Loss stuck at ln(num_classes)	No gradient flow	Remove the accidental `.detach()`
99% test accuracy	Data leakage	`check_data_leakage(train_set, test_set)`
OOM	Batch too large	Halve the batch. `torch.cuda.empty_cache()`
OOM during backward	Gradients not cleared	`optimizer.zero_grad()` each step
Slow, low GPU use	Data loading bottleneck	`num_workers=4` and `pin_memory=True`
Tensor on the wrong device	CPU tensor, GPU model	`check_devices(model, *tensors)`

Command	What it does
`pip install jupyterlab` then `jupyter lab`	Start JupyterLab
`Escape` / `Enter`	Command mode / edit mode
`Shift+Enter`	Run cell, move to next
`A` / `B`	Insert cell above / below
`DD`	Delete cell
`M` / `Y`	Convert to markdown / code
`Shift+Tab`	Show function signature
`Ctrl+Shift+H`	Show all shortcuts

Command	What it does
`%timeit np.random.randn(10000)`	Runs many times and averages. Microbenchmarks
`%%time`	Runs the cell once. Training runs
`%matplotlib inline`	Plots render in the notebook
`!pip install scikit-learn`	`!` runs any shell command
`%env CUDA_VISIBLE_DEVICES`	Check an environment variable
`%who`	List all variables in memory
`!which python`	Check the kernel environment

Setting	Default	Note
CSV	Slow	Large. Human readability, spreadsheets
JSON	Slow	Large. APIs, nested data
Parquet	Fast	Small. Best storage format for AI work
Arrow	Fastest	Small. In memory, inside `datasets`
Train split	80%	Validation 10%, test 10%
Dataset cache	`~/.cache/huggingface/datasets/`	Model cache is `~/.cache/huggingface/hub/`

Task	Code
Ignore weights and data	`.safetensors`, `.pt`, `data/*.parquet`, `models/`
Track with Git LFS	`git lfs track "*.safetensors"`
Commit the LFS rules	`git add .gitattributes`
Start DVC	`dvc init`
Track data with DVC	`dvc add data/training_set.parquet`
Commit the pointer	`git add data/training_set.parquet.dvc data/.gitignore`
Push data to S3	`dvc remote add -d myremote s3://my-bucket/dvc-store` then `dvc push`