Data Quality and Dataset Gates

Load and split data with a seed, repair CSV cells, gate rows on a contract, audit leaks and rehearse migrations.

What each gate builds#

Factor	CSV repair	Contract gate	Split auditor	Migration rehearsal
Builds	Cleaned table and recipe	Accepted and quarantine rows	Leak report per record	Rehearsal receipt
Input	csv text, aliases, date_columns	JSONL plus contract.json	{train,test} or {records}	Database, migration, rollback, invariants
Output	cleaned.csv, recipe.json	accepted.jsonl, quarantine.jsonl	split-audit.json	rehearsal.json, schema-diff.json
Pass signal	No pending cells	`passed` is true	`usable` is true	`release_ready` is true
Gate exit	Nonzero on failure	exit 1 rule, exit 2 input	exit 2 when unusable	Nonzero on failure
Limit	Slash and ISO dates only	No semantic truth	Exact copies, not paraphrases	SQLite policy, not OS isolation

Command	What it does
`python3 scripts/project_test.py <project> --init <workspace>`	Makes a workspace. The first run fails
`python3 scripts/project_test.py <project> --all --solution --strict`	Checks the reference solution
`python3 cli.py samples/input.json --output demo-output`	Runs CSV repair or migration rehearsal
`python3 cli.py sample.csv --question "sum units by region" --output output/report.html`	Answers a CSV question with a receipt
`python3 cli.py samples/input.jsonl --contract samples/contract.json --output demo-output`	Splits rows into accepted and quarantine
`python3 cli.py samples/input.json --output split-audit.json --html split-audit.html`	Audits a train and test split
`python3 cli.py --train train.jsonl --test test.jsonl --check --output split-audit.json`	Audits JSONL partitions in CI
`python code/data_utils.py`	Loads, converts, splits and fingerprints a dataset

Setting	Default	Note
`--limit`	100	Result rows, at most 1000
`max_steps`	100000	SQLite instruction budget
`load_csv` rows	10000	Under 2 MB, at most 50 columns
`--seed`	course	Group hash seed for `--split`
Contract rows	100000	API input cap
SQL deadline	two-second	Bounds statements, not backups
`make_splits` seed	42	Train 0.8, validation 0.1
Local storage	10 GB	Use cloud storage above it

Symptom	Cause	Fix
"ragged row"	A row has the wrong cell count	Fix the source row
Pending "ambiguous date"	Both parts are 12 or less and differ	Give an ISO date in decisions
"stale review"	Table or recipe changed after review	Review the new preview again
"Query rejected: "	SQL left the read boundary	Write one SELECT over `data`
"Aggregate column must contain finite numeric values"	Column is TEXT	Aggregate a REAL column
Contract `--check` exits 1	Rows are in quarantine	Read review.html. Decide on rows apart
Split `usable` is false	Content or group leak	Split by group with `--split`
`release_ready` is false	Rollback is not exact	Write rollback SQL that restores rows

dataset = load_dataset("stanfordnlp/imdb", split="train")

split = dataset.train_test_split(test_size=0.2, seed=42)

train_val = split["train"].train_test_split(test_size=0.125, seed=42)

train_ds = train_val["train"]70 percent of all rows

val_ds = train_val["test"]10 percent

test_ds = split["test"]20 percent

dataset.to_parquet("imdb_train.parquet")best storage format

load_dataset("wikimedia/wikipedia", "20231101.en", split="train", streaming=True)

Task	Code
Profile CSV columns	`profile(text)`
Preview cell changes	`preview(table, recipe)`
Apply reviewed repairs	`repair(table, recipe, decisions=None)`
Check one record	`row_issues(row, contract)`
Find duplicate groups	`duplicate_rows(rows, contract)`
Audit two partitions	`audit(train, test)`
Split by group hash	`split_groups(records, test_fraction=0.2, seed='course')`
Run a bounded read	`run_query(data, sql, max_rows=100, max_steps=100000)`

Command	What it does
`git lfs track "*.safetensors"`	Stores the file outside git with a pointer
`dvc add data/training_set.parquet`	Makes a small `.dvc` pointer file
`dvc remote add -d myremote s3://my-bucket/dvc-store`	Sets the DVC remote
`dvc push`	Uploads tracked data to the remote
`snapshot_download("sentence-transformers/all-MiniLM-L6-v2")`	Caches a full model repo

Recipe order: trim, then alias, then date for each column.
Changes: each change names row, column, before and after.
No coercion: Boolean true fails integer and number fields.
Duplicates: every colliding row is reported, the first one too.
Gate: a dataset passes with no quarantined rows and no dataset issues.
Fingerprint: NFKC, casefold and collapsed spaces, then SHA-256.
Group split: SHA-256 of seed and group over 2**256, below the fraction is test.
Formats: Parquet for storage, Arrow in memory, CSV and JSON to exchange.

Does normalization catch paraphrases?: No. It finds exact normalized copies only.
Does rehearsal change the source database?: No. The source is never opened writable.
Does the model run its SQL freely?: No. SQLite authorization still controls execution.
Can I drop quarantined rows at once?: No. Partitions are evidence, not permission to discard.
Why set a seed?: The same seed produces the same split every time.