A Rust + PyO3 OCR cleanup pipeline
robotdad's timecapsule-data collects temporally-filtered text for LLM training. The Internet Archive index it draws on holds roughly 2.3M items for the years 0–1914, about 1.5M of them eligible to download — and their OCR text is riddled with systematic scanning errors.
The question isn't whether the corpus is big. It's how fast you can clean it.
On a 100,000-document Internet Archive benchmark, the pipeline applied 43.3 million substitutions to 80,447 files — 99.8% of the files it processed — in about 21 minutes.
That kind of throughput is exactly what pure Python couldn't reach.
A corpus of this size needs unicode normalization, language detection, triage, boilerplate stripping and hundreds of correction passes per file. Done in pure Python across an eligible pool of ~1.5M documents, that work doesn't finish on a useful timeline — so the engine couldn't stay in Python.
The fix was to move the hot loop out of Python entirely.
Built by robotdad — Marc Goodner at Microsoft, who authored all 120 commits — timecapsule-data collects temporally-filtered text corpora for LLM training. The OCR-cleanup work landed in a tight burst, January 17–24, 2026.
Inside that burst, the cleanup engine was rewritten from the ground up.
The Rust/PyO3 library was added Jan 20; OCR was consolidated to Rust-only by Jan 23; Rayon parallelization landed Jan 24. It fans work out across 24 threads with par_iter().for_each() over file pairs.
On top of that engine sits the layer that does the actual fixing.
Tier 1 cleanup applies 150+ documented OCR-correction patterns — long-s, ligatures, ll→U, rn→m and word run-togethers. Benchmarked on a Ryzen 5950X with NVMe storage, it sustained 64.3 files/s and 50.9 MB/s.
Fast is only half the story. The other half is being right.
Vocabulary validation over 761.9M words surfaced 1,014,036 suspicious candidates. Only 225 turned out to be dictionary false positives — 99.98% precision. Just 972 confusable words remained, meaning the patterns caught ~99.9% of systematic OCR confusions.
Fast and near-perfect — because a human made the architectural calls.
This is human-directed, AI-executed performance engineering. The pattern is general: when pure Python hits a scaling wall, move the hot loop into a compiled, parallel engine behind a PyO3 boundary — and let a person keep making the architectural calls. robotdad made every one of those calls here.
Tens of millions of fixes in minutes, at precision you can trust.
Data as of: January 24, 2026 (last commit in the source repo)
Feature status: Active / open source — Tier 1 (Rust patterns) & Tier 2 (SymSpell) implemented; Tier 3 (LLM) documented as future work
Primary source: github.com/robotdad/timecapsule-data (MIT) — verified via gh repo view robotdad/timecapsule-data
Research performed:
cat rust-ocr-clean/Cargo.toml; grep -nE 'use pyo3|use rayon|par_iter' src/lib.rsgrep -nE '150\+' README.md docs/OCR_CLEANING.md docs/OCR_VOCAB_ANALYSIS.mdgrep -nE '64 files/s|50.9|24 threads|Ryzen 5950X' docs/OCR_VOCAB_ANALYSIS.mdsed -n '5,30p' docs/OCR_VOCAB_ANALYSIS.md (43,314,454 subs; 99.98%)git log --format='%ci %h %s' | grep -iE 'rayon|rust-only|pyo3'; git rev-list --count HEAD (120)gh api users/robotdad --jq '{login,name,company}' (Marc Goodner, Microsoft)Corpus scale: ~2.3M Internet Archive items indexed for years 0–1914; ~1.5M eligible; measured cleanup run = 100,000 documents.
Gaps: Only one timed run (100k docs, 21 min) is documented; no Python baseline benchmark exists in the repo, so no speedup multiplier is claimed.
Primary contributor: robotdad (Marc Goodner, Microsoft) — 120 of 120 commits.