robotdad / timecapsule-data

Past the 10,000 Wall

A pre-WWI corpus pipeline designed around the throttle

Marc Goodner (Microsoft) · Jan 17–24, 2026

The Goal

To teach a model only the world before 1914, first pull it out of the Internet Archive

Marc Goodner (Microsoft, GitHub robotdad) built timecapsule-data to assemble a temporally-pure pre-1914 text corpus. The Internet Archive is the largest — and messiest — source: ~2.3M pre-WWI text items.

Temporal purity means enumerating a giant, hostile corpus. Just how giant?

~2.3M

pre-WWI items (year 0–1914)

1914

the hard temporal cutoff

Proof of Scale

The pre-1914 catalog really is that big

The finished index catalogs roughly 2.3M items into a ~2.5GB SQLite database. That scale is exactly what makes naive, page-by-page scraping impossible.

So what happened when the first approach tried to page through all of it?

~2.3M

items indexed

~2.5GB

SQLite catalog (ia_index_0_1914.db)

The Wall

advancedsearch.php stops dead at 10,000 results

The original path used the official internetarchive library over advancedsearch.php with page-based pagination — hard-capped at 10,000 results. Anything past row 10,000 returned 0. And IA aggressively throttled the bulk requests on top of it.

You can't enumerate 2.3M through a 10,000-result window. Something had to give.

10,000

hard pagination limit

0

results past row 10,000

The Pivot

Delete the official library; switch to the cursor Scraping API

On 2026-01-17 he deleted the official-library collector (internet_archive_v2.py) and moved to IA's cursor-based Scraping API at /services/search/v1/scrape — no result limit, up to 10,000 per batch.

Convenience traded for control. Now the full catalog is reachable — here's how.

- IA_SEARCH_API = "https://archive.org/advancedsearch.php"
- # rows=&page= pagination, capped at 10,000
+ url = "https://archive.org/services/search/v1/scrape"
+ scrape_batch(query, fields, cursor=None, count=10000)
+ # cursor-based pagination, no result limit
# commit 1828c35 (2026-01-17): delete official
# library version -> full control over requests

Mechanism · Catalog

Catalog before you download — resumably

Phase 1, tc-ia-index, splits the date range into time-chunks of ~100k items each with resumable atomic per-chunk commits, enumerating all ~2.3M items in ~40 min (README estimate).

A resumable index makes the full catalog reachable. Next: don't fetch all of it.

1Chunksplit the range into ~100k-item time chunks
2Scrapecursor-paginate each chunk, no 10,000 wall
3Commitatomic per-chunk write — safe to resume

Mechanism · Filter

Filter, don't fetch-everything

Phase 2/3 (tc-ia-enrich / tc-ia-download) select only the items worth the days of download: quality_score ≥ 0.65 and imagecount ≥ 10 pages — about ~1.5M of the 2.3M.

Selecting before spending keeps it tractable. Now the risky part: the download.

≥ 0.65

minimum quality_score

≥ 10

pages (imagecount)

~1.5M

items selected of ~2.3M

The Payoff

The download layer is built for the ban

One shared global rate limiter spans all 12 workers: when any worker hits a 429, they all back off together. Base delay 2.0s, max 60s, ×3.0 on a rate-limit — and a BannedException after 5 consecutive 429s. Rate-limiting is a design goal, not a measured outcome.

Throttling handled as a first-class design input — the reason the rebuild works.

Shared limiter, 12 workersany one 429 backs off all workers together
2.0s base · 60s max · ×3.0adaptive backoff factor on a 429
BannedException @ 5raised after 5 consecutive 429 errors

The Pattern

When an API fights bulk access, stop scraping as-you-go

Separate Catalog, Filter, and Download. Make the slow part resumable so a restart costs minutes, not days — and make the throttling coordinated, so the whole worker pool treats a single 429 as a signal, not a surprise.

Throttling as a design input, not an accident: that's the reusable pattern.

Sources

Sources & Research Methodology

Data as of: 2026-07-21 · Feature status: Working / shipped tooling

Primary source: robotdad/timecapsule-data (cloned to /tmp/timecapsule-data). Note: the named repo kenotron-ms/amplifier-presentation is a generic Amplifier overview with zero Internet Archive content; it is NOT the source for this story.

Contributors: Marc Goodner <mgoodner@microsoft.com> (GitHub robotdad), sole author — 120 / 120 commits, 2026-01-17 to 2026-01-24.

Commands run:

git -C /tmp/timecapsule-data log --oneline --reverse | head — 120 commits, 8-day sprint
git show 1828c35 -s --format='%ad %b' — delete internet_archive_v2.py (2026-01-17), ban detection
git show 071642c -s --format='%b' — 10,000-result wall; switch to Scraping API
git show b51a394 -s --format='%b' — shared limiter, all 12 workers back off on a 429
grep -n 'scrape|cursor|services/search' src/.../ia_index.py — /services/search/v1/scrape, count up to 10000
grep -n '429|RateLimiter|global_rate_limiter' src/.../ia_download.py — base 2.0s / max 60s / ×3.0
grep -n '0.65|imagecount' docs/IA_INDEX_DATABASE.md — quality_score ≥ 0.65 AND imagecount ≥ 10
grep -n '2.3M|~40' README.md — ~2.3M items, ~2.5GB DB, ~40 min index estimate
gh api users/robotdad --jq '.name, .login' — Marc Goodner / robotdad

Gaps & caveats: Time/scale figures (~2.3M catalog, ~1.5M eligible, ~12–18 day download) are documented estimates; the repo contains no log of a completed full run. Index build time has a repo-internal inconsistency (README ~40 min vs commit body ~15 min) — cited as the ~40 min README estimate. Ban avoidance ("zero 429s") is a design goal, not a measured outcome: the code is built to absorb 429s, but no run log asserts a real run finished without one.