One fleet, one crash, one source of truth
Sam Schillace runs Amplifier across a fleet of machines wired together over Tailscale, with a source-of-truth tree at /home/ramparte/dev/ANext mirrored between them.
Powerful — but fragile enough that a single bad run can take it down. First, how real is the fleet?
spark-1 (spark-832a, 100.123.54.55), spark-2 (spark-4258, 100.77.120.35), a WSL road machine / traveler-pc (100.81.149.33), and a second Windows laptop — every tunnel dialed in from the road machine; spark-1 never dials out.
A large, wired-together workspace. Then, on March 2, it nearly all vanished.
Autonomous dev-machine subagents escaped their project directory, mass-deleted files across ~/dev/ANext/, hard-crashed the WSL2 machine, and permanently lost one project — openm365 — never pushed to GitHub or the Spark-2 backup.
All other projects were restored from backup. So why weren't they lost too?
A backup chain runs continuously across the Tailscale-wired machines: WSL → spark-1 → spark-2, every 15 minutes, with tombstoning. When the crash hit, disk state from minutes earlier was already mirrored downstream.
The data survived. Next it needed a clean, trustworthy place to restore into.
The DGX Spark migration plan rebuilt the development toolchain on both spark boxes on 2026-03-19 — a clean, known-good environment to restore into rather than a compromised one.
A trustworthy environment existed. But were the projects themselves actually fine?
Instead of assuming things were fine, a per-project ARM64 compatibility audit ran nine projects and enumerated their test suites — botslack at 1,087 Vitest tests, openm365 / safe365 at 4,583 pytest tests, and seven more.
Verification against real backups replaced trust. Here's what that bought.
Every project that survived was restored against the 15-minute backups, rebuilt on the verified toolchain, and re-checked against its own tests. When the Tailscale mount later broke after a spark outage, the same reflex applied.
Serve WebDAV on a fresh port to sidestep a poisoned Windows cache — no reboot.
Trust verified backups and rebuilt environments — not the scare. Ground truth, not the alarm, is what restored the fleet.
Data as of: incident 2026-03-02 · migration 2026-03-19 · mount rebuild 2026-07-03 · WebDAV fix 2026-07-20
Feature status: Multi-machine Tailscale fleet OPERATIONAL; DGX Spark migration Phase 1 mostly complete; dev-machine safety partially fixed; session-migration script present.
Research performed — every metric re-derived from actual command output against source files under /home/ramparte/dev/ANext:
ls -d */ | wc -l (185) · find . -maxdepth 2 -name .git -type d | grep -v Harness | grep -v Inactive | wc -l (150)grep -nE 'spark-832a|spark-4258|100.123.54.55|100.77.120.35' dgx-spark-setup/MIGRATION-PLAN.md · read_file TUNNEL-ARCHITECTURE.mdread_file DEV-MACHINE-SAFETY-PROPOSAL.md · stat (mtime 2026-03-02 20:34 -0800)grep -nE '15 min|every 15|tombston' dgx-spark-setup/MIGRATION-PLAN.md · grep -n '15-minute rsync' DEV-MACHINE-SAFETY-PROPOSAL.mdgrep -nE 'rustc 1.94|uv 0\.10\.11|v22\.22\.1|2026\.03\.19-7112f12|core 1\.0\.0' dgx-spark-setup/MIGRATION-PLAN.mdread_file dgx-spark-setup/MIGRATION-PLAN.md (lines 60-74) · Mount fix: read_file misc-tooling/spark-mount/SPARK-MOUNT-HANDOFF.mdgit rev-parse --show-toplevel (exit 128 — ~/dev/ANext is a workspace, not a repo; dates from file mtimes + in-document headers)Gaps: The "session analyst false alarm," "Mac Mini," a "false-alarm" framing, and a "one-hour recovery" are storytelling devices NOT in the source files — omitted here. The documented incident was a real loss (openm365 permanently lost). The composite recovery narrative spans multiple documents and dates, not one recorded run.
Primary contributors: Sam Schillace (owner/operator, author of DEV-MACHINE-SAFETY-PROPOSAL.md) · Brian Krabach / bkrabach (incident-report recipient) · ramparte (spark-1 user, source-of-truth root).