Swarm

Parallel agent tasks that survive crashes

The Setup

Agent workflows pile up tasks. Swarm runs them without losing work when a worker dies.

Amplifier agent workflows generate long lists of tasks to run. Swarm is the parallel executor for that pile — and its whole job is to finish the work even when a worker crashes mid-task.

First question: is it real, and is it really parallel?

Proof — It's Real

A ~2,496-line tool that spawns isolated OS processes, not threads.

amplifier-swarm v0.1.0 is a working tool across 7 Python modules. Each worker is a separate OS process spawned via multiprocessing — the comment in the source reads "Use multiprocessing for proper isolation."

Real parallelism exists. So what was it built to replace?

2,496

lines of Python across 7 modules

v0.1.0

amplifier-swarm, in ramparte/amplifier-toolkit

N

isolated worker processes (multiprocessing.Process)

The Pain It Replaces

The old tool ran serially, raced, and lost work when a worker crashed.

Swarm's documented predecessor, webword-pm, was serial-only with no dashboard, race conditions on concurrent runs, no graceful shutdown, and limited recovery — so a crash mid-task lost its progress.

Swarm has to fix both problems: the serial bottleneck and the lost work.

No parallelism — serial only
No dashboard to watch or control runs
Race conditions with concurrent runs
No graceful shutdown
Limited error recovery — a crash lost the work

Mechanism — Safe Handoff

An atomic claim means two workers can never grab the same task.

The task queue is a durable SQLite database opened with isolation_level 'IMMEDIATE'. To take a task, a worker runs BEGIN IMMEDIATE, then updates one row WHERE status = 'not_started' — the comment: "prevent race conditions under high load."

Coordination is safe. But how does the queue survive a restart?

# claim_task() — atomic, race-free
BEGIN IMMEDIATE
SELECT ... WHERE status = 'not_started'
UPDATE tasks
  SET status = 'claimed'
  WHERE id = ?
    AND status = 'not_started'
  RETURNING *
# SQLite isolation_level = 'IMMEDIATE'

Mechanism — Durable State

Three SQLite tables plus heartbeats track exactly which task each worker holds.

State lives in three tables — tasks, workers, and a timestamped execution_log — backed by six indexes. Workers write a heartbeat every 30 seconds. Because the state is on disk, the queue knows the truth even across restarts.

That durable record is what makes the payoff possible: recovering from a crash.

3

SQLite tables: tasks, workers, execution_log

6

indexes over status, worker, priority & log

30s

worker heartbeat interval

The Payoff — Crash-Proof Queue

A worker dies mid-task, and its work is reclaimed and finished — not lost.

When a worker stops heartbeating for more than 90 seconds it's marked crashed and restarted while the orchestrator lives. Its orphaned task — stuck over 30 minutes — is reset to not_started for retry (up to a default of 2) instead of vanishing.

The crash-proof queue in action. Now, can an operator watch it happen?

1

Worker goes silent — >90s with no heartbeat marks it crashed

2

Orchestrator restarts the worker as "{id}-restarted"

3

Orphaned task stuck >30 min is reset to not_started

4

Task retries up to 2× by default, or is marked failed

Recovery You Can See

A live dashboard lets you watch, retry, kill, and shut down the swarm.

A FastAPI + WebSocket server on port 8765 broadcasts state every 5 seconds — no manual refresh. From it you retry failed tasks, kill stuck ones, and trigger graceful (5-min) or emergency shutdown.

Recovery machinery becomes something an operator can trust and control.

8765

default dashboard port (FastAPI + WebSocket)

5s

live broadcast interval — no manual refresh

5 min

default graceful shutdown before hard stop

What To Keep

A serial pile of agent tasks becomes parallel work a crash can't erase.

Swarm resolves the story: N isolated processes, an atomic SQLite queue, durable state, and automatic recovery. On 3 workers the author estimates ~20-30 tasks/hour — a documented estimate, not a measured benchmark, and it depends on task complexity.

Parallel, durable, crash-resilient — with one honest, qualified figure.

~20-30

tasks/hour on 3 workers (author's estimate)

3-6

concurrent LLM API calls expected

2×

default automatic retries per task

Sources

Research Methodology & Sources

Data as of: based on source at last commit 2026-03-03; no swarm feature commits after 2026-01-26.

Feature status: Working prototype / v0.1.0 — "Phase 1 & 2 Complete – Production Ready" with Phase 3 monitoring added.

Repository: tools/amplifier-swarm in ramparte/amplifier-toolkit (git remote confirmed).

Research performed:

Line counts: wc -l tools/amplifier-swarm/src/amplifier_swarm/*.py (2,496 lines, 7 modules)
Package metadata: cat pyproject.toml; cat __init__.py (name amplifier-swarm, version 0.1.0)
Source read: orchestrator.py, worker.py, database.py, dashboard.py, cli.py
Commit history: git log --format='%h|%ad|%an|%s' --date=short -- tools/amplifier-swarm/
Authors: git log --format='%an' -- tools/amplifier-swarm/ | sort | uniq -c
Predecessor contrast & throughput: sed -n '237,276p' README.md; sed -n '234,245p' EXAMPLE.md
Test presence: find tools/amplifier-swarm -name 'test_*.py' (none found)

Gaps & caveats: The ~20-30 tasks/hour figure is a documented estimate, not a measured benchmark, and depends on task complexity. No automated test files exist in the tool directory. The "~1/N wall-clock" speedup is not measured in the repo.

Primary contributor: Sam Schillace — sole git author of all 3 commits touching tools/amplifier-swarm/ (pyproject lists "Microsoft Amplifier Team").

Status: v0.1.0 · Working Prototype