Lessons in multi-agent orchestration

When Green Tests Lie

Building real orchestration under process discipline

The trap

549 green tests, and the shipped wheel still crashed on startup

Amplifier's own engineering doctrine records it: every unit and integration test passed, yet the installed wheel died at launch. The bugs lived at the Rust↔Python FFI boundary the mocks never touched.

Passing tests can lie. That is the problem this whole story is about.

549

unit + integration tests, all passing

1

wheel that crashed on startup anyway

The evidence

Turned loose with up-front design, the same AI built something that actually runs

The bundle's 77 tests pass against real amplifier-core and amplifier-foundation built from git — not mocks in a vacuum. Independently reproduced: 77 passed in 1.07s.

So what is the thing that runs? Let's name it.

77

tests passed in ~1.07s

2

real git-pinned deps: amplifier-core (f57a59a) & amplifier-foundation (dbcfc79)

The project

colombod/amplifier-bundle-orchestration ships three real, reusable primitives

A runnable Python package, amplifier_orchestration — five source modules, exports you can import. It is a fork/mirror of microsoft/amplifier-bundle-orchestration.

Three primitives. The interesting one is the worker pool.

spawn_bundle — unified session spawning with inheritance controls
EventRouter / triggers — cross-session pub/sub, timer & event triggers
BackgroundSessionManager — worker-pool lifecycle

The pattern

The pool_size model realizes Paul Payne's documented 'foreman' pattern

Paul Payne (payneio) designed the foreman pattern: one coordinator agent talks to the user while delegating to a pool of specialist workers. BackgroundSessionConfig.pool_size enforces the concurrency limit for those worker pools.

A real pattern, on real machinery — is that machinery real too?

# background.py — worker pool
pool_size: int = 1
# "Number of concurrent instances
#  allowed (for worker pools)"

# coordinator + worker-pool
#   = the foreman pattern

The machinery

It isn't fabricated plumbing: session.spawn is a real registered capability

session.spawn is registered by amplifier-app-cli and consumed by tool-task. spawn_bundle() is the orchestration-policy layer built atop foundation's PreparedBundle.spawn() mechanism.

Real pattern, real capability. But then coverage speaks.

# amplifier-core registry
coordinator.register_capability(
  "session.spawn", spawn_fn)

# signature
async (agent_name, task,
       parent_session) -> dict

The turn

Coverage tells on the build: the deepest paths are still driven by mocks

Overall coverage is 52%. The real session-spawning code is barely exercised — spawn.py at 34%, background.py at 30% — still driven by MockCoordinator / MockSession fixtures.

Sound familiar? It's the doctrine's own trap, indoors.

52%

overall coverage (610 stmts, 291 missed)

34%

spawn.py

30%

background.py

96%

events.py (triggers.py 94%)

The payoff

77 green tests — and the real spawning body barely runs.

The doctrine's trap reappears inside the honest build. Discipline bought genuine, reproducible primitives — but not immunity from the very mock-vs-real gap the doctrine teaches you to watch for.

The takeaway

Green tests are a starting line, not a finish

The fix the doctrine names is the one thing mocks can never do for you: run real sessions end-to-end. Process discipline earns honest primitives — running the real path is what closes the gap.

Run the real path. Don't trust the mock.

Mocks pass — the real path can still crash (549 green, wheel dead)
Discipline helps — up-front design, tests-first, real deps
Only end-to-end closes it — run a real session

Sources

Research Methodology

Feature status: Active

Data as of: primary-source repo colombod/amplifier-bundle-orchestration (created 2026-01-26, default branch main), fork/mirror of microsoft/amplifier-bundle-orchestration (created 2026-01-08).

Research performed — commands run:

gh repo view colombod/amplifier-bundle-orchestration --json name,description,url,createdAt,updatedAt,defaultBranchRef
wc -l amplifier_orchestration/*.py — spawn.py 696, background.py 596, triggers.py 401, events.py 294, __init__.py 57
uv run --extra dev python -m pytest tests/ -q — 77 passed in 1.07s
uv run --extra dev --with pytest-cov python -m pytest tests/ --cov=amplifier_orchestration --cov-report=term-missing -q — TOTAL 52%
grep -c 'def test_' tests/*.py — 20 + 35 + 22 = 77
grep -n -i 'foreman' notes/v5/AI-REFERENCE.md; grep -n 'pool_size' amplifier_orchestration/background.py
grep -n 'session.spawn' amplifier-core/docs/CAPABILITY_REGISTRY.md; git log -1 2ef847c
grep -n -i 'wheel crashed|FFI|real session' amplifier-foundation/context/amplifier-dev/testing-patterns.md — 549 tests / wheel crash

Gaps: Coverage is 52%, not 100%; spawn.py (34%) and background.py (30%) deep paths are exercised mainly by MagicMock fixtures. The "13 tests / 100% coverage", "461 tests", and "4 feature PRs merged" figures were NOT found in the repo and are excluded. No class literally named "Foreman" exists; the pattern is realized via BackgroundSessionManager / pool_size.

Primary contributors: Paul Payne (payneio) — 8 commits, orchestration architect and foreman-pattern designer; Amplifier (AI) co-author on 4 "Generated with Amplifier" commits; Salil Das merged the two governance PRs (#5, #6).