Chasing the Ghost

How two engineers debugged a stale-state error together

The Ghost

A transient error appeared after an update — then vanished on restart

After a routine uv tool install --force, Amplifier logged a "Partial provider failure" warning. Restart, and it was gone. A classic stale-state ghost.

A bug you can't reproduce is still a bug worth chasing.

1. Update via uv tool install --force
2. Session warns: "Partial provider failure"
3. Restart — the warning disappears

Proof

The warning is real, and it lives in the CLI app layer

It's a concrete, shipped code path in session_runner.py (lines 491–494) — not a rumor. The session continues with whatever providers loaded.

So the ghost is real. But what was it hiding?

# amplifier-app-cli/session_runner.py:491-494
logger.warning(
  "Partial provider failure: "
  "N/M loaded... Session continuing "
  "with available providers "
  "(self-healing NOT triggered "
  "for partial failure).")

The Complication

One ghost, four interconnected bugs, two repos

The transient warning concealed four bugs spanning the CLI and the kernel/foundation stack — and no single person had the whole picture.

A lone-hero fix wasn't possible. This needed collaboration.

1

Silent auto-configuration of a provider from env vars, bypassing init

2

Aggressive self-healing on partial failures causing cascade failures

3

Broken activator lookup ("No activator found") from AppModuleResolver wrapping BundleModuleResolver

4

Module-ID vs provider-name mismatch making the warning report ALL providers as failed

The Opening Move

Samuel Lee shipped the first self-healing detection in PR #61

Samuel opened PR #61 — "add self-healing for stale install state after reinstall" — authoring 3 commits to catch the config-vs-mounted provider mismatch.

One contributor landed the opening move. Then review widened it.

#61

PR by samueljklee

3

commits (a725eff, 9e8d873, f94bdf6)

Jan 23

opened 2026-01-23

Collaboration in Review

Review expanded the scope, then earned approval

Diego reviewed PR #61 and pushed for broader coverage. Samuel extended the fix to check providers AND tools, then Diego approved and merged it on 2026-01-27.

This is collaboration in practice — review that makes the fix bigger.

Diego: "even if a single provider or a single module fails loading we might have to see for self healing opportunities"

Samuel: "Covered in the latest commit"

Diego: "looks good to me!" (APPROVED)

The Turn

Diego built on Samuel's work — and Samuel reviewed back

PR #71 made self-healing conservative and fixed the broken activator lookup. PR #72 normalized module IDs so the warning stopped falsely blaming every provider. Samuel reviewed #71.

Co-ownership turned a single fix into a hardened stack.

#71 "require init when no provider configured, conservative self-healing" — merged 2026-01-28
#72 "correct provider failure detection by normalizing module IDs" — merged 2026-01-29
↩ Samuel reviewed #71 (reciprocal review)

The Payoff

Diego chased the ghost to its lair — the real root cause

In the second repo, amplifier-foundation, stale install-state.json survived --force so reinstall was skipped. Diego detected environment recreation via the Python executable's mtime (c889e85, fixes microsoft/amplifier#204).

The root cause was reachable only because the collaboration cleared the path.

# amplifier-foundation
# modules/install_state.py
c889e85 "fix: detect environment
recreation via python
executable mtime"

# Fixes microsoft/amplifier#204
# (issue: install-state.json
# survives env recreation)

What to Keep

Collaborative debugging catches what a lone hero can't

Building on each other's PRs and catching issues in review, Diego and Samuel hardened the whole provider/install-state stack. Brian Krabach later carried the fix further with an lstat hardening (342e1cd).

A self-hiding ghost gets caught — and the stack gets durably stronger.

2

engineers co-owning session_runner.py

3

PRs (#61, #71, #72) merged

342e1cd

Brian Krabach's lstat follow-on

Sources

Research Methodology

Data as of: March 2026 | Feature status: Shipped / merged

Repos: microsoft/amplifier-app-cli, microsoft/amplifier-foundation, microsoft/amplifier-core

Commands run:

grep -n 'Partial provider' amplifier_app_cli/session_runner.py (warning at lines 491–494)
gh pr view 61 --repo microsoft/amplifier-app-cli --json author,mergedBy,commits (author samueljklee, mergedBy colombod, 3 commits)
gh api repos/microsoft/amplifier-app-cli/pulls/61/comments ; gh pr view 61 --json reviews (review thread verbatim)
gh pr view 71 --json ... ; gh pr view 72 --json ... (merged 2026-01-28 / 2026-01-29)
git -C amplifier-foundation show -s c889e85 ; git -C amplifier-foundation show -s 342e1cd
gh issue view 204 --repo microsoft/amplifier (CLOSED — install-state survives env recreation)
git -C amplifier-app-cli shortlog -sne --all (Diego 23+3, Samuel 6, Brian 638 commits)

Gaps: The named bundle repo michaeljabbour/amplifier-bundle-collaboration (6 commits, no PRs/issues) does not itself contain this story; the verified story lives in the three Amplifier repos above. Amplifier's investigate/reproduce sessions are attribution-only (Co-Authored-By trailers), not independently verified metrics.

Primary contributors: Samuel Lee (samueljklee), Diego Colombo (colombod); follow-on by Brian Krabach (brkrabac).