The Judge Isn't the Builder

Independent validation for Amplifier shadow environments

Amplifier · Shadow Environments

The setup

Shadow environments test risky changes without touching the host

They run your local, uncommitted, multi-repo Amplifier changes inside OS-level isolated Docker or Podman containers — so you can try dangerous edits safely.

The trick that makes it work is a selective git rewrite. Here it is.

Local & uncommitted — test your exact working-tree state as-is: new files, modifications, and deletions.
Multi-repo — register several repos at once for a change that spans them.
Isolated containers — Docker or Podman, security-hardened with dropped capabilities, no-new-privileges, and resource limits.

Proof · How it works

Git's insteadOf config redirects only your registered repos

A global url.<gitea>.insteadOf rule points registered GitHub repos at an embedded Gitea server on localhost:3000, serving exact working-tree snapshots. Every unregistered dependency still hits real GitHub.

Integration test test_url_rewriting.py asserts this is real — and repo-specific, not a blanket rewrite.

# manager.py:585 — selective rewrite
git config --global --add \
  url."http://localhost:3000".insteadOf \
  "<registered repo>"

# gitea.py:39
base_url = "http://localhost:3000"

# test_url_rewriting.py asserts:
# - localhost:3000 + github.com in config
# - unregistered repo passes through

The complication

The agent that builds the rewrite can't judge its own work

The shadow-operator agent builds that rewrite. But if a rewrite silently fails, you'd unknowingly test upstream code — and the run would still look green.

A builder certifying its own build isn't a verdict you can trust. So who checks it?

Intended: uv/pip "installs from GitHub" but actually pulls YOUR local snapshot.
Silent failure: the rewrite doesn't apply, the install falls back to upstream code — and you never notice.
Result: a green run that never tested your change.

The independent judge

A separate agent runs AFTER the shadow is built

amplifier-foundation ships the amplifier-smoke-test agent (~290 lines at commit 41639d1, by Brian Krabach). Its text: "You run SEPARATELY from shadow-operator to provide unbiased verification. Don't trust claims — verify with evidence."

Independence only matters if it has teeth. Here's where it bites.

290

lines at commit 41639d1 (2026-02-04); 307 at HEAD

8

commits on the agent, all by Brian Krabach

Mechanism

Source Verification checks the exact mechanism that could fail silently

The 25-point Source Verification category inspects the actual git insteadOf rules and compares installed commits to snapshot commits — so a silent rewrite failure scores zero instead of passing unnoticed.

Zero here isn't just a low score — it triggers an automatic FAIL.

Snapshot commits exist — 5 pts
Git URL rewriting configured — 5 pts
Installed package uses snapshot — 10 pts
Unregistered repos NOT redirected — 5 pts

The scored verdict

The rubric is unforgiving by design

100 points across five weighted categories, a pass threshold of 75, and an automatic FAIL if Source Verification hits zero. A scored, thresholded judgment replaces the builder's self-report.

And when it can't even run the tests, it does something stricter still.

Source Verification — 25
Installation Health — 20
Code Execution — 30
Isolation Integrity — 15
No Regressions — 10
Pass threshold: 75 / 100

The payoff

When it can't actually run the shadow tests, it HALTS

No shadow_id, shadow missing, or shadow tool unavailable? The agent returns CANNOT_PROCEED / VERDICT: INCOMPLETE and refuses to test on the host as a workaround — rather than hand out a PASS it can't back up.

Refusing to fake success is exactly what makes the verdict trustworthy.

HALT ON FAILURE — DO NOT WORK AROUND

Status: CANNOT_PROCEED
VERDICT: INCOMPLETE

"NEVER give a PASS verdict if you couldn't actually run shadow tests." (6 NEVER rules in the agent)

The pattern to keep

A verdict you can trust requires that the judge is not the builder

Independent, evidence-based validation that halts beats a self-certified green. Separate the builder from the judge.

Sources

Research Methodology

Data as of: 2026-02-28 (agent HEAD 517f94d = 307 lines; framing version 41639d1, 2026-02-04 = 290 lines).

Feature status: Active — amplifier-smoke-test agent in microsoft/amplifier-foundation; shadow bundle microsoft/amplifier-bundle-shadow present.

Note: samueljklee/amplifier-bundle-smoke-test (v1.0.0) is a separate CLI sanity check with "no containers, no isolation"; the container/shadow/100-point-rubric mechanism lives in amplifier-foundation + amplifier-bundle-shadow.

Commands run:

git show 41639d1:agents/amplifier-smoke-test.md | wc -l → 290; wc -l at HEAD → 307
grep -n 'points)' agents/amplifier-smoke-test.md (rubric 25/20/30/15/10; threshold 75)
git show 41639d1:agents/amplifier-smoke-test.md | grep -n 'independent\|SEPARATELY' (lines 39, 41)
grep -n 'HALT\|NEVER\|CANNOT_PROCEED' agents/amplifier-smoke-test.md (grep -c NEVER = 6)
grep -rn 'insteadOf' src/ (manager.py:585) & grep -n 'localhost:3000\|base_url' gitea.py (line 39)
sed -n '1,60p' tests/integration/test_url_rewriting.py (selective-rewrite assertions)
git log --format='%an' -- agents/amplifier-smoke-test.md | sort | uniq -c (8 commits, Brian Krabach)

Gaps: No live end-to-end shadow run (no Docker/Podman executed); URL-rewriting, snapshot, and isolation claims rest on source code + integration-test assertions, not a live container run. "Silent rewrite failure" wording paraphrases the topic; the agent supports the intent via "Don't trust claims — verify with evidence."

Primary contributors: Brian Krabach (all 8 commits on the validation agent); Samuel Lee (all 5 commits on the separate smoke-test bundle).