Green Checkmarks Lied

Two-layer defense for AI PR review

The Frame

A green check can still crash every user

In AI code review, green checkmarks are supposed to mean safe to ship. This is the story of a routine change that passed every check — and still broke every user. The question the rest of this deck answers: what actually earns trust when the tests go green?

The Deterministic Layer

The process was real and disciplined

During implementation, python_check verified syntax before committing locally. Then a shadow environment confirmed the CLI actually launched.

Verified, credible confidence — before we break it.

1

Run python_check to verify syntax, then commit locally before shadow testing

2

Shadow tests the CLI with amplifier --version and amplifier --help

3

Smoke rubric awards points when CLI tools respond: amplifier --version returns output

The Break

11 of 11 passed — then users crashed

A routine change shadow-tested 11/11 tests passed and was pushed to production as commit 87e42ae. Users updated and crashed — they could not start sessions.

Every check was green. So how did this happen?

11/11

Shadow tests passed before push

87e42ae

Commit pushed to production

P0

Users crashed: cannot start sessions

The Root Cause

The checks proved it launched, not what users do

Testing must match how a real user encounters the feature — a realistic scenario from the user's perspective, not just that the CLI starts.

The pivot: harden both sides of the defense.

Green checkmarks ≠ complete coverage

A named failure mode from the case study: "False confidence from green tests."

Passing tests prove what you tested

"… not what you didn't test." The lesson recorded from the P0 regression.

Layer One — Static

Guard contracts, not implementations

The static side hardened around the design claim: if "X is overridable" is the claim, test the override — and review adversarially every time.

That stops false confidence at the source. Now the dynamic side.

Tests must guard contracts

"If 'X is overridable' is the design claim, test the override" — not just the happy path.

Review adversarially every time

"Each fix round changes the attack surface." Re-review with fresh eyes.

Layer Two — Dynamic

The shadow environment runs a REAL session

It installs the changed code from source and runs an actual amplifier run against the bundle — not a launch-only check.

Real user behavior, exercised. Now, who grades it?

# install the changed code from source
shadow.create(local_sources=[...])
uv tool install git+https://github.com/microsoft/amplifier
amplifier bundle add / use

# run a REAL Amplifier session
amplifier run 'test the bundle'

The Final Defense

An independent agent that won't trust the operator

The amplifier-smoke-test agent runs SEPARATELY from the operator for unbiased verification: "Don't trust claims — verify with evidence." It scores a 100-point rubric before push approval.

An unbiased verifier is what makes green mean green.

100

Point validation rubric

≥ 75

PASS threshold to seek push approval

Phase 6

Independent smoke test = FINAL DEFENSE

The Habit

Every miss becomes the next check

After the regression, the missing pattern was found in under 30 minutes and users were unblocked within 1 hour — then the gap was folded into an updated checklist.

Turn each escape into a permanent guard.

<30 min

To find the missing pattern

1 hr

Until users were unblocked

Updated Process Requirements

"Run smoke test with EXACT user deployment scenario" — added to the Algorithm Fix Validation checklist.

Sources

Research Methodology

Data as of: Feb–Mar 2026 (git history). Feature status: Present and active in microsoft/amplifier-foundation (v1.0.0) — two-layer defense codified across ISSUE_HANDLING Phases 4–6. Active

Primary sources: microsoft/amplifier-foundation (co-primary) and microsoft/amplifier-bundle-recipes.

Commands run (independently re-verified):

grep -nE 'python_check|Commit locally' amplifier-foundation/context/ISSUE_HANDLING.md — deterministic syntax gate (lines 133/134/458)
grep -nE 'amplifier --version|amplifier --help|CLI tools respond' …shadow-tests.md …smoke-test.md — CLI launch checks
sed -n '226,300p' amplifier-foundation/docs/ISSUE_CASE_STUDIES.md — "11/11 tests passed", commit 87e42ae, "Cannot start sessions"
git cat-file -t 87e42ae → commit (dated 2026-01-15 22:14:56)
grep -nE 'run SEPARATELY|Validation Rubric|Total >= 75' …amplifier-smoke-test.md — 100-point rubric, PASS ≥ 75
sed -n '341,363p' …ISSUE_CASE_STUDIES.md — "<30 minutes", "unblocked within 1 hour"
git log --format='%an' … → 24 commits, sole author Brian Krabach

Gaps (disclosed): No explicit ruff invocation is configured in this repo (pyright is, at pyproject.toml line 72); "python_check" is a named gate whose exact commands are not defined inside amplifier-foundation. No single PR number literally passing static+launch checks then breaking was found beyond the documented P0 case study.

Primary contributor: Brian Krabach (24/24 commits across these files).