Reality Check · Technical Walkthrough

The Convergence Gate

How Reality Check turns intent into evidence

microsoft/amplifier-bundle-reality-check · v0.2.0 · Alpha

The Problem

Reality Check tests whether built software actually matches user intent

It is an Amplifier bundle that verifies delivered software against what was asked — instead of trusting an agent that just says “done.”

So how do you trust a verdict when an LLM produced it?

The claim: the agent reports the work is complete.
The gap: “done” is a model's word, not evidence.
The job: confirm intent became working behavior.

Verified Live

One CLI call: exit 0 on valid, exit 1 on invalid

A deterministic checkpoint, amplifier-reality-check, sits between the LLM steps. A bad software_type is rejected by Pydantic — no model opinion required.

A non-LLM gate exists. Now — what is it gating?

# valid acceptance-test suite $ validate-acceptance-tests valid.yaml "valid": true → exit 0 # software_type: mobile_app (invalid) $ validate-acceptance-tests invalid.yaml literal_error: web_app, cli_tool, api_service or library → exit 1 # contract: 0 valid / 1 invalid / 2 error

The Pipeline

Five LLM agents drive the reality-check pipeline

The intent-analyzer writes acceptance tests; three testers run them; the report agent consolidates the results into report.raw.yaml.

Five agents, all authored by an LLM — and that's the weak point.

1intent-analyzer — derives acceptance tests from the spec
2browser-tester — runs browser-type tests
3terminal-tester — runs CLI-type tests
4generic-tester — runs the remaining tests
5report — consolidates results into one raw report

The Weak Point

An LLM writing tests or a report can drift or emit malformed YAML

When the model authors the acceptance tests and the report, its “looks good” proves nothing — the output can wander from the schema or simply be invalid.

The fix isn't a smarter model. It's a checkpoint the model can't argue with.

Gate One

validate-acceptance-tests enforces a strict Pydantic schema

extra='forbid', strict=True forbids unknown keys and refuses to coerce types. Each test needs an 8-char hex id, a valid type, and at least one step.

The intent hand-off is gated. Now gate the report.

extra='forbid' — unknown keys are rejected
strict=True — no silent type coercion
id — pattern ^[0-9a-f]{8}$, auto-injected & idempotent
software_type — web_app, cli_tool, api_service, library

Gate Two

validate-report is the sole producer of the canonical report

It buckets every test into passed / failures / missing, each with an evidence string, and computes a pass_rate like 3/5.

Both authoring hand-offs are gated — but who enforces it?

passed

status = pass, evidence attached

failures

status = fail, evidence attached

missing

no valid raw entry found

3/5

pass_rate = passed / total

The Convergence Gate

The recipe enforces convergence — the model doesn't promise it

Both LLM stages run in while-loops capped at 3 attempts. A bash guard runs exit 1 — failing the whole recipe — if the CLI never exits 0.

Intent becomes evidence-backed pass/fail, not a model's word.

# both loops in reality-check-pipeline.yaml max_while_iterations: 3 while_condition: {{validated}} != true break_when: {{validated}} == true # guard step after the loop ensure-validated: exit 1 # failed CLI validation # after 3 iterations

The Pattern

Put a deterministic gate between LLM hand-offs — and let the recipe enforce it

Don't ask the model to be trustworthy. Wrap each authoring hand-off in a machine-checkable checkpoint and loop until it passes. That's how intent becomes evidence you can trust.

Convergence is a recipe property, not a model promise.

Sources

Research Methodology

Primary sources: microsoft/amplifier-bundle-reality-check (the capability) and bkrabach/reality-check-greeting-app (the 85-line Flask test fixture).

Feature status: Development Status :: 3 — Alpha; tool version 0.2.0 (bundle 0.2.1); authored by the Microsoft MADE:Explorations Team.

Commands run:

ls agents/ and cat AGENTS.md, behaviors/reality-check.yaml — confirmed the 5 LLM agents
uv run amplifier-reality-check --help — 3 subcommands (validate-acceptance-tests, validate-report, schema)
validate-acceptance-tests on valid + invalid YAML; echo $? — observed exit 0 / exit 1
sed -n '1,90p' src/…/acceptance_tests.py — ConfigDict(extra='forbid', strict=True)
grep of report.py — passed/failures/missing buckets and pass_rate formula
cat recipes/reality-check-pipeline.yaml — both loops max_while_iterations: 3 + guard steps
uv run pytest -q — 89 collected, 88 pass, 1 stale version assertion
git log / gh repo view / git shortlog -sne --all — timeline & contributors

Contributors: David Koleczek (25 commits, lead), Brian Krabach (5 commits + sole author of greeting-app), Manoj Prabhakar Paidiparthy (3 commits). 26 commits total in the tool repo.

Gaps: No run log of Reality Check executed end-to-end against the greeting-app was found — only the fixture code and the tool's own test suite were verified. CLI checkpoints gate the two YAML-authoring hand-offs (acceptance tests, report), not every agent invocation.