The 50% Rule

Why AI validators must verify against real code

The stakes

AI writes validator rules in seconds. The hard part is trusting what they flag.

In microsoft/amplifier-foundation, four validate-* recipes check bundles and agents. Generating a rule is cheap. But a validator only helps if its verdict can be believed.

So the real question: were the generated rules actually right?

4

validate-* recipes in the foundation (validate-agents, -bundle, -bundle-repo, -single-bundle)

The measured complication

Half the bundle validator's first rules were wrong against real code.

The Domain Validator Authoring Guide records it plainly: "50% of initial rules were inaccurate when traced against actual code behavior." Plausible-sounding is not the same as correct.

What did those wrong rules actually claim?

50%

of the bundle validator's initial rules were inaccurate when traced against actual code — DOMAIN_VALIDATOR_GUIDE.md, line 121

The wrong rules were concrete

Two rules that sounded right — and contradicted the actual parser.

Commit 0ec3efd ("align validator recipes with actual code behavior", Brian Krabach, 2026-01-28) removed and downgraded rules that the code disproved.

Wrong rules are only half the trust problem, though.

Removed"Inline agents are deprecated" — the code supports both formats, so the check was deleted.
Downgraded"DRY pattern required" — only a convention, so it was marked RECOMMENDED, not required.

The "Always Finds Something" anti-pattern

With no PASS threshold, a validator always finds something — and becomes noise.

The guide names it: "Without explicit PASS thresholds, validators will always find something to suggest. This creates user frustration and distrust." "PASS WITH SUGGESTIONS feels like failure."

The fix has two halves — start with what earns each rule its severity.

Symptom: PASS WITH SUGGESTIONS feels like failure.
Symptom: Unclear when the work is actually done.
Result: user frustration and distrust — the verdict gets ignored.

Fix, half one: verify against code

Let the code decide severity — not a plausible-sounding assumption.

A litmus test sets the level: it's an ERROR only if the code actually raises or fails, a WARNING if only docs enforce it, otherwise a SUGGESTION.

Verifying rules is half the fix. The other half is knowing when to stop.

ERRORCode raises an exception / validation fails.
WARNINGNo code enforcement, but a documented pattern.
SUGGESTIONExpertise-based advice, nothing enforced.

Fix, half two: explicit PASS thresholds

Define "good enough" so a clean agent gets a clean bill of health.

The validate-agents recipe spells out four explicit PASS criteria. Meet them all and the agent passes clean — the validator stops finding things.

But thresholds only earn trust if a deterministic check applies them first.

Strong trigger: MUST / ALWAYS / REQUIRED / PROACTIVELY / DO NOT.
At least one <example> block.
Description is 100 characters or more.
Explicit tools section (even if empty []).

Deterministic first, AI second

A bash Phase 2.5 sets the PASS verdict before any LLM runs.

In validate-agents, Phase 2.5 "Quality Classification" is a deterministic bash/Python step. If every agent passes, it sets requires_llm_analysis = False — and the LLM phases are skipped.

The verdict is earned by code, so the validator's word is trusted again.

# validate-agents.yaml — Phase 2.5 (bash, deterministic)
id: quality-classification
type: bash   # runs python3, no LLM

# all agents meet PASS thresholds →
requires_llm_analysis = False

# LLM phases are gated on the flag:
condition: requires_llm_analysis == false  # quick-approve
condition: requires_llm_analysis == true   # detailed / expert

The portable principle

AI can write the rules. Only real code can rule on them.

Verify every rule against real code, set explicit PASS thresholds, and run deterministic checks first, AI second — that's how a validator earns back trust.

Sources

Research Methodology

Shipped & iterated on main

Source repo: microsoft/amplifier-foundation (origin); ramparte/amplifier-foundation is the "fork" remote. Verified on local branch fix/session-naming-timeout-configurable at HEAD 0965a34.

Data as of: 2026-04-09 · Status: DOMAIN_VALIDATOR_GUIDE.md is the "definitive reference"; validate-agents.yaml at version 1.2.4.

Commands run:

git remote -v — confirmed origin = microsoft/amplifier-foundation.
grep -n '50%' docs/DOMAIN_VALIDATOR_GUIDE.md — line 121: "50% of initial rules were inaccurate."
grep -n 'Always Finds\|distrust' docs/DOMAIN_VALIDATOR_GUIDE.md — lines 37, 39, 43-46.
git show --stat 0ec3efd — "align validator recipes with actual code behavior" (Brian Krabach, 2026-01-28).
sed -n '233,265p' docs/DOMAIN_VALIDATOR_GUIDE.md — ERROR/WARNING/SUGGESTION table + litmus test.
sed -n '38,42p' recipes/validate-agents.yaml — four explicit PASS thresholds.
grep -n 'PHASE 2.5\|quality-classification\|requires_llm_analysis' recipes/validate-agents.yaml — lines 582, 587-588, 702, 809.
wc -l amplifier_foundation/validator.py tests/test_validator.py — 305 / 254 lines.
git shortlog -sn (guide + validate recipes) — 27 Brian Krabach / 1 Diego Colombo / 1 Salil Das.

Primary contributor: Brian Krabach (27 of 29 combined commits, brkrabac@microsoft.com); also Diego Colombo (#123) and Salil Das (#151).

Gaps: "The 50% Rule" name and the And/But/Therefore framing are the storyteller's synthesis — the guide states "50% of initial rules were inaccurate" but not the phrase "The 50% Rule." "Ignored as noise" paraphrases "user frustration and distrust." PR merge status (#123/#127/#151) taken from local git log subjects only, not confirmed via gh.