Catch Broken Recipes Early

A 7-phase validator that gates merges

The Problem

A broken recipe usually fails when it runs, not when it merges

Amplifier teams run real work through recipes — declarative YAML workflows. Three defect classes hide until execution time.

So the question is: can we catch them before merge?

YAML / structural error

Malformed structure or a missing name, version, or step type

Undefined variable

A recipe references a variable that was never defined in its flow

Bad agent namespace

A call to an agent namespace that doesn't exist

Already Shipped

validate-recipes.yaml already ships — real, tested, merged

Merged in PR #50 on 2026-03-14 by Brian Krabach, it emits 51 distinct finding codes and shipped with 117 passing phase tests.

This is a real pipeline — but getting recipe validation right was hard.

51

distinct finding codes

117

passing phase tests

#50

creating PR, merged 2026-03-14

1,671

lines added in the creating PR

Validation Is Hard

Even the engine's own validator carried a blind spot

The engine's validator.py (in the repo since 2025-12-15) never tracked output_exit_code, firing false-positive "variable not defined" errors.

PR #49 fixed it on 2026-03-13 — one day before the pipeline shipped.

1

validator.py initial commit2025-12-15

2

type: bash step type added2025-12-29 — exposed the blind spot

3

PR #49 fixes output_exit_code accumulation2026-03-13

The Pipeline

Seven ordered phases inspect every recipe in a bundle

From environment check through report synthesis, each phase adds a layer of checks the later mechanism and payoff will lean on.

Two of those phases carry the heart of the mechanism.

0

Environment check

1

Discovery

2

Structural validation

3

Best practices

4

Semantic validation

5

Quality classification

6

Report synthesis

Phase 4 — Semantic Validation

Phase 4 flags calls to unknown or bare agent namespaces

It discovers a bundle's local agents from repo_path/agents, builds the known-agent patterns, and catches reference errors that would otherwise blow up at runtime.

And it renders a verdict — deterministically.

UNKNOWN_AGENT · WARNING

A recipe calls an agent that isn't among the discovered known patterns

BARE_AGENT_NAME · WARNING

An agent referenced without its namespace

BROKEN_RECIPE_REF · ERROR

A reference to a recipe that can't be resolved

Quality Classification

Every recipe gets a deterministic verdict — no LLM required

classify_recipe() sorts each recipe into one of four levels straight from its ERROR / WARNING / SUGGESTION findings.

Which sets up the real point: how cheap this gate is.

goodClean bill of health

polishOnly SUGGESTIONs

needs_workWARNINGs present

criticalERRORs present

The Payoff

Mostly deterministic — the LLM fires only as a synthesis gate

10 of the pipeline's 12 steps are plain bash. Clean repos take a bash-only quick-approval fast path and never pay for an LLM call.

Both agent steps use foundation:zen-architect, and both are conditional.

10 / 12

steps are plain type: bash

2

conditional agent (LLM) steps

0

LLM calls for a clean repo's fast path

The Pattern

Broken recipes become pre-merge findings, not production surprises

Wired opt-in (on_error: continue) into amplifier-foundation's validate-bundle-repo.yaml v3.3.0, the gate runs before merge.

A deterministic-first pattern: spend an LLM only when it's actually needed.

Shipped & maintained · v1.0.1

type: recipe

validate-recipes wired as a sub-recipe step

opt-in · on_error: continue

Gated by validation_flags.validate_recipes, timeout 600

validate-bundle-repo.yaml v3.3.0

amplifier-foundation's bundle-repo validator

Sources

Research Methodology

Data as of: 2026-04-22 (softcloud cache of microsoft/amplifier-bundle-recipes @ commit 7963f317, v1.0.1)

Feature status: Shipped and maintained — first merged 2026-03-14 (PR #50), most recently touched 2026-04-07 (PR #60); 8 commits total.

Research performed:

Line count: wc -l recipes/validate-recipes.yaml (2205 lines; 1,671 additions in PR #50)
Phases: grep -n '# PHASE' recipes/validate-recipes.yaml + PR #50 body (7 phases, 0–6)
Step types: grep -nE '^ - id:|^ type:|^ agent:' recipes/validate-recipes.yaml (12 steps, 10 bash, 2 agent)
Finding codes: grep -oE '"code": "[A-Z_]+"' recipes/validate-recipes.yaml | sort -u | wc -l (51)
Tests: grep -rcE 'def test_' recipes/tests/test_phase*.py (6+12+21+21+19+38 = 117)
Validator history: gh api repos/microsoft/amplifier-bundle-recipes/commits?path=.../validator.py; gh pr view 49
PR history: gh pr view 50 --repo microsoft/amplifier-bundle-recipes --json files; PRs #49–#60
Integration: grep -n 'validate-recipes' amplifier-foundation/recipes/validate-bundle-repo.yaml (v3.3.0)
Source-repo negative check: gh repo view bkrabach/develop-pipeline-test (a smoke-test fixture, not this pipeline)

Gaps: "Before merge" is the design intent (opt-in sub-recipe); no CI workflow that auto-runs validate-recipes was found. "Fail silently in production" is narrative framing — repos confirm the failure classes and the pre-merge validator, not that phrasing.

Primary contributor: Brian Krabach (bkrabach) — 7 of 8 commits touching the file, plus the #49 fix. Salil Das — 1 commit (PR #60).