The Tool Tests Itself

A regression suite for the Amplifier CLI

The Problem

Keeping the whole Amplifier CLI from silently breaking is the real problem

The CLI spans three AI providers and a wide set of capabilities. Any of them can regress quietly with the next change.

So how much of that surface is actually covered today?

provider response bash execution agent delegation web search web fetch PDF extraction recipe list bundle context AGENTS.md injection

It Runs Today

One command drives 11 invocations over 9 tasks across 3 providers

The regression benchmark config schedules 11 test invocations over 9 distinct task definitions, spanning anthropic, openai, and gemini.

Real, verified scope — but why not just script it by hand?

11

test invocations (8 + 1 + 1 + 1)

9

distinct regression tasks

3

providers: anthropic, openai, gemini

1

command against a YAML benchmark file

Why Not By Hand

By-hand string matching would flake on non-deterministic LLM output

Custom plumbing across every provider and capability is weeks of work — and brittle assertions break the moment a model phrases an answer differently.

The fix starts with not building it from scratch.

✕

Brittle — exact string matches flake on valid but reworded answers

✕

Wide — every provider and capability needs its own plumbing

✕

Slow — weeks of custom test scaffolding to maintain

The Foundation

The suite stands on Microsoft's eval-recipes, pinned to v0.0.34

eval-recipes is Microsoft's evaluation framework. Rather than reinvent grading and orchestration, the project depends on it directly.

That framework is what makes semantic grading possible.

# pyproject.toml dependency
eval-recipes @ git+https://github.com/
  microsoft/eval-recipes@v0.0.34

Semantic Grading

Each test is judged by meaning, not by matching strings

All 9 regression tests use eval-recipes' semantic_test with an LLM rubric — e.g. provider_responds scores file-exists 40 pts + correct-content 60 pts, summed 0–100.

Grading is solved; next is isolation and speed.

9/9

tests use semantic_test + an LLM RUBRIC

0–100

rubric score (40 + 60 for provider_responds)

Isolation & Speed

Every agent runs in its own Docker image, in parallel

8 of 8 agent configs carry a dockerfile_portion that installs the Amplifier CLI fresh; Docker is a stated prerequisite. A --max-parallel flag runs tasks concurrently.

Which raises the twist behind the whole harness.

8/8

agents build a Docker image installing the CLI

~12–15

default --max-parallel (README 12 / code 15)

The Loop

The benchmarking CLI was generated with Amplifier — and now guards it

The CLI commit (30dceab) is co-authored by Amplifier and marked "Generated with Amplifier." The tool built the harness that now tests the tool.

A self-referential loop — and a pattern worth keeping.

Amplifier generates the benchmarking CLI (commit 30dceab)

↓

The CLI runs the regression suite

↓

The suite guards the Amplifier CLI

The Pattern

Seeded from 3 smoke-test tasks, the pattern generalizes

The suite began Jan 26, 2026 with 3 tasks converted from the amplifier smoke-test bundle (c93e50f); CI followed about an hour later (9aee498). Lean on a framework, grade semantically, isolate in Docker.

The same harness already carries 28 full-benchmark and 23 comparison tasks.

3

tasks in the first regression commit (c93e50f)

~1 hr

later, scheduled CI workflow (9aee498)

28

full_benchmark tasks in the same repo

23

comparison tasks in the same repo

Sources

Research Methodology

Data as of: February 2026 (local clone HEAD 634dedc, 2026-02-10)

Feature status: Real and runnable — working Click CLI, regression.yaml config, 9 YAML tasks, 8 Docker agent configs, eval-recipes v0.0.34. Not executed here (no API keys / Docker run).

Repository: DavidKoleczek/amplifier-app-benchmarks (canonical; bkrabach's is a fork).

Commands run:

ls -d data/tasks/regression/*/ | wc -l — 9 regression tasks
cat data/benchmarks/regression.yaml — 11 invocations across 4 agent blocks
grep -n 'provider|--model' data/agents/amplifier_foundation*/agent.yaml — anthropic, openai, gemini
grep -l semantic_test data/tasks/regression/*/test.py | wc -l — 9/9 semantic_test + RUBRIC
grep -l dockerfile_portion data/agents/*/agent.yaml | wc -l — 8/8 agents
sed -n '56,60p' src/amplifier_app_benchmarks/cli.py — --max-parallel default 15 (README 12)
grep eval-recipes pyproject.toml + gh api .../tags/v0.0.34 — eval-recipes @ v0.0.34
git show --stat 30dceab — CLI commit co-authored by Amplifier
git show --stat c93e50f & git log --format='%h %ci %an %s' — first 3 tasks + CI 9aee498

Gaps: Suite not executed in this environment; Docker-run internals inferred from agent dockerfile_portion + README prerequisite + eval_recipes.semantic_test imports. "Hours" is an order-of-magnitude claim about the initial suite; the full 9-task suite spanned Jan 26–28.

Primary contributor: David Koleczek (author of the regression tasks, multi-provider agents, and CLI). Brian Krabach contributed the README/venv-setup commit + PR #1. Amplifier bot co-authored the initial CLI commit.