Scores You Can Trust

How a rebuilt harness earns its numbers

The Stakes

A score is only worth reading if you can trust it wasn't stolen by a Docker hiccup

This deck isn't about posting a number — it's about earning one. When infrastructure flakiness can masquerade as an agent failing, the score stops meaning anything.

So the first question is simple: is this a real contest at all?

The Contest

Amplifier Foundation runs head-to-head against three other agents on the same tasks

Every comparison task lists the same four agent IDs — this is an apples-to-apples run, not a self-report.

And the workload behind those runs is concrete.

1amplifier_foundation — amplifier run --bundle foundation

2claude_code

3gh_cli — GitHub Copilot CLI

4openai_codex

The Test Bed

The workload is concrete: real office tasks, repeated, plus scored tasks

23 business and office-document tasks run 7 times each, plus 28 scored tasks — all judged on a 0-100 scale where a perfect score is 100.

Which is exactly where the catch hides.

23

office / business tasks

7×

runs per comparison task

28

scored tasks per agent

0–100

score scale

The Catch

A crashed container looks exactly like the agent failing

A Docker issue, an API timeout, or a network error can be indistinguishable from a real agent mistake — and quietly poison the score.

So at v0.0.32 the team did something drastic about it.

×Docker issues — container fails to start

×API timeouts — external service stalls

×Network errors & missing base-image deps

The Rebuild

They threw out the old harness in a single 188-file rewrite

At v0.0.32 (PR #45, David Koleczek, Feb 3 2026) the old Harness and ComparisonHarness were removed and benchmarking was rebuilt on a DAG job runner with parallel execution.

That runner does one crucial new thing with every failure.

188

files changed in PR #45

+8224

lines added

−6925

lines removed

DAG

job runner replaces Harness

Classification

Every failure is read and sorted into one of three categories

The analysis runner classifies each failure — not lumping them together — so the harness can tell an agent's mistake apart from the environment's.

And that distinction drives the one rule that makes the score honest.

AAGENT_ERROR — the agent made a mistake

IINFRASTRUCTURE_ERROR — Docker, API timeout, network, missing deps

TTEST_ISSUE — flaky test or incorrect expectations

The Payoff

A trial is discarded ONLY when the failure is infrastructure

valid_trial is set to false only for INFRASTRUCTURE_ERROR. Infra noise never lands on an agent's score — so the number you read is the number the agent earned.

And that rule isn't a convention — it's baked into the data.

# analysis_runner.py
Set VALID_TRIAL to false
ONLY for INFRASTRUCTURE_ERROR
failures.

# AGENT_ERROR  → trial counts
# TEST_ISSUE   → trial counts
# INFRA_ERROR  → trial discarded

What To Keep

Trust isn't a headline — it's a field in the schema

valid_trial and failure_category are first-class fields in the benchmark result schema. Credibility is engineered into the data model, so trustworthiness ships with every result.

The durable pattern: don't assert trust — build it into the record.

# schemas.py
valid_trial: bool
  # False if failure was
  # infrastructure-related
  # (Docker, API, timeout)

failure_category: str | None
score: float  # 0 to 100

Sources

Research Methodology

Data as of: February 2026. All metrics independently re-derived from command output in a fresh isolated session.

Feature status: microsoft/eval-recipes (MIT, "very early"), benchmarking rewritten at v0.0.32 (2026-02-03); latest verified tag v0.0.36 (2026-02-10). DavidKoleczek/amplifier-app-benchmarks v0.1.0 pins eval-recipes@v0.0.34.

Commands run:

gh api repos/microsoft/eval-recipes/tags --jq '.[].name' — tags v0.0.32–v0.0.36 exist
gh pr view 45 --repo microsoft/eval-recipes --json title,author,mergedAt,additions,deletions,changedFiles — +8224/−6925, 188 files, David Koleczek, merged 2026-02-03
gh release view v0.0.32 --repo microsoft/eval-recipes — complete rewrite; Harness/ComparisonHarness removed
gh api .../job_framework/runner.py?ref=v0.0.36 — "Job runner with DAG resolution and parallel execution"
gh api .../evaluation/analysis_runner.py?ref=v0.0.36 — AGENT_ERROR / INFRASTRUCTURE_ERROR / TEST_ISSUE; VALID_TRIAL false only for infrastructure
gh api .../schemas.py?ref=v0.0.36 — valid_trial, failure_category, score 0–100
cat data/benchmarks/full.yaml — agent_ids [claude_code, gh_cli, amplifier_foundation, openai_codex]; comparison_runs: 7
grep -c 'task_name:' full.yaml — 23 comparison tasks; ls data/tasks/full_benchmark | wc -l — 28 score tasks

Primary contributor: David Koleczek (DavidKoleczek) — top contributor to microsoft/eval-recipes and author of the benchmarking rewrite (PR #45) and amplifier-app-benchmarks.

Gaps: No results artifact (win rankings, per-task scores, or aggregate percentages) was found in the accessible repos; this deck therefore makes no claim about who won or by how much — only about how the harness earns a trustworthy number.

Status: Active · eval-recipes v0.0.36 · MIT