microsoft/eval-recipes · v0.0.28 → v0.0.31

Which Agent Actually Won?

Blind head-to-head benchmarking in eval-recipes

January 2026

The Frame

eval-recipes benchmarks AI agents — but it could only hand you a score, never a winner

It runs an AI agent on a task and judges the work. Through v0.0.27 that judgment was a solo score against a rubric — useful, but silent on which agent is best.

First, the capability the story is built on.

Before · v0.0.27Score an agent's work against a test.py rubric
The gapA score in isolation never names a winner
The windowv0.0.28 → v0.0.31 closes it

Proof · The AND

Score-based evaluation was already real at v0.0.27

Each agent was judged independently by a test.py script. At the v0.0.27 tag the schema used test_script: Path — and grepping the tree for “comparison” returned nothing.

So the machinery existed. But a lone number has a limit…

v0.0.27

Released 2026-01-13, score-only

0

Comparison files in the tree (grep returned nothing)

test_script

TaskConfig field: Path to test.py

Tension · The BUT

A number in isolation
never names a winner.

Put three agents on one task and score each in isolation, and you still can't say who actually won. The real question a score can't answer: relative to the others, who did best?

v0.0.28 set out to answer exactly that.

Setup · The THEREFORE begins

v0.0.28 answered it in one 3,479-line PR #36

“Add comparison based benchmarking” landed as a single large change: run several agents (minimum 2, any number above) on the same task and rank their outputs.

The next question — how do you make that ranking trustworthy?

#36

PR “add comparison based benchmarking”

+3,479

Insertions (28 files, −153 deletions)

≥2

Agents per comparison, ranks any number

ppt-1/2/3

New sample tasks added to exercise it

Mechanism · Trust

It's genuinely blind — outputs are anonymized before the judge sees them

Each agent's directory is copied to a randomly named folder agent_<hex> so the comparison carries no hint of who produced what.

Anonymized inputs are only half of it — who does the ranking?

# semantic_test_comparison.py · _create_anonymous_workspace()
random_suffix = secrets.token_hex(2)
anon_name = f"agent_{random_suffix}"
shutil.copytree(directory, dest_path)
# "randomized name to ensure blind comparison"

Mechanism · Verdict

A Claude-Agent-SDK judge ranks best-to-worst — and may never touch the work

The COMPARISON_SYSTEM_PROMPT drives an LLM judge to create a relative ranking of the agents from best to worst, under one absolute rule.

Now the verdict needs somewhere to land.

The judgeBuilt on ClaudeAgentOptions / ClaudeSDKClient
Its job“Create a relative ranking of the agents from best to worst”
The absolute rule“NEVER under ANY circumstances change the code or files created by the agent”

Payoff · The THEREFORE

The blind verdict lands in a dashboard with a real concordance score

The HTML report computes Average Rank, Win Rate, and Task Wins — plus a genuine Kendall's W, with LLM explanations per task and across all tasks.

From “here's a number” to “here's who won, blindly, and why.”

Avg Rank

Overall average rank per agent

Win Rate

overall_win_rate (0–100)

Task Wins

Per-task head-to-head wins

Kendall's W

Concordance, 0 (none) → 1 (perfect)

Takeaway

The next releases made the winner usable and debuggable

Shipping the capability wasn't the end. v0.0.29 and v0.0.31 turned a working feature into one you can actually drive and diagnose.

From a number, to a ranked winner, to a winner you can explain.

29v0.0.29 · PR #39YAML run definitions (ScoreRunSpec) replaced CLI filters; filters.py (150 lines) deleted
30v0.0.30Maintenance: version references made consistent across docs and code
31v0.0.31 · PR #44Agent run logs passed through to the semantic test — debug why an agent lost

Sources

Research Methodology

Feature status: Active · v0.0.28–v0.0.31

Data as of: 2026-01-27 (v0.0.31 release date). Primary source: microsoft/eval-recipes, cloned to /tmp/eval-recipes and researched directly.

Commands run:

gh release list -R microsoft/eval-recipes — confirmed v0.0.28 (2026-01-16) → v0.0.31 (2026-01-27)
git show --stat 8522922 — PR #36: 28 files, +3,479 / −153
grep -n 'secrets|token_hex|copytree' semantic_test_comparison.py; sed -n '186,206p' — _create_anonymous_workspace(), secrets.token_hex(2)
sed -n '1,60p' semantic_test_comparison.py — ClaudeAgentOptions/ClaudeSDKClient judge, “NEVER…change the code or files” rule
grep -n 'Kendall|Win Rate|Average Rank' create_comparison_html_report.py; sed -n '171,203p' — _calculate_kendalls_w(), W = 12·S/(m²(n³−n))
git ls-tree -r v0.0.27 --name-only | grep comparison — empty; v0.0.27 schema used test_script: Path
git show --stat a1eaa6d (PR #39) · git show --stat bfd3aa1 (PR #44)
git log v0.0.27..v0.0.31 --format='%an' | sort | uniq -c — 6 David Koleczek + 4 GitHub Action

Primary contributor: David Koleczek (DavidKoleczek) — all human commits and releases in the window.

Gaps: Dashboard interactivity/visual quality inferred from create_comparison_html_report.py (1,469 lines) and release text, not rendered. Version tags live in microsoft/eval-recipes, not amplifier-bundle-recipes.