Five words that beat three blind rounds
A team built an AI Storyteller specialist and set out to validate it. Validation is the whole point — and the hard part is telling real proof from a convincing appearance.
So which did they end up with?
10 narrative traps were planted. 10 were excluded — confirmed, not projected. Claims came from findings only, and every statement traced.
But getting there was anything but clean.
Canvas-specialists is a bundle of 6 AI specialist agents — a Researcher, a Formatter, a Data Analyzer, a Competitive Analyst, a Writer, and the newest addition: the Storyteller.
To test it: one frozen 8-document evidence pack, randomized labels, two independent blind judges per run.
[Doc X:LN-LN] citations, randomized X/Y/Z labelsThe blind A/B/C tournament ran the same on-device SLM strategy memo three times. Every variant won exactly once.
If more rounds only muddied the answer, what was the testing actually telling them?
| Run | A | B | C |
|---|---|---|---|
| Run 1 | 35.5 | 34.0 | 46.5 |
| Run 2 | 42.5 | 33.0 | 41.5 |
| Run 3 | 44.5 | 46.5 | 43.5 |
Every variant won once. That's noise, not signal.
Cross-run aggregation of 18 scorecards found a single significant result: Narrative Memorability at p=.011 — Storyteller 4.83 vs Writer-only 3.67 (of 5), Cohen's d > 2.0.
One signal in a mountain of testing. So why did they keep testing?
Run 1 told them: the Storyteller adds narrative lift but bleeds precision when its output feeds the Writer. Runs 2 and 3 generated noise because n=6 is underpowered.
Three rounds to characterize a problem one round already named. More testing wasn't the path.
“How do you know they are all 3 star? Is this confirmed through testing or theoretical?”
Live smoke tests split the result. The fragile part was the old most-used chain, not the new specialist — and the real bug was one missing input type: the Writer had no story-output case.
Solved with one new input type — because someone asked how do you know, and got an honest answer.
Data as of: March 6, 2026 · Status: Narrative artifact / validation case study in the amplifier-stories bundle (live).
Primary source: docs/story-how-do-you-know.html in the checked-out repo ramparte/amplifier-stories, with siblings docs/story-building-the-storyteller.html and case-studies/canvas-becomes-a-specialist-platform.html.
Research performed:
git log --all --follow docs/story-how-do-you-know.html — staged 2026-03-06 by cpark4x (7f5c3f5), promoted 2026-03-11 by sadlilas (95376b8), regenerated spine-first 2026-07-14 by Sam Schillace (0a5aca4).grep confirmed every deck string (10/10, 15/15, 24/24, run scores, p=.011, 4.83 vs 3.67, "story-output", "347 words") present in the deck.grep ai_working/pipeline-batch/corpus-source-map.tsv — self-referential meta-story validating this bundle's own Storyteller specialist (confidence high).Methodology (as stated by the deck): 3 blind runs × 2 independent judges × 3 variants = 18 scorecards; frozen 8-document evidence pack (Doc A–H); randomized X/Y/Z labels; 5 rubric criteria scored 1–5; same scenario (on-device SLM integration strategy memo); Kruskal-Wallis omnibus p=0.352, p=.011 Narrative Memorability; power analysis ~26 scorecards/variant.
Gaps: The canvas-specialists source (agents, blind A/B/C harness, cited PR #16 / commits 9bbb3ba and 4fe5214) is not checked out under this tree and those commits do not resolve via git — treated as deck-asserted references, not source-verified. All tournament statistics are asserted by the deck; the raw scorecards/dataset were not independently re-derivable. Deck's own noted open items: Competitive Analysis format compliance not yet re-addressed; single-domain test scenario; chain smoke tests not independently re-verified.
Primary contributor: Chris Park / cpark4x (product direction, all decisions) with Amplifier AI (analysis, implementation, testing); deck promoted by sadlilas and regenerated spine-first by Sam Schillace.