Teaching AI to Follow Through

When clearer instructions beat code

A Canvas Specialists case study

The Promise

You should get a finished brief, not the machinery

Canvas Specialists chains domain-expert AI agents — researcher, writer, and more — so a user receives one finished, sourced document instead of the raw machine output behind it.

The whole value rides on that chain running end to end. So does it?

It's Real and Shipping

Twelve specialist agents run in the bundle today

The chain is realized as concrete agent files — anchored by a researcher whose structured JSON is the single source of truth, required upstream and forbidden to fabricate.

A trustworthy sourced brief only works if the chain finishes. It didn't always.

12

specialist agent files, incl. story-researcher, technical-writer, case-study-writer

1

researcher: REQUIRED upstream; its JSON is the single source of truth

0

fabricated metrics — missing evidence is reported, never invented

The Break

The chain kept quitting at the researcher

Roughly 40% of the time the coordinator halted at the researcher and surfaced raw structured research blocks instead of completing the chain to the writer.

Which broke the whole promise: users got the machinery, not the document.

~40%

of runs: coordinator halted early and dumped raw research blocks

Raw

machine output surfaced instead of a finished, written brief

The Path Not Taken

The obvious fix was code — the team reached for instructions instead

Routing logic, validation layers, retry mechanisms were all on the table. Instead of building machinery, the team decided to fix the words the coordinator reads.

So where did the change actually live?

Considered: routing logic
Considered: validation layers
Considered: retry mechanisms
Chosen: clearer instructions

Where the Fix Lived

One natural-language instruction file — zero lines of logic

The entire change was 129 insertions and 16 deletions of instruction text. No application logic, no routing engine, no validation code.

The leverage was words in a spec. But words alone still left a gap.

# the whole change
129 insertions
16 deletions
# zero lines of
# application logic

The Loophole

Instructions alone scored 4 out of 5

A five-scenario smoke test still failed one case: the coordinator found a loophole on casual phrasing like "tell me about X" and quit early anyway.

The ambiguity had to be closed — and that took one more sentence.

4/5

smoke-test scenarios passed after the first instruction rewrite

"tell me
about X"

casual phrasing the coordinator exploited to quit early

Smoke-test scores are deck-attested from a test log that is not present locally; treat as directional.

The One-Sentence Fix

"When in doubt, chain." closed the last loophole

A routing decision table with concrete examples plus that single tiebreaker sentence took the smoke test to 5/5 and lifted assessed chain reliability from ~40% to >90%.

Not a routing engine. A sentence.

~40%

assessed reliability before the fix

>90%

assessed reliability after the fix

5/5

smoke-test scenarios passing post-fix

~40% and >90% are qualitative assessments from repeated use, not controlled A/B tests — the deck's own caveat.

The Pattern

When an agent chain misbehaves, teach it

The most impactful change often isn't code — it's clearer instructions. A routing table and one tiebreaker sentence did what a routing engine would have.

Kept honest: the reliability figures are qualitative assessments from repeated use, not controlled A/B tests.

Sources

Research Methodology

Subject: The "Canvas Specialists" bundle case study, as documented in the amplifier-stories bundle (ramparte/amplifier-stories).

Feature status: Shipped / documented as a case study; the specialist-agent chain is present and active in the bundle.

Research performed:

Deck existence & size: wc -l docs/story-teaching-ai-to-follow-through.html → 917 lines
History & authorship: git log --format='%h %an %ad %s' --date=short -- docs/story-teaching-ai-to-follow-through.html → added 2026-03-06 by cpark4x; scrubbed 2026-03-11; regenerated 2026-07-14
Agent inventory: for f in agents/*.md; do grep -m1 'name:' "$f"; done ; ls agents/*.md | wc -l → 12 specialist agent files
Researcher contract: head -40 agents/story-researcher.md ; grep -niE 'REQUIRED|single source of truth|never fabricate' agents/story-researcher.md
Claim extraction: grep -oiE '(~40%|>90|routing decision table|tiebreaker|zero lines)[^<]*' docs/story-teaching-ai-to-follow-through.html
Source-repo scan: git cat-file -t 9bbb3ba / 521c1a0 across all local .git dirs → NOT FOUND

Gaps: The underlying "canvas-specialists" source repo (context/specialists-instructions.md, PR #16, commits 9bbb3ba/521c1a0) and the smoke-test log are not checked out locally. The ~40%/>90% reliability figures and 4/5 & 5/5 smoke-test scores are deck-attested qualitative estimates, not independently measured.

Primary contributors: Chris Park (cpark4x) — product direction and the case-study deck; Michael J. Jabbour — introduced the specialist agent architecture; Sam Schillace — regenerated the deck spine-first.

Active · Shipped as a case study