Amplifier UX Analyzer

Pixels Into Structure

Vision pipeline that lets agents read a UI

The gap

An agent can code a UI from a screenshot — but sees only opaque pixels

AI agents already generate UI code from a screenshot. Yet to a model that screenshot is opaque pixels — no addressable colors, regions, or text. Amplifier UX Analyzer (ramparte/amplifier-ux-analyzer) gives it structure instead.

So what does that structure actually look like coming out?

→

Input: a UI screenshot — pixels only

→

Pipeline: four-stage OpenCV / EasyOCR vision analyzer

→

Output: structured JSON an agent can address

Proof

analyze() emits one structured JSON object straight from a screenshot

The core analyze() pipeline assembles a single result dict — {metadata, colors, regions, text_elements} — and save_json() writes it with json.dump(indent=2). Structured output isn't a plan; it already exists in the code.

Which matters because a raw screenshot gives a model none of this.

result = {
  "metadata": {source, dimensions},
  "colors": [ ... ],
  "regions": [ ... ],
  "text_elements": [ ... ]
}
# save_json() -> json.dump(indent=2)

The tension

Raw pixels give a model nothing to point at

A screenshot is a grid of pixels. There is no palette to reference, no layout bands to name, no readable text, no coordinates. Every one of those handles has to be recovered before an agent can address the design.

Each pipeline stage exists to rebuild exactly one of these handles.

✕

No addressable color palette

✕

No named layout bands

✕

No readable text

✕

No element coordinates

Stage one — color

k-means recovers ten dominant colors as addressable design data

get_dominant_colors() runs cv2.kmeans with KMEANS_PP_CENTERS, default n_colors = 10 (OpenCV, not scikit-learn). Each color is emitted as {hex, rgb, frequency} — a palette the agent can reference by value.

Color recovered — next, where things sit on the screen.

10

dominant colors (default n_colors)

cv2.kmeans

OpenCV, KMEANS_PP_CENTERS

Stage two — regions

Canny edges split the frame into three addressable layout bands

detect_regions() uses cv2.Canny(gray, 50, 150) and fixed height thresholds at 15% and 85% to return three bands — toolbar, content, status_bar — each with bounds and color. UI element detection then runs on the toolbar region via contours, classified by aspect ratio.

Structure recovered — one handle still missing: the words.

1

toolbar

0–15% of height

2

content

15–85% of height

3

status_bar

85–100% of height

Stage four — text

EasyOCR reads the on-screen text with confidence and position

extract_text() calls easyocr.Reader(['en'], gpu=False).readtext() and emits each item as {text, confidence, bounds}. OCR is optional — the HAS_OCR flag returns [] when unavailable — recovering the last missing handle: words plus coordinates.

With color, regions, and text in hand, the payoff closes the loop.

# easyocr.Reader(['en'], gpu=False)
{
  "text": "...",
  "confidence": float,
  "bounds": {x, y, width, height}
}

The payoff — roundtrip

The JSON becomes a spec, then HTML that iterates against the reference

A roundtrip recipe wires it end-to-end: SpecConverter turns the JSON into a YAML spec, then CodeGenerator (default claude-sonnet-4-20250514) generates HTML — iterating up to 3 times (max_iterations) against the reference. Structured data, not pixels, drives the regeneration.

That is the whole point in motion.

1

analyze

screenshot → structured JSON

2

convert

SpecConverter → YAML spec

3

generate

claude-sonnet-4 → HTML, up to 3 iterations

The pattern

Address the data, don't guess the image

Give a model structure instead of pixels and UI generation becomes deterministic input, not guesswork. That is the pattern Amplifier UX Analyzer turns into a working v0.2.0 pipeline.

Sources

Status: Working prototype / v0.2.0

Research Methodology

Primary source: ramparte/amplifier-ux-analyzer (personal account, not microsoft/*), checked out at ~/dev/ANext/amplifier-ux-analyzer. Pipeline claims grounded in source-code reading, not a live run.

Repository: 3 commits, single commit author Sam Schillace; created 2026-02-02, last pushed 2026-06-29.

Commands run:

sed -n '283,322p' amplifier_ux_analyzer/core/analyzer.py — analyze() four-stage pipeline & result dict
grep -n 'kmeans|def get_dominant_colors' amplifier_ux_analyzer/core/analyzer.py — cv2.kmeans, 10 colors
sed -n '91,182p' amplifier_ux_analyzer/core/analyzer.py — three bands (15% / 85%), cv2.Canny
grep -n 'easyocr|readtext' amplifier_ux_analyzer/core/analyzer.py — EasyOCR {text, confidence, bounds}
sed -n '1,60p' amplifier_ux_analyzer/recipes/roundtrip.yaml — roundtrip, max_iterations: 3
gh api repos/ramparte/amplifier-ux-analyzer/commits — commit log, dates, author
find amplifier_ux_analyzer -name '*.py' | xargs wc -l — package line counts, version 0.2.0

Gaps / not verified: README performance figures (~3–6s/screenshot, GPU 10–50× speedup) are documentation claims, not benchmarked here; the pipeline was not executed end-to-end; local .git is corrupt ('bad object HEAD'), so history came from the GitHub API. README line 183 mislabels color extraction 'scikit-learn'; the code uses OpenCV cv2.kmeans (authoritative).

Primary contributor: Sam Schillace (sole commit author of all 3 commits).