Skip to content
UpSkill Design System

ADR-019 — Screenshot baseline visual regression

Date: 2026-07-14 Status: accepted

Context

Visual regression is the one testing tier below industry practice. A CSS Module change to a shared primitive (Box) or a device-token change can silently shift many components with nothing in CI noticing. The existing gates — lint, typecheck, build, metadata validation, a11y tests — all pass invisibly on pixel-level shifts because none of them inspect visual output. The human visualReview record (ADR-010, Airtable checklist item 2) covers only in-review moments, leaving dozens of merged changes to components un-visually-vetted after they ship.

Issue #71 asks for a deterministic script (no agent, no MCP) that snapshots one canonical story per component (light + dark) into committed baselines, diffs fresh screenshots against them, and offers a one-command approval path after an intentional visual change.

Decision

Committed PNG baselines of each component’s canonical --default story, compared via perceptual diff (pixelmatch, threshold 0.1), generated by a pure script (npm run screenshot:check|approve), advisory CI step, Playwright on-demand.

Naming and non-decisions

  • screenshot:check / screenshot:approve, not visual:*. visualReview is the human-led judgment record (JSON stored in .claude/component-review-state.json, written by /add-component and /layout-generation, rendered as checklist item 2 by sense.js); a machine-led visual:check beside it would blur human vs machine. Rejected: baseline:* (collides with the repo’s pervasive “frozen-memory baseline” vocabulary — airtable-governance.json, token-usage.json, etc.) and regression:* (less specific about mechanism). screenshot:* is unused and names the mechanism exactly.
  • No new lifecycle checklist item. The four-item in review checklist (reviewChecklist() in scripts/sense.js:256) stores no per-component state beyond reviewPath/learningsBackfilled/visualReview; the screenshot check folds under checklist item 1 (“Automated gate”). Leave item 1’s label unchanged while the CI step is advisory (a label edit would cascade to docs-check staleness in docs/02 and docs/08); revisit when promoted to blocking.
  • Playwright stays on-demand. The script dynamic-imports Playwright and prints install instructions if absent; package.json does not list it. CI installs it explicitly. One browser instance drives all 54 shots (light+dark for 27 components).
  • Enumeration is directory-driven, not prefix-driven. Component stories live under three title prefixes (Components/, Layout/, Typography/), so filtering index.json on components-* alone misses the layout and typography primitives — the original plan assumed one prefix and its own story-count guard caught the error on first run. Each directory in packages/components/src/components/ must match exactly one --default story by name under any prefix; showcase stories (layout-examples-*) match no directory and fall out naturally.
  • Two-tier diff ratio. Same-machine renders are fully deterministic (a clean check reports 0.000% on every shot), so the local default tolerance is 0.01% of pixels. Calibration during verification showed why 1% is far too loose locally: a 6px button padding change moves only ~0.05% of a full 1440×900 page, so a 1% ratio silently passes exactly the regression class the tool exists to catch. The CI step sets SCREENSHOT_DIFF_RATIO=0.01 (1%) because ubuntu antialiasing differs from the macOS-generated baselines.
  • Deliberately not Chromatic. A paid hosted service is disproportionate for a solo lite system, and a cloud service brings rate limits and service-dependency risk. Committed PNGs + local pixelmatch is cheaper, self-hosted, and explicit in git history.

Consequences

  • Baselines in git history. packages/components/screenshots/ contains 54 PNG files (27 components × light+dark), adding ~200–300 KB to the repo per generation. Git LFS is not configured and not planned — the repo stays standalone and cloneable. Large diffs when baselines update are expected and acceptable.
  • Cross-OS antialiasing drift. Ubuntu font rendering in CI differs from macOS locally. The CI step runs with continue-on-error: true (advisory only) at a loosened 1% ratio; adoption is measured per PR by reading diff ratios. Escalation path: if even 1% is too strict on ubuntu, move to CI-generated baselines (CI becomes the reference, local becomes the canary); once ratios are quiet, promote the step to blocking.
  • Full-page shots make content-height changes a dimension FAIL. Changing line-height, padding, or margin inside a component shifts the full-page height and triggers a FAIL even if the component itself looks identical. Both sizes are printed so the developer can judge intent. This is deliberate — catching unintended layout shifts — but surprises developers who expect smaller crops.
  • No new gate or workflow moment. The script is purely deterministic (no agent, no MCP); approval is a one-liner after a visual change, no new command moment. The refresh lives in the optional per-run loop (npm run screenshot:approve -- --component <Name> after a change, inline with the PR).
  • ~2–4 min added CI time (Playwright install + Storybook build + screenshot run), one-time per workflow. Browser caching and parallel screenshot batches are future optimization paths; captured here only to set expectation.