Skip to content
let-sunny edited this page Apr 18, 2026 · 12 revisions

Measuring AI implementation quality of Figma designs through controlled experiments.

Setup — Install and configure canicode (CLI, MCP token, Claude Code skills) Decision Log — Every major decision with context and rationale Rule Reference — Complete detection conditions, subTypes, and exemptions for all 16 rules Scoring Model — How scores are calculated, severity levels, grade thresholds Calibration — Pipeline structure, agents, debate highlights, fixture analysis Delta-driven Pipeline — How calibration + rule discovery close the feedback loop Calibration 01 — First full calibration: 10/24 fixtures, responsive comparison, avg 93.7% similarity Calibration 02 — size-constraints responsive strip: 6/24 fixtures, missing-size-constraint -5→-8, raw-value -3→-4

Experiments

# Title Key Finding Date
01 Ablation Pilot ΔT is noise, ΔO is the signal (small fixture only) 2026-03-26
02 Large-scale Ablation Pilot conclusions reversed. Structure is dominant. ΔV is strongest signal 2026-03-27
03 Input Method Comparison design-tree >> raw JSON. MCP direct render = 94% 2026-03-27
04 Viewport Test Responsive reveals structure's true value. bad-structure drops -23%p at 500px 2026-03-27
05 Ablation Phase 1 Strip experiments across 3 desktop fixtures 2026-03-28
05b Ablation Phase 2 Generalization across 6 additional fixtures incl. mobile 2026-03-28
06 Calibration Data Validity 3 of 6 responsive deltas invalid — size-constraints strip consistently hurts or is neutral 2026-03-31

Key Numbers

  • design-tree @500px: 95% similarity (root width fix only, flex expands naturally)
  • MCP Vite @500px: 92% (good, hardcoded widths), 75% (bad-structure)
  • MCP requires LLM post-processing for responsive — hardcoded w-[375px] in child elements
  • design-tree is inherently responsive — root width removal is the only change needed
  • Structure (auto-layout) is the only category that affects responsive — token, component, naming are viewport-independent

Project Direction

Help teams implement designer-provided Figma designs exactly as designed, with zero unnecessary AI token cost.

See #121 for full direction statement.

Methodology

OAT (One-at-a-Time) Sensitivity Analysis — remove one design property at a time and measure the impact on AI implementation quality.

Related Issues

  • #108 — Ablation calibration signals
  • #115 — Fixture design spec
  • #117 — Input method comparison
  • #118 — design-tree only for implement
  • #119 — Category weight adjustment
  • #120 — MCP get_metadata limitation
  • #121 — Project direction update

Clone this wiki locally