Skip to content

feat(extract): live-URL token extraction (@tokenscout/extract)#10

Closed
Atroci wants to merge 2 commits into
chore/deslop-docs-and-harden-corefrom
feat/extract-package
Closed

feat(extract): live-URL token extraction (@tokenscout/extract)#10
Atroci wants to merge 2 commits into
chore/deslop-docs-and-harden-corefrom
feat/extract-package

Conversation

@Atroci

@Atroci Atroci commented Jun 4, 2026

Copy link
Copy Markdown
Owner

Ships the headline capability: point tokenscout at a URL and get design tokens. Stacked on #9 (base will retarget to main once #9 merges).

What

  • Workspace split so the core stays zero-dependency: published core moves to packages/core (name stays tokenscout, no consumer breakage); root is now a private npm workspace with a shared tsconfig.base.json.
  • New @tokenscout/extract package, Playwright as a peer dependency:
    • extractSite(url, { breakpoints, top }) drives headless Chromium, reads computed styles (color / background / border, font-size, margin/padding/gap) at each breakpoint with optional same-origin crawling, returns PageExtract[].
    • extractTokens(url, opts) chains that into the core's assembleTokens for a one-call URL → DTCG flow.
    • Aggregation is a pure, unit-tested harvest(); the browser-side collector stays thin. A real-Chromium smoke test runs against a local fixture and asserts the end-to-end path.
  • CI: split into a fast core job and a Chromium-backed extract job. Release: publishes the core from packages/core with provenance.
  • Docs updated to match reality (no more "planned" for things that now exist): README live-URL usage + install + roadmap + limitations, ARCHITECTURE real two-package layout + CI, ROADMAP Phase 3 in progress, CHANGELOG.

Verification

  • npm run build, typecheck, lint, format:check: clean.
  • Tests: core 57/57, extract 3/3 (incl. the live-browser smoke test). Zero em-dashes repo-wide.

Usage

import { extractTokens } from "@tokenscout/extract";
const tokens = await extractTokens("https://example.com", { breakpoints: [1280, 375], top: 1 });

Not in this PR (still roadmap)

Image/asset harvesting and animation capture. Sitemap-driven discovery and tech-stack profiling.

🤖 Generated with Claude Code

Atroci and others added 2 commits June 4, 2026 13:39
Ship the headline capability: point at a URL, get design tokens.

Workspace split (so the core stays zero-dependency):
- Move the published core to packages/core (name stays "tokenscout",
  no consumer breakage); root becomes a private npm workspace with a
  shared tsconfig.base.json and dev tooling.

New @tokenscout/extract package (Playwright as a peer dependency):
- extractSite(url, { breakpoints, top }) drives headless Chromium, reads
  computed styles (color/background/border, font-size, margin/padding/gap)
  at each breakpoint with optional same-origin crawling, and returns
  PageExtract[].
- extractTokens() chains that into the core's assembleTokens for a one-call
  URL -> DTCG flow.
- Aggregation is a pure harvest() function (unit-tested); the browser-side
  collector is kept thin. A smoke test runs real Chromium against a local
  fixture and asserts the end-to-end path.

CI/release:
- ci.yml split into a fast core job and a Chromium-backed extract job.
- release.yml publishes the core from packages/core with provenance.

Docs updated to match: README (live-URL usage, install, roadmap, limits),
ARCHITECTURE (real two-package layout + CI), ROADMAP (Phase 3 in progress),
CHANGELOG.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bring README.pt-BR up to parity with the rewritten English README (live-URL
usage, two-package install, results, use cases, limitations). Fix the English
README example links, which broke when examples/ moved to
packages/core/examples/.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Atroci Atroci deleted the branch chore/deslop-docs-and-harden-core June 4, 2026 12:14
@Atroci Atroci closed this Jun 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant