Skip to content

Latest commit

 

History

69 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

BrandRadar

Competitive visibility for D2C brands — powered by custom Bright Data Scraper Studio collectors that self-heal when sites redesign.

Status Shipped · stable for the Lumen D2C beauty cohort
Repo https://github.com/priyanshuchawda/brandradar
License MIT

BrandRadar watches a fixed cohort of rivals on their own public sites (not marketplaces), normalizes shelf and changelog data into a weekly snapshot, scores how clearly you can still “see” the market, and pushes a Monday brief to Discord. When a layout change empties an extract, the same Studio collector id (c_*) is healed in place — no redeploy, no new scraper identity.

What ships in this release

  1. Custom Bright Data Scraper Studio collectors per rival (own-domain serums + blogs)
  2. Gemini-assisted deep pass (sublink ranking + evidence-bound brief)
  3. Discord delivery into the channels the team already uses
  4. Monday Diff respond / watch / hold plays
  5. Heal Lab proof — break → zero → heal on the same c_*

Who this is for

Founders, growth leads, and category managers at regional D2C brands who already know their own numbers and do not have a research team.

They need three answers every week:

  1. What changed on rival shelves and update pages?
  2. Where are we expensive, thin, or poorly rated?
  3. What should we do on Monday?

Spreadsheets go stale. Homegrown browser automation breaks on the first class rename. Marketplace library scrapers target the wrong competitive set. BrandRadar’s wedge is custom Studio collectors + self-heal + decision-ready plays.


Product surfaces

Surface Path Role
Control Tower /control-tower Ops board for the POV brand Lumen — serum-shelf ₹ table, mission runs, Gemini-assisted plans
Monday Diff /monday-diff Week-over-week rival update diffs, visibility score, plays
Heal Lab /heal-lab/before · /after Owned changelog pages used to prove break → heal → recover on the same c_*
Discord BrandRadar · Lumen server Per-rival channels, #monday-diff, #heal-alerts, slash commands

POV brand: Lumen (clean beauty D2C, India).
Watchlist: Mamaearth, Plum Goodness, Minimalist, Foxtale — each rival’s own serum collection + blog/news URLs.


How it works (architecture)

Rival public URLs (own domains only)
        │
        ▼
Bright Data Scraper Studio collectors (c_*)
   Discovery listings · optional PDP · intel updates · heal-lab
        │
        ▼
Normalize → Brand / Intel snapshot (Zod schemas)
        │
        ├──► Diff + visibility + plays
        ├──► Control Tower ₹ shelf
        ├──► Discord embeds + slash commands
        └──► Week cache (local disk / optional Vercel Blob)

Self-heal path (same collector id forever):
  QA flags empty / null_rate / row_collapse
        → bdata scraper heal <same c_*> "…" --url <page>
        → approve / auto-approve
        → re-run → rows back

Detail: docs/architecture.md · Bright Data internals: docs/brightdata.md

End-to-end data path

  1. Create a custom Studio collector (CLI or Studio UI) for one URL shape — e.g. Plum serums listing.
  2. Run via CLI (bdata scraper run) or app (POST /dca/trigger through lib/brightdata.ts).
  3. Normalize raw Studio JSON (lib/studio-normalize.ts) into typed rows (name, ₹ price, URL).
  4. Persist intel weeks under data/intel/<ISO-week>/ (or Blob in production).
  5. Diff vs prior week → visibility score → deterministic plays (lib/plays.ts).
  6. Deliver Control Tower UI + Discord (lib/discord-embeds.ts).
  7. Heal on QA failure with the same c_* (lib/heal-engine.ts + lib/studio.ts).

Collector ids are the production API handle. Healing never mints a new id.


Quick start

git clone https://github.com/priyanshuchawda/brandradar.git
cd brandradar
cp .env.example .env.local
npm install
npm run dev

Open http://localhost:3000 (or http://127.0.0.1:3000).

npm test                 # unit tests
npm run lint
npm run discord:bootstrap   # create Discord layout + pins (needs bot token)
npm run discord:tidy        # prune junk channels + post example intel

Secrets stay in .env.local — never commit tokens.


Bright Data Scraper Studio

BrandRadar uses custom Scraper Studio collectors, not pre-built marketplace library scrapers.

CLI (terminal-first)

# login once
npx -p @brightdata/cli bdata login

# create a Discovery collector for a listing page
npx -p @brightdata/cli bdata scraper create \
  "https://plumgoodness.com/collections/serums" \
  "Extract serum products: product_name, price, list_price, currency, product_url. Listing page only." \
  --name brandradar-plum-discovery --pretty

# run
npx -p @brightdata/cli bdata scraper run c_YOUR_ID \
  "https://plumgoodness.com/collections/serums" --pretty

# heal in place (same id)
npx -p @brightdata/cli bdata scraper heal c_YOUR_ID \
  "Extract serum products: product_name, price, list_price, currency, product_url." \
  --url "https://plumgoodness.com/collections/serums" --pretty

npx -p @brightdata/cli bdata scraper approve c_YOUR_ID \
  --url "https://plumgoodness.com/collections/serums" --pretty

In the app, runs go through lib/brightdata.ts: SDK realtime first, then REST fallback POST /dca/trigger → poll GET /dca/dataset.

Full collector guide: docs/collectors.md

Current watchlist collectors (Lumen cohort)

Pinned in config/rivals.json and/or .env.local:

Rival Surface Collector env / id
Mamaearth Face serum collection COLLECTOR_MAMAEARTH_DISCOVERY · c_msxk5ul22mk8gvsdgy
Plum Serums collection COLLECTOR_PLUM_DISCOVERY · c_mt5dogddx8lyp1q7f
Minimalist Face serum collection COLLECTOR_MINIMALIST_DISCOVERY · c_mt5gcpsg1hqf5xpv81
Foxtale Serums collection COLLECTOR_FOXTALE_DISCOVERY · c_mt5h0alj2rhv1aozp0
Heal Lab Owned before/after pages COLLECTOR_HEAL_LAB · c_mt3ekwjs2lzsn3dwl7
Intel updates Rival blogs/news COLLECTOR_INTEL_UPDATES

Sample Plum extract (sanitized): data/demo/plum-serums-sample.json

{
  "product_name": "10% Niacinamide & Rice Water Brightening Face Serum…",
  "price": 307,
  "list_price": 349,
  "currency": "INR",
  "product_url": "https://plumgoodness.com/products/…"
}

Configuration

Bright Data

Variable Purpose
BRIGHT_DATA_API_TOKEN Studio + SDK (also set BRIGHTDATA_API_KEY to the same value for CLI)
COLLECTOR_*_DISCOVERY / COLLECTOR_*_PDP Per-rival / vertical collectors
COLLECTOR_INTEL_UPDATES Monday Diff update-page pulls
COLLECTOR_HEAL_LAB Heal Lab owned-site collector
USE_MOCK=false Prefer live Studio when ids are set

Discord

Variable Purpose
DISCORD_BOT_TOKEN Bot token
DISCORD_GUILD_ID Server id
DISCORD_APPLICATION_ID Application / client id
DISCORD_PUBLIC_KEY Slash-command signature verify
DISCORD_CHANNEL_ID #monday-diff
DISCORD_HEAL_CHANNEL_ID #heal-alerts
DISCORD_RIVAL_*_CHANNEL_ID Per-rival channels (optional)

Setup: docs/discord.md

Optional

Variable Purpose
GEMINI_API_KEY Optional heal-prompt drafting + play copy rewrite
CRON_SECRET Auth for /api/cron/monday-diff
BLOB_READ_WRITE_TOKEN Week snapshots on Vercel
BRANDRADAR_API_KEY Lock mutating routes when the API is exposed

Self-healing

Sites redesign. Extracts go empty. BrandRadar’s contract:

Step Behavior
Detect Listing QA (lib/extract-qa.ts) — empty, high null rate, row collapse
Repair bdata scraper heal on the same c_* with plain-language field intent
Verify Re-run Collection API; proof is row counts returning (e.g. 5 → 0 → 5)
Notify Discord #heal-alerts broken / recovered embeds

Shared loop: lib/heal-engine.ts (≤2 heal passes locally; tighter budget on Vercel). Gemini is opt-in for prompt text only — it does not invent prices or URLs.

Owned proof pages (local after npm run dev):

Guide: docs/heal-lab.md


Monday Diff

Weekly competitive intel on public update pages (blogs / guides / changelogs):

  1. Load cohort from config/rivals.json
  2. Pull listing rows per rival (Studio or cached week)
  3. Diff vs prior week → added / removed / modified
  4. Visibility score 0–100 + per-rival health
  5. Emit plays (attack / watch / fill) — rule-based, not LLM-invented
  6. Optional cron → Discord #monday-diff

Guide: docs/monday-diff.md · Example: examples/intel-snapshot.json


HTTP API

Method Path Role
POST /api/intel Pull / cache cohort intel (forceMock, persist, refresh)
POST /api/intel/heal Heal COLLECTOR_INTEL_UPDATES
GET/POST /api/heal-lab Fixtures or live heal loop on owned pages
POST /api/scan Catalog arena scan
POST /api/discord/setup Bootstrap server + slash commands
POST /api/discord/interactions /intel, /rivals, /schema, /help
POST /api/cron/monday-diff Weekly pull → Discord (CRON_SECRET)

All Bright Data and Gemini calls are server-side only. The browser talks to /api/*.


Data schema

Listing / serum row (Studio Discovery):

{
  "product_name": "",
  "price": 307,
  "list_price": 349,
  "currency": "INR",
  "product_url": "https://…"
}

Intel listing row (update pages):

{
  "title": "",
  "url": "https://…",
  "published_at": "2026-08-01",
  "summary": ""
}

Intel snapshot (data/intel/<week>/snapshot.json): rivals[], diff[], visibility, plays[], health.collector_ids[].

Schemas: lib/intel-schema.ts · lib/schema.ts


AI usage (disclosure)

Component AI? Notes
Bright Data Scraper Studio create / heal Yes Studio AI Flow generates / repairs collector logic
Monday Diff plays + visibility No Deterministic rules from diff + row health
Discord embeds No Templates from snapshot JSON
Gemini Flash Optional Heal prompt drafting when QA flags break (useGemini)
Gemini Flash-Lite Optional Arena rival pick + play copy rewrite — numbers stay on extracted rows

We do not invent prices, URLs, or rival posts. Mock paths are labeled mode: fixture / mode: mock.


Stack

  • Next.js App Router + TypeScript + Zod
  • Bright Data Scraper Studio CLI (@brightdata/cli) + SDK (@brightdata/sdk)
  • Discord HTTP Interactions (Ed25519) — not a Gateway bot for slash commands
  • Optional Google Gemini for copy / heal prompts
  • Tests Vitest (npm test)

See docs/stack.md.


Data policy

  • Public HTTPS pages only — no logins, paywalls, or personal data
  • Each rival’s own domain — not aggregators or marketplace scrapers as the product extractor
  • Mask secrets in recordings; rotate any leaked tokens

Security notes: docs/security.md


Documentation

Doc Contents
docs/operations.md Local runbook, Studio + Discord verify
docs/architecture.md System diagram, pipelines, modules
docs/product.md Problem, users, plays, verticals
docs/collectors.md Studio create / run / heal
docs/brightdata.md Studio types, /dca/trigger, CLI
docs/heal-lab.md Self-heal proof on owned pages
docs/monday-diff.md Cohort intel, visibility, cron
docs/discord.md Server layout, embeds, slash commands
docs/integrations.md Bright Data + Gemini boundaries
docs/security.md Auth, rate limits, URL policy
docs/stack.md Language and runtime choices
examples/ Sample snapshot + PDP JSON

Roadmap (direction)

  • More verticals (edtech, food) with the same Discovery → PDP → heal contract
  • Stronger week memory and alerting without a heavy warehouse
  • Deeper Discord ops (threaded shelf diffs, scheduled missions)
  • Hardening production auth and multi-tenant tenancy (out of scope today)

Contributing

See CONTRIBUTING.md. Issues and PRs welcome.

npm test && npm run lint

License

MIT

About

Open-source competitive intel for D2C brands — custom Bright Data Scraper Studio collectors, self-healing extracts, Monday Diff, and Discord ops.

Topics

Resources

Contributing

Security policy

Stars

2 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages