Sapu

Sapu — sweep articles off the web. Stealth Puppeteer scraper with URL classification, multi-site scheduling, and RabbitMQ workers.

A configuration-driven news scraper. The complex stuff lives in the database as JSON; the codebase stays small.

Most scrapers accumulate abstractions: workers, queues, classifiers, schedulers, each their own service. Sapu collapses that. One TypeScript service, one Postgres database, one RabbitMQ exchange, and a small Vite admin panel.

Three things drove the design:

1. Configuration over code — site rules, processor definitions, URL patterns live in the database as JSON. Add a new site by adding a JSON file to config/sites/.
2. Stealth by default — puppeteer-extra with the stealth plugin, automatic resource blocking (images, CSS, fonts), and per-site timeouts.
3. Schedule-aware — every site gets its own cron. The scheduler re-reads its own config at runtime.

git clone https://github.com/ahsanatha/sapu.git
cd sapu
cp .env.example .env
docker compose up -d

The API comes up on :3000, the admin panel on :4321. Bring Postgres and RabbitMQ up via the included docker-compose.yml.

pnpm install
pnpm db:init          # create schema + seed reference data
pnpm dev              # API server (tsx watch)
pnpm worker           # scraper worker process
pnpm web:dev          # admin panel (Vite + React)

Requires Node 22+, pnpm 10+, Postgres 15+, RabbitMQ 3.13+.

┌─────────────────────┐
│  Scheduler (cron)   │   per-site cron, re-reads config at runtime
└──────────┬──────────┘
           ↓ publishes to sapu.jobs
┌─────────────────────┐
│  RabbitMQ exchange  │   routing keys: scraping, url_collection
└──────────┬──────────┘
           ↓
┌─────────────────────┐         ┌──────────────────┐
│  Worker pool        │  ←────  │  Auto-scaler     │  queue-depth aware
│  (stealth puppeteer)│         └──────────────────┘
└──────────┬──────────┘
           ↓
┌─────────────────────┐         ┌──────────────────┐
│  Postgres           │  ←────  │  Admin panel     │  React + Vite
│  - articles         │         │  (port 4321)     │
│  - processors       │         └──────────────────┘
│  - url_patterns     │
│  - articles         │
└─────────────────────┘

Public endpoints that can be placed behind x402 middleware and priced per route:

Example:

curl -s http://localhost:3000/x402/health

curl -s http://localhost:3000/x402/classify-url \
  -H 'content-type: application/json' \
  -d '{"url":"https://example.com/news/article"}'

curl -s http://localhost:3000/x402/extract/article \
  -H 'content-type: application/json' \
  -d '{"url":"https://example.com","includeText":true}'

curl -s http://localhost:3000/x402/extract/article/browser \
  -H 'content-type: application/json' \
  -d '{"url":"https://example.com","includeText":true}'

curl -s http://localhost:3000/x402/extract/article/browser/long \
  -H 'content-type: application/json' \
  -d '{"url":"https://example.com","timeoutMs":90000,"includeText":true}'

curl -s 'http://localhost:3000/x402/cost/estimate?endpoint=fetch-browser&seconds=10'

Operational guardrail: cheap HTTP extraction routes must not launch browser rendering. Cheap browser routes use Cloudflare Browser Run. Long-running browser routes use local Puppeteer and are priced separately.

The server includes x402 v2 Express middleware. It is disabled by default for local development and turns on only when both conditions are true:

X402_ENABLED=true
X402_PAY_TO=0xYourSellerWalletAddress

Default seller config:

X402_NETWORK=eip155:84532
X402_FACILITATOR_URL=https://x402.org/facilitator
X402_RESOURCE_BASE_URL=https://sapu.rekursa.id

GET /x402/health and GET /x402/cost/estimate stay free. The paid routes return x402 v2 payment requirements and expect the PAYMENT-SIGNATURE header. Successful paid responses include the x402 PAYMENT-RESPONSE header.

For deployment, prefer the stateless x402 entrypoint:

pnpm build
pnpm start:x402

This starts apis/x402-server.ts, which does not import Postgres, RabbitMQ, or the admin UI. Use Dockerfile.x402 for Railway.

Cheap browser-priced routes use Cloudflare Browser Run instead of local Chromium:

CLOUDFLARE_ACCOUNT_ID=...
CLOUDFLARE_BROWSER_API_TOKEN=...
CLOUDFLARE_BROWSER_WAIT_UNTIL=domcontentloaded
X402_CLOUDFLARE_BROWSER_MAX_TIMEOUT_MS=45000
X402_LOCAL_BROWSER_MAX_TIMEOUT_MS=120000

The implementation calls Cloudflare's Browser Run REST /content Quick Action, which captures fully rendered HTML after JavaScript execution. Use an API token with Browser Rendering - Edit permission. The public Sapu response contract does not change: browser routes still return html, title, contentHash, elapsedMs, and retrievedAt. When Cloudflare returns it, Sapu also includes cloudflareBrowserMsUsed for unit-cost tracking.

Use /x402/fetch/browser/long or /x402/extract/article/browser/long for slow pages that need a local long-running browser task.

The schema (schema.sql) is configuration-as-data:

Run pnpm db:init to apply schema and seed reference data.

All runtime config via .env:

# Database
DATABASE_URL=postgresql://postgres@localhost:5432/sapu
RABBITMQ_URL=amqp://sapu:sapu123@localhost:5672/sapu

# Embeddings (via OpenRouter)
OPENROUTER_API_KEY=sk-or-v1-your-key-here
OPENROUTER_EMBEDDING_MODEL=openai/text-embedding-3-large

# Admin auth (HMAC signature)
ADMIN_PASSWORD=sapu123

# Telegram (optional)
TELEGRAM_ENABLED=false

This was built for a specific workload and shaped by it:

• Single-region, single-process — fine for hundreds of sites, not designed for thousands.
• Postgres-only persistence — no Redis; the queue is the only ephemeral state.
• Stealth ≠ magic — puppeteer-extra mitigates, but dedicated anti-bot services can still detect it.

If you need horizontal scaling or stronger anti-detection, swap RabbitMQ for Kafka/NATS and look at puppeteer-extra + residential proxy rotation.

MIT — see LICENSE.