crw-mcp

MCP (Model Context Protocol) server for the CRW web scraper.

Overview

crw-mcp is a self-contained MCP server that gives any MCP-compatible AI client (Claude Code, Claude Desktop, Cursor, Windsurf, Cline, Continue.dev, OpenAI Codex CLI) 4 web scraping tools. No external server needed — just add and go.

Two modes:

4 MCP tools:

Installation

cargo install crw-mcp

Quick Start (Embedded Mode)

No server to run. Just add crw-mcp to your AI client:

# Claude Code
claude mcp add crw -- crw-mcp

# With custom config via env vars
claude mcp add \
  -e CRW_CRAWLER__MAX_CONCURRENCY=5 \
  -e CRW_RENDERER__LIGHTPANDA__WS_URL=ws://127.0.0.1:9222 \
  crw -- crw-mcp

Proxy Mode (Remote Server)

Connect to fastcrw.com or any remote CRW instance:

# Cloud server
claude mcp add \
  -e CRW_API_URL=https://fastcrw.com/api \
  -e CRW_API_KEY=fc-xxx \
  crw -- crw-mcp

# Local crw-server on custom port
claude mcp add \
  -e CRW_API_URL=http://localhost:4000 \
  crw -- crw-mcp

Or use the HTTP transport directly (no crw-mcp binary needed):

claude mcp add --transport http crw http://localhost:3000/mcp

CLI Options

Embedded mode configuration

In embedded mode, crw-mcp loads configuration the same way as crw-server: config.default.toml → config.local.toml → environment variable overrides. Env vars use CRW_ prefix with __ as separator:

CRW_CRAWLER__MAX_CONCURRENCY=5
CRW_RENDERER__LIGHTPANDA__WS_URL=ws://127.0.0.1:9222
CRW_CRAWLER__USER_AGENT="MyBot/1.0"

Feature Flags

Build a slim proxy-only binary without the embedded engine:

cargo build -p crw-mcp --no-default-features --release

Setup by Client

Claude Code

claude mcp add crw -- crw-mcp

Claude Desktop

Edit your config file:

{
  "mcpServers": {
    "crw": {
      "command": "crw-mcp"
    }
  }
}

Cursor

Edit ~/.cursor/mcp.json (global) or .cursor/mcp.json (project):

{
  "mcpServers": {
    "crw": {
      "command": "crw-mcp"
    }
  }
}

Windsurf

Edit ~/.codeium/windsurf/mcp_config.json:

{
  "mcpServers": {
    "crw": {
      "command": "crw-mcp"
    }
  }
}

Cline (VS Code)

{
  "mcpServers": {
    "crw": {
      "command": "crw-mcp",
      "alwaysAllow": ["crw_scrape", "crw_map"],
      "disabled": false
    }
  }
}

Continue.dev (VS Code / JetBrains)

Edit ~/.continue/config.yaml:

mcpServers:
  - name: crw
    command: crw-mcp

OpenAI Codex CLI

Edit ~/.codex/config.toml:

[mcp_servers.crw]
command = "crw-mcp"

Any MCP client

{
  "mcpServers": {
    "crw": {
      "command": "crw-mcp"
    }
  }
}

Tip: For clients that support HTTP transport, you can still use http://localhost:3000/mcp directly with a running crw-server — no stdio binary needed.

With Proxy Mode Authentication

{
  "mcpServers": {
    "crw": {
      "command": "crw-mcp",
      "env": {
        "CRW_API_URL": "https://fastcrw.com/api",
        "CRW_API_KEY": "fc-your-api-key"
      }
    }
  }
}

Part of CRW

This crate is part of the CRW workspace — a fast, lightweight, Firecrawl-compatible web scraper built in Rust.

License

AGPL-3.0 — see LICENSE.