xscrape

Asynchronous Python client for collecting public data from X (Twitter).

xscrape is a lightweight library for asynchronously collecting public data from X (Twitter): posts, profiles, threads, replies, and media. It supports account rotation, a built-in rate limiter, and export to JSON / CSV / SQLite.

• Features
• Installation
• Quick Start
• How It Works
• Configuration
• CLI
• Examples
• Docker
• Testing
• Project Layout
• Roadmap
• Contributing
• Disclaimer
• License

• 🔍 Post search — by keywords, hashtags, and operators (from:, since:, until:)
• 👤 User profiles — metadata, followers, activity counters
• 🧵 Threads and replies — reconstruction of conversation chains
• 🖼 Media — extraction of image and video links
• 🔄 Account rotation — session pool with automatic failover
• ⏱ Rate limiting — adaptive control of request frequency
• 💾 Storage backends — JSON, CSV, SQLite out of the box
• 🧩 Plugins — custom handlers and exporters
• 🖥 CLI — ready-to-use command line interface
• 🐳 Docker-ready — single command deployment
• 📊 Structured logging — JSON logs with request tracing

git clone https://github.com/kloxeld/xscrape.git
cd xscrape
pip install -e .

• Python 3.10+
• aiohttp, pydantic, tenacity, orjson

pip install "xscrape[socks]"    # SOCKS proxy support
pip install "xscrape[dev]"      # development tools
pip install "xscrape[docs]"     # documentation builders

# Search posts
xscrape search "python asyncio" --limit 50 --out tweets.json

# User profile
xscrape user elonmusk

# User timeline
xscrape timeline elonmusk --limit 200 --out timeline.csv

# Reconstruct a thread
xscrape thread 1234567890123456789 --out thread.json

# Collect by hashtag into SQLite
xscrape hashtag "#opensource" --limit 1000 --db hashtag.db

# Multi-account pool
XSCRAPE_POOL=accounts.json xscrape search "data engineering" --limit 2000

Run xscrape --help for the full command reference.

xscrape talks to public GraphQL endpoints of X using session cookies. The pipeline looks like this:

┌────────────┐   ┌──────────────┐   ┌────────────┐   ┌────────────┐
│  Client    │──▶│  AuthPool    │──▶│  Fetcher   │──▶│  Parser    │
└────────────┘   └──────────────┘   └────────────┘   └────────────┘
                                            │                │
                                            ▼                ▼
                                     ┌────────────┐   ┌────────────┐
                                     │ RateLimiter│   │  Storage   │
                                     └────────────┘   └────────────┘

1. Client — public interface (search, user, thread, replies).
2. AuthPool — session pool; picks a free account, handles 429/401.
3. Fetcher — low-level HTTP requests with retries and exponential backoff.
4. Parser — normalizes raw responses into typed models (Tweet, User).
5. RateLimiter — per-account token bucket plus a global cap.
6. Storage — serialization of results into the chosen format.

See docs/ARCHITECTURE.md for details.

All options are read from environment variables (see .env.example):

Full reference: docs/CONFIGURATION.md.

The examples/ directory contains ready-to-run scripts:

• search_tweets.py — search with pagination and filters
• user_timeline.py — collect a user's timeline
• export_to_csv.py — dump results to CSV
• export_to_sqlite.py — persist results into SQLite
• thread_dump.py — reconstruct a full thread
• hashtag_monitor.py — long-running hashtag watcher
• multi_account_pool.py — usage of an account pool

Run:

python examples/search_tweets.py --query "openai" --limit 200

docker build -f docker/Dockerfile -t xscrape:latest .

cp .env.example .env
docker compose up --build

docker compose -f docker-compose.dev.yml up --build

See docs/EXAMPLES.md for advanced Docker workflows.

pytest -q                    # run everything
pytest tests/unit            # unit tests only
pytest tests/integration     # integration tests only

Coverage:

pytest --cov=xscrape --cov-report=html

xscrape/
├── xscrape/          # library source
│   ├── client.py     # public client
│   ├── auth.py       # session pool
│   ├── parser.py     # response parsing
│   ├── ratelimit.py  # rate limiter
│   ├── storage.py    # storage backends
│   ├── plugins/      # plugin system
│   └── exporters/    # pluggable exporters
├── tests/            # unit + integration tests
├── examples/         # ready-to-run scripts
├── docs/             # documentation
├── docker/           # Dockerfiles
├── scripts/          # helper shell scripts
└── .github/          # CI workflows, templates

• Search and profiles
• Account pool and rate limiter
• JSON / CSV / SQLite export
• CLI
• Docker support
• Plugin exporter system
• Media download support
• Webhook notifications
• Web monitoring dashboard
• Prometheus metrics
• GraphQL query cache

Full roadmap: docs/ROADMAP.md.

We welcome contributions. Please read CONTRIBUTING.md and CODE_OF_CONDUCT.md before opening a PR.

This project is intended for educational purposes and work with public data only. Use it in accordance with the laws of your jurisdiction and the platform's rules. The authors are not responsible for any consequences of use.

MIT — see LICENSE.