Scrapy, a fast high-level web crawling & scraping framework for Python.

An interactive TLS-capable intercepting HTTP proxy for penetration testers and software developers.

Asynchronous HTTP client/server framework for asyncio and Python

ChatterBot is a machine learning, conversational dialog engine for creating chat bots

Visual scraping for Scrapy

The property-based testing library for Python

Event-driven networking engine written in Python.

Library for building WebSocket servers and clients in Python

Command line interface to the freedesktop.org trashcan.

Lightweight, scriptable browser as a service with an HTTP API

Up-to-date simple useragent faker with real world database

High level Python client for Elasticsearch

WebSocket client for Python

python parser for human readable dates

WebSocket and WAMP in Python for Twisted and asyncio

🎭 Playwright integration for Scrapy

Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

The web framework for inventors

NO LONGER MAINTAINED - A Flask extension for creating simple ReSTful JSON APIs from SQLAlchemy models.

Extract embedded metadata from HTML markup

HTTP API for Scrapy spiders

Zyte Smart Proxy Manager (formerly Crawlera) middleware for Scrapy

Generator of User-Agent header

Scrapy spider middleware to ignore requests to pages containing items seen in previous crawls

JSON (de)serialization, GraphQL and JSON schema generation using Python typing.

Convert Javascript code to an XML document

Python bindings to the Brotli compression library

Scrapinghub Command Line Client