🕷️ Discover and use popular web crawlers across various programming languages to efficiently extract data from the web.
-
Updated
Feb 11, 2026 - HTML
🕷️ Discover and use popular web crawlers across various programming languages to efficiently extract data from the web.
Web Crawling/Scraping Framework [WIP: Distributed Architecture]
List of libraries, tools and APIs for web scraping and data processing.
A flexible and composeable web crawler for .NET
A Python framework to build polite, but tenacious crawlers / scrapers with a MariaDB backend
List of best web crawlers to extract data from the web. Find web crawling tools for different needs.
🚀 OFFICIAL STARTER TEMPLATE FOR BOTASAURUS SCRAPING FRAMEWORK 🤖
This Repo is part of a tech assessment for a Web & Social Scraping application made with Tonik Studio
SpideyX a multipurpose Web Penetration Testing tool with asynchronous concurrent performance with multiple mode and configurations.
Python library to create web Crawlers which aims to be powerful yet simple.
Web crawling & scraping framework for Node.js on top of headless Chrome browser
🚀 SCRAPE 1000'S OF PRODUCTS FROM DENTALKART 🤖
An intelligent proxy server. Provide durable, real-time, high-quality proxies as a middleman or datasource server.
🌌 High productivity semi-automatic crawler generator 🛠️🧰
Laravel adapter for the crwlr/crawler package.
Sneakpeek is a framework that helps to quickly and conviniently develop scrapers. It’s the best choice for scrapers that have some specific complex scraping logic that needs to be run on a constant basis
🚀 THIS WEB SCRAPING TEMPLATE PROVIDES YOU WITH A GREAT STARTING POINT WHEN CREATING WEB SCRAPING BOTS. 🤖
Async Python 3.6+ web scraping micro-framework based on asyncio
a scrapping bot that generates internal links & open assessable API report for any website by crawling every page.
Crawlzone is a fast asynchronous internet crawling framework for PHP.
Add a description, image, and links to the crawling-framework topic page so that developers can more easily learn about it.
To associate your repository with the crawling-framework topic, visit your repo's landing page and select "manage topics."