Scrapy, a fast high-level web crawling & scraping framework for Python.
-
Updated
Oct 27, 2025 - Python
#
web-scraping
Here are 3,776 public repositories matching this topic...
Best and simplest tool for website change detection, web page monitoring, and website change alerts. Perfect for tracking content changes, price drops, restock alerts, and website defacement monitoring—all for free or enjoy our SaaS plan!
notifications rss monitoring self-hosted web-scraping website-monitor url-monitor change-alert change-detection website-change-monitor website-change-tracker website-monitoring change-monitoring website-watcher website-change-detector restock-monitor website-change-detection website-change-notification back-in-stock website-defacement-monitoring
-
Updated
Nov 13, 2025 - Python
Python scraper based on AI
markdown crawler web-crawler scraping web-scraper web-scraping data-extraction webscraping web-data-extraction web-search ai-search rag automated-scraper web-data scraping-python web-crawlers llm ai-crawler large-language-model ai-scraping
-
Updated
Nov 6, 2025 - Python
🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。
python api crawler scraper spider async web-scraping douyin tiktok fastapi tiktok-scraper tiktok-api douyin-api pywebio tiktok-signature no-watermark online-parsing douyin-tiktok-api douyin-tiktok-download douyin-scraper
-
Updated
Oct 12, 2025 - Python
Python APIs for web automation, testing, and bypassing bot-detection with ease.
python webdriver selenium test-automation pytest web-scraping chromedriver webkit pytest-plugin cdp behave bot-detection web-automation python-scraper selenium-python e2e-testing cloudflare-bypass seleniumbase anti-detection web-scraping-python
-
Updated
Nov 13, 2025 - Python
🕷️ An undetectable, powerful, flexible, high-performance Python library to make Web Scraping Easy and Effortless as it should be!
python crawler data automation ai mcp scraping crawling web-scraper web-scraping selectors xpath data-extraction stealth webscraping crawling-python playwright web-scraping-python ai-scraping mcp-server
-
Updated
Nov 10, 2025 - Python
Lighter web automation with Python
-
Updated
Nov 10, 2025 - Python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
python crawler scraper automation web-crawler headless scraping crawling pip web-scraping beautifulsoup web-crawling hacktoberfest headless-chrome apify playwright
-
Updated
Nov 11, 2025 - Python
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
python crawler machine-learning scraper automation ai scraping artificial-intelligence web-scraping scrape webscraping webautomation
-
Updated
Jun 9, 2025 - Python
Pydoll is a library for automating chromium-based browsers without a WebDriver, offering realistic interactions.
testing crawler automation webdriver headless scraping selenium chromium e2e-tests web-scraping testing-tools fingerprinting cdp browser-automation puppeteer recaptcha-v3 anti-detection playwright turnstile-solver
-
Updated
Nov 13, 2025 - Python
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
nlp crawler text-mining html-to-markdown scraping news-aggregator text-extraction web-scraping rss-feed readability tei html2text news-crawler corpus-builder corpus-tools article-extractor text-cleaning rag text-preprocessing llm
-
Updated
Sep 12, 2025 - Python
Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints.
http curl https http-client web-scraping fingerprinting ja3 tls-fingerprint curl-impersonate ja3-fingerprint http2-fingerprint akamai-fingerprint
-
Updated
Nov 11, 2025 - Python
Convert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detection
github python pdf documentation automation ocr mcp code-analysis web-scraping ast-parser documentation-generator conflict-detection multi-source github-scraper ai-tools claude-ai mcp-server claude-skills
-
Updated
Nov 12, 2025 - Python
Web Scraping Framework
python crawler framework spider asynchronous network python-library scraping crawling http-client python3 web-scraping pycurl urllib3
-
Updated
Sep 19, 2025 - Python
Free Trial Amazon Scraper API for extracting search, product, offer listing, reviews, question and answers, best sellers and sellers data.
python amazon-api web-scraping price-scraper e-commerce-api amazon-scraper scraping-api amazon-price-tracker amazon-scraping-library amazon-scraping amazon-product-scraper amazon-scraper-api price-scraper-api how-to-scrape-amazon scrape-amazon scraping-amazon amazon-products-api amazon-scrape-api amazon-aip amazon-api-data
-
Updated
Sep 25, 2025 - Python
Structured data gathering from any website using AI-powered scraper, crawler, and browser automation. Scraping and crawling with natural language prompts. Equip your LLM agents with fresh data. AI Studio python SDK for intelligent web data gathering.
web-scraping ai-search proxy-scraper python-ai ai-tools web-scraping-python ai-crawler ai-web-scraper ai-scraping ai-scraper web-scraping-ai
-
Updated
Nov 13, 2025 - Python
Learn step-by-step how to scrape Google Trends data and make a result comparison using Python and Oxylabs SERP API. Extract keywords, their popularity, breakdown by region, related queries, and more.
python scraper google scraping web-scraping google-trends google-trends-api google-trend-strategy scraping-api scraper-api google-trend google-trends-scraper google-trends-crawler google-trend-api
-
Updated
Sep 23, 2025 - Python
AgentQL is a suite of tools for connecting your AI to the web. Featuring a query language and Playwright integrations for interacting with elements and extracting data quickly, precisely, and at scale. Includes REST API, Python and JavaScript SDKs, browser debugger.
javascript python agent automation web ai scraping web-scraping web-scrapping rpa playwright web-scraping-python web-scraping-javascript aiagent webagent web-scraping-colabs
-
Updated
Nov 13, 2025 - Python
Undetected Python version of the Playwright testing and automation library.
bot chrome automation webdriver browser bots chromium cloudflare web-scraping chromedriver stealth webscraping botting web-automation webautomation undetected undetectable playwright cloudflare-by
-
Updated
Nov 11, 2025 - Python
Improve this page
Add a description, image, and links to the web-scraping topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the web-scraping topic, visit your repo's landing page and select "manage topics."