Scrapy, a fast high-level web crawling & scraping framework for Python.
-
Updated
Nov 27, 2025 - Python
#
web-scraping
Here are 7,891 public repositories matching this topic...
The all-in-one Desktop & Docker AI application with built-in RAG, AI agents, No-code agent builder, MCP compatibility, and more.
mcp web-scraping no-code ai-agents kimi multimodal rag moonshot vector-database llm localai local-llm ollama lmstudio deepseek llama3 custom-ai-agents mcp-servers qwen3
-
Updated
Nov 27, 2025 - JavaScript
🔥 The Web Data API for AI - Turn entire websites into LLM-ready markdown or structured data
markdown crawler scraper ai html-to-markdown web-crawler scraping web-scraper web-scraping data-extraction webscraping web-data-extraction ai-agents web-search ai-search web-data llm ai-crawler ai-scraping
-
Updated
Nov 29, 2025 - TypeScript
🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。
python api crawler scraper spider async web-scraping douyin tiktok fastapi tiktok-scraper tiktok-api douyin-api pywebio tiktok-signature no-watermark online-parsing douyin-tiktok-api douyin-tiktok-download douyin-scraper
-
Updated
Oct 12, 2025 - Python
The Python Code Tutorials
python python-tutorials machine-learning natural-language-processing computer-vision text-classification tutorials python3 web-scraping face-detection scapy network-analysis network-programming programming-tutorial ethical-hacking network-security socket-programming scapy-tutorials
-
Updated
Oct 6, 2025 - Jupyter Notebook
Python scraper based on AI
markdown crawler web-crawler scraping web-scraper web-scraping data-extraction webscraping web-data-extraction web-search ai-search rag automated-scraper web-data scraping-python web-crawlers llm ai-crawler large-language-model ai-scraping
-
Updated
Nov 26, 2025 - Python
Best and simplest tool for website change detection, web page monitoring, and website change alerts. Perfect for tracking content changes, price drops, restock alerts, and website defacement monitoring—all for free or enjoy our SaaS plan!
notifications rss monitoring self-hosted web-scraping website-monitor url-monitor change-alert change-detection website-change-monitor website-change-tracker website-monitoring change-monitoring website-watcher website-change-detector restock-monitor website-change-detection website-change-notification back-in-stock website-defacement-monitoring
-
Updated
Nov 28, 2025 - Python
Python APIs for web automation, testing, and bypassing bot-detection with ease.
python webdriver selenium test-automation pytest web-scraping chromedriver webkit pytest-plugin cdp behave bot-detection web-automation python-scraper selenium-python e2e-testing cloudflare-bypass seleniumbase anti-detection web-scraping-python
-
Updated
Nov 29, 2025 - Python
Learn Python for the next 30 (or so) Days.
python api flask automation tutorial csv jupyter rest-api selenium pandas python3 web-scraping selenium-webdriver fastapi
-
Updated
Feb 27, 2024 - HTML
Turn any website into clean, contextualized data pipelines for your workflows
api scraper automation browser web-scraper self-hosted web-scraping data-extraction webscraping agents hacktoberfest browser-automation no-code web-automation rpa robotic-process-automation nocode playwright hacktoberfest-accepted no-code-web-scraper
-
Updated
Nov 29, 2025 - TypeScript
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.
nodejs javascript npm crawler scraper automation typescript web-crawler headless scraping crawling web-scraping web-crawling headless-chrome apify puppeteer playwright
-
Updated
Nov 28, 2025 - TypeScript
General Assembly's 2015 Data Science course in Washington, DC
python data-science machine-learning natural-language-processing course clustering naive-bayes linear-regression scikit-learn jupyter-notebook pandas data-visualization web-scraping data-analysis ensemble-learning logistic-regression decision-trees regular-expressions data-cleaning model-evaluation
-
Updated
Jun 5, 2024 - Jupyter Notebook
Collection of scripts corresponding to LucidProgramming YouTube tutorials
python python3 web-scraping youtube-tutorial python-tutorial ctci-solutions lucidprogramming python3-tutorial technical-interview
-
Updated
Oct 26, 2022 - Python
List of libraries, tools and APIs for web scraping and data processing.
crawler spider scraping crawling web-scraping captcha-recaptcha webscraping crawling-framework scraping-framework captcha-bypass scraping-tool crawling-tool scraping-python crawling-python
-
Updated
Oct 13, 2025 - Makefile
PHP Curl Class makes it easy to send HTTP requests and integrate with web APIs
api php http client json framework curl xml proxy restful class http-client http-proxy api-client web-scraper requests web-scraping php-curl web-service php-curl-library
-
Updated
Nov 24, 2025 - PHP
A Smart, Automatic, Fast and Lightweight Web Scraper for Python
python crawler machine-learning scraper automation ai scraping artificial-intelligence web-scraping scrape webscraping webautomation
-
Updated
Jun 9, 2025 - Python
🔥 Official Firecrawl MCP Server - Adds powerful web scraping and search to Cursor, Claude and any other LLM clients.
mcp web-crawler web-scraping data-collection batch-processing content-extraction search-api claude llm-tools firecrawl model-context-protocol mcp-server firecrawl-ai javascript-rendering
-
Updated
Nov 22, 2025 - JavaScript
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
python crawler scraper automation web-crawler headless scraping crawling pip web-scraping beautifulsoup web-crawling hacktoberfest headless-chrome apify playwright
-
Updated
Nov 28, 2025 - Python
Lighter web automation with Python
-
Updated
Nov 10, 2025 - Python
Convert documentation websites, GitHub repositories, and PDFs into Claude AI skills with automatic conflict detection
github python pdf documentation automation ocr mcp code-analysis web-scraping ast-parser documentation-generator conflict-detection multi-source github-scraper ai-tools claude-ai mcp-server claude-skills
-
Updated
Nov 29, 2025 - Python
Improve this page
Add a description, image, and links to the web-scraping topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the web-scraping topic, visit your repo's landing page and select "manage topics."