This collection is a curated list of websites that employ the robots.txt file to restrict access to AI Agents, AI crawlers and GPTs.
It will be updated monthly.
The robots.txt file allows website owners to control and limit the access of these user agents to certain areas of their website by specifying rules and directives.
# OpenAI’s web crawler: GPT3.5, GPT4, ChatGPT
# https://platform.openai.com/docs/bots
User-agent: GPTBot
# ChatGPT plugins
# https://platform.openai.com/docs/bots
User-agent: ChatGPT-User
# OpenAI Search bot
# https://platform.openai.com/docs/bots
User-agent: OAI-SearchBot
# Google's web crawler: Bard, VertexAI, Gemini
# https://blog.google/technology/ai/an-update-on-web-publisher-controls/
User-agent: Google-Extended
# Apple's web crawler, dedicated to GenAI projects
# https://support.apple.com/en-us/119829
User-agent: Applebot-Extended
# Claude
User-agent: anthropic-ai
# Claude Bot
User-agent: ClaudeBot
# Claude web
User-agent: Claude-Web
# Amazonbot
# https://developer.amazon.com/amazonbot
User-agent: Amazonbot
# Cohere
User-agent: Cohere-ai
# Perplexity
User-agent: PerplexityBot
# You
# https://about.you.com/fr/youbot/
User-agent: YouBot
# Common Crawl
# https://commoncrawl.org/ccbot
User-agent: CCBot
# Omglibot: webz.io
# https://webz.io/blog/web-data/what-is-the-omgili-bot-and-why-is-it-crawling-your-website/
User-agent: Omgilibot
User-agent: Omgili
User-agent: Webzio-Extended
# Facebook: Llama
# https://developers.facebook.com/docs/sharing/bot/
User-agent: FacebookBot
# Facebook
# https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/
User-agent: Meta-ExternalAgent
# ByteDance: Duobao
User-agent: Bytespider
# Ai2
# https://allenai.org/crawler
User-agent: Ai2bot
User-agent: Ai2Bot-Dolma
# Diffbot
User-agent: Diffbot
# Huawei
# https://darkvisitors.com/agents/pangubot
User-agent: PanguBot
# Petal Search
# https://datadome.co/learning-center/how-to-block-petal-bot/
User-agent: PetalBot
Timpibot
# https://darkvisitors.com/agents/timpibot
User-agent: Timpibot
# Censorship area
Disallow: /Please note that this blocklist is intended for informational purposes only. Despite the provoking project name, it's fine to disallow web crawling and protect content ownership.
Scanned: 66
✅ Passing: 30 %
🔐 Blocked: 70 %
❓ Unknown: 0 %
Category: Video on demand
Scanned: 9
✅ Passing: 33 %
🔐 Blocked: 67 %
❓ Unknown: 0 %
Scanned: 6
✅ Passing: 50 %
🔐 Blocked: 50 %
❓ Unknown: 0 %
Scanned: 8
✅ Passing: 75 %
🔐 Blocked: 25 %
❓ Unknown: 0 %
Scanned: 6
✅ Passing: 83 %
🔐 Blocked: 17 %
❓ Unknown: 0 %
Scanned: 5
✅ Passing: 100 %
🔐 Blocked: 0 %
❓ Unknown: 0 %
Scanned: 13
✅ Passing: 31 %
🔐 Blocked: 62 %
❓ Unknown: 8 %
Scanned: 42
✅ Passing: 67 %
🔐 Blocked: 26 %
❓ Unknown: 7 %
Scanned: 3
✅ Passing: 100 %
🔐 Blocked: 0 %
❓ Unknown: 0 %
Scanned: 28
✅ Passing: 64 %
🔐 Blocked: 32 %
❓ Unknown: 4 %
Scanned: 3
✅ Passing: 100 %
🔐 Blocked: 0 %
❓ Unknown: 0 %
Scanned: 19
✅ Passing: 58 %
🔐 Blocked: 42 %
❓ Unknown: 0 %
Scanned: 1
✅ Passing: 0 %
🔐 Blocked: 100 %
❓ Unknown: 0 %
Name
Country
Status
Indeed 🇺🇸
🔐
Looking for contributions:
Enrich website database
Chinese websites
New categories
Please open issues!
Don't hesitate ;)
python -m venv venv
source ./venv/bin/activate
pip3 install -r requirements.txt
python3 scrape.py
# then copy the last version into readme
Give a ⭐️ if this project helped you!
Copyright © 2024 Samuel Berthe .
This project is MIT licensed.