See how any site
handles AI
How they configure AI crawler access. Their robots.txt policies. The choices that shape AI visibility.
Real data on real companies
See how major sites handle AI crawlers
robots.txt policies from the web's top sites. Updated daily.
Explore the AI Ecosystem
Directories and resources for understanding AI agents
AI Bot Directory
95+ crawlers with purpose, operator & behavior
🌐Site Database
1M+ domains with AI policy analysis
⚖️Compare Policies
Side-by-side AI policy comparison
📋llms.txt Directory
Sites with declared AI permissions
🧩Tech Database
700+ technologies detected across the web
🔌MCP Servers
100+ sites with verified remote MCP servers
Understand any site's AI policy
Deep analysis of how sites handle AI crawlers
Training & Crawl Policies
Full robots.txt analysis against 95+ AI crawlers. See exactly how each bot is configured. and show exactly which bots can access each site.
AI Search Visibility
Will ChatGPT cite this site? Can Perplexity answer questions about it? See which AI search engines can access and reference content.
Take control of AI on your site
Don't just watch — decide which bots access your content, get alerts when new crawlers appear, and enforce your AI policy automatically.
- 📊See every visit from OpenAI, Anthropic, Perplexity, and more
- 🚨Get alerts when new AI crawlers appear
- 🛡️One-click block or allow — enforce your policy instantly
- 📈Track when ChatGPT and Claude cite your content
Free tier: 10K events/month • 1-line install • No credit card
StackFox Research
The AI Divide: Who's Blocking, Who's Open?
We analyzed the top 10,000 websites to understand how the web is responding to AI. The results reveal a fascinating split between industries.
Ready to control AI crawlers on your site?
Start with a free analysis, then track and manage your own site's AI bot traffic.