Languages

robots.txt Tester

Free, no signup. robots.txt gives no warning when a misplaced Disallow hides half a site from Google — it just quietly works. Test any list of URLs against it with a matcher ported from Google's own open-source parser — longest-match-wins, wildcards, sub-agent fallback, and status-code semantics handled exactly the way Google does it.

Built by Patrick Stox

1. robots.txt
2. Pages to test
3. Bots

Example data — replace with your own

Runs entirely in your browser — nothing you paste is uploaded or stored. The server is only used to fetch a site's robots.txt or sitemap (browsers can't — no CORS). Anonymous run-level outcome counters may be used for aggregate research; URLs, domains, IPs, and identifiers are never included, and no statistic is released below 100 runs.

Feedback
Report a bug

Found something broken in Robots Txt Tester? Let us know what happened — this goes straight to a private triage queue, not a public list.

What will be sent 
 No tool inputs, uploads, pasted source, complete results, query parameters, or URL fragments are attached automatically. You can edit or remove the selected passage above. Browser and anti-abuse metadata is processed for spam prevention. 
  
 
  
 
 
  
 
 
   
    
 
Sample report Example data — real, verified result
 
Given this robots.txt:
 
 
User-agent: *
Disallow: /search
Allow: /search/howsearchworks
 
 
…then test /search/howsearchworks and /search?q=redirects against Googlebot. The matrix comes back:
  
     
  • Longest match wins, not first match. /search/howsearchworks is
            allowed even though Disallow: /search also matches it — the longer
Allow: /search/howsearchworks on line 3 is more specific and overrides it.
    •  
  • The second URL has no exception. /search?q=redirects is blocked by
Disallow: /search on line 2 — nothing more specific matches it.
    •  
  • Click a verdict to see why. Clicking either cell highlights the exact line that
            decided it — see what "blocked"/"allowed" and the L-numbers mean
and the common-mistakes list below for how order-independent matching trips people up.
    •  
  • Background reading. The robots.txt guideA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.
covers every directive this matcher supports.
    •  
 
Accidental-indexing example: robots.txt hides the noindex
 

A staging template returns a public 200 response with this directive:

 
 
<meta name="robots" content="noindex">
 
 
But the production robots.txt also contains:
 
 
User-agent: *
Disallow: /staging/
 
 

Test /staging/pricing/ and this tool correctly reports it as
blocked. That is evidence about crawling only—not evidence that the URL is
          excluded from search. Because Google cannot fetch the page, it cannot see the
noindex. Links or an old sitemap can still expose the URL for indexing.

 
     
  1. Remove the Disallow: /staging/ rule temporarily while keeping the server-delivered noindex.
    2.  
  2. Rerun this tool and confirm the URL is crawlable, then inspect the live response and request recrawling.
    3.  
  3. After removal is verified, protect private staging environments with authentication—not robots.txt.
    4.  
 
 
 
How to use it
 
     
  1. Load a robots.txt. Enter a domain to fetch the live file, or switch to
Paste to test a draft that isn't published yet. In fetch mode, full URLs on
            other subdomains automatically use that origin’s own /robots.txt.
    2.  
  2. Add the pages to test. Paste URLs (full or bare paths), upload a
.txt/.csv list, or pull them straight from the site's
sitemap.xml.
    3.  
  3. Pick the bots. Choose the crawlers you care about, or hit
AI crawler audit to load the AI bots (GPTBot, ClaudeBot, Google-Extended, and
            the rest) in one click.
    4.  
  4. Read the matrix. Every URL × bot cell shows allowed or blocked. Click a cell
            to see the exact rule that decided it, highlighted in the file.
    5.  
 
 
 
       
     
       
 
How it works
 

The matcher is ported from Google's open-source robots.txt parser, so it decides
          allow/block the way Googlebot actually does — not with a naive "first match wins" rule that most
          testers use. That means longest-match-wins (the most specific
Allow/Disallow path takes precedence regardless of order),
* and $ wildcards, percent-encoding normalisation, and
sub-agent fallback (a bot with no group of its own falls back to
User-agent: *).

 
 Crawl-delay is crawler-specific, not part of this Google matcher. Google does not process it.
Bing documents 1–20 second values.
          This tool reports Google-style Allow/Disallow decisions and never pretends a Crawl-delay line changes Googlebot’s result.

 

All of that runs in your browser. The server is only touched to fetch a live
          robots.txt or sitemap, because browsers can't request them cross-origin. It also models
          robots.txt status-code semantics: most 4xx responses are treated as "no robots.txt,"
          but 429 is a temporary failure like a 5xx or timeout. Temporary failures can initially pause crawling;
          Google may also reuse its last known good file, which a point-in-time tester cannot observe.

 
Features
 
     
  • Google-accurate matching: longest-match-wins, wildcards, sub-agent fallback, status-code rules.
    •  
  • Blocked/allowed matrix across as many URLs and bots as you like.
    •  
  • Click any verdict to see the exact matching line highlighted in the file.
    •  
  • One-click AI crawler audit for the major LLM and AI-search bots.
    •  
  • Built-in linter that flags common robots.txt mistakes.
    •  
  • Fetch by domain, upload a URL list, or pull URLs from a sitemap — including a separate live robots.txt for each entered subdomain (up to 10 origins per run).
    •  
  • Privacy: matching is fully client-side — pasted files never leave your machine.
    •  
 
Common mistakes it catches
 
     
  • Assuming rule order matters — it doesn't for Google; the most specific path wins.
    •  
  • Blocking a resource (CSS/JS) that the page needs to render, then wondering why rankings drop.
    •  
  • Using robots.txt to keep a page out of the index — disallow stops crawling, not indexing; use
NoindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. for that.
    •  
  • A stray Disallow: / under User-agent: * that quietly blocks the whole site.
    •  
 
Limitations
 

This checks Google-style robots.txt matching only — it tells you whether a bot is allowed to crawl a
          URL, not whether the page is indexable (a page can be crawlable and still carry a
noindex). Fetch-by-domain reads the robots.txt exactly as served; if a site returns
          different robots.txt to different user-agents or regions, you'll see the version your fetch
          received. A robots.txt file is scoped to its origin, so fetch mode requests each
          entered subdomain’s own file. Paste mode deliberately uses the one pasted draft for every URL.
          Non-Google crawler behavior is not inferred here; consult that crawler’s own documentation before
          relying on system-specific extensions.

 
Frequently asked questions
 
Does robots.txt stop a page from being indexed?
No. Disallow blocks crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., not indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. If other pages link to a disallowed URL, Google can still index it (usually without a snippet). To keep a page out of the index, let it be crawled and add a noindex meta tag or X-Robots-Tag header — which means you must not also disallow it, or Google can never see the noindex.
Does the order of rules in robots.txt matter?
Not for Google. Google applies the most specific rule — the longest matching path — regardless of where it sits in the file, so an Allow can override an earlier Disallow. This tool matches the same way. Some other crawlers use first-match-wins, so keep rules unambiguous.
What is the difference between Disallow and noindex?
Disallow (in robots.txt) tells crawlers not to fetch a URL. noindex (a meta tag or HTTP header on the page) tells them not to keep it in the index. They solve different problems, and combining them backfires: a disallowed page is never fetched, so its noindex is never seen.
Will blocking CSS or JavaScript in robots.txt hurt SEO?
It can. Google renders pages to understand them, so if it cannot fetch the CSS and JS a page needs, it may misjudge the layout, content, or mobile-friendliness. Allow the resources required to render your pages and only block genuinely non-essential paths.
How do I block AI crawlers with robots.txt?
Add a user-agent group for each AI bot — such as GPTBot, Google-Extended, ClaudeBot, CCBot, and PerplexityBot — with Disallow: /. Use the AI crawler audit button to load them and confirm your rules match. Note that robots.txt compliance is voluntary; reputable bots honour it, but it is not an access control.
 
 
 
     
Next steprobots.txt Generator — generate a replacement robots.txt with the directives you meant to ship.

People also explore
ToolLog File AnalyzerToolAI-Crawler Access CheckerToolGooglebot VerifierArticleAI CrawlersArticleUser AgentArticleNuxt SEO
 
 
 
Feature requests for Robots Txt Tester
 

Upvote what you want most. New ideas can be submitted from the floating Feedback menu;
      requests appear here once approved, and the most-wanted rise to the top.

   
Loading…
  
  ➕ Request a feature   
New requests are reviewed before they appear here.
 
     
  
  
 
 
 
 
   
   
 
Where this tool helps
Common use cases
Check a risky robots.txt change before publishing
Paste the current and proposed files in diff mode to see which URL-and-bot verdicts would change before the new rules reach production.
Find why important pages are not being crawled
Test a URL list against Googlebot and open any blocked result to see the exact Allow or Disallow rule that won.
Audit AI crawler access
Load the major AI crawler user agents in one click and compare how the same paths are treated across GPTBot, ClaudeBot, Google-Extended, and others.
Catch staging and noindex conflicts
Identify cases where robots.txt prevents a crawler from reaching a noindex directive, leaving a supposedly private or removed URL exposed to indexing.
Watch the full workflow
robots.txt Tester walkthrough
Read the transcript

      
      

        
robots.txt Tester

        

        
Hi, this is the robots dot TXT Tester. It checks whether a crawler is allowed to request a page under a specific robots dot TXT file. I’ll explain the basics, run a safe example, show the exact rule behind each result, and compare a proposed change before it goes live.

      

      

        
Step 1

        

        
A robots dot TXT file is a public set of instructions for crawlers. A crawler is software that requests pages, such as Googlebot. This tester answers one clear question: under the file you provide, may this named crawler request this page?

      

      

        
Step 2

        

        
Use it before publishing a risky rules change, when an important page stops being crawled, when search and AI crawlers need different access, or when a staging block hides a noindex instruction. In every case, the goal is to connect a result to the exact rule that caused it.

      

      

        
Step 3

        

        
After adding rules and pages, choose the crawlers here. A user agent is simply the crawler name used for matching, such as Googlebot or GPTBot. Use a preset for a related group, add a custom name when needed, then choose “Test.”

      

      

        
Step 4

        

        
Let’s start with the button labeled “Try an example.” It loads a fictional file, two sample paths, and Googlebot, then runs the check automatically. Example data makes the walkthrough repeatable and keeps real websites or customer information off screen.

      

      

        
Step 5

        

        
The first rule says every crawler should avoid paths beginning with slash search. “Disallow” means do not request a matching path. The next rule creates a narrower exception for slash search slash how search works. We’ll test that exception and a regular search address.

      

      

        
Step 6

        

        
The result is a table. Pages run down the left, selected crawlers run across the top, and every cell says “allowed” or “blocked.” Here one page is allowed and one is blocked. That is a crawling result only; it does not tell us whether either page appears in search.

      

      

        
Step 7

        

        
Now open the red “blocked” cell. The evidence panel repeats the page and crawler, names the matching user-agent group, and identifies “Disallow: slash search” on line two as the winning rule. You do not have to guess which line mattered.

      

      

        
Step 8

        

        
Below the verdict, the tool separates observed evidence from advice. It shows the checked URL, the selected crawler, the winning directive, and the applied group. If the block is accidental, narrow that rule or add a more specific exception, then rerun the same page and crawler.

      

      

        
Step 9

        

        
Open the green result and the highlighted line changes to “Allow: slash search slash how search works.” When more than one rule matches, Google uses the most specific matching path. That longer exception wins over the broader block, regardless of which rule appears first.

      

      

        
Step 10

        

        
The button labeled “Try accidental indexing” shows a common trap. A staging page is blocked from crawling, so Google cannot fetch it to see a noindex instruction on the page. Robots dot TXT can stop a crawl while still leaving the address eligible to appear in search.

      

      

        
Step 11

        

        
For a broader policy review, choose “AI crawler audit.” One click selects the built-in AI crawler names, including GPTBot, ClaudeBot, CCBot, PerplexityBot, and Google-Extended. Run the test to compare how the same pages are treated for every selected name.

      

      

        
Step 12

        

        
If a crawler is missing, type its user-agent name and choose “Add.” The new name becomes a selected chip beside the built-in choices. This is useful for an internal crawler or a newly launched bot, but confirm the correct name with its operator first.

      

      

        
Step 13

        

        
Diff mode compares the current file with a proposed replacement. Here the proposal removes the useful exception and blocks that path too. After “Test,” the summary counts results that would become blocked or allowed, and changed table cells are outlined for review.

      

      

        
Step 14

        

        
The warning says one page-and-crawler result would become blocked and none would become allowed. That small count is the review target. Open the changed cell, confirm whether the new block is intentional, and do not publish until every important change has an owner and a reason.

      

      

        
Step 15

        

        
“File health” checks syntax and directive problems. This example is clean even though one page is blocked, because the valid rule intentionally blocks it. A clean file does not mean every important page is crawlable; use the table for policy results and the linter for file problems.

      

      

        
Step 16

        

        
When the result is ready, choose “Download CSV” to keep the page-by-crawler table or “Copy share link” to hand the setup to someone else. The link stores its state in the address itself rather than saving the audit on the site. Use CSV when the list is too large for a link.

      

      

        
Step 17

        

        
The limits are important. Robots dot TXT controls crawling, not indexing, privacy, or access. A live fetch cannot reveal an older file Google may have cached during an outage, and the test cannot prove a crawler will obey the instructions. Use authentication for private systems and Search Console when indexing is the question.

      

      

        
Step 18

        

        
A good next step is simple: record the page, crawler, verdict, and winning line. Change the narrowest rule that fixes the intent, then rerun this exact case. After publishing, confirm the live file and monitor real crawl behavior separately.

      

      

        
Test the page. Inspect the rule. Verify the fix.

        

        
That’s the complete workflow. Test the pages and crawlers that matter, open every surprising result, make the narrowest safe change, and rerun the same test. Remember that robots dot TXT controls crawling, not indexing or security.

      

    
 
          
 
Common issues & how to fix them
 
     
  •  Warning Directive is misspelled Fix: Correct the directive spelling to a supported name so crawlers do not ignore it. 
  •  Warning Unsupported noindex directive is used Fix: Remove noindex from robots.txt and use a meta robots tag or X-Robots-Tag on crawlable URLs instead. 
  •  Information Crawl-delay is ignored by Google Fix: Remove Crawl-delay for Google and manage Googlebot crawl rate through Search Console or server capacity controls. 
  •  Error Rule appears before any user-agent group Fix: Move the rule below the User-agent line for the crawler group that should apply it. 
  •  Warning Line has no colon separator Fix: Add the missing colon between the directive name and value so parsers can read the line. 
  •  Information Unknown directive is ignored Fix: Remove the nonstandard directive or replace it with a supported robots.txt directive that expresses the intended crawl rule. 
  •  Information Crawler group contains no rules Fix: Add the intended Allow or Disallow rules to this group, or remove the group so the crawler can fall through to the wildcard group. 
  •  Error Rule pattern can never match Fix: Start the path pattern with / or * so matching can begin at the start of the URL path. 
  •  Warning Dollar anchor appears inside a pattern Fix: Move $ to the end of the pattern if it should anchor the path, or remove it if a literal dollar sign was not intended. 
  •  Information User-agent is split across groups Fix: Merge rules for the same user-agent into one group so the effective policy is clear and maintainable. 
  •  Error Sitemap URL is not absolute Fix: Replace the relative sitemap location with a fully qualified URL including https:// and the hostname. 
  •  Error Robots file exceeds 500 KiB Fix: Reduce robots.txt below 500 KiB by consolidating patterns and removing redundant rules before Google’s truncation point. 
  •  Information Robots file is empty Fix: Leave the file empty if all crawling is intentionally allowed; otherwise add explicit user-agent groups and the minimum required rules. 
  •  Error Test URL is blocked for the selected crawler Fix: Remove or narrow the winning Disallow rule, or add a more specific Allow rule, when the tested URL should be crawlable. 
    •