Languages

Log File Analyzer

Free, no signup. Server logs answer a question your analytics can't: what are crawlers actually doing on your site. Drop one in and see crawl budget by bot and by section, how much is wasted on 404s and redirects, how AI crawlers compare to search engines, and which requests are impostors faking a crawler user-agent.

Built by Patrick Stox

Handles nginx / Apache (combined & common), IIS / W3C Extended, and JSON logs (incl. Cloudflare Logpush) — format auto-detected. Crawler ranges updated 18 days ago.

Drop your access log here
or ·
Multiple files allowed. Big logs stream in chunks.

Runs entirely in your browser — nothing you paste is uploaded or stored. Spot-checking a suspicious IP against the Googlebot Verifier sends only that IP to our server. Anonymous run-level outcome counters may be used for aggregate research; URLs, domains, IPs, and identifiers are never included, and no statistic is released below 100 runs.

Feedback
Report a bug

Found something broken in Log File Analyzer? Let us know what happened — this goes straight to a private triage queue, not a public list.

What will be sent 
 No tool inputs, uploads, pasted source, complete results, query parameters, or URL fragments are attached automatically. You can edit or remove the selected passage above. Browser and anti-abuse metadata is processed for spam prevention. 
  
 
  
 
 
  
 
 
      
 
Sample report
 
Say your log contains two lines — a genuine Googlebot hit and a Googlebot impostor:
 
 
66.249.66.1 - - [08/Jul/2026:09:14:22 +0000] "GET /blog/technical-seo-guide HTTP/1.1" 200 8213 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
193.42.33.7 - - [08/Jul/2026:09:14:41 +0000] "GET /wp-login.php HTTP/1.1" 404 402 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
 
 
…and the analyzer returns:
 
Example data — captured by running this page's own analyzer against the two lines above
 
 
 
2
requests parsed
all lines parsed
 
2
crawler requests
100% of traffic
 
1
verified
IP in published ranges
 
1
likely spoofed
crawler UA, wrong network
 
 
 
BotTypeRequestsVerifiedSpoofedWaste
  
 Googlebot
Google
 Search 2 1 1 50% 
  
 
 
IPClaimed to beRequestsUser-agent
  
 193.42.33.7 Googlebot 1 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) 
  
 
 
1 requests spoofed a verified crawler
 
1 IP sent a search/AI-crawler User-Agent from outside that operator's published ranges — robots.txt won't stop them (impostors ignore it). Block or rate-limit these at the firewall/WAF by IP or ASN. Top offenders: 193.42.33.7 (as Googlebot, ×1). Run each through the Googlebot Verifier for the owning network (ASN) before blocking.
 
 
 
     
  • Both lines roll up under one Googlebot row — the table groups by crawler
            identity, not by verification status, so the genuine 200 and the spoofed 404 both count toward
            the same bot's totals. That's why Waste is 50% (1 of 2 requests was non-200), not
            0% — see what waste means ↓.
    •  
  • Verified vs. spoofed is per-request, shown per-IP in the spoofer table below the
            bot row: 193.42.33.7 isn't in Google's published ranges, so its Googlebot user-agent
            is flagged rather than trusted.
    •  
  • The recommendation is severity-ranked — a spoofed crawler hitting
/wp-login.php is high severity because robots.txt can't stop an impostor
            that ignores it; only a firewall/WAF IP or ASN block does. Confirm the real owner first with the
Googlebot Verifier.
    •  
 
 
 
How to use it
 
     
  1. Export an access log from your server, CDN, or Cloudflare Logpush. Common formats
            (nginx/Apache, IIS/W3C, JSON) all work — the format is auto-detected.
    2.  
  2. Drop the file onto the upload box, or browse to it. You can
            drop several files at once; large logs stream in chunks. Nothing is uploaded — parsing runs in
            your browser. No log handy? Click load demo data.
    3.  
  3. Read the summary cards first (requests parsed, crawler share, verified vs.
            spoofed), then check the llms.txt fetch evidence before working down through
            crawl budget by bot, the spoofer report, section budget, and the most-wasted URLs.
    4.  
  4. Check the Recommendations panel for ranked fixes and copy-paste robots.txt
            lines. In the spoofer report, hit Verify IP to run a live reverse-DNS + ASN
            spot-check on any suspicious address.
    5.  
  5. Use Export bots CSV or Export wasted URLs CSV to pull the
            findings into a spreadsheet or ticket.
    6.  
 
 
      
      
 

A server access log is the ground truth for how crawlers actually behave on your site — more
          complete than any report, because it records every single request. This tool turns a raw log into
          a crawl-budget picture: who's crawling, what they're wasting budget on, and who's faking a crawler
          identity. For the reasoning behind each metric, see the
AI crawler log analysis guide and the
crawl budget explainer.

 
What the results mean
 
The verification badges on each bot and IP tell you how much to trust a request's identity:
 
     
  • Verified — the source IP falls inside the crawler
            operator's published ranges. This is a genuine crawler.
    •  
  • Spoofed — a crawler user-agent from an IP
outside that operator's ranges. A likely impostor; block by IP or ASN.
    •  
  • Unverifiable — a known crawler whose operator
            publishes no IP ranges to check against (e.g. Bytespider), so genuine and fake can't be
            separated by list alone.
    •  
 
Crawlers are grouped by type, and each group is colour-coded:
 
     
  • Search — traditional search-index crawlers (Googlebot, bingbot).
    •  
  • AI training — bots that fetch pages to train models (e.g. GPTBot).
    •  
  • AI search — bots that fetch pages to answer live AI queries (e.g. OAI-SearchBot).
    •  
  • User-triggered — fetches a person kicked off, such as a citation preview.
    •  
  • Other / unclassified — no known crawler user-agent: human browsers and untracked tools.
    •  
 
 Waste is the share of a bot's, section's, or URL's requests that returned a
          redirect (excluding 304), 4xx, or 5xx. The status mix bar breaks all responses down: green 2xx, amber 3xx, red 4xx, dark-red 5xx.
          Recommendations carry a severity — high (red), medium (amber),
          or info (blue) — reflecting how much budget the fix reclaims.

 
How it works
 
     
  1. Parsed in your browser. A Web Worker streams the file chunk by chunk and
            parses each line — nginx/Apache combined & common, IIS/W3C Extended, and JSON (including
            Cloudflare Logpush). The file is never uploaded.
    2.  
  2. Every request classified. The user-agent is matched to a known crawler, then
            the source IP is checked against that operator's published ranges — the same weekly-refreshed
            lists the Googlebot Verifier uses. A crawler UA from an IP outside
            its ranges is flagged as a likely spoofer.
    3.  
  3. Aggregated, not stored. Requests roll up into crawl budget, status-code
            waste, section and AI-vs-search breakdowns, and a spoofer report. Close the tab and it's gone.
    4.  
  4. Optional spot-checks. For a suspicious IP you can run a full
            forward-confirmed reverse-DNS check and ASN lookup — the only step that contacts a server,
            rate-limited, one IP at a time.
    5.  
 

Crawler IP ranges were last refreshed
18 days ago.

 
Features
 
     
  • Auto-detects nginx/Apache combined & common, IIS/W3C Extended, and JSON (Cloudflare Logpush) logs; multiple files at once.
    •  
  • Crawl budget by bot with verified/spoofed/unverifiable counts, unique IPs, status mix, and waste percentage.
    •  
  • AI-vs-search breakdown so you can see how much of your budget AI crawlers now consume versus traditional search.
    •  
  • Spoofer report listing crawler-UA requests from out-of-range IPs, with a live per-IP reverse-DNS + ASN verify button.
    •  
  • Crawl budget by section and a most-wasted-URLs table pinpointing the highest-leverage fixes.
    •  
  • Daily crawl-over-time trend chart when the log carries timestamps.
    •  
  • Ranked recommendations with copy-paste robots.txt lines, plus CSV export of bots and wasted URLs.
    •  
 
Limitations
 

It analyses what's in the log — it can't see requests your server never logged, and it can't tell
          you why a crawler chose a URL. Verification is only as good as the operators' published
          ranges: crawlers with no published ranges are marked unverifiable rather than judged, and a
          determined impostor using a genuine-crawler IP won't be caught by list matching alone (use the
          reverse-DNS spot-check for those). Sections are inferred from URL paths, so your site's structure
          determines how clean the grouping is. It reports crawl behaviour, not indexing outcomes — pair it
          with Search Console to connect a crawled URL to whether it actually ranks.

 
Frequently asked questions
 
Are my log files uploaded anywhere?
No. Parsing and analysis run entirely in your browser through a Web Worker — the file is read from disk and streamed in chunks locally, and nothing is sent to a server. The one exception is the optional per-IP spot-check in the spoofer report, which sends a single IP address (not your log) to the verifier API to run a forward-confirmed reverse-DNS and ASN lookup. Close the tab and all parsed data is gone.
What log formats does it support?
The format is auto-detected. It reads nginx and Apache combined and common formats, IIS / W3C Extended, and JSON logs including Cloudflare Logpush. You can drop multiple files at once, and large logs stream in chunks so a multi-gigabyte access log will not lock up the tab.
How does it tell a real crawler from a spoofed one?
Each request is matched to a known crawler by user-agent, then the source IP is checked against that operator’s published IP ranges — the same weekly-refreshed lists the GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. Verifier uses. A request that sends, say, a Googlebot user-agent from an IP outside Google’s ranges is flagged as a likely spoofer. Some operators (like ByteDance’s Bytespider) publish no ranges, so those hits are marked unverifiable rather than spoofed.
What counts as wasted crawl budget?
Any crawler requestA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. that gets a redirect (except a useful 304 Not ModifiedHTTP 304 Not Modified is the response to a conditional GET/HEAD whose condition evaluates false, in the 3xx class but not a redirect: it has no Location header and no body. It tells a client its cached copy is still valid. For SEO it has no direct ranking effect and no indexing effect beyond a possible signal recalculation, though it can help crawl efficiency indirectly on large sites. revalidation), a 4xx error such as 404 or 410, or a 5xx server error. The tool rolls waste up per bot, per section, and per individual URL so you can see where the highest-leverage fixes are.
Can I block spoofers with robots.txt?
No. Impostors send a crawler user-agent precisely so they can ignore robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere., so a Disallow line does nothing to them — block them by IP or ASN at the edge or firewall instead. robots.txt is the right tool for genuine crawlers wasting budget on sections you don’t want crawled, and the Recommendations panel generates copy-paste Disallow lines for exactly those cases.
 
 
 
       
Next steprobots.txt Generator — generate the corrected version.

People also explore
ToolGooglebot VerifierToolrobots.txt TesterToolAI-Crawler Access CheckerArticleUser AgentArticleCrawlingArticleCrawl Budget
 
 
 
Feature requests for Log File Analyzer
 

Upvote what you want most. New ideas can be submitted from the floating Feedback menu;
      requests appear here once approved, and the most-wanted rise to the top.

   
Loading…
  
  ➕ Request a feature   
New requests are reviewed before they appear here.
 
     
  
  
 
 
 
 
     
 
Where this tool helps
Common use cases
Measure crawler activity
Parse nginx, Apache, IIS/W3C, or JSON access logs to see crawl volume by named bot and site section.
Find crawl-budget waste
Surface repeated requests to redirects, errors, parameters, and low-value sections that deserve investigation.
Compare AI and search crawlers
Break down observed requests by crawler family without treating user-agent strings as verified identity.
Flag possible bot spoofing
Use the spoofer report to prioritize IP and reverse-DNS verification for clients claiming a crawler user agent.
Watch the full workflow
Log File Analyzer walkthrough
Read the transcript

      
      

        
Log File Analyzer

        

        
Analytics misses much of what crawlers request. I’ll show you when server-log analysis helps, how to load files and expected U-R-Ls, read the built-in demo, interpret bot identity and crawl waste, use the main reports and exports, respect the limitations, and plan the next fixes.

      

      

        
Step 1

        

        
A server access log records requests that analytics may never execute or classify. This analyzer summarizes recognized crawlers, verification, response waste, sections, and suspicious identities. Use it for crawl-budget investigations, migrations, recurring errors, A-I crawler monitoring, and security triage.

      

      

        
Step 2

        

        
Drop nginx, Apache, I-I-S, W-three-C, J-S-O-N, or Cloudflare logs; formats are auto-detected and multiple large files stream locally. Choose a representative date range. Optionally paste expected sitemap paths and select section depth to compare known U-R-Ls with crawler requests.

      

      

        
Step 3

        

        
Select Load demo data to explore safely, or browse for your own file. The demo is generated inside the page and processed by the real browser worker. Your logs remain local. Only an optional single-I-P reverse-D-N-S and A-S-N spot-check contacts the verifier service.

      

      

        
Step 4

        

        
Begin with total parsed requests, recognized crawler share, verified hits, likely spoofed hits, and the observed date range. These cards define the denominator for every later percentage. Confirm the parser recognized enough lines and the window matches the period you intended to investigate.

      

      

        
Step 5

        

        
The category view separates traditional search, A-I training, A-I search, user-triggered fetches, and unclassified traffic. Use it to understand crawler mix, not user demand. Check the l-l-m-s dot t-x-t fetch evidence separately because a request proves fetching, not compliance or downstream use.

      

      

        
Step 6

        

        
The bot table groups requests by claimed crawler and shows verification, unique I-Ps, response mix, and waste. Waste includes redirects other than three-oh-four revalidation, four-hundred errors, and five-hundred errors. Prioritize high request volume and high waste together, not a large percentage from one hit.

      

      

        
Step 7

        

        
A crawler user agent from outside its operator’s published ranges is flagged as likely spoofed. Robots dot t-x-t will not stop an impostor. Before blocking, spot-check the I-P owner, confirm repeated behavior, and apply a scoped firewall or edge rule. Unverifiable means no usable published range, not malicious.

      

      

        
Step 8

        

        
Section and crawler-by-page-type reports reveal where each bot concentrates requests. Look for templates producing repeated redirects, soft migrations, parameter traps, or errors. Path-based grouping is an approximation, so choose section depth that matches your site architecture and validate specific U-R-L examples.

      

      

        
Step 9

        

        
The most-wasted U-R-L table ranks redirect and error destinations by crawler request count. Confirm the intended final response, repair internal links, maps, canonicals, sitemaps, or routing rules, and avoid blocking a path that should instead return useful content. Export rows for tickets and owners.

      

      

        
Step 10

        

        
Main features include streamed multi-format parsing, trend charts, crawler verification, section and template matrices, expected-U-R-L coverage, A-I-invented-path candidates, ranked recommendations, robots suggestions for genuine crawlers, C-S-V exports, and private baseline snapshots. Simulated path probes are weaker than observed log evidence.

      

      

        
Step 11

        

        
The tool only sees recorded requests and cannot explain why a crawler chose a U-R-L or whether the page was indexed. Identity depends on published ranges and optional network checks. Path sections are inferred, sampling or retention can bias counts, and observed A-I requests do not prove training, citation, or answer use.

      

      

        
Step 12

        

        
Record the log source, time zone, window, parser coverage, crawler, verification state, response code, section, and U-R-L. Separate spoofing controls from crawl-efficiency fixes. Pair findings with Search Console and server configuration, assign the highest-volume causes, save a baseline, and analyze a comparable later window.

      

      

        
Fix the largest verified waste first.

        

        
Export the bot and wasted-U-R-L evidence, verify suspicious networks before blocking, and fix high-volume error or redirect patterns at their source. Then collect another representative log window and compare the same sections, bots, statuses, and coverage to confirm the change.

      

    
 
          
 
Common issues & how to fix them
 
     
  •  Warning Crawler claim is outside the operator’s published range Fix: Treat the crawler as unverified and block or investigate it until its IP matches the operator’s published range. 
  •  Warning Crawler spends requests on non-useful URLs Fix: Disallow or canonicalize low-value parameter, filter, search, and duplicate URLs, then strengthen links to the URLs crawlers should prioritize. 
  •  Warning Crawler repeatedly receives error responses Fix: Repair or redirect the most-requested crawler URLs returning 4xx/5xx responses and remove internal links to URLs that should stay gone. 
  •  Needs review Crawler encounters excessive redirects Fix: Update internal links and sitemap entries to point directly to final URLs, and collapse redirect chains to one hop. 
  •  Needs review Crawler visits URL outside the expected URL set Fix: Either add the crawler-discovered URL to the canonical internal-link and sitemap set or redirect/remove it if it is unintended. 
  •  Needs review Expected important URL has no crawler hits Fix: Add crawlable internal links and a canonical sitemap entry for the expected URL, then check robots and authentication barriers. 
  •  Warning Crawler activity drops materially Fix: Compare the drop date with robots, status, sitemap, deployment, and server-capacity changes, then restore the affected crawl path. 
  •  Needs review Crawler activity spikes materially Fix: Identify the URL pattern causing the spike and contain newly exposed parameters, traps, redirects, or duplicate crawl paths. 
  •  Needs review Crawler response times are slow Fix: Cache or optimize the slowest crawler endpoints and reduce origin latency before requesting a fresh crawl. 
    •