A powerful and customizable sitemap generator written in PHP (PHP 8+).
It crawls one or multiple domains, respects robots.txt, follows meta directives, supports resumable sessions, sends logs by email, and can even notify search engines when the sitemap is ready. It is optimized for large websites and offers advanced crawl controls, meta/robots filtering, JSON/HTML export, and more.

• 🔗 Multi-domain support (comma-separated URLs)
• 📑 Combined sitemap for all domains
• 📑 Automatically creates multiple sitemap files if more than 50,000 URLs are found
• 🧭 Crawling depth control
• 🔍 robots.txt and <meta name="robots"> handling
• 🔁 Resumable crawl via cache (optional)
• 💣 --resetcache to force full crawl (new!)
• 💣 --resetlog to delete old log files (new!)
• 🧠 Dynamic priority & changefreq rules (via config or patterns)
• 🧹 Pretty or single-line XML output
• 📦 GZIP compression (optional)
• 📧 Log by email
• 🛠 Health check report
• 📡 Ping Google/Bing/Yandex
• 🧪 Debug mode with detailed logs
• 📑 Structured logging with timestamps and levels (info, error, debug, etc.)
• 📑 Log export: JSON format + HTML report (logs/crawl_log.json, logs/crawl_log.html)
• 📑 Visual crawl map generation (crawl_graph.json, crawl_map.html)
• 📧 Flattened email reports with attached crawl logs
• 📧 Customizable sender email via --from
• 📑 Public base URL for sitemap/map references via --publicbase

• PHP 8.0 or newer
• curl and dom extensions enabled
• Write permissions to the script folder (for logs/cache/sitemaps)

php sitemap.php \
  --url=https://yourdomain.com,https://blog.yourdomain.com \
  --key=YOUR_SECRET_KEY \
  [options]

sitemap.php?url=https://yourdomain.com&key=YOUR_SECRET_KEY&gzip&prettyxml

• sitemap.xml or sitemap-*.xml
• sitemap.xml.gz (optional)
• sitemap_index.xml (if split)
• cache/visited.json → stores crawl progress (used with --resume)
• logs/crawl_log.txt → full crawl log
• logs/crawl_log.json → Structured log as JSON
• logs/crawl_log.html → Visual HTML report of the crawl log
• logs/crawl_report_*.txt → emailed attachment
• logs/health_report.txt → summary of crawl (errors, speed, blocks)
• crawl_graph.json → Graph structure for visualization
• crawl_map.html → Interactive crawl map

Create a config/filter.json to define your own include/exclude patterns and dynamic rules:

{
  "excludeExtensions": ["jpg", "png", "zip", "docx"],
  "excludePatterns": ["*/private/*", "debug"],
  "includeOnlyPatterns": ["blog", "news"],
  "priorityPatterns": {
    "high": ["blog", "news"],
    "low": ["impressum", "privacy"]
  },
  "changefreqPatterns": {
    "daily": ["blog", "news{
      "excludeExtensions": ["jpg", "png", "docx", "zip"],
      "excludePatterns": [],
      "includeOnlyPatterns": [],
      "priorityPatterns": {
        "high": [
          "news",
          "blog",
          "offers"
        ],
        "low": [
          "terms-and-conditions",
          "legal-notice",
          "privacy-policy"
        ]
      },
      "changefreqPatterns": {
        "daily": [
          "news",
          "blog",
          "offers"
        ],
        "monthly": [
          "terms-and-conditions",
          "legal-notice",
          "privacy-policy"
        ]
      }
      }"],
    "monthly": ["impressum", "agb"]
  }
}

Activate with:

--filters --priorityrules --changefreqrules

With --ping enabled, the script will notify:

• Yandex: https://webmaster.yandex.com/ping

As of 2023/2024:

• ❌ Google and Bing ping endpoints are deprecated (410 Gone)
• ✅ Use robots.txt with a Sitemap: entry
• ✅ Optionally submit in Webmaster Tools

The script requires a secret key (--key= or key=) to run.
Set it inside the script:

$authorized_hash = 'YOUR_SECRET_KEY';

Send crawl reports to your inbox with:

--email=you@yourdomain.com

Your server must support the mail() function.

Enable --debug to log everything:

• Pattern matches
• Skipped URLs
• Meta robots blocking
• Robots.txt interpretation
• Response times
• Log file resets

If more than 50,000 URLs are crawled (the limit of a single sitemap file per sitemaps.org spec),
the script will automatically create multiple sitemap files:

• sitemap-1.xml, sitemap-2.xml, ...
• Or domain-a-1.xml, domain-a-2.xml, ... if --splitbysite is active
• These are automatically referenced from a sitemap_index.xml

No configuration is needed – the split is automatic.

When using --splitbysite, the crawler will:

1. Create a separate sitemap file for each domain (e.g., /sitemaps/domain1.xml, /sitemaps/domain2.xml)
2. Automatically generate a sitemap_index.xml file in the root directory
3. Ping search engines (Google, Bing, Yandex) with the sitemap_index.xml URL instead of individual sitemap files

This is useful when crawling multiple domains in a single run.

If you enable --graph, the crawler will export:

• graph.json – link structure as raw data
• crawl_map.html – interactive map powered by D3.js

You can explore your site structure visually, zoom in/out, drag nodes, and inspect links. Useful for spotting crawl traps, dead ends, and structure gaps.

📍 Tip: For large sites, open the HTML file in Chrome or Firefox.

User-agent: *
Disallow:

Sitemap: https://yourdomain.com/sitemap.xml

MIT License
Feel free to fork, modify, or contribute!

Built by Gilles Dumont (Qiubits SARL)
Contributions and feedback welcome.