🐘 Simple PHP library to help developers 🍻 do better on-page SEO optimization 🤖

Determine if a page may be crawled from robots.txt, robots meta tags and robot headers

Php class for robots.txt parse

A complete SEO solution for Symfony projects. This bundle handles meta tags, Open Graph, Twitter Cards, canonical URLs, sitemaps, and more—helping your app stay search-engine friendly and socially shareable out of the box.

An extensible robots.txt parser and client library, with full support for every directive and specification.

Manage the robots.txt from the Kirby config file

A Multisite Robots.txt Manager - Quickly and easily manage all robots.txt files on a WordPress Multisite Website Network.

Laravel package to manage robots

Behat extension for testing some On-Page SEO factors: meta title/description, canonical, hreflang, meta robots, robots.txt, redirects, sitemap validation, HTML validation, performance...

Simple robots generation module for Silverstripe (SS 4 and above)

TYPO3 sitemap crawler

Optimizes your site's robots.txt to reduce server load and CO2 footprint by blocking unnecessary crawlers while allowing major search engines and specific tools.

Robots Exclusion Standard/Protocol Parser for Web Crawling/Scraping

Declarative, scriptable web robot (crawler) and scrapper

Robots for Kirby CMS

This is Pico's official robots plugin to add a robots.txt and sitemap.xml to your website. Pico is a stupidly simple, blazing fast, flat file CMS.

Middleware to avoid search engine indexing with PSR-7 using robots.txt and X-Robots-Tag

A modern, fluent PHP package for managing robots.txt rules with type safety and great developer experience

This is ready to use template to quickly start selling domain with minimum setup.

PSR-15 middleware to enable/disable the robots of the search engines