Robots.txt parser

This is a collection of robots.txt templates

Provides python access to Googles parser for robot.txt files as used by their GoogleBot webscraper.

The repository contains Google-based robots.txt parser and matcher as a C++ library (compliant to C++17).

🚫🤖 Override /robots.txt to disallow all web crawlers, regardless settings stored in the database. Compatible with Liferay 7.0, 7.1, 7.2, 7.3 and 7.4.

This is a python crawler that disregards robots.txt rules and downloads disallowed resources

Movie web-application. Inspiration from Hemmakvälls website. Created with Vite, implementing Redux, SEO, and tests with Cypress. Using TMDB api. Styling under progress.

SixArm.com » Apache webserver » robots.txt configuration file

Fully native robots.txt parsing component without any dependencies.

Website sitemap parser

Robots.txt and sitemap.xml generator

SEO Master is a powerful all-in-one tool developed to boost your website's visibility and rankings. With features like automatic sitemap generation, customizable robots.txt creation, SEO-optimized metadata, Image assets generation and seamless integration with major search engines.

Scripts to create a robots.txt file from building blocks

The Robots.txt Generator tool helps you to create the Robots.txt file for your website.

Determining bias to search engines from Robots.txt

Sharp SEO Tools is collection of free web tools completely written in Javascript (19 tools available), feel free to use

sitemap2posts is designed to identify all posts for a blog, using a specified URL path, along with titles and published dates, ready to be added to history4feed.

Integrations dedicated to search engines and social media plattforms for all sites of the WordPress multisite network figuren.theater

A simple script to open all the pages in a website's robots.txt files

The Blogger Robots.txt Generatois a tool designed to simplify the process of creating a robots.txt file for websites hosted on the Blogger platform. A robots.txt file is crucial for controlling how search engines index your site. This generator allows users to customize and generate a robots.txt file tailored to their specific needs.