Robots.txt parser / generator

Front-end workflow to start a new project with Eleventy and Webpack.

Generates a robots.txt

The repository contains Google-based robots.txt parser and matcher as a C++ library (compliant to C++17).

🚫🤖 Override /robots.txt to disallow all web crawlers, regardless settings stored in the database. Compatible with Liferay 7.0, 7.1, 7.2, 7.3 and 7.4.

This is a python crawler that disregards robots.txt rules and downloads disallowed resources

A tool for debugging robots.txt

🌐 Displays the contents of robots.txt and sitemap.xml files of a website google extension

Movie web-application. Inspiration from Hemmakvälls website. Created with Vite, implementing Redux, SEO, and tests with Cypress. Using TMDB api. Styling under progress.

Scripts to create a robots.txt file from building blocks

documents my master's level thesis work on building continous, topical web crawler based on mercator 1999

The Robots.txt Generator tool helps you to create the Robots.txt file for your website.

Determining bias to search engines from Robots.txt

Sharp SEO Tools is collection of free web tools completely written in Javascript (19 tools available), feel free to use

Open source robots.txt denylist for uncredited AI crawlers

sitemap2posts is designed to identify all posts for a blog, using a specified URL path, along with titles and published dates, ready to be added to history4feed.

Integrations dedicated to search engines and social media plattforms for all sites of the WordPress multisite network figuren.theater

Website of Alp Ozkan: Product Manager living in Istanbul, passionate about UX, Software Development and Web3.

YiraBot: Simplifying Web Scraping for All. A user-friendly tool for developers and enthusiasts, offering command-line ease and Python integration. Ideal for research, SEO, and data collection.