Here are
11 public repositories
matching this topic...
Parsers for robots.txt (aka Robots Exclusion Standard / Robots Exclusion Protocol), Robots Meta Tag, and X-Robots-Tag
PowerShell module for reading robots.txt files
Tool to analyze and parse website robots.txt for crawler rules.
Parse robots.txt and traverse sitemaps.
An admin extension for Optimizely CMS 12+ for managing robots.txt on a per site basis.
A friendly tool for creating dynamic robots.txt files in Umbraco
Provides robots.txt middleware for .NET core
This package helps you to add meta-tags, sitemap.xml and robots.txt into your project easily.
A "robots.txt" parsing and querying library for .NET
.Net Core Plugin Manager, extend web applications using plugin technology enabling true SOLID and DRY principles when developing applications
A simple but powerful web crawler library for .NET
Improve this page
Add a description, image, and links to the
robots-txt
topic page so that developers can more easily learn about it.
Curate this topic
Add this topic to your repo
To associate your repository with the
robots-txt
topic, visit your repo's landing page and select "manage topics."
Learn more
You can’t perform that action at this time.