Go robots.txt parser

Robots.txt parser and generator - Work in progress

Package robots implements robots.txt file parsing and matching based on Google's specification.

A simple and flexible web crawler that follows the robots.txt policies and crawl delays.

Polite, slim and concurrent web crawler.

grobotstxt is a native Go port of Google's robots.txt parser and matcher library.

robots.txt parser

The robots.txt exclusion protocol implementation for Go language

Enumerate old versions of robots.txt paths using Wayback Machine for content discovery

Go language library for parsing Sitemaps

Parsero is a free script written in Golang which reads the Robots.txt file of a web server and looks at the Disallow entries. The Disallow entries tell the search engines what directories or files hosted on a web server mustn't be indexed.