robots-txt

Here are 3 public repositories matching this topic...

Privacy Web Search Engine (not meta, own crawler)

Lightweight R wrapper around rep-cpp for robot.txt (Robots Exclusion Protocol) parsing and path testing in R

The repository contains Google-based robots.txt parser and matcher as a C++ library (compliant to C++17).

Add a description, image, and links to the robots-txt topic page so that developers can more easily learn about it.

To associate your repository with the robots-txt topic, visit your repo's landing page and select "manage topics."