A simple but powerful web crawler library for .NET
-
Updated
Dec 15, 2023 - C#
#
robots-txt
Here are 11 public repositories matching this topic...
A "robots.txt" parsing and querying library for .NET
-
Updated
Sep 3, 2024 - C#
.Net Core Plugin Manager, extend web applications using plugin technology enabling true SOLID and DRY principles when developing applications
-
Updated
May 14, 2026 - C#
This package helps you to add meta-tags, sitemap.xml and robots.txt into your project easily.
-
Updated
May 26, 2023 - C#
A friendly tool for creating dynamic robots.txt files in Umbraco
-
Updated
Feb 27, 2022 - C#
An admin extension for Optimizely CMS 12+ for managing robots.txt on a per site basis.
-
Updated
Aug 26, 2026 - C#
Parse robots.txt and traverse sitemaps.
sitemap spider crawl robots-txt robots useragent robots-parser useragentparser robotstxt sitemap-url
-
Updated
Feb 7, 2025 - C#
Parsers for robots.txt (aka Robots Exclusion Standard / Robots Exclusion Protocol), Robots Meta Tag, and X-Robots-Tag
robots-txt robots meta-tags robots-parser robots-exclusion-standard robotstxt robots-tag robots-exclusion-protocol x-robots-tag
-
Updated
Apr 11, 2026 - C#
Tool to analyze and parse website robots.txt for crawler rules.
-
Updated
Nov 25, 2024 - C#
Distributed, context-aware search engine built in .NET with an async pipeline and event-driven architecture.
search-engine distributed-systems microservices scalable csharp pipeline dotnet fault-tolerance high-performance web-crawler concurrency bloom-filter channels robots-txt event-driven aspnet-core pub-sub deduplication content-extraction context-aware-ranking
-
Updated
Apr 5, 2026 - C#
Add this topic to your repo
To associate your repository with the robots-txt topic, visit your repo's landing page and select "manage topics."