A set of reusable Java components that implement functionality common to any web crawler
-
Updated
Dec 16, 2024 - Java
#
robots-txt
Here are 6 public repositories matching this topic...
Java sitemap generator. This library generates a web sitemap, can ping Google, generate RSS feed, robots.txt and more with friendly, easy to use Java 8 functional style of programming
java rss sitemap aws-lambda lambda-functions robots-txt robots-generator java-8 sitemap-generator rss-generator java-8-lambda web-sitemap java-sitemap-generator
-
Updated
Dec 10, 2022 - Java
🤖 robots.txt as a service. Crawls robots.txt files, downloads and parses them to check rules through an API
kotlin java api docker redis crawler spring-boot gradle docker-compose makefile postgresql robots-txt antlr4 spiders robots-parser crawler-engine redis-stream redis-streams
-
Updated
Dec 2, 2020 - Java
NextTypes is a standards based information storage, processing and transmission system that integrates the characteristics of other systems such as databases, programming languages, communication protocols, file systems, document managers, operating systems, frameworks, file formats and hardware in a single tightly integrated system using a comm…
javascript java http sitemap json sql database html5 rest icalendar css3 xml utf-8 caldav postgresql webdav x509 robots-txt smtp full-text-search
-
Updated
Dec 13, 2024 - Java
🚫🤖 Override /robots.txt to disallow all web crawlers, regardless settings stored in the database. Compatible with Liferay 7.0, 7.1, 7.2, 7.3 and 7.4.
web osgi robots-txt crawlers liferay liferay-portal liferay-dxp liferay-7 liferay71 liferay70 liferay72 liferay-71 liferay-72 liferay73 webcrawlers liferay-73 liferay-74 liferay74 liferay-70
-
Updated
Sep 4, 2021 - Java
This tool, written in Java, downloads website source code and stores in a MySQL database for processing.
-
Updated
Jul 13, 2017 - Java
Improve this page
Add a description, image, and links to the robots-txt topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the robots-txt topic, visit your repo's landing page and select "manage topics."