advertools - online marketing productivity and analysis tools

Opt-Out tool to check Copyright reservations in a way that even machines can understand.

Ultimate Website Sitemap Parser

Open-Source Python Based SEO Web Crawler

🤖 A curated list of websites that restrict access to AI Agents, AI crawlers and GPTs

ScrapeGPT is a RAG-based Telegram bot designed to scrape and analyze websites, then answer questions based on the scraped content. The bot utilizes Retrieval Augmented Generation and webscraping to return natural language answers to the user's queries.

Python-based web crawling script with randomized intervals, user-agent rotation, and proxy server IP rotation to outsmart website bots and prevent blocking.

Dark Web Informationgathering Footprinting Scanner and Recon Tool Release. Dark Web is an Information Gathering Tool I made in python 3. To run Dark Web, it only needs a domain or ip. Dark Web can work with any Linux distros if they support Python 3. Author: AKASHBLACKHAT(help for ethical hackers)

A python script to check if URLs are allowed or disallowed by a robots.txt file.

YiraBot: Simplifying Web Scraping for All. A user-friendly tool for developers and enthusiasts, offering command-line ease and Python integration. Ideal for research, SEO, and data collection.

a tool that gets all paths at robots.txt and opens it in the browser.

🕵️‍♂️ɪɴғᴏʀᴍᴀᴛɪᴏɴ ɢᴀᴛʜᴇʀɪɴɢ ᴛᴏᴏʟ🕵️‍♂️

Monitor and report changes across one or more robots.txt files.

pyrobotstxt: Python Package for robots.txt Files

Scan websites for multiple things like honeypot, whois , port scan etc...

ROBOTS.TXT SCANNER

Visual App for Testing URLs and User-agents blocked by robots.txt Files

Fast and reliable python tool that grabs robots.txt files from a bunch of subdomains asynchronously

Datasette plugin that blocks robots and crawlers using robots.txt

AI Data Guard