robots.txt snippets to block crawlers and bots.

Trying to “compete” in the world of writing, art, music etc is hopeless in the era of the global village where people can spend their time reading texts from any place and any era and even have AI-generated, tailor-made entertainment.

Write for yourself and for your friends.

Test and validate your robots.txt with this testing tool. Check if a URL is blocked, which statement is blocking it and for which user agent. You can also check if the resources for the page (CSS, JavaScript, images) are disallowed.