VirtualTam's bookmarks

1. robots.txt - AI / LLM User-Agents: Blocking Guide

   2025-04-29
   network/http robots.txt web/crawling web/scraping

   Permalink
   • A summary of my bot defence systems
     • Butlerian Jihad - Blog posts on the topic of fighting off spam bots, search engine spiders and other non-humans wasting the precious resources we have on Earth
   • EmacsWiki's robots.txt