VirtualTam's bookmarks

1. robots.txt - AI / LLM User-Agents: Blocking Guide 2025-04-29

   tags: network/http, robots.txt, web/crawling, web/scraping

   [ permalink ]

   • A summary of my bot defence systems
     • Butlerian Jihad - Blog posts on the topic of fighting off spam bots, search engine spiders and other non-humans wasting the precious resources we have on Earth
   • EmacsWiki's robots.txt
2. Analyzing One Million robots.txt Files 2022-06-19

   tags: parser, programming/c++, programming/python, robots.txt

   [ permalink ]

   • https://docs.python.org/3/library/urllib.robotparser.html
   • https://github.com/scrapy/scrapy/issues/3656
   • https://github.com/scrapy/scrapy/issues/5226
   • https://github.com/seomoz/reppy
   • https://github.com/seomoz/reppy/issues/132