VirtualTam's bookmarks

1. robots.txt - AI / LLM User-Agents: Blocking Guide

   2025-04-29
   network/http robots.txt web/crawling web/scraping

   Permalink
   • A summary of my bot defence systems
     • Butlerian Jihad - Blog posts on the topic of fighting off spam bots, search engine spiders and other non-humans wasting the precious resources we have on Earth
   • EmacsWiki's robots.txt
2. A different approach to blocking bad webbots by IP address

   2025-04-28
   feed network/autonomous-system network/http network/ip web/crawling web/scraping

   Permalink
   • Now a bit about feed readers
   • A deeper dive into mapping web requests via ASN, not by IP address
   • Notes on blocking spam by filtering on ASN
3. Archiving URLs | Gwern

   2025-04-09
   network/http personal-knowledge/bookmark personal-knowledge/digital-garden programming/go programming/html programming/java programming/python web web/scraping

   Permalink
   • Archiving Web Sites (LWN)
   • Robots.txt meant for search engines don’t work well for web archives

   Web page archive formats:

   • ARC (1996)
   • WARC (BnF, 2009)
     • WARC Specifications
     • The WARC Ecosystem

   Tools for crawling, scraping and archiving Web pages:

   • internetarchive/heritrix3 - Extensible, web-scale, archival-quality web crawler project (Java)
   • internetarchive/Zeno - State-of-the-art web crawler (Go)
     • internetarchive/gowarc - Read and write WARC files in Go
   • webrecorder/pywb - Web Archiving Toolkit for replay and recording of web archives (Python)

   Self-hosted solutions:

   • ArchiveBox - A self-hosted app that lets you preserve content from websites in a variety of formats
   • Wallabag - Save and classify articles, read them later
4. Scrapy | A Fast and Powerful Scraping and Web Crawling Framework

   2018-05-15
   network/http programming/python programming/xml web web/scraping

   Permalink
   • https://github.com/scrapy/scrapy
   • https://doc.scrapy.org/en/latest/