VirtualTam's bookmarks

1. Archiving URLs | Gwern

   2025-04-09
   network/http personal-knowledge/bookmark personal-knowledge/digital-garden programming/go programming/html programming/java programming/python web web/scraping

   Permalink
   • Archiving Web Sites (LWN)
   • Robots.txt meant for search engines don’t work well for web archives

   Web page archive formats:

   • ARC (1996)
   • WARC (BnF, 2009)
     • WARC Specifications
     • The WARC Ecosystem

   Tools for crawling, scraping and archiving Web pages:

   • internetarchive/heritrix3 - Extensible, web-scale, archival-quality web crawler project (Java)
   • internetarchive/Zeno - State-of-the-art web crawler (Go)
     • internetarchive/gowarc - Read and write WARC files in Go
   • webrecorder/pywb - Web Archiving Toolkit for replay and recording of web archives (Python)

   Self-hosted solutions:

   • ArchiveBox - A self-hosted app that lets you preserve content from websites in a variety of formats
   • Wallabag - Save and classify articles, read them later