VirtualTam's bookmarks

1. Analyzing One Million robots.txt Files 2022-06-19

   tags: parser, programming/c++, programming/python, robots.txt

   [ permalink ]

   • https://docs.python.org/3/library/urllib.robotparser.html
   • https://github.com/scrapy/scrapy/issues/3656
   • https://github.com/scrapy/scrapy/issues/5226
   • https://github.com/seomoz/reppy
   • https://github.com/seomoz/reppy/issues/132