VirtualTam's bookmarks

1. Analyzing One Million robots.txt Files

   2022-06-19
   parser programming/c++ programming/python robots.txt

   Permalink
   • https://docs.python.org/3/library/urllib.robotparser.html
   • https://github.com/scrapy/scrapy/issues/3656
   • https://github.com/scrapy/scrapy/issues/5226
   • https://github.com/seomoz/reppy
   • https://github.com/seomoz/reppy/issues/132