VirtualTam's bookmarks

1. robots.txt - AI / LLM User-Agents: Blocking Guide

   2025-04-29
   network/http robots.txt web/crawling web/scraping

   Permalink
   • A summary of my bot defence systems
     • Butlerian Jihad - Blog posts on the topic of fighting off spam bots, search engine spiders and other non-humans wasting the precious resources we have on Earth
   • EmacsWiki's robots.txt
2. Archiving URLs | Gwern

   2025-04-09
   network/http personal-knowledge/bookmark personal-knowledge/digital-garden programming/go programming/html programming/java programming/python web web/scraping

   Permalink
   • Archiving Web Sites (LWN)
   • Robots.txt meant for search engines don’t work well for web archives

   Web page archive formats:

   • ARC (1996)
   • WARC (BnF, 2009)
     • WARC Specifications
     • The WARC Ecosystem

   Tools for crawling, scraping and archiving Web pages:

   • internetarchive/heritrix3 - Extensible, web-scale, archival-quality web crawler project (Java)
   • internetarchive/Zeno - State-of-the-art web crawler (Go)
     • internetarchive/gowarc - Read and write WARC files in Go
   • webrecorder/pywb - Web Archiving Toolkit for replay and recording of web archives (Python)

   Self-hosted solutions:

   • ArchiveBox - A self-hosted app that lets you preserve content from websites in a variety of formats
   • Wallabag - Save and classify articles, read them later
3. BeamNG.tech - A real time soft-body physics with a comprehensive, expandable platform

   2024-10-13
   artificial-intelligence car image programming/c++ programming/lua programming/python research

   Permalink
   • BeamNG.tech Technical Paper (2021-06-21)
   • Publications
   • BeamNG.tech Blog
   • Project References
   • BeamNG/BeamNGpy - Python API for BeamNG.tech
   • BeamNG/beamng-ros2-integration - Robot Operating System integration
4. Analyzing One Million robots.txt Files

   2022-06-19
   parser programming/c++ programming/python robots.txt

   Permalink
   • https://docs.python.org/3/library/urllib.robotparser.html
   • https://github.com/scrapy/scrapy/issues/3656
   • https://github.com/scrapy/scrapy/issues/5226
   • https://github.com/seomoz/reppy
   • https://github.com/seomoz/reppy/issues/132
5. pabot: Parallel executor for Robot Framework test cases.

   2019-05-20
   parallel programming/python programming/testing robot-framework

   Permalink
6. [JENKINS-32118] Document Jenkins security settings so Robot Framework log/report can be opened - Jenkins JIRA

   2019-01-23
   continuous-integration/jenkins plugin programming/testing robot-framework

   Permalink
   • https://issues.jenkins-ci.org/browse/JENKINS-38338
7. Bot Summit

   2018-09-11
   bot chat text

   Permalink
   • http://tinysubversions.com/botsummit/2016/
   • http://tinysubversions.com/botsummit/2014/
   • http://opentranscripts.org/sources/bot-summit/bot-summit-2014/
   • https://docs.google.com/document/d/1bka4o1RE9RPUeoUzgpTIKRWsgWHzZEKEADialnv7haQ/view
   • https://github.com/dariusk/corpora
   • https://brianshumate.com/articles/twitter-bots/
   • https://brianshumate.com/projects/index.html#-twitter-robots
8. FOSSCON 2018: Where Open Source and LEGO Collide | Hackaday

   2018-09-10
   computer-aided-design hardware lego robot sandbox

   Permalink
9. RED - Robot Editor

   2018-09-03
   behavior-driven development eclipse editor programming/test-driven-development programming/testing

   Permalink
   • https://nokia.github.io/RED/
10. Robot Framework

    2018-04-05
    automation programming/python programming/testing qa selenium

    Permalink
    • https://github.com/robotframework/robotframework
11. OpenCat - Arduino Project Hub

    2018-02-23
    artificial-intelligence automation cat embedded/arduino robot

    Permalink
12. Noumena | Robotic Habitats

    2018-01-31
    artificial-intelligence autonomous drone life robot

    Permalink
    • https://www.engadget.com/2018/01/29/robotic-habitats-noumena-the-big-picture/
13. Monitoring the Dark Army with Kibana on Mr. Robot | Elastic

    2017-11-22
    dashboard data engine observability/elastic/elasticsearch observability/elastic/kibana search series visualization

    Permalink
    • https://medium.com/@ryankazanciyan/mr-robot-disassembled-3-4-runtime-error-r00-9319c70c30f2
14. http://myrobotlab.org

    2017-08-20
    artificial-intelligence programming/java robot tts web

    Permalink
    • https://github.com/MyRobotLab
15. Zero-K

    2017-08-08
    game game/real-time-strategy open-source robot

    Permalink
16. HUBOT - A customizable life embetterment robot

    2016-12-20
    bot chat source-code-management/github

    Permalink
    • https://github.com/spajus/hubot-control
17. No. Fun. Allowed. Ever.

    2016-08-24
    bot fun meme no robot

    Permalink

    via https://www.reddit.com/r/NoFunAllowed via https://www.reddit.com/r/ExpectedFactorialBot/comments/4yx9qm/frequently_asked_questions/

18. Les androïdes rêvent-ils de moutons électriques ? | Tracks | Culture | fr - ARTE

    2014-06-21
    art automation hack mobile/android painting robot tweak

    Permalink
19. Flying Robot Rockstars - YouTube

    2014-04-24
    automation flying robot

    Permalink
20. Find out your SEXY AWESOME robot name!

    2014-01-15
    bank comics itsatrap privacy robot

    Permalink