VirtualTam's bookmarks

1. Cloudflare - Over 700 million events/second: How we make sense of too much data 2025-02-04

   tags: cloud-computing/cloudflare, observability/logging, programming/go

   [ permalink ]
2. lnav - Logfile navigator 2025-01-30

   tags: command-line, database/sqlite, observability/logging, programming/c++, programming/json

   [ permalink ]

   • The Logfile Navigator
   • Command Line Interface
   • Log Formats
   • SQLite Interface
3. wirelogd - Logging daemon for Wireguard 2025-01-04

   tags: network/vpn/wireguard, observability/logging, programming/go

   [ permalink ]

   Wirelogd relies on the latest handshake to determine if a peer is active or inactive. While there is trafic the handshake should be renewed every 2 minutes. If there is no trafic handshake is not renewed. Based on this behavior we assume that if there is no new handshake after a while, the client is probably inactive.

4. Crowdsec - Curated Threat Intelligence Powered by the Crowd 2024-05-29

   tags: bsd, linux/iptables, network/firewall, observability/logging, programming/go

   [ permalink ]

   • crowdsecurity/crowdsec
   • Documentation
     • Docker / Podman Deployment
     • Data Sources
     • Parsers
     • Configuration
     • Observability / Prometheus
     • OpenAPI Specification
   • Blog
     • Building a Network for Small Businesses Part 1: Hardware
     • Building a Network for Small Businesses Part 2: Running Configurations
     • Building a Network for Small Businesses Part 3: Attack Scenarios
   • Discussion on HN (2020)
5. elastic-integration-corpus-generator-tool - Generate an event corpus given a specific integration 2024-05-15

   tags: observability/elastic/elasticsearch, observability/logging, programming/go, programming/json

   [ permalink ]

   • quickwit-oss/benchmarks/tree/main/scripts/create-generated-log-dataset
6. All you need is Wide Events, not "Metrics, Logs and Traces" 2024-04-19

   tags: database, distributed-system, observability, observability/logging, observability/metrics, observability/tracing

   [ permalink ]

   • Scuba: Diving into Data at Facebook - International Conference on Very Large Data Bases (VLDB) (PDF, Talk)
   • HN discussion
   • Axiom - Stop sampling, observe every event
     • Adaptive _time compression — A database hacker story
   • Eventrelay - An event streaming/storage system for the rest of us
   • Honeycomb - Identify Outliers
   • Rill - Fast operational dashboards that your team will actually use
   • Strangeloop 17 - Why We Built Our Own Distributed Column Store by Sam Stokes
7. spacer - CLI tool to insert spacers when command output stops 2024-02-25

   tags: command-line, linux, observability/logging, programming/rust

   [ permalink ]
8. Distributed Tracing in Rust 2023-12-17

   tags: distributed-system, observability/logging, observability/opentelemetry, observability/tracing, programming/rust

   [ permalink ]

   1. Logging Basics
   2. Tracing Basics
   3. Inter-Service Tracing
   4. Correlating Logs and Traces
9. Elasticsearch - Grok processor 2023-11-19

   tags: observability/elastic/elasticsearch, observability/elastic/logstash, observability/logging, programming/go, programming/java, programming/regular-expressions, programming/rust

   [ permalink ]

   • Elastic - Grokking Grok
   • Kibana - Debug grok expressions
   • Grok Constructor
   • Elasticsearch Grok Patterns
   • Logstash Grok Patterns

   Alternative implementations:

   • vjeantet/grok - Simple library to use/parse grok patterns with Go
     • GuanceCloud/grok - Fork of the above
     • logrusorgru/grokky - Pure golang grok-like library
     • Telegraf's Grok Parser Plugin
   • Vector Remap Language - Datadog Grok Parser
10. Vector - A lightweight, ultra-fast tool for building observability pipelines 2022-02-01

    tags: linux, observability/elastic/elasticsearch, observability/logging/grafana-loki, observability/metrics, observability/time-series/prometheus, syslog

    [ permalink ]
11. How to successfully correlate metrics, logs, and traces in Grafana 2020-12-21

    tags: observability, observability/grafana, observability/logging, observability/logging/grafana-loki, observability/time-series/prometheus, observability/tracing, observability/tracing/grafana-tempo, performance

    [ permalink ]
12. Grafana Tempo | a high volume, minimal dependency trace storage 2020-12-21

    tags: observability, observability/grafana, observability/logging, observability/logging/grafana-loki, observability/tracing, observability/tracing/grafana-tempo, performance, programming/go

    [ permalink ]

    • https://grafana.com/docs/tempo/latest/getting-started/
    • https://grafana.com/docs/tempo/latest/configuration/
    • https://github.com/grafana/tempo
    • https://grafana.com/blog/2020/10/27/announcing-grafana-tempo-a-massively-scalable-distributed-tracing-system/
    • https://grafana.com/blog/2020/11/09/trace-discovery-in-grafana-tempo-using-prometheus-exemplars-loki-2.0-queries-and-more/
    • https://grafana.com/go/observabilitycon/observability-with-logs-grafana/
    • https://grafana.com/go/observabilitycon/tracing-made-simple-with-grafana/
    • https://gouthamve.dev/tempo-a-game-of-trade-offs/
13. Open Distro for Elasticsearch | Open Distro 2020-12-20

    tags: cloud-computing, cloud-computing/aws, machine-learning, observability/elastic/elasticsearch, observability/logging, observability/tracing, programming/java, security/authentication, security/authorization

    [ permalink ]

    • https://github.com/opendistro-for-elasticsearch
    • https://aws.amazon.com/blogs/aws/new-open-distro-for-elasticsearch/
    • https://opendistro.github.io/for-elasticsearch/blog/releases/2020/12/announcing-trace-analytics/
    • https://logz.io/blog/open-distro-for-elasticsearch/
14. Sending logs from syslog-ng to Grafana Loki - Blog - syslog-ng Community - syslog-ng Community 2020-12-11

    tags: observability, observability/grafana, observability/logging, observability/logging/grafana-loki, syslog

    [ permalink ]
15. Using Grafana Loki 2.0 as an Alert Source - ScyllaDB 2020-12-11

    tags: observability, observability/grafana, observability/logging/grafana-loki, observability/time-series/prometheus

    [ permalink ]
16. Introduction to HAProxy Logging - HAProxy Technologies 2020-12-11

    tags: network/http, network/http-server/haproxy, network/reverse-proxy, network/tcp, observability/logging

    [ permalink ]

    • https://www.haproxy.com/documentation/hapee/latest/administration/docker-logging/
17. IoT logging with MQTT, Fluentd / Fluent Bit and Loki 2020-08-15

    tags: embedded, fluentd, internet-of-things, network/mqtt, observability/grafana, observability/logging, observability/logging/grafana-loki

    [ permalink ]

    Open questions, potential solutions or workarounds:

    • Loki does not support out-of-order message ingestion
      • https://github.com/grafana/loki/issues/1544
      • https://github.com/project-hatohol/fluent-plugin-sort
    • There is not Java client for Loki
      • https://github.com/grafana/loki/issues/1158
      • https://grafana.com/docs/loki/latest/api/
      • https://github.com/grafana/loki/tree/master/cmd/fluent-bit
    • The Wiced MQTT queue might be limited depending on the volume of logs
      • https://iotexpert.com/2017/03/08/embedded-world-2017-wiced-wifi-mqtt-and-the-amazon-iot-cloud/
18. Percona Toolkit Documentation 2020-01-21

    tags: database, database/mysql, observability/logging, performance, programming/sql, query

    [ permalink ]

    • https://www.percona.com/blog/2018/09/13/analyzing-amazon-aurora-slow-logs-pt-query-digest/
    • https://www.jeffgeerling.com/blog/2018/analyzing-mysql-slow-query-log-pt-query-digest
19. Git double-dot and triple-dot notations 2020-01-14

    tags: observability/logging, source-code-management, source-code-management/git, text/diff, tree

    [ permalink ]

    • https://stackoverflow.com/questions/7251477/what-are-the-differences-between-double-dot-and-triple-dot-in-git-dif
    • https://stackoverflow.com/questions/462974/what-are-the-differences-between-double-dot-and-triple-dot-in-git-com
20. trailscraper: A command-line tool to get valuable information out of AWS CloudTrail 2019-08-08

    tags: cloud-computing/aws, observability/logging, programming/python, security, user

    [ permalink ]