VirtualTam's bookmarks

121. Migrating a 130TB Cluster from Elasticsearch 2 to 5 in 20 Hours with 0 Downtime and a Rollback… 2018-01-31

     tags: data, database, distributed-system/cluster, index, observability/elastic, observability/elastic/elasticsearch, search

     [ permalink ]

     • https://thoughts.t37.net/how-we-reindexed-36-billions-documents-in-5-days-within-the-same-elasticsearch-cluster-cd9c054d1db8
     • https://thoughts.t37.net/how-we-upgraded-a-22tb-mysql-cluster-from-5-6-to-5-7-in-9-months-cc41b391895d
     • https://www.elastic.co/blog/hot-warm-architecture-in-elasticsearch-5-x
     • https://www.graylog.org/post/back-to-basics-using-a-hot-warm-elasticsearch-cluster
     • https://github.com/mraad/es-hot-warm
122. Distributed environments and database migrations 2017-11-29

     tags: data, database, database/migration, distributed, framework/django, linux/container/docker

     [ permalink ]

     Actors:

     • database server / cluster
     • load-balanced application servers relying on a database schema with migrations
     • reverse proxy

     Issue:

     • managing database schema migrations

     Possible sequence:

     • create the database
     • run a service that executes database schema migrations
     • start application servers
     • start the load-balancer / reverse-proxy

     Resources:

     • https://www.reddit.com/r/docker/comments/2t2pnf/what_are_best_strategies_to_migrate_relational/
     • https://softwareengineering.stackexchange.com/questions/357929/db-migration-strategy-for-docker-containers-in-aws-ecs
     • https://stackoverflow.com/questions/41592091/run-a-database-migration-command-when-deploying-a-docker-container-to-aws
     • https://ltd-keeper.lsst.io/gke-migrations.html
     • https://stackoverflow.com/questions/31715239/docker-compose-database-migrations-and-other-pre-post-scripts
     • https://stackoverflow.com/questions/38089999/docker-compose-rails-best-practice-to-migrate
     • https://stackoverflow.com/questions/37058812/how-best-to-run-one-off-migration-tasks-in-a-kubernetes-cluster
123. Django, Docker and MySQL 2017-11-29

     tags: database, database/migration, database/mysql, framework/django, linux/container/docker, programming/python, programming/sql

     [ permalink ]

     Django ORM:

     • https://docs.djangoproject.com/en/1.11/ref/databases/#mysql-notes
     • https://stackoverflow.com/questions/34777755/how-to-config-django-using-pymysql-as-driver
     • https://stackoverflow.com/questions/19189813/setting-django-up-to-use-mysql

     MySQL / MariaDB / Percona connectors:

     • https://stackoverflow.com/questions/4960048/python-3-and-mysql
     • https://github.com/PyMySQL/PyMySQL
     • https://pypi.python.org/pypi/mysql-connector-python

     Docker images:

     • https://store.docker.com/images/mariadb
     • https://store.docker.com/images/mysql
     • https://store.docker.com/images/percona

     Compose and Django ORM database migrations:

     • https://docs.docker.com/compose/django/
     • https://stackoverflow.com/questions/33992867/how-do-you-perform-django-database-migrations-when-using-docker-compose
     • https://www.stavros.io/posts/how-deploy-django-docker/
     • https://docs.docker.com/compose/startup-order/
124. calibre-web: Web app for browsing, reading and downloading eBooks stored in a Calibre database 2017-11-25

     tags: book, framework/flask, library, linux/container/docker, network/opds, programming/python, self-hosting

     [ permalink ]

     • https://github.com/Technosoft2000/docker-calibre-web
     • https://hub.docker.com/r/technosoft2000/calibre-web/
     • https://www.reddit.com/r/selfhosted/comments/8cdzma/calibreserver_troubles_replacement_or_workaround/
125. tarpas/pytest-testmon: taking TDD to a new level with testmon and py.test for Python 2017-11-06

     tags: database, programming/python, programming/python/pytest, programming/testing

     [ permalink ]

     • https://github.com/tarpas/pytest-testmon/wiki/Determining-affected-tests
     • https://medium.com/@boxed/vastly-faster-python-integration-tests-9d8106b3693c
126. Python REST API Framework: Eve, the Simple Way to REST. — Eve 0.8-dev documentation 2017-11-06

     tags: database/mongodb, framework/flask, network/http, programming/api, programming/api/rest, programming/python

     [ permalink ]

     • https://github.com/pyeve/eve
     • http://docs.python-cerberus.org/en/stable/
     • https://speakerdeck.com/nicola/developing-restful-web-apis-with-python-flask-and-mongodb
     • https://www.reddit.com/r/Python/comments/1m3br9/eve_python_rest_api_framework_powered_by_flask/
127. Exoplanet sonification 2017-10-07

     tags: planet, sound, visualization, visualization/d3.js

     [ permalink ]

     • http://phl.upr.edu/projects/habitable-exoplanets-catalog/data/database
     • http://kyma.symbolicsound.com/
     • https://xkcd.com/1071/
128. Netdata, Prometheus, and Grafana Stack 2017-09-29

     tags: database, observability, observability/grafana, observability/time-series/prometheus, sysadmin, visualization

     [ permalink ]

     • https://sistemcim.wordpress.com/2017/04/07/prometheus-grafana-netdata/
     • https://medium.com/vimeo-engineering-blog/graphing-systems-metrics-with-netdata-prometheus-and-grafana-29ba9ec6bc98
129. Change MySQL default character set to UTF-8 in my.cnf? - Stack Overflow 2017-09-05

     tags: database, database/mysql, text/unicode

     [ permalink ]
130. Docker and database persistence 2017-09-04

     tags: data, database, linux, linux/container, linux/container/docker, volume

     [ permalink ]

     • https://myopsblog.wordpress.com/2017/02/06/why-databases-is-not-for-containers/
     • https://stackoverflow.com/questions/18496940/how-to-deal-with-persistent-storage-e-g-databases-in-docker
     • https://stackoverflow.com/questions/30109561/how-persistent-are-docker-data-only-containers
     • https://kubernetes.io/docs/concepts/storage/persistent-volumes/
     • http://rancher.com/introducing-convoy-a-docker-volume-driver-for-backup-and-recovery-of-persistent-data/
131. PostgreSQL: Documentation: 9.6: Schemas 2017-08-20

     tags: database, database/postgresql, framework/flask, programming/python, programming/sql, schema, sqlalchemy

     [ permalink ]

     • https://wiki.postgresql.org/wiki/Database_Schema_Recommendations_for_an_Application
     • https://stackoverflow.com/questions/9298296/sqlalchemy-support-of-postgres-schemas
     • https://github.com/bernardopires/django-tenant-schemas
132. Python | General knowledge & job interview questions 2017-08-20

     tags: exercise, knowledge, programming/job-interview, programming/python

     [ permalink ]

     • https://www.quora.com/What-are-good-Python-interview-questions
     • https://www.reddit.com/r/Python/comments/1knw7z/python_interview_questions/
     • https://resources.workable.com/python-developer-interview-questions
       • https://www.reddit.com/r/Python/comments/dxvtv/favorite_python_interview_question/
       • https://softwareengineering.stackexchange.com/questions/21917/python-interview-questions
       • http://www.ilian.io/python-interview-question-and-answers/
     • https://stackoverflow.com/questions/2573135/python-progression-path-from-apprentice-to-guru/2576240#2576240
     • https://docs.python.org/3/tutorial/
     • https://www.reddit.com/r/Python/comments/6v0amj/the_more_i_learn_about_python_the_more_i_realized/
     • https://github.com/00111000/Imports-in-Python

     Language:

     • GIL
     • memory management, object references
     • data structures
     • duck typing
     • monkey patching
     • generators
     • list & dict comprehensions
     • decorators
     • introspection
     • ...

     Development tools:

     • debugging
     • packaging
     • lint
     • testing: unitary, functional, TDD, BDD

     Libraries:

     • data parsing
     • data scraping
     • database management
     • http requests
     • web frameworks
     • ...
133. Turning the database inside-out with Apache Samza - Confluent 2017-08-10

     tags: apache, commit, data, database, processing

     [ permalink ]
134. Ansible setup mysql root password - Stack Overflow 2017-06-26

     tags: configuration-management/ansible, database/mysql, security/password

     [ permalink ]

     • https://docs.ansible.com/ansible/mysql_user_module.html
     • https://docs.ansible.com/ansible/ini_file_module.html
135. Django vs Flask — git-pull.com 2017-06-23

     tags: comparison, database, framework, framework/django, framework/flask, programming/python, programming/sql, web

     [ permalink ]
136. python - When scattering Flask Models, RuntimeError: 'application not registered on db' was raised - Stack Overflow 2017-05-31

     tags: context, database, framework/flask, model, programming/python

     [ permalink ]
137. Build a CRUD Web App With Python and Flask ― Scotch 2017-05-23

     tags: bootstrap, database, framework/flask, programming/css, programming/html, programming/python, programming/sql, security/authentication, tutorial, web, wsgi

     [ permalink ]

     Part 1 - https://scotch.io/tutorials/build-a-crud-web-app-with-python-and-flask-part-one

     • https://github.com/mbithenzomo/project-dream-team-one
     • http://flask.pocoo.org/docs/0.12/config/
     • http://flask.pocoo.org/docs/0.12/patterns/appfactories/
     • http://flask.pocoo.org/docs/0.12/blueprints/
     • http://flask-sqlalchemy.pocoo.org/2.2/config/

     Part 2 - https://scotch.io/tutorials/build-a-crud-web-app-with-python-and-flask-part-two

     • https://github.com/mbithenzomo/project-dream-team-two

     Part 3 - https://scotch.io/tutorials/build-a-crud-web-app-with-python-and-flask-part-three

     • https://github.com/mbithenzomo/project-dream-team-three
138. JSON data with SQLAlchemy and Flask 2017-05-19

     tags: data, database, framework/flask, programming/json, programming/python, programming/sql

     [ permalink ]

     • https://stackoverflow.com/questions/27682685/how-to-store-and-access-json-array-data-in-postgresql-with-sqlalchemy
     • https://pypi.python.org/pypi/SQLAlchemy-JSONField/0.6.3
     • http://variable-scope.com/posts/creating-a-json-column-type-for-sqlalchemy
     • https://pypi.python.org/pypi/flask_jsontools
139. How do I declare a base model class in Flask-SQLAlchemy? - Stack Overflow 2017-05-19

     tags: abstract, database, framework/flask, mixin, model, programming/python, programming/sql

     [ permalink ]

     • https://stackoverflow.com/questions/25055877/flask-sqlalchemy-abstract-base-model
     • http://docs.sqlalchemy.org/en/latest/orm/extensions/declarative/api.html#abstract
     • http://docs.sqlalchemy.org/en/latest/orm/extensions/declarative/mixins.html#declarative-mixins
140. appsec - How to securely hash passwords? - Information Security Stack Exchange 2017-05-15

     tags: cryptography/hash, cryptography/salt, database, security, security/password

     [ permalink ]