Simon Willison's Weblog: deployment

MRSK

2023-04-29T23:54:40+00:00

A new open source web application deployment tool from 37signals, developed to help migrate their Hey webmail app out of the cloud and onto their own managed hardware. The key feature is one that I care about deeply: it enables zero-downtime deploys by running all traffic through a Traefik reverse proxy in a way that allows requests to be paused while a new deployment is going out—so end users get a few seconds delay on their HTTP requests before being served by the replaced application.

Tags: 37-signals, deployment, ops, zero-downtime, traefik

Quoting Avery Pennarun

2021-08-30T05:46:10+00:00

Unshipped work is inventory and it costs you money as it spoils

— Avery Pennarun

Tags: deployment

Running Datasette on DigitalOcean App Platform

2020-10-07T02:52:54+00:00

Running Datasette on DigitalOcean App Platform

I spent some time with DigitalOcean’s new App Platform today, which is a Heroku-style PaaS that starts at $5/month. It looks like it could be a really good fit for Datasette. Disk is ephemeral, but if you’re publishing read-only data that doesn’t matter since you can build the SQLite database as part of the deployment and bundle it up in the Docker/Kubernetes container.

Via @simonw

Tags: deployment, heroku, datasette, digitalocean

Zero Downtime Release: Disruption-free Load Balancing of a Multi-Billion User Website

2020-08-05T03:27:27+00:00

Zero Downtime Release: Disruption-free Load Balancing of a Multi-Billion User Website

I remain fascinated by techniques for zero downtime deployment—once you have it working it makes shipping changes to your software so much less stressful, which means you can iterate faster and generally be much more confident in shipping code.

Facebook have invested vast amounts of effort into getting this right, and their new paper for the ACM SIGCOMM conference goes into detail about how it all works.

Via Cindy Sridharan

Tags: deployment, zero-downtime

Weeknotes: Datasette Cloud and zero downtime deployments

2020-01-21T20:56:46+00:00

Yesterday's piece on Tracking FARA by deploying a data API using GitHub Actions and Cloud Run was originally intended to be my weeknotes, but ended up getting a bit too involved.

Aside from playing with GitHub Actions and Cloud Run, my focus over the past week has been working on Datasette Cloud. Datasette Cloud is the current name I'm using for my hosted Datasette product - the idea being that I'll find it a lot easier to get feedback on Datasette from journalists if they can use it without having to install anything!

My MVP for Datasette Cloud is that I can use it to instantly provision a new, private Datasette instance for a journalist (or team of journalists) that they can then sign into, start playing with and start uploading their data to (initially as CSV files).

I have to solve quite a few problems to get there:

Secure, isolated instances of Datasette. A team or user should only be able to see their own files. I plan to solve this using Docker containers that are mounted such that they can only see their own dedicated volumes.
The ability to provision new instances as easily as possible - and give each one its own HTTPS subdomain.
Authentication: users need to be able to register and sign in to accounts. I could use datasette-auth-github for this but I'd like to be able to support regular email/password accounts too.
Users need to be able to upload CSV files and have them converted into a SQLite database compatible with Datasette.

Zero downtime deployments

I have a stretch goal which I'm taking pretty seriously: I want to have a mechanism in place for zero-downtime deployments of new versions of the software.

Arguable this is an unneccessary complication for an MVP. I may not fully implement it, but I do want to at least know that the path I've taken is compatible with zero downtime deployments.

Why do zero downtime deployments matter so much to me? Because they are desirable for rapid iteration, and crucial for setting up continuious deployment. Even a couple of seconds of downtime during a deployment leaves a psychological desire not to deploy too often. I've seen the productivity boost that deploying fearlessly multiple times a day brings, and I want it.

So I've been doing a bunch of research into zero downtime deployment options (thanks to some great help on Twitter) and I think I have something that's going to work for me.

The first ingredient is Traefik - a new-to-me edge router (similar to nginx) which has a delightful focus on runtime configuration based on automatic discovery.

It works with a bunch of different technology stacks, but I'm going to be using it with regular Docker. Traefik watches for new Docker containers, reads their labels and uses that to reroute traffic to them.

So I can launch a new Docker container, apply the Docker label "traefik.frontend.rule": "Host:subdomain.mydomain.com" and Traefik will start proxying traffic to that subdomain directly to that container.

Traefik also has extremely robust built-in support for Lets Encrypt to issue certificates. I managed to issue a wildcard TLS certificate for my entire domain, so new subdomains are encrypted straight away. This did require me to give Traefik API access to modify DNS entries - I'm running DNS for this project on Digital Ocean and thankfully Traefik knows how to do this by talking to their API.

That solves provisioning: when I create a new account I can call the Docker API (from Python) to start up a new, labelled container on a subdomain protected by a TLS certificate.

I still needed a way to run a zero-downtime deployment of a new container (for example when I release a new version of Datasette and want to upgrade everyone). After quite a bit of research (during which I discovered you can't modify the labels on a Docker container without restarting it) I settled on the approach described in this article.

Essentially you configure Traefik to retry failed requests, start a new, updated container with the same routing information as the existing one (causing Traefik to load balance HTTP requests across both), then shut down the old container and trust Traefik to retry in-flight requests against the one that's still running.

Rudimentary testing with ab suggested that this is working as desired.

One remaining problem: if Traefik is running in a Docker container and proxying all of my traffic, how can I upgrade Traefik itself without any downtime?

Consensus on Twitter seems to be that Docker on its own doesn't have a great mechanism for this (I was hoping I could re-route port 80 traffic to the host to a different container in an atomic way). But... iptables has mechanisms that can re-route traffic from one port to another - so I should be able to run a new Traefik container on a different port and re-route to it at the operating system level.

That's quite enough yak shaving around zero time deployments for now!

datasette-upload-csvs

A big problem I'm seeing with the current Datasette ecosystem is that while Datasette offers a web-based user interface for querying and accessing data, the tools I've written for actually creating those databases are decidedly command-line only.

Telling journalists they have to learn to install and run software on the command-line is way too high a barrier to entry.

I've always intended to have Datasette plugins that can handle uploading and converting data. It's time to actually build one!

datasette-upload-csvs is what I've got so far. It has a big warning not to use it in the README - it's very alpha sofware at the moment - but it does prove that the concept can work.

It uses the asgi_wrapper plugin hook to intercept requests to the path /-/upload-csv and forward them on to another ASGI app, written using Starlette, which provides a basic upload form and then handles the upload.

Uploaded CSVs are converted to SQLite using sqlite-utils and written to the first mutable database attached to Datasette.

It needs a bunch more work (and tests) before I'm comfortable telling people to use it, but it does at least exist as a proof of concept for me to iterate on.

datasette-auth-django-cookies

No code for this yet, but I'm beginning to flesh it out as a concept.

I don't particularly want to implement user registration and authentication and cookies and password hashing. I know how to do it, which means I know it's not something you shuld re-roll for every project.

Django has a really well designed, robust authentication system. Can't I just use that?

Since all of my applications will be running on subdomains of a single domain, my current plan is to have a regular Django application which handles registration and logins. Each subdomain will then run a custom piece of Datasette ASGI middleware which knows how to read and validate the Django authentication cookie.

This should give me single sign-on with a single, audited codebase for registration and login with (hopefully) the least amount of work needed to integrate it with Datasette.

Code for this will hopefully follow over the next week.

Niche Museums - now publishing weekly

I hit a milestone with my Niche Museums project: the site now lists details of 100 museums!

For the 100th entry I decided to celebrate with by far the most rewarding (and exclusive) niche museum experience I've ever had: Ray Bandar's Bone Palace.

You should read the entry. The short version is that Ray Bandar collected 7,000 animals skulls over a sixty year period, and Natalie managed to score us a tour of his incredible basement mere weeks before the collection was donated to the California Academy of Sciences.

Posting one museum a day was taking increasingly more of my time, as I had to delve into the depths of my museums-I-have-visited backlog and do increasing amounts of research. Now that I've hit 100 I'm going to switch to publishing one a week, which should also help me visit new ones quickly enough to keep the backlog full!

So I only posted four this week:

The ruins of Llano del Rio in Los Angeles County
Cleveland Hungarian Museum in Cleveland
New Orleans Historic Voodoo Museum in New Orleans
Ray Bandar's Bone Palace in San Francisco

I also built a simple JavaScript image gallery to better display the 54 photos I published from our trip to Ray Bandar's basement.

Tags: csv, deployment, museums, projects, zero-downtime, docker, datasette, weeknotes, traefik, datasette-cloud, digitalocean

How to do Zero Downtime Deployments of Docker Containers

2020-01-16T23:12:35+00:00

How to do Zero Downtime Deployments of Docker Containers

I’m determined to get reliable zero-downtime deploys working for a new project, because I know from experience that even a few seconds of downtime during a deploy changes the project mentality from “deploy any time you want” to “don’t deploy too often”. I’m using Docker containers behind Traefik, which means new containers should have traffic automatically balanced to them by Traefik based on their labels. After much fiddling around the pattern described by this article worked best for me: it lets me start a new container, then stop the old one and have Traefik’s “retry” mechanism send any requests to the stopped container over to the new one instead.

Tags: deployment, zero-downtime, docker, traefik

How we use "ship small" to rapidly build new features at GitHub

2020-01-02T04:30:53+00:00

How we use "ship small" to rapidly build new features at GitHub

Useful insight into how GitHub develop new features. They make aggressive use of feature flags, shipping a rough skeleton of a new feature to production as early as possible and actively soliciting feedback from other employees as they iterate on the feature. They static JSON mocks of APIs to unblock their frontend engineers and iterate on the necessary data structures while the real backend is bring implemented.

Via Jeff Triplett

Tags: deployment, github, feature-flags

Azure Readiness Checklist

2019-10-26T20:32:46+00:00

Azure Readiness Checklist

I love a good comprehensive checklist. This one is focused on large projects running on Azure but it’s still fun to browse through if you are hosting elsewhere, mainly as a reminder of quite how much still goes into deploying large web services into production.

Via Charity Majors

Tags: azure, checklists, deployment

Quoting Charity Majors

2018-08-27T21:00:46+00:00

In too many organizations, deploy code is a technical backwater, an accumulation of crufty scripts and glue code, forked gems and interns’ earnest attempts to hack up Capistrano. It usually gives off a strong whiff of “sloppily evolved from many 2 am patches with no code review”.

This is insane. Deploy software is the most important software you have. Treat it that way: recruit an owner, allocate real time for development and testing, bake in metrics and track them over time.

— Charity Majors

Tags: code-review, deployment, charity-majors

Quoting Dan Manges

2011-06-30T21:27:00+00:00

We can deploy new versions of our software, make database schema changes, or even rotate our primary database server, all without failing to respond to a single request. We can accomplish this because we gave ourselves the ability suspend our traffic, which gives us a window of a few seconds to make some changes before letting the requests through. To make this happen, we built a custom HTTP server and application dispatching infrastructure around Python’s Tornado and Redis.

— Dan Manges, Braintree

Tags: deployment, http, redis, tornado, recovered

What to do when PyPI goes down

2010-07-21T10:19:00+00:00

What to do when PyPI goes down

My deployment scripts tend to rely on PyPI these days (they install dependencies in to a virtualenv) which makes me distinctly uncomfortable. Jacob explains how to use the PyPI mirrors that are starting to come online, but that won’t help if the PyPI listing links to an externally hosted file which starts to 404, as happened with the python-openid package quite recently (now fixed). The comments on the post discuss workarounds, including hosting your own PyPI mirror or bundling tar.gz files of your dependencies with your project.

Tags: deployment, jacob-kaplan-moss, packaging, pip, pypi, python, setuptools, recovered

Automated deployments with Fabric - tips and tricks

2010-03-16T11:19:58+00:00

Automated deployments with Fabric - tips and tricks

“If it’s not in a Fabric fabfile, it’s not deployable”—I’m slowly applying this philosophy to my personal projects.

Tags: deployment, fabric, python, sysadmin

Automate EC2 Instance Setup with user-data Scripts

2010-03-11T12:31:50+00:00

Automate EC2 Instance Setup with user-data Scripts

I knew about EC2’s user-data feature—what I didn’t know is that the Alestic and Canonical images are configured so that if the user-data starts with #! the instance will automatically execute it as a shell script as soon as it boots up (after networking has been configured).

Via Paul Smith

Tags: deployment, ec2, sysadmin, userdata

jacobian's django-deployment-workshop

2010-02-19T14:28:35+00:00

jacobian's django-deployment-workshop

Notes and resources from Jacob’s 3 hour Django deployment workshop at PyCon, including example configuration files for Apache2 + mod_wsgi, nginx, PostgreSQL and pgpool.

Tags: apache, deployment, django, jacob-kaplan-moss, modwsgi, nginx, pgpool, postgresql, pycon, python, sysadmin

Why toppcloud will not be agnostic

2010-02-12T09:21:32+00:00

Why toppcloud will not be agnostic

Ian Bicking’s toppcloud aims to offer deployment with the ease of use of AppEngine against a standard, open source Ubuntu + Python 2.6 + mod_wsgi + Varnish stack. Here he explains why he’s not going to vary the required components: keeping everything completely standardised means everyone gets the same bugs (and the same fixes).

Tags: deployment, django, google-app-engine, ian-bicking, modwsgi, python, toppcloud, ubuntu, varnish, wsgi

Hot Code Loading in Node.js

2010-01-31T13:57:53+00:00

Hot Code Loading in Node.js

Blaine Cook’s patch for Node.js that enables Erlang-style hot code loading, so you can switch out your application logic without restarting the server or affecting existing requests. This could make deploying new versions of Node applications trivial. I’d love to see a Node hosting service that allows you to simply upload a script file and have it execute on the Web.

Tags: blaine-cook, deployment, erlang, javascript, nodejs

Fabric 0.9.0

2009-11-09T14:02:53+00:00

Fabric 0.9.0

A Python-based SSH automation and deployment tool. Released today, 0.9.0 is finally the official “stable” release—which is good, as it breaks API compatibility with previous versions and caused me all sorts of confusion when I tried to learn Fabric recently.

Tags: deployment, fabric, python, ssh

Automating web site deployment at Barcamp Brighton

2009-09-06T14:16:30+00:00

Automating web site deployment at Barcamp Brighton

I’m determined to start using Fabric and proper deployment scripts for my personal projects.

Tags: barcamp, barcampbrighton, brighton, deployment, fabric, gareth-rushgrove

Fabric, Django, Git, Apache, mod_wsgi, virtualenv and pip deployment

2009-07-28T11:56:09+00:00

Fabric, Django, Git, Apache, mod_wsgi, virtualenv and pip deployment

I’m slowly working my way through this stack at the moment—next stop, fabric.

Tags: apache, deployment, django, fabric, gareth-rushgrove, git, modwsgi, pip, python, virtualenv, wsgi

Tools of the Modern Python Hacker: Virtualenv, Fabric and Pip

2009-07-09T11:40:58+00:00

Tools of the Modern Python Hacker: Virtualenv, Fabric and Pip

Ashamed to say I’m not using any of these yet—for Django projects, my manage.py inserts an “ext” directory at the beginning of the Python path which contains my dependencies for that project.

Tags: deployment, django, fabric, pip, python, pythonpath, tools, virtualenv

Phusion Passenger for nginx

2009-04-20T04:53:55+00:00

Phusion Passenger for nginx

Passenger (aka mod_rails / mod_rack) enables easy deployment of Rails and Ruby apps under Apache... and the latest version adds support for nginx as well. It works as an HTTP proxy and process manager, spawning worker processes and forwarding HTTP requests to them via a request queue. It can also handle Python WSGI applications—anyone tried it out for that yet?

Tags: apache, deployment, nginx, passenger, python, rails, ruby

How to use Django with Apache and mod_wsgi

2009-04-01T00:24:04+00:00

How to use Django with Apache and mod_wsgi

My favourite deployment option is now included in the official Django docs, thanks to Alex Gaynor. I tend to run a stripped down Apache with mod_wsgi behind an nginx proxy, and have nginx serve static files directly. This avoids the need for a completely separate media server (although a separate media domain is still a good idea for better client-side performance).

Tags: alex-gaynor, deployment, django, modwsgi, nginx, proxies, python, wsgi

Webistrano

2007-08-31T23:05:54+00:00

Webistrano

Web based interface for managing Capistrano deployments. Cal recommends having a “deploy to live site” button in his book; this looks like an easy way to build that.

Tags: cal-henderson, capistrano, deployment, webistrano

Quick Django Benching

2007-02-17T16:56:28+00:00

Quick Django Benching

Django under Apache/mod_python outperforms nginx/FastCGI and LightTPD/FastCGI once you ramp up the concurrency levels. My setup for this site (Apache/mod_python behind an nginx proxy, with nginx handling static files) should give the best of both worlds.

Tags: apache, deployment, django, fastcgi, lighttpd, modpython, nginx

The Django Book: Deploying Django

2007-01-26T12:38:46+00:00

The Django Book: Deploying Django

Solid advice based on years of experience at the Journal-World and the Washington Post.

Tags: deployment, django, django-book