DSQL Vignette: Reads and Compute

2024-12-06T17:12:10+00:00

Marc Brooker is one of the engineers behind AWS's new Aurora DSQL horizontally scalable database. Here he shares all sorts of interesting details about how it works under the hood.

The system is built around the principle of separating storage from compute: storage uses S3, while compute runs in Firecracker:

Each transaction inside DSQL runs in a customized Postgres engine inside a Firecracker MicroVM, dedicated to your database. When you connect to DSQL, we make sure there are enough of these MicroVMs to serve your load, and scale up dynamically if needed. We add MicroVMs in the AZs and regions your connections are coming from, keeping your SQL query processor engine as close to your client as possible to optimize for latency.

We opted to use PostgreSQL here because of its pedigree, modularity, extensibility, and performance. We’re not using any of the storage or transaction processing parts of PostgreSQL, but are using the SQL engine, an adapted version of the planner and optimizer, and the client protocol implementation.

The system then provides strong repeatable-read transaction isolation using MVCC and EC2's high precision clocks, enabling reads "as of time X" including against nearby read replicas.

The storage layer supports index scans, which means the compute layer can push down some operations allowing it to load a subset of the rows it needs, reducing round-trips that are affected by speed-of-light latency.

The overall approach here is disaggregation: we’ve taken each of the critical components of an OLTP database and made it a dedicated service. Each of those services is independently horizontally scalable, most of them are shared-nothing, and each can make the design choices that is most optimal in its domain.

Tags: architecture, aws, databases, ec2, postgresql, s3, scaling, firecracker

Why We Replaced Firecracker with QEMU

2023-07-10T15:09:03+00:00

Why We Replaced Firecracker with QEMU

Hocus are building a self-hosted alternative to cloud development environment tools like GitPod and Codespaces. They moved away from Firecracker because it’s optimized for short-running (AWS Lambda style) functions—which means it never releases allocated RAM or storage volume space back to the host machine unless the container is entirely restarted. It also lacks GPU support.

Via Hacker News

Tags: qemu, virtualization, firecracker

Weeknotes: Building Datasette Cloud on Fly Machines, Furo for documentation

2022-05-26T04:35:11+00:00

Hosting provider Fly released Fly Machines this week. I got an early preview and I've been working with it for a few days - it's a fascinating new piece of technology. I'm using it to get my hosting service for Datasette ready for wider release.

Datasette Cloud

Datasette Cloud is the name I've given my forthcoming hosted SaaS version of Datasette. I'm building it for two reasons:

This is an obvious step towards building a sustainable business model for my open source project. It's a reasonably well-trodden path at this point: plenty of projects have demonstrated that offering paid hosting for an open source project can build a valuable business. GitLab are an especially good example of this model.
There are plenty of people who could benefit from Datasette, but the friction involved in hosting it prevents them from taking advantage of the software. I've tried to make it as easy to host as possible, but without a SaaS hosted version I'm failing to deliver value to the people that I most want the software to help.

My previous alpha was built directly on Docker, running everything on a single large VPS. Obviously it needed to scale beyond one machine, and I started experimenting with Kubernetes to make this happen.

I also want to allow users to run their own plugins, without risk of malicious code causing problems for other accounts. Docker and Kubernetes containers don't offer the isolation that I need to feel comfortable doing this, so I started researching Firecracker - constructed by AWS to power Lambda and Fargate, so very much designed with potentially malicious code in mind.

Spinning up Firecracker on a Kubernetes cluster is no small lift!

And then I heard about Fly Machines. And it looks like it's exactly what I need to get this project to the next milestone.

Fly Machines

Fly's core offering allows you to run Docker containers in regions around the world, compiled (automatically by Fly) to Firecracker containers with geo-load-balancing so users automatically get routed to an instance running near them.

Their new Fly Machines product gives you a new way to run containers there: you get full control over when containers are created, updated, started, stopped and destroyed. It's the exact level of control I need to build Datasette Cloud.

It also implements scale-to-zero: you can stop a container, and Fly will automatically start it back up again for you (generally in less than a second) when fresh traffic comes in.

(I had built my own version of this for my Datasette Cloud alpha, but the spin up time took more like 10s and involved showing the user a custom progress bar to help them see what was going on.)

Being able to programatically start and stop Firecracker containers was exactly what I'd been trying to piece together using Kubernetes - and the ability to control which global region they go in (with the potential for Litestream replication between regions in the future) is a feature I hadn't expected to be able to offer for years.

So I spent most of this week on a proof of concept. I've successfully demonstrated that the Fly Machines product has almost exactly the features that I need to ship Datasette Cloud on Fly Machines - and I've confirmed that the gaps I need to fill are on Fly's near-term roadmap.

I don't have anything to demonstrate publicly just yet, but I do have several new TILs.

If this sounds interesting to you or your organization and you'd like to try it out, drop me an email at swillison @ Google's email service.

The Furo theme for Sphinx

My shot-scraper automated screenshot tool's README had got a little too long, so I decided to upgrade it to a full documentation website.

I chose to use MyST and Sphinx for this, hosted on Read The Docs.

MyST adds Markdown syntax to Sphinx, which is easier to remember (and for people to contribute to) than reStructuredText.

After putting the site live, Adam Johnson suggested I take a look at the Furo theme. I'd previously found Sphinx themes hard to navigate because they had so much differing functionality, but a personal recommendation turned out to be exactly what I needed.

Furo is really nice - it fixed a slight rendering complaint I had about nested lists in the theme I was using, and since it doesn't use web fonts it dropped the bytes transferred for a page of documentation by more than half!

I switched shot-scraper over to Furo, and liked it so much that I switched over Datasette and sqlite-utils too.

Here's what the shot-scraper documentation looks like now:

Screenshot taken using shot-scraper itself, like this:

shot-scraper \
  https://shot-scraper.datasette.io/en/latest/ \
  --retina --height 1200

Full details of those theme migrations (including more comparative screenshots) can be found in these issues:

Releases this week

datasette-unsafe-expose-env: 0.1 - 2022-05-25
Datasette plugin to expose some environment variables at /-/env for debugging
shot-scraper: 0.14.1 - (16 releases total) - 2022-05-22
A comand-line utility for taking automated screenshots of websites
google-calendar-to-sqlite: 0.1a0 - 2022-05-21
Create a SQLite database containing your data from Google Calendar
datasette-upload-dbs: 0.1.1 - (2 releases total) - 2022-05-17
Upload SQLite database files to Datasette
datasette-insert: 0.7 - (7 releases total) - 2022-05-16
Datasette plugin for inserting and updating data

TIL this week

Tags: documentation, projects, datasette, weeknotes, datasette-cloud, fly, firecracker

Behind the scenes, AWS Lambda

2021-07-10T19:40:06+00:00

Behind the scenes, AWS Lambda

Bruno Schaatsbergen pulled together details about how AWS Lambda works under the hood from a detailed review of the AWS documentation, the Firecracker paper and various talks at AWS re:Invent.