Simon Willison's Weblog: api-design

Quoting Addy Osmani

2026-01-04T16:40:39+00:00

With enough users, every observable behavior becomes a dependency - regardless of what you promised. Someone is scraping your API, automating your quirks, caching your bugs.

This creates a career-level insight: you can’t treat compatibility work as “maintenance” and new features as “real work.” Compatibility is product.

Design your deprecations as migrations with time, tooling, and empathy. Most “API design” is actually “API retirement.”

— Addy Osmani, 21 lessons from 14 years at Google

Tags: api-design, google, careers, addy-osmani

Jiff

2024-07-22T04:48:35+00:00

Jiff

Andrew Gallant (aka BurntSushi) implemented regex for Rust and built the fabulous ripgrep, so it's worth paying attention to their new projects.

Jiff is a brand new datetime library for Rust which focuses on "providing high level datetime primitives that are difficult to misuse and have reasonable performance". The API design is heavily inspired by the Temporal proposal for JavaScript.

The core type provided by Jiff is Zoned, best imagine as a 96-bit integer nanosecond time since the Unix each combined with a geographic region timezone and a civil/local calendar date and clock time.

The documentation is comprehensive and a fascinating read if you're interested in API design and timezones.

Via @burntsushi5

Tags: api-design, timezones, rust, andrew-gallant

Designing a write API for Datasette

2022-11-09T19:44:32+00:00

Building out Datasette Cloud has made one thing clear to me: Datasette needs a write API for ingesting new data into its attached SQLite databases.

I had originally thought that this could be left entirely to plugins: my datasette-insert plugin already provides a JSON API for inserting data, and other plugins like datasette-upload-csvs also implement data import functionality.

But some things deserve to live in core. An API for manipulating data is one of them, because it can hopefully open up a floodgate of opportunities for other plugins and external applications to build on top of it.

I've been working on this over the past two weeks, in between getting distracted by Mastodon (it's just blogs!).

Designing the API

You can follow my progress in this tracking issue: Write API in Datasette core #1850. I'm building the new functionality in a branch (called 1.0-dev, because this is going to be one of the defining features of Datasette 1.0 - and will be previewed in alphas of that release).

Here's the functionality I'm aiming for in the first alpha:

API for writing new records (singular or plural) to a table
API for updating an existing record
API for deleting an existing record
API for creating a new table - either with an explicit schema or by inferring it from a set of provided rows
API for dropping a table

I have a bunch of things I plan to add later, but I think the above represents a powerful, coherent set of initial functionality.

In terms of building this, I have a secret weapon: sqlite-utils. It already has both a Python client library and a comprehensive CLI interface for inserting data and creating tables. I've evolved the design of those over multiple major versions, and I'm confident that they're solid. Datasette's write API will mostly implement the same patterns I've eventually settled on for sqlite-utils.

I still need to design the higher level aspects of the API though - the endpoint URLs and the JSON format that will be used.

This is still in flux, but my current design looks like this.

To insert records:

POST /database/table/-/insert

{
    "rows": [
        {"id": 1, "name": "Simon"},
        {"id": 2, "name": "Cleo"}
    ]
}

Or use "row": {...} to insert a single row.

To create a new table with an explicit schema:

POST /database/-/create

{
    "name": "people",
    "columns": [
        {
            "name": "id",
            "type": "integer"
        },
        {
            "name": "title",
            "type": "text"
        }
    ]
   "pk": "id"
}

To create a new table with a schema automatically derived from some initial rows:

POST /database/-/create

{
    "name": "my new table",
    "rows": [
        {"id": 1, "name": "Simon"},
        {"id": 2, "name": "Cleo"}
    ]
   "pk": "id"
}

To update a record:

POST /database/table/134/-/update

{
    "update": {
        "name": "New name"
    }
}

Where 134 in the URL is the primary key of the record. Datasette supports compound primary keys too, so this could be /database/docs/article,242/-/update for a table with a compound primary key.

I'm using a "update" nested object here rather than having everything at the root of the document because that frees me up to add extra future fields that control the update - "alter": true to specify that the table schema should be updated to add new columns, for example.

To delete a record:

POST /database/table/134/-/delete

I thought about using the HTTP DELETE verb here and I'm ready to be convinced that it's a good idea, but thinking back over my career I can't see any times where I've seen DELETE offered a concrete benefit over just sticking with POST for this kind of thing.

This isn't going to be a pure REST API, and I'm OK with that.

So many details

There are so many interesting details to consider here - especially given that Datasette is designed to support ANY schema that's possible in SQLite.

Should you be allowed to update the primary key of an existing record?
What happens if you try to insert a record that violates a foreign key constraint?
What happens if you try to insert a record that violates a unique constraint?
How should inserting binary data work, given that JSON doesn't have a binary type?
What permissions should the different API endpoints require (I'm looking to add a bunch of new ones)
How should compound primary keys be treated?
Should the API return a copy of the records that were just inserted? Initially I thought yes, but it turns out to be a big impact on insert speeds, at least in SQLite versions before the RETURNING clause was added in SQLite 3.35.0 (in March 2021, so not necessarily widely available yet).
How should the interactive API explorer work? I've been building that in this issue.

I'm working through these questions in the various issues attached to my tracking issue. If you have opinions to share you're welcome to join me there!

Token authentication

This is another area that I've previously left to plugins. datasette-auth-tokens adds Authorization: Bearer xxx authentication to Datasette, but if there's a write API in core there really needs to be a default token authentication mechanism too.

I've implemented a default mechanism based around generating signed tokens, described in issue #1852 and described in this in-progress documentation.

The basic idea is to support tokens that are signed JSON objects (similar to JWT but not JWT, because JWT is a flawed standard - I rolled my own using itsdangerous).

The signed content of a token looks like this:

{
    "a": "user_id",
    "t": 1668022423,
    "d": 3600
}

The "a" field captures the ID of the user created that token. The token can then inherit the permissions of that user.

The "t" field shows when the token was initially created.

The "d" field is optional, and indicates after how many seconds duration the token should expire. This allows for the creation of time-limited tokens.

Tokens can be created using the new /-/create-token page or the new datasette create-token CLI command.

It's important to note that this is not intended to be the only way tokens work in Datasette. There are plenty of applications where database-backed tokens makes more sense, since it allows tokens to be revoked individually without rotating secrets and revoking every issued token at once. I plan to implement this pattern myself for Datasette Cloud.

But I think this is a reasonable default scheme to include in Datasette core. It can even be turned off entirely using the new --setting allow_signed_tokens off option.

I'm also planning a variant of these tokens that can apply additional restrictions. Let's say you want to issue a token that acts as your user but is only allowed to insert rows into the docs table in the primary database. You'll be able to create a token that looks like this:

{
    "a": "simonw",
    "t": 1668022423,
    "r": {
        "t": {
            "primary: {
                "docs": ["ir"]
            }
        }
    }
}

"r" means restrictions. The "t" key indicates per-table restrictions, and the "ir" is an acronym for the insert-row permission.

I'm still fleshing out how this will work, but it feels like an important feature of any permissions system. I find it frustrating any time I'm working with a a system that doesn't allow me to create scoped-down tokens.

Releases this week

json-flatten: 0.3 - (2 releases total) - 2022-10-29
Python functions for flattening a JSON object to a single dictionary of pairs, and unflattening that dictionary back to a JSON object
datasette-edit-templates: 0.1 - (2 releases total) - 2022-10-27
Plugin allowing Datasette templates to be edited within Datasette
datasette: 0.63 - (116 releases total) - 2022-10-27
An open source multi-tool for exploring and publishing data
sqlite-utils: 3.30 - (104 releases total) - 2022-10-25
Python CLI utility and library for manipulating SQLite databases
datasette-indieauth: 1.2.1 - (10 releases total) - 2022-10-25
Datasette authentication using IndieAuth and RelMeAuth
shot-scraper: 1.0.1 - (24 releases total) - 2022-10-24

TIL this week

Tags: api-design, projects, datasette, weeknotes, datasette-cloud, sqlite-utils

Quoting Hyrum's Law

2019-11-21T22:45:55+00:00

With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody.

— Hyrum's Law

Tags: api-design

Quoting Hyrum's Law

2018-08-11T00:33:29+00:00

With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody.

— Hyrum's Law

Tags: api-design, brandur-leach

Quoting Malte Ubl

2018-04-15T17:23:18+00:00

The way I would talk about myself as a senior engineer is that I’d say “I know how I would solve the problem” and because I know how I would solve it I could also teach someone else to do it. And my theory is that the next level is that I can say about myself “I know how others would solve the problem”. Let’s make that a bit more concrete. You make that sentence: “I can anticipate how the API choices that I’m making, or the abstractions that I’m introducing into a project, how they impact how other people would solve a problem.”

— Malte Ubl

Tags: api-design, careers

Implementing Stripe-like Idempotency Keys in Postgres

2017-10-27T17:51:52+00:00

Implementing Stripe-like Idempotency Keys in Postgres

Having clients send “idempotency keys” with API requests in order to be able to safely retry them if something’s goes wrong is a really neat trick for making transactional APIs more robust. Here Brandur Leach talks implementation strategies.

Via Hacker News

Tags: api-design, idempotency, postgresql, brandur-leach, stripe

jQuery 1.4.3 Released

2010-10-17T00:15:00+00:00

jQuery 1.4.3 Released

Once again, the thing that impresses me most about this jQuery release is how stable the core API is. Hardly any new methods added, but the existing methods are made faster, more flexible and more predictable. The same as been true for the past several releases as well. It just keeps getting more and more polished.

Tags: api-design, javascript, jquery, recovered

Unimpressed by NodeIterator

2009-06-19T21:53:58+00:00

Unimpressed by NodeIterator

John Resig, one of the most talented API designers I’ve ever come across, posts some well earned criticism of the document.createNodeIterator DOM traversal API.

Tags: api-design, dom, javascript, john-resig, nodeiterator

The Little Manual of API Design (PDF)

2009-05-18T10:14:24+00:00

The Little Manual of API Design (PDF)

A concise, highly readable guide to designing APIs that are “Complete, Easy to learn and memorize, lead to readable code, hard to misuse, and easy to extend”, based on lessons learnt over many years of development of the Qt framework.

Tags: api-design, apis, programming, qt

Building a JavaScript Library

2007-08-24T16:02:59+00:00

Building a JavaScript Library

Slides from John Resig’s Google Tech Talk. Some great tips in here, including: make your APIs orthogonal, look for common patterns, keep things extensible and write the documentation yourself.

Tags: api-design, googletechtalk, javascript, john-resig, jquery, libraries, slideshare