Simon Willison's Weblog: paul-ford

Quoting Paul Ford

2026-02-23T16:00:32+00:00

The paper asked me to explain vibe coding, and I did so, because I think something big is coming there, and I'm deep in, and I worry that normal people are not able to see it and I want them to be prepared. But people can't just read something and hate you quietly; they can't see that you have provided them with a utility or a warning; they need their screech. You are distributed to millions of people, and become the local proxy for the emotions of maybe dozens of people, who disagree and demand your attention, and because you are the one in the paper you need to welcome them with a pastor's smile and deep empathy, and if you speak a word in your own defense they'll screech even louder.

— Paul Ford, on writing about vibe coding for the New York Times

Tags: new-york-times, paul-ford, vibe-coding

The A.I. Disruption We’ve Been Waiting for Has Arrived

2026-02-18T17:07:31+00:00

The A.I. Disruption We’ve Been Waiting for Has Arrived

New opinion piece from Paul Ford in the New York Times. Unsurprisingly for a piece by Paul it's packed with quoteworthy snippets, but a few stood out for me in particular.

Paul describes the November moment that so many other programmers have observed, and highlights Claude Code's ability to revive old side projects:

[Claude Code] was always a helpful coding assistant, but in November it suddenly got much better, and ever since I’ve been knocking off side projects that had sat in folders for a decade or longer. It’s fun to see old ideas come to life, so I keep a steady flow. Maybe it adds up to a half-hour a day of my time, and an hour of Claude’s.

November was, for me and many others in tech, a great surprise. Before, A.I. coding tools were often useful, but halting and clumsy. Now, the bot can run for a full hour and make whole, designed websites and apps that may be flawed, but credible. I spent an entire session of therapy talking about it.

And as the former CEO of a respected consultancy firm (Postlight) he's well positioned to evaluate the potential impact:

When you watch a large language model slice through some horrible, expensive problem — like migrating data from an old platform to a modern one — you feel the earth shifting. I was the chief executive of a software services firm, which made me a professional software cost estimator. When I rebooted my messy personal website a few weeks ago, I realized: I would have paid $25,000 for someone else to do this. When a friend asked me to convert a large, thorny data set, I downloaded it, cleaned it up and made it pretty and easy to explore. In the past I would have charged $350,000.

That last price is full 2021 retail — it implies a product manager, a designer, two engineers (one senior) and four to six months of design, coding and testing. Plus maintenance. Bespoke software is joltingly expensive. Today, though, when the stars align and my prompts work out, I can do hundreds of thousands of dollars worth of work for fun (fun for me) over weekends and evenings, for the price of the Claude $200-a-month plan.

He also neatly captures the inherent community tension involved in exploring this technology:

All of the people I love hate this stuff, and all the people I hate love it. And yet, likely because of the same personality flaws that drew me to technology in the first place, I am annoyingly excited.

Tags: new-york-times, paul-ford, careers, ai, generative-ai, llms, ai-assisted-programming, ai-ethics, coding-agents, claude-code, deep-blue, november-2025-inflection

Quoting Paul Ford

2020-12-16T05:35:47+00:00

I get asked a lot about learning to code. Sure, if you can. It's fun. But the real action, the crux of things, is there in the database. Grab a tiny, free database like SQLite. Import a few million rows of data. Make them searchable. It's one of the most soothing activities known to humankind, taking big piles of messy data and massaging them into the rigid structure required of a relational database. It's true power.

— Paul Ford

Tags: databases, paul-ford, sqlite

Fun with binary data and SQLite

2020-07-30T23:22:17+00:00

This week I've been mainly experimenting with binary data storage in SQLite. sqlite-utils can now insert data from binary files, and datasette-media can serve content over HTTP that originated as binary BLOBs in a database file.

Paul Ford piqued my interest in this when he tweeted about loading thousands of PDF documents into a SQLite database:

I made a shell script that loads thousands of PDF docs into SQLite databases. That means I can have a web server that produces infinite randomly chosen US Military PowerPoint slides that I can scroll on my phone when I'm around the house.
— Paul Ford (@ftrain) July 26, 2020

The SQLite documentation claims that serving smaller binary files from BLOB columns can be 35% faster than the filesystem. I've done a little bit of work with binary files in SQLite - the datasette-render-binary and datasette-render-images both help display BLOB data - but I'd never really dug into it in much detail.

sqlite-utils insert-files

The first step was to make it easier to build database files that include binary data.

sqlite-utils is my combination Python library and CLI tool for building SQLite databases. I've been steadily evolving it for a couple of years now, and it's the engine behind my Dogsheep collection of tools for personal analytics.

The new insert-files command can be used to insert content from binary files into a SQLite database, along with file metadata.

The most basic usage looks like this:

sqlite-utils insert-files gifs.db images *.gif

By default, this creates a database table like so:

CREATE TABLE [images] (
    [path] TEXT PRIMARY KEY,
    [content] BLOB,
    [size] INTEGER
);

You can customize this table to include other file metadata using the -c (short for --column) option:

sqlite-utils insert-files gifs.db images *.gif \
    -c path -c md5 -c last_modified:mtime -c size --pk=path

This creates a table with the following schema:

CREATE TABLE [images] (
    [path] TEXT PRIMARY KEY,
    [md5] TEXT,
    [last_modified] FLOAT,
    [size] INTEGER
);

If you pass a directory instead of a file name the command will recursively add every file in that directory.

I also improved sqlite-utils with respect to outputting binary data. The new --raw option dumps the binary contents of a column directly to standard out, so you can read an image back out of one of the above tables like so:

sqlite-utils photos.db \
    "select content from images where path=:path" \
    -p path 'myphoto.jpg' \
    --raw > myphoto.jpg

This example also demonstrates the new support for :parameters passed using the new -p option, see #124.

sqlite-utils usually communicates using JSON, but JSON doesn't have the ability to represent binary values. Datasette outputs binary values like so:

"data": {
  "$base64": true,
  "encoded": "iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAY..."
}

I added support for the same format to sqlite-utils - so you can now query binary columns and get out that nested object, or pipe JSON with that nested structure in to sqlite-utils insert and have it stored as a binary BLOB in the database.

datasette-media

datasette-media is a plugin for serving binary content directly from Datasette on a special URL. I originally built it while working on Dogsheep Photos - given a SQLite file full of Apple Photos metadata I wanted to be able to serve thumbnails of the actual images via my Datasette web server.

Those photos were still stored on disk - the plugin lets you configure a SQL query like this which will cause hits to /-/media/photos/$UUID to serve that file from disk:

{
    "plugins": {
        "datasette-media": {
            "photo": {
                "sql": "select filepath from apple_photos where uuid=:key"
            }
        }
    }
}

Issue #14 added support for BLOB columns as well. You can now configure the plugin like this to serve binary content that was stored in the database:

{
    "plugins": {
        "datasette-media": {
            "thumb": {
                "sql": "select content from thumbnails where uuid=:key"
            }
        }
    }
}

This would serve content from a BLOB column in a thumbnails table from the URL /-/media/thumb/$UUID.

I really like this pattern of configuring plugins using SQL queries, where the returned column names have special meaning that is interpreted by the plugin. datasette-atom and datasette-ics use a similar trick.

I expanded datasette-media with a few other related features:

Return a content_url column and it will proxy content from that URL
Set "enable_transform": true for a media bucket to enable ?w= and ?h= and ?format= parameters for transforming the image before it is served to the user
Return a content_filename column to set a download file name (in a content-disposition HTTP header) prompting the user's browser to download the file

See the README or release notes for more details.

Also this week

I renamed datasette-insert-api to just datasette-insert, reflecting my plans to add non-API features to that plugin in the future.

In doing so I had to figure out how to rename a PyPI package such that dependent projects would continue to work. I ended up building a pypi-rename cookiecutter template encoding what I learned.

I enabled PostgreSQL full-text search for my blog's Django Admin interface, and wrote a TIL on how I did it.

I added compound primary key support to db-to-sqlite, so now it can convert PostgreSQL or MySQL databases to SQLite if they use compound primary keys.

TIL this week

Releases this week

sqlite-utils 2.13 - 2020-07-30
datasette-media 0.5 - 2020-07-29
db-to-sqlite 1.3 - 2020-07-27
datasette-media 0.4 - 2020-07-27
sqlite-utils 2.12 - 2020-07-27
datasette-render-images 0.3.1 - 2020-07-27
datasette-render-images 0.3 - 2020-07-27
datasette-auth-passwords 0.3.1 - 2020-07-26
datasette-insert 0.5 - 2020-07-25

Tags: paul-ford, plugins, projects, sqlite, datasette, weeknotes, sqlite-utils

Why I (Still) Love Tech: In Defense of a Difficult Industry

2019-05-15T15:45:20+00:00

Why I (Still) Love Tech: In Defense of a Difficult Industry

If you only read one longform piece this week, make it this one. Utterly delightful prose and a bunch of different messages that resonated with me deeply.

Via Daring Fireball

Tags: paul-ford

Quoting Paul Ford

2019-05-15T15:44:01+00:00

Imagine if you were really into the group Swervedriver in the mid-’90s but by 2019 someone was on CNBC telling you that Swervedriver represented, I don’t know, 10 percent of global economic growth, outpacing returns in oil and lumber. That’s the tech industry.

— Paul Ford

Tags: paul-ford

Usable Data

2019-01-11T18:33:18+00:00

Usable Data

A Paul Ford essay from February 2016 in which he advocates for SQLite as the ideal format for sharing interesting data. I don’t know how I missed this one—it predates Datasette, but it perfectly captures the benefits that I’m trying to expose with the project. “In my dream universe, there would be a massive searchable torrent site filled with open, explorable data sets, in SQLite format, some with full text search indexes already in place.”

Via Twitter

Tags: open-data, paul-ford, sqlite, datasette

Quoting Paul Ford

2018-03-10T11:34:30+00:00

Consider Bitcoin a grand middle finger. It’s a prank, almost a parody of the global financial system, that turned into a bubble. “You plutocrats of Davos may think you control the global money supply,” the pranksters seem to say. “But humans will make an economy out of anything. Even this!”

— Paul Ford

Tags: economics, paul-ford, bitcoin, blockchain

How I made a Who's On First subset database

2018-02-03T05:25:25+00:00

How I made a Who's On First subset database

Inspired by Paul Ford on Twitter, I tried out a new trick with SQLite: connect to a database containing JSON, attach a brand new empty database file using “attach database”, then populate it using INSERT INTO ... SELECT plus the json_extract() function to extract out a subset of the JSON properties into a new table in the new database.

Tags: json, paul-ford, sqlite, whosonfirst

Quoting Paul Ford

2018-01-29T16:29:36+00:00

[On SQLite] The JSON interface is like, "we save the text and when you retrieve it we parse the JSON at several hundred MB/s and let you do path queries against it please stop overthinking it, this is filing cabinet."

— Paul Ford

Tags: json, paul-ford, sqlite

Datasette: Ability to customize presentation of specific columns in HTML view

2017-11-30T07:27:58+00:00

Datasette: Ability to customize presentation of specific columns in HTML view

Still a work in progress, but Datasette master now allows you to inject links to one or more additional CSS and JavaScript resources (optionally with SRI hashes) which will be included on every page. Each template also
now provides CSS classes on the body element derived from the current database and table names to provide hooks for custom styling. Next up: custom template support.

Tags: css, paul-ford, datasette

Quoting Paul Ford

2017-11-17T01:10:49+00:00

If you’re a public data provider—and many large NGOs, government organizations, cultural organizations, historical archives, media organizations, medical orgs, and academic institutions are exactly that—you can publish gigabytes of data, and make it available as an API, and make it easy to browse on the web, too, with extremely low effort. Put it into SQLite, point this little guy at it, and you’ve just radically increased the accessibility and utility of your data. Because messing around in SQL from a web browser is orders of magnitude more immediately useful than downloading a CSV, processing it, and figuring out what comes next.

— Paul Ford

Tags: paul-ford, sqlite, datasette

Learning to Fear the Semantic Web

2008-10-23T16:14:03+00:00

Learning to Fear the Semantic Web

Paul Ford raises the liability issue with regards to building sites around other people’s metadata, pointing out that OpenCalais is owned by Thomson Reuters who have a bad track record with regards to intellectual property lawsuits elsewhere in the organisation.

Tags: intellectualproperty, opencalais, paul-ford, semanticweb, thomson-reuters

Harper's Magazine

2007-04-14T00:05:37+00:00

Harper's Magazine

The site with the best metadata on the Web just relaunched, with even MORE metadata.

Via Launch (Ftrain.com)

Tags: harpers, metadata, paul-ford

Quoting Paul Ford

2007-01-08T13:46:04+00:00

If your average iPod weighs five ounces with packaging, then Apple has moved about 21,875,000 pounds of them, equivalent in weight to 1,325 full-grown male African elephants, 35 times as many as Hannibal's force.

— Paul Ford

Tags: apple, hannibal, ipod, paul-ford

Gothamist Interviews: Paul Ford

2004-02-25T23:47:10+00:00

Gothamist Interviews: Paul Ford

“the only way to ignore advertising is to die”

Tags: paul-ford