Simon Willison's Weblog: tom-macwright

Tom MacWright: Observable Notebooks 2.0

2025-08-06T16:37:13+00:00

Tom MacWright: Observable Notebooks 2.0

Observable announced Observable Notebooks 2.0 last week - the latest take on their JavaScript notebook technology, this time with an open file format and a brand new macOS desktop app.

Tom MacWright worked at Observable during their first iteration and here provides thoughtful commentary from an insider-to-outsider perspective on how their platform has evolved over time.

I particularly appreciated this aside on the downsides of evolving your own not-quite-standard language syntax:

Notebook Kit and Desktop support vanilla JavaScript, which is excellent and cool. The Observable changes to JavaScript were always tricky and meant that we struggled to use off-the-shelf parsers, and users couldn't use standard JavaScript tooling like eslint. This is stuff like the viewof operator which meant that Observable was not JavaScript. [...] Sidenote: I now work on Val Town, which is also a platform based on writing JavaScript, and when I joined it also had a tweaked version of JavaScript. We used the @ character to let you 'mention' other vals and implicitly import them. This was, like it was in Observable, not worth it and we switched to standard syntax: don't mess with language standards folks!

Tags: javascript, observable, tom-macwright, val-town

Directive prologues and JavaScript dark matter

2025-06-02T18:30:31+00:00

Directive prologues and JavaScript dark matter

Tom MacWright does some archaeology and describes the three different magic comment formats that can affect how JavaScript/TypeScript files are processed:

"a directive"; is a directive prologue, most commonly seen with "use strict";.

/** @aPragma */ is a pragma for a transpiler, often used for /** @jsx h */.

//# aMagicComment is usually used for source maps - //# sourceMappingURL=<url> - but also just got used by v8 for their new explicit compile hints feature.

Via Jim Nielsen

Tags: javascript, programming-languages, v8, typescript, tom-macwright

A warning about tiktoken, BPE, and OpenAI models

2024-11-21T06:13:51+00:00

A warning about tiktoken, BPE, and OpenAI models

Tom MacWright warns that OpenAI's tiktoken Python library has a surprising performance profile: it's superlinear with the length of input, meaning someone could potentially denial-of-service you by sending you a 100,000 character string if you're passing that directly to tiktoken.encode().

There's an open issue about this (now over a year old), so for safety today it's best to truncate on characters before attempting to count or truncate using tiktoken.

Tags: denial-of-service, python, security, tom-macwright, openai

Quoting Tom MacWright

2024-11-03T16:36:13+00:00

Building technology in startups is all about having the right level of tech debt. If you have none, you’re probably going too slow and not prioritizing product-market fit and the important business stuff. If you get too much, everything grinds to a halt. Plus, tech debt is a “know it when you see it” kind of thing, and I know that my definition of “a bunch of tech debt” is, to other people, “very little tech debt.”

— Tom MacWright

Tags: tom-macwright, technical-debt

Quoting Tom MacWright

2024-08-12T20:17:08+00:00

But [LLM assisted programming] does make me wonder whether the adoption of these tools will lead to a form of de-skilling. Not even that programmers will be less skilled, but that the job will drift from the perception and dynamics of a skilled trade to an unskilled trade, with the attendant change - decrease - in pay. Instead of hiring a team of engineers who try to write something of quality and try to load the mental model of what they're building into their heads, companies will just hire a lot of prompt engineers and, who knows, generate 5 versions of the application and A/B test them all across their users.

— Tom MacWright

Tags: ai, tom-macwright, generative-ai, llms, ai-assisted-programming

The first four Val Town runtimes

2024-02-08T18:38:39+00:00

The first four Val Town runtimes

Val Town solves one of my favourite technical problems: how to run untrusted code in a safe sandbox. They're on their fourth iteration of this now, currently using a Node.js application that launches Deno sub-processes using the node-deno-vm npm package and runs code in those, taking advantage of the Deno sandboxing mechanism and terminating processes that take too long in order to protect against while(true) style attacks.

Via @tmcw

Tags: javascript, nodejs, sandboxing, deno, tom-macwright, val-town

Playing with ActivityPub

2022-12-10T00:58:42+00:00

Playing with ActivityPub

Tom MacWright describes his attempts to build the simplest possible ActivityPub publication—for a static site powered by Jekyll, where he used Netlify functions to handle incoming subscriptions (storing them in PlanetScale via their Deno API library) and wrote a script which loops through and notifies all of his subscriptions every time he publishes something new.

Via lobste.rs

Tags: deno, tom-macwright, mastodon, activitypub

Quoting Tom MacWright

2022-03-04T16:11:08+00:00

Working with the web platform is dealing with history, with the accumulated matter of quirksmode and good-enough standards. In exchange for the ability to deliver instantly-updating software directly to customers with no middlemen and no installation, you have to absorb a great deal of nearly-useless information that’s entirely about dodging meaningless traps.

— Tom MacWright

Tags: web, tom-macwright

lon lat lon lat lon

2022-02-10T16:32:49+00:00

lon lat lon lat lon

Tom MacWright’s definitive guide to the (latitude, longitude) v.s. (longitude, latitude) debate. The answer is frustrating: both orders are used by significant software, so there’s no single answer that will satisfy everyone. I’ve recently been mostly convinced over to the longitude, latitude side mainly because that’s a better fit for the non-geospatial x, y pattern.

Tags: geospatial, tom-macwright

GitHub Burndown

2022-02-10T16:29:04+00:00

GitHub Burndown

Neat Observable notebook by Tom MacWright—give it a GitHub access token and the name of a repo and it pulls the details of every issue and plots a burndown chart over time, showing how long issues stay open for. The code is worth spending some time with—the way it fetches data from the paginated JSON API is a really great example of using generators with Observable, and the chart itself is a lovely clear example of Observable Plot.

Via @tmcw

Tags: github, observable, tom-macwright, observable-plot

Serving map tiles from SQLite with MBTiles and datasette-tiles

2021-02-04T01:09:30+00:00

Working on datasette-leaflet last week re-kindled my interest in using Datasette as a GIS (Geographic Information System) platform. SQLite already has strong GIS functionality in the form of SpatiaLite and datasette-cluster-map is currently the most downloaded plugin. Most importantly, maps are fun!

MBTiles

I was talking to Tom MacWright on Monday and I mentioned that I'd been thinking about how SQLite might make a good mechanism for distributing tile images for use with libraries like Leaflet. "I might be able to save you some time there" he said... and he showed me MBTiles, a specification he started developing ten years ago at Mapbox which does exactly that - bundles tile images up in SQLite databases.

(My best guess is I read about MBTiles a while ago, then managed to forget about the spec entirely while the idea of using SQLite for tile distribution wedged itself in my head somewhere.)

The new datasette-tiles plugin

I found some example MBTiles files on the internet and started playing around with them. My first prototype used the datasette-media plugin, described here previously in Fun with binary data and SQLite. I used some convoluted SQL to teach it that hits to /-/media/tiles/{z},{x},{y} should serve up content from the tiles table in my MBTiles database - you can see details of that prototype in this TIL: Serving MBTiles with datasette-media.

The obvious next step was to write a dedicated plugin: datasette-tiles. Install it and run Datasette against any MBTiles database file and the plugin will set up a /-/tiles/db-name/z/x/y.png endpoint that serves the specified tiles.

It also adds a tile explorer view with a pre-configured Leaflet map. Here's a live demo serving up a subset of Stamen's toner map - just zoom levels 6 and 7 for the country of Japan.

Here's how to run this on your own computer:

# Install Datasette
brew install datasette
# Install the plugin
datasette install datasette-tiles
# Download the japan-toner.db database
curl -O https://datasette-tiles-demo.datasette.io/japan-toner.db
# Launch Datasette and open a browser
datasette japan-toner.db -o
# Use the cog menu to access the tile explorer
# Or visit http://127.0.0.1:8001/-/tiles/japan-toner

Creating MBTiles files with my download-tiles tool

A sticking point when I started playing with MBTiles was finding example files to work with.

After some digging, I came across the amazing HOT Export Tool. It's a project by the Humanitarian OpenStreetMap Team that allows anyone to export subsets of data from OpenStreetMap in a wide variety of formats, including MBTiles.

I filed a minor bug report against it, and in doing so took a look at the source code (it's all open source)... and found the code that assembles MBTiles files. It uses another open source library called Landez, which provides functions for downloading tiles from existing providers and bundling those up as an MBTiles SQLite file.

I prefer command-line tools for this kind of thing over using Python libraries directly, so I fired up my click-app cookiecutter template and built a thin command-line interface over the top of the library.

The new tool is called download-tiles and it does exactly that: downloads tiles from a tile server and creates an MBTiles SQLite database on disk containing those tiles.

Please use this tool responsibly. Downloading large numbers of tiles is bad manners. Be sure to familiarize yourself with the OpenStreetMap Tile Usage Policy, and use the tool politely when pointing it at other tile servers.

Basic usage is as follows:

download-tiles world.mbtiles

By default the tool pulls tiles from OpenStreetMap. The above command will fetch zoom levels 0-3 of the entire world - 85 tiles total, well within acceptable usage limits.

Various options (described in the README) can be used to customize the tiles that are downloaded. Here's how I created the japan-toner.db demo database, linked to above:

download-tiles japan-toner.mbtiles \ 
    --zoom-levels 6-7 \
    --country Japan \
    --tiles-url "http://{s}.tile.stamen.com/toner/{z}/{x}/{y}.png" \
    --tiles-subdomains "a,b,c,d" \
    --attribution 'Map tiles by Stamen Design, under CC BY 3.0. Data by OpenStreetMap, under CC BY SA.'

The --country Japan option here looks up the bounding box for Japan using Nominatim. --zoom-levels 6-7 fetches zoom levels 6 and 7 (in this case that makes for 193 tiles total). --tiles-url and --tiles-subdomain configure the tile server to fetch them from. The --attribution option bakes that string into the metadata table for the database - which is then used to display it correctly in the tile explorer (and eventually in other Datasette plugins).

datasette-basemap

Out of the box, Datasette's current Leaflet plugins (datasette-cluster-map, datasette-leaflet-geojson and so on) serve tiles directly from the OpenStreetMap tile server.

I've never felt particularly comfortable about this. Users can configure the plugins to run against other tile servers, but pointing to OpenStreetMap as a default was the easiest way to ensure these plugins would work for people who just wanted to try them out.

Now that I have the tooling for bundling map subsets, maybe I can do better.

datasette-basemap offers an alternative: it's a plugin that bundles a 22.7MB SQLite file containing zoom levels 0-6 of OpenStreetMap - 5,461 tiles total.

Running pip install datasette-basemap (or datasette install datasette-basemap) will install the plugin, complete with that database - and register it with Datasette.

Start Datasette with the plugin installed and /basemap will expose the bundled database. Install datasette-tiles and you'll be able to browse it as a tile server: here's a demo.

(I recommend also installing datasette-render-images so you can see the tile images themselves in the regular table view, like this.)

Zoom level 6 is close enough that major cities and the roads between them are visible, for all of the countries in the world. Not bad for 22.7MB!

This is the first time I've built a Datasette plugin that bundles a full SQLite database as part of the Python package. The pattern seems to work well - I'm excited to explore it further with other projects.

Bonus feature: tile stacks

I added one last feature to datasette-tiles before writing everything up for my blog. I'm calling this feature tile stacks - it lets you serve tiles from multiple MBTiles files, falling back to other files if a tile is missing.

Imagine you had a low-zoom-level world map (similar to datasette-basemap) and a number of other databases providing packages of tiles for specific countries or cities. You could run Datasette like this:

datasette basemap.mbtiles japan.mbtiles london.mbtiles tokyo.mbtiles

Hitting /-/tiles-stack/1/1/1.png would seek out the specified tile in the tokyo.mbtiles file, then fall back to london.mbtiles and then japan.mbtiles and finally basemap.mbtiles if it couldn't find it.

For a demo, visit https://datasette-tiles-demo.datasette.io/-/tiles-stack and zoom in on Japan. It should start to display the Stamen toner map once you get to zoom levels 6 and 7.

Next steps

I've been having a lot of fun exploring MBTiles - it's such a natural fit for Datasette, and it's exciting to be able to build new things on top of nearly a decade of innovation by other geo-hackers.

There are plenty of features missing from datasette-tiles.

It currently only handles .png image data, but the MBTiles 1.3 specification also defines .jpg and .webp tiles, plus vector tiles using Mapbox's .pbf gzip-compressed protocol buffers.

UTFGrid is a related specification for including "rasterized interaction data" in MBTiles databases - it helps efficiently provide maps with millions of embedded objects.

As a newcomer to the MBTiles world I'd love to hear suggestions for new features and feedback on how I can improve what I've got so far in the datasette-tiles issues.

Being able to serve your own map tiles like this feels very much in the spirit of the OpenStreetMap project. I'm looking forward to using my own tile subsets for any future projects that fit within a sensible tile subset.

Tags: geospatial, mapping, projects, sqlite, datasette, tom-macwright, leaflet

Quoting Tom MacWright

2020-05-11T00:03:42+00:00

And for what? Again - there is a swath of use cases which would be hard without React and which aren’t complicated enough to push beyond React’s limits. But there are also a lot of problems for which I can’t see any concrete benefit to using React. Those are things like blogs, shopping-cart-websites, mostly-CRUD-and-forms-websites. For these things, all of the fancy optimizations are optimizations to get you closer to the performance you would’ve gotten if you just hadn’t used so much technology.

— Tom MacWright

Tags: react, tom-macwright

Things I learned about shapefiles building shapefile-to-sqlite

2020-02-19T05:25:58+00:00

The latest in my series of x-to-sqlite tools is shapefile-to-sqlite. I learned a whole bunch of things about the ESRI shapefile format while building it.

Governments really love ESRI shapefiles. There is a huge amount of interesting geospatial data made available in the format - 4,614 on Data.gov!

shapefile-to-sqlite

shapefile-to-sqlite loads the data from these files into a SQLite database, turning geometry properties into database columns and the geometry itself into a blob of GeoJSON. Let's try it out on a shapefile containing the boundaries of US national parks.

$ wget http://nrdata.nps.gov/programs/lands/nps_boundary.zip
...
Saving to: ‘nps_boundary.zip’
nps_boundary.zip                           100%[=====================================================================================>]  12.61M   705KB/s    in 22s     
2020-02-18 19:59:22 (597 KB/s) - ‘nps_boundary.zip’ saved [13227561/13227561]

$ unzip nps_boundary.zip 
Archive:  nps_boundary.zip
inflating: temp/Current_Shapes/Data_Store/06-06-12_Posting/nps_boundary.xml  
inflating: temp/Current_Shapes/Data_Store/06-06-12_Posting/nps_boundary.dbf  
inflating: temp/Current_Shapes/Data_Store/06-06-12_Posting/nps_boundary.prj  
inflating: temp/Current_Shapes/Data_Store/06-06-12_Posting/nps_boundary.shp  
inflating: temp/Current_Shapes/Data_Store/06-06-12_Posting/nps_boundary.shx

$ shapefile-to-sqlite nps.db temp/Current_Shapes/Data_Store/06-06-12_Posting/nps_boundary.shp
temp/Current_Shapes/Data_Store/06-06-12_Posting/nps_boundary.shp
[####################################]  100%

$ datasette nps.db
Serve! files=('nps.db',) (immutables=()) on port 8003
INFO:     Started server process [33534]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8001 (Press CTRL+C to quit)

I recommend installing the datasette-leaflet-geojson plugin, which will turn any column containing GeoJSON into a Leaflet map.

If you've installed SpatiaLite (installation instructions here) you can use the --spatialite option to instead store the geometry in a SpatiaLite column, unlocking a bewildering array of SQL geometry functions.

$ shapefile-to-sqlite nps.db temp/Current_Shapes/Data_Store/06-06-12_Posting/nps_boundary.shp --spatialite --table=nps-spatialite
temp/Current_Shapes/Data_Store/06-06-12_Posting/nps_boundary.shp
[##################################--]   94%  00:00:00

I deployed a copy of the resulting database using Cloud Run:

$ datasette publish cloudrun nps.db \
    --service national-parks \
    --title "National Parks" \
    --source_url="https://catalog.data.gov/dataset/national-parks" \
    --source="data.gov" \
    --spatialite \
    --install=datasette-leaflet-geojson \
    --install=datasette-render-binary \
    --extra-options="--config max_returned_rows:5"

I used max_returned_rows:5 there because these geometrries are pretty big - without it a page with 100 rows on it can return over 90MB of HTML!

You can browse the GeoJSON version of the table here and the SpatiaLite version here.

The SpatiaLite version defaults to rendering each geometry as an ugly binary blob. You can convert them to GeoJSON for compatibility with datasette-leaflet-geojson using the SpatiaLite AsGeoJSON() function:

select id, UNIT_NAME, AsGeoJSON(geometry)
from [nps-spatialite]

Here's the result of that query running against the demo.

Understanding shapefiles

The most confusing thing about shapefiles is that they aren't a single file. A shapefile comes as a minimum of three files: foo.shp containing geometries, foo.shx containing an index into those geometries (really more of an implementation detail) and foo.dbf contains key/value properties for each geometry.

They often come bundled with other files too. foo.prj is a WKT projection for the data for example. Wikipedia lists a whole bunch of other possibilities.

As a result, shapefiles are usually distributed as a zip file. Some shapefile libraries can even read directly from a zip.

The GeoJSON format was designed as a modern alternative to shapefiles, so understanding GeoJSON really helps in understanding shapefiles. In particular the GeoJSON geometry types: Point, LineString, MultiLineString, Polygon and MultiPolygon match how shapefile geometries work.

An important detail in shapefiles is that data in the .shp and .dbf files is matched by array index - so the first geometry can be considered as having ID=0, the second ID=1 and so on.

You can read the properties from the .dbf file using the dbfread Python module like this:

$ ipython
In [1]: import dbfread
In [2]: db = dbfread.DBF("temp/Current_Shapes/Data_Store/06-06-12_Posting/nps_boundary.dbf")
In [3]: next(iter(db))
Out[3]: 
OrderedDict([('UNIT_TYPE', 'Park'),
            ('STATE', ''),
            ('REGION', 'NC'),
            ('UNIT_CODE', 'NACC'),
            ('UNIT_NAME', 'West Potomac Park'),
            ('DATE_EDIT', None),
            ('GIS_NOTES', ''),
            ('CREATED_BY', 'Legacy'),
            ('METADATA', ''),
            ('PARKNAME', '')])

Reading shapefiles in Python

I'm a big fan of the Shapely Python library, so I was delighted to see that Sean Gillies, creator of Shapely, also created a library for reading and writing shapefiles: Fiona.

GIS with Python, Shapely, and Fiona by Tom MacWright was particularly useful for figuring this out. I like how he wrote that post in 2012 but added a note in 2017 that it's still his recommended way of getting started with GIS in Python.

Projections

The trickiest part of working with any GIS data is always figuring out how to deal with projections.

GeoJSON attempts to standardize on WGS 84, otherwise known as the latitude/longitude model used by GPS. But... shapefiles frequently use something else. The Santa Clara county parks shapefiles for example use EPSG:2227, also known as California zone 3.

(Fun fact: ESPG stands for European Petroleum Survey Group, a now defunct oil industry group that today lives on only as a database of projected coordinate systems.)

I spent quite a while thinking about how to best handle projections. In the end I decided that I'd follow GeoJSON's lead and attempt to convert everything to WGS 84, but allow users to skip that behaviour using --crs=keep or to specify an alternative projection to convert to with --crs=epsg:2227 or similar.

SpatiaLite creates its geometry columns with a baked in SRID (a code which usually maps to the EPSG identifier). You can see which SRID was used for a specific geometry using the srid() function:

select srid(geometry) from "nps-spatialite" limit 1

SpatiaLite can also convert to another projection using the Transform() function:

select ':' || AsGeoJSON(Transform(geometry, 2227)) from "nps-spatialite" limit 1

(I'm using ':' || AsGeoJSON(...) here to disable the datasette-leaflet-geojson plugin, since it can't correctly render data that has been transformed to a non-WGS-84 proection.)

Pulling it all together

I now have two tools for imorting geospatial data into SQLite (or SpatiaLite) databases: shapefile-to-sqlite and geojson-to-sqlite.

I'm excited about Datasette's potential as a tool for GIS. I started exploring this back in 2017 when I used it to build a location to timezone API - but adding easy shapefile imports to the toolchain should unlock all kinds of interesting new geospatial projects.

Tags: geospatial, projects, shapefiles, spatialite, sqlite, geojson, weeknotes, tom-macwright, leaflet

togeojson

2019-01-18T23:50:00+00:00

togeojson

Handy JavaScript library and command-mine tool for converting KML and GPX to GeoJSON, by Tom MacWright

Via @tmcw

Tags: geospatial, kml, geojson, tom-macwright

Observable Beta

2018-01-31T16:46:38+00:00

Observable Beta

Observable just released their beta, and it’s quite something. It’s by Mike Bostock (d3), Jeremy Ashkenas (Backbone, CoffeeScript) and Tom MacWright (Mapbox Studio). The easiest way to describe it is Jupyter notebooks for JavaScript supporting reactive programming—so code is evaluated as you type and you can add interactive widgets (like sliders and canvas views) to construct explorable visualizations on the fly.

Via Hacker News

Tags: javascript, jeremy-ashkenas, d3, jupyter, observable, mike-bostock, tom-macwright