Simon Willison's Weblog: github-codespaces

Coding agents for data analysis

2026-03-16T20:12:32+00:00

Here's the handout I prepared for my NICAR 2026 workshop "Coding agents for data analysis" - a three hour session aimed at data journalists demonstrating ways that tools like Claude Code and OpenAI Codex can be used to explore, analyze and clean data.

Here's the table of contents:

Coding agents

Warmup: ChatGPT and Claude

Setup Claude Code and Codex

Asking questions against a database

Exploring data with agents

Cleaning data: decoding neighborhood codes

Creating visualizations with agents

Scraping data with agents

I ran the workshop using GitHub Codespaces and OpenAI Codex, since it was easy (and inexpensive) to distribute a budget-restricted API key for Codex that attendees could use during the class. Participants ended up burning $23 of Codex tokens.

The exercises all used Python and SQLite and some of them used Datasette.

One highlight of the workshop was when we started running Datasette such that it served static content from a viz/ folder, then had Claude Code start vibe coding new interactive visualizations directly in that folder. Here's a heat map it created for my trees database using Leaflet and Leaflet.heat, source code here.

I designed the handout to also be useful for people who weren't able to attend the session in person. As is usually the case, material aimed at data journalists is equally applicable to anyone else with data to explore.

Tags: data-journalism, geospatial, python, speaking, sqlite, ai, datasette, generative-ai, llms, github-codespaces, nicar, coding-agents, claude-code, codex-cli, leaflet

simonw/codespaces-llm

2025-08-13T05:39:07+00:00

simonw/codespaces-llm

GitHub Codespaces provides full development environments in your browser, and is free to use with anyone with a GitHub account. Each environment has a full Linux container and a browser-based UI using VS Code.

I found out today that GitHub Codespaces come with a GITHUB_TOKEN environment variable... and that token works as an API key for accessing LLMs in the GitHub Models collection, which includes dozens of models from OpenAI, Microsoft, Mistral, xAI, DeepSeek, Meta and more.

Anthony Shaw's llm-github-models plugin for my LLM tool allows it to talk directly to GitHub Models. I filed a suggestion that it could pick up that GITHUB_TOKEN variable automatically and Anthony shipped v0.18.0 with that feature a few hours later.

... which means you can now run the following in any Python-enabled Codespaces container and get a working llm command:

pip install llm
llm install llm-github-models
llm models default github/gpt-4.1
llm "Fun facts about pelicans"

Setting the default model to github/gpt-4.1 means you get free (albeit rate-limited) access to that OpenAI model.

To save you from needing to even run that sequence of commands I've created a new GitHub repository, simonw/codespaces-llm, which pre-installs and runs those commands for you.

Anyone with a GitHub account can use this URL to launch a new Codespaces instance with a configured llm terminal command ready to use:

codespaces.new/simonw/codespaces-llm?quickstart=1

While putting this together I wrote up what I've learned about devcontainers so far as a TIL: Configuring GitHub Codespaces using devcontainers.

Tags: github, projects, python, ai, til, openai, generative-ai, llms, llm, github-codespaces, anthony-shaw

Weeknotes: Datasette Studio and a whole lot of blogging

2024-06-19T04:30:26+00:00

I'm still spinning back up after my trip back to the UK, so actual time spent building things has been less than I'd like. I presented an hour long workshop on command-line LLM usage, wrote five full blog entries (since my last weeknotes) and I've also been leaning more into short-form link blogging - a lot more prominent on this site now since my homepage redesign last week.

Datasette Studio

I ran a workshop for a data journalism class recently which included having students try running structured data extraction using datasette-extract. I didn't want to talk them through installing Python etc on their own machines, so I instead took advantage of a project I've been tinkering with for a little while called Datasette Studio.

Datasette Studio is actually two things. The first is a distribution of Datasette which bundles the core application along with a selection of plugins that greatly increase its capabilities as a tool for cleaning and analyzing data. You can install that like this:

pipx install datasette-studio

Then run datasette-studio to start the server or datasette-studio install xyz to install additional plugins.

Datasette Studio runs the latest Datasette 1.0 alpha, and will upgrade to 1.0 stable as soon as that is released.

Quoting the pyproject.toml file, the current list of plugins is this:

I plan to grow this list over time. A neat thing about datasette-studio is that the entire application is defined by a single pyproject.toml that lists those dependecies and sets up the datasette-studio CLI console script, which is then published to PyPI.

The second part of Datasette Studio is a GitHub repository that's designed to help run it in GitHub Codespaces, with a very pleasing URL:

https://github.com/datasette/studio

Visit that page, click the green "Code" button and click "Create codespace on main" to launch a virtual machine running in GitHub's Azure environment, preconfigured to launch a private instance of Datasette as soon as the Codespace has started running.

You can then start using it directly - uploading CSVs or JSON data, or even set your own OpenAI key (using the "Manage secrets" menu item) to enable OpenAI features such as GPT enrichments and structured data extraction.

I'm still fleshing out the idea, but I really like this as a starting point for a completely free Datasette trial environment that's entirely hosted (and paid for) by Microsoft/GitHub!

More blog improvements

In addition to the redesign of the homepage - moving my linkblog and quotations out of the sidebar and into the main content, at least on desktop - I've made a couple of other tweaks.

I added optional descriptions to my tags, so now pages like /tags/datasette/ or /tags/sqliteutils/ can clarify themselves and link to the relevant projects.
I started displaying images in more places. I've been creating "social media card" images for many of my posts for a few years, to show up when those URLs are shared in places like Mastodon or Twitter or Discord or Slack. Those images now display in various places on my blog as well, including the homepage, search results and the tag pages. My annotatedtalks tag page looks a whole lot more interesting with accompanying presentation title slides.

Blog entries

Releases

datasette-faiss 0.2.1 - 2024-06-17
Maintain a FAISS index for specified Datasette tables
datasette-cluster-map 0.18.2 - 2024-06-13
Datasette plugin that shows a map for any data with latitude/longitude columns
datasette 0.64.7 - 2024-06-12
An open source multi-tool for exploring and publishing data
datasette-studio 0.1a4 - 2024-06-05
Datasette pre-configured with useful plugins. Experimental alpha.

TILs

Upgrade Postgres.app on macOS - 2024-06-16
Cloudflare redirect rules with dynamic expressions - 2024-05-29

Tags: blogging, github, projects, datasette, weeknotes, github-codespaces

datasette/studio

2024-03-10T03:03:42+00:00

datasette/studio

I’m trying a new way to make Datasette available for small personal data manipulation projects, using GitHub Codespaces.

This repository is designed to be opened directly in Codespaces—detailed instructions in the README.

When the container starts it installs the datasette-studio family of plugins—including CSV upload, some enrichments and a few other useful feature—then starts the server running and provides a big green button to click to access the server via GitHub’s port forwarding mechanism.

Tags: projects, datasette, github-codespaces

Exploring codespaces as temporary dev containers

2024-01-26T18:46:13+00:00

Exploring codespaces as temporary dev containers

DJ Adams shows how to use GitHub Codespaces without interacting with their web UI at all: you can run “gh codespace create --repo ...” to create a new instance, then SSH directly into it using “gh codespace ssh --codespace codespacename”.

This turns Codespaces into an extremely convenient way to spin up a scratch on-demand Linux container where you pay for just the time that the machine spends running.

Via lobste.rs

Tags: github, github-codespaces

Exploration de données avec Datasette

2023-05-27T00:36:52+00:00

Exploration de données avec Datasette

One of the great delights of open source development is seeing people run workshops on your project, even more so when they’re in a language other than English! Romain Clement presented this French workshop for the Python Grenoble meetup on 25th May 2023, using GitHub Codespaces as the environment. It’s pretty comprehensive, including a 300,000+ row example table which illustrates Datasette plugins such as datasette-cluster-map and datasette-leaflet-geojson.

Tags: tutorials, datasette, github-codespaces, leaflet

codespaces-jupyter

2023-04-14T22:38:21+00:00

codespaces-jupyter

This is really neat. Click “Use this template” -> “Open in a codespace” and you get a full in-browser VS Code interface where you can open existing notebook files (or create new ones) and start playing with them straight away.

Via @simon thread about online notebooks

Tags: github, python, jupyter, github-codespaces

Teaching News Apps with Codespaces

2023-03-23T00:39:33+00:00

Teaching News Apps with Codespaces

Derek Willis used GitHub Codespaces for the latest data journalism class he taught, and it eliminated the painful process of trying to get students on an assortment of Mac, Windows and Chromebook laptops all to a point where they could start working and learning together.

Via @derekwillis

Tags: data-journalism, derek-willis, github, teaching, github-codespaces

Using Datasette in GitHub Codespaces

2023-02-24T00:40:11+00:00

Using Datasette in GitHub Codespaces

A new Datasette tutorial showing how it can be run inside GitHub Codespaces—GitHub’s browser-based development environments—in order to explore and analyze data. I’ve been using Codespaces to run tutorials recently and it’s absolutely fantastic, because it puts every tutorial attendee on a level playing field with respect to their development environments.

Tags: github, tutorials, datasette, github-codespaces

Weeknotes: CDC vaccination history fixes, developing in GitHub Codespaces

2021-09-28T01:53:49+00:00

I spent the last week mostly surrounded by boxes: we're completing our move to the new place and life is mostly unpacking now. I did find some time to fix some issues with my CDC vaccination history Datasette instance though.

Fixing my CDC vaccination history site

I started tracking changes made to the CDC's COVID Data Tracker website back in Feburary. I created a git scraper repository for it as part of my five minute lightning talk on git scraping (notes and video) at this year's NICAR data journalism conference.

Since then it's been quietly ticking along, recording the latest data in a git repository that now has 335 commits.

In March I added a script to build the collected historic data into a SQLite database and publish it to Vercel using GitHub. That started breaking a few weeks ago, and it turnoud out that was because the database file had grown in size to the point where it was too large to deploy to Vercel (~100MB).

I got a bug report about this, so I took some time to move the deployment over to Google Cloud Run which doesn't have a documented size limit (though in my experience starts to creak once you go above about 2GB.)

I also started publishing the raw collected data directly as a CSV file, partly as an excuse to learn how to publish to Google Cloud Storage.

datasette-template-request

I released an extremely simple plugin this week called datasette-template-request - all it does is expose Datasette's request object in the context passed to custom templates, for people who want to update their custom page based on incoming request parameters.

More notable is how I built the plugin: this is the first plugin I've developed, tested and released entirely in my browser using the new GitHub Codespaces online development environment.

I created the new repo using my Datasette plugin template repository, opened it up in Codespaces, implemented the plugin and tests, tried it out using the port forwarding feature and then published it to PyPI using the publish.yml workflow.

Not having to even open a text editor on my laptop (let alone get a new Python development environment up and running) felt really good. I should turn this into a tutorial.

Releases this week

datasette-template-request: 0.1 - 2021-09-23
Expose the Datasette request object to custom templates
datasette-notebook: 0.1a1 - (2 releases total) - 2021-09-22
A markdown wiki and dashboarding system for Datasette
datasette-render-markdown: 2.0 - (8 releases total) - 2021-09-22
Datasette plugin for rendering Markdown
sqlite-utils: 3.17.1 - (87 releases total) - 2021-09-22
Python CLI utility and library for manipulating SQLite databases
twitter-to-sqlite: 0.22 - (28 releases total) - 2021-09-21
Save data from Twitter to a SQLite database

TIL this week

Tags: github, projects, weeknotes, covid19, git-scraping, github-codespaces

Datasette on Codespaces, sqlite-utils API reference documentation and other weeknotes

2021-08-14T04:57:12+00:00

This week I broke my streak of not sending out the Datasette newsletter, figured out how to use Sphinx for Python class documentation, worked out how to run Datasette on GitHub Codespaces, implemented Datasette column metadata and got tantalizingly close to a solution for an elusive Datasette feature.

API reference documentation for sqlite-utils using Sphinx

I've never been a big fan of Javadoc-style API documentation: I usually find that documentation structured around classes and methods fails to show me how to actually use those classes to solve real-world problems. I've tended to avoid it for my own projects.

My sqlite-utils Python library has a ton of functionality, but it mainly boils down to two classes: Database and Table. Since it already has pretty comprehesive narrative documentation explaining the different problems it can solve, I decided to try experimenting with the Sphinx autodoc module to produce some classic API reference documentation for it:

Since autodoc works from docstrings, this was also a great excuse to add more comprehensive docstrings and type hints to the library. This helps tools like Jupyter notebooks and VS Code display more useful inline help.

This proved to be time well spent! Here's what sqlite-utils looks like in VS Code now:

Running mypy against the type hints also helped me identify and fix a couple of obscure edge-case bugs in the existing methods, detailed in the 3.15.1 release notes. It's taken me a few years but I'm finally starting to come round to Python's optional typing as being worth the additional effort!

Figuring out how to use autodoc in Sphinx, and then how to get the documentation to build correctly on Read The Docs took some effort. I wrote up what I learned in this TIL.

Datasette on GitHub Codespaces

GitHub released their new Codespaces online development environments to general availability this week and I'm really excited about it. I ran a team at Eventbrite for a while resonsible for development environment tooling and it really was shocking how much time and money was lost to broken local development environments, even with a significant amount of engineering effort applied to the problem.

Codespaces promises a fresh, working development environment on-demand any time you need it. That's a very exciting premise! Their detailed write-up of how they convinced GitHub's own internal engineers to move to it is full of intriguing details - getting an existing application working with it is no small feat, but the pay-off looks very promising indeed.

So... I decided to try and get Datasette running on it. It works really well!

You can run Datasette in any Codespace environment using the following steps:

Open the terminal. Three-bar-menu-icon, View, Terminal does the trick.
In the terminal run pip install datasette datasette-x-forwarded-host (more on this in a moment).
Run datasette - Codespaces will automatically setup port forwarding and give you a link to "Open in Browser" - click the link and you're done!

You can pip install sqlite-utils and then use sqlite-utils insert to create SQLite databases to use with Datasette.

There was one catch: the first time I ran Datasette, clicking on any of the internal links within the web application took me to http://localhost/ pages that broke with a 404.

It turns out the Codespaces proxy sends a host: localhost header - which Datasette then uses to incorrectly construct internal URLs.

So I wrote a tiny ASGI plugin, datasette-x-forwarded-host, which takes the incoming X-Forwarded-Host provided by Codespaces and uses that as the Host header within Datasette itself. After that everything worked fine.

sqlite-utils insert --flatten

Early this week I finally figured out Cloud Run logging. It's actually really good! In doing so, I worked out a convoluted recipe for tailing the JSON logs locally and piping them into a SQLite database so that I could analyze them with Datasette.

Part of the reason it was convoluted is that Cloud Run logs feature nested JSON, but sqlite-utils insert only works against an array of flat JSON objects. I had to use this jq monstrosity to flatten the nested JSON into key/value pairs.

Since I've had to solve this problem a few times now I decided to improve sqlite-utils to have it do the work instead. You can now use the new --flatten option like so:

sqlite-utils insert logs.db logs log.json --flatten

To create a schema that flattens nested objects into a topkey_nextkey structure like so:

CREATE TABLE [logs] (
   [httpRequest_latency] TEXT,
   [httpRequest_requestMethod] TEXT,
   [httpRequest_requestSize] TEXT,
   [httpRequest_status] INTEGER,
   [insertId] TEXT,
   [labels_service] TEXT
);

Full documentation for --flatten.

Datasette column metadata

I've been wanting to add this for a while: Datasette's main branch now includes an implementation of column descriptions metadata for Datasette tables. This is best illustrated by a screenshot (of this live demo):

You can add the following to metadata.yml (or .json) to specify descriptions for the columns of a given table:

databases:
  fixtures:
    roadside_attractions:
      columns:
        name: The name of the attraction
        address: The street address for the attraction

Column descriptions will be shown in a <dl> at the top of the page, and will also be added to the menu that appears when you click on the cog icon at the top of a column.

Getting closer to query column metadata, too

Datasette lets you execute arbitrary SQL queries, like this one:

select
  roadside_attractions.name,
  roadside_attractions.address,
  attraction_characteristic.name
from
  roadside_attraction_characteristics
  join roadside_attractions on roadside_attractions.pk = roadside_attraction_characteristics.attraction_id
  join attraction_characteristic on attraction_characteristic.pk = roadside_attraction_characteristics.characteristic_id

You can try that here. It returns the following:

name	address	name
The Mystery Spot	465 Mystery Spot Road, Santa Cruz, CA 95065	Paranormal
Winchester Mystery House	525 South Winchester Boulevard, San Jose, CA 95128	Paranormal
Bigfoot Discovery Museum	5497 Highway 9, Felton, CA 95018	Paranormal
Burlingame Museum of PEZ Memorabilia	214 California Drive, Burlingame, CA 94010	Museum
Bigfoot Discovery Museum	5497 Highway 9, Felton, CA 95018	Museum

The columns it returns have names... but I've long wanted to do more with these results. If I could derive which source columns each of those output columns were, there are a bunch of interesting things I could do, most notably:

If the output column is a known foreign key relationship, I could turn it into a hyperlink (as seen on this table page)
If the original table column has the new column metadata, I could display that as additional documentation

The challenge is: given an abitrary SQL query, how can I figure out what the resulting columns are going to be and how to tie those back to the original tables?

Thanks to a hint from the SQLite forum I'm getting tantalizingly close to a solution.

The trick is to horribly abuse SQLite's explain output. Here's what it looks like for the example query above:

addr	opcode	p1	p2	p3	p4	p5
0	Init	0	15	0		0
1	OpenRead	0	47	0	2	0
2	OpenRead	1	45	0	3	0
3	OpenRead	2	46	0	2	0
4	Rewind	0	14	0		0
5	Column	0	0	1		0
6	SeekRowid	1	13	1		0
7	Column	0	1	2		0
8	SeekRowid	2	13	2		0
9	Column	1	1	3		0
10	Column	1	2	4		0
11	Column	2	1	5		0
12	ResultRow	3	3	0		0
13	Next	0	5	0		1
14	Halt	0	0	0		0
15	Transaction	0	0	35	0	1
16	Goto	0	1	0		0

The magic is on line 12: ResultRow 3 3 means "return a result that spans three columns, starting at register 3" - so that's register 3, 4 and 5. Those three registers are populated by the Column operations on line 9, 10 and 11 (the register they write into is in the p3 column). Each Column operation specifies the table (as p1) and the column index within that table (p2). And those table references map back to the OpenRead lines at the start, where p1 is that table register (referered to by Column) and p1 is the root page of the table within the schema.

Running select rootpage, name from sqlite_master where rootpage in (45, 46, 47) produces the following:

rootpage	name
45	roadside_attractions
46	attraction_characteristic
47	roadside_attraction_characteristics

Tie all of this together, and it may be possible to use explain to derive the original tables and columns for each of the outputs of an arbitrary query!

I was almost ready to declare victory, until I tried running it against a query with an order by column at the end... and the results no longer matched up.

You can follow my ongoing investigation here - the short version is that I think I'm going to have to learn to decode a whole bunch more opcodes before I can get this to work.

This is also a very risk way of attacking this problem. The SQLite documentation for the bytecode engine includes the following warning:

This document describes SQLite internals. The information provided here is not needed for routine application development using SQLite. This document is intended for people who want to delve more deeply into the internal operation of SQLite.

The bytecode engine is not an API of SQLite. Details about the bytecode engine change from one release of SQLite to the next. Applications that use SQLite should not depend on any of the details found in this document.

So it's pretty clear that this is a highly unsupported way of working with SQLite!

I'm still tempted to try it though. This feature is very much a nice-to-have: if it breaks and the additional column context stops displaying it's not a critical bug - and hopefully I'll be able to ship a Datasette update that takes into account those breaking SQLite changes relatively shortly afterwards.

If I can find another, more supported way to solve this I'll jump on it!

In the meantime, I did use this technque to solve a simpler problem. Datasette extracts :named parameters from arbitrary SQL queries and turns them into form fields - but since it uses a simple regular expression for this it could be confused by things like a literal 00:04:05 time string contained in a SQL query.

The explain output for that query includes the following:

addr	opcode	p1	p2	p3	p4	p5	comment
...	...	...	...	...	...	...	...
27	Variable	1	12	0	:text	0

So I wrote some code which uses explain to extract just the p4 operands from Variable columns and treats those as the extracted parameters! This feels a lot safer than the more complex ResultRow/Column logic - and it also falls back to the regular expression if it runs into any SQL errors. More in the issue.

TIL this week

Releases this week

datasette-x-forwarded-host: 0.1 - 2021-08-12
Treat the X-Forwarded-Host header as the Host header
sqlite-utils: 3.15.1 - (84 releases total) - 2021-08-10
Python CLI utility and library for manipulating SQLite databases
datasette-query-links: 0.1.2 - (3 releases total) - 2021-08-09
Turn SELECT queries returned by a query into links to execute them
datasette: 0.59a1 - (96 releases total) - 2021-08-09
An open source multi-tool for exploring and publishing data
datasette-pyinstrument: 0.1 - 2021-08-08
Use pyinstrument to analyze Datasette page performance

Tags: documentation, github, sql, sqlite, datasette, weeknotes, sqlite-utils, mypy, github-codespaces

GitHub’s Engineering Team has moved to Codespaces

2021-08-11T16:53:06+00:00

GitHub’s Engineering Team has moved to Codespaces

My absolute dream development environment is one where I can spin up a new, working development environment in seconds—to try something new on a branch, or because I broke something and don’t want to spend time figuring out how to fix it. This article from GitHub explains how they got there: from a half-day setup to a 45 minute bootstrap in a codespace, then to five minutes through shallow cloning and a nightly pre-built Docker image and finally to 10 seconds be setting up “pools of codespaces, fully cloned and bootstrapped, waiting to be connected with a developer who wants to get to work”.

Tags: github, docker, github-codespaces