Simon Willison's Weblog: andrew-godwin

Understanding a Protocol

2022-12-06T00:50:13+00:00

Andrew’s latest notes on how ActivityPub and Mastodon work under the hood, based on his extensive development work building out Takahē.

Tags: andrew-godwin, mastodon, activitypub

Quoting Andrew Godwin

2022-11-19T16:02:57+00:00

... it [ActivityPub] is crucially good enough. Perfect is the enemy of good, and in ActivityPub we have a protocol that has flaws but, crucially, that works, and has a standard we can all mostly agree on how to implement - and eventually, I hope, agree on how to improve.

— Andrew Godwin

Tags: andrew-godwin, mastodon, activitypub

Django: Added support for asynchronous views and middleware

2020-03-19T03:43:40+00:00

Django: Added support for asynchronous views and middleware

An enormously consequential feature just landed in Django, and is set to ship as part of Django 3.1 in August. Asynchronous views will allow Django applications to define views using “async def myview(request)”—taking full advantage of Python’s growing asyncio ecosystem and providing enormous performance improvements for Django sites that do things like hitting APIs over HTTP. Andrew has been puzzling over this for ages and it’s really exciting to see it land in a form that should be usable in a stable Django release in just a few months.

Via @andrewgodwin

Tags: andrew-godwin, async, django

Porting Datasette to ASGI, and Turtles all the way down

2019-06-23T21:39:00+00:00

This evening I finally closed a Datasette issue that I opened more than 13 months ago: #272: Port Datasette to ASGI. A few notes on why this is such an important step for the project.

ASGI is the Asynchronous Server Gateway Interface standard. It’s been evolving steadily over the past few years under the guidance of Andrew Godwin. It’s intended as an asynchronous replacement for the venerable WSGI.

Turtles all the way down

Ten years ago at EuroDjangoCon 2009 in Prague I gave a talk entitled Django Heresies. After discussing some of the design decisions in Django that I didn’t think had aged well, I spent the last part of the talk talking about Turtles all the way down. I wrote that idea up here on my blog (see also these slides).

The key idea was that Django would be more interesting if the core Django contract - a function that takes a request and returns a response - was extended to more places in the framework. The top level site, the reusable applications, middleware and URL routing could all share that same contract. Everything could be composed from the same raw building blocks.

I’m excited about ASGI because it absolutely fits the turtles all the way down model.

The ASGI contract is an asynchronous function that takes three arguments:

async def application(scope, receive, send):
    ...

scope is a serializable dictionary providing the context for the current connection. receive is an awaitable which can be used to recieve incoming messages. send is an awaitable that can be used to send replies.

It’s a pretty low-level set of primitives (and less obvious than a simple request/response) - and that’s because ASGI is about more than just the standard HTTP request/response cycle. This contract works for HTTP, WebSockets and potentially any other protocol that needs to asynchronously send and receive data.

It’s an extremely elegant piece of protocol design, informed by Andrew’s experience with Django Channels, SOA protocols (we are co-workers at Eventbrite where we’ve both been heavily involved in Eventbrite’s SOA mechanism) and Andrew’s extensive conversations with other maintainers in the Python web community.

The ASGI protocol really is turtles all the way down - it’s a simple, well defined contract which can be composed together to implement all kinds of interesting web architectural patterns.

My asgi-cors library was my first attempt at building an ASGI turtle. The implementation is a simple Python decorator which, when applied to another ASGI callable, adds HTTP CORS headers based on the parameters you pass to the decorator. The library has zero installation dependencies (it has test dependencies on pytest and friends) and can be used on any HTTP ASGI project.

Building asgi-cors completely sold me on ASGI as the turtle pattern I had been desiring for over a decade!

Datasette plugins and ASGI

Which brings me to Datasette.

One of the most promising components of Datasette is its plugin mechanism. Based on pluggy (extracted from pytest), Datasette Plugins allow new features to be added to Datasette without needing to change the underlying code. This means new features can be built, packaged and shipped entirely independently of the core project. A list of currently available plugins can be found here.

WordPress is very solid blogging engine. Add in the plugin ecosystem around it and it can be used to build literally any CMS you can possibly imagine.

My dream for Datasette is to apply the same model: I want a strong core for publishing and exploring data that’s enhanced by plugins to solve a huge array of data analysis, visualization and API-backed problems.

Datasette has a range of plugin hooks already, but I’ve so far held back on implementing the most useful class of hooks: hooks that allow developers to add entirely new URL routes exposing completely custom functionality.

The reason I held back is that I wanted to be confident that the contract I was offering was something I would continue to support moving forward. A plugin system isn’t much good if the core implementation keeps on changing in backwards-incompatible ways.

ASGI is the exact contract I’ve been waiting for. It’s not quite ready yet, but you can follow #520: prepare_asgi plugin hook (thoughts and suggestions welcome!) to be the first to hear about this hook when it lands. I’m planning to use it to make my asgi-cors library available as a plugin, after which I’m excited to start exploring the idea of bringing authentication plugins to Datasette (and to the wider ASGI world in general).

I’m hoping that many Datasette ASGI plugins will exist in a form that allows them to be used by other ASGI applications as well.

I also plan to use ASGI to make components of Datasette itself available to other ASGI applications. If you just want a single instance of Datasette’s table view to be embedded somewhere in your URL configuration you should be able to do that by routing traffic directly to the ASGI-compatible view class.

I’m really excited about exploring the intersection of ASGI turtles-all-the-way-down and pluggy’s powerful mechanism for gluing components together. Both WSGI and Django’s reusable apps have attempted to create a reusable ecosystem in the past, to limited levels of success. Let’s see if ASGI can finally make the turtle dream come true.

Quoting Andrew Godwin

2019-05-10T02:00:03+00:00

... the overall conclusion I reach is that we have so much to gain from making Django async-capable that it is worth the large amount of work it will take. I also believe, crucially, that we can undertake this change in an iterative, community-driven way that does not rely solely on one or two long-time contributors burning themselves out.

— Andrew Godwin

Tags: andrew-godwin, async, django

How I moderated the State of Django panel at DjangoCon US.

2018-10-22T19:48:53+00:00

On Wednesday last week I moderated the State of Django panel as the closing session for DjangoCon US 2018.

I think it went well, so I’m writing some notes on exactly how we did it. In my experience it’s worth doing this for things like public speaking: in six months time I might moderate another panel and I’ll be desperately trying to remember what went right last time.

Panels are hard. Bad panels are way too common, to the point that some good conferences actively avoid having panels at all.

In my opinion, a good panel has a number of important attributes:

The panel needs a coherent theme. It shouldn’t just be several independent speakers that happen to be sharing the same time slot.
Panels need to be balanced. Having just one or two of the speakers monopolize the conversation is bad for the audience and bad for the panelists themselves.
The moderator is there to facilitate the conversation, not to be the center of attention. I love public speaking so I feel the need to be particularly cautious here.
Panelists need to have diverse perspectives on the topics under discussion. A panel where everyone agrees with each other and makes the same points is a very boring panel indeed.

I originally pitched the panel to the DjangoCon US organizing committee as a “State of Django” conversation where core maintainers would talk about the current state of the project.

They countered with a much better idea: a panel that encompassed both the state of the Django framework and the community and ecosystem that it exists within. Since DjangoCon is primarily about bringing that community together this was a much better fit for the conference, and would make for a much more interesting and relevant discussion.

I worked together with the conference organizers to find our panelists. Nicholle James in particular was the driving force behind assembling the panelists and ensuring everything was in place for the panel to succeed.

We ended up with a panel representing a comprehensive cross-section of the organizations that make the Django community work:

Andrew Godwin, representing Django Core
Anna Makarudze, representing the DSF, Django Girls and Python Africa
Frank Wiles, President of the DSF
Jeff Triplett, President of DEFNA, member of the Board of Directors for the PSF
Josue Balandrano Coronel, representing DEFNA
Katherine Michel, representing DEFNA and DjangoCon US Website Chair
Kojo Idrissa, DEFNA North American Ambassador
Rachell Calhoun, representing Django Girls and PyLadies

As it was the closing session for the conference, I wanted the panel to both celebrate the progress of the community and project and to explore what we need to do next: what should we be working on to improve Django and it’s community in the future?

I had some initial thoughts on topics, but since the panel was scheduled for the last session of the conference I decided to spend the conference itself firming up the topics that would be discussed. This was a really good call: we got to create an agenda for the panel that was almost entirely informed by the other conference sessions combined with hot topics from the halfway track. We also asked conference attendees for their suggestions via an online form, and used those suggestions to further inform the topics that were discussed.

I made sure to have a 10-15 minute conversation one-on-one with each of the panelists during the conference. We then got together for an hour at lunch before the panel to sync up with the topics and themes we would be discussing.

Topic and themes

Our pre-panel conversations highlighted a powerful theme for the panel itself, which I ended up summarizing as “What can the Django project learn from the Django community?” This formed the framework for the other themes of the panel.

The themes themselves were:

Diversity and education - inspired by the work of Django Girls
Events and international focus, lead by DEFNA
Django governance: In particular James Bennett’s proposal to split up core
Big feature ideas for Django (guided by Andrew Godwin’s proposed Django async roadmap)
Money: who has it, and who needs it - understanding the role of the DSF.

One of the hardest parts for me was figuring out the order in which we would tackle these themes. I ended up settling on the above order about half an hour before the panel started.

Opening and closing

With eight panelists, ensuring that introductions didn’t drag on too long was particularly important. I asked each panelist to introduce themselves with a couple of sentences that highlighted the organizations they were affiliated with that were relevant to the panel. For our chosen group of panelists this was definitely the right framing.

I then asked each panelist to be prepared to close the panel with a call to action: something an audience member could actively do that would support the community going forward. This worked really well: it provided a great, actionable note to end both the panel and the conference.

Preparing and running the panel

We used our panel lunch together to check that no one would have calls to action that overlapped too much, and to provide a rough indication of who had things to say about each topic we planned to discuss.

This turned out to be essential: I’ve been on smaller panels where the panelists have been able to riff easily on each other’s points, but with eight panelists it turned out not everyone could even see each other, so as panel moderator it fell on me to direct questions to individuals and then prompt others for follow-ups. Thankfully the panel lunch combined with the one-to-one conversations gave me the information I needed for this.

I had written down a selection of questions for each of the themes. Having a selection turned out to be crucial: a few times the panelists talked about material that I had planned to cover in a later section, so I had to adapt as we went along. In the future I’ll spend more time on this: these written ideas were a crucial component in keeping the panel flowing in the right direction.

With everything in place the panel itself was a case of concentrating on what everyone was saying and using the selection of the next questions (plus careful ad-libbing) to guide the conversation along the preselected themes. I also tried to keep mental track of who had done the most speaking so I could ensure the conversation stayed as balanced as possible by inviting other panelists into the conversation.

The video of the panel should be out in around three weeks time, at which point you can evaluate for yourself if we managed to do a good job of it. I’m really happy with the feedback we got after the panel, and I plan to use a similar process for panels I’m involved with in the future.

Tags: andrew-godwin, django, djangocon, frank-wiles, panels, speaking, my-talks, jeff-triplett

Andrew Godwin's www-router Docker container

2018-02-21T05:04:02+00:00

Andrew Godwin's www-router Docker container

Really clever Docker trick: a container that runs Nginx and uses it to route traffic to other containers based on the hostname—but the hostnames to be routed are configured using environment variables which the run-nginx.py CMD script uses to dynamically construct an nginx config when the container starts.

Via @andrewgodwin on Twitter

Tags: andrew-godwin, nginx, docker

Python & Async Simplified

2018-02-20T00:30:03+00:00

Python & Async Simplified

Andrew Godwin: “Python’s async framework is actually relatively simple when you treat it at face value, but a lot of tutorials and documentation discuss it in minute implementation detail, so I wanted to make a higher-level overview that deliberately ignores some of the small facts and focuses on the practicalities of writing projects that mix both kinds of code.” ‪This is really useful: clearly explains the two separate worlds of Python (sync and async functions) and describes Andrew’s clever sync_to_async and async_to_sync decorators as well.‬

Tags: andrew-godwin, async, python, python3

asgiref: AsyncToSync and SyncToAsync

2018-02-02T19:06:48+00:00

asgiref: AsyncToSync and SyncToAsync

Andrew’s classes in asgiref that can turn a synchronous callable into an awaitable (that runs in a thread pool) or an awaitable callable into a synchronous callable, using Python 3 futures and asyncio.

Via Andrew Godwin

Tags: andrew-godwin, async, python3

Channels 2.0

2018-02-02T18:19:54+00:00

Channels 2.0

Andrew just shipped Channels 2.0—a major rewrite and redesign of the Channels project he started back in 2014. Channels brings async to Django, providing a logical, standardized way of supporting things like WebSockets and asynchronous execution on top of a Django application. Previously it required you to run a separate Twisted server and redis/RabbitMQ queue, but thanks to Python 3 async everything can now be deployed as a single process. And the new ASGI spec means its turtles all the way down! Everything from URL routing to view functions to middleware can be composed together using the same ASGI interface.

Tags: andrew-godwin, async, django, python, python3, websockets

The First Few Weeks - ep.io

2011-01-13T04:25:00+00:00

The First Few Weeks - ep.io

Another take on managed Python Django/WSGI hosting, from Andrew Godwin and Ben Firshman.

Tags: andrew-godwin, ben-firshman, django, hosting, python, wsgi, recovered

On Django And Migrations

2010-06-02T16:27:00+00:00

On Django And Migrations

South author Andrew Godwin on the plans for migrations in Django. His excellent South migration library will be split in to two parts—one handling database abstraction, dependency resolution and history tracking and the other providing autodetection and the South user interface. The former will go in to Django proper, encouraging other migration libraries to share the same core abstractions.

Tags: andrew-godwin, django, migrations, orm, south, recovered

Announcing Heechee

2009-09-11T02:16:56+00:00

Announcing Heechee

“Heechee is a transparent mercurial-as-subversion gateway”—you can use it to allow subversion clients to check out a mercurial repository, meaning svn:externals can work against projects hosted by mercurial. It’s very young code but I’ve already seen it out-perform regular subversion for checkout speed.

Tags: andrew-godwin, heechee, mercurial, subversion

South's Design

2009-05-13T12:30:45+00:00

South's Design

Andrew Godwin explains why South resorts to parsing your models.py file in order to construct information about for creating automatic migrations.

Tags: andrew-godwin, django, models, orm, parsing, python, south

Southerly Breezes

2009-03-15T13:17:20+00:00

Southerly Breezes

Andrew Godwin is slowly assimilating the best ideas from other Django migration systems in to South—the latest additions include ORM Freezing from Migratory and automatic change detection. Exciting stuff.

Tags: andrew-godwin, databases, django, migrations, orm, south

South

2008-08-08T11:42:00+00:00

South

A brand new light-weight Django migrations tool from Andrew Godwin. On first glance, this is spookily similar to the system we’ve been putting together at GCap.

Tags: andrew-godwin, django, gcap, migrations, south

LastGraph 3

2008-05-25T14:05:20+00:00

LastGraph 3

Andrew Godwin’s last.fm profile visualisation tool, now in its third incarnation.

Tags: andrew-godwin, lastfm, lastgraph, lastgraph3

Graphication

2008-03-30T19:05:36+00:00

Graphication

Andrew Godwin’s Python graphing library, based on Cairo. Responsible for the very handsome graphs on The Carbon Account.

Tags: andrew-godwin, cairo, graphication, graphing, python, thecarbonaccount

LastGraph. Now Available.

2007-10-15T22:02:03+00:00

LastGraph. Now Available.

Andrew Godwin has relaunched his LastGraph Last.fm graphing application. The new version is built on Django and S3 and uses Andrew’s Graphication graphing library based on Cairo.

Tags: andrew-godwin, cairo, django, graphication, graphing, lastfm, lastgraph, python, s3

Simon Willison's Weblog: andrew-godwin

Understanding a Protocol

Quoting Andrew Godwin

Django: Added support for asynchronous views and middleware

Porting Datasette to ASGI, and Turtles all the way down

Turtles all the way down

Datasette plugins and ASGI

Further reading

Quoting Andrew Godwin

How I moderated the State of Django panel at DjangoCon US.

Topic and themes

Opening and closing

Preparing and running the panel

Andrew Godwin's www-router Docker container

Python & Async Simplified

asgiref: AsyncToSync and SyncToAsync

Channels 2.0

The First Few Weeks - ep.io

On Django And Migrations

Announcing Heechee

South's Design

Southerly Breezes

South

LastGraph 3

Graphication

LastGraph. Now Available.