Simon Willison's Weblog: middleware

Single sign-on against GitHub using ASGI middleware

2019-07-14T01:18:56+00:00

I released Datasette 0.29 last weekend, the first version of Datasette to be built on top of ASGI (discussed previously in Porting Datasette to ASGI, and Turtles all the way down).

This also marked the introduction of the new asgi_wrapper plugin hook, which allows plugins to wrap the entire Datasette application in their own piece of ASGI middleware.

To celebrate this new capability, I also released two new plugins: datasette-cors, which provides fine-grained control over CORS headers (using my asgi-cors library from a few months ago) and datasette-auth-github, the first of hopefully many authentication plugins for Datasette.

datasette-auth-github

The new plugin is best illustrated with a demo.

Visit https://datasette-auth-demo.now.sh/ and you will be redirected to GitHub and asked to approve access to your account (just your e-mail address, not repository access).

Agree, and you’ll be redirected back to the demo with a new element in the Datasette header: your GitHub username, plus a “log out” link in the navigation bar at the top of the screen.

Controlling who can access

The default behaviour of the plugin is to allow in anyone with a GitHub account. Since the primary use-case for the plugin (at least for the moment) is restricting access to view data to a trusted subset of people, the plugin lets you configure who is allowed to view your data in three different ways:

You can restrict access to a specific list of GitHub accounts, using the allow_users configuration option.
You can restrict access to members of one or more GitHub organizations, with allow_orgs.
You can restrict access to members of specific teams within an organization, using allow_teams.

Datasette inherits quite a sophisticated user management system from GitHub, with very little effort required from the plugin. The user_is_allowed() method that implements all three of the above options against the GitHub API in just 40 lines of code.

These options can be set using the "plugins" section of the Datasette metadata.json configuration file. Here’s an example:

{
  "plugins": {
    "datasette-auth-github": {
      "client_id": {"$env": "GITHUB_CLIENT_ID"},
      "client_secret": {"$env": "GITHUB_CLIENT_SECRET"},
      "allow_users": ["simonw"]
    }
  }
}

This also illustrates a new Datasette feature: the ability to set secret plugin configuration values. {"$env": "GITHUB_CLIENT_SECRET"} means "read this configuration option from the environment variable GITHUB_CLIENT_SECRET".

Automatic log in

Like many OAuth providers, GitHub only asks the user for their approval the first time they log into a given app. Any subsequent times they are redirected to GitHub it will skip the permission screen and redirect them right back again with a token.

This means we can implement automatic log in: any time a visitor arrives who does not have a cookie we can bounce them directly to GitHub, and if they have already consented they will be logged in instantly.

This is a great user-experience - provided the user is logged into GitHub they will be treated as if they are logged into your application - but it does come with a downside: what if the user clicks the “log out” link?

For the moment I’ve implemented this using another cookie: if the user clicks “log out”, I set an asgi_auth_logout cookie marking the user as having explicitly logged out. While they have that cookie they won’t be logged in automatically, instead having to click an explicit link. See issue 41 for thoughts on how this could be further improved.

One pleasant side-effect of all of this is that datasette-auth-github doesn’t need to persist the users GitHub access_token anywhere - it uses it during initil authentication check for any required organizations or teams, but then it deliberately forgets the token entirely.

OAuth access tokens are like passwords, so the most resonsible thing for a piece of softare to do with them is avoid storing them anywhere at all unless they are explicitly needed.

What happens when a user leaves an organization?

When building against a single sign-in provider, consideration needs to be given to offboarding: when a user is removed from a team or organization they should also lose access to their SSO applications.

This is difficult when an application sets its own authentication cookies, like datasette-auth-github does.

One solution would be to make an API call on every request to the application, to verify that the user should still have access. This would slow everything down and is likely to blow through rate limits as well, so we need a more efficient solution.

I ended up solving this with two mechanisms. Since we have automatic log in, our cookies don’t actually need to last very long - so by default the signed cookies set by the plugin last for just one hour. When a user’s cookie has expired they will be redirected back through GitHub - they probably won’t even notice the redirect, and their permissions will be re-verified as part of that flow.

But what if you need to invalidate those cookies instantly?

To cover that case, I’ve incorporated an optional cookie_version configuration option into the signatures on the cookies. If you need to invalidate every signed cookie that is out there - to lock out a compromised GitHub account owner for example - you can do so by changing the cookie_version configuration option and restarting (or re-deploying) Datasette.

These options are all described in detail in the project README.

Integration with datasette publish

The datasette publish command-line tool lets users instantly publish a SQLite database to the internet, using Heroku, Cloud Run or Zeit Now v1. I’ve added suppor for setting secret plugin configuration directly to that tool, which means you can publish an authentication-protected SQLite database to the internet with a shell one-liner, using --install=datasette-auth-github to install the plugin and --plugin-secret to configure it:

$ datasette publish cloudrun fixtures.db \
   --install=datasette-auth-github \
   --name datasette-auth-protected \
   --service datasette-auth-protected \
   --plugin-secret datasette-auth-github allow_users simonw \
   --plugin-secret datasette-auth-github client_id 85f6224cb2a44bbad3fa \
   --plugin-secret datasette-auth-github client_secret ...

This creates a Cloud Run instance which only allows GitHub user simonw to log in. You could instead use --plugin-secret datasette-auth-github allow_orgs my-org to allow any users from a specific GitHub organization.

Note that Cloud Run does not yet give you full control over the URL that will be assigned to your deployment. In this case it gave me https://datasette-auth-protected-j7hipcg4aq-uc.a.run.app - which works fine, but I needed to update my GitHub OAuth application’s callback URL manually to https://datasette-auth-protected-j7hipcg4aq-uc.a.run.app/-/auth-callback after deploying the application in order to get the authentication flow to work correctly.

Add GitHub authentication to any ASGI application!

datasette-auth-github isn’t just for Datasette: I deliberately wrote the plugin as ASGI middleware first, with only a very thin layer of extra code to turn it into an installable plugin.

This means that if you are building any other kind of ASGI app (or using an ASGI-compatible framework such as Starlette or Sanic) you can wrap your application directly with the middleware and get the same authentication behaviour as when the plugin is added to Datasette!

Here’s what that looks like:

from datasette_auth_github import GitHubAuth
from starlette.applications import Starlette
from starlette.responses import HTMLResponse
import uvicorn

app = Starlette(debug=True)


@app.route("/")
async def homepage(request):
    return HTMLResponse("Hello, {}".format(
        repr(request.scope["auth"])
    ))


authenticated_app = GitHubAuth(
    app,
    client_id="986f5d837b45e32ee6dd",
    client_secret="...",
    require_auth=True,
    allow_users=["simonw"],
)

if __name__ == "__main__":
    uvicorn.run(authenticated_app, host="0.0.0.0", port=8000)

The middleware adds a scope["auth"] key describing the logged in user, which is then passed through to your application. More on this in the README.

Your security reviews needed!

Since datasette-auth-github adds authentication to Datasette, it is an extremely security-sensitive piece of code. So far I’m the only person who has looked at it: before I start widely recommending it to people I’d really like to get some more eyes on it to check for any potential security problems.

I’ve opened issue #44 encouraging security-minded developers to have a dig through the code and see if there’s anything that can be tightened up or any potential vulnerabilities that need to be addressed. Please get involved!

It’s a pretty small codebase, but here are some areas you might want to inspect:

At a high level: is the way I’m verifying the user through the GitHub API and then storing their identity in a signed cookie the right way to go?
The cookie signing secret is derived from the GitHub OAuth application’s client_id and client_secret (because that secret is already meant to be a secret), combined with the cookie_version option described above - implementation here. Since this is a derived secret I’m using pbkdf2_hmac with 100,000 iterations. This is by far the most cryptographically interesting part of the code, and could definitely do with some second opinions.
The code used to sign and verify cookies is based on Django’s (thoroughly reviewed) implementation, but could benefit from a sanity check.
I wanted this library to work on Glitch, which currently only provides Python 3.5.2. Python’s asyncio HTTP librarys such as http3 and aiohttp both require more modern Pythons, so I ended up rolling my own very simple async HTTP function which uses urllib.request inside a loop.run_in_executor thread pool. Is that approach sound? Rolling my own HTTP client in this way feels a little hairy.

This has been a really fun project so far, and I’m very excited about the potential for authenticated Datasette moving forward - not to mention the possibilites unlocked by an ASGI middleware ecosystem with strong support for wrapping any application in an authentication layer.

Tags: github, middleware, projects, security, datasette, asgi

Debugging Django in Production Revisited

2009-09-07T05:21:04+00:00

Debugging Django in Production Revisited

Eric Holscher expands his show-technical-errors-to-superusers middleware to only show them to users in the group named “Technical Errors”.

Tags: debugging, django, eric-holscher, middleware, python

Finding and fixing memory leaks in Python

2009-04-22T12:16:32+00:00

Finding and fixing memory leaks in Python

Using Dozer, a clever piece of WSGI middleware which displays sparklines of Python object counts and allows you to introspect them, using the gc module under the hood.

Tags: amir-salihefendic, debugging, memory, memoryleaks, middleware, python, wsgi

Integrating Facebook Connect with Django in 15 minutes

2008-12-17T13:18:38+00:00

Integrating Facebook Connect with Django in 15 minutes

Django authentication middleware that calls the Facebook REST API using a cookie set by Facebook Connect and checks if that person is your Facebook friend. Despite most of the magic happening on the server you still need Facebook’s JavaScript to set that cookie in the first place.

Tags: cookies, django, facebook, facebookconnect, javascript, middleware

csrf_protect.php

2008-09-24T14:52:36+00:00

csrf_protect.php

A PHP class for applying CSRF protection to existing PHP applications, using output buffering to rewrite any POST forms on a page. Heavily inspired by Django’s CSRF middleware. Tell me if you spot any bugs!

Tags: code, csrf, middleware, outputbuffering, php, projects, security

minidetector

2008-08-15T08:21:59+00:00

minidetector

Neat piece of Django middleware that adds a “mobile = True” attribute to the request object if the request’s user-agent matches a list of strings of known low-power browsers in mobiles, PDAs or game consoles.

Tags: django, middleware, minidetector, mobile, python, user-agents

Super User Conditional Page Exception Reporting

2008-07-31T21:06:30+00:00

Super User Conditional Page Exception Reporting

The name is almost as long as the code snippet: this serves Django’s debug page to logged in super-users, falling back to the default 500 template for everyone else.

Tags: debugging, django, exceptions, middleware, python

mod_rpaf for Apache

2008-06-24T17:02:55+00:00

mod_rpaf for Apache

A more secure alternative to Django’s equivalent middleware: sets the REMOTE_ADDR of incoming requests from whitelisted load balancers to the X-Forwarded-For header, without any risk that if the load balancers are missing attackers could abuse it to spoof their IP addresses.

Tags: apache, django, http, load-balancing, middleware, modrpaf, rpaf, security, xforwardedfor

DebugFooter middleware with Pygments sql syntax highlighting

2008-06-14T10:04:08+00:00

DebugFooter middleware with Pygments sql syntax highlighting

Andreas Marr has enhanced my Django DebugFooter middleware with proper syntax highlighting for the logged SQL.

Tags: andreas-marr, debugfooter, django, middleware, python

Debugging Django

2008-05-22T00:35:40+00:00

I gave a talk on Debugging Django applications at Monday's inaugural meeting of DJUGL, the London Django Users Group. I wanted to talk about something that wasn't particularly well documented elsewhere, so I pitched the talk as "Bug Driven Development" - what happens when Test Driven Development goes the way of this unfortunate pony.

The slides are up on SlideShare, but don't provide quite enough context so I'm going to cover the tips in full here.

Making the most of the error page

Django's default error page is great - it provides a detailed traceback with local variables, lets you expand out the lines of code around the problem, provides a plain text exception suitable for e-mailing to colleagues and even a one-click button to send details to http://dpaste.com/ so you can go and talk about the error on IRC. It also serves the same purpose as phpinfo() - it shows you your application's settings, the GET, POST and COOKIE data from the request and the all important META fields assembled from the HTTP environment (great for remembering how to miss-spell HTTP_REFERER).

Useful tip number one is that you can trigger the error page from any view just by adding the following line:

assert False

You can serve up an expression with the assertion as well; it will be displayed at the top of the error page:

assert False, request.GET

One particularly useful place to use this is when you are building a complex form. If you want to see the data that was submitted, drop an assert False in to the view that the form targets and use the error page to inspect the data.

Logging to the development server console

If you want to know what's going on inside a view, the quickest way is to drop in a print statement. The development server outputs any print statements directly to the terminal; it's the server-side alternative to a JavaScript alert().

If you want to be a bit more sophisticated with your logging, it's worth turning to Python's logging module (part of the standard library). You can configure it in your settings.py:

import logging
logging.basicConfig(
    level = logging.DEBUG,
    format = '%(asctime)s %(levelname)s %(message)s',
)

Then call it from any of your views:

def my_view(request):
    import logging
    logging.debug("A log message")
    ...

Again, this will log things to the terminal where the development server is running. If you want to log things to a file you can do so by extending the basicConfig call:

logging.basicConfig(
    level = logging.DEBUG,
    format = '%(asctime)s %(levelname)s %(message)s',
    filename = '/tmp/myapp.log',
    filemode = 'w'
)

You can then use tail -f /tmp/myapp.log to see log lines being appended to that file in real-time. This can be used in production as well as development.

The above just scratches the surface of Python's logging module; with a bit of digging around in the documentation you can use it to rotate log files, send log messages over the network and even POST log events to an HTTP server somewhere.

Often you find yourself dealing with an error that only occurs in certain circumstances - a function might be called from dozens of different places in your program but only runs in to trouble in a very specific case. You can use the traceback module to log the current stack, which will allow you to tell how a function was called when something went wrong:

import logging, traceback, pprint

def my_buggy_function(arg):
    ...
    if error_condition:
        stack = pprint.pformat(traceback.extract_stack())
        logging.debug('An error occurred: %s' % stack)

The tuple returned by traceback.extract_stack() includes line numbers, function names and paths to Python files so you can use it to reconstruct a good amount of information about your program.

Using the debugger

By far the most powerful weapon in my debugging toolkit is the Python debugger, pdb. Again, this ships with the standard library so there's nothing extra to install. pdb is a command line debugger (if you want a GUI options include PyEclipse and Komodo, but I haven't used either myself). There are a bunch of ways to activate pdb, but the most straight forward is to simply drop the following line directly in to a Django view function:

import pdb; pdb.set_trace()

If you try to load that page in your browser, the browser will hang - the page will appear to be loading extremely slowly. What's actually happened is the developer server has paused execution and thrown up the pdb interface - you can switch over to your console and start interacting directly with the server mid view.

Did I mention you should never, ever leave this on in production?

So, you've got a hung development server and a pdb prompt. What can you do with it? The answer is pretty much anything. I won't provide a full pdb tutorial here (this is a good introduction), but the commands I find most useful are the following:

list: Shows the lines of source code around your current point of execution. You can run it multiple times to increase the amount of source code displayed.
n: Execute the next line
s: Same as n, but steps in to any functions that are called. You can quickly get lost in a twisty maze of code with this command, but that's OK because...
r: Continues execution until the current function returns
u: Goes UP one level in the stack - so you can see the function that called the function you are currently in
d: Goes DOWN again
locals(): not a pdb command, but handy for seeing what's in your current scope

The pdb docs have a full list of commands.

The pdb prompt doubles up as a full Python interactive shell, so you can not only access variables but you can modify them, call functions and generally mess around with the internals of your application as much as you like, while it's running. It's kind of a poor man's imitation of being a Smalltalk developer.

Remember though, the whole time you're messing around in pdb your browser is still stuck there, waiting for the HTTP request to come back. If you hit "c" (for continue) your application will kick in again, the request will be served and your browser will breathe a sigh of relief.

Thankfully you don't have to use pdb in a way that freezes your development server; it also works great in the interactive shell. If you've got a buggy function, one way to explore it is to run it interactively, then use the following idiom:

>>> def function_that_raises_an_exception():
...   assert False
... 
>>> function_that_raises_an_exception()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in function_that_raises_an_exception
AssertionError
>>> import pdb; pdb.pm()
> <stdin>(2)function_that_raises_an_exception()
(Pdb)

pdb.pm() stands for post-mortem, and is probably my favourite feature of the debugger - it lets you jump back in to debug the most recently raised exception, even if you hadn't imported pdb at the time the exception was raised.

One last pdb tip: you can use it to debug Python command line scripts such as Django's custom ./manage.py commands. The trick is to run the script like this:

python -i manage.py buggy_command

The -i argument causes Python to drop in to the interactive prompt after executing the script. If the script raised an exception, you can then use pdb.pm() to debug it.

Handling errors in production

Django's default behaviour in production (that is, when the DEBUG setting is set to False) is to e-mail exception reports to anyone listed in the ADMINS section. You can also turn on e-mail reports on every 404 error with the SEND_BROKEN_LINK_EMAILS setting, which will send them to addresses in the MANAGERS setting. As far as I know these settings don't do anything else - they're a pretty ancient bit of Django.

On a high traffic site you probably don't want to be e-mailed on every server error. One neat alternative is David Cramer's django-db-log, which logs exceptions to a database table. It cleverly uses an MD5 hash of the traceback to aggregate many reports of the same error. More importantly though, it acts as a really straight forward example of how to use Django middleware's process_exception hook to roll your own error reporting. Take a look at the code to see how simple this is.

More useful middleware

In the talk I demoed a couple of other handy pieces of middleware. The first was the ProfilerMiddleware (one of several profiling tools on Django Snippets) which allows you to add ?prof to the end of any URL to see the output of Python's cProfile module run against that request. The second is one that I've just released: DebugFooter, which adds a footer showing exactly which templates were loaded from where (handy for debugging complex template paths) as well as every executed SQL query and how long each one took.

Abusing the test client

A final tip for exploring your application interactively is to learn to use Django's TestClient. Although designed for use in unit tests, this tool is equally useful for use at the interactive prompt. It allows you to simulate an in-process request against your application from within your Python code. Here's an example:

>>> from django.test.client import Client
>>> c = Client()
>>> response = c.get("/") # The homepage
>>> response
<django.http.HttpResponse object at 0x2300470>
>>> print response
Vary: Cookie
Content-Type: text/html; charset=utf-8

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"
    "http://www.w3.org/TR/html4/strict.dtd">
<html>
...

The response object you get back is the HttpResponse returned by the view, ready to be explored interactively.

There's another function from the unit testing tools that can help with interactively exploring an application: setup_test_environment(). This function monkey-patches in some additional hooks used by the unit tests, including one that intercepts template render calls and adds information on them to the request object. Here's an example:

>>> from django.test.utils import setup_test_environment
>>> setup_test_environment()
>>> from django.test.client import Client
>>> c = Client()
>>> response = c.get("/")
>>> response.template
[<django.template.Template object at 0x2723dd0>,
 <django.template.Template object at 0x2723f30>,
 <django.template.Template object at 0x273ee10>]
>>> response.context
[ list of Context objects ]

This allows you to explore not just the HTML returned by a view, but also the templates and contexts that were used to render it.

Your tips welcome

If you have any useful tips on debugging Django applications, please share them in the comments on this entry.

Tags: debugging, django, djugl, middleware, my-talks, testing

django-db-log

2008-05-13T08:07:49+00:00

django-db-log

Middleware that logs Django exceptions to the database, using a clever scheme based on an MD5 of the traceback text to group duplicate errors in to batches.

Tags: david-cramer, django, djangodblog, exceptions, logging, middleware

Wait For It

2007-06-09T16:53:14+00:00

Wait For It

Neat WSGI middleware from Ian Bicking that launches a thread for every incoming request and watches for slow responses; if something is taking too long it returns a “please wait” page to the user and polls for completion.

Via Ian Bicking

Tags: ian-bicking, middleware, python, wsgi

Avoid IE Brokenness When using Vary and Attachments

2007-04-09T09:41:29+00:00

Avoid IE Brokenness When using Vary and Attachments

Django middleware that works around a bug in IE where external applications fail to load content that was served with a Vary header.

Via django-developers

Tags: django, http, internet-explorer, middleware, python