Simon Willison's Weblog: proxies

httpjail

2025-09-19T21:57:29+00:00

Here's a promising new (experimental) project in the sandboxing space from Ammar Bandukwala at Coder. httpjail provides a Rust CLI tool for running an individual process against a custom configured HTTP proxy.

The initial goal is to help run coding agents like Claude Code and Codex CLI with extra rules governing how they interact with outside services. From Ammar's blog post that introduces the new tool, Fine-grained HTTP filtering for Claude Code:

httpjail implements an HTTP(S) interceptor alongside process-level network isolation. Under default configuration, all DNS (udp:53) is permitted and all other non-HTTP(S) traffic is blocked.

httpjail rules are either JavaScript expressions or custom programs. This approach makes them far more flexible than traditional rule-oriented firewalls and avoids the learning curve of a DSL.

Block all HTTP requests other than the LLM API traffic itself:
$ httpjail --js "r.host === 'api.anthropic.com'" -- claude "build something great"

I tried it out using OpenAI's Codex CLI instead and found this recipe worked:

brew upgrade rust
cargo install httpjail # Drops it in `~/.cargo/bin`
httpjail --js "r.host === 'chatgpt.com'" -- codex

Within that Codex instance the model ran fine but any attempts to access other URLs (e.g. telling it "Use curl to fetch simonwillison.net)" failed at the proxy layer.

This is still at a really early stage but there's a lot I like about this project. Being able to use JavaScript to filter requests via the --js option is neat (it's using V8 under the hood), and there's also a --sh shellscript option which instead runs a shell program passing environment variables that can be used to determine if the request should be allowed.

At a basic level it works by running a proxy server and setting HTTP_PROXY and HTTPS_PROXY environment variables so well-behaving software knows how to route requests.

It can also add a bunch of other layers. On Linux it sets up nftables rules to explicitly deny additional network access. There's also a --docker-run option which can launch a Docker container with the specified image but first locks that container down to only have network access to the httpjail proxy server.

It can intercept, filter and log HTTPS requests too by generating its own certificate and making that available to the underlying process.

I'm always interested in new approaches to sandboxing, and fine-grained network access is a particularly tricky problem to solve. This looks like a very promising step in that direction - I'm looking forward to seeing how this project continues to evolve.

Via Fine-grained HTTP filtering for Claude Code

Tags: http, javascript, proxies, sandboxing, security, v8, rust, claude-code, codex-cli

Weeknotes: Apache proxies in Docker containers, refactoring Datasette

2021-11-22T05:43:44+00:00

Updates to six major projects this week, plus finally some concrete progress towards Datasette 1.0.

Fixing Datasette's proxy bugs

Now that Datasette has had its fourth birthday I've decided to really push towards hitting the 1.0 milestone. The key property of that release will be a stable JSON API, stable plugin hooks and a stable, documented context for custom templates. There's quite a lot of mostly unexciting work needed to get there.

As I work through the issues in that milestone I'm encountering some that I filed more than two years ago!

Two of those made it into the Datasette 0.59.3 bug fix release earlier this week.

The majority of the work in that release though related to Datasette's base_url feature, designed to help people who run Datasette behind a proxy.

base_url lets you run Datasette like this:

datasette --setting base_url=/prefix/ fixtures.db

When you do this, Datasette will change its URLs to start with that prefix - so the hompage will live at /prefix/, the database index page at /prefix/fixtures/, tables at /prefix/fixtures/facetable etc.

The reason you would want this is if you are running a larger website, and you intend to proxy traffic to /prefix/ to a separate Datasette instance.

The Datasette documentation includes suggested nginx and Apache configurations for doing exactly that.

This feature has been a magnet for bugs over the years! People keep finding new parts of the Datasette interface that fail to link to the correct pages when run in this mode.

The principle cause of these bugs is that I don't use Datasette in this way myself, so I wasn't testing it nearly as thoroughly as it needed.

So the first step in finally solving these issues once and for all was to get my own instance of Datasette up and running behind an Apache proxy.

Since I like to deploy live demos to Cloud Run, I decided to try and run Apache and Datasette in the same container. This took a lot of figuring out. You can follow my progress on this in these two issue threads:

The short version: I got it working! My Docker implementation now lives in the demos/apache-proxy directory and the live demo itself is deployed to datasette-apache-proxy-demo.fly.dev/prefix/.

(I ended up deploying it to Fly after running into a bug when deployed to Cloud Run that I couldn't replicate on my own laptop.)

My final implementation uses a Debian base container with Supervisord to manage the two processes.

With a working live environment, I was finally able to track down the root cause of the bugs. My notes on #1519: base_url is omitted in JSON and CSV views document how I found and solved them, and updated the associated test to hopefully avoid them ever coming back in the future.

The big Datasette table refactor

The single most complicated part of the Datasette codebase is the code behind the table view - the page that lets you browse, facet, search, filter and paginate through the contents of a table (this page here).

It's got very thorough tests, but the actual implementation is mostly a 600 line class method.

It was already difficult to work with, but the changes I want to make for Datasette 1.0 have proven too much for it. I need to refactor.

Apart from making that view easier to change and maintain, a major goal I have is for it to support a much more flexible JSON syntax. I want the JSON version to default to just returning minimal information about the table, then allow ?_extra=x parameters to opt into additional information - like facets, suggested facets, full counts, SQL schema information and so on.

This means I want to break up that 600 line method into a bunch of separate methods, each of which can be opted-in-to by the calling code.

The HTML interface should then build on top of the JSON, requesting the extras that it knows it will need and passing the resulting data through to the template. This helps solve the challenge of having a stable template context that I can document in advance of Datasette 1.0

I've been putting this off for over a year now, because it's a lot of work. But no longer! This week I finally started to get stuck in.

I don't know if I'll stick with it, but my initial attempt at this is a little unconventional. Inspired by how pytest fixtures work I'm experimenting with a form of dependency injection, in a new (very alpha) library I've released called asyncinject.

The key idea behind asyncinject is to provide a way for class methods to indicate their dependencies as named parameters, in the same way as pytest fixtures do.

When you call a method, the code can spot which dependencies have not yet been resolved and execute them before executing the method.

Crucially, since they are all async def methods they can be executed in parallel. I'm cautiously excited about this - Datasette has a bunch of opportunities for parallel queries - fetching a single page of table rows, calculating a count(*) for the entire table, executing requested facets and calculating suggested facets are all queries that could potentially run in parallel rather than in serial.

What about the GIL, you might ask? Datasette's database queries are handled by the sqlite3 module, and that module releases the GIL once it gets into SQLite C code. So theoretically I should be able to use more than one core for this all.

The asyncinject README has more details, including code examples. This may turn out to be a terrible idea! But it's really fun to explore, and I'll be able to tell for sure if this is a useful, maintainable and performant approach once I have Datasette's table view running on top of it.

git-history and sqlite-utils

I made some big improvements to my git-history tool, which automates the process of turning a JSON (or other) file that has been version-tracked in a GitHub repository (see Git scraping) into a SQLite database that can be used to explore changes to it over time.

The biggest was a major change to the database schema. Previously, the tool used full Git SHA hashes as foreign keys in the largest table.

The problem here is that a SHA hash string is 40 characters long, and if they are being used as a foreign key that's a LOT of extra weight added to the largest table.

sqlite-utils has a table.lookup() method which is designed to make creating "lookup" tables - where a string is stored in a unique column but an integer ID can be used for things like foreign keys - as easy as possible.

That method was previously quite limited, but in sqlite-utils 3.18 and 3.19 - both released this week - I expanded it to cover the more advanced needs of my git-history tool.

The great thing about building stuff on top of your own libraries is that you can discover new features that you need along the way - and then ship them promptly without them blocking your progress!

Some other highlights

s3-credentials 0.6 adds a --dry-run option that you can use to show what the tool would do without making any actual changes to your AWS account. I found myself wanting this while continuing to work on the ability to specify a folder prefix within S3 that the bucket credentials should be limited to.
datasette-publish-vercel 0.12 applies some pull requests from Romain Clement that I had left unreviewed for far too long, and adds the ability to customize the vercel.json file used for the deployment - useful for things like setting up additional custom redirects.
datasette-graphql 2.0 updates that plugin to Graphene 3.0, a major update to that library. I had to break backwards compatiblity in very minor ways, hence the 2.0 version number.

csvs-to-sqlite 1.3 is the first relase of that tool in just over a year. William Rowell contributed a new feature that allows you to populate "fixed" database columns on your imported records, see PR #81 for details.

TIL this week

Releases this week

datasette-publish-vercel: 0.12 - (18 releases total) - 2021-11-22
Datasette plugin for publishing data using Vercel
git-history: 0.4 - (6 releases total) - 2021-11-21
Tools for analyzing Git history using SQLite
sqlite-utils: 3.19 - (90 releases total) - 2021-11-21
Python CLI utility and library for manipulating SQLite databases
datasette: 0.59.3 - (101 releases total) - 2021-11-20
An open source multi-tool for exploring and publishing data
datasette-redirect-to-https: 0.1 - 2021-11-20
Datasette plugin that redirects all non-https requests to https
s3-credentials: 0.6 - (6 releases total) - 2021-11-18
A tool for creating credentials for accessing S3 buckets
csvs-to-sqlite: 1.3 - (13 releases total) - 2021-11-18
Convert CSV files into a SQLite database
datasette-graphql: 2.0 - (32 releases total) - 2021-11-17
Datasette plugin providing an automatic GraphQL API for your SQLite databases
asyncinject: 0.2a0 - (2 releases total) - 2021-11-17
Run async workflows using pytest-fixtures-style dependency injection

Tags: apache, proxies, refactoring, supervisord, docker, datasette, weeknotes, git-scraping, sqlite-utils

Weeknotes: Fun with Unix domain sockets

2021-07-13T18:57:10+00:00

A small enhancement to Datasette this week: I've added support for proxying via Unix domain sockets.

This started out as a feature request from Aslak Raanes: #1388: Serve using UNIX domain socket.

I've not worked with these much before so it was a good opportunity to learn something new. Unix domain sockets provide a mechanism whereby different processes on a machine can communicate with each over over a mechanism similar to TCP, but via a file path instead.

I've encountered these before with the Docker daemon, which listens on path /var/run/docker.sock and can be communicated with using curl like so:

curl --unix-socket /var/run/docker.sock \
  http://localhost/v1.41/containers/json

Plenty more examples in the Docker documentation if you click the 'HTTP' tab.

It turns out both nginx and Apache have the ability to proxy traffic to a Unix domain socket rather than to an HTTP port, which makes this a useful mechanism for running backend servers without attaching them to TCP ports.

Implementing this in Datasette

Datasette uses the excellent Uvicorn Python web server to serve traffic out of the box, and Uvicorn already includes support for UDS - so adding support to Datasette was pretty easy - here's the full implementation. I've added a new --uds option, so now you can run Datasette like this:

datasette --uds /tmp/datasette.sock fixtures.db

Datasette will "listen" on /tmp/datasette.sock - which means you can run requests via curl like so:

curl --unix-socket /tmp/datasette.sock \
  http://localhost/fixtures.json | jq

More importantly, it means you can configure nginx or Apache to proxy to the Datasette server like this (nginx):

daemon off;
events {
  worker_connections  1024;
}
http {
  server {
    listen 80;
    location / {
      proxy_pass http://datasette;
      proxy_set_header Host $host;
    }
  }
  upstream datasette {
    server unix:/tmp/datasette.sock;
  }
}

Or like this (Apache):

ProxyPass / unix:/tmp/datasette.sock|http://localhost/

Writing tests

The implementation was only a few lines of code (to pass the uds option to Uvicorn) but adding a test proved a little more challenging. I used this pytest fixture to spin up a server process:

@pytest.fixture(scope="session")
def ds_unix_domain_socket_server(tmp_path_factory):
    socket_folder = tmp_path_factory.mktemp("uds")
    uds = str(socket_folder / "datasette.sock")
    ds_proc = subprocess.Popen(
        ["datasette", "--memory", "--uds", uds],
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT,
        cwd=tempfile.gettempdir(),
    )
    # Give the server time to start
    time.sleep(1.5)
    # Check it started successfully
    assert not ds_proc.poll(), ds_proc.stdout.read().decode("utf-8")
    yield ds_proc, uds
    # Shut it down at the end of the pytest session
    ds_proc.terminate()

I use a similar pattern for some other tests, to exercise the --ssl-keyfile and --ssl-certfile options added in #1221.

The test itself looks like this, taking advantage of HTTPX's ability to make calls against Unix domain sockets:

@pytest.mark.serial
@pytest.mark.skipif(not hasattr(socket, "AF_UNIX"), reason="Requires socket.AF_UNIX support")
def test_serve_unix_domain_socket(ds_unix_domain_socket_server):
    _, uds = ds_unix_domain_socket_server
    transport = httpx.HTTPTransport(uds=uds)
    client = httpx.Client(transport=transport)
    response = client.get("http://localhost/_memory.json")
    assert {
        "database": "_memory",
        "path": "/_memory",
        "tables": [],
    }.items() <= response.json().items()

The skipif decorator avoids running this test on platforms which don't support Unix domain sockets (which I think includes Windows, see this comment).

The @pytest.mark.serial decorator applies a "mark" that can be used to selectively run the test. I do this because Datasette's tests run in CI using pytest-xdist, but that's not compatible with this way of spinning up a temporary server. Datasette actually runs the tests in GitHub Actions like so:

- name: Run tests
  run: |
    pytest -n auto -m "not serial"
    pytest -m "serial"

The pytest -n auto -m "not serial" line runs almost all of the tests using pytest-xdist across an automatically selected number of processes, but skips the ones marked with @pytest.mark.serial. Then the second line runs the remaining serial tests without any additional concurrency.

Documenation and example configuration for this feature can be found in the Running Datasette behind a proxy documentation. Thanks to Aslak for contributing the notes on Apache configuration.

TIL this week

Tags: proxies, datasette, weeknotes

Building a stateless API proxy

2019-05-30T04:28:55+00:00

Building a stateless API proxy

This is a really clever idea. The GitHub API is infuriatingly coarsely grained with its permissions: you often end up having to create a token with way more permissions than you actually need for your project. Thea Flowers proposes running your own proxy in front of their API that adds more finely grained permissions, based on custom encrypted proxy API tokens that use JWT to encode the original API key along with the permissions you want to grant to that particular token (as a list of regular expressions matching paths on the underlying API).

Via @theavalkyrie

Tags: apis, encryption, github, proxies, security, jwt

Charles Proxy now available on iOS

2018-03-28T15:57:34+00:00

Charles Proxy now available on iOS

I didn’t think this was possible, but the Charles debugging proxy is now available for iOS. It works by setting itself up as a VPN such that all app traffic runs through it. You can also optionally turn on SSL decryption for specific hosts by installing a special certificate (which involves jumping through several hoops). It won’t work for apps that implement SSL certificate pinning but from playing with it for a few minutes it looks like most apps haven’t done that, even apps from Google. Well worth $8.99.

Via @codepo8

Tags: charles, proxies, ios

Velocity: Forcing Gzip Compression

2010-09-30T17:45:00+00:00

Velocity: Forcing Gzip Compression

Almost every browser supports gzip these days, but 15% of web requests have had their Accept-Encoding header stripped or mangled, generally due to poorly implemented proxies or anti-virus software. Steve Souders passes on a trick used by Google Search, where an iframe is used to test the browser’s gzip support and set a cookie to force gzipping of future pages.

Tags: browsers, gzip, performance, proxies, steve-souders, recovered

nodejitsu's node-http-proxy

2010-07-28T23:34:00+00:00

nodejitsu's node-http-proxy

Exactly what I’ve been waiting for—a robust HTTP proxy library for Node that makes it trivial to proxy requests to a backend with custom proxy behaviour added in JavaScript. The example app adds an artificial delay to every request to simulate a slow connection, but other exciting potential use cases could include rate limiting, API key restriction, logging, load balancing, lint testing and more besides.

Via The Changelog

Tags: http, javascript, nodejs, proxies, recovered

A HTTP Proxy Server in 20 Lines of node.js

2010-04-28T13:24:58+00:00

A HTTP Proxy Server in 20 Lines of node.js

Proxying is definitely a sweet spot for Node.js. Peteris Krummins takes it a step further, adding host blacklists and an IP whitelist as configuration files and using Node’s watchFile method to automatically reload changes to them.

Tags: http, javascript, nodejs, peteris-krummins, proxies

Using Django as a Pass Through Image Proxy

2010-03-22T07:18:18+00:00

Using Django as a Pass Through Image Proxy

Neat idea for running development environments against data copied from a live production site—a static file serving handler which uses a local cache but copies in user-uploaded files from the production site the first time they are requested.

Via Django snippets

Tags: django, proxies

Traffic Server

2009-11-01T12:15:27+00:00

Traffic Server

Mark Nottingham explains the release of Traffic Server, a new Apache Incubator open source project donated by Yahoo! using code originally developed at Inktomi around a decade ago. Traffic Server is a HTTP proxy/cache, similar to Squid and Varnish (though Traffic Server acts as both a forward and reverse proxy, whereas Varnish only handles reverse).

Tags: apache, cache, http, inktomi, mark-nottingham, open-source, proxies, squid, trafficserver, varnish, yahoo

Exploring OAuth-Protected APIs

2009-08-23T11:06:42+00:00

Exploring OAuth-Protected APIs

One of the downsides of OAuth is that it makes debugging APIs in your browser much harder. Seth Fitzsimmons’ oauth-proxy solves this by running a Twisted-powered proxy on your local machine which OAuth-signs every request going through it using your consumer key, secret and tokens for that API. Using it with a browsers risks exposing your key and token (but not secret) to sites you accidentally browse to—it would be useful if you could pass a whitelist of API domains as a command line option to the proxy.

Tags: apis, oauth, proxies, python, seth-fitzsimmons, twisted

Yahoo! proposal to open source "Traffic Server" via the ASF

2009-07-07T12:37:02+00:00

Yahoo! proposal to open source "Traffic Server" via the ASF

Traffic Server is a “fast, scalable and extensible HTTP/1.1 compliant caching proxy server” (presumably equivalent to things like Squid and Varnish) originally acquired from Inktomi and developed internally at Yahoo! for the past three years, which has been benchmarked handling 35,000 req/s on a single box. No source code yet but it looks like the release will arrive pretty soon.

Tags: apache, asf, caching, open-source, proxies, squid, trafficserver, varnish, yahoo

How to use Django with Apache and mod_wsgi

2009-04-01T00:24:04+00:00

How to use Django with Apache and mod_wsgi

My favourite deployment option is now included in the official Django docs, thanks to Alex Gaynor. I tend to run a stripped down Apache with mod_wsgi behind an nginx proxy, and have nginx serve static files directly. This avoids the need for a completely separate media server (although a separate media domain is still a good idea for better client-side performance).

Tags: alex-gaynor, deployment, django, modwsgi, nginx, proxies, python, wsgi

Sloppy - the slow proxy

2009-01-13T16:17:41+00:00

Sloppy - the slow proxy

Java Web Start GUI application which runs a proxy to the site of your choice simulating lower connection speeds—great for testing how well your ajax holds up under poor network conditions.

Tags: ajax, java, javascript, javawebstart, performance, proxies, richard-dallaway, sloppy

ratproxy

2008-07-03T14:35:25+00:00

ratproxy

“A semi-automated, largely passive web application security audit tool”—watches you browse and highlights potential XSS, CSRF and other vulnerabilities in your application. Created by Michal Zalewski at Google.

Tags: csrf, google, michal-zalewski, proxies, ratproxy, security, testing, xss

Apache proxy auto-re-loader

2008-02-18T09:44:02+00:00

Apache proxy auto-re-loader

Neat trick: set your 502 (Bad Gateway) error document to include a meta refresh tag, automating the refresh needed should a server you are proxying to be temporarily unavailable.

Tags: apache, metarefresh, ned-batchelder, proxies

A Fair Proxy Balancer for Nginx and Mongrel

2007-12-09T14:57:44+00:00

A Fair Proxy Balancer for Nginx and Mongrel

nginx uses round robin for proxying by default; this extension module ensures requests are queued up and sent through to backend mongrel servers that aren’t currently busy. I don’t see any reason this wouldn’t work with servers other than mongrel.

Tags: fair, load-balancing, mongrel, nginx, proxies

The State of Proxy Caching

2007-06-21T14:18:50+00:00

The State of Proxy Caching

If you’ve always wondered exactly what intermediate proxies are going to do to your carefully constructed Web application, here’s your answer.

Tags: caching, http, mark-nottingham, proxies

Online and offline development with the YUI and Charles

2007-05-15T14:41:39+00:00

Online and offline development with the YUI and Charles

Stuart Colville shows how the Charles debugging proxy can be used to serve up hosted YUI files while developing offline.

Via Yahoo! User Interface Blog

Tags: charles, debugging, javascript, offline, proxies, stuart-colville, yui