Simon Willison's Weblog: ripgrep

How to add a directory to your PATH

2025-02-14T02:40:11+00:00

Classic Julia Evans piece here, answering a question which you might assume is obvious but very much isn't.

Plenty of useful tips in here, plus the best explanation I've ever seen of the three different Bash configuration options:

Bash has three possible config files: ~/.bashrc, ~/.bash_profile, and ~/.profile.

If you're not sure which one your system is set up to use, I'd recommend testing this way:

add echo hi there to your ~/.bashrc

Restart your terminal

If you see "hi there", that means ~/.bashrc is being used! Hooray!

Otherwise remove it and try the same thing with ~/.bash_profile

You can also try ~/.profile if the first two options don't work.

This article also reminded me to try which -a again, which gave me this confusing result for datasette:

% which -a datasette
/opt/homebrew/Caskroom/miniconda/base/bin/datasette
/Users/simon/.local/bin/datasette
/Users/simon/.local/bin/datasette

Why is the second path in there twice? I figured out how to use rg to search just the dot-files in my home directory:

rg local/bin -g '/.*' --max-depth 1

And found that I have both a .zshrc and .zprofile file that are adding that to my path:

.zshrc.backup
4:export PATH="$PATH:/Users/simon/.local/bin"

.zprofile
5:export PATH="$PATH:/Users/simon/.local/bin"

.zshrc
7:export PATH="$PATH:/Users/simon/.local/bin"

Tags: bash, julia-evans, ripgrep, zsh

Weeknotes: Datasette 0.63.3, datasette-ripgrep

2022-12-20T14:54:35+00:00

We're back in the UK to see family over Christmas (our first trip back since 2019). Here are a few notes from the past couple of weeks.

Datasette 0.63.3

In addition to the Datasette 1.02a2 alpha (described in detail here) I also published a small bug fix release for the 0.63.x stable branch. Quoting the release notes in full:

Fixed a bug where datasette --root, when running in Docker, would only output the URL to sign in as root when the server shut down, not when it started up. (#1958)

You no longer need to ensure await datasette.invoke_startup() has been called in order for Datasette to start correctly serving requests - this is now handled automatically the first time the server receives a request. This fixes a bug experienced when Datasette is served directly by an ASGI application server such as Uvicorn or Gunicorn. It also fixes a bug with the datasette-gunicorn plugin. (#1955)

That second fix ended up taking longer than expected.

The root of that fix was that back in Datasette 0.63 I introduced the need to call await datasette.invoke_startup() as part of Datasette's setup process - mainly to trigger plugins that might need to run their own async setup code.

This turned out to break a bunch of unexpected things - most notably, it affected any time people wanted to run Datasette using an ASGI handler such us Gunicorn or Uvicorn.

It broke my own datasette-gunicorn plugin too.

The core problem was that the Datasette() class constructor can be called synchronously, but needed a subsequent await ... call to run those async def setup methods.

I realized that a neater way to handle this would be to introduce a mechanism such that the first time anyone attempted to run an HTTP request through Datasette - an operation that always involved an await - the invoke_startup() method would be called automatically.

I got that working, but in doing so I ran into a longer-running set of problems.

Datasette has around 1,200 tests at this point, and parts of the test suite date back to the start of the project and no longer reflect my preferred way of writing tests.

I've started running into "too many open files" errors running the test suite on macOS, and have so far not quite tracked down the best way to keep open file handles under control.

Test failures were hampering my efforts to fix the issue, so I used this as the impetus to refactor a large chunk of the test suite.

Several hundred of Datasette's tests now share a single in-memory fixtures database - previously, these tests were using a fixtures.db database file created in a temporary directory.

There's still more test refactoring that I want to do, described in this issue, but I'm happy with the progress I've made so far.

datasette-ripgrep, cosmetic upgrade

I built datasette-ripgrep a couple of years ago - it's a Datasette plugin that provides a UI for running ripgrep code search queries - and linking to the results. It's very handy for finding uses of APIs that I might want to deprecate.

In using it to investigate Datasette's error output I spotted that the results would be more readable if they included a gap between non-consecutive line numbers, so I shipped an update with that improvement.

Releases this week

datasette-gunicorn: 0.1.1 - (2 releases total) - 2022-12-18
Plugin for running Datasette using Gunicorn
datasette: 0.63.3 - (122 releases total) - 2022-12-18
An open source multi-tool for exploring and publishing data
datasette-ripgrep: 0.8 - (13 releases total) - 2022-12-15
Web interface for searching your code using ripgrep, built as a Datasette plugin
datasette-media: 0.5.1 - (7 releases total) - 2022-12-13
Datasette plugin for serving media based on a SQL query
datasette-secret-santa: 0.1 - (2 releases total) - 2022-12-11
Run secret santa gift circles using Datasette
datasette-render-binary: 0.3.1 - (4 releases total) - 2022-12-10
Datasette plugin for rendering binary data

TIL this week

Tags: datasette, weeknotes, ripgrep

Weeknotes: github-to-sqlite workflows, datasette-ripgrep enhancements, Datasette 0.52

2020-12-06T05:46:11+00:00

This week: Improvements to datasette-ripgrep, github-to-sqlite and datasette-graphql, plus Datasette 0.52 and a flurry of dot-releases.

datasette-ripgrep 0.5 and 0.6

datasette-ripgrep (introduced last week) landed on Hacker News, and the comments there inspired me to build a few new features. The interface looks like this now:

I added options for literal searches (as opposed to a regex match) and ignoring case, and a field that lets you filter to just a specific file pattern, for example test*.py.

These are already features of ripgrep so adding them was a case of hooking up the interface and using it to modify the command-line arguments passed to the underlying tool.

github-to-sqlite workflows

github-to-sqlite is my command-line tool for importing data from the GitHub API into a SQLite database, for analysis with Datasette.

I released github-to-sqlite 2.8 this week with two new commands: github-to-sqlite pull-requests, contributed by Adam Jonas, and github-to-sqlite workflows.

The new workflows command can be run against one or more repositories and will fetch their GitHub Actions workflow YAML files, parse them and use them to populate new database tables called workflows, jobs and steps.

If you run workflows across a bunch of different repositories this means you can analyze your workflow usage using SQL!

My github-to-sqlite demo now includes workflows from my core Datasette and Dogsheep projects. Some example queries:

My most commonly used action steps - the top two are actions/checkout@v2 and actions/cache@v2.
All steps using actions/cache@v1, which need to be upgraded to v2 (this link will likely soon stop returning any results as I apply those updates).
My workflows that use the workflow_dispatch trigger.

The implementation is a good example of my sqlite-utils library in action - I pass the extracted YAML data straight to the .insert(data, alter=True) method which creates the correct table schema automatically, altering it if there are any missing columns.

datasette-graphql 1.3

datasette-graphql 1.3 has one tiny feature which I find enormously satisfying.

The plugin provides a GraphQL interface to any table in Datasette. The latest versions use the new "table actions" menu (accessible through a cog icon in the page heading) to provide a link to an example query for that table.

I added the example queries in 1.2, but in 1.3 the example has been expanded to include examples of foreign key references. For a table like this one of GitHub commits the example query now looks like this:

{
  commits {
    totalCount
    pageInfo {
      hasNextPage
      endCursor
    }
    nodes {
      sha
      message
      author_date
      committer_date
      raw_author {
        id
        name
      }
      raw_committer {
        id
        name
      }
      repo {
        id
        name
      }
      author {
        id
        login
      }
      committer {
        id
        login
      }
    }
  }
}

The new foreign key references include the repo, author and committer fields. The example query now demonstrates the most interesting feature of datasette-graphql - its ability to automatically convert foreign key relationships in your database into nested GraphQL fields.

It's a small change, but it makes me really happy.

Datasette 0.52

I shipped Datasette 0.52, a relatively minor release which mainly kicked off an effort to rebrand "configuration" as "settings".

I'm doing this as part of my effort to fix Datasette's "metadata" concept. The metadata.json file started out as a way to add metadata - title, description, license and source information. Over time the file expanded to cover things like default facet displays and sort orders... and then when plugins came along it grew to cover plugin configuration as well.

This is really confusing. Editing metadata.json to configure a plugin doesn't make a great deal of sense.

For Datasette 1.0 I want to clean this up. I'm planning on splitting metadata and configuration into separate mechanisms.

There's just one problem: Datasette already has a "configuration" concept in the form of the --config command-line option which can be used to set some very fundamental options for the Datasette server - things like the SQL time limit and the maximum allowed CSV download size.

I want to call plugin configuration settings "configuration", so I've renamed --config to --settings - see the new settings documentation for details.

This also gave me the chance to clean up a weird design decision. Datasette's configuration options looked like this:

datasette data.db --config sql_time_limit_ms:1000

The new --setting replacement instead looks like this:

datasette data.db --setting sql_time_limit_ms 1000

Note the lack of a colon here - having an option take two arguments is a perfectly cromulent way of using Click, but it's one I wasn't aware of when I first released Datasette.

The old --config mechanism continues to work, but it now displays a deprecation warning - it will be removed in Datasette 1.0.

Datasette dot-releases

0.52 has already had more dot-releases than any other version of Datasette. These are all pure bug fixes, mostly for obscure bugs that are unlikely to have affected anyone. To summarize the release notes:

0.52.1 updated the testing plugins documentation to promote datasette.client, fixed a bug with the display of compound foreign keys and improved the locations searched by the datasette --load-module=spatialite shortcut.
0.52.2 fixed support for the generated columns feature added in SQLite 3.31.0, fixed a 500 error on OPTIONS requests, added support for >32MB database file downloads on Cloud Run and shipped a CSS fix to the cog menus contributed by Abdussamet Koçak.
0.52.3 fixed a fun bug with Datasette installed on Amazon Linux running on ARM where static assets would 404. I eventually tracked that down to an unexpected symlink in the site-packages directory.
0.52.4 now writes errors logged by Datasette to stderr, not stdout. It also fixes a startup error on Windows, another contribution from Abdussamet Koçak.

Broken Dogsheep

My personal Dogsheep broke this week. I've been running it on an Amazon Lightsail instance, and this week I learned that Lightsail has a baseline CPU mechanism which grants your instance burst capacity but shuts it down if it exceeds that capacity too often!

So I'm moving it to a DigitalOcean droplet which won't do that, and trying to figure out enough Ansible to completely automate the process.

My ideal server is one that is configured entirely from files in source control, and updates itself by pulling new configuration from that repository. I plan to use ansible-pull for this, once I've put together the necessary playbooks.

Tags: projects, graphql, datasette, dogsheep, weeknotes, ripgrep

datasette-ripgrep: deploy a regular expression search engine for your source code

2020-11-28T06:51:06+00:00

This week I built datasette-ripgrep - a web application for running regular expression searches against source code, built on top of the amazing ripgrep command-line tool.

datasette-ripgrep demo

I've deployed a demo version of the application here:

ripgrep.datasette.io/-/ripgrep?pattern=pytest

The demo runs searches against the source code of every one of my GitHub repositories that start with datasette - 61 repos right now - so it should include all of my Datasette plugins plus the core Datasette repository itself.

Since it's running on top of ripgrep, it supports regular expressions. This is absurdly useful. Some examples:

Every usage of the .plugin_config( method: plugin_config\(
Everywhere I use async with httpx.AsyncClient (usually in tests): async with.*AsyncClient
All places where I use a Jinja | filter inside a variable: \{\{.*\|.*\}\}

I usually run ripgrep as rg on the command-line, or use it within Visual Studio Code (fun fact: the reason VS Code's "Find in Files" is so good is it's running ripgrep under the hood).

So why have it as a web application? Because this means I can link to it, bookmark it and use it on my phone.

Why build this?

There are plenty of great existing code search tools out there already: I've heard great things about livegrep, and a quick Google search shows a bunch of other options.

Aside from being a fun project, datasette-ripgrep has one key advantage: it gets to benefit from Datasette's publishing mechanism, which means it's really easy to deploy.

That ripgrep.datasette.io demo is deployed by checking out the source code to be searched into a all directory and then using the following command:

datasette publish cloudrun \
    --metadata metadata.json \
    --static all:all \
    --install=datasette-ripgrep \
    --service datasette-ripgrep \
    --apt-get-install ripgrep

all is a folder containing the source code to be searched. metadata.json contains this:

{
    "plugins": {
        "datasette-ripgrep": {
            "path": "/app/all",
            "time_limit": 3.0
        }
    }
}

That's all there is to it! The result is a deployed code search engine, running on Google Cloud Run.

(If you want to try this yourself you'll need to be using the just-released Datasette 0.52.)

The GitHub Action workflow that deploys the demo also uses my github-to-sqlite tool to fetch my repos and then shallow-clones the ones that begin with datasette.

If you have your own Google Cloud Run credentials, you can run your own copy of that workflow against your own repositories.

A different kind of Datasette plugin

Datasette is a tool for publishing SQLite databases, so most Datasette plugins integrate with SQLite in some way.

datasette-ripgrep is different: it makes no use of SQLite at all, but instead takes advantage of Datasette's URL routing, datasette publish deployments and permissions system.

The plugin implementation is currently 134 lines of code, excluding tests and templates.

While the plugin doesn't use SQLite, it does share a common philosophy with Datasette: the plugin bundles the source code that it is going to search as part of the deployed application, in a similar way to how Datasette usually bundles one or more SQLite database files.

As such, it's extremely inexpensive to run and can be deployed to serverless hosting. If you need to scale it, you can run more copies.

This does mean that the application needs to be re-deployed to pick up changes to the searchable code. I'll probably set my demo to do this on a daily basis.

Controlling processes from asyncio

The trickiest part of the implementation was figuring out how to use Python's asyncio.create_subprocess_exec() method to safely run the rg process in response to incoming requests.

I don't want expensive searches to tie up the server, so I implemented two limits here. The first is a time limit: by default, searches have a second to run after which the rg process will be terminated and only results recieved so far will be returned. This is achieved using the asyncio.wait_for() function.

I also implemented a limit on the number of matching lines that can be returned, defaulting to 2,000. Any more than that and the process is terminated early.

Both of these limits can be customized using plugin settings (documented in the README). You can see how they are implemented in the async def run_ripgrep(pattern, path, time_limit=1.0, max_lines=2000) function.

Highlighted linkable line numbers

The other fun implementation detail is the way the source code listings are displayed. I'm using CSS to display the line numbers in a way that makes them visible without them breaking copy-and-paste (inspired by this article by Sylvain Durand).

code:before {
    content: attr(data-line);
    display: inline-block;
    width: 3.5ch;
    -webkit-user-select: none;
    color: #666;
}

The HTML looks like this:

<pre><code id="L1" data-line="1">from setuptools import setup</code>
<code id="L2" data-line="2">import os</code>
<code id="L3" data-line="3">&nbsp;</code>
<code id="L4" data-line="4">VERSION = &#34;0.1&#34;</code>
...

I wanted to imitate GitHub's handling of line links, where adding #L23 to the URL both jumps to that line and causes the line to be highlighted. Here's a demo of that - I use the following JavaScript to update the contents of a <style id="highlightStyle"></style> element in the document head any time the URL fragment changes:

<script>
var highlightStyle = document.getElementById('highlightStyle');
function highlightLineFromFragment() {
    if (/^#L\d+$/.exec(location.hash)) {
        highlightStyle.innerText = `${location.hash} { background-color: yellow; }`;
    }
}
highlightLineFromFragment();
window.addEventListener("hashchange", highlightLineFromFragment);
</script>

It's the simplest way I could think of to achieve this effect.

Update 28th November 2020: Louis Lévêque on Twitter suggested using the CSS :target selector instead, which is indeed MUCH simpler - I deleted the above JavaScript and replaced it with this CSS:

:target {
    background-color: #FFFF99;
}

Next steps for this project

I'm pleased to have got datasette-ripgrep to a workable state, and I'm looking forward to using it to answer questions about the growing Datasette ecosystem. I don't know how much more time I'll invest in this - if it proves useful then I may well expand it.

I do think there's something really interesting about being able to spin up this kind of code search engine on demand using datasette publish. It feels like a very useful trick to have access to.

Better URLs for my TILs

My other project this week was an upgrade to til.simonwillison.net: I finally spent the time to design nicer URLs for the site.

Before:

til.simonwillison.net/til/til/javascript_manipulating-query-params.md

After:

til.simonwillison.net/javascript/manipulating-query-params

The implementation for this takes advantage of a feature I sneaked into Datasette 0.49: Path parameters for custom page templates. I can create a template file called pages/{topic}/{slug}.html and Datasette use that template to handle 404 errors that match that pattern.

Here's the new pages/{topic}/{slug}.html template for my TIL site. It uses the sql() template function from the datasette-template-sql plugin to retrieve and render the matching TIL, or raises a 404 if no TIL can be found.

I also needed to setup redirects from the old pages to the new ones. I wrote a TIL on edirects for Datasette explaining how I did that.

TIL this week

Redirects for Datasette

Releases this week

datasette-ripgrep 0.2 - 2020-11-27
datasette-ripgrep 0.1 - 2020-11-26
datasette-atom 0.8.1 - 2020-11-25
datasette-ripgrep 0.1a1 - 2020-11-25
datasette-ripgrep 0.1a0 - 2020-11-25
datasette-graphql 1.2.1 - 2020-11-24

Tags: async, css, projects, python, datasette, weeknotes, cloudrun, ripgrep, baked-data

How FZF and ripgrep improved my workflow

2019-07-05T17:51:09+00:00

How FZF and ripgrep improved my workflow

I’m already a keen user of ripgrep (a crazy-fast grep alternative) but fzf was new to me: it’s a CLI utility that lets you pipe in a list of strings, then gives you a typeahead search interface to search and select a string before returning the selected string to stdout when you hit enter. This means you can pipe it together with other tools to add a dynamic selection step, which has all kinds of delightful combinations. “vi $(find . | fzf)” for example opens vi against the file you selected.

Via Hacker News

Tags: cli, unix, ripgrep

ripgrep is faster than {grep, ag, git grep, ucg, pt, sift}

2019-04-16T17:52:53+00:00

ripgrep is faster than {grep, ag, git grep, ucg, pt, sift}

Andrew Gallant's post from September 2016 introducing ripgrep, the command-line grep tool he wrote using Rust (on top of the Rust regular expression library also written by Andrew). ripgrep is a beautifully designed CLI interface and is crazy fast, and this post describes how it gets its performance in a huge amount of detail, right down to comparing the different algorithmic approaches used by other similar tools.

I recently learned that ripgrep ships as part of VS Code, which is why VS Code's search-across-project feature is so fast. In fact, if you dig around in the OS X package you can find the rg binary already installed on your mac:

find /Applications/Visual* | grep bin/rg

Via Ripgrep 11 Released

Tags: cli, rust, ripgrep, vs-code, andrew-gallant