Simon Willison's Weblog: releases

Large Language Models can run tools in your terminal with LLM 0.26

2025-05-27T20:35:20+00:00

LLM 0.26 is out with the biggest new feature since I started the project: support for tools. You can now use the LLM CLI tool - and Python library - to grant LLMs from OpenAI, Anthropic, Gemini and local models from Ollama with access to any tool that you can represent as a Python function.

LLM also now has tool plugins, so you can install a plugin that adds new capabilities to whatever model you are currently using.

There's a lot to cover here, but here are the highlights:

LLM can run tools now! You can install tools from plugins and load them by name with --tool/-T name_of_tool.
You can also pass in Python function code on the command-line with the --functions option.
The Python API supports tools too: llm.get_model("gpt-4.1").chain("show me the locals", tools=[locals]).text()
Tools work in both async and sync contexts.

Here's what's covered in this post:

Trying it out

First, install the latest LLM. It may not be on Homebrew yet so I suggest using pip or pipx or uv:

uv tool install llm

If you have it already, upgrade it.

uv tool upgrade llm

Tools work with other vendors, but let's stick with OpenAI for the moment. Give LLM an OpenAI API key

llm keys set openai
# Paste key here

Now let's run our first tool:

llm --tool llm_version "What version?" --td

Here's what I get:

llm_version is a very simple demo tool that ships with LLM. Running --tool llm_version exposes that tool to the model - you can specify that multiple times to enable multiple tools, and it has a shorter version of -T to save on typing.

The --td option stands for --tools-debug - it causes LLM to output information about tool calls and their responses so you can peek behind the scenes.

This is using the default LLM model, which is usually gpt-4o-mini. I switched it to gpt-4.1-mini (better but fractionally more expensive) by running:

llm models default gpt-4.1-mini

You can try other models using the -m option. Here's how to run a similar demo of the llm_time built-in tool using o4-mini:

llm --tool llm_time "What time is it?" --td -m o4-mini

Outputs:

Tool call: llm_time({})

  {
    "utc_time": "2025-05-27 19:15:55 UTC",
    "utc_time_iso": "2025-05-27T19:15:55.288632+00:00",
    "local_timezone": "PDT",
    "local_time": "2025-05-27 12:15:55",
    "timezone_offset": "UTC-7:00",
    "is_dst": true
  }

The current time is 12:15 PM PDT (UTC−7:00) on May 27, 2025, which corresponds to 7:15 PM UTC.

Models from (tool supporting) plugins work too. Anthropic's Claude Sonnet 4:

llm install llm-anthropic -U
llm keys set anthropic
# Paste Anthropic key here
llm --tool llm_version "What version?" --td -m claude-4-sonnet

Or Google's Gemini 2.5 Flash:

llm install llm-gemini -U
llm keys set gemini
# Paste Gemini key here
llm --tool llm_version "What version?" --td -m gemini-2.5-flash-preview-05-20

You can even run simple tools with Qwen3:4b, a tiny (2.6GB) model that I run using Ollama:

ollama pull qwen3:4b
llm install 'llm-ollama>=0.11a0'
llm --tool llm_version "What version?" --td -m qwen3:4b

Qwen 3 calls the tool, thinks about it a bit and then prints out a response:

More interesting tools from plugins

This demo has been pretty weak so far. Let's do something a whole lot more interesting.

LLMs are notoriously bad at mathematics. This is deeply surprising to many people: supposedly the most sophisticated computer systems we've ever built can't multiply two large numbers together?

We can fix that with tools.

The llm-tools-simpleeval plugin exposes the simpleeval "Simple Safe Sandboxed Extensible Expression Evaluator for Python" library by Daniel Fairhead. This provides a robust-enough sandbox for executing simple Python expressions.

Here's how to run a calculation:

llm install llm-tools-simpleeval
llm -T simpleeval

Trying that out:

llm -T simple_eval 'Calculate 1234 * 4346 / 32414 and square root it' --td

I got back this - it tried sqrt() first, then when that didn't work switched to ** 0.5 instead:

Tool call: simple_eval({'expression': '1234 * 4346 / 32414'})
  165.45208860368976


Tool call: simple_eval({'expression': 'sqrt(1234 * 4346 / 32414)'})
  Error: Function 'sqrt' not defined, for expression 'sqrt(1234 * 4346 / 32414)'.


Tool call: simple_eval({'expression': '(1234 * 4346 / 32414) ** 0.5'})
  12.862818066181678

The result of (1234 * 4346 / 32414) is approximately
165.45, and the square root of this value is approximately 12.86.

I've released four tool plugins so far:

llm-tools-simpleeval - as shown above, simple expression support for things like mathematics.
llm-tools-quickjs - provides access to a sandboxed QuickJS JavaScript interpreter, allowing LLMs to run JavaScript code. The environment persists between calls so the model can set variables and build functions and reuse them later on.
llm-tools-sqlite - read-only SQL query access to a local SQLite database.
llm-tools-datasette - run SQL queries against a remote Datasette instance!

Let's try that Datasette one now:

llm install llm-tools-datasette
llm -T 'Datasette("https://datasette.io/content")' --td "What has the most stars?"

The syntax here is slightly different: the Datasette plugin is what I'm calling a "toolbox" - a plugin that has multiple tools inside it and can be configured with a constructor.

Specifying --tool as Datasette("https://datasette.io/content") provides the plugin with the URL to the Datasette instance it should use - in this case the content database that powers the Datasette website.

Here's the output, with the schema section truncated for brevity:

This question triggered three calls. The model started by guessing the query! It tried SELECT name, stars FROM repos ORDER BY stars DESC LIMIT 1, which failed because the stars column doesn't exist.

The tool call returned an error, so the model had another go - this time calling the Datasette_schema() tool to get the schema of the database.

Based on that schema it assembled and then executed the correct query, and output its interpretation of the result:

The repository with the most stars is "datasette" with 10,020 stars.

Getting to this point was a real Penny Arcade Minecraft moment for me. The possibilities here are limitless. If you can write a Python function for it, you can trigger it from an LLM.

Ad-hoc command-line tools with `--functions`

I'm looking forward to people building more plugins, but there's also much less structured and more ad-hoc way to use tools with the LLM CLI tool: the --functions option.

This was inspired by a similar feature I added to sqlite-utils a while ago.

You can pass a block of literal Python code directly to the CLI tool using the --functions option, and any functions defined there will be made available to the model as tools.

Here's an example that adds the ability to search my blog:

llm --functions '
import httpx

def search_blog(q):
    "Search Simon Willison blog"
    return httpx.get("https://simonwillison.net/search/", params={"q": q}).content
' --td 'Three features of sqlite-utils' -s 'use Simon search'

This is such a hack of an implementation! I'm literally just hitting my search page and dumping the HTML straight back into tho model.

It totally works though - it helps that the GPT-4.1 series all handle a million tokens now, so crufty HTML is no longer a problem for them.

(I had to add "use Simon search" as the system prompt because without it the model would try to answer the question itself, rather than using the search tool I provided. System prompts for tools are clearly a big topic, Anthropic's own web search tool has 6,471 tokens of instructions!)

Here's the output I got just now:

Three features of sqlite-utils are:

It is a combined CLI tool and Python library for manipulating SQLite databases.

It can automatically add columns to a database table if you attempt to insert data that doesn't quite fit (using the alter=True option).

It supports plugins, allowing the extension of its functionality through third-party or custom plugins.

A better search tool would have more detailed instructions and would return relevant snippets of the results, not just the headline and first paragraph for each result. This is pretty great for just four lines of Python though!

Tools in the LLM Python API

LLM is both a CLI tool and a Python library at the same time (similar to my other project sqlite-utils). The LLM Python library grew tool support in LLM 0.26 as well.

Here's a simple example solving one of the previously hardest problems in LLMs: counting the number of Rs in "strawberry":

import llm

def count_char_in_text(char: str, text: str) -> int:
    "How many times does char appear in text?"
    return text.count(char)

model = llm.get_model("gpt-4.1-mini")
chain_response = model.chain(
    "Rs in strawberry?",
    tools=[count_char_in_text],
    after_call=print
)
for chunk in chain_response:
    print(chunk, end="", flush=True)

The after_call=print argument is a way to peek at the tool calls, the Python equivalent of the --td option from earlier.

The model.chain() method is new: it's similar to model.prompt() but knows how to spot returned tool call requests, execute them and then prompt the model again with the results. A model.chain() could potentially execute dozens of responses on the way to giving you a final answer.

You can iterate over the chain_response to output those tokens as they are returned by the model, even across multiple responses.

I got back this:

Tool(name='count_char_in_text', description='How many times does char appear in text?', input_schema={'properties': {'char': {'type': 'string'}, 'text': {'type': 'string'}}, 'required': ['char', 'text'], 'type': 'object'}, implementation=<function count_char_in_text at 0x109dd4f40>, plugin=None) ToolCall(name='count_char_in_text', arguments={'char': 'r', 'text': 'strawberry'}, tool_call_id='call_DGXcM8b2B26KsbdMyC1uhGUu') ToolResult(name='count_char_in_text', output='3', tool_call_id='call_DGXcM8b2B26KsbdMyC1uhGUu', instance=None, exception=None)

There are 3 letter "r"s in the word "strawberry".

LLM's Python library also supports asyncio, and tools can be async def functions as described here. If a model requests multiple async tools at once the library will run them concurrently with asyncio.gather().

The Toolbox form of tools is supported too: you can pass tools=[Datasette("https://datasette.io/content")] to that chain() method to achieve the same effect as the --tool 'Datasette(...) option from earlier.

Why did this take me so long?

I've been tracking llm-tool-use for a while. I first saw the trick described in the ReAcT paper, first published in October 2022 (a month before the initial release of ChatGPT). I built a simple implementation of that in a few dozen lines of Python. It was clearly a very neat pattern!

Over the past few years it has become very apparent that tool use is the single most effective way to extend the abilities of language models. It's such a simple trick: you tell the model that there are tools it can use, and have it output special syntax (JSON or XML or tool_name(arguments), it doesn't matter which) requesting a tool action, then stop.

Your code parses that output, runs the requested tools and then starts a new prompt to the model with the results.

This works with almost every model now. Most of them are specifically trained for tool usage, and there are leaderboards like the Berkeley Function-Calling Leaderboard dedicated to tracking which models do the best job of it.

All of the big model vendors - OpenAI, Anthropic, Google, Mistral, Meta - have a version of this baked into their API, either called tool usage or function calling. It's all the same underlying pattern.

The models you can run locally are getting good at this too. Ollama added tool support last year, and it's baked into the llama.cpp server as well.

It's been clear for a while that LLM absolutely needed to grow support for tools. I released LLM schema support back in February as a stepping stone towards this. I'm glad to finally have it over the line.

As always with LLM, the challenge was designing an abstraction layer that could work across as many different models as possible. A year ago I didn't feel that model tool support was mature enough to figure this out. Today there's a very definite consensus among vendors about how this should work, which finally gave me the confidence to implement it.

I also presented a workshop at PyCon US two weeks ago about Building software on top of Large Language Models, which was exactly the incentive I needed to finally get this working in an alpha! Here's the tools section from that tutorial.

Is this agents then?

Sigh.

I still don't like using the term "agents". I worry that developers will think tools in a loop, regular people will think virtual AI assistants voiced by Scarlett Johansson and academics will grumble about thermostats. But in the LLM world we appear to be converging on "tools in a loop", and that's absolutely what this.

So yes, if you want to build "agents" then LLM 0.26 is a great way to do that.

What's next for tools in LLM?

I already have a LLM tools v2 milestone with 13 issues in it, mainly around improvements to how tool execution logs are displayed but with quite a few minor issues I decided shouldn't block this release. There's a bunch more stuff in the tools label.

I'm most excited about the potential for plugins.

Writing tool plugins is really fun. I have an llm-plugin-tools cookiecutter template that I've been using for my own, and I plan to put together a tutorial around that soon.

There's more work to be done adding tool support to more model plugins. I added details of this to the advanced plugins documentation. This commit adding tool support for Gemini is a useful illustratino of what's involved.

And yes, Model Context Protocol support is clearly on the agenda as well. MCP is emerging as the standard way for models to access tools at a frankly bewildering speed. Two weeks ago it wasn't directly supported by the APIs of any of the major vendors. In just the past eight days it's been added by OpenAI, Anthropic and Mistral! It's feeling like a lot less of a moving target today.

I want LLM to be able to act as an MCP client, so that any of the MCP servers people are writing can be easily accessed as additional sources of tools for LLM.

If you're interested in talking more about what comes next for LLM, come and chat to us in our Discord.

Tags: projects, releases, ai, openai, generative-ai, llms, llm, anthropic, gemini, llm-tool-use, ai-agents, ollama

llm-openrouter 0.3

2024-12-08T23:56:14+00:00

llm-openrouter 0.3

New release of my llm-openrouter plugin, which allows LLM to access models hosted by OpenRouter.

Quoting the release notes:

Enable image attachments for models that support images. Thanks, Adam Montgomery. #12

Provide async model access. #15

Fix documentation to list correct LLM_OPENROUTER_KEY environment variable. #10

Tags: plugins, releases, ai, generative-ai, llms, llm, openrouter

Prompts.js

2024-12-07T20:35:26+00:00

I've been putting the new o1 model from OpenAI through its paces, in particular for code. I'm very impressed - it feels like it's giving me a similar code quality to Claude 3.5 Sonnet, at least for Python and JavaScript and Bash... but it's returning output noticeably faster.

I decided to try building a library I've had in mind for a while - an await ... based alternative implementation of the browser's built-in alert(), confirm() and prompt() functions.

Short version: it lets you do this:

await Prompts.alert(
    "This is an alert message!"
);

const confirmedBoolean = await Prompts.confirm(
    "Are you sure you want to proceed?"
);

const nameString = await Prompts.prompt(
    "Please enter your name"
);

Here's the source code and a a live demo where you can try it out:

I think there's something really interesting about using await in this way.

In the past every time I've used it in Python or JavaScript I've had an expectation that the thing I'm awaiting is going to return as quickly as possible - that I'm really just using this as a performance hack to unblock the event loop and allow it to do something else while I'm waiting for an operation to complete.

That's not actually necessary at all! There's no reason not to use await for operations that could take a long time to complete, such as a user interacting with a modal dialog.

Having LLMs around to help prototype this kind of library idea is really fun. This is another example of something I probably wouldn't have bothered exploring without a model to do most of the code writing work for me.

I didn't quite get it with a single prompt, but after a little bit of back-and-forth with o1 I got what I wanted - the main thing missing at first was sensible keyboard support (in particular the Enter and Escape keys).

My opening prompt was the following:

Write me a JavaScript library - no extra dependencies - which gives me the following functions:

await Prompts.alert("hi there"); -> displays a modal with a message and waits for you to click OK on it
await Prompts.confirm("Are you sure") -> an OK and cancel option, returns true or false<br>
await Prompts.prompt("What is your name?") -> a form asking the user's name, an OK button and cancel - if cancel returns null otherwise returns a string

These are equivalent to the browser builtin alert() and confirm() and prompt() - but I want them to work as async functions and to implement their own thing where they dull out the screen and show as a nicely styled modal

All CSS should be set by the Javascript, trying to avoid risk of existing CSS interfering with it

Here's the full shared ChatGPT/o1 transcript.

I then got Google's new gemini-exp-1206 model to write the first draft of the README, this time via my LLM tool:

cat index.js | llm -m gemini-exp-1206 -s \
  'write a readme for this suitable for display on npm'

Here's the response. I ended up editing this quite a bit.

I published the result to npm as prompts-js, partly to exercise those muscles again - this is only the second package I've ever published there (the first was a Web Component).

This means it's available via CDNs such as jsDelivr - so you can load it into a page and start using it like this:

<script
  src="https://cdn.jsdelivr.net/npm/prompts-js"
></script>

I haven't yet figured out how to get it working as an ES module - there's an open issue for that here.

Update: 0.0.3 switches to dialog.showModal()

I got some excellent feedback on Mastodon and on Twitter suggesting that I improve its accessibility by switching to using the built-in browser dialog.showModal().

This was a great idea! I ran a couple of rounds more with o1 and then switched to Claude 3.5 Sonnet for one last bug fix. Here's a PR where I reviewed those changes.

I shipped that as release 0.0.3, which is now powering the demo.

I also hit this message, so I guess I won't be using o1 as often as I had hoped!

Upgrading to unlimited o1 currently costs $200/month with the new ChatGPT Pro.

Things I learned from this project

Outsourcing code like this to an LLM is a great way to get something done quickly, and for me often means the difference between doing a project versus not bothering at all.

Paying attention to what the model is writing - and then iterating on it, spotting bugs and generally trying to knock it into shape - is also a great way to learn new tricks.

Here are some of the things I've learned from working on Prompts.js so far:

The const name = await askUserSomething() pattern really does work, and it feels great. I love the idea of being able to await a potentially lengthy user interaction like this.
HTML <dialog> elements are usable across multiple browsers now.
Using a <dialog> means you can skip implementing an overlay that dims out the rest of the screen yourself - that will happen automatically.
A <dialog> also does the right thing with respect to accessibility and preventing keyboard access to other elements on the page while that dialog is open.
If you set <form method="dialog"> in a form inside a dialog, submitting that form will close the dialog automatically.
The dialog.returnValue will be set to the value of the button used to submit the form.
I also learned how to create a no-dependency, no build-step single file NPM package and how to ship that to NPM automatically using GitHub Actions and GitHub Releases. I wrote that up in this TIL: Publishing a simple client-side JavaScript package to npm with GitHub Actions.

Tags: javascript, projects, releases, npm, openai, llms, ai-assisted-programming, llm, gemini, claude-3-5-sonnet, o1

New Gemini model: gemini-exp-1206

2024-12-06T18:05:30+00:00

New Gemini model: gemini-exp-1206

Google's Jeff Dean:

Today’s the one year anniversary of our first Gemini model releases! And it’s never looked better.

Check out our newest release, Gemini-exp-1206, in Google AI Studio and the Gemini API!

I upgraded my llm-gemini plugin to support the new model and released it as version 0.6 - you can install or upgrade it like this:

llm install -U llm-gemini

Running my SVG pelican on a bicycle test prompt:

llm -m gemini-exp-1206 "Generate an SVG of a pelican riding a bicycle"

Provided this result, which is the best I've seen from any model:

Here's the full output - I enjoyed these two pieces of commentary from the model:

<polygon>: Shapes the distinctive pelican beak, with an added line for the lower mandible.
[...]
transform="translate(50, 30)": This attribute on the pelican's <g> tag moves the entire pelican group 50 units to the right and 30 units down, positioning it correctly on the bicycle.

The new model is also currently in top place on the Chatbot Arena.

Update: a delightful bonus, here's what I got from the follow-up prompt:

llm -c "now animate it"

Transcript here.

Tags: google, releases, svg, ai, generative-ai, llms, llm, gemini, pelican-riding-a-bicycle, llm-release, chatbot-arena

datasette-enrichments-llm

2024-12-05T23:46:48+00:00

datasette-enrichments-llm

Today's new alpha release is datasette-enrichments-llm, a plugin for Datasette 1.0a+ that provides an enrichment that lets you run prompts against data from one or more column and store the result in another column.

So far it's a light re-implementation of the existing datasette-enrichments-gpt plugin, now using the new llm.get_async_models() method to allow users to select any async-enabled model that has been registered by a plugin - so currently any of the models from OpenAI, Anthropic, Gemini or Mistral via their respective plugins.

Still plenty to do on this one. Next step is to integrate it with datasette-llm-usage and use it to drive a design-complete stable version of that.

Tags: plugins, projects, releases, ai, datasette, generative-ai, llms, llm, enrichments

First impressions of the new Amazon Nova LLMs (via a new llm-bedrock plugin)

2024-12-04T15:50:50+00:00

Amazon released three new Large Language Models yesterday at their AWS re:Invent conference. The new model family is called Amazon Nova and comes in three sizes: Micro, Lite and Pro.

I built a new LLM plugin called llm-bedrock for accessing the models in the terminal via boto3 and the Amazon Bedrock API.

My initial impressions from trying out the models are that they're mainly competitive with the Google Gemini family. They are extremely inexpensive - Nova Micro slightly undercuts even previously cheapest model Gemini 1.5 Flash-8B - can handle quite large context and the two larger models can handle images, video and PDFs.

Pricing and capabilities

Amazon list their pricing in price per 1,000 input tokens. Almost everyone else uses price per million, so I've done the conversion.

Here's a table comparing the least expensive models from the largest providers:

Provider	Model	Cents per million input	Cents per million output
OpenAI	GPT-4o Mini	15	60
Anthropic	Claude 3 Haiku	25	125
Anthropic	Claude 3.5 Haiku	80	400
Google	Gemini 1.5 Flash-8B	3.75	15
Google	Gemini 1.5 Flash	7.5	30
Amazon	Nova Micro	3.5	14
Amazon	Nova Lite	6	24

As you can see, Nova Micro is .25 of a cent cheaper on input and 1 cent cheaper on output than previous champion Gemini 1.5 Flash-8B.

And the more expensive models:

Provider	Model	Cents per million input	Cents per million output
OpenAI	GPT-4o	250	1000
OpenAI	o1-mini	300	1200
OpenAI	o1-preview	1500	6000
Anthropic	Claude 3.5 Sonnet	300	1500
Anthropic	Claude 3 Opus	1500	7500
Google	Gemini 1.5 Pro	125	500
Amazon	Nova Pro	80	320

So Nova Pro isn't priced the same as those top-tier models, in fact it's priced about the same as Claude 3.5 Haiku (after their new 20% price drop).

The best model in the Nova family is still to come - from their post:

Amazon Nova Premier – Our most capable multimodal model for complex reasoning tasks and for use as the best teacher for distilling custom models. Amazon Nova Premier is still in training. We’re targeting availability in early 2025.

And from the press release, which presumably is talking about Nova Premier as well:

In early 2025, Amazon will support context length of over 2M input tokens.

Comparing models based on token pricing doesn't capture the entire picture because models use different tokenizers and hence may use up different numbers of tokens for the same input, especially with images and video added to the mix. I'd like to implement a good way to compare those counts in the future.

In terms of capabilities: all of the above models can handle image inputs now, with the exception of Amazon Nova Micro and o1-mini and o1-preview which are text only. Video support is currently unique to the Gemini and Nova models - Anthropic and OpenAI don't have any that handle video yet. See Amazon Nova video understanding limitations for more details.

Nova models can't handle audio, and in fact ignore audio in video that they analyze (treating it as a sequence of images). Gemini models have audio input, as does OpenAI's gpt-4o-audio-preview model ($100/$200 for input/output of those audio tokens).

Nova mini has a 128,000 input token limit, while Lite and Pro are both 300,000 tokens. This isn't quite in the same league as Gemini (2 million for Pro and Flash, 1 million for Flash-8B) but beats the standard models from both OpenAI and Anthropic. Anthropic have an "Enterprise" tier with 500,000 but I've not had a chance to try that myself.

Using Nova from the terminal with LLM

Install LLM, then install the plugin like this:

llm install llm-bedrock

Next comes the hard part: you'll need AWS credentials that have permission to access Bedrock models on your behalf.

If you've previously configured the AWS CLI you may be able to use a shortcut: request access to the Bedrock models (Nova models are approved instantly) and the tool should be able to pick up your existing credentials.

I got stuck on this for a while, then Nils Durner came to the rescue with step-by-step instructions (17 steps in total, screenshots included) for requesting access to the models, creating an IAM user, creating a JSON IAM policy that allows access to the bedrock: actions, assigning that policy to the new user and then creating and storing an access key and access secret for calling the API.

Now that you have those credentials, you can provide them to LLM by running:

llm keys set bedrock

And then pasting in the access_key:secret_access_key - the two keys with a colon in the middle.

Run llm models to confirm that the models are installed correctly - you should see output that ends like this:

Bedrock: us.amazon.nova-micro-v1:0 (aliases: nova-micro)
Bedrock: us.amazon.nova-lite-v1:0 (aliases: nova-lite)
Bedrock: us.amazon.nova-pro-v1:0 (aliases: nova-pro)

And now you can run prompts! A simple text prompt looks like this - I'm setting an optional system prompt too:

llm -m nova-micro 'Three ways to impress a pelican' -s 'short sentences'

Offer a fish.

Dance like a clown.

Wear bright colors.

That's using nova-micro, the cheapest and fastest model. Trying that again with -m nova-pro (the best model) gave me:

Offer fresh fish.

Perform mimicry of pelican calls.

Create a calm, respectful environment.

My plugin supports LLM attachments, so you can attach images, video and PDF files to a prompt using the -a option which accepts either a URL or a file path.

Trying against my standard low-resolution pelican group photo:

llm -m nova-lite 'describe image' \
  -a https://static.simonwillison.net/static/2024/pelicans.jpg -u

The -u option there outputs token usage. I got back:

The image depicts a large gathering of birds on a rocky cliff adjacent to a body of water, likely a sea or a large lake. The birds, primarily pelicans, appear to be resting and socializing in a tightly packed group. Some of the birds are standing while others are perched on the rocks, and a few are near the edge of the cliff, possibly preparing to dive into the water below. The cliff is rocky and has a rugged appearance, with some areas showing signs of erosion or weathering. The water beneath the birds is calm and appears to be a deep blue color, reflecting the clear sky above. The birds' feathers are visible in various shades of brown, gray, and white, adding to the natural and picturesque scene.

Token usage: 1,668 input, 148 output

Using my LLM pricing calculator (now upgraded to support Nova) I get 0.0136 cents - that's just over 1/100th of a cent!

For the same prompt using -m nova-pro for the more expensive Nova Pro model:

The image depicts a large gathering of birds on a rocky shore or beach, adjacent to a body of water. The scene is bustling with a variety of birds, primarily pelicans, which are identifiable by their large size, long necks, and distinctive beaks. The pelicans are positioned both standing and sitting on the rocks, with some standing tall and others hunched over.

In addition to pelicans, there are numerous smaller birds, likely seagulls, characterized by their gray and white plumage. These birds are densely packed together, creating a vivid scene of avian life. The seagulls are scattered throughout the rocky area, some standing, others sitting, and a few engaged in various activities such as preening their feathers or searching for food.

The rocky shore is uneven and covered with a mix of pebbles and larger stones, providing a natural habitat for the birds. The body of water in the background appears calm, with a reflection of the sky and the rocky shore. The water's surface is smooth, indicating little to no wind.

The sky above is clear, suggesting fair weather conditions. The lighting in the image suggests it is daytime, likely in the early morning or late afternoon, given the soft light and the absence of harsh shadows.

In summary, the image captures a lively scene of wildlife, with a focus on pelicans and seagulls congregating on a rocky shore next to a calm body of water under a clear sky. The diverse bird population and their interactions with the environment create a dynamic and engaging visual experience.

Token usage: 1,668 input, 305 output

That came in at a whopping 0.231 cents - nearly a quarter of a cent!

Let's try something absurdly ambitious. I ran some random Google searches for government military PDFs and stumbled across this one. Let's convert that to Markdown:

llm -u -m nova-pro 'convert this document to markdown' \
  -a https://www.darpa.mil/attachments/DARPA_FY18_Presidents_Budget_Request.pdf

You can see the result here. It stopped around page 9 and missed some of the tables, but this was an absurdly challenging document so I'm not surprised it didn't manage a perfect conversion. It cost me 176,386 input, 4,283 output which is a full 15.4814 cents.

Finally, let's see if it can code:

llm -m nova-pro 'python function to domnload a CSV file from a URL, create a corresponding table in SQLite and import the data'

24 input, 616 output = 0.0087 cents.

Full output here, but this is the code it wrote:

import requests
import csv
import sqlite3
import os
from io import StringIO

def download_csv_and_import_to_sqlite(url, db_name, table_name):
    # Download the CSV file from the URL
    response = requests.get(url)
    response.raise_for_status()  # Ensure we notice bad responses...
    
    # Read the CSV data
    csv_data = response.content.decode('utf-8')
    csv_reader = csv.reader(StringIO(csv_data))
    
    # Get the header (column names) from the CSV
    header = next(csv_reader)
    
    # Create the SQLite database and connect to it
    conn = sqlite3.connect(db_name)
    cursor = conn.cursor()
    
    # Create the table
    create_table_query = f"CREATE TABLE IF NOT EXISTS {table_name} ("
    create_table_query += ",".join([f"{col} TEXT" for col in header])
    create_table_query += ")"
    cursor.execute(create_table_query)
    
    # Insert the data into the table
    insert_query = f"INSERT INTO {table_name} VALUES ({','.join(['?' for _ in header])})"
    for row in csv_reader:
        cursor.execute(insert_query, row)
    
    # Commit the changes and close the connection
    conn.commit()
    conn.close()
    
    print(f"Data successfully imported into {db_name} in table {table_name}.")

# Example usage
url = "https://datasette.io/content/stats.csv"
db_name = "example.db"
table_name = "data_table"

download_csv_and_import_to_sqlite(url, db_name, table_name)

That's spot on. I may have to retire that benchmark question, it's not enough of a challenge for modern models.

Can they produce an SVG of a pelican on a bicycle? Not very well.

Closing thoughts: GAMOA

My experiments here aren't particularly comprehensive - I look forward from hearing from people who have better challenges and a more disciplined way of applying them. LiveBench have some useful results here already.

As a general initial vibe check though these Nova models pass with flying colors.

Google Gemini now has competition in terms of pricing. This is a relief to me, Gemini is so cheap I've been nervous that they'll jack the prices up through lack of competition!
These appear to be good multi-modal models - their image handling feels solid and it's neat to have a new video-input model (even if it's quite limited compared to Gemini)
Anthropic's decision to raise the price for Claude 3.5 Haiku isn't looking great, given the continued competition at the bottom end of the market. Just a few months ago Claude 3 Haiku was the cheapest model available, now it's hardly worth considering, undercut by even GPT-4o mini.
Gaining API access to anything on AWS remains horrifyingly difficult - seriously, 17 steps! Some day a PM at AWS is going to figure out that fixing this will have a material impact on Amazon's bottom line.
From the Nova press release in the what's next section:

Amazon will introduce an Amazon Nova speech-to-speech model in the first quarter of 2025. [...] Amazon is also developing a novel model that can take text, images, audio, and video as input, and generate outputs in any of these modalities. This Amazon Nova model with native multimodal-to-multimodal – or “any-to-any” modality capabilities – will be introduced mid-year 2025.

This reminded me that OpenAI's GPT-4o was meant to be able to provide image output, which they demonstrated at their launch and as far as I'm aware have not mentioned since. Maybe this will encourage them to finally ship that feature!
The cost of multi-modal models - first Gemini, now Nova - continues to leave me almost speechless. At 0.0136 cents per image I could generate a description of every photo in my 67,771 personal photo library using Nova Mini for $9.21.

With this release I think Amazon may have earned a spot among the top tier of model providers. Maybe we need a new FAANG acronym that covers OpenAI, Anthropic, Google, Meta and Amazon. I like GAMOA.

Tags: amazon, projects, releases, ai, openai, generative-ai, llms, llm, anthropic, gemini, vision-llms, llm-pricing, multi-modal-output, llm-release

datasette-queries

2024-12-03T23:59:26+00:00

datasette-queries

I released the first alpha of a new plugin to replace the crusty old datasette-saved-queries. This one adds a new UI element to the top of the query results page with an expandable form for saving the query as a new canned query:

It's my first plugin to depend on LLM and datasette-llm-usage - it uses GPT-4o mini to power an optional "Suggest title and description" button, labeled with the becoming-standard ✨ sparkles emoji to indicate an LLM-powered feature.

I intend to expand this to work across multiple models as I continue to iterate on llm-datasette-usage to better support those kinds of patterns.

For the moment though each suggested title and description call costs about 250 input tokens and 50 output tokens, which against GPT-4o mini adds up to 0.0067 cents.

Tags: plugins, projects, releases, ai, datasette, openai, generative-ai, llms, llm

datasette-llm-usage

2024-12-02T21:33:05+00:00

datasette-llm-usage

I released the first alpha of a Datasette plugin to help track LLM usage by other plugins, with the goal of supporting token allowances - both for things like free public apps that stop working after a daily allowance, plus free previews of AI features for paid-account-based projects such as Datasette Cloud.

It's using the usage features I added in LLM 0.19.

The alpha doesn't do much yet - it will start getting interesting once I upgrade other plugins to depend on it.

Design notes so far in issue #1.

Tags: plugins, projects, releases, ai, datasette, datasette-cloud, generative-ai, llms, llm

LLM 0.19

2024-12-01T23:59:45+00:00

LLM 0.19

I just released version 0.19 of LLM, my Python library and CLI utility for working with Large Language Models.

I released 0.18 a couple of weeks ago adding support for calling models from Python asyncio code. 0.19 improves on that, and also adds a new mechanism for models to report their token usage.

LLM can log those usage numbers to a SQLite database, or make then available to custom Python code.

My eventual goal with these features is to implement token accounting as a Datasette plugin so I can offer AI features in my SaaS platform without worrying about customers spending unlimited LLM tokens.

Those 0.19 release notes in full:

Tokens used by a response are now logged to new input_tokens and output_tokens integer columns and a token_details JSON string column, for the default OpenAI models and models from other plugins that implement this feature. #610

llm prompt now takes a -u/--usage flag to display token usage at the end of the response.

llm logs -u/--usage shows token usage information for logged responses.

llm prompt ... --async responses are now logged to the database. #641

llm.get_models() and llm.get_async_models() functions, documented here. #640

response.usage() and async response await response.usage() methods, returning a Usage(input=2, output=1, details=None) dataclass. #644

response.on_done(callback) and await response.on_done(callback) methods for specifying a callback to be executed when a response has completed, documented here. #653

Fix for bug running llm chat on Windows 11. Thanks, Sukhbinder Singh. #495

I also released three new plugin versions that add support for the new usage tracking feature: llm-gemini 0.5, llm-claude-3 0.10 and llm-mistral 0.9.

Tags: cli, projects, releasenotes, releases, ai, generative-ai, llms, llm

Datasette 1.0a15

2024-08-16T05:06:51+00:00

Datasette 1.0a15

Mainly bug fixes, but a couple of minor new features:

Datasette now defaults to hiding SQLite "shadow" tables, as seen in extensions such as SQLite FTS and sqlite-vec. Virtual tables that it makes sense to display, such as FTS core tables, are no longer hidden. Thanks, Alex Garcia. (#2296)
The Datasette homepage is now duplicated at /-/, using the default index.html template. This ensures that the information on that page is still accessible even if the Datasette homepage has been customized using a custom index.html template, for example on sites like datasette.io. (#2393)

Datasette also now serves more user-friendly CSRF pages, an improvement which required me to ship asgi-csrf 0.10.

Tags: csrf, projects, releases, datasette

Datasette 1.0a14: The annotated release notes

2024-08-05T23:20:01+00:00

Released today: Datasette 1.0a14. This alpha includes significant contributions from Alex Garcia, including some backwards-incompatible changes in the run-up to the 1.0 release.

Metadata now lives in a database
datasette-remote-metadata 0.2a0
SQLite isolation_level="IMMEDIATE"
Updating the URLs
Everything else
Tricks to help construct the release notes

Metadata now lives in a database

The biggest change in the alpha concerns how Datasette's metadata system works.

Datasette can record and serve metadata about the databases, tables and columns that it is serving. This includes things like the source of the data, the license it is made available under and descriptions of the tables and columns.

Historically this has been powered by a metadata.json file. Over time, this file grew to include all sorts of things that weren't strictly metadata - things like plugin configuration. Cleaning this up is a major breaking change for Datasette 1.0, and Alex has been working on this across several alphas.

The latest alpha adds a new upgrade guide describing changes plugin authors will need to make to support the new metadata system.

The big change in 1.0a14 is that metadata now lives in Datasette's hidden _internal SQLite database, in four new tables called metadata_instance, metadata_databases, metadata_resources and metadata_columns. The schema for these is now included in the documentation (updated using this Cog code), but rather than accessing those tables directly plugins are encouraged to use the new set_*_metadata() and get_*_metadata() methods on the Datasette class.

I plan to use these new tables to build a new performant, paginated homepage that shows all of the databases and tables that Datasette is serving, complete with their metadata - without needing to make potentially hundreds of calls to the now-removed get_metadata() plugin hook.

datasette-remote-metadata 0.2a0

When introducing new plugin internals like this it's always good to accompany them with a plugin that exercises them. datasette-remote-metadata is a few years old now, and provides a mechanism for hosting the metadata for a Datasette instance at a separate URL. This means you can deploy a stateless Datasette instance with a large database and later update the attached metadata without having to re-deploy the whole thing.

I released a new alpha of that plugin which switches over to the new metadata mechanism. The core code ended up looking like this, imitating code Alex wrote for Datasette Core:

async def apply_metadata(datasette, metadata_dict):
    for key in metadata_dict or {}:
        if key == "databases":
            continue
        await datasette.set_instance_metadata(key, metadata_dict[key])
    # database-level
    for dbname, db in metadata_dict.get("databases", {}).items():
        for key, value in db.items():
            if key == "tables":
                continue
            await datasette.set_database_metadata(dbname, key, value)
        # table-level
        for tablename, table in db.get("tables", {}).items():
            for key, value in table.items():
                if key == "columns":
                    continue
                await datasette.set_resource_metadata(dbname, tablename, key, value)
            # column-level
            for columnname, column_description in table.get("columns", {}).items():
                await datasette.set_column_metadata(
                    dbname, tablename, columnname, "description", column_description
                )

SQLite isolation_level="IMMEDIATE"

Sylvain Kerkour wrote about the benefits of IMMEDIATE transactions back in February. The key issue here is that SQLite defaults to starting transactions in DEFERRED mode, which can lead to SQLITE_BUSY errors if a transaction is upgraded to a write transaction mid-flight. Starting in IMMEDIATE mode for Datasette's dedicated write connection should help avoid this.

Frustratingly I failed to replicate the underlying problem in my own tests, despite having anecdotally seen it happen in the past.

After spending more time than I had budgeted for on this, I decided to ship it as an alpha to get it properly exercised before the 1.0 stable release.

Updating the URLs

Here's another change that was important to get out before 1.0.

Datasette's URL design had a subtle blemish. The following page had two potential meanings:

/databasename - list all of the tables in the specified database
/databasename?sql= - execute an arbitrary SQL query against that database

This also meant that the JSON structure returned by /database.json v.s. /database.json?sql= was different.

Alex and I decided to fix that. Alex laid out the new design in issue #2360 - there are quite a few other changes, but the big one is that we are splitting out the SQL query interface to a new URL: /databasename/-/query?sql= - or /databasename/-/query.json?sql= for the JSON API.

We've added redirects from the old URLs to the new ones, so existing links should continue to work.

Everything else

Fix for a bug where canned queries with named parameters could fail against SQLite 3.46. (#2353)

This reflects a bug fix that went out in Datasette 0.64.7.

Datasette now serves E-Tag headers for static files. Thanks, Agustin Bacigalup. (#2306)

There's still more to be done making Datasette play well with caches, but this is a great, low-risk start.

Dropdown menus now use a z-index that should avoid them being hidden by plugins. (#2311)

A cosmetic bug that showed up on Datasette Cloud when using the datasette-cluster-map plugin.

Incorrect table and row names are no longer reflected back on the resulting 404 page. (#2359)

This was reported as a potential security issue. The table names were correctly escaped, so this wasn't an XSS, but there was still potential for confusion if an attacker constructed a URL along the lines of /database-does-not-exist-visit-www.attacker.com-for-more-info. A similar fix went out in Datasette 0.64.8.

Improved documentation for async usage of the track_event(datasette, event) hook. (#2319)

Fixed some HTTPX deprecation warnings. (#2307)

Datasette now serves a <html lange="en"> attribute. Thanks, Charles Nepote. (#2348)

Datasette's automated tests now run against the maximum and minimum supported versions of SQLite: 3.25 (from September 2018) and 3.46 (from May 2024). Thanks, Alex Garcia. (#2352)

Fixed an issue where clicking twice on the URL output by datasette --root produced a confusing error. (#2375)

Tricks to help construct the release notes

I still write the Datasette release notes entirely by hand (aside from a few words auto-completed by GitHub Copilot) - I find the process of writing them to be really useful as a way to construct a final review of everything before it goes out.

I used a couple of tricks to help this time. I always start my longer release notes with an issue. The GitHub diff view is useful for seeing what's changed since the last release, but I took it a step further this time with the following shell command:

git log --pretty=format:"- %ad: %s %h" --date=short --reverse 1.0a13...81b68a14

This outputs a summary of each commit in the range, looking like this (truncated):

- 2024-03-12: Added two things I left out of the 1.0a13 release notes 8b6f155b
- 2024-03-15: Fix httpx warning about app=self.app, refs #2307 5af68377
- 2024-03-15: Fixed cookies= httpx warning, refs #2307 54f5604c
...

Crucially, the syntax of this output is in GitHub Flavored Markdown - and pasting it into an issue comment causes both the issue references and the commit hashes to be expanded into links that look like this:

It's a neat way to get a quick review of what's changed, and also means that those issues will automatically link back to the new issue where I'm constructing the release notes.

I wrote this up in a TIL here, along with another trick I used where I used LLM to get Claude 3.5 Sonnet to summarize my changes for me:

curl 'https://github.com/simonw/datasette/compare/1.0a13...2ad51baa3.diff' \
  | llm -m claude-3.5-sonnet --system \
  'generate a short summary of these changes, then a bullet point list of detailed release notes'

Tags: projects, releases, sqlite, datasette, annotated-release-notes, llm, sqlite-busy

Datasette 0.64.8

2024-06-21T23:48:43+00:00

Datasette 0.64.8

A very small Datasette release, fixing a minor potential security issue where the name of missing databases or tables was reflected on the 404 page in a way that could allow an attacker to present arbitrary text to a user who followed a link. Not an XSS attack (no code could be executed) but still a potential vector for confusing messages.

Tags: projects, releases, security, datasette

Datasette 1.0 alpha series leaks names of databases and tables to unauthenticated users

2023-08-22T17:44:31+00:00

Datasette 1.0 alpha series leaks names of databases and tables to unauthenticated users

I found and fixed a security vulnerability in the Datasette 1.0 alpha series, described in this GitHub security advisory.

The vulnerability allowed unauthenticated users to see the names of the databases and tables in an otherwise private Datasette instance—though not the actual table contents.

The fix is now shipped in Datasette 1.0a4.

The vulnerability affected Datasette Cloud as well, but thankfully I was able to analyze the access logs and confirm that no unauthenticated requests had been made against any of the affected endpoints.

Tags: releases, security, datasette

LLM 0.4

2023-06-17T22:58:34+00:00

LLM 0.4

I released a major update to my LLM CLI tool today—version 0.4, which adds conversation mode and prompt templates so you can store and re-use interesting prompts, plus a whole bunch of other large and small improvements.

I also released 0.4.1 with some minor fixes and the ability to install the tool using Hombrew: brew install simonw/llm/llm

Tags: cli, projects, releases, ai, openai, generative-ai, chatgpt, llms

Django 1.2 release notes

2010-05-17T21:11:00+00:00

Django 1.2 release notes

Released today, this is a terrific upgrade. Multiple database connections, model validation, improved CSRF protection, a messages framework, the new smart if template tag and lots, lots more. I’ve been using the 1.2 betas for a major new project over the past few months and it’s been smooth sailing all the way.

Via Django 1.2 released

Tags: csrf, django, multidb, open-source, python, releases, recovered

What's new in Django 1.2 alpha 1

2010-01-07T19:31:50+00:00

What's new in Django 1.2 alpha 1

Multiple database support, improved CSRF prevention, a messages framework (similar to the Rails “flash” feature), model validation, custom e-mail backends, template caching for much faster handling of the include and extends tags, read only fields in the admin, a better if tag and more. Very exciting release.

Via Django Weblog

Tags: alpha, csrf, django, django-admin, python, releases

Django 1.1 release notes

2009-07-29T09:34:04+00:00

Django 1.1 release notes

Django 1.1 is out! Congratulations everyone who worked on this, it’s a fantastic release. New features include aggregate support in the ORM, proxy models, deferred fields and some really nice admin improvements. Oh, and the testing framework is now up to 10 times thanks to smart use of transactions.

Via Django | Weblog | Django 1.1 released

Tags: aggregates, django, django-admin, open-source, orm, python, releases

Nmap 5.00 Release Notes

2009-07-16T19:40:10+00:00

Nmap 5.00 Release Notes

Released today, “the most important Nmap release since 1997”. New features include Ncat, a powerful netcat alternative, Ndiff, a utility for comparing scan results so you can spot changes to your network, and a new Nmap Scripting Engine using Lua.

Tags: lua, ncat, ndiff, netcat, nmap, releases, security, tools

What's New In Python 3.1

2009-06-28T15:02:09+00:00

What's New In Python 3.1

Lots of stuff, but the best bits are an ordered dictionary type (congrats, Armin), a Counter class for counting unique items in an iterable (I do this on an almost daily basis) and a bunch of performance improvements including a rewrite of the Python 3.0 IO system in C.

Tags: armin-ronacher, performance, python, python3, python31, releases

Dojo 1.3 now available

2009-04-01T00:19:25+00:00

Dojo 1.3 now available

Looks like an excellent release. dojo.create is particularly nice—I’d be interested to know why something similar has never shipped with jQuery (presumably there’s a reason) as it feels a lot more elegant than gluing together an HTML-style string. Also interesting: you can swap between Dojo’s Acme selector engine and John Resig’s sizzle.

Tags: acme, dojo, dojocreate, javascript, jquery, releases, selectors, sizzle

Python 3.0

2008-12-04T12:38:41+00:00

Python 3.0

“We are pleased to announce the release of Python 3.0 (final), a new production-ready release, on December 3rd, 2008.”

Tags: python, python3, releases

Django 1.0.2 released

2008-11-19T08:46:27+00:00

Django 1.0.2 released

An update to last week’s 1.0.1 release, which I failed to link to. 1.0.2 mainly fixes some packaging issues, while 1.0.1 contains “over two hundred fixes to the original Django 1.0 codebase”. The team are holding up the promise to move to a regular release cycle after 1.0.

Tags: django, python, releases

What's New in Python 2.6

2008-10-02T11:47:21+00:00

What's New in Python 2.6

Python 2.6 final has been released (the last 2.x version before 3.0). multiprocessing and simplejson (as json) are now in the standard library, any backwards compatible 3.0 features have been added and the official docs are now powered by Sphinx (used by Django 1.0 as well). There’s plenty more.

Via Python 2.6 Release

Tags: django, json, multiprocessing, python, releases, simplejson, sphinx-docs

Django's release process

2008-09-20T19:16:17+00:00

Django's release process

Django is moving to time-based releases, with minor releases (new features but no backwards incompatible changes) approximately every six months.

Tags: django, python, releases

[Release] CouchDB 0.7.0

2007-11-17T00:25:10+00:00

[Release] CouchDB 0.7.0

This is a huge milestone for the project—it’s the first official release to include the JSON REST API instead of XML, and it’s also the first release that is “intended for widespread use”.

Tags: couchdb, jan-lehnardt, json, releases, rest, xml

Opera 9.5 (Kestrel)

2007-09-04T10:49:00+00:00

Opera 9.5 (Kestrel)

The latest Opera alpha includes a bunch of CSS3 features (including an almost full implementation of CSS3 Selectors) as well as the ability to use SVG for scalable background images.

Tags: anne-van-kesteren, browsers, css3, opera, opera95, releases, selectors, svg

Satchmo 0.5 Release

2007-08-22T22:36:16+00:00

Satchmo 0.5 Release

Django powered e-commerce application, “the webshop for perfectionists with deadlines”.

Tags: django, python, releases, satchmo

Simon Willison's Weblog: releases

Large Language Models can run tools in your terminal with LLM 0.26

Trying it out

More interesting tools from plugins

Ad-hoc command-line tools with --functions

Tools in the LLM Python API

Why did this take me so long?

Is this agents then?

What's next for tools in LLM?

llm-openrouter 0.3

Prompts.js

Update: 0.0.3 switches to dialog.showModal()

Things I learned from this project

New Gemini model: gemini-exp-1206

datasette-enrichments-llm

First impressions of the new Amazon Nova LLMs (via a new llm-bedrock plugin)

Pricing and capabilities

Using Nova from the terminal with LLM

Multi-modal prompting with images and documents

Closing thoughts: GAMOA

datasette-queries

datasette-llm-usage

LLM 0.19

Datasette 1.0a15

Datasette 1.0a14: The annotated release notes

Metadata now lives in a database

datasette-remote-metadata 0.2a0

SQLite isolation_level="IMMEDIATE"

Updating the URLs

Everything else

Tricks to help construct the release notes

Datasette 0.64.8

Datasette 1.0 alpha series leaks names of databases and tables to unauthenticated users

LLM 0.4

Django 1.2 release notes

What's new in Django 1.2 alpha 1

Django 1.1 release notes

Nmap 5.00 Release Notes

What's New In Python 3.1

Dojo 1.3 now available

Python 3.0

Django 1.0.2 released

What's New in Python 2.6

Django's release process

[Release] CouchDB 0.7.0

Opera 9.5 (Kestrel)

Satchmo 0.5 Release

Ad-hoc command-line tools with `--functions`