Simon Willison's Weblog: linux

Quoting Willy Tarreau

2026-04-03T21:48:22+00:00

On the kernel security list we've seen a huge bump of reports. We were between 2 and 3 per week maybe two years ago, then reached probably 10 a week over the last year with the only difference being only AI slop, and now since the beginning of the year we're around 5-10 per day depending on the days (fridays and tuesdays seem the worst). Now most of these reports are correct, to the point that we had to bring in more maintainers to help us.

And we're now seeing on a daily basis something that never happened before: duplicate reports, or the same bug found by two different people using (possibly slightly) different tools.

— Willy Tarreau, Lead Software Developer. HAPROXY

Tags: security, linux, generative-ai, ai, llms, ai-security-research

Quoting Greg Kroah-Hartman

2026-04-03T21:44:41+00:00

Months ago, we were getting what we called 'AI slop,' AI-generated security reports that were obviously wrong or low quality. It was kind of funny. It didn't really worry us.

Something happened a month ago, and the world switched. Now we have real reports. All open source projects have real reports that are made with AI, but they're good, and they're real.

— Greg Kroah-Hartman, Linux kernel maintainer (bio), in conversation with Steven J. Vaughan-Nichols

Tags: security, linux, generative-ai, ai, llms, ai-security-research

Quoting Julian Andres Klode

2025-11-01T17:34:34+00:00

I plan to introduce hard Rust dependencies and Rust code into APT, no earlier than May 2026. This extends at first to the Rust compiler and standard library, and the Sequoia ecosystem.

In particular, our code to parse .deb, .ar, .tar, and the HTTP signature verification code would strongly benefit from memory safe languages and a stronger approach to unit testing.

If you maintain a port without a working Rust toolchain, please ensure it has one within the next 6 months, or sunset the port.

— Julian Andres Klode, debian-devel mailing list

Tags: open-source, debian, rust, linux

No More Blue Fridays

2024-07-22T18:33:00+00:00

No More Blue Fridays

Brendan Gregg: "In the future, computers will not crash due to bad software updates, even those updates that involve kernel code. In the future, these updates will push eBPF code."

New-to-me things I picked up from this:

eBPF - a technology I had thought was unique to the a Linux kernel - is coming Windows!
A useful mental model to have for eBPF is that it provides a WebAssembly-style sandbox for kernel code.
eBPF doesn't stand for "extended Berkeley Packet Filter" any more - that name greatly understates its capabilities and has been retired. More on that in the eBPF FAQ.
From this Hacker News thread eBPF programs can be analyzed before running despite the halting problem because eBPF only allows verifiably-halting programs to run.

Via Hacker News

Tags: linux, security, windows, webassembly, brendan-gregg

Hello World

2024-04-09T01:06:46+00:00

Hello World

Lennon McLean dives deep down the rabbit hole of what happens when you execute the binary compiled from “Hello world” in C on a Linux system, digging into the details of ELF executables, objdump disassembly, the C standard library, stack frames, null-terminated strings and taking a detour through musl because it’s easier to read than Glibc.

Via Hacker News

Tags: c, linux

iSH: The Linux shell for iOS

2023-12-31T04:20:00+00:00

iSH: The Linux shell for iOS

Installing this iOS app gives you a full Linux shell environment running on your phone, using a “usermode x86 emulator”. You can even install packages: “apk add python3” gave me a working Python 3.9 interpreter, installed from the apk.ish.app repository.

I didn’t think this kind of thing was allowed by the App Store, but that’s not been the case for a few years now: Section 4.5.2 of the App Store guidelines clarifies that “Educational apps designed to teach, develop, or allow students to test executable code may, in limited circumstances, download code provided that such code is not used for other purposes.”

Via Hacker News

Tags: appstore, emulator, linux, python, ios

Dynamic linker tricks: Using LD_PRELOAD to cheat, inject features and investigate programs

2023-09-08T22:05:10+00:00

Dynamic linker tricks: Using LD_PRELOAD to cheat, inject features and investigate programs

This tutorial by Rafał Cieślak from 2013 filled in a bunch of gaps in my knowledge about how C works on Linux.

Via Hacker News

Tags: c, linux

Lima VM - Linux Virtual Machines On macOS

2023-07-10T19:01:21+00:00

Lima VM - Linux Virtual Machines On macOS

This looks really useful: “brew install lima” to install, then “limactl start default” to start an Ubuntu VM running and “lima” to get a shell. Julia Evans wrote about the tool this morning, and here Adam Gordon Bell includes details on adding a writable directory (by default lima mounts your macOS home directory in read-only mode).

Via Lima: a nice way to run Linux VMs on Mac

Tags: linux, macos, virtualization, julia-evans

Weeknotes: Getting my personal Dogsheep up and running again

2021-08-22T04:46:13+00:00

I gave a talk about Dogsheep at Noisebridge's Five Minutes of Fame on Thursday. Just one problem: my regular Dogsheep demo was broken, so I ended up building it from scratch again. In doing so I fixed a few bugs in some Dogsheep tools.

Dogsheep on a Digital Ocean droplet

The latest iteration of my personal Dogsheep runs on a $20/month 4GB/2CPU Digital Ocean Droplet running Ubuntu 20.04 LTS.

It runs a private Datasette instance and a bunch of cron jobs to fetch data from Twitter, GitHub, Foursquare Swarm, Pocket and Hacker News.

It also has copies of my Apple Photos and Apple HealthKit data which I upload manually - plus a copy of my genome for good measure.

Some abbreviated notes on how I set it up, copied from a private GitHub Issues thread:

Create a new Ubuntu droplet, and configure its IP address as the A record for dogsheep.simonwillison.net
Install Python 3 and NGINX and SQLite: apt-get install python3 python3-venv nginx sqlite -y
Use letsencrypt to get an HTTPS certificate for it: apt-get update and then apt install certbot python3-certbot-nginx -y, then certbot --nginx -d dogsheep.simonwillison.net
I had to remove the ipv6only=on; bit from the NGINX configuration due to this bug
Created a dogsheep user, useradd -s /bin/bash -d /home/dogsheep/ -m -G
As that user, created a virtual environment: python3 -mvenv datasette-venv and then datasette-venv/bin/pip install wheel and datasette-venv/bin/pip install datasette datasette-auth-passwords
Created a /etc/systemd/system/datasette.service file with this contents

Created a set of blank SQLite database files in WAL mode in /home/dogsheep using the following:

for f in beta.db twitter.db healthkit.db github.db \
  swarm.db photos.db genome.db simonwillisonblog.db \
  pocket.db hacker-news.db memories.db
do
  sqlite3 $f vacuum
  # And enable WAL mode:
  sqlite3 $f 'PRAGMA journal_mode=WAL;'
done

Started the Datasette service: service datasette start
Configured NGINX to proxy to localhost port 8001, using this configuration

It's a few more steps than I'd like, but the end result was a password-protected Datasette instance running against a bunch of SQLite database files on my new server.

With Datasette up and running, the next step was to start loading in data.

Importing my tweets

I started with Twitter. I dropped my Twitter API access credentials into an auth.json file (as described here) and ran the following:

source /home/dogsheep/datasette-venv/bin/activate
pip install twitter-to-sqlite
twitter-to-sqlite user-timeline /home/dogsheep/twitter.db \
  -a /home/dogsheep/auth.json
@simonw [###############################-----] 26299/29684 00:02:06

That pulled in all 29,684 of my personal tweets.

(Actually, first it broke with an error, exposing a bug that had already been reported. I shipped a fix for that and tried again and it worked.)

Favourited tweets were a little harder - I have 39,904 favourited tweets, but the Twitter API only returns the most recent 3,200. I grabbed those more recent ones with:

twitter-to-sqlite favorites /home/dogsheep/twitter.db \
  -a /home/dogsheep/auth.json

Then I requested my Twitter archive, waited 24 hours and uploaded the resulting like.js file to the server, then ran:

twitter-to-sqlite import twitter.db /tmp/like.js

This gave me an archive_like table with the data from that file - but it wasn't the full tweet representation, just the subset that Twitter expose in the archive export.

The README shows how to inflate those into full tweets:

twitter-to-sqlite statuses-lookup twitter.db \
  --sql='select tweetId from archive_like' \
  --skip-existing
Importing 33,382 tweets [------------------------------------] 0% 00:18:28

Once that was done I wrote additional records into the favorited_by table like so:

sqlite3 twitter.db '
  INSERT OR IGNORE INTO favorited_by (tweet, user)
  SELECT tweetId, 12497 FROM archive_like
'

(12497 is my Twitter user ID.)

I also came up with a SQL view that lets me see just media attached to tweets:

sqlite-utils create-view twitter.db media_details "
select
  json_object('img_src', media_url_https, 'width', 400) as img,
  tweets.full_text,
  tweets.created_at,
  tweets.id as tweet_id,
  users.screen_name,
  'https://twitter.com/' || users.screen_name || '/status/' || tweets.id as tweet_url
from
  media
  join media_tweets on media.id = media_tweets.media_id
  join tweets on media_tweets.tweets_id = tweets.id
  join users on tweets.user = users.id
order by
  tweets.id desc
"

Now I can visit /twitter/media_details?_where=tweet_id+in+(select+tweet+from+favorited_by+where+user+=+12497) to see the most recent media tweets that I've favourited!

Swarm checkins

Swarm checkins were a lot easier. I needed my previously-created Foursquare API token, and swarm-to-sqlite:

pip install swarm-to-sqlite
swarm-to-sqlite /home/dogsheep/swarm.db --token=...

This gave me a full table of my Swarm checkins, which I can visualize using datasette-cluster-map:

Apple HealthKit

I don't yet have full automation for my Apple HealthKit data (collected by my Apple Watch) or my Apple Photos - both require me to run scripts on my laptop to create the SQLite database file and then copy the result to the server via scp.

healthkit-to-sqlite runs against the export.zip that is produced by the Apple Health app on the iPhone's export data button - for me that was a 158MB zip file which I AirDropped to my laptop and converted (after fixing a new bug) like so:

healthkit-to-sqlite ~/Downloads/export.zip healthkit.db
Importing from HealthKit  [-----------------------------]    2%  00:02:25

I uploaded the resulting 1.5GB healthkit.db file and now I can do things like visualize my 2017 San Francisco Half Marathon run on a map:

Apple Photos

For my photos I use dogsheep-photos, which I described last year in Using SQL to find my best photo of a pelican according to Apple Photos. The short version: I run this script on my laptop:

# Upload original photos to my S3 bucket
dogsheep-photos upload photos.db \
  ~/Pictures/Photos\ Library.photoslibrary/originals
dogsheep-photos apple-photos photos.db \
  --image-url-prefix "https://photos.simonwillison.net/i/" \
  --image-url-suffix "?w=600"
scp photos.db dogsheep:/home/dogsheep/photos.db

photos.db is only 171MB - it contains the metadata, including the machine learning labels, but not the photos themselves.

And now I can run queries for things like photos of food I've taken in 2021:

Automation via cron

I'm still working through the last step, which involves setting up cron tasks to refresh my data periodically from various sources. My crontab currently looks like this:

# Twitter
1,11,21,31,41,51 * * * * /home/dogsheep/datasette-venv/bin/twitter-to-sqlite user-timeline /home/dogsheep/twitter.db -a /home/dogsheep/auth.json --since
4,14,24,34,44,54 * * * * run-one /home/dogsheep/datasette-venv/bin/twitter-to-sqlite mentions-timeline /home/dogsheep/twitter.db -a /home/dogsheep/auth.json --since
11 * * * * run-one /home/dogsheep/datasette-venv/bin/twitter-to-sqlite user-timeline /home/dogsheep/twitter.db cleopaws -a /home/dogsheep/auth.json --since
6,16,26,36,46,56 * * * * run-one /home/dogsheep/datasette-venv/bin/twitter-to-sqlite favorites /home/dogsheep/twitter.db -a /home/dogsheep/auth.json --stop_after=50

# Swarm
25 */2 * * * /home/dogsheep/datasette-venv/bin/swarm-to-sqlite /home/dogsheep/swarm.db --token=... --since=2w

# Hacker News data every six hours
35 0,6,12,18 * * * /home/dogsheep/datasette-venv/bin/hacker-news-to-sqlite user /home/dogsheep/hacker-news.db simonw

# Re-build dogsheep-beta search index once an hour
32 * * * * /home/dogsheep/datasette-venv/bin/dogsheep-beta index /home/dogsheep/beta.db /home/dogsheep/dogsheep-beta.yml

I'll be expanding this out as I configure more of the Dogsheep tools for my personal instance.

TIL this week

Releases this week

healthkit-to-sqlite: 1.0.1 - (9 releases total) - 2021-08-20
Convert an Apple Healthkit export zip to a SQLite database
twitter-to-sqlite: 0.21.4 - (27 releases total) - 2021-08-20
Save data from Twitter to a SQLite database
datasette-block-robots: 1.0 - (5 releases total) - 2021-08-19
Datasette plugin that blocks robots and crawlers using robots.txt
sqlite-utils: 3.16 - (85 releases total) - 2021-08-18
Python CLI utility and library for manipulating SQLite databases
datasette-debug-asgi: 1.1 - (3 releases total) - 2021-08-17
Datasette plugin for dumping out the ASGI scope

Tags: linux, nginx, datasette, dogsheep, weeknotes

TLDR pages

2017-11-24T05:38:04+00:00

TLDR pages

This is an absurdly good idea: a community maintained set of alternative man pages for common commands with a focus on usage examples, plus a “tldr netstat” command to see them. The man pages themselves are maintained on GitHub.

Tags: documentation, github, linux

Docker Containers on the Desktop

2017-11-05T04:16:38+00:00

Docker Containers on the Desktop

Jessie Frazelle’s classic explanation from 2015 of how she runs every desktop application on her Linux machine in its own Docker container.

Via Jeff Triplett

Tags: linux, docker

What is the difference between Windows and Linux for web hosting, in other words, what are the pros and cons of each, each’s limitations, performance development environment and deployment between Windows and Linux?

2013-08-05T12:24:00+00:00

My answer to What is the difference between Windows and Linux for web hosting, in other words, what are the pros and cons of each, each’s limitations, performance development environment and deployment between Windows and Linux? on Quora

Any and every operation you perform on a Linux server can be trivially automated by copying the commands you ran in to a text file. I haven't managed a Windows server in years and I hear PowerShell is pretty great these days but an OS based around a GUI is always going to be harder to automate than one based around a command line.

Linux has a much stronger complement of high quality open source software - and new open source server software (stuff like node.js) usually becomes available for Linux first.

If you don't know anything about Linux but are comfortable with Windows, you'll find it easier to manage a Windows server in the short term.

Tags: hosting, linux, web-development, windows, quora, ecommerce

How can I download a web server's directory and all subdirectories with one command?

2012-01-15T18:55:00+00:00

My answer to How can I download a web server's directory and all subdirectories with one command? on Quora

Use wget (you can install it with apt-get install wget)

$ wget --recursive http://example.com

That will create a directory called example.com and put the mirrored downloaded files in the right sub-directories inside it.

If you just want to download a subdirectory, do this:

$ wget --recursive http://example.com/subdirectory --no-parent

The --no-parent option ensures wget won't follow links up to parent directories of the one you want to download.

Tags: http, linux, ubuntu, quora

What is a painless way for a non-Unix programmer to get started learning Unix or GNU/Linux?

2010-10-04T17:01:00+00:00

My answer to What is a painless way for a non-Unix programmer to get started learning Unix or GNU/Linux? on Quora

I'd suggest getting yourself an Ubuntu virtual machine running on your own laptop - VirtualBox is free, so that's a good starting point. That way you can play with Linux all you like without fear of breaking anything, since you can always delete the image or roll back to a snapshot.

Tags: linux, programming, software-engineering, quora

Undelete!

2010-08-21T10:56:00+00:00

Undelete!

How to undelete a file accidentally removed using rm on Linux, by grepping through the raw bytes on the hard drive searching for a unique string that was contained in the file. “grep -a -B 25 -A 100 ’some string in the file’ /dev/sda1 > results.txt”

Tags: linux, sysadmin, recovered

Running Processes

2010-03-02T09:55:18+00:00

Running Processes

I’ve been searching for a good solution to this problem (“run this program, and restart it if it falls over”) for years. I’m currently using god which works pretty well, but according to this article I should be learning upstart instead. It never ceases to amaze me how difficult this is, and how obtuse the tools are.

Tags: god, linux, processes, ubuntu, unix, upstart

Linux performance basics

2010-01-24T13:50:39+00:00

Linux performance basics

This kind of Linux knowledge is rapidly becoming a key skill for server-side web development.

Tags: jonathan-ellis, linux, ops, performance, sysadmin

The Maximal Usage Doctrine for Open Source

2010-01-06T17:23:31+00:00

The Maximal Usage Doctrine for Open Source

Yehuda Katz shares my own philosophy on Open Source licensing—stick BSD or MIT on it to maximise the number of people who can use it. The projects I work on are small enough that I don’t care if someone makes big private improvements and refuses to share them. I can see how much larger projects like Linux would disagree though.

Tags: bsd, licensing, linux, mit, open-source, yehuda-katz

Using Graphics Card Memory as Swap

2009-11-03T11:01:18+00:00

Using Graphics Card Memory as Swap

Interesting idea: “Graphic cards contain a lot of very fast RAM, typically between 64 and 512 MB. With Linux, it’s possible to use it as swap space, or even as RAM disk.”

Via Anchor Hosting

Tags: graphicscards, linux, memory, ops, performance, ram, sysadmin

Learning to compile things from source (on Unix/Linux/OSX)

2009-07-27T16:21:47+00:00

Learning to compile things from source (on Unix/Linux/OSX)

I asked on serverfault.com for tips on learning how to solve configure/make/install problems on my own, and got some extremely useful replies.

Tags: compiling, linux, macos, questions, serverfault

Ubuntu brings advanced Screen features to the masses

2009-04-28T21:52:50+00:00

Ubuntu brings advanced Screen features to the masses

Ubuntu 9.04’s screen-profiles package adds a taskbar to screen and emulates the gnome panel. You can even add a widget showing the cost of your current EC2 session.

Tags: ec2, linux, screen, ubuntu

Introducing the Karmic Koala, our mascot for Ubuntu 9.10

2009-02-21T17:19:18+00:00

Introducing the Karmic Koala, our mascot for Ubuntu 9.10

Ubuntu 9.10 will have a strong focus on cloud computing, including tools for easily creating EC2 AMIs and Eucalyptus, an open-source system for running an EC2-compatible cloud in your own data centre.

Via Simon Wardley

Tags: cloud-computing, ec2, eucalyptus, karmickoala, linux, mark-shuttleworth, ubuntu

Quoting Jeremy Allison

2008-12-09T08:03:09+00:00

[In Mali...] The outcome of this rampant illegal software copying is that Windows is seen as "the first world standard" and any attempt to push a cheaper alternative is strongly resisted. They consider it trying to cheat local people out of getting the same quality of software that is used in the developed world, even though it's a legal way of getting quality software for free.

— Jeremy Allison

Tags: mali, africa, linux, open-source, windows, piracy, jeremy-allison

Ubuntu and Debian AMIs for Amazon EC2

2008-12-08T18:04:09+00:00

Ubuntu and Debian AMIs for Amazon EC2

Exactly what it says on the tin.

Tags: amis, debian, ec2, linux, ubuntu

Happy Run Some Old Web Browsers Day!

2008-03-31T17:54:56+00:00

Happy Run Some Old Web Browsers Day!

jwz has recreated home.mcom.com, the original home of the Mosaic Communications Corporation, using a snapshot from 21st October 1994 and a domain borrowed from current owner AOL. Also includes instructions on running 1994 Mosaic Netscape binaries under a modern Linux distro.

Tags: aol, browsers, history, jwz, linux, mosaic, netscape

Quoting Miguel de Icaza

2008-01-04T12:42:35+00:00

From my perspective, it is crucial for Linux to have good support for Silverlight because I do not want Linux on the desktop to become a second class citizen ever again. [...] The core of the debate is whether Microsoft will succeed in establishing Silverlight as a RIA platform or not. You believe that without Moonlight they would not have a chance of success, and I believe that they would have regardless of us.

— Miguel de Icaza

Tags: miguel-de-icaza, roberto-callahan, silverlight, moonlight, microsoft, open-source, linux, ria

Quoting Tim Bray

2008-01-03T13:08:59+00:00

The strain due to the fact that most business desktops are locked into the Microsoft platform, at a time when both the Apple and GNU/Linux alternatives are qualitatively safer, better, and cheaper to operate, will start to become impossible to ignore.

— Tim Bray

Tags: tim-bray, predictions, microsoft, windows, linux, apple, macos

The future of web standards

2007-12-17T13:16:43+00:00

The future of web standards

Nice analysis from James Bennett, who suggests that successful open source projects (Linux, Python, Perl etc) could be used as the model for a more effective standards process, and points out that Ian Hickson is something of a BDFL for the WHAT-WG.

Tags: bdfl, ian-hickson, james-bennett, linux, open-source, perl, python, standards, w3c, web-standards, whatwg

Quoting Linus Torvalds

2007-12-16T21:53:51+00:00

Don't EVER make the mistake that you can design something better than what you get from ruthless massively parallel trial-and-error with a feedback cycle. That's giving your intelligence much too much credit.

— Linus Torvalds

Tags: linus-torvalds, evolution, open-source, linux, programming

BBC iPlayer now supports streaming Flash for Mac and Linux

2007-12-14T12:36:47+00:00

BBC iPlayer now supports streaming Flash for Mac and Linux

Absolutely fantastic—it Just Works, you hit the homepage and you can be watching video in seconds. No need to even sign up for an account. I imagine IP ranges are used to block access from outside the UK.

Tags: bbc, flash, iplayer, linux, mac, macos