Simon Willison's Weblog: mark-pilgrim

Can coding agents relicense open source through a “clean room” implementation of code?

2026-03-05T16:49:33+00:00

Over the past few months it's become clear that coding agents are extraordinarily good at building a weird version of a "clean room" implementation of code.

The most famous version of this pattern is when Compaq created a clean-room clone of the IBM BIOS back in 1982. They had one team of engineers reverse engineer the BIOS to create a specification, then handed that specification to another team to build a new ground-up version.

This process used to take multiple teams of engineers weeks or months to complete. Coding agents can do a version of this in hours - I experimented with a variant of this pattern against JustHTML back in December.

There are a lot of open questions about this, both ethically and legally. These appear to be coming to a head in the venerable chardet Python library.

chardet was created by Mark Pilgrim back in 2006 and released under the LGPL. Mark retired from public internet life in 2011 and chardet's maintenance was taken over by others, most notably Dan Blanchard who has been responsible for every release since 1.1 in July 2012.

Two days ago Dan released chardet 7.0.0 with the following note in the release notes:

Ground-up, MIT-licensed rewrite of chardet. Same package name, same public API — drop-in replacement for chardet 5.x/6.x. Just way faster and more accurate!

Yesterday Mark Pilgrim opened #327: No right to relicense this project:

[...] First off, I would like to thank the current maintainers and everyone who has contributed to and improved this project over the years. Truly a Free Software success story.

However, it has been brought to my attention that, in the release 7.0.0, the maintainers claim to have the right to "relicense" the project. They have no such right; doing so is an explicit violation of the LGPL. Licensed code, when modified, must be released under the same LGPL license. Their claim that it is a "complete rewrite" is irrelevant, since they had ample exposure to the originally licensed code (i.e. this is not a "clean room" implementation). Adding a fancy code generator into the mix does not somehow grant them any additional rights.

Dan's lengthy reply included:

You're right that I have had extensive exposure to the original codebase: I've been maintaining it for over a decade. A traditional clean-room approach involves a strict separation between people with knowledge of the original and people writing the new implementation, and that separation did not exist here.

However, the purpose of clean-room methodology is to ensure the resulting code is not a derivative work of the original. It is a means to an end, not the end itself. In this case, I can demonstrate that the end result is the same — the new code is structurally independent of the old code — through direct measurement rather than process guarantees alone.

Dan goes on to present results from the JPlag tool - which describes itself as "State-of-the-Art Source Code Plagiarism & Collusion Detection" - showing that the new 7.0.0 release has a max similarity of 1.29% with the previous release and 0.64% with the 1.1 version. Other release versions had similarities more in the 80-93% range.

He then shares critical details about his process, highlights mine:

For full transparency, here's how the rewrite was conducted. I used the superpowers brainstorming skill to create a design document specifying the architecture and approach I wanted based on the following requirements I had for the rewrite [...]

I then started in an empty repository with no access to the old source tree, and explicitly instructed Claude not to base anything on LGPL/GPL-licensed code. I then reviewed, tested, and iterated on every piece of the result using Claude. [...]

I understand this is a new and uncomfortable area, and that using AI tools in the rewrite of a long-standing open source project raises legitimate questions. But the evidence here is clear: 7.0 is an independent work, not a derivative of the LGPL-licensed codebase. The MIT license applies to it legitimately.

Since the rewrite was conducted using Claude Code there are a whole lot of interesting artifacts available in the repo. 2026-02-25-chardet-rewrite-plan.md is particularly detailed, stepping through each stage of the rewrite process in turn - starting with the tests, then fleshing out the planned replacement code.

There are several twists that make this case particularly hard to confidently resolve:

Dan has been immersed in chardet for over a decade, and has clearly been strongly influenced by the original codebase.
There is one example where Claude Code referenced parts of the codebase while it worked, as shown in the plan - it looked at metadata/charsets.py, a file that lists charsets and their properties expressed as a dictionary of dataclasses.
More complicated: Claude itself was very likely trained on chardet as part of its enormous quantity of training data - though we have no way of confirming this for sure. Can a model trained on a codebase produce a morally or legally defensible clean-room implementation?
As discussed in this issue from 2014 (where Dan first openly contemplated a license change) Mark Pilgrim's original code was a manual port from C to Python of Mozilla's MPL-licensed character detection library.
How significant is the fact that the new release of chardet used the same PyPI package name as the old one? Would a fresh release under a new name have been more defensible?

I have no idea how this one is going to play out. I'm personally leaning towards the idea that the rewrite is legitimate, but the arguments on both sides of this are entirely credible.

I see this as a microcosm of the larger question around coding agents for fresh implementations of existing, mature code. This question is hitting the open source world first, but I expect it will soon start showing up in Compaq-like scenarios in the commercial world.

Once commercial companies see that their closely held IP is under threat I expect we'll see some well-funded litigation.

Update 6th March 2026: A detail that's worth emphasizing is that Dan does not claim that the new implementation is a pure "clean room" rewrite. Quoting his comment again:

A traditional clean-room approach involves a strict separation between people with knowledge of the original and people writing the new implementation, and that separation did not exist here.

I can't find it now, but I saw a comment somewhere that pointed out the absurdity of Dan being blocked from working on a new implementation of character detection as a result of the volunteer effort he put into helping to maintain an existing open source library in that domain.

I enjoyed Armin's take on this situation in AI And The Ship of Theseus, in particular:

There are huge consequences to this. When the cost of generating code goes down that much, and we can re-implement it from test suites alone, what does that mean for the future of software? Will we see a lot of software re-emerging under more permissive licenses? Will we see a lot of proprietary software re-emerging as open source? Will we see a lot of software re-emerging as proprietary?

Update 27th March 2026: Here's a comment from Richard Fontana, one of the authors of the GPLv3 and LGPLv3 licenses, providing his own TINLA ("This Is Not Legal Advice") take on the situation:

[...] FWIW, IANDBL, TINLA, etc., I don't currently see any basis for concluding that chardet 7.0.0 is required to be released under the LGPL. AFAIK no one including Mark Pilgrim has identified persistence of copyrightable expressive material from earlier versions in 7.0.0 nor has anyone articulated some viable alternate theory of license violation. I don't think I personally would have used the MIT license here, even if I somehow rewrote everything from scratch without the use of AI in a way that didn't implicate obligations flowing from earlier versions of chardet, but that's irrelevant.

Tags: licensing, mark-pilgrim, open-source, ai, generative-ai, llms, ai-assisted-programming, ai-ethics, coding-agents, vibe-porting

SpatiaLite — Datasette documentation

2018-05-30T04:34:06+00:00

SpatiaLite — Datasette documentation

Datasette’s documentation now includes extensive coverage of the SpatiaLite extension for SQLite: how to install it, how to import latitude/longitude points, shapefiles and GeoJSON data into SpatiaLite tables, and how to run SQL queries against it that take advantage of spatial indexes. I’m learning SpatiaLite at the moment and filling out the documentation with each new trick I learn as I go—as Mark Pilgrim once taught me, the best way to learn a new technology is to write about it.

Tags: documentation, mark-pilgrim, spatialite, sqlite, datasette

An implausibly illustrated introduction to HTML5 Web Workers

2010-08-15T23:14:00+00:00

An implausibly illustrated introduction to HTML5 Web Workers

By Mark Pilgrim.

Tags: html5, mark-pilgrim, webworkers, recovered

Quoting Mark Pilgrim

2010-06-08T20:48:00+00:00

I’m renaming the book to “Dive Into HTML 5” for better SEO. This is not a joke. The book is the #5 search result for “HTML5” (no space) but #13 for “HTML 5” (with a space). I get 514 visitors a day searching Google for “HTML5” but only 53 visitors a day searching for “HTML 5”.

— Mark Pilgrim

Tags: diveintohtml5, html5, mark-pilgrim, seo, recovered

The All-In-One Almost-Alphabetical No-Bullshit Guide to Detecting Everything

2010-05-14T14:17:00+00:00

The All-In-One Almost-Alphabetical No-Bullshit Guide to Detecting Everything

Appendix A of Dive Into HTML5.

Tags: html5, javascript, mark-pilgrim, recovered, detecting, diveintohtml5

Video on the Web - Dive Into HTML5

2010-03-24T00:50:39+00:00

Video on the Web - Dive Into HTML5

Everything a web developer needs to know about video containers, video codecs, adio containers, audio codecs, h.264, theora, vorbis, licensing, encoding, batch encoding and the html5 video element.

Tags: audio, h264, html5, mark-pilgrim, theora, video

flashblockdetector

2010-03-13T10:44:58+00:00

flashblockdetector

Mark Pilgrim’s JavaScript library for detecting if the user has a Flash blocker enabled, such as FlashBlock for Firefox and Chrome or ClickToFlash for Safari. One good use of this would be to inform users that they need to opt-in to Flash for unobtrusive Flash enhancements (such as invisible audio players) to work on that page.

Tags: clicktoflash, flash, flashblock, javascript, mark-pilgrim

Quoting Mark Pilgrim

2009-11-03T07:20:26+00:00

HTML has always been a conversation between browser makers, authors, standards wonks, and other people who just showed up and liked to talk about angle brackets. Most of the successful versions of HTML have been “retro-specs,” catching up to the world while simultaneously trying to nudge it in the right direction. Anyone who tells you that HTML should be kept “pure” (presumably by ignoring browser makers, or ignoring authors, or both) is simply misinformed. HTML has never been pure, and all attempts to purify it have been spectacular failures, matched only by the attempts to replace it.

— Mark Pilgrim

Tags: html, html5, standards, mark-pilgrim

Quoting Mark Pilgrim

2009-10-21T09:59:50+00:00

You count the "value" that is lost by people who would have made money selling rival goods, but can't now because they can't compete with free. But you don't count the value that is created by people who build upon the freely given goods. [...] In other words, you only look at the first-order effects. It's the same mistake a lot of people make when they accuse open source developers of "dumping" and ruining the market for competing software. That's true, in a very narrow sense, but it ignores all the other people who took that software and used it to create something else of value.

— Mark Pilgrim

Tags: mark-pilgrim, open-source, free

Dive Into HTML 5

2009-08-20T14:40:23+00:00

Dive Into HTML 5

Mark Pilgrim’s free online book on HTML 5—currently just one chapter on canvas (which neatly illustrates the coordinate system using a diagram rendered using canvas itself) but certain to become an invaluable resource for anyone looking to take advantage of HTML 5.

Tags: books, canvas, html5, mark-pilgrim, web-standards

Quoting Alan Storm

2009-07-04T12:51:15+00:00

Yes, it'd be nice if everyone kept up to date on the progress of the various W3C working groups. They don't. There are a lot of people who asked what professional markup looked like and were told (right or wrong) that XHTML was the future. So they went ahead and learned XHTML, built their websites and chose watching a DVD or spending time with their kids over watching Mark Pilgrim and Sam Ruby do battle over Postel's Law. Now all of a sudden they're told XHTML is dead. Some wailing and gnashing of teeth is to be expected. What's needed is less "boy aren't I smarter than them" snideness, and more Hey, here's what's up.

— Alan Storm

Tags: xhtml, html5, mark-pilgrim, sam-ruby, postelslaw, xhtml2, web-standards, w3c, alan-storm

Magic properties make Firefox synchronously load the Java plugin

2009-02-27T16:03:13+00:00

Magic properties make Firefox synchronously load the Java plugin

Even defining a function called sun() (or several other symbols) will trigger the Java VM to be loaded, dramatically hurting the performance of your page.

Tags: firefox, java, javascript, mark-pilgrim, performance

Quoting Mark Pilgrim

2009-02-25T22:06:07+00:00

Some people, when confronted with a problem, think "I know, I'll quote Jamie Zawinski." Now they have two problems.

— Mark Pilgrim

Tags: mark-pilgrim, funny, jamie-zawinski

Dive into Python 3

2009-01-26T18:10:13+00:00

Dive into Python 3

Mark Pilgrim’s seminal work taught me Python nearly eight years ago. Now he’s updating it to cover Python 3. It’s just a table of contents at the moment, but the chapter on “Packaging Python libraries” has me very excited.

Tags: diveintopython, mark-pilgrim, packaging, python, python3

Quoting Mark Pilgrim

2008-12-13T23:36:20+00:00

There. Is. No. Long-Term. Data. Storage. Solution. There is only a series of short-term solutions punctuated by data migration from one medium to the next.

— Mark Pilgrim

Tags: mark-pilgrim, backups

This Week in HTML 5 - Episode 7: Clickjacking

2008-10-01T01:48:15+00:00

This Week in HTML 5 - Episode 7: Clickjacking

Clickjacking is when a third party site is embedded in an iframe with opacity 0 and positioned such that a click on the page actually hits a button on the now invisible third party site. Mark Pilgrim explains how the NoScript site uses this in a non malicious way to for the “install now!” button.

Tags: clickjacking, html5, iframes, mark-pilgrim, noscript, opacity, phishing, security

This Week in HTML 5 - Episode 1

2008-08-07T07:57:11+00:00

This Week in HTML 5 - Episode 1

It looks like the most controversial aspect of the HTML 5 spec has been addressed - now, instead of omitting the alt attribute for user generated content that has no relevant information available, sites are advised to provide an indication of the kind of image expected surrounded by braces, for example alt="{uploaded photo}".

Tags: accessibility, alt-text, html5, mark-pilgrim, whatwg

Quoting Mark Pilgrim

2008-08-05T22:52:05+00:00

My Universal Feed Parser was conceived as a weapon against what I considered the gravest error of XML: draconian error handling. Recently, someone asked me to implement a switch that makes it not fall back on lax parsing in the case of an XML wellformedness error. I said no, not because it would be difficult to implement, but because that defeats its entire reason for being.

— Mark Pilgrim

Tags: xml, mark-pilgrim, universalfeedparser, feeds, draconian, wellformedness, python

Microformats and accessibility: the soap opera that never ends

2008-06-29T08:44:46+00:00

Microformats and accessibility: the soap opera that never ends

“Be sure to tune in next week, when we’ll drown a leading accessibility expert to see if she’s a witch.”

Tags: accessibility, funny, mark-pilgrim, microformats, witch

goog/useragent/iphoto.js

2008-05-14T21:21:51+00:00

goog/useragent/iphoto.js

The Goog library includes code to detect the user’s installed version of iPhoto, based on reverse engineering the Mac.com Gallery RSS feeds. This has Mark Pilgrim written all over it.

Tags: goog, googledoctyp, iphoto, javascript, mark-pilgrim

Google Doctype

2008-05-14T20:30:27+00:00

Google Doctype

So now we know what Mark Pilgrim’s been doing at Google... heading up a project to create an encyclopaedia of web development. The JavaScript UI for browsing it is a bit weird (though you do at least get real pages if you disable JavaScript in your browser).

Tags: documentation, google, googledoctype, mark-pilgrim

Quoting Mark Pilgrim

2008-04-01T22:58:52+00:00

NOTE TO INTERNATIONAL DEVELOPERS: PLEASE DO NOT MAKE SERIOUS ANNOUNCEMENTS ON INTERNET JACKASS DAY.

— Mark Pilgrim

Tags: aprilfools, mark-pilgrim

Quoting Mark Pilgrim

2008-03-10T14:01:28+00:00

For the record, my site is valid HTML 5, except the parts that aren't. My therapist says I shouldn't rely so much on external validation.

— Mark Pilgrim

Tags: mark-pilgrim, validation, html5, standards

Gmail Greasemonkey API

2007-11-07T10:38:38+00:00

Gmail Greasemonkey API

The new version of Gmail includes API hooks for Greasemonkey script authors. The documentation is by Mark Pilgrim, author of Greasemonkey Hacks.

Via Mihai Parparita

Tags: gmail, google, greasemonkey, javascript, mark-pilgrim, mihai-parparita

Quoting Mark Pilgrim

2007-10-05T16:32:49+00:00

I thought the big draw for Apple hardware was that "It Just Works." By breaking it, you must know you’re giving up the "Just Works" factor, so what’s left? Rounded corners?

— Mark Pilgrim

Tags: mark-pilgrim, apple

The longdesc lottery

2007-09-14T11:44:42+00:00

The longdesc lottery

Mark Pilgrim is now writing for the WHATWG blog. Here he makes the case for replacing the longdesc attribute with a better solution, based on ten years of developer ignorance and misuse. As always with that site, check the comments for a microcosm of the larger debate.

Tags: accessibility, html, html5, longdesc, mark-pilgrim, whatwg

Silly season

2007-05-02T20:29:53+00:00

Silly season

Mark expresses exactly what I’ve been thinking. The fawning over Silverlight and Apollo is incredibly short sighted.

Tags: apollo, mark-pilgrim, silverlight

Two visions

2007-03-20T08:32:03+00:00

Two visions

It looks like Mark Pilgrim is going to be joining Hixie at Google.

Tags: google, ian-hickson, mark-pilgrim

IE and 2-letter domain-names

2007-02-15T00:33:13+00:00

IE and 2-letter domain-names

IE won’t let you set a cookie on XX.YY, where YY is anything other than .pl or .gr. Other browsers have better exception lists.

Via Mark Pilgrim

Tags: cookies, dns, internet-explorer, mark-pilgrim

Quoting Mark Pilgrim

2007-01-12T09:51:00+00:00

Apple doesn't give a damn. Steve Jobs doesn't build platforms, except by accident. He doesn't care about your thriving metropolis. All you independent Mac developers: you're all sharecroppers, and your rent just went up. Way up.

— Mark Pilgrim

Tags: open-source, iphone, sharecropping, steve-jobs, apple, mark-pilgrim, macos