Simon Willison's Weblog: internet-archive

Quoting Joseph Weizenbaum

2026-03-08T14:59:48+00:00

What I had not realized is that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people.

— Joseph Weizenbaum, creator of ELIZA, in 1976 (via)

Tags: computer-history, internet-archive, ai, ai-ethics

Spotlighting The World Factbook as We Bid a Fond Farewell

2026-02-05T00:23:38+00:00

Spotlighting The World Factbook as We Bid a Fond Farewell

Somewhat devastating news today from CIA:

One of CIA’s oldest and most recognizable intelligence publications, The World Factbook, has sunset.

There's not even a hint as to why they decided to stop maintaining this publication, which has been their most useful public-facing initiative since 1971 and a cornerstone of the public internet since 1997.

In a bizarre act of cultural vandalism they've not just removed the entire site (including the archives of previous versions) but they've also set every single page to be a 302 redirect to their closure announcement.

The Factbook has been released into the public domain since the start. There's no reason not to continue to serve archived versions - a banner at the top of the page saying it's no longer maintained would be much better than removing all of that valuable content entirely.

Up until 2020 the CIA published annual zip file archives of the entire site. Those are available (along with the rest of the Factbook) on the Internet Archive.

I downloaded the 384MB .zip file for the year 2020 and extracted it into a new GitHub repository, simonw/cia-world-factbook-2020. I've enabled GitHub Pages for that repository so you can browse the archived copy at simonw.github.io/cia-world-factbook-2020/.

Here's a neat example of the editorial voice of the Factbook from the What's New page, dated December 10th 2020:

Years of wrangling were brought to a close this week when officials from Nepal and China announced that they have agreed on the height of Mount Everest. The mountain sits on the border between Nepal and Tibet (in western China), and its height changed slightly following an earthquake in 2015. The new height of 8,848.86 meters is just under a meter higher than the old figure of 8,848 meters. The World Factbook rounds the new measurement to 8,849 meters and this new height has been entered throughout the Factbook database.

Via Hacker News

Tags: cia, github, internet-archive

Reddit will block the Internet Archive

2025-08-11T18:11:49+00:00

Reddit will block the Internet Archive

Well this sucks. Jay Peters for the Verge:

Reddit says that it has caught AI companies scraping its data from the Internet Archive’s Wayback Machine, so it’s going to start blocking the Internet Archive from indexing the vast majority of Reddit. The Wayback Machine will no longer be able to crawl post detail pages, comments, or profiles; instead, it will only be able to index the Reddit.com homepage, which effectively means Internet Archive will only be able to archive insights into which news headlines and posts were most popular on a given day.

Tags: internet-archive, reddit, scraping, ai, training-data, ai-ethics

TextSynth Server

2024-11-21T05:16:55+00:00

TextSynth Server

I'd missed this: Fabrice Bellard (yes, that Fabrice Bellard) has a project called TextSynth Server which he describes like this:

ts_server is a web server proposing a REST API to large language models. They can be used for example for text completion, question answering, classification, chat, translation, image generation, ...

It has the following characteristics:

All is included in a single binary. Very few external dependencies (Python is not needed) so installation is easy.

Supports many Transformer variants (GPT-J, GPT-NeoX, GPT-Neo, OPT, Fairseq GPT, M2M100, CodeGen, GPT2, T5, RWKV, LLAMA, Falcon, MPT, Llama 3.2, Mistral, Mixtral, Qwen2, Phi3, Whisper) and Stable Diffusion.

[...]

Unlike many of his other notable projects (such as FFmpeg, QEMU, QuickJS) this isn't open source - in fact it's not even source available, you instead can download compiled binaries for Linux or Windows that are available for non-commercial use only.

Commercial terms are available, or you can visit textsynth.com and pre-pay for API credits which can then be used with the hosted REST API there.

This is not a new project: the earliest evidence I could find of it was this July 2019 page in the Internet Archive, which said:

Text Synth is build using the GPT-2 language model released by OpenAI. [...] This implementation is original because instead of using a GPU, it runs using only 4 cores of a Xeon E5-2640 v3 CPU at 2.60GHz. With a single user, it generates 40 words per second. It is programmed in plain C using the LibNC library.

Via They all use it - Thorsten Ball

Tags: internet-archive, ai, generative-ai, local-llms, llms, gpt-2, fabrice-bellard

Wayback Machine: Models - Anthropic (8th October 2024)

2024-10-22T22:42:17+00:00

Wayback Machine: Models - Anthropic (8th October 2024)

The Internet Archive is only intermittently available at the moment, but the Wayback Machine just came back long enough for me to confirm that the Anthropic Models documentation page listed Claude 3.5 Opus as coming “Later this year” at least as recently as the 8th of October, but today makes no mention of that model at all.

October 8th 2024

October 22nd 2024

Claude 3 came in three flavors: Haiku (fast and cheap), Sonnet (mid-range) and Opus (best). We were expecting 3.5 to have the same three levels, and both 3.5 Haiku and 3.5 Sonnet fitted those expectations, matching their prices to the Claude 3 equivalents.

It looks like 3.5 Opus may have been entirely cancelled, or at least delayed for an unpredictable amount of time. I guess that means the new 3.5 Sonnet will be Anthropic's best overall model for a while, maybe until Claude 4.

Tags: internet-archive, ai, generative-ai, llms, anthropic, claude

Today's research challenge: why is August 1st "World Wide Web Day"?

2024-08-01T17:34:29+00:00

Today's research challenge: why is August 1st "World Wide Web Day"?

Here's a fun mystery. A bunch of publications will tell you that today, August 1st, is "World Wide Web Day"... but where did that idea come from?

It's not an official day marked by any national or international organization. It's not celebrated by CERN or the W3C.

The date August 1st doesn't appear to hold any specific significance in the history of the web. The first website was launched on August 6th 1991.

I posed the following three questions this morning on Mastodon:

Who first decided that August 1st should be "World Wide Web Day"?
Why did they pick that date?
When was the first World Wide Web Day celebrated?

Finding answers to these questions has proven stubbornly difficult. Searches on Google have proven futile, and illustrate the growing impact of LLM-generated slop on the web: they turn up dozens of articles celebrating the day, many from news publications playing the "write about what people might search for" game and many others that have distinctive ChatGPT vibes to them.

One early hint we've found is in the "Bylines 2010 Writer's Desk Calendar" by Snowflake Press, published in January 2009. Jessamyn West spotted that on the book's page in the Internet Archive, but it merely lists "World Wide Web Day" at the bottom of the July calendar page (clearly a printing mistake, the heading is meant to align with August 1st on the next page) without any hint as to the origin:

I found two earlier mentions from August 1st 2008 on Twitter, from @GabeMcCauley and from @iJess.

Our earliest news media reference, spotted by Hugo van Kemenade, is also from August 1st 2008: this opinion piece in the Attleboro Massachusetts Sun Chronicle, which has no byline so presumably was written by the paper's editorial board:

Today is World Wide Web Day, but who cares? We'd rather nap than surf. How about you? Better relax while you can: August presages the start of school, a new season of public meetings, worries about fuel costs, the rundown to the presidential election and local races.

So the mystery remains! Who decided that August 1st should be "World Wide Web Day", why that date and how did it spread so widely without leaving a clear origin story?

If your research skills are up to the challenge, join the challenge!

Tags: history, internet-archive, w3c, web, mastodon, slop

Quoting quora.com/robots.txt

2024-03-19T23:09:31+00:00

People share a lot of sensitive material on Quora - controversial political views, workplace gossip and compensation, and negative opinions held of companies. Over many years, as they change jobs or change their views, it is important that they can delete or anonymize their previously-written answers.

We opt out of the wayback machine because inclusion would allow people to discover the identity of authors who had written sensitive answers publicly and later had made them anonymous, and because it would prevent authors from being able to remove their content from the internet if they change their mind about publishing it.

— quora.com/robots.txt

Tags: internet-archive, robots-txt, quora

Internet Archive Software Library: Flash

2020-11-19T21:19:51+00:00

Internet Archive Software Library: Flash

A fantastic new initiative from the Internet Archive: they’re now archiving Flash (.swf) files and serving them for modern browsers using Ruffle, a Flash Player emulator written in Rust and compiled to WebAssembly. They are fully interactive and audio works too. Considering the enormous quantity of creative material released in Flash over the decades this helps fill a big hole in the Internet’s cultural memory.

Via Jason Scott

Tags: flash, internet-archive, jason-scott, rust, webassembly

Usage of ARIA attributes via HTTP Archive

2018-07-12T03:16:26+00:00

Usage of ARIA attributes via HTTP Archive

A neat example of a Google BigQuery query you can run against the HTTP Archive public dataset (a crawl of the “top” websites run periodically by the Internet Archive, which captures the full details of every resource fetched) to see which ARIA attributes are used the most often. Linking to this because I used it successfully today as the basis for my own custom query—I love that it’s possible to analyze a huge representative sample of the modern web in this way.

Tags: aria, http, internet-archive, big-data

Elaborate Halloween Costume Tips from a 19th-Century Guide to Fancy Dress

2017-10-26T14:01:40+00:00

Elaborate Halloween Costume Tips from a 19th-Century Guide to Fancy Dress

The gilded age had some ridiculous parties. Here are highlights of the most popular costume guide of the era, now available on the Internet Archive.

Via But, what are we to wear? | MetaFilter

Tags: history, internet-archive

Recovering missing content from the Internet Archive

2017-10-08T19:08:57+00:00

When I restored my blog last weekend I used the most recent SQL backup of my blog’s database from back in 2010. I thought it had all of my content from before I started my 7 year hiatus, but in watching the 404 logs I started seeing the occasional hit to something that really should have been there but wasn’t. Turns out the SQL backup I was working from was missing some content.

Thank goodness then for the Wayback Machine at the Internet Archive! I tried some of the missing URLs there and found they had been captured and preserved. But how to get them back?

A quick search turned up wayback-machine-downloader, an open-source Ruby script that claims to be able to Download an entire website from the Internet Archive Wayback Machine. I gem installed it and tried it out (after some cargo cult incantations to work around some weird certificate errors I was seeing)

rvm osx-ssl-certs update all
gem update --system
gem install wayback_machine_downloader

wayback_machine_downloader http://simonwillison.net/

And it worked! I left it running overnight and came back to a folder containing 18,952 HTML files, neatly arranged in a directory structure that matched my site:

$ find . | more
.
./simonwillison.net
./simonwillison.net/2002
./simonwillison.net/2002/Aug
./simonwillison.net/2002/Aug/1
./simonwillison.net/2002/Aug/1/cetis
./simonwillison.net/2002/Aug/1/cetis/index.html
./simonwillison.net/2002/Aug/1/cssSelectorsTutorial
./simonwillison.net/2002/Aug/1/cssSelectorsTutorial/index.html
...

I tarred them up into an archive and backed them up to Dropbox.

Next challenge: how to restore the missing content?

I’m a recent and enthusiastic adopter of Jupyter notebooks. As a huge fan of development in a REPL I’m shocked I was so late to this particular party. So I fired up Jupyter and used it to start playing with the data.

Here’s the final version of my notebook. I ended up with a script that did the following:

Load in the full list of paths from the tar archive, and filter for just the ones matching the /YYYY/Mon/DD/slug/ format used for my blog content
Talk to my local Django development environment and load in the full list of actual content URLs represented in that database.
Calculate the difference between the two - those are the 213 items that need to be recovered.
For each of those 213 items, load the full HTML that had been saved by the Internet Archive and feed it into the BeautifulSoup HTML parsing library.
Detect if each one is an entry, a blogmark or a quotation. Scrape the key content out of each one based on the type.
Scrape the tags for each item, using this delightful one-liner: [a.text for a in soup.findAll('a', {'rel': 'tag'})]
Scrape the comments for each item separately. These were mostly spam, so I haven’t yet recovered these for publication (I need to do some aggressive spam filtering first). I have however stashed them in the database for later processing.
Write all of the scraped data out to a giant JSON file and upload it to a gist (a nice cheap way of giving it a URL).

Having executed the above script, I now have a JSON file containing the parsed content for all of the missing items found in the Wayback Machine. All I needed then was a script which could take that JSON and turn it into records in the database. I implemented that as a custom Django management command and deployed it to Heroku.

Last step: shell into a Heroku dyno (using heroku run bash) and run the following:

./manage.py import_blog_json \
    --url_to_json=https://gist.github.com/simonw/5a5bc1f58297d2c7d68dd7448a4d6614/raw/28d5d564ae3fe7165802967b0f9c4eff6091caf0/recovered-blog-content.json \
    --tag_with=recovered

The result: 213 recovered items (which I tagged with recovered so I could easily browse them). Including the most important entry on my whole site, my write-up of my wedding!

So thank you very much to the Internet Archive team, and thank you Hartator for your extremely useful wayback-machine-downloader tool.

Tags: beautifulsoup, internet-archive, urls, jupyter

tr.im is "discontinuing service"

2009-08-10T11:06:41+00:00

tr.im is "discontinuing service"

“However, all tr.im links will continue to redirect, and will do so until at least December 31, 2009.Your tweets with tr.im URLs in them will not be affected.”—these statements seem to contradict themselves. Will tr.im URLs in tweets stop working after December 31st or not? Any chance they could hand the domain over to the Internet Archive? At any rate, this is exactly why centralised URL shorteners are a harmful trend.

Tags: internet-archive, redirects, trim, twitter, urls, urlshorteners

A new leaf.

2009-04-28T00:55:18+00:00

A new leaf.

George Oates is now heading up the Open Library project at the Internet Archive. Sounds like a perfect match.

Tags: george-oates, internet-archive, openlibrary

TinyURL - Archiveteam

2009-04-03T23:11:39+00:00

TinyURL - Archiveteam

Excellent: the Internet Archive are crawling TinyURL (and hopefully other URL shortening services as well). The wiki page was created back in January. UPDATE from comments: Archiveteam are a separate organisation from the Internet Archive.

Tags: archive, archiveteam, internet-archive, tinyurl

Quoting Me

2009-03-08T14:59:34+00:00

The Internet Archive should actively partner with bit.ly / tinyurl.com / icanhaz.com etc. and maintain a mirror database of their redirects

— Me, on Twitter

Tags: bitly, icanhaz, internet-archive, me, tinyurl, twitter, urlshorteners

My Future of Web Apps talk as a slidecast

2007-03-12T23:57:25+00:00

The team at Carson Systems have a pretty quick turnaround on their podcasts; they've had full recordings of every speaker up for a few days now. I spent a bunch of time over the weekend splicing the recording of my talk together with my slides, and the result is now available at The Future of OpenID (a slidecast).

I managed to crunch it down to a 41.2 MB H.264 MPEG file; there is also a Flash video version are available on the Internet Archive.

A quick aside: I'm hosting the main video file in the Internet Archive's Open Source Movies collection. They actively encourage people to submit their own digital artifacts, and once you've uploaded something they'll automatically create thumbnails, derive an FLV version, mirror it to a bunch of places and much more besides. If you've got a large video to distribute this is a great way to share it.

Tags: future-of-web-apps, internet-archive, openid, presenting, slidecast, my-talks