Simon Willison's Weblog: adrian-holovaty

Talk Python: Celebrating Django's 20th Birthday With Its Creators

2025-08-29T20:02:50+00:00

Talk Python: Celebrating Django's 20th Birthday With Its Creators

I recorded this podcast episode recently to celebrate Django's 20th birthday with Adrian Holovaty, Will Vincent, Jeff Triplet, and Thibaud Colas.

We didn’t know that it was a web framework. We thought it was a tool for building local newspaper websites. [...]

Django’s original tagline was ‘Web development on journalism deadlines’. That’s always been my favorite description of the project.

Tags: adrian-holovaty, django, python, podcast-appearances

Happy 20th birthday Django! Here's my talk on Django Origins from Django's 10th

2025-07-13T18:47:13+00:00

Today is the 20th anniversary of the first commit to the public Django repository!

Ten years ago we threw a multi-day 10th birthday party for Django back in its birthtown of Lawrence, Kansas. As a personal celebration of the 20th, I'm revisiting the talk I gave at that event and writing it up here.

Here's the YouTube video. Below is a full transcript, plus my slides and some present-day annotations.

Django Origins (and some things I have built with Django)

Presented 11th July 2015 at Django Birthday in Lawrence, Kansas

My original talk title, as you'll see on your programs, was "Some Things I've Built with Django." But then I realized that we're here in the birthplace of Django, celebrating the 10th birthday of the framework, and nobody's told the origin story yet. So, I've switched things around a little bit. I'm going to talk about the origin story of Django, and then if I have time, I'll do the self-indulgent bit and talk about some of the projects I've shipped since then.

I think Jacob's introduction hit on something I've never really realized about myself. I do love shipping things. The follow-up and the long-term thing I'm not quite so strong on. And that came to focus when I was putting together this talk and realized that basically every project I'm going to show you, I had to dig out of the Internet Archive.

Ten years on from writing this talk I'm proud that I've managed to overcome my weakness in following-up - I'm now actively maintaining a bewildering array of projects, having finally figured out how to maintain things in addition to creating them!

But that said, I will tell you the origin story of Django.

For me, the story starts very much like Jacob's. I was reading RSS feeds back in 2003, and I saw this entry on Adrian's blog, talking about a job opportunity for a web programmer or developer in Lawrence, Kansas.

Now, I was in England. I was at university. But my university had just given me the opportunity to take a year abroad, to take a year out to do an internship year in industry. My girlfriend at the time was off to Germany to do her year in industry. So I was like, well, you know, do I stay at university? And then this comes along.

So I got in touch with Adrian and said, you know, could this work as a year-long internship instead? And he was reading my blog and I was reading his blog, and we knew that we aligned on a bunch of things. So we thought we'd give it a go.

Now, if you look through this job ad, you'll see that this is all about expert knowledge of PHP and experience designing and maintaining databases, particularly MySQL. So this was a PHP and MySQL gig.

But when I arrived in Kansas, we quickly realized that we were both kind of over PHP. You know, we'd both built substantial systems in PHP, and we were running up against the limits of what you can do in PHP and have your code still be clean and maintainable.

And at the same time, we were both reading Mark Pilgrim's blog (archive link). Mark Pilgrim had been publishing Dive into Python and making a really strong case for why Python was a great web language.

So we decided that this was the thing we wanted to do. But we both had very strong opinions about how you should build websites. Things like URL design matters, and we care about the difference between get and post, and we want to use this new thing called CSS to lay out our web pages. And none of the existing Python web frameworks really seemed to let us do what we wanted to do.

Now, before I talk more about that, I'll back up and talk about the organization we're working for, the Lawrence Journal World.

David gave a great introduction to why this is an interesting organization. Now, we're talking about a newspaper with a circulation of about 10,000, like a tiny newspaper, but with a world-class newsroom, huge amounts of money being funneled into it, and like employing full-time software developers to work at a local newspaper in Kansas.

And part of what was going on here was this guy. This is Rob Curley. He's been mentioned a few times before already.

And yeah, Rob Curley set this unofficial mission statement that we "build cool shit". This is something that Adrian would certainly never say. It's not really something I'd say. But this is Rob through and through. He was a fantastic showman.

And this was really the appeal of coming out to Lawrence, seeing the stuff they'd already built and the ambitions they had.

This is Lawrence.com. This is actually the Lawrence.com written in PHP that Adrian had built as the sole programmer at the Lawrence Journal World. And you should check this out. Like, even today, this is the best local entertainment website I have ever seen. This has everything that is happening in the town of Lawrence, Kansas population, 150,000 people. Every gig, every venue, all of the stuff that's going on.

And it was all written in PHP. And it was a very clean PHP code base, but it was really stretching the limits of what it's sensible to do using PHP 4 back in 2003.

So we had this goal when we started using Python. We wanted to eventually rebuild Lawrence.com using Python. But in order to get there, we had to first build -- we didn't even know it was a web framework. We called it the CMS.

And so when we started working on Django, the first thing that we shipped was actually this website. We had a lot of the six-news Lawrence. This is the six-news Lawrence -- six-news is the TV channel here -- six-news Lawrence weather page.

And I think this is pretty cool. So Dan Cox, the designer, was a fantastic illustrator. We actually have this illustration of the famous Lawrence skyline with each panel could be displayed with different weather conditions depending on the weather.

And in case you're not from Kansas, you might not have realized that the weather is a big deal here. You know, you have never seen more excited weathermen than when there's a tornado warning and they get to go on local news 24 hours a day giving people updates.

So we put the site live first. This was the first ever Django website. We then did the rest of the 6 News Lawrence website.

And this -- Adrian reminded me this morning -- the launch of this was actually delayed by a week because the most important feature on the website, which is clearly the photograph of the news people who are on TV, they didn't like their hairdos. They literally told us we couldn't launch the website until they'd had their hair redone, had the headshots retaken, had a new image put together. But, you know, image is important for these things.

So anyway, we did that. We did six-news Lawrence. And by the end of my year in Kansas, Adrian had rewritten all of Lawrence.com as well.

So this is the Lawrence.com powered by Django. And one thing I think is interesting about this is when you talk to like David Heinemeier Hansson about Rails, he'll tell you that Rails is a framework that was extracted from Basecamp. They built Basecamp and then they pulled out the framework that they used and open sourced it.

I see Django the other way around. Django is a framework that was built up to create Lawrence.com. Lawrence.com already existed. So we knew what the web framework needed to be able to do. And we just kept on iterating on Django or the CMS until it was ready to produce this site here.

And for me, the moment I realized that we were onto something special was actually when we built this thing. This is a classic Rob Curley project. So Rob was the boss. He had the crazy ideas and he didn't care how you implemented them. He just wanted this thing done.

And he came to us one day and said, you know, the kids' little league season is about to start. Like kids playing softball or baseball. Whatever the American kids with bats thing is. So he said, kids' little league season is about to start. And we are going to go all out.

I want to treat these kids like they're the New York Yankees. We're going to have player profiles and schedules and photos and results.

And, you know, we're going to have the ability for parents to get SMS notifications whenever their kid scores.

And we're going to have 360 degree, like, interactive photos of all of the pitches in Lawrence, Kansas, that these kids are playing games on.

They actually did send a couple of interns out with a rig to take 360 degree virtual panoramas of Fenway Park and Lawrence High School and all of these different places.

And he said -- and it starts in three days. You've got three days to put this all together.

And we pulled it off because Django, even at that very early stage, had all of the primitives you needed to build 360 degree interactives. That was all down to the interns. But we had all of the pieces we needed to pull this together.

So when we were working on it back then, we called it the CMS.

A few years ago, Jacob found a wiki page with some of the names that were being brainstormed for the open source release. And some of these are great. There's Brazos -- I don't know where that came from -- Webbing, Physique, Anson.

This is my favorite name. I think this is what I proposed -- is the Tornado Publishing System.

And the reason is that I was a really big fan of Office Space. And if we had the Tornado, we could produce TPS reports, which I thought would be amazing.

But unfortunately, this being Kansas, the association of Tornadoes isn't actually a positive one.

Private Dancer, Physgig, Lavalia, Pithy -- yeah. I'm very, very pleased that they picked the name that they did.

So one of our philosophies was build cool shit. The other philosophy we had was what we called "Wouldn't it be cool if?"

So there were no user stories or careful specs or anything. We'd all sit around in the basement and then somebody would go "Wouldn't it be cool if...", and they'd say something. And if we thought it was a cool idea, we'd build it and we'd ship it that day.

And my favorite example of "Wouldn't it be cool if?" -- this is a classic Adrian one -- is "Wouldn't it be cool if the downloads page on Lawrence.com featured MP3s you could download of local bands?" And seeing as we've also got the schedule of when the bands are playing, why don't we feature the audio from bands who you can go and see that week?

So this page will say, "OK Jones are playing on Thursday at the Bottleneck. Get their MP3. Listen to the radio station." We had a little MP3 widget in there. Go and look at their band profile. All of this stuff.

Really, these kinds of features are what you get when you take 1970s relational database technology and use it to power websites, which -- back in 2003, in the news industry -- still felt incredibly cutting edge. But, you know, it worked.

And that philosophy followed me through the rest of my career, which is sometimes a good idea and often means that you're left maintaining features that seemed like a good idea at the time and quickly become a massive pain!

After I finished my internship, I finished my degree in England and then ended up joining up with Yahoo. I was actually working out of the Yahoo UK office but for a R&D team in the States. I was there for about a year and a half.

One of the things I learned is that you should never go and work for an R&D team, because the problem with R&D teams is you never ship. I was there for a year and a half and I basically have nothing to show for it in terms of actual shipped features.

We built some very cool prototypes. And actually, after I left, one of the projects I worked on, Yahoo FireEagle, did end up getting spun out and turned into a real product.

But there is one project -- the first project I built at Yahoo using Django that I wanted to demonstrate. This was for Yahoo's internal hack day. And so Tom Coates and myself, who were working together, we decided that we were going to build a mashup, because it was 2005 and mashups were the cutting edge of computer science.

So we figured, OK, let's take the two most unlikely Yahoo products and combine them together and see what happens. My original suggestion was that we take Yahoo Dating and Yahoo Pets. But I was told that actually there was this thing called Dogster and this other thing called Catster, which already existed and did exactly that.

So the next best thing, we went for Yahoo News and Yahoo Horoscopes. And what we ended up building -- and again, this is the first Django application within Yahoo -- was Yahoo Astronewsology.

And the idea was you take the news feed from Yahoo News, you pull out anything that looks like it's a celebrity's name, look up their birth date, use that to look up their horoscope, and then combine them on the page. And in a massive stroke of luck, we built this the week that Dick Cheney shot his friend in the face while out hunting.

Dick Cheney's horoscope for that week says, "A very close friend who means a great deal to you has found it necessary to go out of their way to tick you off. You're not just angry, you're furious. Before you let go and let them have it, be sure you're right. Feeling righteous is far better than feeling guilty."

And so if Dick Cheney had only had only been reading his horoscopes, maybe that whole situation would have ended very differently.

So after Yahoo, I spent a while doing consulting and things, mainly around OpenID because I was determined to make OpenID work. I was absolutely convinced that if OpenID didn't take off, just one company would end up owning single sign-on for the entire internet, and that would be a total disaster.

And with hindsight, it didn't quite happen. Facebook login looked like it was going to do that a few years ago, but these days there's enough variety out there that I don't feel like we all have to submit to our Facebook masters.

But, you know, I was enjoying freelancing and consulting and so on. And then I ended up going for coffee with somebody who worked for The Guardian.

I'm sure you've all heard of The Guardian. It's one of England's most internationally focused newspapers. It's a very fine publication. And I realized that I really missed working in a newsroom environment. And I was incredibly jealous of people like Adrian, who'd gone off to the Washington Post and was doing data journalism there, and Derek Willis as well, who bounced from the Post and The New York Times. There was all of this cool data journalism stuff going on.

And The Guardian's pitch was basically, we've been building a CMS from scratch in Java with a giant team of engineers, and we've built it and it's really cool, but we're not shipping things quickly. We want to start exploring this idea of building things much faster to fit in with the news cycle.

And that was a very, very tempting thing for me to get involved with. So I went to work for The Guardian.

And The Guardian have a really interesting way of doing onboarding of new staff. The way they do it is they set you up on coffee dates with people from all over the organization. So one day you'll be having coffee with somebody who sells ads, and the next day it'll be the deputy editor of the newsroom, and the next day it'll be a journalist somewhere. And each of these people will talk to you and then they'll suggest other people for you to meet up with. So over the first few weeks that you're there, you meet a huge variety of people.

And time and time again, as I was talking to people, they were saying, "You know what? You should go and talk to Simon Rogers, this journalist in the newsroom."

This is Simon Rogers. I went down to talk to him, and we had this fascinating conversation. So Simon is a journalist. He worked in the newsroom, and his speciality was gathering data for The Guardian's infographics. Because they are in the paper. They post, they have graphs and charts and all sorts of things like that that they publish.

It turns out that Simon was the journalist who knew how to get that data out of basically any source you can imagine. If you wanted data, he would make some phone calls, dig into some government contacts and things, and he'd get those raw numbers. And all of the other journalists thought he was a bit weird, because he liked hanging out and editing Excel spreadsheets and stuff.

So I said to him halfway through this conversation, "Just out of interest, what do you do with those Excel spreadsheets?" And he's like, "Oh, I keep them all on my hard drive." And showed me this folder with hundreds and hundreds of meticulously researched, properly citable news quality spreadsheets full of data about everything you could imagine. And they lived on his hard drive and nowhere else.

And I was like, "Have you ever talked to anyone in the engineering department upstairs?" And we made this connection.

And so from then on, we had this collaboration going where he would get data and he'd funnel it to me and see if we could, see if I or someone else in the engineering department at Guardian could do something fun with it.

And so that was some of the most rewarding work of my career, because it's journalism, you know, it's news, it's stuff that matters. The deadlines are ridiculous. If a news story breaks and it takes you three weeks to turn around a piece of data journalism around it, why did you even bother? And it's perfect for applying Django to.

So the first story I got to work on at the Guardian was actually one of the early WikiLeaks things. This is before WikiLeaks was like massively high profile. But quite early on, WikiLeaks leaked a list of all of the members of the British National Party, basically the British Nazis. They leaked a list of all of their names and addresses.

And the Guardian is an ethical newspaper, so we're not going to just publish 18,000 people's names and addresses. But we wanted to figure out if there was something we could do that would make use of that data but wouldn't be violating anyone's individual privacy.

And so what we did is we took all of the addresses, geocoded them, figured out which parliamentary constituency they lived in, and used that to generate a heat map that's actually called a choropleth map, I think, of the UK showing where the hotspots of BNP activity were.

And this works because in the UK a parliamentary constituency is, they're designed to all have around about the same population. So if you just like make the color denser for the larger numbers of BNP members, you get this really interesting heat map of the country.

And what was really cool about this is that I did this using SVG, because we have an infographics department with Illustrator who are good at working with SVG. And it's very easy with an SVG file with the right class names on things to set colors on different regions.

And because we produced it in SVG, we could then hand it over to the print department, and the next day it was out in the paper. It was like a printed thing on paper, on like dead trees distributed all over the country, which I thought was super cool.

So that was the first data journalism project that we did at The Guardian. And it really helped prove that given the right data sets and like the right tools and a bit of freedom, you can do some really cool things.

The first few times I did this, I did it by hand. Then we had The Guardian's first hack day and I was like, well okay, I'm going to build a little self-service tool for our infographics journalists to like dump in a bunch of CSV numbers and get one of these maps out of it.

So I built this tool. I didn't have anywhere official to deploy it, so I just ran it on my Linux desktop underneath my desk. And they started using it and putting things in the paper and I kind of forgot about it. And every now and then I get a little feature request.

A few years after I left The Guardian, I ran into someone who worked there. And he was like, yeah, you know that thing that you built? So we had to keep your desktop running for six months after you left. And then we had to like convert it into a VMware instance. And as far as I know, my desktop is still running as a VMware instance somewhere in The Guardian.

Which ties into the Simon database, I guess. The hard thing is building stuff is easy. Keeping it going it turns out is surprisingly difficult.

This was my favorite project at The Guardian. There was a scandal in the UK a few years ago where it turned out that UK members of parliament had all been fiddling their expenses.

And actually the background on this is that they're the lowest paid MPs anywhere in Europe. And it seems like the culture had become that you become an MP and on your first day somebody takes you aside and goes, look, I know the salary is terrible. But here's how to fill your expenses and make up for it.

This was a scandal that was brewing for several years. The Guardian had actually filed freedom of information requests to try and get these expense reports. Because they were pretty sure something dodgy was going on. The government had dragged their heels in releasing the documents.

And then just when they were a month before they finally released the documents, a rival newspaper, the Telegraph, managed to get hold of a leaked copy of all of these expenses. And so the Telegraph had 30 days lead on all of the other newspapers to dig through and try and find the dirt.

So when they did release the expenses 30 days later, we had a race on our hands because we needed to analyze 20,000 odd pages of documents. Actually, here it says 450,000 pages of documents in order to try and find anything left that was newsworthy.

And so we tackled this with crowdsourcing. We stuck up a website. We told people, we told Guardian readers, come to this website, hit the button, we'll show you a random page from someone's expenses. And then you can tell us if you think it's not interesting, interesting, or we should seek an investigative reporter on it.

And one of the smartest things we did with this is we added a feature where you could put in your postcode, we'd figure out who your MP was, and then we would show you their smug press photo. You know, their smug face next to all of their expense claims that they'd filed.

And this was incredibly effective. People were like, "Ooh, you look so smug. I'm going to get you." And once we put this up, and within 18 hours, our community had burned through hundreds of thousands of pages of expense documents trying to find this stuff.

And again, this was built in Django. We had, I think, five days warning that these documents are coming out. And so it was a total, like, I think I built a proof of concept on day one. That was enough to show that it was possible. So I got a team with a designer and a couple of other people to help out. And we had it ready to go when the document dump came out on that Friday.

And it was pretty successful. We dug up some pretty interesting stories from it. And it was also just a fantastic interactive way of engaging our community. And, you know, the whole crowdsourcing side of it was super fun.

So I guess the thing I've learned from that is that, oh, my goodness, it's fun working for newspapers. And actually, if you -- the Lawrence Journal world, sadly, no longer has its own technology team. But there was a period a few years ago where they were doing some cracking data journalism work. Things like tracking what the University of Kansas had been using its private jet for, and letting people explore the data around that and so on.

The other thing we did at the Guardian, this is going back to Simon Rogers, is he had all of these spreadsheets on his hard drive. And we're like, okay, we should really try and publish this stuff as raw data. Because living on your hard drive under your head is a crying shame.

And the idea we came up with was essentially to start something we called the Data blog and publish them as Google spreadsheets. You know, we spent a while thinking, well, you know, what's the best format to publish these things in? And we're like, well, they're in Excel. Google spreadsheets exists and it's pretty good. Let's just put a few of them up as Google sheets and see what people do with them.

And it turns out that was enough to build this really fun community of data nerds around the Guardian's data blog who would build their own visualizations. They'd dig into the data. And it meant that we could get all sorts of -- like, we could get so much extra value from the work that we were already doing to gather these numbers for the newspaper. That stuff was super fun.

Now, while I was working at the Guardian, I also got into the habit of building some projects with my girlfriend at the time, now my wife Natalie. So Natalie and I have skill sets that fit together very nicely. She's a front-end web developer. I do back-end Django stuff. I do just enough ops to be dangerous. And so between the two of us, we can build websites.

The first things we worked on together is a site which I think some people here should be familiar with, called Django People. The idea was just, you know, the Django community appears to be quite big now. Let's try and get people to stick a pin on a map and tell us where they are.

Django People still exists today. It's online thanks to a large number of people constantly bugging me at Django Cons and saying, look, just give us the code and the data and we'll get it set up somewhere so it can continue to work. And that's great. I'm really glad I did that because this is the one project that I'm showing you today which is still available on the web somewhere. (2025 update: the site is no longer online.)

But Django People was really fun. And the thing we learned from this, my wife and I, is that we can work together really well on things.

The other side project we did was much more of a collaborative effort. Again, this no longer exists, or at least it's no longer up on the web. And I'm deeply sad about this because it's my favorite thing I'm going to show you.

But before I show you the project, I'll show you how we built it. We were at a BarCamp in London with a bunch of our new friends and somebody was showing photographs of this Napoleonic sea fortress that they had rented out for the weekend from an organization in the UK called the Landmark Trust, who basically take historic buildings and turn them into vacation rentals as part of the work to restore them.

And we were like, "Oh, wouldn't it be funny if we rented a castle for a week and all of us went out there and we built stuff together?" And then we were like, "That wouldn't be funny. That would be freaking amazing."

So we rented this place. This is called Fort Clonque. It's in the Channel Islands, halfway between England and France. And I think it cost something like $2,000 for the week, but you split that between a dozen people and it's like youth hostel prices to stay in a freaking fortress.

So we got a bunch of people together and we went out there and we just spent a week. We called it /dev/fort. We spent a week just building something together.

And the thing we ended up building was called Wildlife Near You. And what Wildlife Near You does is it solves the eternal question, "Where is my nearest llama?"

Once again, this is a crowdsourcing system. The idea is that you go to wildlifenearyou.com and you've just been on a trip to like a nature park or a zoo or something. And so you create a trip report saying, "I went to the Red Kite feeding station and I saw a common raven and a common buzzard and a red kite." And you import any of your photos from Flickr and so forth.

And you build up this profile saying, "Here are all the places I've been and my favorite animals and things I've seen."

And then once we've got that data set, we can solve the problem. You can say, "Search for llamas near Brighton." And it'll say, "Your nearest llama is 18 miles away and it'll show you pictures of llamas and all of the llama things."

And we have species pages. So here's the red panda page. 17 people love red pandas. You can see them at Taronga Zoo.

And then our most viral feature was we had all of these photos of red pandas, but how do we know which is the best photo of a red panda that we should highlight on the red panda page? So we basically built Hot or Not for photographs of wildlife.

So it's like, "Which marmot photo is better?" And you say, "Well, clearly the one on the right." And it's like, "Okay, which skunk photo is better?"

I was looking at the logs and people would go through hundreds and hundreds of photos. And you'd get scores and you can see, "Oh, wow, my marmot photo is the second best marmot photo on the whole website."

So that was really fun. And then we eventually took it a step further and said, "Okay, well, this is really fun, but this is a website that you have to type on, right?" And meanwhile, mobile phones are now getting HTML5 geolocation and stuff. So can we go a step further?

So we built owlsnearyou.com. And what owlsnearyou.com does is you type in the location, and it says, "Your nearest owl is 49 miles away." It's a spectacle owl at London Zoo. It was spotted one year ago by Natalie.

And if you went here on a mobile phone-- If you went here on a device that supported geolocation, it doesn't even ask you where you live. It's just like, "Oh, okay, here's your nearest owl."

And I think we shipped lions near you and monkeys near you and a couple of other domains, but owlsnearyou.com was always my favorite.

So looking at this now, we should really get this stuff up and running again. It was freaking amazing. Like, this for me is the killer app of all killer apps.

(We did eventually bring this idea back as www.owlsnearme.com, using data from iNaturalist - that's online today.)

So there have actually been a bunch of Devforts since then. One of the things we learned from Devfort is that building applications-- If you want to do a side project, doing one with user accounts and logins and so on, it's a freaking nightmare. It actually took us almost a year after we finished on the fort to finally ship Wildlife Near You because there were so many complexities. And then we had to moderate it and keep an eye on it and so on.

So if you look at the more recent Devforts, they've taken that to heart. And now they try and ship things which just work and don't require ongoing users logging in and all of that kind of rubbish.

But one of the other projects I wanted to show you that came out of a Devfort was something called Bugle. And the idea of Bugle is Bugle is a Twitter-like application for groups of hackers collaborating in a castle, fort, or other defensive structure who don't have an internet connection.

This was basically to deal with Twitter withdrawal when we were all on the fort together and we had an internal network. So Bugle, looking at it now, we could have been Slack! We could have been valued at $2 billion.

Yeah, Bugle is like an internal Twitter clone with a bunch of extra features like it's got a paste bin and to-do lists and all sorts of stuff like that.

And does anyone here know Ben Firshman? I think quite a few people do. Excellent. So Ben Firshman was out on a Devfort and I did a "Wouldn't it be cool if" on him. I said, "Wouldn't it be cool if all of our Twitter apps and our phones talked to Bugle instead on the network?"

And so if you go and look on GitHub, I bet this doesn't work anymore. But he did add magic Twitter support where you could run a local DNS server, redirect Twitter to Bugle and we cloned, he cloned enough of the Twitter API that like Twitter apps would work and it would be able to Bugle instead.

We wanted to do a Devfort in America. You don't really have castles and forts that you can rent for the most part. If anyone knows of one, please come and talk to me because there's a distinct lack of defensible structures at least of the kind that we are used to back in Europe.

So I'm running out of time, but that's OK because the most recent project, Lanyrd, is something which most people here have probably encountered.

I will tell a little bit of the backstory of Lanyrd because it's kind of fun.

Lanyrd was a honeymoon project.

Natalie and I got married. The wildlife near you influence affected our wedding - it was a freaking awesome wedding! You know, in England, you can get a man with a golden eagle and a barn owl and various other birds to show up for about $400 for the day. And then you get to take photos like this.

So anyway, we got married, we quit our jobs, I had to leave the Guardian because we wanted to spend the next year or two of our lives just traveling around the world, doing freelancing work on our laptops and so on.

We got as far as Morocco, we were six months in, when we contracted food poisoning in Casablanca and we were too sick to keep on travelling, so we figured we needed to like, you know, and it was also Ramadan, so it was really hard to get food and stuff. So we rented an apartment for two weeks and said, "Okay, well, since we're stuck for two weeks, let's like finish that side project we've been talking about and ship it and see if anyone's interested."

So we shipped Lanyrd, which was built around the idea of helping people who use Twitter find conferences and events to go to. What we hadn't realised is that if you build something around Twitter, especially back in 2010, it instantly goes viral amongst people who use Twitter.

So that ended up cutting our honeymoon short, and we actually applied for Y Combinator from Egypt and ended up spending three years building a startup and like hiring people and doing that whole thing.

(Natalie wrote more about our startup in Lanyrd: from idea to exit - the story of our startup.)

The only thing I'll say about that is everything in the... Startups have to give the impression that everything's super easy and fun and cool all the time, because people say, "How's your startup going?" And the only correct answer is, "Oh man, it's amazing. It's doing so well." Because everyone has to lie about the misery, pain, anguish and stress that's happening behind the scenes.

So it was a very interesting three years, and we built some cool stuff and we learnt a lot, and I don't regret it, but do not take startups lightly.

So a year and a half ago, we ended up selling Lanyrd to Eventbrite and moving out to San Francisco. And at Eventbrite, I've been mostly on the management team building side of things, but occasionally managing to sneak some code out as well.

The one thing I want to show you from Eventbrite, because I really want to open source this thing, is again at Hack Day, we built a tool called the Tikibar, which is essentially like the Django debug toolbar, but it's designed to be run in production. Because the really tough things to debug don't happen in your dev environment. They happen in production when you're hitting a hundred million row database or whatever.

And so the Tikibar is designed to add as little overhead as possible, but to still give you detailed timelines of SQL queries that are executing and service calls and all of that kind of stuff. It's called the Tikibar because I really like Tikibars.

And the best feature is if a page takes over 500 milliseconds to load, the eyes on the Tiki God glow red in disapproval at you.

If anyone wants a demo of that, come and talk to me. I would love to get a few more instrumentation hooks into Django to make this stuff easier.

(The Tikibar was eventually open sourced as eventbrite/tikibar on GitHub.)

This has been a whistle-stop tour of the highlights of my career working with Django.

And actually, in putting this presentation together, I realized that really it's that Rob Curley influence from all the way back in 2003. The reason I love Django is it makes it really easy to build cool shit and to ship it. And, you know, swearing aside, I think that's a reasonable moral to take away from this.

Colophon

I put this annotated version of my 10 year old talk together using a few different tools.

I fetched the audio from YouTube using yt-dlp:

yt-dlp -x --audio-format mp3 \
  "https://youtube.com/watch?v=wqii_iX0RTs"

I then ran the mp3 through MacWhisper to generate an initial transcript. I cleaned that up by pasting it into Claude Opus 4 with this prompt:

Take this audio transcript of a talk and clean it up very slightly - I want paragraph breaks and tiny edits like removing ums or "sort of" or things like that, but other than that the content should be exactly as presented.

I converted a PDF of the slides into a JPEG per page using this command (found with the llm-cmd plugin):

pdftoppm -jpeg -jpegopt quality=70 django-birthday.pdf django-birthday

Then I used my annotated presentations tool (described here) to combine the slides and transcript, making minor edits and adding links using Markdown in that interface.

Tags: adrian-holovaty, devfort, django, history, jacob-kaplan-moss, lawrence, lawrence-com, lawrence-journal-world, python, my-talks, the-guardian, annotated-talks

Adding a feature because ChatGPT incorrectly thinks it exists

2025-07-07T15:29:38+00:00

Adding a feature because ChatGPT incorrectly thinks it exists

Adrian Holovaty describes how his SoundSlice service saw an uptick in users attempting to use their sheet music scanner to import ASCII-art guitar tab... because it turned out ChatGPT had hallucinated that as a feature SoundSlice supported and was telling users to go there!

So they built that feature. Easier than convincing OpenAI to somehow patch ChatGPT to stop it from hallucinating a feature that doesn't exist.

Adrian:

To my knowledge, this is the first case of a company developing a feature because ChatGPT is incorrectly telling people it exists. (Yay?)

Via Hacker News

Tags: adrian-holovaty, ai, openai, generative-ai, chatgpt, llms, ai-ethics, hallucinations

State-of-the-art music scanning by Soundslice

2024-06-20T04:37:28+00:00

State-of-the-art music scanning by Soundslice

It's been a while since I checked in on Soundslice, Adrian Holovaty's beautiful web application focused on music education.

The latest feature is spectacular. The Soundslice music editor - already one of the most impressive web applications I've ever experienced - can now import notation directly from scans or photos of sheet music.

The attention to detail is immaculate. The custom machine learning model can handle a wide variety of notation details, and the system asks the user to verify or correct details that it couldn't perfectly determine using a neatly designed flow.

Free accounts can scan two single page documents a month, and paid plans get a much higher allowance. I tried it out just now on a low resolution image I found on Wikipedia and it did a fantastic job, even allowing me to listen to a simulated piano rendition of the music once it had finished processing.

It's worth spending some time with the release notes for the feature to appreciate how much work they've out into improving it since the initial release.

If you're new to Soundslice, here's an example of their core player interface which syncs the display of music notation to an accompanying video.

Adrian wrote up some detailed notes on the machine learning behind the feature when they first launched it in beta back in November 2022.

OMR [Optical Music Recognition] is an inherently hard problem, significantly more difficult than text OCR. For one, music symbols have complex spatial relationships, and mistakes have a tendency to cascade. A single misdetected key signature might result in multiple incorrect note pitches. And there’s a wide diversity of symbols, each with its own behavior and semantics — meaning the problems and subproblems aren’t just hard, there are many of them.

Tags: adrian-holovaty, machine-learning, music, ocr, ai

Let websites framebust out of native apps

2022-08-10T22:29:42+00:00

Let websites framebust out of native apps

Adrian Holovaty makes a compelling case that it is Not OK that we allow native mobile apps to embed our websites in their own browsers, including the ability for them to modify and intercept those pages (it turned out today that Instagram injects extra JavaScript into pages loaded within the Instagram in-app browser). He compares this to frame-busting on the regular web, and proposes that the X-Frame-Options: DENY header which browsers support to prevent a page from being framed should be upgraded to apply to native embedded browsers as well.

I’m not convinced that reusing X-Frame-Options: DENY would be the best approach—I think it would break too many existing legitimate uses—but a similar option (or a similar header) specifically for native apps which causes pages to load in the native OS browser instead sounds like a fantastic idea to me.

Via @adrianholovaty

Tags: adrian-holovaty, browsers, privacy, security

Twenty years of my blog

2022-06-12T22:59:31+00:00

I started this blog on June 12th 2002 - twenty years ago today! To celebrate two decades of blogging, I decided to pull together some highlights and dive down a self-indulgent nostalgia hole.

Some highlights

Some of my more influential posts, in chronological order.

A new XML-RPC library for PHP - 2nd September 2002

I was really excited about XML-RPC, one of the earliest technologies for building Web APIs. IXR, the Incutio library for XML-RPC, was one of my earliest ever open source library releases. Here's a capture of the old site.

I've not touched anything relating to this project in over 15 years now, but it has lived on in both WordPress and Drupal (now only in Drupal 6 LTS).

It's also been responsible for at least one CVE vulnerability in those platforms!
getElementsBySelector() - 25th March 2003

Andrew Hayward had posted a delightful snippet of JavaScript called document.getElementsByClassName() - like document.getElementsByTagName() but for classes instead.

Inspired by this, I built document.getElementsBySelector() - a function that could take a CSS selector and return all of the matching elements.

This ended up being very influential indeed! Paul Irish offers a timeline of JavaScript CSS selector engines which tracks some of what happens next. Most notably, getElementsBySelector() was part of John Resig's inspiration in creating the first version of jQuery. To this day, the jQuery source includes this testing fixture which is derived from my original demo page.

I guess you could call document.getElementsBySelector() the original polyfill for document.querySelectorAll().
I'm in Kansas - 27th August 2003

In May 2003 Adrian Holovaty posted about a job opportunity for a web developer at at the Lawrence Journal-World newspaper in Lawrence, Kansas.

This coincided with my UK university offering a "year in industry" placement, which meant I could work for a year anywhere in the world with a student visa program. I'd been reading Adrian's blog for a while and really liked the way he thought about building for the web - we were big fans of Web Standards and CSS and cleanly-designed URLs, all of which were very hot new things at the time!

So I talked to Adrian about if this could work as a year-long opportunity, and we figured out how to make it work.

At the Lawrence Journal-Word Adrian and I decided to start using Python instead of PHP, in order to build a CMS for that local newspaper...
Introducing Django - 17th July 2005

... and this was the eventual outcome! Adrian and I didn't even know we were building a web framework at first - we called it "the CMS". But we kept having to solve new foundational problems: how should database routing work? What about templating? What's the best way to represent the incoming HTTP request?

I had left the Lawrence Journal-World in 2004, but by 2005 the team there had grown what's now known as Django far beyond where it was when I had left, and they got the go-ahead from the company to release it as open source (partly thanks to the example set by Ruby on Rails, which first released in August 2004).

In 2010 I wrote up a more detailed history of Django in a Quora answer, now mirrored to my blog.
Finally powered by Django - 15th December 2006

In which I replaced my duct-tape-and-mud PHP blogging engine with a new Django app. I sadly don't have the version history for this anymore (this was pre-git, I think I probably had it in Subversion or Mercurial somewhere) but today's implementation is still based on the same code, upgraded to Django 1.8 in 2015.

That 2006 version did include a very pleasing Flickr integration to import my photos (example on the Internet Archive):
How to turn your blog in to an OpenID - 19th December 2006

In late 2006 I got very, very excited about OpenID. I was convinced that Microsoft Passport was going to take over SSO on the internet, and that the only way to stop that was to promote an open, decentralized solution. I wrote posts about it, made screencasts (that one got 840 diggs! Annoyingly I was serving it from the Internet Archive who appear to have deleted it) and gave a whole bunch of conference talks about it too.

I spent the next few years advocating for OpenID - in particular the URL-based OpenID mechanism where any website can be turned into an identifier. It didn't end up taking off, and with hindsight I think that's likely for the best: expecting people to take control of their own security by chosing their preferred authentication provider sounded great to me in 2006, but I can understand why companies chose to instead integrate with a smaller, tightly controlled set of SSO partners over time.
A few notes on the Guardian Open Platform - 10th March 2009

In 2009 I was working at the Guardian newspaper in London in my first proper data journalism role - my work at the Lawrence Journal-World had hinted towards that a little, but I spent the vast majority of my time there building out a CMS.

In March we launched two major initiatives: the Datablog (also known as the Data Store) and the Guardian's Open Platform (an API that is still offered to this day).

The goal of the Datablog was to share the data behind the stories. Simon Rogers, the Guardian's data editor, had been collecting meticulous datasets about the world to help power infographics in the paper for years. The new plan was to share that raw data with the world.

We started out using Google Sheets for this. I desperately wanted to come up with something less proprietary than that - I spent quite some time experimenting with CouchDB - but Google Sheets was more than enough to get the project started.

Many years later my continued mulling of this problem formed part of the inspiration for my creation of Datasette, a story I told in my 2018 PyBay talk How to Instantly Publish Data to the Internet with Datasette.
Why I like Redis - 22nd October 2009

I got interested in NoSQL for a few years starting around 2009. I still think Redis was the most interesting new piece of technology to come out of that whole movement - an in-memory data structure server exposed over the network turns out to be a fantastic complement for other data stores, and even though I now default to PostgreSQL or SQLite for almost everything else I can still find problems for which Redis is a great solution.

In April 2010 I gave a three hour Redis tutorial at NoSQL Europe which I wrote up in Comprehensive notes from my three hour Redis tutorial.
Node.js is genuinely exciting - 23rd November 2009

In December 2009 I found out about Node.js. As a Python web developer I had been following the evolution of Twisted with great interest, but I'd also run into the classic challenge that once you start using event-driven programming almost every library you might want to use likely doesn't work for you any more.

Node.js had server-side event-driven programming baked into its very core. You couldn't accidentally make a blocking call and break your event loop because it didn't ever give you the option to do so!

I liked it so much I switched out my talk for Full Frontal 2009 at the last minute for one about Node.js instead.

I think this was an influential decision. I won't say who they are (for fear of mis-representing or mis-quoting them), but I've talked to entrepreneurs who built significant products on top of server-side JavaScript who told me that they heard about Node.js from me first.
Crowdsourced document analysis and MP expenses - 20th December 2009

This was my biggest data journalism project at the Guardian.

The UK government had finally got around to releasing our Member of Parliament expense reports, and there was a giant scandal brewing about the expenses that had been claimed. We recruited our audience to help dig through 10,000s of pages of PDFs to help us find more stories.

The first round of the MP's expenses crowdsourcing project launched in June, but I was too busy working on it to properly write about it! Charles Arthur wrote about it for the Guardian in The breakneck race to build an application to crowdsource MPs' expenses.

In December we launched round two, and I took the time to write about it properly.

Here's a Google Scholar search for guardian mps expenses - I think it was pretty influential. It's definitely one of the projects I'm most proud of in my career so far.
WildlifeNearYou: It began on a fort... - 12th January 2010

In October 2008 I participated in the first /dev/fort - a bunch of nerds rent a fortress (or similar historic building) for a week and hack on a project together.

Following that week of work it took 14 months to add the "final touches" before putting the site we had built live (partly because I insisted on implementing OpenID for it) but in January 2010 we finally went live with WildlifeNearYou.com (sadly no longer available). It was a fabulous website, which crowdsourced places that people had seen animals in order to answer the crucial question "where is my nearest Llama?".

Here's what it looked like:

Although it shipped after the Guardian MP's expenses project most of the work on WildlifeNearYou had come before that - building WildlifeNearYou (in Django) was the reason I was confident that the MP's expenses project was feasible.
Getting married and going travelling - 21st June 2010

One June 5th 2010 I married Natalie Downe, and we both quit our jobs to set off travelling around the world and see how far we could get.

We got as far as Casablanca, Morocco before we accidentally launched a startup together: Lanyrd, launched in August 2010. "Sign in with Twitter to see conferences that your friends are speaking at, attending or tracking, then add your own events."

We ended up spending the next three years on this: we went through Y Combinator, raised a sizable seed round, moved to London, hired a team and shipped a LOT of features. We even managed to ship some features that made the company money!

This also coincided with me putting the blog on the back-burner for a few years.

Here's an early snapshot:

In 2013 we sold Lanyrd to Eventbrite, and moved our entire team (and their families) from London to San Francisco. It had been a very wild ride.

Sadly the site itself is no longer available: as Eventbrite grew it became impossible to justify the work needed to keep Lanyrd maintained, safe and secure. Especially as it started to attract overwhelming volumes of spam.

Natalie told the full story of Lanyrd on her blog in September 2013: Lanyrd: from idea to exit - the story of our startup.
Scraping hurricane Irma - 10th September 2017

In 2017 hurricane Irma devastated large areas of the Caribbean and the southern USA.

I got involved with the Irma Response project, helping crowdsource and publish critical information for people affected by the storm.

I came up with a trick to help with scraping: I ran scrapers against important information sources and recorded the results to a git repository, in order to cheaply track changes to those sources over time.

I later coined the term "Git scraping" for this technique, see my series of posts about Git scraping over time.
Getting the blog back together - 1st October 2017

Running a startup, and then working at Eventbrite afterwards, had resulted in an almost 7 year gap in blogging for me. In October 2017 I decided to finally get my blog going again. I also back-filled content for the intervening years by scraping my content from Quora and from Ask Metafilter.

If you've been meaning to start a new blog or revive an old one this is a trick that I can thoroughly recommend: just because you initially wrote something elsewhere doesn't mean you shouldn't repost it on a site you own.
Recovering missing content from the Internet Archive - 8th October 2017

The other step in recovering my old blog's content was picking up some content that was missing from my old database backup. Here's how I pulled in that content by scraping the Internet Archive.
Implementing faceted search with Django and PostgreSQL - 5th October 2017

I absolutely love building faceted search engines. I realized a while ago that most of my career has been spent applying the exact same trick - faceted search - to different problem spaces. WildlifeNearYou offered faceted search over animal sightings. MP's expenses had faceted search across crowdsourced expense analysis. Lanyrd was faceted search for conferences.

I implemented faceted search for this blog on top of PostgreSQL, and wrote about how I did it.
Datasette: instantly create and publish an API for your SQLite databases - 13th November 2017

I shipped the first release of simonw/datasette in Nevember 2017. Nearly five years later it's now my number-one focus, and I don't see myself losing interest in it for many decades to come.

Datasette was inspired by the Guardian Datablog, combined with my realization that Zeit Now (today called Vercel) meant you could bundle data up in a SQLite database and deploy it as part of an exploratory application almost for free.

My blog has 284 items tagged datasette at this point.
Datasette Facets - 20th May 2018

Given how much I love faceted search, it's surprising it took me until May 2018 to realize that I could bake them into Datasette itself - turning it into a tool for building faceted search engines against any data. It turns out to be my ideal solution to my favourite problem!
Documentation unit tests - 28th July 2018

I figured out a pattern for using unit tests to ensure that features of my projects were covered by the documentation. Four years later I can confirm that this technique works really well - though I wish I'd called it Test-driven documentation instead!
Letterboxing on Lundy - 18th September 2018

A brief foray into travel writing: Natalie and I spent a few days staying in a small castle on the delightful island of Lundy off the coast of North Devon, and I used it as an opportunity to enthuse about letterboxing and the Landmark Trust.
sqlite-utils: a Python library and CLI tool for building SQLite databases - 25th February 2019

Datasette helps you explore and publish data stored in SQLite, but how do you get data into SQLite in the first place?

sqlite-utils is my answer to that question - a combined CLI tool and Python library with all sorts of utilites for working with and creating SQLite databases.

It recently had its 100th release!
I commissioned an oil painting of Barbra Streisand’s cloned dogs - 7th March 2019

Not much I can add that's not covered by the title. It's a really good painting!
My JSK Fellowship: Building an open source ecosystem of tools for data journalism - 10th September 2019

In late 2019 I left Eventbrite to join the JSK fellowship program at Stanford. It was an opportunity to devote myself full-time to working on my growing collection of open source tools for data journalism, centered around Datasette.

I jumped on that opportunity with both hands, and I've been mostly working full-time on Datasette and associated projects (without being paid for it since the fellowship ended) ever since.
Weeknotes: ONA19, twitter-to-sqlite, datasette-rure - 13th September 2019

At the start of my fellowship I decided to publish weeknotes, to keep myself accountable for what I was working on now that I didn't have the structure of a full-time job.

I've managed to post them roughly once a week ever since - 128 posts and counting.

I absolutely love weeknotes as a format. Even if no-one else ever reads them, I find them really useful as a way to keep track of my progress and ensure that I have motivation to get projects to a point where I can write about them at the end of the week!
Using a self-rewriting README powered by GitHub Actions to track TILs - 20th April 2020

In April 2020 I started publishing TILs - Today I Learneds - at til.simonwillison.net.

The idea behind TILs is to dramatically reduce the friction involved in writing a blog post. If I learned something that was useful to me, I'll write it up as a TIL. These often take less than ten minutes to throw together and I find myself referring back to them all the time.

My main blog is a Django application, but my TILs run entirely using Datasette. You can see how that all works in the simonw/til GitHub repository.
Using SQL to find my best photo of a pelican according to Apple Photos - 21st May 2020

Dogsheep is my ongoing side project in which I explore ways to analyze my own personal data using SQLite and Datasette.

dogsheep-photos is my tool for extracting metadata about my photos from the undocumented Apple Photos SQLite database (building on osxphotos by Rhet Turnbull). I had been wanting to solve the photo problem for years and was delighted when osxphotos provided the capability I had been missing. And I really like pelicans, so I celebrated by using my photos of them for the demo.
Git scraping: track changes over time by scraping to a Git repository - 9th October 2020

If you really want people to engage with a technique, it's helpful to give it a name. I defined Git scraping in this post, and I've been promoting it heavily ever since.

There are now 275 public repositories on GitHub with the git-scraping topic, and if you sort them by recently updated you can see the scrapers on there that most recently captured some new data.
Personal Data Warehouses: Reclaiming Your Data - 14th November 2020

I gave this talk for GitHub's OCTO (previously Office of the CTO, since rebranded to GitHub Next) speaker series.

It's the Dogsheep talk, with a better title (thanks, Idan!) It includes a full video demo of my personal Dogsheep instance, including my dog's Foursquare checkins, my Twitter data, Apple Watch GPS trails and more.

I also explain why I called it Dogsheep: it's a devastatingly terrible pun on Wolfram.

I'm frustrated when information like this is only available in video format, so when I give particularly information-dense talks I like to turn them into full write-ups as well, providing extra notes and resources alongside screen captures from the talk.

For this one I added a custom template mechanism to my blog, to allow me to break out of my usual entry page design.
Trying to end the pandemic a little earlier with VaccinateCA - 28th February 2021

In February 2021 I joined the VaccinateCA effort to try and help end the pandemic a little bit earlier by crowdsourcing information about the best places to get vaccinated. It was a classic match-up for my skills and interests: a huge crowdsourcing effort that needed to be spun up as a fresh Django application as quickly as possible.

Django SQL Dashboard was one project that spun directly out of that effort.
The Baked Data architectural pattern - 28th July 2021

My second attempt at coining a new term, after Git scraping: Baked Data is the name I'm using for the architectural pattern embodied by Datasette where you bundle a read-only copy of your data alongside the code for your application, as part of the same deployment. I think it's a really good idea, and more people should be doing it.
How I build a feature - 12th January 2022

Over the years I’ve evolved a processes for feature development that works really well for me, and scales down to small personal projects as well as scaling up to much larger pieces of work. I described that in detail in this post.

Picking out these highlights wasn't easy. I ended up setting myself a time limit (to ensure I could put this post live within a minute of midnight UTC time on my blog's 20th birthday) so there's plenty more that I would have liked to dig up.

My tags index page includes a 2010s-style word cloud that you can visit if you want to explore the rest of my content. Or use the faceted search!

A few more project release highlights:

GraphQL in Datasette with the new datasette-graphql plugin - 7th August 2020
git-history: a tool for analyzing scraped data collected using Git and SQLite - 7th December 2021
shot-scraper: automated screenshots for documentation, built on Playwright - 10th March 2022
Django SQL Dashboard - 10th May 2021
Datasette Desktop—a macOS desktop application for Datasette - 8th September 2021
Datasette Lite: a server-side Python web application running in a browser - 4th May 2022

Evolution over time

I started my blog in my first year of as a student studying computer science at the University of Bath.

You can tell that Twitter wasn't a thing yet, because I wrote 107 posts in that first month. Lots of links to other people's blog posts (we did a lot of that back then) with extra commentary. Lots of blogging about blogging.

That first version of the site was hosted at http://www.bath.ac.uk/~cs1spw/blog/ - on my university's student hosting. Sadly the Internet Archive doesn't have a capture of it there, since I moved it to http://simon.incutio.com/ (my part-time employer at the time) in September 2002. Here's my note from then about rewriting it to use MySQL instead of flat file storage.

This is the earliest capture I could find on the Internet Archive, from June 2003:

Full entry on Using bookmarklets to experiment with CSS.

By November 2006 I had redesigned from orange to green, and started writing Blogmarks - the name I used for small, bookmark-style link posts. I've collected 6,304 of them over the years!

By 2010 I'd reached more-or-less my current purple on white design, albeit with the ability to sign in with OpenID to post a comment. I dropped comments entirely when I relaunched in 2017 - constantly fighting against spam comments makes blogging much less fun.

The source code for the current iteration of my blog is available on GitHub.

Taking screenshots of the Internet Archive with shot-scraper

Here's how I generated the screenshots in this post, using shot-scraper against the Internet Archive but with a line of JavaScript to hide the banner the display at the top of every archived page:

shot-scraper 'https://web.archive.org/web/20030610004652/http://simon.incutio.com/' \
  --javascript 'document.querySelector("#wm-ipp-base").style.display="none"' \
   --width 800 --height 600 --retina

mgdlbp on Hacker News pointed out that you can instead add if_ to the date part of the archive URLs to hide the banner, like this:

shot-scraper 'https://web.archive.org/web/20030610004652if_/http://simon.incutio.com/' \
   --width 800 --height 600 --retina

Tags: adrian-holovaty, blogging, personal-news

What is the history of the Django web framework? Why has it been described as "developed in a newsroom"?

2010-08-24T18:15:00+00:00

My answer to What is the history of the Django web framework? Why has it been described as "developed in a newsroom"? on Quora

I was there!

Django started when Adrian Holovaty and I were working together for a year at the Lawrence Journal-World newspaper back in 2003-2004.

I was there on a year long internship (my UK university offered the chance to work abroad). Adrian had already created Lawrence.com using PHP, and I had about 5 years of PHP experience. Both of us were fed up with trying to maintain large sites in PHP (this was before namespaces, PHP5 etc so it might be better now, but at the time it started to drag once you got above a certain size of codebase). We had both fallen in love with Python, probably thanks to Mark Pilgrim's Dive Into Python book.

We wanted to switch to doing web development in Python, but we both had very strong opinions about how web development should work - thinks things like the importance of well designed URLs, and making good use of CSS (back in 2004 the Web Standards movement was still a relatively new thing).

We looked at the existing set of Python tools for web development, but none of them fitted the way we wanted to work. We also looked in to Python deployment options and found that the best bet appeared to be mod_python.

The original reason for creating Django was that we weren't convinced that mod_python would scale, so we wanted to have our own thin abstraction layer between it and our code just in case we needed to ditch mod_python for something else. That's where the Django HttpRequest / HttpResponse objects came from. (WSGI didn't exist yet - in fact, we got involved on the initial Python Web-SIG mailing list to talk about what became WSGI based on what we were learning while building Django, but that process took several years and the eventual spec didn't look much like our request/response objects).

We never really intended to build a web framework - for the longest time, the code that became Django was referred to as "the CMS". We pair programmed some of the core aspects of Django - the Request/Response objects, the URL resolving and the template language in particular.

The ORM started life as some very unpleasant repetitive classes that we kept on having to copy-and-paste for each of our persistent models. We were sure there was a better way of doing it, but we didn't know quite what it was.

Then I went away to SxSW 2004 for a week, and when I came back Adrian had written a code generator that spat out all of our model classes for us - I think he might even have had it spitting out simple versions of the admin pages as well!

The aim with Django was always to port lawrence.com to it - so you could say that while Rails was extracted out of BaseCamp, Django was built up to fit the functionality of lawrence.com. We also built several new sites on Django - the first ever Django site to go live was 6newslawrence.com (now no longer available.)

My internship only lasted a year, and a month before we left we hired Jacob Kaplan-Moss as my replacement. Jacob and Adrian continued to develop Django, and about a year later convinced the World Company (the owners of the newspaper) to open source it - the successful story of Ruby on Rails was a useful argument there, but I understand the World Company were also convinced by the amount of free software the company had benefited from previously, and the desire to give something back to that community.

Adrian named it after his favourite guitarist, Django Reinhardt. At some point before then the code generator had been replaced with a much smarter system based on metaclasses - I think Adrian's conversations with Ian Bicking may have influenced that.

After that, everything is pretty much on the public record.

Tags: adrian-holovaty, django, frameworks, me, quora

Data Is Journalism: MSNBC.com Acquires Everyblock

2009-08-18T12:10:11+00:00

Data Is Journalism: MSNBC.com Acquires Everyblock

Congratulations Adrian, Wilson and the team! Brady Forrest reports the acquisition within the larger context of the rise of data-driven journalism.

Tags: adrian-holovaty, brady-forrest, data-journalism, everyblock, msnbc, wilson-miner

Django tip: Caching and two-phased template rendering

2009-05-19T01:34:35+00:00

Django tip: Caching and two-phased template rendering

Neat trick for expensive pages which can be mostly cached with the exception of the “logged in as” bit—run them through the template system twice, caching the intermediary generated template.

Tags: adrian-holovaty, caching, django, performance, python, template

The Django Book: Version 2.0

2009-01-09T14:54:24+00:00

The Django Book: Version 2.0

Adrian’s working on a new edition of the Django Book updated to cover version 1.0. As with the first edition, it will be available free online in addition to a published Apress paperback. The first three chapters are now available.

Via Adrian Holovaty

Tags: adrian-holovaty, apress, books, django, python

Django tickets with keyword "djangocon"

2008-09-08T03:02:11+00:00

Django tickets with keyword "djangocon"

Adrian and Jacob ran an “I want a pony” session during their closing keynote at DjangoCon—I’ve filed the feature requests as tickets tagged with the “djangocon” keyword.

Tags: adrian-holovaty, django, djangocon, djangocon08, iwantapony, jacob-kaplan-moss, python, tickets

Cyberstar

2008-08-18T23:56:42+00:00

Cyberstar

Adrian made the front cover of the Chicago Tribune magazine!

Tags: adrian-holovaty, django, everyblock

themaneater.com Launch

2008-02-10T08:10:42+00:00

themaneater.com Launch

The Maneater’s online edition is where Adrian cut his web development teeth, so it’s great to see them up and running on Django. Important to note that KeepAlive can completely murder Apache/Django performance.

Tags: adrian-holovaty, apache, django, keepalive, themaneater

Introducing EveryBlock

2008-01-23T21:56:55+00:00

Introducing EveryBlock

EveryBlock launched! Adrian Holovaty, Wilson Miner, Paul Smith and Daniel X. O’Neil’s startup which answers the question, “What’s happening in my neighborhood?” Cities covered by the launch are Chicago, New York and San Francisco.

Tags: adrian-holovaty, chicago, daniel-x-oneil, django, everyblock, new-york, paul-smith, san-francisco, wilson-miner

Chatting with Adrian Holovaty

2008-01-01T11:44:21+00:00

Chatting with Adrian Holovaty

Fabio Akita interviews Adrian about Django and related topics.

Tags: adrian-holovaty, django, fabioakita, python

"The Definitive Guide to Django" is now shipping from Amazon

2007-12-11T21:12:37+00:00

"The Definitive Guide to Django" is now shipping from Amazon

The book looks absolutely fantastic (bias disclosure: I contributed the newforms chapter)—huge congratulations to Adrian and Jacob.

Tags: adrian-holovaty, amazon, apress, books, django, django-book, jacob-kaplan-moss

Django Book Update

2007-11-14T00:59:55+00:00

Django Book Update

It’s done! Went to the printer on Friday, due in bookstores in the second week of December (just in time for Christmas). Congrats to Adrian and Jacob.

Tags: adrian-holovaty, django, django-book, jacob-kaplan-moss, python

Knight Foundation grant

2007-05-24T16:27:04+00:00

Knight Foundation grant

Adrian’s leaving the Washington Post to found EveryBlock, a startup focusing on local news and information in the style of chicagocrime.org.

Tags: adrian-holovaty, chicagocrime, everyblock, startup, washington-post

Web Focus Leads Newspapers to Hire Programmers for Editorial Staff

2007-03-08T00:27:26+00:00

Web Focus Leads Newspapers to Hire Programmers for Editorial Staff

It’s great to see this trend taking off. A newsroom is an excellent place to work as a programmer.

Tags: adrian-holovaty, jacob-kaplan-moss, jobs, newspapers, programmers

Graphing requests with Tamper Data

2006-10-17T18:21:06+00:00

I spent the weekend in Boston, speaking at GBC/ACM's Deep Ajax seminar with Alex Russell and Adrian Holovaty. I'll be posting some notes on this later, but I wanted to share a really neat Firefox extension that Alex showed me: Tamper Data.

Tamper Data is an extension for intercepting HTTP requests and modifying them. I have very little interest in this functionality myself, but hidden deep within the extension is the ability to do this:

That's a graph showing what happens when you load up www.yahoo.com. It shows every component of the page - JavaScript, CSS, images - and when each component started and finished loading. You can use it to get an idea for how long it took between the HTML starting to load and the browser beginning to pull in the CSS, then the images, and so on. It's a superb visualization of what happens when a page is loaded.

Unfortunately, if you install and run the extension (Tools -> Tamper Data) you'll see this instead:

To get the graph, you have to right click in the main data grid and select "Graph All" from the context menu. Be sure to hit "clear" before loading a page that you want to graph or you'll end up seeing data from other pages too (you should shut down GMail or similar to prevent their polling requests from polluting the graph).

It's a great tool but it's pretty well hidden. If you're looking for a side project, implementing the same functionality in a smaller extension (maybe as an extra tab in the Page Info screen) would be a significant service to the web development community.

Tags: adrian-holovaty, alex-russell, extensions, firefox, my-talks, tamperdata

The programmer as journalist

2006-06-06T10:51:01+00:00

The programmer as journalist

An interview with Adrian Holovaty.

Tags: adrian-holovaty

How I'm using Amazon S3 to serve media files

2006-04-07T18:51:53+00:00

How I'm using Amazon S3 to serve media files

Adrian’s saving server overhead on ChicagoCrime by serving media from S3.

Tags: adrian-holovaty, amazon, aws, chicagocrime, s3

Exciting developments with Django

2005-08-03T16:56:34+00:00

The amount of activity surrounding the Django web framework since its not-quite release a few weeks ago is amazing. Adrian, Jacob and Wilson have been working over-time, with 395 check-ins to source control since the 13th of July. They've added WSGI support, a development web server, unit-tests, a ton of documentation, SQLite support, database introspection and dozens of other feature tweaks and bug fixes. Check out the project Timeline for an idea of just how frenetic things have been.

The emerging Django community has been kicking in as well. There's a significant community-led initiative to get internationalisation and localisation going, and a wide number of unofficial tutorials have emerged to complement the one on the site.

Here's where things get really interesting: changes at the Journal-World have kick-started the Django job market. Rob Curley, formally in charge of the World Company's web activities, recently took up a new position at the Naples Daily News in Florida. Rob just hired Eric Moritz, a regular on the #django IRC channel, to work on Django-powered projects there.

Meanwhile, Adrian Holovaty has taken a new job at the Washington Post as "Editor, Editorial Innovations" - a role that is sure to involve some very innovative use of Django (Adrian built chicagocrime.org). Adrian's departure means that the Journal-World are looking for a new developer - here's why you should apply.

One thing's for certain: we're going to see some very exciting Django-powered sites in the next few months.

Tags: adrian-holovaty, django, sqlite, wsgi

Adrian is leaving the Journal-World

2005-08-03T14:50:51+00:00

Adrian is leaving the Journal-World

This opens an excellent Python job opportunity in Kansas.

Tags: adrian-holovaty

Introducing Django

2005-07-17T11:59:57+00:00

You may know that I spent a year working in Kansas for a local newspaper - the Lawrence Journal-World. I'm delighted to announce that a decent chunk of the software I worked on there is now available as open-source, in the form of the Django web framework.

Django is an MVC Python web development framework with a strong emphasis on content management. While comparisons with Rails are inevitable (I plan to post one myself shortly), it should be emphasized that Django is by no means a clone of Rails - in fact, development on Django started in October 2003, months before the first public Rails release. That the two frameworks share so many ideas is, I feel, a testament to the design of both.

Django was initially developed by Adrian Holovaty and myself, with Jacob Kaplan-Moss joining the team shortly before my departure. The framework evolved during the construction of a number of sites, which included:

6newslawrence.com, local TV news, the first site to launch using the then nascent Django framework.
lawrence.com, a local entertainment site. Every town should have a site like this; sadly, very few do.
VisitLawrence.com, a local tourism site.
KUSports.com, a sports news site covering the University of Kansas.
LJWorld.com, a local news site and the flagship for the Journal-World.

It's also purring along under the hood of chicagocrime.org and the Django project site itself.

Adrian and I developed Django to allow us to create reasonably complex database driven web applications in as little time as possible. A local newspaper produces a huge amount of information - news stories, events listings, obituaries, sports results, marriage announcements, photo galleries, polls, weather reports and dozens of other bits and pieces. To maximise their value and potential for reuse, these things need to be stored in a database. That database needs a user interface for people to add and modify content, and of course the information needs to be published to a website somewhere.

Django automates most of this - and makes the rest as easy as possible. You create a data description (kind of like an SQL table schema but with additional information about validation rules and interface widgets) and load it in to Django. Django then creates the database tables, model classes and a comprehensive web-based administration interface for your site's staff. All that's left for you to do manually is the code for the public site, which is generally a case of writing a few lines of controller code, configuring some URLs and knocking out some templates.

To give you an idea of how much this speeds up the creation of new sites, consider the case of Game. Game was a site created to provide detailed coverage of little-league games in and around Lawrence (for non-Americans, little-league is baseball and softball for children aged around 5 to 11 - it's amazingly popular). The entire Game site, including news, fixtures, match results, team profiles and even weather forecasts for forthcoming matches took two days to develop.

I haven't been involved with Django since leaving the Journal-World back in September, but now that the framework is open-sourced I look forward to contributing to its further development - both in terms of documentation and actual code. There's already plenty to look at on the Django site (designed by the ever-talented Wilson Miner); I suggest the overview as your first port of call.

Expect to hear a lot more about the framework in the next few weeks.

Tags: adrian-holovaty, django, jacob-kaplan-moss, lawrence-journal-world, projects, highlights

Why Greasemonkey is good for publishers

2005-04-13T16:09:41+00:00

Why Greasemonkey is good for publishers

Free usability tips and bug fixing—your users know more than you do.

Via rc3.org

Tags: adrian-holovaty, greasemonkey

Again, a newspaper PDF experiment is fatally flawed

2005-03-02T11:25:28+00:00

Again, a newspaper PDF experiment is fatally flawed

Adrian nails the reason PDF editions will always play second fiddle to the real web.

Tags: adrian-holovaty

'Game': Fun with databases

2004-06-12T00:14:34+00:00

'Game': Fun with databases

Adrian discusses our latest site launch.

Tags: adrian-holovaty

Installing psycopg on Red Hat 9

2003-12-31T04:43:03+00:00

Adrian Holovaty and I spent some time today figuring out how to get the psycopg Postgres module to install on Red Hat 9. It took a while, but eventually we tweaked the spec file and used it to compile our own RPM. I've posted our modified spec file to the psycopg mailing list. More for my own record than anything else, the arcane incantations needed to create the RPM went roughly as follows:

Copy the modified psycopg.spec file in to /usr/src/redhat/SPECS/
Copy psycopg-1.1.11.tar.gz in to /usr/src/redhat/SOURCES
In the SPECS directory, run rpmbuild -ba psycopg.spec
Install the resulting RPM, which materialised in /usr/src/redhat/RPMS

The above was recreated by digging through my bash history - hopefully I haven't missed a step. The technique appears to work for Fedora as well.

Tags: adrian-holovaty, linux, postgresql, python, red-hat

Lawrence web meetup

2003-10-18T23:06:57+00:00

My colleague Adrian is putting together a meetup for web designers and developers in the Lawrence, Kansas area on Monday evening. The venue is yet to be decided but we're looking to book somewhere for dinner with an atmosphere that's conducive to lively conversation. Likely attendants so far include Adrian Holovaty, Travis Beckham, Anitra Pavka, Dan Cox (designer for ljworld.com, lawrence.com and kusports.com) and myself. If you're a web developer/designer in the area and this sounds like your kind of thing drop Adrian a line for more information. It should be a lot of fun.

Update: Adrian's site now has a more details of confirmed attendants.

Tags: adrian-holovaty, lawrence