Simon Willison's Weblog: couchdb

What are the advantages and disadvantages of using MongoDB vs CouchDB vs Cassandra vs Redis?

2010-12-01T12:54:00+00:00

My answer to What are the advantages and disadvantages of using MongoDB vs CouchDB vs Cassandra vs Redis? on Quora

I see Redis as a different category from the other three - kind of like you wouldn't say "what are the advantages of MySQL v.s. Memcached". Redis makes an excellent complement to pretty much any other persistent storage mechanism. I expanded on this here: http://simonwillison.net/2009/Oc...

Tags: cassandra, couchdb, mongodb, nosql, redis, quora

A Gentle Introduction to CouchDB for Relational Practitioners

2010-09-22T21:51:00+00:00

A Gentle Introduction to CouchDB for Relational Practitioners

By “High Performance MySQL” author Baron Schwartz—a smart, concise overview that touches pretty much everything that’s interesting about CouchDB.

Tags: couchdb, databases, recovered

What is the largest production deployment of CouchDB for online use?

2010-08-25T09:23:00+00:00

My answer to What is the largest production deployment of CouchDB for online use? on Quora

The BBC have a pretty big CouchDB cluster, which they use mostly as a replicated key-value store. It's used by their new identity platform which includes customisation features for iPlayer.

Tags: couchdb, databases, nosql, scaling, quora

CouchDB View Cookbook for SQL Jockeys

2009-11-26T13:20:25+00:00

CouchDB View Cookbook for SQL Jockeys

This demystified CouchDB views for me. From “CouchDB: The Definitive Guide”, the free online manual.

Tags: cookbook, couchdb, sqll, views

Node.js is genuinely exciting

2009-11-23T12:50:22+00:00

I gave a talk on Friday at Full Frontal, a new one day JavaScript conference in my home town of Brighton. I ended up throwing away my intended topic (JSONP, APIs and cross-domain security) three days before the event in favour of a technology which first crossed my radar less than two weeks ago.

That technology is Ryan Dahl's Node. It's the most exciting new project I've come across in quite a while.

At first glance, Node looks like yet another take on the idea of server-side JavaScript, but it's a lot more interesting than that. It builds on JavaScript's excellent support for event-based programming and uses it to create something that truly plays to the strengths of the language.

Node describes itself as "evented I/O for V8 javascript". It's a toolkit for writing extremely high performance non-blocking event driven network servers in JavaScript. Think similar to Twisted or EventMachine but for JavaScript instead of Python or Ruby.

Evented I/O?

As I discussed in my talk, event driven servers are a powerful alternative to the threading / blocking mechanism used by most popular server-side programming frameworks. Typical frameworks can only handle a small number of requests simultaneously, dictated by the number of server threads or processes available. Long-running operations can tie up one of those threads - enough long running operations at once and the server runs out of available threads and becomes unresponsive. For large amounts of traffic, each request must be handled as quickly as possible to free the thread up to deal with the next in line.

This makes certain functionality extremely difficult to support. Examples include handling large file uploads, combining resources from multiple backend web APIs (which themselves can take an unpredictable amount of time to respond) or providing comet functionality by holding open the connection until a new event becomes available.

Event driven programming takes advantage of the fact that network servers spend most of their time waiting for I/O operations to complete. Operations against in-memory data are incredibly fast, but anything that involves talking to the filesystem or over a network inevitably involves waiting around for a response.

With Twisted, EventMachine and Node, the solution lies in specifying I/O operations in conjunction with callbacks. A single event loop rapidly switches between a list of tasks, firing off I/O operations and then moving on to service the next request. When the I/O returns, execution of that particular request is picked up again.

(In the talk, I attempted to illustrate this with a questionable metaphor involving hamsters, bunnies and a hyperactive squid).

What makes Node exciting?

If systems like this already exist, what's so exciting about Node? Quite a few things:

JavaScript is extremely well suited to programming with callbacks. Its anonymous function syntax and closure support is perfect for defining inline callbacks, and client-side development in general uses event-based programming as a matter of course: run this function when the user clicks here / when the Ajax response returns / when the page loads. JavaScript programmers already understand how to build software in this way.
Node represents a clean slate. Twisted and EventMachine are hampered by the existence of a large number of blocking libraries for their respective languages. Part of the difficulty in learning those technologies is understanding which Python or Ruby libraries you can use and which ones you have to avoid. Node creator Ryan Dahl has a stated aim for Node to never provide a blocking API - even filesystem access and DNS lookups are catered for with non-blocking callback based APIs. This makes it much, much harder to screw things up.
Node is small. I read through the API documentation in around half an hour and felt like I had a pretty comprehensive idea of what Node does and how I would achieve things with it.
Node is fast. V8 is the fast and keeps getting faster. Node's event loop uses Marc Lehmann's highly regarded libev and libeio libraries. Ryan Dahl is himself something of a speed demon - he just replaced Node's HTTP parser implementation (already pretty speedy due to it's Ragel / Mongrel heritage) with a hand-tuned C implementation with some impressive characteristics.
Easy to get started. Node ships with all of its dependencies, and compiles cleanly on Snow Leopard out of the box.

With both my JavaScript and server-side hats on, Node just feels right. The APIs make sense, it fits a clear niche and despite its youth (the project started in February) everything feels solid and well constructed. The rapidly growing community is further indication that Ryan is on to something great here.

What does Node look like?

Here's how to get Hello World running in Node in 7 easy steps:

git clone git://github.com/ry/node.git (or download and extract a tarball)
./configure
make (takes a while, it needs to compile V8 as well)
sudo make install
Save the below code as helloworld.js
node helloworld.js
Visit http://localhost:8080/ in your browser

Here's helloworld.js:

var sys = require('sys'), 
  http = require('http');

http.createServer(function(req, res) {
  res.sendHeader(200, {'Content-Type': 'text/html'});
  res.sendBody('<h1>Hello World</h1>');
  res.finish();
}).listen(8080);

sys.puts('Server running at http://127.0.0.1:8080/');

If you have Apache Bench installed, try running ab -n 1000 -c 100 'http://127.0.0.1:8080/' to test it with 1000 requests using 100 concurrent connections. On my MacBook Pro I get 3374 requests a second.

So Node is fast - but where it really shines is concurrency with long running requests. Alter the helloworld.js server definition to look like this:

http.createServer(function(req, res) {
  setTimeout(function() {
    res.sendHeader(200, {'Content-Type': 'text/html'});
    res.sendBody('<h1>Hello World</h1>');
    res.finish();
  }, 2000);
}).listen(8080);

We're using setTimeout to introduce an artificial two second delay to each request. Run the benchmark again - I get 49.68 requests a second, with every single request taking between 2012 and 2022 ms. With a two second delay, the best possible performance for 1000 requests 100 at a time is 1000 requests / (1000 / 100) * 2 seconds = 50 requests a second. Node hits it pretty much bang on the nose.

The most important line in the above examples is res.finish(). This is the mechanism Node provides for explicitly signalling that a request has been fully processed and should be returned to the browser. By making it explicit, Node makes it easy to implement comet patterns like long polling and streaming responses - stuff that is decidedly non trivial in most server-side frameworks.

djangode

Node's core APIs are pretty low level - it has HTTP client and server libraries, DNS handling, asynchronous file I/O etc, but it doesn't give you much in the way of high level web framework APIs. Unsurprisingly, this has lead to a cambrian explosion of lightweight web frameworks based on top of Node - the projects using node page lists a bunch of them. Rolling a framework is a great way of learning a low-level API, so I've thrown together my own - djangode - which brings Django's regex-based URL handling to Node along with a few handy utility functions. Here's a simple djangode application:

var dj = require('./djangode');

var app = dj.makeApp([
  ['^/$', function(req, res) {
    dj.respond(res, 'Homepage');
  }],
  ['^/other$', function(req, res) {
    dj.respond(res, 'Other page');
  }],
  ['^/page/(\\d+)$', function(req, res, page) {
    dj.respond(res, 'Page ' + page);
  }]
]);
dj.serve(app, 8008);

djangode is currently a throwaway prototype, but I'll probably be extending it with extra functionality as I explore more Node related ideas.

nodecast

My main demo in the Full Frontal talk was nodecast, an extremely simple broadcast-oriented comet application. Broadcast is my favourite "hello world" example for comet because it's both simpler than chat and more realistic - I've been involved in plenty of projects that could benefit from being able to broadcast events to their audience, but few that needed an interactive chat room.

The source code for the version I demoed can be found on GitHub in the no-redis branch. It's a very simple application - the client-side JavaScript simply uses jQuery's getJSON method to perform long-polling against a simple URL endpoint:

function fetchLatest() {
  $.getJSON('/wait?id=' + last_seen, function(d) {
    $.each(d, function() {
      last_seen = parseInt(this.id, 10) + 1;
      ul.prepend($('<li></li>').text(this.text));
    });
    fetchLatest();
  });
}

Doing this recursively is probably a bad idea since it will eventually blow the browser's JavaScript stack, but it works OK for the demo.

The more interesting part is the server-side /wait URL which is being polled. Here's the relevant Node/djangode code:

var message_queue = new process.EventEmitter();

var app = dj.makeApp([
  // ...
  ['^/wait$', function(req, res) {
    var id = req.uri.params.id || 0;
    var messages = getMessagesSince(id);
    if (messages.length) {
      dj.respond(res, JSON.stringify(messages), 'text/plain');
    } else {
      // Wait for the next message
      var listener = message_queue.addListener('message', function() {
        dj.respond(res, 
          JSON.stringify(getMessagesSince(id)), 'text/plain'
        );
        message_queue.removeListener('message', listener);
        clearTimeout(timeout);
      });
      var timeout = setTimeout(function() {
        message_queue.removeListener('message', listener);
        dj.respond(res, JSON.stringify([]), 'text/plain');
      }, 10000);
    }
  }]
  // ...
]);

The wait endpoint checks for new messages and, if any exist, returns immediately. If there are no new messages it does two things: it hooks up a listener on the message_queue EventEmitter (Node's equivalent of jQuery/YUI/Prototype's custom events) which will respond and end the request when a new message becomes available, and also sets a timeout that will cancel the listener and end the request after 10 seconds. This ensures that long polls don't go on too long and potentially cause problems - as far as the browser is concerned it's just talking to a JSON resource which takes up to ten seconds to load.

When a message does become available, calling message_queue.emit('message') will cause all waiting requests to respond with the latest set of messages.

Talking to databases

nodecast keeps track of messages using an in-memory JavaScript array, which works fine until you restart the server and lose everything. How do you implement persistent storage?

For the moment, the easiest answer lies with the NoSQL ecosystem. Node's focus on non-blocking I/O makes it hard (but not impossible) to hook it up to regular database client libraries. Instead, it strongly favours databases that speak simple protocols over a TCP/IP socket - or even better, databases that communicate over HTTP. So far I've tried using CouchDB (with node-couch) and redis (with redis-node-client), and both worked extremely well. nodecast trunk now uses redis to store the message queue, and provides a nice example of working with a callback-based non-blocking database interface:

var db = redis.create_client();
var REDIS_KEY = 'nodecast-queue';

function addMessage(msg, callback) {
  db.llen(REDIS_KEY, function(i) {
    msg.id = i; // ID is set to the queue length
    db.rpush(REDIS_KEY, JSON.stringify(msg), function() {
      message_queue.emit('message', msg);
      callback(msg);
    });
  });
}

Relational databases are coming to Node. Ryan has a PostgreSQL adapter in the works, thanks to that database already featuring a mature non-blocking client library. MySQL will be a bit tougher - Node will need to grow a separate thread pool to integrate with the official client libs - but you can talk to MySQL right now by dropping in DBSlayer from the NY Times which provides an HTTP interface to a pool of MySQL servers.

Mixed environments

I don't see myself switching all of my server-side development over to JavaScript, but Node has definitely earned a place in my toolbox. It shouldn't be at all hard to mix Node in to an existing server-side environment - either by running both behind a single HTTP proxy (being event-based itself, nginx would be an obvious fit) or by putting Node applications on a separate subdomain. Node is a tempting option for anything involving comet, file uploads or even just mashing together potentially slow loading web APIs. Expect to hear a lot more about it in the future.

Simple CouchDB multi-master clustering via Nginx

2009-11-19T16:37:36+00:00

Simple CouchDB multi-master clustering via Nginx

An impressive combination. CouchDB can be easily set up in a multi-master configuration, where writes to one master are replicated to the other and vice versa. This makes setting up a reliable CouchDB cluster is as simple as putting two such servers behind a single nginx proxy.

Tags: cluster, couchdb, load-balancing, multimaster, nginx, replication

Desktop Couch initial code

2009-07-09T11:34:19+00:00

Desktop Couch initial code

More from Stuart Langridge on the project to make CouchDB available as a desktop service, providing free synchronisation between machines and a way for different applications to interrogate each other’s structured data.

Tags: couchdb, desktop, stuart-langridge, synchronisation

MongoDB

2009-06-30T19:13:04+00:00

MongoDB

Lots of discussions about this at EuroPython today—it’s a document database, very similar to CouchDB but significantly faster and suggested for production use. Best of all, trying it out on OS X is as easy as extracting the tarball and running “bin/mongod --dbpath /tmp/test-mongo-db run”.

Tags: couchdb, documentstore, europython, json, keyvaluestore, macos, mongodb, nonrelationaldatabase

whine flu, railsmalefail 2009

2009-04-29T11:39:38+00:00

whine flu, railsmalefail 2009

Danny quotes the smartest take on the CouchDB/pr0n controversy: “It’s about presenting women as ’the other,’ not ’us.’ It would have been just as offensive if all the women shown were domineering mothers in aprons, shaking their fingers and threatening with rolling pins.”

Tags: couchdb, danny-obrien, rails, sexism

Installing CouchDB from source on OS X

2009-04-17T16:22:41+00:00

Installing CouchDB from source on OS X

So far I’ve just been playing with it in an Ubuntu virtual machine.

Tags: building, couchdb, macos, ubuntu

New PylonsHQ Site Launches

2009-01-22T18:33:17+00:00

New PylonsHQ Site Launches

The new site uses CouchDB instead of a relational database, and the code for the site is open source so you can see how it all works.

Tags: ben-bangert, couchdb, nonrelational, pylons, python

Rules of Database App Aging

2009-01-18T09:09:43+00:00

Rules of Database App Aging

Peter Harkins: All fields become optional, all relationships become many-to-many, chatter always expands. This is why document oriented databases such as CouchDB are looking more and more attractive.

Tags: couchdb, databases, peter-harkins

GeoCouch: Geospatial queries with CouchDB

2008-10-27T23:48:46+00:00

GeoCouch: Geospatial queries with CouchDB

Interesting approach: uses “external2”, a branch that allows external services to be called from CouchDB. SQLite’s SpatiaLite extension is then used as an external spacial index.

Tags: couchdb, erlang, external2, geocouch, spatialite, sqlite

Persevere adds Comet Support

2008-05-13T08:09:29+00:00

Persevere adds Comet Support

Persevere sounds neat: a RESTful HTTP/JSON data store (the interface reminds me of CouchDB) which recently gained the ability to “subscribe” to a resource and receive notifications of updates via comet.

Tags: comet, couchdb, javascript, json, persevere, rest, restful

Multi-Inflection-Point Alert

2008-04-26T18:48:57+00:00

Multi-Inflection-Point Alert

Dammit, Tim, stop giving away our competitive advantages!

Tags: bigtable, couchdb, java, python, rails, rest, simpledb, soap, tim-bray

What Sucks About Erlang

2008-03-11T05:45:16+00:00

What Sucks About Erlang

Damien Katz shares his greatest frustrations from working with Erlang on CouchDB.

Tags: couchdb, damien-katz, erlang

CouchDB, XML, and E4X

2008-03-05T00:31:34+00:00

CouchDB, XML, and E4X

Brilliant—CouchDB now enables SpiderMonkey’s E4X support, meaning CouchDB views can easily query XML documents stored inside JSON objects using E4X syntax.

Tags: christopher-lenz, couchdb, e4x, javascript, json, spidermonkey, xml

Damien Katz: New Gig

2008-01-02T20:35:24+00:00

Damien Katz: New Gig

IBM have employed Damien Katz to work full time on CouchDB. The work will be under the Apache license with the ASF owning the copyright.

Tags: apache, asf, couchdb, damien-katz, ibm

CouchDB Roundup

2007-12-06T15:48:47+00:00

CouchDB Roundup

The CouchDB project is interested in contributions from people who can write a large file driver for Erlang, help figure out the CouchDB security model and build scripts to help benchmark performance, scalability and reliability.

Tags: couchdb, erlang, helpwanted

Call for Participation for XTech 2008

2007-12-05T15:28:22+00:00

Call for Participation for XTech 2008

XTech 2008 will be in Dublin, Ireland from the 6th to the 9th of May. Lots of really interesting topics in the CfP (OpenID, OAuth, Comet, CouchDB...)—deadline for submissions is the 25th of January.

Tags: call-for-proposals, comet, conferences, couchdb, oauth, openid, xtech

[Release] CouchDB 0.7.0

2007-11-17T00:25:10+00:00

[Release] CouchDB 0.7.0

This is a huge milestone for the project—it’s the first official release to include the JSON REST API instead of XML, and it’s also the first release that is “intended for widespread use”.

Tags: couchdb, jan-lehnardt, json, releases, rest, xml

CouchDB "Joins"

2007-10-25T08:27:27+00:00

CouchDB "Joins"

Different approaches to indexing a blog post and its associated comments in the non-relational CouchDB.

Tags: christopher-lenz, couchdb, databases, erlang, joins, relational, views

Quoting Jacob Kaplan-Moss

2007-10-20T13:46:56+00:00

Django may be built for the Web, but CouchDB is built of the Web. I've never seen software that so completely embraces the philosophies behind HTTP. CouchDB makes Django look old-school in the same way that Django makes ASP look outdated.

— Jacob Kaplan-Moss

Tags: asp, couchdb, django, http, jacob-kaplan-moss, python

CouchDB first impressions

2007-10-19T11:43:11+00:00

CouchDB first impressions

Jacob’s been poking at CouchDB. Inserting data is slow, but everything else looks pretty slick considering how recently the JSON / JavaScript views functionality was added.

Tags: couchdb, erlang, jacob-kaplan-moss, javascript, json

A CouchDB GUI front end

2007-09-12T23:59:48+00:00

A CouchDB GUI front end

Written in C# and .NET. It looks like writing frontends for CouchDB could make an excellent project for learning a new GUI environment.

Tags: ciarangultnieks, couchdb, csharp, dotnet

CouchDb: Some Context

2007-09-04T02:31:10+00:00

CouchDb: Some Context

CouchDb developer Jan Lehnardt wrote up detailed notes on slides from a presentation he gave back in June, explaining most of what’s interesting about CouchDb (although without the new JavaScript function query language).

Tags: couchdb, jan-lehnardt, javascript, nonrelational, slides

CouchDB: Thinking beyond the RDBMS

2007-09-03T09:48:43+00:00

CouchDB: Thinking beyond the RDBMS

CouchDB is a fascinating project—an Erlang powered non-relational database with a JSON API that lets you define “views” (really computed tables) based on JavaScript functions that execute using map/reduce. Damien Katz, the main developer currently works for MySQL and used to work on Lotus Notes.

Tags: couchdb, damien-katz, databases, erlang, javascript, json, lotusnotes, mapreduce, mysql