Simon Willison's Weblog: sphinx-search

How can you build a search engine for a website built in PHP/MySQL?

2012-02-11T18:39:00+00:00

My answer to How can you build a search engine for a website built in PHP/MySQL? on Quora

There are a bunch of options.

The easiest to implement is to build search on top of MySQL LIKE queries - performance will be pretty terrible (since every search will require a full table scan) but provided your tables only have a few thousand records on them and your site doesn't have to cope with more than a dozen or so hits a second it should work fine.

Next easiest: use MySQL's built-in full text indexing feature. It's not particularly good, and it requires you to use MyISAM tables (InnoDB is much more reliable, but doesn't support full text indexing) - but it will do the job. You could always keep your main site data in InnoDB and denormalise in to a MyISAM table just for search - or you could use the trick Flickr used to use, which is to set up MySQL replication and run MyISAM on one of the slaves purely to support fulltext search.

Past that, you're looking at adding another component to the stack. Sphinx can integrate directly with MySQL and lets you run SQL-style queries against a proper full text index. Personally I'm a big fan of Solr, which runs as a separate (Java) server and requires you to index documents over HTTP. The great thing about Solr is that you can talk to it from any language that has an HTTP client library.

The last option is to go for a hosted solution. Google Custom Search is free, but not particularly flexible. IndexTank was a good option here but they were acquired by LinkedIn and are shutting down the hosted service - they've since open sourced their software and other companies such as http://www.searchify.com/ are starting to offer it as a hosted solution.

Tags: mysql, php, search-engines, sphinx-search, quora

Who are major competitors to Solr?

2010-09-02T18:01:00+00:00

My answer to Who are major competitors to Solr? on Quora

ElasticSearch is a really interesting one - it's the same underlying search library (Lucene) and the same integration model (an HTTP interface) but takes quite a different approach. It hasn't been around for a long time but it looks very impressive: http://www.elasticsearch.com/

Other than that, popular open source search engines include Sphinx and Xapian. I'm a big fan of talking to a search engine via HTTP, so I've been keeping an eye on the http://www.flax.co.uk/ project which does that for Xapian.

Tags: apache, lucene, search, search-engines, solr, sphinx-search, xapian, quora

How do Solr, Lucene, Sphinx and Searchify compare?

2010-08-26T14:14:00+00:00

My answer to How do Solr, Lucene, Sphinx and Searchify compare? on Quora

Lucene is a Java library for creating and searching through a full text index. If you want to make use of it, you'll need to write your own Java code that integrates with it.

Solr is a web service that is built on top of the Lucene library. You can talk to it over HTTP from any programming language - so you can take advantage of the power of Lucene without having to write any Java code at all. Solr also adds a number of features that Lucene leaves out such as sharding and replication.

Tags: databases, lucene, search, search-engines, solr, sphinx-search, web-development, quora

Ravelry

2009-09-03T18:50:20+00:00

Ravelry

Tim Bray interviews Casey Forbes, the single engineer behind Ravelry, the knitting community that serves 10 million Rails requests a day using just seven physical servers, MySQL, Sphinx, memcached, nginx, haproxy, passenger and Tokyo Cabinet.

Tags: caseyforbes, haproxy, memcached, mysql, nginx, passenger, rails, ravelry, scaling, sphinx-search, tim-bray, tokyocabinet, tokyotyrant

Sphinx 0.9.9-rc2 is out

2009-04-08T13:59:26+00:00

Sphinx 0.9.9-rc2 is out

Interesting new feature: the Sphinx search server now supports the MySQL binary protocol, so you can talk to it using a regular MySQL client library and fire off search queries using SELECT syntax and the new SphinxQL query language.

Tags: full-text-search, mysql, search, sphinx-search, sql

In-Depth django-sphinx Tutorial

2008-03-05T00:03:45+00:00

In-Depth django-sphinx Tutorial

Another neat Django extension from the guys at Curse: easy integration with the sphinx full text search engine.

Tags: curse, david-cramer, django, python, search, sphinx-search

django-sphinx

2007-09-09T00:35:19+00:00

django-sphinx

More code from Curse Gaming; this time a really nice API for adding Sphinx full-text search to a Django model.

Via David Cramer

Tags: cursegaming, david-cramer, django, full-text-search, orm, python, search, sphinx-search