Simon Willison's Weblog: elasticsearch

GitHub Issues search now supports nested queries and boolean operators: Here’s how we (re)built it

2025-05-26T07:23:17+00:00

GitHub Issues search now supports nested queries and boolean operators: Here’s how we (re)built it

GitHub Issues got a significant search upgrade back in January. Deborah Digges provides some behind the scene details about how it works and how they rolled it out.

The signature new feature is complex boolean logic: you can now search for things like is:issue state:open author:rileybroughten (type:Bug OR type:Epic), up to five levels of nesting deep.

Queries are parsed into an AST using the Ruby parslet PEG grammar library. The AST is then compiled into a nested Elasticsearch bool JSON query.

GitHub Issues search deals with around 2,000 queries a second so robust testing is extremely important! The team rolled it out invisibly to 1% of live traffic, running the new implementation via a queue and competing the number of results returned to try and spot any degradations compared to the old production code.

Tags: elasticsearch, github, ops, parsing, ruby, scaling, search, github-issues

Elasticsearch is open source, again

2024-08-29T20:50:41+00:00

Elasticsearch is open source, again

Three and a half years ago, Elastic relicensed their core products from Apache 2.0 to dual-license under the Server Side Public License (SSPL) and the new Elastic License, neither of which were OSI-compliant open source licenses. They explained this change as a reaction to AWS, who were offering a paid hosted search product that directly competed with Elastic's commercial offering.

AWS were also sponsoring an "open distribution" alternative packaging of Elasticsearch, created in 2019 in response to Elastic releasing components of their package as the "x-pack" under alternative licenses. Stephen O'Grady wrote about that at the time.

AWS subsequently forked Elasticsearch entirely, creating the OpenSearch project in April 2021.

Now Elastic have made another change: they're triple-licensing their core products, adding the OSI-complaint AGPL as the third option.

This announcement of the change from Elastic creator Shay Banon directly addresses the most obvious conclusion we can make from this:

“Changing the license was a mistake, and Elastic now backtracks from it”. We removed a lot of market confusion when we changed our license 3 years ago. And because of our actions, a lot has changed. It’s an entirely different landscape now. We aren’t living in the past. We want to build a better future for our users. It’s because we took action then, that we are in a position to take action now.

By "market confusion" I think he means the trademark disagreement (later resolved) with AWS, who no longer sell their own Elasticsearch but sell OpenSearch instead.

I'm not entirely convinced by this explanation, but if it kicks off a trend of other no-longer-open-source companies returning to the fold I'm all for it!

Via Hacker News

Tags: aws, elasticsearch, open-source

The Cloud and Open Source Powder Keg

2019-03-17T19:08:18+00:00

The Cloud and Open Source Powder Keg

Stephen O’Grady’s analysis of the Elastic v.s. AWS situation, where Elastic started mixing their open source and non-open source code together and Amazon responded by releasing their own forked “open distribution for Elasticsearch”. World War One analogies included!

Via @jessfraz

Tags: aws, elasticsearch, open-source

django-zombodb

2019-02-13T22:14:53+00:00

django-zombodb

The hardest part of working with an external search engine like Elasticsearch is always keeping that index synchronized with your relational database. ZomboDB is a PostgreSQL extension which lets you create a new type of index backed by an external Elasticsearch cluster. Updated rows will be pushed to the index automatically, and custom SQL syntax can then be used to execute searches. django-zombodb is a brand new library by Flávio Juvenal which integrates ZomboDB directly into the Django ORM, letting you add Elasticsearch-backed functionality with just a few lines of extra configuration. It even includes custom Django migrations for enabling the extension in PostgreSQL!

Via @flaviojuvenal

Tags: django, elasticsearch, postgresql

elasticsearch-dump

2018-04-09T22:10:36+00:00

elasticsearch-dump

Neat open source utility by TaskRabbit for importing and exporting data in bulk from Elasticsearch. It can copy data from one Elasticsearch cluster directly to another or to an intermediary file, making it a swiss-army knife for migrating data around. I successfully used the “docker run” incantation to execute it without needing to worry about having the correct version of Node.js installed.

Tags: elasticsearch, nodejs, docker

Building a Full-Text Search App Using Docker and Elasticsearch

2018-02-01T15:41:03+00:00

Building a Full-Text Search App Using Docker and Elasticsearch

Deep, comprehensive tutorial from Patrick Triest showing how to use docker-compose to run three containers (Node API, nginx static content, elasticsearch) and then use that to build a neat Vue.js web search UI against 100 books from Project Gutenberg.

Tags: elasticsearch, nodejs, docker

Quoting Jason Tedor

2017-11-14T22:40:00+00:00

We are actively developing cross datacenter replication (internally we are calling it "cross cluster replication" so you will likely see it referred to this in the future but of course this is subject to change). I can not give a timeframe, but it is one of the top features on the Elasticsearch roadmap.

— Jason Tedor

Tags: elasticsearch

gillyb/sensitive: A native desktop version of the kibana sense plugin

2017-11-04T19:35:41+00:00

gillyb/sensitive: A native desktop version of the kibana sense plugin

I love using the Sense UI for developing against Elasticsearch, but it’s infuriatingly hard to obtain these days. You can install it as a Kibana plugin but I work with multiple Elasticsearch instances and I don’t want to have to get it installed on all of them. Until recently I was using a Chrome extension for it, but that’s now been disabled as containing malware and removed from the Chrome extension store. I’ve now switched to Sensitive, which packages Sense up as a native OS X application using Electron.

Tags: elasticsearch, electron

The denormalized query engine design pattern

2017-08-16T22:49:22+00:00

I presented this talk at DjangoCon 2017 in Spokane, Washington. Below is the abstract, the slides and the YouTube video of the talk.

Abstract

Most web applications need to offer search functionality. Open source tools like Solr and Elasticsearch are a powerful option for building custom search engines… but it turns out they can be used for way more than just search.

By treating your search engine as a denormalization layer, you can use it to answer queries that would be too expensive to answer using your core relational database. Questions like “What are the top twenty tags used by my users from Spain?” or “What are the most common times of day for events to start?” or “Which articles contain addresses within 500 miles of Toronto?”.

With the denormalized query engine design pattern, modifications to relational data are published to a denormalized schema in Elasticsearch or Solr. Data queries can then be answered using either the relational database or the search engine, depending on the nature of the specific query. The search engine returns database IDs, which are inflated from the database before being displayed to a user - ensuring that users never see stale data even if the search engine is not 100% up to date with the latest changes. This opens up all kinds of new capabilities for slicing, dicing and exploring data.

In this talk, I’ll be illustrating this pattern by focusing on Elasticsearch - showing how it can be used with Django to bring new capabilities to your application. I’ll discuss the challenge of keeping data synchronized between a relational database and a search engine, and show examples of features that become much easier to build once you have this denormalization layer in place.

Video

Slides

Tags: architecture, definitions, denormalisation, design-patterns, django, djangocon, elasticsearch, my-talks

elasticsearch: Percolator

2011-02-08T23:16:00+00:00

elasticsearch: Percolator

Another fascinating elasticsearch feature: Percolator lets you register searches with your elasticsearch cluster, then pass in a document and have the matching query IDs returned. It’s an upside down search engine. I’m sure there are some very neat things you could build with this, I just haven’t figured out what they are just yet.

Tags: elasticsearch, search, recovered

ElasticSearch memcached module

2010-05-15T10:17:00+00:00

ElasticSearch memcached module

Fascinating idea: the ElasticSearch search server provides an optional memcached protocol plugin for added performance which maps simple HTTP to memcached. GET is mapped to memcached get commands, POST is mapped to set commands. This means you can use any memcached client to communicate with the search server.

Tags: elasticsearch, http, memcached, protocol, recovered

Search Engine Time Machine

2010-02-17T22:32:55+00:00

Search Engine Time Machine

Detailed explanation of how ElasticSearch provides high availability, through clever sharding and replication strategies and configurable gateways for long-term persistent storage.

Tags: elasticsearch, highavailability, scaling, search

ElasticSearch: Your Data, Your Search

2010-02-12T15:22:09+00:00

ElasticSearch: Your Data, Your Search

A neat example of how ElasticSearch’s schemaless indexes and native JSON support make it ridiculously easy to index different types of data and run queries across them.

Tags: elasticsearch, java, json, schemaless, search

Elastic Search

2010-02-11T18:33:14+00:00

Elastic Search

Solr has competition! Like Solr, Elastic Search provides a RESTful JSON HTTP interface to Lucene. The focus here is on distribution, auto-sharding and high availability. It’s even easier to get started with than Solr, partly due to the focus on providing a schema-less document store, but it’s currently missing out on a bunch of useful Solr features (a web interface and faceting are the two that stand out). The high availability features look particularly interesting. UPDATE: I was incorrect, basic faceted queries are already supported.

Via ElasticSearch blog

Tags: elasticsearch, http, java, json, lucene, rest, scaling, search, sharding, solr