Playing with distributed percolator from elasticsearch

I have recently attended an elasticsearch meetup in Amsterdam, where I heard the first time about a very cool feature of elasticsearch, called percolator.
Using the percolator, you can register queries against one or multiple indices and then you can send percolate requests with a document which return which registered queries it matched.

The percolator has been available in the product since the version 0.15, however starting from version 1.0 a fully distributed version of percolator will be available. The redesigned percolator will have a _percolator type mapping instead of a _percolator index, and the query and data will coexist with the same index. The redesigned percolator is also fixing the confusing part of the query registration API where the index name was represented as the type in the _percolator index. In the redesigned percolator this confusion is gone, because the _percolator became a type

	curl -XPUT 'http://localhost:9200/households/_percolator/my_id' -d {
	"query": {
	...
	}
	}

view raw request.sh hosted with ❤ by GitHub

There are many use cases for percolator. For example in case or a real estate website, registered users save their preferences (a query) about their dream house which they want to buy. When a new house is added to the website, a notification is sent to the registered users if there is match with their preferences (using percolate requests).
Let's consider the following model

	public class Household {
	private String id;
	private String description;
	private Double lat;
	private Double lon;
	private int price;
	private Date postDate;
	}

view raw Household.java hosted with ❤ by GitHub

Since automatic mapping of "geo enabled" properties has been disabled, you have to provide the correct mapping for geo properties. For the sake of simplicity XContentBuilder, a built-in utility from elasticsearch, is used to construct the JSON representation of the model. Another alternative would be to use Jackson.

	private void addGeoTypeMapping() throws IOException {
	XContentBuilder mapping = jsonBuilder()
	.startObject()
	.startObject("household")
	.startObject("properties")
	.startObject("location")
	.field("type", "geo_point")
	.endObject()
	.endObject()
	.endObject()
	.endObject();
	client.admin().indices().prepareCreate("households")
	.addMapping("household", mapping).execute().actionGet();
	}

view raw PercolateTest.java hosted with ❤ by GitHub

Let's consider that one user is interested in houses within 5km of Utrecht central station prices between 250000 and 300000. We could have the following filter

	private FilterBuilder filterBuilder() {
	return boolFilter()
	.must(rangeFilter("price").from("250000").to("300000"))
	.must(geoDistanceFilter("location")
	.point(52.09, 5.11)
	.distance(5, DistanceUnit.KILOMETERS)
	);
	}

view raw PercolateTest.java hosted with ❤ by GitHub

Using the above filter we can register the following query. The user id is saved in this query, in order to know to which user it belongs.

	client.prepareIndex("households", "_percolator", "myQuery")
	.setSource(jsonBuilder()
	.startObject()
	.field("query", constantScoreQuery(filterBuilder()))
	.field("user_id", "123")
	.endObject())
	.setRefresh(true) // Needed when the query shall be available immediately
	.execute().actionGet();

view raw PercolateTest.java hosted with ❤ by GitHub

When a new house is added then we can a send percolate request with the new document. Note that the new document is not added to the index.

	// create a source of the percolate request
	XContentBuilder docBuilder = jsonBuilder()
	.startObject()
	.field("doc")
	.startObject()
	.startObject("location")
	.field("lat", 52.10)
	.field("lon", 5.12)
	.endObject()
	.field("price", 290000)
	.endObject()
	.endObject();
	// send the percolate request
	PercolateResponse response = client.preparePercolate()
	.setIndices("households")
	.setDocumentType("household")
	.setSource(docBuilder)
	.execute().actionGet();
	assertThat(response.getCount(), is(1L));

view raw PercolateTest.java hosted with ❤ by GitHub

If there was a match we need to check to which user the query belongs in order to send notification.

	GetResponse matchedQuery = client.prepareGet("households","_percolator", "myQuery")
	.setFetchSource("user_id", "query")
	.execute().actionGet();
	assertThat(((String) matchedQuery.getSource().get("user_id")), is("123"));
	assertThat(matchedQuery.getSource().get("query"), is(nullValue()));

view raw PercolateTest.java hosted with ❤ by GitHub

A working example you find on my github account. In order to try it, you need to clone elasticsearch and build it locally, since the redesigned percolator will be available in the 1.0 release.