Refactoring our search capabilities using Elasticsearch and two new Spatie packages

At Flare, we're continously making improvements to our code base. A lot of the code has gone through multiple iterations and improvements. A part of our code base that was in need of some love was the errors & occurrences search. The original code was a Frankenstein mix of a deprecated Eloquent search query package and ElasticSearch with some custom SQL sprinkled on top.

The past few weeks we've spent time refactoring this code. Recentely, we deployed our improved-search branch to production. Let's dive into the changes in this refactor and the new packages that were born from it.

A re-usable strategy for search

Flare features three search fields throughout the application: one for projects, one for errors and one for error occurrences. The biggest annoyance for us was that all three of these search solutions were built differently.

The project search is a simple WHERE LIKE SQL query to filter on project name. It's basically a one-liner, so we left it that way.

The errors and error occurences search on the other hand are both complex search fields with optional filters, grouping and fuzzy filtering on multiple fields. In true DRY (don't repeat yourself) spirit we decided to build a unified search solution for both of these using ElasticSearch.

Additionally, by open-sourcing this solution we've extracted a lot of the complicated search code from our codebase into a package. This isolated package code is easier to test and easier to re-use by other developers. In turn, we get valueable feedback and free bugfixes in the form of PRs.

ElasticSearch using a custom query language

When talking about search, it's hard to ignore ElasticSearch. Having used it before for error occurrences we quickly decided to start indexing errors in ElasticSearch as well and fully rely on it as our search driver. ElasticSearch can handle most complex search queries you can come up with, if you know how to formulate it in a JSON search request.

Sadly, having a search field that only takes JSON is bad UX. We still need a way to convert a query string like class:QueryException status 404 group_by:user_id to the JSON request that ElasticSearch can understand. This is where the spatie/elasticsearch-search-string-parser package comes in handy.

Introducing spatie/elasticsearch-search-string-parser

Our new spatie/elasticsearch-search-string-parser package can dissect a search string like the example above into different "directives" like class=QueryException, grouping=user_id and the remaining status 404 for fuzzy filtering.

These directives can be dynamically defined and applied to a search string. Dissecting or parsing these directives works using regular expressions. This allows us to match basic directives like /class:(.*)/ but also more complex syntax if we wanted too.

Finally, every directive also knows how to add itself to an ElasticSearch query builder. For example, the class directive (class=QueryException) will add the following match query to the ElasticSearch builder:

{ 
  query: {
	"match": {
	  "class_name": {
		"query": "QueryException"
      }
	}
  }
}

Implementing search-string-parser in Flare

Using the search-string-parser package, it suddenly becomes really easy to replace both search controllers in Flare with two SearchQuerys and a couple directives. We even managed to re-use a couple of directives as both the errors and occurrences have a ClassNameDirective to search by exception class and a MessageDirective to search by exception message.

Finally, the search-string-parser packages uses another Spatie package under the hood: the elasticsearch-query-builder. It's features a fluent (Eloquent-like) API to to build ElasticSearch queries. It even comes with pagination support out of the box. This means it plugged right in to our custom paginator in Flare.

Future plans

Spoilers ahead! As we've now got access to the full power of ElasticSearch in both of our most important search fields, you'll see some cool search features coming up soon. One of the things we're most excited about is auto-completion on the directives in the search field. More on that soon™️!