Building a better search with Monaco and amCharts

A couple of months ago we refactored a considerable part of Flare's search to better utilise ElasticSearch's amazing search capabilities. We introduced better filters with autocompletion and fuzzy matching. Using our elasticsearch-search-string-parser package we also created a custom search language to query ElasticSearch.

Inspired by GitHub's issue and PR search, we decided to keep the UI as simple as possible: a plain text field with a simple dropdown for basic autocompletion. You can read all about this blog post.

Even though we were pretty happy with the results, we felt that we could do better. Let's git checkout search-v2 and see what's going on.

A better date range selector

For a start, using a plaintext query to specify a timeframe is not very fun. Queries like is:unresolved received_at:>2021-08-01 received_at:<=2021-08-31 are a pain to read –let alone write– especially without syntax highlighting.

We quickly decided to replace the received_at filters with a fancy UI component we had been eyeing for a while: amCharts' zoomable/scrollable charts and their range selector plugin.

This doesn't only solve the time range selector problem, but also provides a neat graph with the number of occurrences for each error matched by the search query. It also comes with some unique features like being able to handle 43.000 data points (one per minute for 30 days) and automatic grouping per day, hour or 10-minute interval, based on the zoom-level.

Resolving the chart data points was a bit tricky. We really wanted to filter the charts data using the ElasticSearch query. However, the occurrence counts and timestamps are only stored in the MySQL database. As a solution we ended up querying ElasticSearch for only the error IDs and nothing else. This kept the query short and the response data small. We then query MySQL for the actual occurrence counts per minute for only those error IDs.

Depending on the complexity of the search query, this whole process takes somewhere around 450ms. Because each query is pretty unique (different time frames, different search query, errors coming in all the time) we're not attempting to cache the graph data at all. In theory we could hash all these variables together and cache the graph data, but the cache hit percentage would be too low to make it worth it. That doesn't mean we can't manually cache some of the most frequent queries. For example, the occurrence graph data for the default search query (is:unresolved is:unsnoozed) is fetched entirely from cache without every hitting the ElasticSearch server.

A better search field

At this point we had a cool looking occurrence graph next to a very plain search field. Inspired by Sourcegraph's search and this cool React package we decided to try to compress the entire Monaco editor (the open-source editor that powers VS Code) down to a single line and use it as an input field. As the Monaco editor is super extensible we can easily provide context-aware autocompletion, syntax highlighting and other goodies. Let's dive into the details.

Installing Monaco

The Monaco editor is no lightweight package. It comes in at 3MB (ungzipped) and includes a Webpack loader plugin to help shave off some of those unnecessary kilobytes. However, to avoid having to deal with Webpack and it's infinite complaints about our bundle, we decided to go with a suren-atoyan/monaco-react. As the name suggests, this is a React wrapper around the Monaco editor. More importantly: it includes its own monaco-loader that works out of the box and without having to deal with Webpack.

Monaco on a single line

We can't simply set the editor's height to 1em to turn it into a single line input field. There are a couple gotchas that need to be taken care of as well.

For starters, pressing the enter key in most code editors will insert a newline. It's really hard to get rid of that behaviour in Monaco without hacking into its internal event listeners and keybindinds (believe me, we tried). The easy way around is to hook into the editor's onChange event and literally replace every newline character with an empty string: value.replace(/[\n\r]/g, '').

<Editor
    height="16px"
    value={value}
    onChange={value => setValue(value.replace(/[\n\r]/g, ''))}
    options={options}
/>

While this is a somewhat unconventional solution, it still works with no performance impact or fanthom newlines.

Apart from setting the editor height to 16px (= single line-height) and removing any newlines, we're passing in some options too. This options object contains a lot of the same keys you might recognise from VS Code's settings. For example, we had to disable line numbers using lineNumbers: 'off'. We also hid the default minimap, disabled all rulers, unregistered all keybinds and disabled the context menu. The devil really is in the details. After all those changes, our <SingleLineMonacoEditor /> component finally rendered a blank 16px high rectangle that can be typed in. Perfect!

Adding a custom language for search

To be able to highlight custom syntax, you need to be able to parse the custom syntax. Or so we thought. In practice it's really a lot easier. The Monaco editor only expects a tokensProvider that can convert a document into an array of tokens. Even easier: it comes with a token provider called Monarch out of the box and it honestly feels like cheating. Monarch is configured using only JSON and the Monarch config for highlighting all JavaScript code is only 150 lines (switch to the JavaScript sample).

For Flare's query language, we don't even need most of Monarch's features. We'll only provide an array of keywords (each search filter is a keyword) and a rule to apply a filter tag to keyword tokens. Take a look at the literal 13 lines of code for Flare's tokeniser:

{
    keywords: filter.map(filter => filter.keyword),
    tokenizer: {
        root: [
            [
                /\w+/, { cases: { 
	                '@keywords': 'directive', 
	                '@default': 'identifier' 
				}},
            ],
        ],
    },
}

In the tokeniser above, we're once again taking the easy route. The \w+ regex will first match every "word" in the code. The tokeniser will then look at each of those words and determine if it's one of the @keywords we specified earlier. If it is, we're applying the directive type to the token. If it's not, we'll fall back to the @default case and apply the identifier type.

Finally, VS Code or Monaco are able to style these keywords using the configured a color scheme. We slapped some Flare purple on the default VS Code color scheme, registered it and called it a day:

monaco.editor.defineTheme('flare-light', {
    base: 'vs',
    inherit: true,
    rules: [{ token: 'directive', foreground: '#7900f5' }],
    colors: { foreground: '#332f51' },
});

Autocompletion and code suggestions

Getting suggestions from an external API to show up at the right places in the editor was probably the hardest part of this entire setup. Once again we feared having to dive into the world of tokenisers and abstract syntax trees to provide suggestions based on the current token and its context. Luckily, Monaco doesn't really care about any of that stuff and just expects us to return a Array<string> with suggestions based on the cursor position. Here's a quick example that will suggest foo and bar anywhere in the editor:

monaco.langagues.registerCompletionItemProvider('flare-query-language', {
    provideCompletionItems: (model, position) => {
	    suggestions: ['foo', 'bar']
    }
})

For Flare specifically, we're only able to suggest either filter keywords (e.g. class, version, ...) or filter values. The filter keyword suggestions are all a hardcoded array but the filter values need to be queried from ElasticSearch using our back-end API. We'll use the following regex to determine whether our cursor is currently in a filter keyword or in a filter value:

/\b(?<filter>[^\s:]+):?(?<value>\S+)?/g

If you don't speak regex, this just means to match any key:value occurrence in the query. You can also see the : and value parts are optional. This is because we also want to match the start of a new filter statement when the user is still typing. We can than then loop over all matches and compare the regex match.index and match.length to the cursor's position to determine if we're in a filter keyword or a filter value and make suggestions accordingly.

As mentioned before: if we're autocompleting a filter keyword we can simply return the a hardcoded array of filter keywords. For suggesting filter values, we need to execute and asynchronous API call. Luckily, Monaco supports returning a Promise<CompletionList> from the provideCompletions method. In pseudo-code, all of that looks like this:

const filters = ['version', 'class', 'type', 'is'];

async function provideCompletions(code position) {
 	// If we're autocompleting filter values, make an API request
    if (isFilterValue(code, position)) {
        const suggestions = await fetchSuggestions(query, position);
        
        return {
            suggestions,
        };
    }

  	// We're autocompleting a new filter or random text, return all available filters
    return {
        suggestions: filters,
    };
}

Finally, using some black magic and a stroke of luck we also managed to add a debounce in there without upsetting React's delicate rendering cycle.

Bonus round: tooltips on hover

Most of Flare's filters are pretty self explanatory and we initially weren't planning to show tooltips. Despite that, we figured that we're already downloading 3MB of Monaco editor including tooltip support so we might as well use it. /s

Once again, Monaco makes it really easy to hook in a custom hoverProvider. Just like the suggestion provider, the hover provider is given the entire document and the current mouse position as if it were a cursor. This means hovering over the 5th character in the document will pass 5 for the position. It then expects us to return the contents of a tooltip, if we want to show one.

We can re-use the filter:value regex from the suggestion provider above to once again determine if we're hovering over a filter keyword or a value. We'll then provide tooltips for just the filter keywords:

function provideHover(code, position) {
	// Use regex to extract the filter keyword or value the cursor is in:
	const currentWord = getCurrentWord(code, position);

	if (word.type !== 'keyword') {
		return null;
	}

    return {
        range: new Monaco.Range(1, word.start, 1, word.end),
        contents: [
            { value: `**${word.value}**` },
            { value: keywords[word.value]?.description }
        ],
    };
},

Lazy loading the input field

Apart from the single-line Monaco configuration and the custom language definition, we're using a couple other tweaks to make the experience as smooth as possible. One of these tweaks is the loading indicator – or rather, lack thereof.

The monaco-react package already show a loading placeholder that'll show a Loading... label by default. Instead of that label, we can also pass in a React component like a fully working input field styled to look exactly like the editor:

<Editor
    value={value}
    onChange={handeOnChange}
		// ...
    loading={<InputField value={value} onChange={value => handleOnChange(value)} />}
/>

The onChange callback can even be hooked up to capture any changes to the input field before the Monaco editor even loads. Once loaded it can then transition seamlessly into the full-blown editor.

So you spent 2 months installing a graph component and copying VS Code?

Yes! And we're pretty proud of it too. In the coming weeks we'll also bring this occurrence graph and the new search field to the error occurrence detail page. We're also looking at what other data we can index to provide useful search filters. Feel free to start a free 10-day trial (no credit card required) to play around with the new search features and be sure to let us know your feedback and ideas for interesting filters!