Skip to content

Elasticsearch Series 04 – Building Autocomplete API with Completion Suggester in ASP.NET Core

When we look at many popular commercial websites such as Amazon, Netflix and eBay, we can clearly see that autocomplete(search suggestion) boxes are important for the companies.

We know that a good search result is also important for end users. In terms of commercial websites, when we are able to direct the end user to the right product or category quickly, this situation will affect sales rates positively.

To be able to see related suggestions while we are writing, isn’t it great?

And I decided to write an autosuggest sample using Elasticsearch – Completion Suggester and .NET Core according to e-mails which followers requested.

There are several different ways to implement autocomplete/suggest feature in elasticsearch such as “ngrams“, “prefix queries” and “completion suggester“. Also, different implementations have some tradeoffs such as resulting and indexing speeds, document sizes etc… For “autocomplete/search-as-you-type” functionality example in this article, I will try to show how we can implement autocomplete feature in a most performant way(I think) using “completion suggester” feature.

Completion Suggester

I think resulting speed is important when it’s about to give an instant feedback to end users with a good suggestion result. At this point, completion suggester works differently from other implementation ways. All combinations to be suggested, need to be indexed on elasticsearch with a “completion” type mapping. Completion suggester uses an in-memory data structure called FST(Finite-state transducer) to provide fast lookup operation.
Thus, it can perform prefix lookup operation faster than other term-based queries.

Let’s take a look at the example on the elastic engineering blog to understand it working logic better. Suppose that “hotel“, “marriot“, “mercure“, “munchen” and “munich” words are on a FST.

On the in-memory graph above, suggester performs a matching process from left to right according to the text input which entered by a user. For example, when the user enters “h” as an input, this word will be completed immediately because the only possible matching option is “hotel“. If the user enters “m“, this time the suggester will list all words which start with “m“.

The disadvantage of completion suggester that matching process always starts from left to right as like above. For example “Sam” text will be matched with “Samsung Note 8” not with “Note 8 Samsung“. In such cases, term-based queries work more efficiently. However, as I mentioned above, we can also perform the match operation of “Note 8 Samsung” word by indexing all related combinations to be suggested on a single suggest output with completion suggester. I will mention this in an example later.

It is also possible to specify “Fuzzy Matching” with completion suggester and “Weights” for scoring operations.

Scenario

Let’s assume we are working at an e-commerce company. Product owner, responsible for search domain, asked us to create an autocomplete feature that gives a result close to real-time based on the “brand” and “product name” fields.

First we need to feed items, that we want to suggest on elasticsearch.

1) Creation of Mapping and Index

We have to create a mapping, which has completion type, to use completion suggester. For that, first create a .NET Core class library called “Autocomplete.Business.Objects” and include “NEST” library using NuGet package manager.

We created this library as separately, because we will use the models which we define here, in both feeder application and autocomplete API.

First, define “Product” and “ProductSuggestResponse” models as follows.

We will use the “Suggest” property in the “Product” model for texts that we want to suggest during autocomplete.

Create a new .NET Core class library called “Autocomplete.Business“, then include the “Autocomplete.Business.Objects” and “NEST” libraries. After that let’s define an interface called “IAutocompleteService“.

and implement it as follows.

If we look at the “CreateIndexAsync” method above, we did mapping process of “Product” model. Also, we specified “Suggest” property in the “Product” model as completion field. At this point, default “simple” analyzer is used as the analyzer. Simple analyzer divides all text into terms as lower case. If you want, it is possible to replace the analyzer with the “Analyzer” method on the completion.

The “SuggestAsync” method will be used during autocomplete. Actually, we specified that we will perform completion operation with the “Suggest” field. Also, we set text input that user will enter, to the completion using the “Prefix” method. I guess, “Fuzzy” is great feature while autocomplete process. Sometimes we can make simple typo mistakes when we write, right? 🙂

And the last one is, we mapped “Text” and “Score” properties of suggestion options which are come from “searchResponse“, to “ProductSuggest” model.

NOTE: It is also possible to access original document of suggested text from “Source” property.

Finally, we can start to develop console application which will feed suggestion documents to elasticsearch. So, let’s create a .NET Core console application project called “Autocomplete.Feed“, then include the “Autocomplete.Business.Objects” and “Autocomplete.Business” libraries.

First of all, we need an elasticsearch instance to test. So let’s run the following command on Docker.

and change the “Program.cs” class with the following code blog.

If we look at code blog above, firstly we created the products that will be used in autocomplete. Also while creating the products, we set the inputs, these are wanted to match with the related product, to “CompletionField” property of each product. Thus, if a user writes an input such as “Galaxy Note 8” or “Note 8“, we can provide matching these inputs with “Samsung Galaxy Note 8“.

After creation of the products, we have provided feed operation to elasticsearch using “AutocompleteService” class, which created before.

So we have a feeder project now, and let’s run it. If it runs successfully, we can see the “product_suggest” index on the elasticsearch with the following mapping.

GET product_suggest/_mapping

2) Designing of Autocomplete API

Now, all we need is to do design an API, to expose autocomplete feature. Let’s create a .NET Core Web API project called “Autocomplete.API“, then include the “Autocomplete.Business.Objects“, “Autocomplete.Business” and “NEST” libraries.

After that, create a controller called “ProductSuggests“.

In the “Get” method, we just return the “ProductSuggestResponse” using the “IAutocompleteService“.

Also in the “Startup” class, we need to inject services into service collection.

That’s all.

Let’s run the API project for testing autocomplete feature, and then assume a user entered “iph“.

GEThttp://localhost:5000/api/product-suggests?keyword=iph

If we look at the response above, we can see that the related results for the “iph” such as “Iphone 8“, “Iphone X” and “iPad Pro“.

Now if we assume the user entered “app” instead of the “iph“:

and this time “Apple Iphone 8“, “Apple Iphone X” and “Apple iPad Pro” results will suggested to the user.

Conclusion

As I mentioned the beginning of this article, there are many ways implement to autocomplete feature in the elasticsearch with some tradeoffs. Completion suggester works faster than term-based queries. Also, it is a disadvantage of completion suggester to start matching operation at the beginning of the text. In addition, sort order options are limited.

https://github.com/GokGokalp/Elasticsearch-Autocomplete-API-Sample

References

https://www.elastic.co/blog/you-complete-me

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-suggesters-completion.html

Bu makale toplam (4536) kez okunmuştur.

26
0



Published inASP.NET CoreSearch Engine

10 Comments

  1. Kirill Kirill

    Thanks for the great article! You are describing a lot. I only wanted to ask what if we are using some sort of db with a big amount of info in it. How would it be right to create suggesters – I see you inputing them by yourself

    • Hi Kirill, thanks for your interest. Could you please give me a little more information about your question? What you mean with “I see you inputting them by yourself”? Is it about indexing data to the elassticssearch from the db or something else?

      • Kirill Kirill

        Well, you have your own examples in your code. I’ve changed your solution a bit and now I have an opportunity to post some kind of news with fields like Name, Tags, Short Description. I’ve wanted to ask if you know how to connect ES with MS SQL for example(I’ve read smth about logstash). And is it possible to suggest not only Name but Description too? Thanks for your time

        • Hi Kirill, I’m sorry for late reply, these days I a little bit busy with marriage preparing. 🙂 So, yes you can set multiple inputs to one suggestion item. E.g : Input = new[] { “Name”, “Tags”, “Description” } If these fields match any search term, you can change will be displayed text what you want. Is the same thing with my example. Thanks.

  2. hamid hamid

    Hi.
    Thanks for your nice article.
    I have a one problem for implementing autocomplete suggester in asp.net core and sql server.
    How to create index of multiple fields of some tables?
    For example, I search by Product name or category name or product attributes and etc.
    Thanks.

    • Hi Hamid, thanks.

      To do that, there are a few ways you can choose.
      For example, while you prepare index data, you can add more input text for search by product name, category name or attributes etc…

      products.Add(new Product()
      {
      Id = 3,
      Name = "Apple Iphone 8",
      Suggest = new CompletionField()
      {
      Input = new[] { "Apple Iphone 8", "Iphone 8", "Cell Phones", "128GB Iphone 8", "Silver 128GB Iphone 8", "ADD WHAT YOU WANT" }
      }
      });

      or you can create different index and use term analyser instead of completion suggester. Thus you can search by product name, category name or product attributes as parallel, then you can aggregate the result simultaneously.

      Regards

  3. crbn crbn

    Merhaba,
    ES direk bizim var olan bir dbdeki tabloyu içine alıp her insert update delete yaptığımızda da ES’yi de mi güncellememiz gerekiyor kurulu bir sistemde ürünler tablosunda ki milyonlarca satır arasından arama yapması için ES nasıl kullanilabilir.

    • Merhaba evet, nerede arama yapmasını istiyorsanız onunla ilgili index’lerinizi oluşturmanız ve her değişimde o index’leri up-to-date tutmanız gerekmektedir.

  4. Serhat Serhat

    Hocam merhaba,
    Suggest ile aramada sorun yok ama ben buna ek bir field daha ilave etmek istediğimde hata alıyorum sürekli.
    Yani suggest ile arasın ama sadece userid =5 olanları getirsin şeklinde bir sorgu yazamadım bir türlü.

    Yardımcı olabilir misin?

    • Merhaba, completion suggestor FST yapısını kullandığı için bu iş için uygun mudur bilemedim. Ek filtreler takmak istiyorsanız eğer, completion suggestor yerine term-based bir yapı kullanmanızı önerebilirim.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.