Elasticsearch plugin for UBI: Analyze user behavior from search queries

Want to get Elastic certified? Find out when the next Elasticsearch Engineer training is running!

Elasticsearch is packed with new features to help you build the best search solutions for your use case. Dive into our sample notebooks to learn more, start a free cloud trial, or try Elastic on your local machine now.

UBI (User Behavioral Insights) is an emerging standard to help search engineers capture and track usage events from search applications, from the front end all the way to the search engine. In this article, we will explore the UBI standard and explain how to capture analytics using the Elasticsearch plugin.

UBI explained

UBI can be used to understand what the user is doing on the application and provide information about what the user is searching, where on the site the user is interacting (search bar, search results, add to cart button, etc.), what result(s) the user is clicking, which results are coming back in that moment for that query, along with any other metadata you find useful to tune your search experience.

The most common metrics to capture and use are:

Top search queries: What is being searched for the most
No result queries: Queries that return no results
Most clicked: Results with the most clicks

Some examples of how the captured UBI can be used are:

Visualizations: Creating dashboards with the analytics data to make strategic decisions.
Relevance tuning: Applying query rules or synonyms based on your no-result queries, or adjusting queries.
LTR: Using UBI to create judgment lists to feed the LTR model and boost results based on criteria like clicks/views or other business requirements.

UBI is maintained on GitHub under the Apache 2.0 license.

UBI schemes

UBI proposes schemas for queries, which are triggered when a user searches, and for events, which are triggered by other interactions within the context of that search.

For instance, if a user searches for “shoe”, a query-type document is captured. Then, when the user clicks a result, an event-type document is created—correlated with the query-type document.

Query schema

The query schema stores the search text and result shown to users. It includes the following fields:

application
query_id
client_id
user_query Required
query_attributes
object_id_field
timestamp
query_response_id
query_response_hit_ids

A good starting point is storing user_query, timestamp, query_id, which will give you the time a query was executed, and the ID you can use to correlate with search events.

For more details on these attributes, check out the latest schema.

Event schema

The event schema captures all users' actions that follow a search, such as clicks and purchases. It includes the following fields:

application
action_name Required
query_id
session_id
client_id
user_id
timestamp Required
message_type
message
user_query
event_attributes

You can find the definition of each field here.

UBI in Elasticsearch

Since UBI is a standard and not a tool or library, we just need two components to implement it in Elasticsearch:

Application: We need to produce usage events that comply with the UBI standard from an application
Index: We need Elasticsearch indices to hold the data, and we can visualize the events with Kibana later on

The UBI team created an Elasticsearch plugin that automatically produces the indices and stores queries on them based on the _search requests the cluster receives. We will go this route. Alternatively, you can build your own events collector and send them with the right schemas to Elasticsearch.

To start capturing UBI events, we will go through the following steps:

A future article will cover the visualization part. Stay tuned!

Install the Elasticsearch plugin for UBI

The User Behavior Insights (UBI) plugin for Elasticsearch is designed to capture search queries and develop a better understanding of user behavior. While this plugin focuses on capturing server-side queries, the o19s/ubi repository handles client-side event capture.

The way it works is by accepting an additional ext parameter in the search query body, which allows sending a UBI event. When searching, it will internally create a document with the query executed and any other fields from the schema you put within ext.ubi. You can use the query_id field to correlate different events from the same search.

Compile the plugin

Start by cloning the repository:

Build the jar package with the plugin:

The bundle is created under the /build/distributions folder and is a zip file like elasticsearch-ubi-1.0.0-SNAPSHOT.zip. Save it for the next steps.

Install plugin

We are going to install the plugin on an Elastic Cloud instance. For self-managed deployments, you can follow the steps in the documentation.

1. Log in to Elastic Cloud, and go to Extensions:

2. Click “Upload extension,” select the elasticsearch-ubi-1.0.0-SNAPSHOT.zip file, and fill in the details. At the time of writing, the plugin Elasticsearch version is 8.15.2, so you must use that version or compile for a different one. You can check the Elasticsearch version of the latest code of the plugin by looking at the gradle.properties file.

3. After successfully uploading the extension, click on the Elastic logo to go to Elastic’s Cloud main page

4. On your working deployment, click Manage

5. Click on the Actions dropdown, then click on Edit deployment

6. Under Elasticsearch, click the Manage user settings and extensions link

7. Click on the Extensions tab and select the recently installed extension. In this case, it is ubi8. Then, click “Back” and save the changes.

Load sample data

Run the following command in the Kibana DevTools Console to load a couple of books to test the plugin. This will create a new index named “books” for this example.

Test the plugin

When calling the _search API, you can send an additional ext parameter—an object containing the fields described at the beginning of the article.

Indexing queries

Let’s write a query and use the plugin to capture behavioral metadata.

object_id_field: Is the field we want to use to identify our results and store them on the ubi_queries index as a result set.
user_query: What the user typed in the search bar.
client_id: Originator of the query
query_attributes: Arbitrary key/values.

Along with the results, the response will include a query_id that we can use to inspect the ubi_queries index:

The document stored in ubi_queries will look like this:

The document will contain the query executed by Elasticsearch, the metadata we provided, the list of documents matched, and also an additional identifier for the query response.

With this data, we can now analyze trends like top queries, top queries with no results, top documents in the results set, and apply filters on things such as client application, application component, and time-base filters.

Capturing events

Let’s imagine that during that same search, the user clicked one of the results, and we want to capture that event and correlate it with the query we just indexed. To capture client-side events (e.g, clicks on the results), the application must send the event to the ubi_events index:

Now we’ve captured the document, the click on that search, the position (position.ordinal), and the original value of id (object_id).

This event allows us to identify the most clicked documents, the user’s queries that generate the most clicks, and whether users are clicking results in the top positions or not.

Insights analysis

We can use ES|QL to analyze our data. Let’s look for the top 5 queries by counting the ubi_queries index documents by user_query:

You will get your results formatted as a table:

Conclusion

In this article, we learned how to capture UBI queries and events using the Elasticsearch plugin, and how to work with the basic fields needed to extract meaningful insights.

In the next article, we’ll take this further by capturing richer metadata, writing more advanced ES|QL queries, and building Kibana dashboards to help us better understand our users and deliver an improved search experience.

Report an issue