Working with your App Search data in a post Enterprise Search world

Want to get Elastic certified? Find out when the next Elasticsearch Engineer training is running!

Elasticsearch is packed with new features to help you build the best search solutions for your use case. Dive into our sample notebooks to learn more, start a free cloud trial, or try Elastic on your local machine now.

App Search is discontinued with the Elastic Stack 9.0. Wait—what? If you want to upgrade to Elasticsearch 9, what can you do with all the data and functionality you have using App Search? Fear not, you can still use it and in this blog we’ll show you how.

If you’ve been a user of Elastic’s Enterprise Search, starting in Enterprise Search version 8.16.0, you have seen the decommissioning callouts in the App Search and Workplace Search UIs. Those callouts alert you to the fact that Enterprise Search will no longer ship any new major versions beyond 8.x. Nonetheless, you can still access and work with your data from your previous Enterprise Search installation if you do decide to upgrade the rest of your Elastic Stack to 9.0 and beyond.

Though Enterprise Search will not work directly with Elasticsearch 9.0 and beyond, any previous data that you’ve had in your Enterprise Search instances will still be available for you to use going forward. More information about this can be found in the blog post: From App Search to Elasticsearch — Tap into the future of search.

Engines and documents

Just a note before we get started: all underlying Enterprise Search system indices will start with the prefix .ent-search-actastic-*, and most have a version number assigned to them.

For the first step, we’ll get a list of all the engines in your Enterprise Search instance. To do this, perform a search on the .ent-search-actastic-engines_v26 index.

Don’t worry too much about the _v* suffixes you might see in the index names. These simply indicate the various revisions Enterprise Search has gone through over the years, with version 26 of the engines index being the latest.

After retrieving the results of the search, you will see the records for the engines in your instance. The fields we want to return and take note of are specifically the engine ID, the “slug” (the canonical name used in our URLs), and the actual engine name. For the purposes of this example, two sample engine documents look like:

And:

If your App Search instance is using any meta engines, you can see the engines that are included in the meta engine by also getting the source_engine_ids in the above call.

In this example, we have one meta engine named “test-meta-engine” and we can see that the source IDs are the two other engines.

Retrieving your documents

Now that you have the list of your engines and their respective IDs, you can query the indices for the documents that belong to those engines. The documents for an engine are stored in an index named enterprise-search-engine-{slug}, where {slug} is the item from above.

For example:

This will return the documents in the “crawled-docs” index. The actual content and structure of the documents in your index will depend on the source data that you have ingested.

Synonyms

To see your synonym sets, you can query on the .ent-search-actastic-synonyms index. Note that this will return all synonym sets for all engines. If you only want to display the synonyms for a specific engine, you can use a match query to filter them. For example, for the “national-parks-demo” engine above, we can use the engine ID in a match query:

The returns will be a set of documents with the synonym information. Each set will contain the engine ID for the synonym set, and the terms in the set. For example:

With this information, you can use the Elasticsearch synonyms API to create synonym sets that will work with any index.

Curations

To get the list of curations for a given engine, query via:

This will return the queries that are set for curation and the order of those queries. For example:

Note also that we also get back the document_position_id, which will then allow us to list the promoted documents for a given query. To retrieve a list of all promoted documents for queries and their positions for a particular engine, you can provide the engine ID in a match query, and query on:

Which will return results with the format of:

Note the document IDs in the “modified” field above, which relate to the document IDs as they are stored in the documents index for this engine.

App Search curations in Elasticsearch 9 can be achieved by using query rules. Query rules allow you to tailor your results for queries in much of the same manner as curations would.

Relevance settings

If you have applied any relevance tuning to your engines, you can also retrieve that information by querying the engine itself. For example, on our sample engine, we want to display the search_setting field to see what we applied there:

In this example, this returned our defined search fields and any boosts we might have applied:

For creating dynamic and relevant queries using your App Search data, see the section titled “How to build App Search like queries” in the App Search Engine Exporter notebook.

Analytics

All Enterprise Search analytic logs are stored in datastreams. Specifically for App Search logs, the logs-app_search.analytics-default datastream indices that are used to construct the views in the “Analytics” tab within App Search.

To query for the datastream details, you can use:

The datastream acts like any regular index, but is append-only. You can search across the records in the datastream using the _search operator like any other index. E.g.:

What about the App Search user interface?

Much of the user-facing functionality you had with App Search can still be gained using the Elastic Search UI. Now that you know where your data lives and how to access it, you can use the Elastic Search UI to provide your users with a first-class search experience on your App Search data. You can play around with configuring the Elastic Search UI using our sandbox.

For the back-end configuration, query tuning, and index options, you will need to use the underlying Elasticsearch API calls and query language to better tune your experience for the end user. Much of this information can be found in the App Search Engine Exporter notebook.

Conclusion

Hopefully with this information, you will be able to work with any App Search data after upgrading to the Elastic 9 stack. You can see a list of Elasticsearch equivalent functionality as found in this table in the Elastic documentation. For some additional information, check out the couple of notebooks linked below for additional migration information.

If you are starting with a fresh installation from version 9.0 and after, note that Enterprise Search will not be included, and none of the above indices will be created. If you have an existing Enterprise Search installation that you would like to export and copy data into your new Elastic stack, check out the App Search Engine exporter notebook. This notebook talks about many of the items discussed here and shows examples of how to export and migrate those from App Search to Elasticsearch. It also goes into detail about the underlying structures and analyzers used in App Search indexes, so you can see what is happening underneath the hood.

Also, for any crawler data and schedules you may have, see the App Search Crawler to Open Crawler migration notebook. This notebook walks you through exporting your App Search crawler setup, and how to apply that to using the Elastic Open Crawler.