How to display fields of an Elasticsearch index

Want to get Elastic certified? Find out when the next Elasticsearch Engineer training is running!

Elasticsearch is packed with new features to help you build the best search solutions for your use case. Dive into our sample notebooks to learn more, start a free cloud trial, or try Elastic on your local machine now.

In this article, we will discuss how to display fields of an Elasticsearch index. This can be useful for understanding the structure of your data, identifying specific fields, and troubleshooting issues. We will cover the following topics:

1. Using the _mapping API to retrieve field information
2. Using the _search API to display field values
3. Displaying sub-fields
4. synthetic _source
5. Runtime fields

1. Using the _mapping API to retrieve field information

The _mapping API allows you to retrieve the mapping definition for an index or multiple indices. This includes information about the fields, their data types, and other properties. To retrieve the mapping for a specific index, use the following request:

For example, if you have an index named my_index, you can retrieve its mapping with the following request:

The response will include the mapping definition for the index, which contains information about the fields and their properties.

It is also possible to retrieve the mapping of one specific field. This can be useful if your mapping is quite big and you only want to focus on a specific field. To retrieve the mapping of a specific field, use the following request:

You can also retrieve the mappings of several fields by separating their names with commas, as in the following request:

2. Using the _search API to display field values

To display the values of fields in an Elasticsearch index, you can use the _search API. The _search API gives you multiple ways to control which fields are returned; the two main ones are:

1. _source: The _source field contains the original JsON document body exactly as it was indexed, including any changes made by ingestion pipelines or pre-processing steps. To display specific fields from the source document, implement source filtering as we will see below.
2. fields: The fields parameter lets you retrieve specific fields from your documents when performing a search, based on the index mapping. Unlike _source, fields can also return values from stored fields, doc values, or runtime fields without referencing the _source, though for standard fields without doc values or stored settings, it falls back to _source. This can bring many benefits such as performance and more, as we will see below.

Using the _source field

By default, the _search API returns the _source field, which contains the original JsON document that was indexed. To display specific fields, you can add filters in the _source parameter of the search request; this is called source filtering.

Here’s an example of a search request that returns the values of the title and author fields for documents in the my_index index:

In this example, the _source parameter specifies the fields to be returned.

If you need even more control, you can use the _source object’s includes and excludes properties. For example, the query below returns the top-level title field and all the sub-fields of author except for author.description.

In this example, we use the author.* pattern to retrieve every direct sub-field of the author object. Then we explicitly exclude author.description so that only the other author fields are returned. Note that this doesn’t have any performance enhancements since it still has to load and parse the source JsON, but it can lessen the size of the response sent over the network.

Using the fields parameter

You can use the fields parameter to filter the fields returned in the search response. Using fields over _source offers several benefits, including:

• Improved performance: fields can return values directly from stored fields or doc values without having to load the full _source, making the response payload size smaller.
• Formatted output: For standard fields, fields may fall back to _source to grab the values, but it looks at the index mapping to properly format the output, such as formatted dates, making them consistent with what’s used for aggregations and sorting.
• Access to runtime fields: fields can return runtime fields, which don’t exist on the original _source.
• More benefits can be found here.

For example, to return only the title and author fields in the my_index index, you can use the following search request:

In the query above, we set the _source field to false so we don’t return the source document. This can dramatically minimize the payload size for the response, but remember this only works because the fields title and author are of the keyword field type, which have doc_values enabled by default. If the field doesn’t have doc_values enabled and the _source was set to false, Elasticsearch would have no way to retrieve them and they would be skipped in the response.

It’s important to note that the fields response always returns an array of values for each field, even if there’s only a single value. This is due to Elasticsearch having no dedicated array type, and any field could have multiple values. For more information on arrays in Elasticsearch, click here.

Other ways to retrieve fields

Although retrieving fields using _source or fields are the recommended methods, there are different methods available for specific use-cases, such as:

Doc value fields: If you want to avoid _source altogether, you can search using the docvalue_fields parameter. Doc values store the same field values as _source but in an on-disk data structure, optimized for sorting and aggregations.

since it’s separate from the values stored with _source, you can request specific fields without loading the entire _source. This is useful if you’re querying large documents but only need a few small fields that support doc values. Another use case for using docvalue_fields is when you want to use custom formatting on date and numeric fields, as we will see in the example below.

Note that this only works for fields that you enable doc_values for or for field types that have it enabled by default such as keyword, date, numeric types, and boolean, not for text or annotated_text.

In this example, we use the docvalue_fields parameter to retrieve the title, author, and published fields without loading the full _source document:

When this query runs, Elasticsearch grabs the values directly from its on-disk columnar store instead of referencing the _source for each document. The published field is returned with the epoch_millis format instead of the default format, thanks to the format parameter provided in the query.

stored fields: If you explicitly marked specific fields as stored in the mapping, you can use the stored_fields parameter to filter for those fields. This is useful if you want light responses with just those specific fields or for fields you deliberately stored for later retrieval. It’s stored separately from _source, so this method is also useful for avoiding the need to load _source.

It’s important to note that this option is off by default and generally not recommended. Use source filtering instead to return certain subsets of the original source document.

In the example query below, we use the stored_fields parameter to retrieve the summary field, which has the index mapping configuration of ”store”: true.

When this query runs, Elasticsearch looks to see if this field was marked with ”store”: true, if it doesn’t find it, it will skip the field entirely.

3. Displaying sub-fields

If your index contains sub-fields, you can use the dot notation to specify the field path in the fields parameter. Note that sub-fields are different from the nested field type. For example, if you have a sub-field named address.city, you can include it in the search response like this:

In this example, the search response will include the values of the title, author, and address.city fields.

4. synthetic _source

If you want to keep the functionality of using _source but also save disk space, you have the option of using synthetic _source in your index mapping. synthetic _source is a feature that allows Elasticsearch to reconstruct the _source from existing data like stored fields and doc values, even when _source is disabled. This allows you to save a lot of storage space at the cost of slightly lower speeds at query time as the reconstruction happens on-the-fly. Enable this feature by using the values below in your index settings:

some benefits of using synthetic _source include: full document display when using the _search API, source filtering and compatibility with other features and tools like Kibana that expect _source to be available, all while avoiding the need to store the full _source document.

5. Runtime fields

Runtime fields let you define scripted fields at query time or in your index mapping under a runtime block. These fields are never indexed, so adding a runtime field doesn’t increase index size but won’t ever show up in _source. Runtime fields defined in the mapping are persistent and available for all queries, while runtime fields defined at query time are temporary and only available in that search request.

The main benefit of using runtime fields is the ability to add fields to documents after you’ve already ingested them, simplifying your mapping decisions. Runtime fields are also great for enriching your documents with values that don’t exist on the original document but are generated using a script, such as formatting a string or calculating a score.

It is also worth noting that runtime fields can harm performance as a script will need to be run for every document in the result set. To retrieve a runtime field, you can also use the fields parameter on the _search API.

Conclusion

Displaying fields of an Elasticsearch index can range from simply retrieving values using the index mapping or the _source, to more advanced methods using fields, docvalue_fields, or runtime fields for greater control and efficiency. Understanding the tradeoffs between different methods is key to optimizing your search experiences. Whether you’re optimizing payloads, enriching documents or using synthetic _source to save storage, Elasticsearch gives you multiple tools and features to find the data you need, in the way you need it. These techniques can help you understand the structure of your data, identify specific fields, and troubleshoot issues.