Introducing a more powerful, resilient, and observable ES|QL in Elasticsearch 8.19 & 9.1

Elasticsearch is packed with new features to help you build the best search solutions for your use case. Dive into our sample notebooks to learn more, start a free cloud trial, or try Elastic on your local machine now.

We are excited to announce the general availability of Elasticsearch 8.19 and 9.1. This release represents a significant leap forward for the Elasticsearch Query Language (ES|QL), already in use on over 10,000 clusters each week after going GA in 8.14 last year. We've delivered major enhancements across four key themes: the production-readiness of data enrichment with lookup joins, built-in resilience to failures, new capabilities for monitoring and observability, and powerful new features for operating ES|QL at scale, with ES|QL CCS going GA.

Let's dive into the details.

Production-ready data enrichment: LOOKUP JOIN is now GA

The most anticipated feature in this release is that the LOOKUP JOIN is now Generally Available (GA). LOOKUP JOIN fundamentally simplifies data correlation, removing the need for data denormalization or complex, inefficient client-side joins. If you're new to this concept, our original blog post on lookup joins provides excellent background.

A key advantage of this approach is its immediacy. Unlike the ingest-time enrich processor, which requires managing policies and waiting for them to execute, a lookup index can be updated directly. Then, the next LOOKUP JOIN query you execute will reflect those changes, making it ideal for dynamic enrichment scenarios.

The LOOKUP JOIN command allows you to enrich streaming data from a source index with contextual information from a lookup index, all within a single query.

Example: Enrich security logs with employee directory information.

For its GA release, we’ve made LOOKUP JOIN even more powerful and flexible to handle the complexities of real-world data:

Joins on mixed numeric types: Your data isn't always perfect. Now, you can seamlessly join on numeric fields of different but compatible types—for instance, joining a long field with an integer field. ES|QL LOOKUP JOIN handles the conversion automatically.
Index alias support: You can now use an index alias as the target for your lookup, as long as the alias points to a valid lookup index. This allows for cleaner queries and more flexible data management.
High-precision date_nanos joins: For use cases like finance or high-frequency observability, lookup joins now fully support the date_nanos field type, ensuring no loss of accuracy.

Under the hood, new optimizations make joins faster by purging unnecessary operations and reordering query execution. On data nodes, only fields required to perform the join will be loaded from indices; if they are missing entirely, the join will be replaced by a fast no-op that places nulls in the lookup columns. To achieve this, ES|QL’s query optimizer performs multiple passes of query rewriting. As part of the optimizations, we postpone the execution of KEEP/DROP/RENAME commands until after the join, because they trigger loading all fields that participate in the query from the indices. Another optimizer pass is performed on each data node, where statistics about the index shards are used to determine whether the join key field is missing, in which case the LOOKUP JOIN is replaced by a fast EVAL that just creates null values for the lookup fields.

For a complete guide, explore the LOOKUP JOIN documentation. Stay tuned—we have more enhancements and other join types on our roadmap!

A more resilient query engine, by default

Distributed systems must be resilient. With this release, ES|QL is now significantly more robust.

We have introduced a new allow_partial_results setting, which is enabled by default. This means that if a shard is temporarily unavailable, your ES|QL query will now succeed with results from all available shards rather than failing completely.

Also, the skip_unavailable setting (enabled by default) now covers most runtime errors on remote clusters, allowing the queries in which one of the remote clusters failed to deliver the result from the rest of the clusters. You can check the is_partial flag returned in the response to determine if the result is partial or complete.

Furthermore, ES|QL now automatically retries on shard-level failures. This is invaluable during operational procedures like rolling restarts or in the case of temporary node unavailability, as queries have a much higher chance of successful completion without requiring any manual intervention.

Significant performance and scalability improvements

In versions 8.19 and 9.1, ES|QL is smarter, faster, and more scalable. With over 30 individual enhancements merged, the query engine is more intelligent about how it plans execution and allocates cluster resources.

Aggressive pushdowns to Lucene: Big performance gains come from pushing more operations down to the Lucene storage engine. Filters using == on text, LIKE, STARTS_WITH, and ENDS_WITH are now executed by Lucene at the shard level, plus case-insensitive regexes in 9.1. This drastically reduces the amount of data that needs to be moved to the coordinating node for processing, leading to major speed improvements (86x improvement for == and ROUND_TO is over 170x faster than CASE).

Smarter query planning and execution: ES|QL now uses more advanced heuristics to choose the most efficient partitioning scheme for query execution. It also prioritizes querying hotter data tiers first to return results faster.

ES|QL query execution is split between a Coordinating node (in order to parse the query, analyze it, prepare the query plan, and finally merge/aggregate results) and Data nodes for the actual execution (query the corresponding Lucene data structures for the necessary data). Starting in 8.19 and 9.1, a query is routed differently to the data nodes. For non-aggregation and non-sorting queries, we target hot data nodes first.

Additionally, we execute only a few sub-requests at once. This allows us to compute responses for simple queries like `FROM index WHERE @timestamp …` or `FROM index* WHERE @timestamp …` faster, creating less load on the cluster overall, as well as avoiding cold and frozen nodes, as they might be much more expensive to query in case of empty caches. More complex queries (such as `FROM index | STATS ..` or `FROM index | SORT ...` that require us to inspect all data to compute correct query results) are routed to all nodes containing relevant data immediately to ensure better parallelism and faster query response time.

Optimized memory and CPU usage: We’ve optimized the internals of ES|QL of many common operations to significantly improve performance (with some improvements even backported to 9.0).

VALUES aggregation for ordinal inputs is about 2.4× faster, building on earlier improvements. String conversion functions like TO_LOWER and TO_UPPER were optimized by keeping data in ordinal form. In cases with repeated values, this reduces overhead. Benchmarks show an effective drop from ~30–35 ns/op to ~0.025–0.027 ns/op, though this reflects amortized cost per value due to ordinal reuse, not literal per-value execution time.
We also sped up loading stored fields, which greatly cuts query latency when retrieving large result sets. A query returning ~10% of documents in a one-million-document index now completes in ~55 ms (down from 397 ms), and loading ~99% of docs is nearly 10× faster.
Finally, we’ve improved memory and CPU efficiency: the TO_IP function executes ~2× faster, more efficient node-to-node serialization, and updated behavior for REPLACE, which now enforces memory limits at fold-time to prevent clusters reaching OOM before query execution begins.

ES|QL at scale: Cross-Cluster Search (CCS) is now GA

For our enterprise users, the General Availability of Cross-Cluster Search (CCS) for ES|QL is a critical milestone. This powerful feature allows you to run a single ES|QL query that seamlessly targets petabytes of data across multiple, independent Elasticsearch clusters.

This is essential for organizations that manage large, distributed environments. You can now query data across geographically separate data centers, break down data silos between clusters used for different workloads (e.g., security vs. observability), or federate queries across your entire infrastructure from a single pane of glass, with ES|QL.

Enhanced observability: See what your queries are doing

Understanding query behavior is key to managing a healthy, performant cluster. This release introduces powerful new observability features for ES|QL.

ES|QL query logs: You can now enable logging of all ES|QL queries to a dedicated file. Ingesting this log file back into Elasticsearch provides a historical record, allowing you to identify slow or expensive queries and analyze usage trends over time.
Real-time monitoring & profiling (tech preview): A new API to List Queries provides a simple, direct way to see all ES|QL queries currently running on your cluster. We have also enhanced the query profile output with more granular detail to help you and our support teams debug the performance of any specific query.

Here is a basic example. (Please note, List Queries API is still in tech preview and could change.)

New functions and foundational capabilities

This release also broadens the power of ES|QL with new capabilities and functions:

LLM completion support: 9.1 adds the tech preview of the COMPLETION command to access LLM completion within your query, unlocking powerful new use cases—stay tuned for an upcoming blog on completions.
Full-text functions GA, now with MATCH_PHRASE: The powerful MATCH, QSTR, and KQL functions are now GA. We’ve also added the highly-requested MATCH_PHRASE function for performing phrase queries.
FORK: A new command in technical preview for 9.1 so you can run multiple queries, differentiate, mutate, and merge them together.
New commands and aggregations: Explore data on massive datasets with the new SAMPLE command for random sampling. A new ML Change Point Aggregation also helps you automatically detect anomalous changes in your time-series data.
New analytical functions: We've added a host of new functions, including ROUND_TO for flexible rounding, an enhanced LIKE that works with lists of patterns, and new mathematical functions like COPY_SIGN and SCALB.

Get started today

There is too much to cover in a single blog post, so we encourage you to consult the Elasticsearch 8.19 and 9.1 release notes for a complete list of all the changes.