WhatR17;s new in 8.15

Here are the highlights of whatR17;s new and improved in Elasticsearch 8.15! For detailed information about this release, see the Release notes and Migration guide.

Other versions:

8.14 | 8.13 | 8.12 | 8.11 | 8.10 | 8.9 | 8.8 | 8.7 | 8.6 | 8.5 | 8.4 | 8.3 | 8.2 | 8.1 | 8.0

If a multi-repo get-snapshots request encounters a failure in one of the targeted repositories then earlier versions of Elasticsearch would proceed as if the faulty repository did not exist, except for a per-repository failure report in a separate section of the response body. This makes it impossible to paginate the results properly in the presence of failures. In versions 8.15.0 and later this APIR17;s failure handling behaviour has been made stricter, reporting an overall failure if any targeted repositoryR17;s contents cannot be listed.

#107191

This change introduces a new index mode named logsdb. When the new index mode is enabled then the following storage savings features are enabled automatically:

The index.mode index setting set to logsdb should be configured in index templates or defined when creating a plain index. Benchmarks and other tests have shown that logs data sets use around 2.5 times less storage with the new index mode enabled compared to not configuring it. The new logsdb index mode is a tech preview feature.

#108896

New int4 (half-byte) scalar quantization support via two knew index types: int4_hnsw and int4_flat. This gives an 8x reduction from float32 with some accuracy loss. In addition to less memory required, this improves query and merge speed significantly when compared to raw vectors.

#109317

This PR marks query rules as Generally Available. All APIs are no longer in tech preview.

#110004

This adds bit vector support by adding element_type: bit for vectors. This new element type works for indexed and non-indexed vectors. Additionally, it works with hnsw and flat index types. No quantization based codec works with this element type, this is consistent with byte vectors.

bit vectors accept up to 32768 dimensions in size and expect vectors that are being indexed to be encoded either as a hexidecimal string or a byte[] array where each element of the byte array represents 8 bits of the vector.

bit vectors support script usage and regular query usage. When indexed, all comparisons done are xor and popcount summations (aka, hamming distance), and the scores are transformed and normalized given the vector dimensions.

For scripts, l1norm is the same as hamming distance and l2norm is sqrt(l1norm). dotProduct and cosineSimilarity are not supported.

Note, the dimensions expected by this element_type are always to be divisible by 8, and the byte[] vectors provided for index must be have size dim/8 size, where each byte element represents 8 bits of the vectors.

#110059

The Redact processor uses the Grok rules engine to obscure text in the input document matching the given Grok patterns. The Redact processor was initially released as Technical Preview in 8.7.0, and is now released as Generally Available.

#110395

This introduces a new custom parser for ISO-8601 datetimes, for the iso8601, strict_date_optional_time, and strict_date_optional_time_nanos built-in date formats. This provides a performance improvement over the default Java date-time parsing. Whilst it maintains much of the same behaviour, the new parser does not accept nonsensical date-time strings that have multiple fractional seconds fields or multiple timezone specifiers. If the new parser fails to parse a string, it will then use the previous parser to parse it. If a large proportion of the input data consists of these invalid strings, this may cause a small performance degradation. If you wish to force the use of the old parsers regardless, set the JVM property es.datetime.java_time_parsers=true on all ES nodes.

#106486

Following on from #106486, this extends the custom ISO-8601 datetime parser to cover the strict_year, strict_year_month, strict_date_time, strict_date_time_no_millis, strict_date_hour_minute_second, strict_date_hour_minute_second_millis, and strict_date_hour_minute_second_fraction date formats. As before, the parser will use the existing java.time parser if there are parsing issues, and the es.datetime.java_time_parsers=true JVM property will force the use of the old parsers regardless.

#108606

As a Technical Preview, the geoip processor can now use the commercial GeoIP2 Connection Type, GeoIP2 Domain, and GeoIP2 ISP databases from MaxMind.

#108683

Elasticsearch is now updated using the latest Lucene version 9.11. Here are the full release notes: But, here are some particular highlights: - Usage of MADVISE for better memory management: https://github.com/apache/lucene/pull/13196 - Use RWLock to access LRUQueryCache to reduce contention: https://github.com/apache/lucene/pull/13306 - Speedup multi-segment HNSW graph search for nested kNN queries: https://github.com/apache/lucene/pull/13121 - Add a MemorySegment Vector scorer - for scoring without copying on-heap vectors: https://github.com/apache/lucene/pull/13339

#109219

There are multiple improvements to synthetic _source functionality:

#109501

Index sorting is now supported for indexes with mappings containing nested objects. The index sort spec (as specified by index.sort.field) canR17;t contain any nested fields, still.

#110251