How to increase primary shard count in Elasticsearch

It is not possible to increase the primary shard number of an existing index, meaning an index must be recreated if you want to increase the primary shard count. There are 2 methods that are generally used in these situations: the _reindex API and the _split API.

The _split API is often a faster method than the _reindex API. Indexing must be stopped before both operations, otherwise, the source_index and target_index document counts will differ.

Method 1 – using the split API

The split API is used to create a new index with the desired number of primary shards by copying the settings and mapping an existing index. The desired number of primary shards can be set during creation. The following settings should be checked before implementing the split API:

The source index must be read-only. This means that the indexing process needs to be stopped.
The number of primary shards in the target index must be a multiple of the number of primary shards in the source index. For example, if the source index has 5 primary shards, the target index primary shards can be set to 10,15,20, and so on.

Note: If only the primary shard number needs to be changed, the split API is preferred as it is much faster than the Reindex API.

Implementing the split API

Create a test index:

The source index must be read-only in order to be split:

Settings and mappings will be copied automatically from the source index:

You can check the progress with:

Since settings and mappings are copied from the source indices, the target index is read-only. Let’s enable the write operation for the target index:

Check the source and target index docs.count before deleting the original index:

Index name and alias name can’t be the same. You need to delete the source index and add the source index name as an alias to the target index:

After adding the test_split_source alias to the test_split_target index, you should test it with:

Method 2 – using the reindex API

By creating a new index with the Reindex API, any number of primary shard counts can be given. After creating a new index with the intended number of primary shards, all data in the source index can be re-indexed to this new index.

In addition to the split API features, the data can be manipulated using the ingest_pipeline in the reindex AP. With the ingest pipeline, only the specified fields that fit the filter will be indexed into the target index using the query. The data content can be changed using a painless script, and multiple indices can be merged into a single index.

Implementing the reindex API

Create a test reindex:

Copy the settings and mappings from the source index:

Create a target index with settings, mappings, and the desired shard count:

*Note: setting number_of_replicas: 0 and refresh_interval: -1 will increase reindexing speed.

Start the reindex process. Setting requests_per_second=-1 and slices=auto will tune the reindex speed.

You will see the task_id when you run the reindex API. Copy that and check with _tasks API:

Update the settings after reindexing has finished:

Check the source and target index docs.count before deleting the original index, it should be the same:

The index name and alias name can’t be the same. Delete the source index and add the source index name as an alias to the target index:

After adding the test_split_source alias to the test_split_target index, test it using:

Summary

If you want to increase the primary shard count of an existing index, you need to recreate the settings and mappings to a new index. There are 2 primary methods for doing so: the reindex API and the split API. Active indexing must be stopped before using either method.

Want to get Elastic certified? Find out when the next Elasticsearch Engineer training is running!

Elasticsearch is packed with new features to help you build the best search solutions for your use case. Dive into our sample notebooks to learn more, start a free cloud trial, or try Elastic on your local machine now.

Report an issue