Get ready for production

Many teams rely on elasticsearch to run their key services. To keep these services running, you can design your elasticsearch deployment to keep elasticsearch available, even in case of large-scale outages. To keep it running fast, you also can design your deployment to be responsive to production workloads.

elasticsearch is built to be always available and to scale with your needs. It does this using a distributed architecture. By distributing your cluster, you can keep Elastic online and responsive to requests.

In case of failure, elasticsearch offers tools for cross-cluster replication and cluster snapshots that can help you fall back or recover quickly. You can also use cross-cluster replication to serve requests based on the geographic location of your users and your resources.

elasticsearch also offers security and monitoring tools to help you keep your cluster highly available.

You can add servers (nodes) to a cluster to increase capacity, and elasticsearch automatically distributes your data and query load across all of the available nodes.

Elastic is able to distribute your data across nodes by subdividing an index into shards. Each index in elasticsearch is a grouping of one or more physical shards, where each shard is a self-contained Lucene index containing a subset of the documents in the index. By distributing the documents in an index across multiple shards, and distributing those shards across multiple nodes, elasticsearch increases indexing and query capacity.

There are two types of shards: primaries and replicas. Each document in an index belongs to one primary shard. A replica shard is a copy of a primary shard. Replicas maintain redundant copies of your data across the nodes in your cluster. This protects against hardware failure and increases capacity to serve read requests like searching or retrieving a document.

Shard copies in your cluster are automatically balanced across nodes to provide scale and high availability. All nodes are aware of all the other nodes in the cluster and can forward client requests to the appropriate node. This allows elasticsearch to distribute indexing and query load across the cluster.

If you’re exploring elasticsearch for the first time or working in a development environment, then you can use a cluster with a single node and create indices with only one shard. However, in a production environment, you should build a cluster with multiple nodes and indices with multiple shards to increase performance and resilience.

To learn about optimizing the number and size of shards in your cluster, refer to Size your shards. To learn about how read and write operations are replicated across shards and shard copies, refer to Reading and writing documents. To adjust how shards are allocated and balanced across nodes, refer to Shard allocation, relocation, and recovery.

To effectively distribute read and write operations across nodes, the nodes in a cluster need good, reliable connections to each other. To provide better connections, you typically co-locate the nodes in the same data center or nearby data centers.

Co-locating nodes in a single location exposes you to the risk of a single outage taking your entire cluster offline. To maintain high availability, you can prepare a second cluster that can take over in case of disaster by implementing cross-cluster replication (CCR).

CCR provides a way to automatically synchronize indices from your primary cluster to a secondary remote cluster that can serve as a hot backup. If the primary cluster fails, the secondary cluster can take over.

You can also use CCR to create secondary clusters to serve read requests in geo-proximity to your users.

Learn more about cross-cluster replication and about designing for resilience.

As with any enterprise system, you need tools to secure, manage, and monitor your elasticsearch clusters. Security, monitoring, and administrative features that are integrated into elasticsearch enable you to use Kibana as a control center for managing a cluster.

Learn about securing an elasticsearch cluster.

Learn about monitoring your cluster.

elasticsearch offers many options that allow you to configure your cluster to meet your organization’s goals, requirements, and restrictions. You can review the following guides to learn how to tune your cluster to meet your needs:

Many elasticsearch options come with different performance considerations and trade-offs. The best way to determine the optimal configuration for your use case is through testing with your own data and queries.