The Challenge
How do you deliver high-performance search across a continuously growing database without overloading operational resources?
The Solution
By using Elastic Cloud — the hosted and managed elasticsearch service by Elastic — to deliver a scalable, seamless, and highly available search and discovery experience to the growing Docker community.
Case Study Highlights
- Easily and cost-effectively scale a search application to meet growing volumes of data
- Ensure excellent search and discovery experience
- Manage operational complexities
- Use elasticsearch to serve better search results faster
- Choose a hosting service that ensures a stable and persistent search experience
- Reduce operational overhead by delegating elasticsearch infrastructure to Elastic Cloud
Docker, Inc. is the company behind the Docker open source platform and is the chief sponsor of the Docker ecosystem. Used by tens of thousands of organizations like Baidu, BBC, eBay, Goldman Sachs, ING, Spotify, and Yelp, Docker is an open platform for distributed applications that allows developers and IT administrators to build, ship and run applications anywhere.
The Docker community churns out containers for everything from load balancers to databases to time-keeping services at a rapid pace. These building blocks can be easily deployed and arranged, with substantial security, performance, and organizational benefits.
elasticsearch gives us more flexibility over the queries that we need to perform, letting us specify where we can boost fields, sort and score documents.
The Problem: Outgrowing an RDBMS
Docker’s container library needed to be discoverable, and given the sheer quantity of them it was decided that search was the best user experience for this feature. An initial search implementation on a traditional RDBMS was not meeting the performance and quality requirements Docker required. As Docker developer Ken Cochrane noted, "We were using the database before, it was pretty limiting. It was adding load to our database, and was getting slower as we added more and more data."
After the decision to move away from their RDBMS was made, Docker evaluated both elasticsearch and other search technologies. They decided on elasticsearch as their next-generation platform.
Having made the decision to move to elasticsearch, Docker next considered the operational work required. Managing a distributed database like elasticsearch presents some tricky operational challenges, such as maintaining a viable quorum, ensuring that replication is configured correctly for high availability and archiving/managing backups. Given these and other complexities, the decision was made to delegate their elasticsearch infrastructure to a fully hosted provider.
Elastic Cloud's emphasis on service and fast response times helped ensure that Docker's rollout was a success. Support is very responsive…for me it's perfect!
The Solution: Isolating Key Concerns
Docker evaluated the available options for hosted elasticsearch by looking at a variety of different criteria. According to Docker developer John Costa: "We evaluated a few other hosted elasticsearch vendors. During the analysis we considered the location, number of indexes (development, staging, production), available resources (memory), high availability options and price. Elastic Cloud was the best option."
Consistent performance and reliability were key concerns for Docker, making Elastic Cloud's dedicated elasticsearch clusters a good fit for two key reasons. First, Elastic Cloud's hosting model based on dedicated clusters with reserved memory and CPU, gave them assurances that their application would be consistently performant. Second, Elastic Cloud's high availability options gave Docker added assurances that even in the event of a full data center outage their search database would remain available.
Additionally, Docker's index allocation requirements were a good fit for Elastic Cloud's architecture. Since each Elastic Cloud customer is allocated a dedicated elasticsearch cluster with no restrictions on its use, they were free to allocate indexes as they saw fit.
After evaluating a few providers, we decided that Elastic Cloud matched our needs based on fault tolerance, the prompt support and ease of operations.
Elastic Cloud's emphasis on service and fast response times helped ensure that Docker's rollout was a success. Support is very responsive…for me it's perfect!
The Benefits: Moving to Production
Moving to elasticsearch in production affected the performance gains Docker was looking for. Load dropped on their production RDBMS significantly, and search latency and throughput were massively improved. Additionally, Docker was able to greatly improve search result quality by using elasticsearch’s field boosting and function score queries to promote more popular and relevant search results.
The legwork of deploying the Docker Index on Elastic Cloud was quick and simple. As John Costa of Docker noted: "Moving to production was easy as it only required us to add a credit card for billing and then grab the production settings. We had a few questions about indexing, memory settings and reliability settings. The Elastic Cloud support staff was prompt and detailed in their responses."
With their new infrastructure Docker is able to serve better search results faster. This means greater ease of use for developers using the Docker platform. For Docker this is critical; a tool built around providing power and convenience must also have supporting services which possess those characteristics. Ultimately, elasticsearch has proved a powerful tool for the company, helping them deliver on an important part of their core mission: delivering great developer experiences.
