OpenTelemetry demo with the Elastic Distributions of OpenTelemetry

Recently, Elastic introduced the Elastic Distributions (EDOT) for various OpenTelemetry components, we are proud to announce that these EDOT components are now available in the Elastic's fork of the OpenTelemetry demo. We've also made public a Kibana endpoint, allowing you to dive into the demo’s live data and explore its capabilities firsthand. In this blog post, we'll elaborate on the reasons behind the fork and explore the powerful new features it introduces. We'll also provide a comprehensive overview of how these enhancements can be leveraged with the Elastic Distributions of OpenTelemetry (EDOT) for advanced error detection, as well as the EDOT Collector—a cutting-edge evolution of the Elastic Agent—for seamless data collection and analysis.

What is the OpenTelemetry demo?

The OpenTelemetry demo is a microservices-based application created by OpenTelemetry's community to showcase its capabilities in a realistic, and distributed system environment. This demo application, known as the OpenTelemetry Astronomy Shop, simulates an e-commerce website composed of over 10 interconnected microservices (written in multiple languages: Go, Java, .NET, Node.js, etc.), communicating via HTTP and gRPC. Each service is fully instrumented with OpenTelemetry, generating comprehensive traces, metrics, and logs. The demo serves as an invaluable resource for understanding how to implement and use OpenTelemetry in real-world applications.

One of the microservices, called

, automatically starts generating requests to the various endpoints of the demo, simulating a real-world environment where multiple clients are interacting with the system. This helps replicate the behavior of a busy, live application with concurrent user activity.

Elastic's fork

Elastic recognized an opportunity to enhance the OpenTelemetry demo by forking it and integrating advanced Elastic features for deeper observability and simpler monitoring. While forking is the recommended OpenTelemetry approach, we aim to leverage the robust foundation and latest updates from the upstream version as much as possible. To achieve this, Elastic’s fork of the OpenTelemetry demo performs daily pulls from upstream, seamlessly integrating them with Elastic-specific changes. To avoid conflicts, we continuously contribute upstream, ensuring Elastic's modifications are always additive or configurable through environment variables. One such contribution is the .env.override file, designed exclusively for vendor forks to override the microservices images and configuration files used in the demo.

Deeper Insights with Elastic Distributions

In our current update of Elastic's OpenTelemetry demo fork, we have replaced some of the microservices OTel SDKs used for instrumentation with Elastic's specialized distributions. These changes ensure deeper integration with Elastic's observability tools, offering richer insights and more robust monitoring capabilities. These are some of the fork's changes:

Java services: The Ad, Fraud Detection, and Kafka services now utilize the Elastic distribution of the OpenTelemetry Java Agent. One of the included features in the distribution are stack traces, which provides precise information of where in the code path a span was originated. Learn more about the Elastic Java Agent here.

The Cart service has been upgraded to use the Elastic distribution of the OpenTelemetry .NET Agent. This replacement gives visibility on how the Elastic Distribution of OpenTelemetry .NET (EDOT .NET) can be used to get started using OpenTelemetry in your .NET applications with zero code changes. Discover more about the Elastic .NET Agent in this blog post.

In the Payment service, we've configured the Elastic distribution of the OpenTelemetry Node.js Agent. The distribution ships with the host-metrics extension, and Kibana provides a curated service metrics UI. Read more about the Elastic Node.js Agent here.

The Recommendation service now leverages the EDOT Python, replacing the standard OpenTelemetry Python agent. The Python distribution is another example of a Zero-code (or Automatic) instrumentation, meaning that the distribution will set up the OpenTelemetry SDK and enable all the recommended instrumentations for you. Find out more about the Elastic Python Agent in this blog post.

It's important to highlight that Elastic Distributions of OpenTelemetry don't bundle proprietary software, they have been build on top of the vanilla OTel SDKs but they offer some advantages, such as single package for installation, easy auto-instrumentation with reasonable default configuration, automatic logs telemetry sending, and many more. Along these lines, the ultimate goal is to contribute as many features from EDOT's back to the upstream OpenTelemetry agents; they are designed in such a way that the additional features, realized as extensions, work directly with the OTel SDKs.

Collecting Data with the Elastic Collector Distribution

The OpenTelemetry demo applications generate and send their signals to an OpenTelemetry Collector OTLP endpoint. In the demo's fork, the EDOT collector is set up to forward all OTLP signals from the microservices to an APM server OTLP endpoint. Additionally, it sends all other metrics and logs collected by the collector to an Elasticsearch endpoint.

If the fork is deployed in a Kubernetes environment, the collector will automatically start collecting the system's metrics. The collector will be configured to use the hostmetrics receivers to monitor all the K8s node's metrics, the kuebeletstats receiver to retrieve Kubelet's metrics and the filelog receiver, that will collect all cluster's.

Both the signals generated by the microservices and those collected by the EDOT collector are enriched with Kubernetes metadata, allowing users to correlate them seamlessly. This makes it easy to track and observe which Kubernetes nodes and pods each service is running on, providing deep insights into both application performance and infrastructure health.

Learn more about the Elastic's OpenTelemetry Collector distribution: https://www.elastic.co/observability-labs/blog/elastic-distribution-opentelemetry-collector

Error detection with Elastic

The OpenTelemetry demo incorporates flagd, a feature flag evaluation engine used to simulate error scenarios. For example, the

flag will force an error for every request to the payment service endpoint. Since the service is instrumented with OpenTelemetry, the error will be captured in the generated traces. We can then use Kibana's powerful visualization and search tools to trace the error back to its root cause.

Another available flag is named

, which causes a high CPU load in the ad service. This increased CPU usage can be monitored either through the service's metrics or the related metrics of its Kubernetes pod:

The full list of simulated scenarios can be found at this link.

Start your own exploration

Ready to explore the OpenTelemetry demo with Elastic and its enhanced observability capabilities? Follow the link to Kibana and begin your own exploration of how Elastic and OpenTelemetry can transform your approach to observability.

Live demo: https://ela.st/demo-otel

But that's not all—if you want to take it a step further, you can deploy the OpenTelemetry demo directly with your own Elasticsearch stack. Follow the steps provided here to set it up and start gaining valuable insights from your own environment.