This is a cache of https://www.elastic.co/observability-labs/blog/elastic-opentelemetry-otel-operator. It is a snapshot of the page at 2024-11-23T00:45:52.112+0000.
Native OTel-based K8s & App Observability in 3 Steps with Elastic — Elastic Observability Labs
Bahubali Shetti

Native OTel- based K8s & App Observability in 3 Steps with Elastic

Elastic's Distributions of OpenTelemetry are now supported with the OTel Operator, providing auto instrumentation of applications with EDOT SDKs, and deployment and lifecycle management of the EDOT OTel Collector for Kubernetes Observability. Learn how to configure this in 3 easy steps

6 min read
KubernetesOpenTelemetry
Native OTel-based K8s & App Observability in 3 Steps with Elastic

Elastic recently released its Elastic Distributions of OpenTelemetry (EDOT) which have been developed to enhance the capabilities of standard OpenTelemetry distributions and improve existing OpenTelemetry support from Elastic. EDOT helps Elastic deliver its new Unified OpenTelemetry Experience. SRE’s are no longer burdened with a set of tedious steps instrumenting and ingesting OTel data into Observability. SREs get a simple and frictionless way to instrument the OTel collector, and applications, and ingest all the OTel data into Elastic. The components of this experience include: (detailed in the overview blog)

  • Elastic Distributions for OpenTelemetry (EDOT)

  • Elastic’s configuration for the OpenTelemetry Operator providing:

    • OTel Lifecycle management for the OTel collector and SDKs

    • Auto instrumentation of apps, which most developers will not instrument

  • Pre-packaged receivers, processors, exporters, and configuration for the OTel Kubernetes Collector

  • Out-of-the-box OTel-based K8S dashboards for metrics and logs

  • Discovered inventory views for services, hosts, and containers

  • Direct OTel ingest into Elasticsearch (by passing APM) - all your data (logs, metrics, and traces) is now stored in Elastic’s Search AI Lake

In this blog we will cover how to ingest OTel for K8S and your application in 3 easy steps:

  1. Copy the install commands from the UI

  2. Add the OpenTelemetry helm charts, Install the OpenTelemetry Operator with Elastic’s helm configuration & set your Elastic endpoint and authentication

  3. Annotate the app services you want to be auto-instrumented 

Then you can easily see K8S metrics, logs and application logs, metrics, and traces in Elastic Observability.

To follow this blog you will need to have:

  1. An account on cloud.elastic.co, with access to get the Elasticsearch endpoint and authentication (api key)

  2. A non-instrumented application with services based on Go, dotnet, Python, or Java. Auto-instrumentation through the OTel operator. In this example, we will be using the Elastiflix application. 

  3. A Kubernetes cluster, we used EKS in our setup

  4. Helm and Kubectl loaded

To find the authentication, you can find it in the integrations section of Elastic. More information is also available in the documentation.

K8S and Application Observability in Elastic:

Before we walk you through the steps, let's show you what is visible in Elastic.

Once the Operator starts the OTel Collector, you can see the following in Elastic:

Kubernetes metrics:

Using an out-of-the-box dashboard, you can see node metrics, overall cluster metrics, and status across pods, deployments, etc.

Discovered Inventory for Hosts, services, and containers:

This can be found at Observability->Inventory on the UI

Detailed metrics, logs, and processor info on hosts:

This can be found at Observability->Infrastructure->Hosts

K8S and application logs in Elastic’s New Discover (called Explorer)

This can be found on Observability->Discover

Application Service views (logs, metrics, and traces):

This can be found on Observability->Application

Then select the service and drill down into different aspects.

Above we are showing how traces are shown using Native OTel data.

Steps to install

Step 0. Follow the commands listed in the UI

Under Add data->Kubernetes->Kubernetes Monitoring with EDOT

You will find the following instructions, which we will follow here.

Step 1. Install the EDOT config for the OpenTelemetry Operator

Run the following commands. Please make sure that you have already authenticated in your K8s Cluster and this is where you will run the helm commands provided below.

# Install helm repo needed
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts --force-update
# Install needed secrets. Provide the Elasticsearch Endpoint URL and API key you have noted in previous steps
kubectl create ns opentelemetry-operator-system
kubectl create -n opentelemetry-operator-system secret generic elastic-secret-otel \
    --from-literal=elastic_endpoint='YOUR_ELASTICSEARCH_ENDPOINT' \
    --from-literal=elastic_api_key='YOUR_ELASTICSEARCH_API_KEY'
# Install the EDOT Operator
helm install opentelemetry-kube-stack open-telemetry/opentelemetry-kube-stack --namespace opentelemetry-operator-system --create-namespace --values https://raw.githubusercontent.com/elastic/opentelemetry/refs/heads/main/resources/kubernetes/operator/helm/values.yaml --version 0.3.0

The values.yaml file configuration can be found here.

Step 1b: Ensure OTel data is arriving in Elastic

The simplest way to check is to go to Menu > Dashboards > [OTEL][Metrics Kubernetes] Cluster Overview, and ensure you see the following dashboard being populated

Step 2: Annotate the application with auto-instrumentation

For this example, we’re only going to annotate one service, the favorite-java service in the Elastiflix application

Use the following commands to initiate auto-instrumentation:

#Annotate Java namespace
kubectl annotate namespace java instrumentation.opentelemetry.io/inject-java="opentelemetry-operator-system/elastic-instrumentation"
#Restart the java-app to get the new annotation
kubectl rollout restart deployment java-app -n java

You can also modify the yaml for your pod with the annotation

metadata:
 name: my-app
 annotations:
   instrumentation.opentelemetry.io/inject-python: "true"

These instructions are provided in the UI:

Check out the service data in Elastic APM

Once the OTel data is in Elastic, you can see:

  • Out-of-the-box dashboards for OTel-based Kubernetes metrics

  • Discovered resources such as services, hosts, and containers that are part of the Kubernetes clusters

  • Kubernetes metrics, host metrics, logs, processor info, anomaly detection, and universal profiling.

  • Log analytics in Elastic Discover

  • APM features that show app overview, transactions, dependencies, errors, and more:

Try it out

Elastic’s Distribution of OpenTelemetry (EDOT) transforms the observability experience by streamlining Kubernetes and application instrumentation. With EDOT, SREs and developers can bypass complex setups, instantly gain deep visibility into Kubernetes clusters, and capture critical metrics, logs, and traces—all within Elastic Observability. By following just a few simple steps, you’re empowered with a unified, efficient monitoring solution that brings your OpenTelemetry data directly into Elastic. With robust, out-of-the-box dashboards, automatic application instrumentation, and seamless integration, EDOT not only saves time but also enhances the accuracy and accessibility of observability across your infrastructure. Start leveraging EDOT today to unlock a frictionless observability experience and keep your systems running smoothly and insightfully.

Additional resources:

Share this article

XFacebookLinkedInReddit