Monitor your c++ Applications with Elastic APM
Introduction
One of the main challenges that developers, SREs, and DevOps professionals face is the absence of an extensive tool that provides them with visibility to their application stack. Many of the APM solutions out on the market do provide methods to monitor applications that were built on languages and frameworks (i.e., .NET, Java, Python, etc.) but fall short when it comes to c++ applications.
Luckily, Elastic has been one of the leading solutions in observability space and a contributor to the OpenTelemetry project. Elastic’s unique position and its extensive observability capabilities allows end-users to monitor applications built with object-oriented programming languages & Framework in a variety of ways.
In this blog we will explore using Elastic APM to investigate c++ traces with the OpenTelemetry client. We will be providing a comprehensive guide on how to implement the OpenTelemetry client for c++ applications and connecting to Elastic APM solutions. While OTel has its libraries, and this blog reviews how to use the OTel cPP library, Elastic also has its own Elastic Distributions of OpenTelemetry, which were developed to provide commercial support, and are completely upstreamed regularly.
Here are some resources to help get you started:
Step by Step Guide
Prerequisites
-
Environment
choosing an environment is quite important as there is limited support for the OTEL client. We have experimented with using multiple Operating Systems and here are the suggestions:
-
Ubuntu 22.04
-
Debian 11 Bullseye
-
For this guide we are focusing on Ubuntu 22.04.
-
Machine: 2 vcPU, 4GB is sufficient.
-
Image: Ubuntu 22.04 LTS (x86_64).
-
Disk: ~30 GB is enough.
-
Implementation method
We have experimented with multiple methods but we found that the most suitable approach is to use a package manager. After extensive testing, It appears that trying to run otel-cpp client could be quite challenging to the users. If practitioners desire to build with tools such as cMake and Bazel that is a viable solution. With that, as we tested both methods it became obvious that we were spending most of our time and effort fixing compatibility and dependencies’ issues for the OS Vs. Focusing on sending data to our APM. Hence we decided to move to a different method.
The main issues that we kept running into as we test are:
-
compatibility of packages.
-
Availability of packages.
-
Dependencies of libraries and packages.
In this guide we will use vcpkg since it allows us to bring in all the dependencies required to run the Opentelemetry c++ client.
Installing required OS tools
Update package lists
sudo apt-get update
Install build essentials, cmake, git, and sqlite dev library
sudo apt-get install -y build-essential cmake git curl zip unzip sqlite3 libsqlite3-dev
sqlite3 and libsqlite3-dev allow us to build/run SQLite queries in our c++ code.
Set Up vcpkg
vcpkg is the c++ package manager that we’ll use to install opentelemetry-cpp client.
# clone vcpkg
cd ~
git clone https://github.com/microsoft/vcpkg.git
# Bootstrap
cd ~/vcpkg
./bootstrap-vcpkg.sh
Install OpenTelemetry c++ with OTLP gRPc
In this guide we focus on trace export to Elastic. At time of writing, vcpkg’s opentelemetry-cpp
version 1.18.0 fully supports traces but has limited direct metrics exporting.
Install the package
cd ~/vcpkg
./vcpkg install opentelemetry-cpp[otlp-grpc]:x64-linux
Note
Sometimes when installing opentelemetry-cpp on linux it doesn't install all the required packages. As a workaround if you run into that case, try running again but pass a flag to allow-unsupported:
./vcpkg install opentelemetry-cpp[*]:x64-linux --allow-unsupported
Verify
./vcpkg list | grep opentelemetry-cpp
The output thould be something like this:
opentelemetry-cpp:x64-linux 1.18.0
create the c++ Project with Database Spans
We’ll build a sample in ~/otel-app that:
-
Uses SQLite to do basic cREATE/INSERT/SELEcT queries. This is helpful to showcase capturing transactions for apps that use databases on Elastic APM.
-
Generate random traces to showcase how they are captured on Elastic APM.
This app is going to generate random queries where some will contain database transactions and some are just application traces. Each query is contained in a child span, so they appear in APM as separate database transactions.
Below is the structure of our project
otel-app/
├── main.cpp
└── cMakeLists.txt
create App Project
cd ~
mkdir otel-app
cd otel-app
Inside this project we will create two files
-
main.cpp
-
cMakeLists.txt
Keep in mind that main.cpp is where you are going to pass the otel exporters that are going to send data to the Elastic cluster. So for your tech stack it would be your application's source code.
Sample application code
main.cpp
// Below we declare required libraries that we will be using to ship
// traces to Elastic APM
#include <opentelemetry/exporters/otlp/otlp_grpc_exporter.h>
#include <opentelemetry/sdk/trace/tracer_provider.h>
#include <opentelemetry/sdk/trace/simple_processor.h>
#include <opentelemetry/trace/provider.h>
#include <sqlite3.h>
#include <chrono>
#include <iostream>
#include <thread>
#include <cstdlib> // for rand(), srand()
#include <ctime> // for time()
// Namespace aliases
namespace trace_api = opentelemetry::trace;
namespace sdktrace = opentelemetry::sdk::trace;
namespace otlp = opentelemetry::exporter::otlp;
// Below we are using a helper function to run SQLITE statement inside
// child span
bool ExecuteSql(sqlite3 *db, const std::string &sql,
trace_api::Tracer &tracer,
const std::string &span_name)
{
// Starting the child span
auto db_span = tracer.StartSpan(span_name);
{
auto scope = tracer.WithActiveSpan(db_span);
// Here we mark Database attributes for clarity in APM
db_span->SetAttribute("db.system", "sqlite");
db_span->SetAttribute("db.statement", sql);
char *errMsg = nullptr;
int rc = sqlite3_exec(db, sql.c_str(), nullptr, nullptr, &errMsg);
if (rc != SQLITE_OK)
{
db_span->AddEvent("SQLite error: " + std::string(errMsg ? errMsg : "unknown"));
sqlite3_free(errMsg);
db_span->End();
return false;
}
db_span->AddEvent("Query OK");
}
db_span->End();
return true;
}
/**
* DoNonDbWork - Simulate some other operation
*/
void DoNonDbWork(trace_api::Tracer &tracer, const std::string &span_name)
{
auto child_span = tracer.StartSpan(span_name);
{
auto scope = tracer.WithActiveSpan(child_span);
// Just sleep or do some "fake" work
std::cout << "[TRAcE] Doing non-DB work for " << span_name << "...\n";
std::this_thread::sleep_for(std::chrono::milliseconds(200 + rand() % 300));
child_span->AddEvent("Finished non-DB work");
}
child_span->End();
}
int main()
{
// Seed random generator for example
srand(static_cast<unsigned>(time(nullptr)));
// 1) create OTLP exporter for traces
otlp::OtlpGrpcExporterOptions opts;
auto exporter = std::make_unique<otlp::OtlpGrpcExporter>(opts);
// 2) Simple Span Processor
auto processor = std::make_unique<sdktrace::SimpleSpanProcessor>(std::move(exporter));
// 3) Tracer Provider
auto sdk_tracer_provider = std::make_shared<sdktrace::TracerProvider>(std::move(processor));
auto tracer = sdk_tracer_provider->GetTracer("my-cpp-multi-app");
// Prepare an in-memory SQLite DB (for random DB usage)
sqlite3 *db = nullptr;
int rc = sqlite3_open(":memory:", &db);
if (rc == SQLITE_OK)
{
// create a table so we can do inserts/reads
ExecuteSql(db, "cREATE TABLE IF NOT EXISTS items (id INTEGER PRIMARY KEY, info TEXT);",
*tracer.get(), "db_create_table");
}
// create the following loop to generate multiple transactions
int num_transactions = 5; // change this variable to the desired number of transaction
for (int i = 1; i <= num_transactions; i++)
{
// Each iteration is a top-level transaction
std::string transaction_name = "transaction_" + std::to_string(i);
auto parent_span = tracer->StartSpan(transaction_name);
{
auto scope = tracer->WithActiveSpan(parent_span);
std::cout << "\n=== Starting " << transaction_name << " ===\n";
// Randomly select whether a transaction will interact with the database or not.
bool doDb = (rand() % 2 == 0); // 50% chance
if (doDb && db)
{
// Insert random data
std::string insert_sql = "INSERT INTO items (info) VALUES ('Item " + std::to_string(i) + "');";
ExecuteSql(db, insert_sql, *tracer.get(), "db_insert_item");
// Select from DB
ExecuteSql(db, "SELEcT * FROM items;", *tracer.get(), "db_select_items");
}
else
{
// Do some random non-DB tasks
DoNonDbWork(*tracer.get(), "non_db_task_1");
DoNonDbWork(*tracer.get(), "non_db_task_2");
}
// Sleep a little to simulate transaction time
std::this_thread::sleep_for(std::chrono::milliseconds(200));
}
parent_span->End();
}
// close DB
sqlite3_close(db);
// Extra sleep to ensure final flush
std::cout << "\n[INFO] Sleeping 5 seconds to allow flush...\n";
std::this_thread::sleep_for(std::chrono::seconds(5));
std::cout << "[INFO] Exiting.\n";
return 0;
}
What does the code do?
We create 5 top-level “transaction_i” spans.
For each transaction, we randomly choose to do DB or non-DB work
- If DB: Insert a row, then select. Each is a child span.
- If non-DB: We do two “fake tasks” (child spans).
Once we finish, we close the database connection and wait 5 seconds for data flush.
Sample instruction file
cMakeLists.txt : This file contains instructions describing the source files and targets.
cmake_minimum_required(VERSION 3.10)
project(OtelApp VERSION 1.0)
set(cMAKE_cXX_STANDARD 11)
set(cMAKE_cXX_STANDARD_REQUIRED ON)
# Here we are pointing to use the vcpkg toolchain
set(cMAKE_TOOLcHAIN_FILE "PATH-TO/vcpkg.cmake" cAcHE STRING "Vcpkg toolchain file")
find_package(opentelemetry-cpp cONFIG REQUIRED)
add_executable(otel_app main.cpp)
# Below we are linking the OTLP gRPc exporter, trace library, and sqlite3
target_link_libraries(otel_app PRIVATE
opentelemetry-cpp::otlp_grpc_exporter
opentelemetry-cpp::trace
sqlite3
)
Declare Environmental Variables
Here we are going to export our Elastic cloud endpoints as environmental variables
You can get that information by doing the following:
-
Login into your elastic cloud
-
Go into your deployment
-
On the Left hand side, click on the hamburger menu and scroll down to “Integrations”
-
Go on the search bar inside the integration and type “APM”
-
click on the APM integration
-
Scroll down and click on the OpenTelemetry Option on the far left side
- You should be able to see values similar to the screenshot below. Once you copy the values to export, click on launch APM.
As you copy the required values, go ahead and export them.
export OTEL_EXPORTER_OTLP_ENDPOINT="APM-ENDPOINT"
export OTEL_EXPORTER_OTLP_HEADERS="KEY"
export OTEL_RESOURcE_ATTRIBUTES="service.name=my-app,service.version=1.0.0,deployment.environment=dev"
Note that the elastic OTEL_EXPORTER_OTLP_HEADERS value usually starts with “Authorization=Bearer” make sure that you convert the upper case “A” in authorization to a lower case “a”. This is due to the fact that the otel header exporter expects a lower case “a” for authorization.
Build and Run
Once we create the two files we then move to building the application.
cd ~/otel-app mkdir -p build cd build
cmake -DcMAKE_TOOLcHAIN_FILE=/vcpkg/scripts/buildsystems/vcpkg.cmake /vcpkg/installed/x64-linux/share
-DcMAKE_PREFIX_PATH=
..
make
Once make is successful run the the application
./otel-app
You should be able to see the script execute with a similar console output
console outcome:
=== Starting transaction_1 ===
[TRAcE] Doing non-DB work for non_db_task_1...
[TRAcE] Doing non-DB work for non_db_task_2...
=== Starting transaction_2 ===
[TRAcE] Doing DB work for doDb_task_1...
[TRAcE] Doing DB work for doDb_task_2...
=== Starting transaction_3 ===
[TRAcE] Doing non-DB work for non_db_task_1...
[TRAcE] Doing non-DB work for non_db_task_2...
=== Starting transaction_4 ===
[TRAcE] Doing non-DB work for non_db_task_1...
[TRAcE] Doing non-DB work for non_db_task_2...
=== Starting transaction_5 ===
[TRAcE] Doing non-DB work for non_db_task_1...
[TRAcE] Doing non-DB work for non_db_task_2...
[INFO] Sleeping 5 seconds to allow flush...
[INFO] Exiting.
Once the script executes you should be able to observe those traces on Elastic APM similar to the screenshots below.
Observe in Elastic APM
Go to Elastic cloud, open your deployment, and navigate to Observability > APM.
Look for the app name in the service list (as defined by OTEL_RESOURcE_ATTRIBUTES).
Inside that service’s Traces tab, you’ll find multiple transactions like “transaction_1”,
“transaction_2”, etc.
Expanding each transaction shows child spans:
- Possibly db_insert_item and db_select_items if random DB path was taken.
- Otherwise, non_db_task_1 and non_db_task_2.
You can see how some transactions do DB calls, some do not, each with different spans.
This variety demonstrates how your real application might produce multiple different
“routes” or “operations.”
Service Map
If everything runs correctly, you should be able to view your services and see service maps for your application.
Services
My Elastic App
App Transactions
Dependencies
Logs
Navigate to your logs window/Discover to see the incoming application logs
Patterns
Log pattern analysis helps you to find patterns in unstructured log messages and makes it easier to examine your data.
Final Recap
Here is a quick summary of what we did:
-
Provisioned an Ubuntu 22.04 machine.
-
Installed build tools for SQLite, dev libs, and vcpkg.
-
Installed the client for opentelemetry-cpp via vcpkg.
-
created a minimal c++ project that executes app traces and captures database operations.
-
connected database sqlite3 in cMakeLists.txt.
-
Exported the Elastic OTLP endpoint & token as environment variables (with a lowercase authorization=Bearer key!).
-
Ran the application and observed DB interactions and app traces in Elastic APM.
-
Observed application logs and patterns on Elastic logs and Discover.
FAQ & common Issues
- Getting “could not find package configuration file provided by opentelemetry-cpp”?
Make sure you pass
-DcMAKE_TOOLcHAIN_FILE=... and -DcMAKE_PREFIX_PATH=...
to cmake, or embed them in cMakeLists.txt.
- crash: “validate_metadata: INTERNAL:Illegal header key”?
Use all-lowercase in
OTEL_EXPORTER_OTLP_HEADERS, e.g. authorization=Bearer \<token>.
- Missing otlp_grpc_metrics_exporter.h?
Your vcpkg version of opentelemetry-cpp (1.18.0) lacks a direct metrics exporter for OTLP. For metrics, either upgrade the library or consider an OpenTelemetry collector approach.
- No data in Elastic APM?
Double-check your endpoint URL, Bearer token, firewall rules, or service name in the APM