Traces

A trace is a group of transactions and spans with a common root. Each trace tracks the entirety of a single request. It describes the individual operations and their causality that ensue from a single logical operation.

When a trace travels through multiple services, as is common in a microservice architecture, it is known as a distributed trace. A distributed trace comprises operations across multiple distributed components, crossing process, network, and security boundaries.

Distributed tracing enables you to analyze performance throughout your microservice architecture by tracing the entirety of a request̴1;R12;̴1;from the initial web request on your front-end service all the way to database queries made on your back-end services.

Tracking requests as they propagate through your services provides an end-to-end picture of where your application is spending time, where errors are occurring, and where bottlenecks are forming. Distributed tracing eliminates individual serviceR17;s data silos and reveals whatR17;s happening outside of service borders.

For supported technologies, distributed tracing works out-of-the-box, with no additional configuration required.

Distributed tracing works by injecting a custom traceparent HTTP header into outgoing requests. This header includes information, like trace-id, which is used to identify the current trace, and parent-id, which is used to identify the parent of the current span on incoming requests or the current span on an outgoing request.

When a service is working on a request, it checks for the existence of this HTTP header. If itR17;s missing, the service starts a new trace. If it exists, the service ensures the current action is added as a child of the existing trace, and continues to propagate the trace.

In this example, ElasticR17;s Ruby agent communicates with ElasticR17;s Java agent. Both support the traceparent header, and trace data is successfully propagated.

In this example, ElasticR17;s Ruby agent communicates with OpenTelemetryR17;s Java agent. Both support the traceparent header, and trace data is successfully propagated.

In this example, the trace meets a piece of middleware that doesnR17;t propagate the traceparent header. The distributed trace ends and any further communication will result in a new trace.

All Elastic agents now support the official W3C Trace Context specification and traceparent header. See the table below for the minimum required agent version:

The Applications UIR17;s timeline visualization provides a visual deep-dive into each of your applicationR17;s traces:

Elastic agents automatically propagate distributed tracing context for supported technologies. If your service communicates over a different, unsupported protocol, you can manually propagate distributed tracing context from a sending service to a receiving service with each agentR17;s API.

Sending services must add the traceparent header to outgoing requests.

transaction := apm.DefaultTracer().StartTransaction("GET /", "request") 
traceContext := transaction.TraceContext() 

// Send TraceContext to receiving service
traceparent := apmhttp.FormatTraceparentHeader(traceContext) 
tracestate := traceContext.State.String()

import OpenTelemetryApi
import OpenTelemetrySdk

struct BasicSetter: Setter { 
    func set(carrier: inout [String: String], key: String, value: String) {
        carrier[key] = value
    }
}

let span : Span = ... 
let setter = BasicSetter()
let propagator = W3CTraceContextPropagator()
var headers = [String:String]()

propagator.inject(spanContext: span.context, carrier: &headers, setter:setter) 

let request = URLRequest(...)
request.allHTTPHeaderFields = headers
... // make network request
span.end()

// Hook into a callback provided by the RPC framework that is called on outgoing requests
public Response onOutgoingRequest(Request request) throws Exception {
  Span span = ElasticApm.currentSpan() 
          .startSpan("external", "http", null)
          .setName(request.getMethod() + " " + request.getHost());
  try (final Scope scope = transaction.activate()) {
      span.injectTraceHeaders((name, value) -> request.addHeader(name, value)); 
      return request.execute();
  } catch (Exception e) {
      span.captureException(e);
      throw e;
  } finally {
      span.end(); 
  }
}

string outgoingDistributedTracingData =
    (Agent.Tracer.CurrentSpan?.OutgoingDistributedTracingData
        ?? Agent.Tracer.CurrentTransaction?.OutgoingDistributedTracingData)?.SerializeToString();
// Now send `outgoingDistributedTracingData` to the receiving service

agent.startTransaction('my-service-a-transaction'); 
const traceparent = agent.currentTraceparent; 
sendMetadata(`traceparent: ${traceparent}\n`); 

$distDataAsString = ElasticApm::getSerializedCurrentDistributedTracingData(); 

client.begin_transaction('new-transaction')

elasticapm.get_trace_parent_header('new-transaction') 

# Send `trace_parent_str` to another service

ElasticAPM.with_span "Name" do |span| 
  header = span.trace_context.traceparent.to_header 
  # send the TraceContext Header to a receiving service...
end

Receiving services must parse the incoming traceparent header, and start a new transaction or span as a child of the received context.

// Receive incoming TraceContext
traceContext, _ := apmhttp.ParseTraceparentHeader(r.Header.Get("Traceparent")) 
traceContext.State, _ = apmhttp.ParseTracestateHeader(r.Header["Tracestate"]...) 

opts := apm.TransactionOptions{
	TraceContext: traceContext, 
}
transaction := apm.DefaultTracer().StartTransactionOptions("GET /", "request", opts) 

// Hook into a callback provided by the framework that is called on incoming requests
public Response onIncomingRequest(Request request) throws Exception {
    // creates a transaction representing the server-side handling of the request
    Transaction transaction = ElasticApm.startTransactionWithRemoteParent(request::getHeader, request::getHeaders); 
    try (final Scope scope = transaction.activate()) { 
        String name = "a useful name like ClassName#methodName where the request is handled";
        transaction.setName(name); 
        transaction.setType(Transaction.TYPE_REQUEST); 
        return request.handle();
    } catch (Exception e) {
        transaction.captureException(e);
        throw e;
    } finally {
        transaction.end(); 
    }
}

var transaction2 = Agent.Tracer.StartTransaction("Transaction2", "TestTransaction",
     DistributedTracingData.TryDeserializeFromString(serializedDistributedTracingData));

const traceparent = readTraceparentFromUDPPacket() 
agent.startTransaction('my-service-b-transaction', { childOf: traceparent }) 

$receiverTransaction = ElasticApm::beginCurrentTransaction( 
    'GET /data-api',
    'data-layer',
    /* timestamp */ null,
    $distDataAsString 
);

parent = elasticapm.trace_parent_from_headers(headers_dict) 
client.begin_transaction('processors', trace_parent=parent) 

# env being a Rack env
context = ElasticAPM::TraceContext.parse(env: env) 

ElasticAPM.with_transaction("Do things", trace_context: context) do 
  ElasticAPM.with_span("Do nested thing", trace_context: context) do 
  end
end

Some additional setup may be required to correlate requests correctly with the Real User Monitoring (RUM) agent.

See the RUM distributed tracing guide for information on enabling cross-origin requests, setting up server configuration, and working with dynamically-generated HTML.