Introduction
Trace context propagation is essential for distributed tracing in microservices architectures. It ensures that trace information is passed across service boundaries, allowing you to track requests as they flow through multiple services. This comprehensive guide covers trace context propagation using OpenTelemetry, W3C Trace Context, and custom propagation mechanisms in Java.
Core Concepts
1. Trace Context Components
- Trace ID: Unique identifier for the entire request flow
- Span ID: Identifier for a single operation within a trace
- Trace Flags: Sampling and other flags
- Trace State: Additional vendor-specific trace information
2. Propagation Standards
- W3C Trace Context: Modern standard using
traceparentandtracestateheaders - B3 Propagation: Zipkin format with single and multi-header variants
- Jaeger Propagation: Uber's Jaeger format
- OpenTelemetry: Unified standard supporting multiple formats
Project Setup and Dependencies
1. Maven Dependencies
<dependencies> <!-- OpenTelemetry --> <dependency> <groupId>io.opentelemetry</groupId> <artifactId>opentelemetry-api</artifactId> <version>1.34.1</version> </dependency> <dependency> <groupId>io.opentelemetry</groupId> <artifactId>opentelemetry-sdk</artifactId> <version>1.34.1</version> </dependency> <dependency> <groupId>io.opentelemetry</groupId> <artifactId>opentelemetry-exporter-otlp</artifactId> <version>1.34.1</version> </dependency> <dependency> <groupId>io.opentelemetry</groupId> <artifactId>opentelemetry-semconv</artifactId> <version>1.23.1-alpha</version> </dependency> <!-- Spring Boot --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-actuator</artifactId> </dependency> <!-- HTTP Client --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-webflux</artifactId> </dependency> <!-- Messaging --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-amqp</artifactId> </dependency> <!-- Testing --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <scope>test</scope> </dependency> </dependencies>
2. Application Configuration
# application.yml opentelemetry: service-name: trace-context-demo endpoint: http://localhost:4317 protocol: grpc management: endpoints: web: exposure: include: health,metrics,info endpoint: health: show-details: always logging: level: io.opentelemetry: INFO com.example.tracing: DEBUG
OpenTelemetry Configuration
1. OpenTelemetry Setup
@Configuration
@Slf4j
public class OpenTelemetryConfig {
@Bean
public OpenTelemetry openTelemetry(@Value("${opentelemetry.service-name}") String serviceName,
@Value("${opentelemetry.endpoint}") String endpoint) {
// Resource configuration
Resource resource = Resource.getDefault()
.merge(Resource.builder()
.put(SemanticResourceAttributes.SERVICE_NAME, serviceName)
.put(SemanticResourceAttributes.SERVICE_VERSION, "1.0.0")
.put(SemanticResourceAttributes.DEPLOYMENT_ENVIRONMENT, "development")
.build());
// Span exporter
SpanExporter spanExporter = OtlpGrpcSpanExporter.builder()
.setEndpoint(endpoint)
.build();
// Span processor
SpanProcessor spanProcessor = BatchSpanProcessor.builder(spanExporter)
.setScheduleDelay(100, TimeUnit.MILLISECONDS)
.build();
// Tracer provider
SdkTracerProvider tracerProvider = SdkTracerProvider.builder()
.addSpanProcessor(spanProcessor)
.setResource(resource)
.build();
// Context propagators
TextMapPropagator propagator = TextMapPropagator.composite(
W3CTraceContextPropagator.getInstance(),
W3CBaggagePropagator.getInstance(),
B3Propagator.injectingMultiHeaders()
);
// OpenTelemetry instance
OpenTelemetrySdk openTelemetry = OpenTelemetrySdk.builder()
.setTracerProvider(tracerProvider)
.setPropagators(ContextPropagators.create(propagator))
.build();
// Register as global instance
OpenTelemetrySdk.setGlobal(openTelemetry);
log.info("OpenTelemetry configured for service: {}", serviceName);
return openTelemetry;
}
@Bean
public Tracer tracer(OpenTelemetry openTelemetry) {
return openTelemetry.getTracer("trace-context-demo");
}
@Bean
public TextMapPropagator textMapPropagator(OpenTelemetry openTelemetry) {
return openTelemetry.getPropagators().getTextMapPropagator();
}
}
2. Trace Context Models
@Data
@NoArgsConstructor
@AllArgsConstructor
@Builder
public class TraceContext {
private String traceId;
private String spanId;
private String traceFlags;
private String traceState;
private Map<String, String> baggage;
public static TraceContext fromCurrentSpan() {
Span currentSpan = Span.current();
if (currentSpan.getSpanContext().isValid()) {
return TraceContext.builder()
.traceId(currentSpan.getSpanContext().getTraceId())
.spanId(currentSpan.getSpanContext().getSpanId())
.traceFlags(currentSpan.getSpanContext().getTraceFlags().asHex())
.traceState(currentSpan.getSpanContext().getTraceState().toString())
.baggage(extractBaggage())
.build();
}
return null;
}
private static Map<String, String> extractBaggage() {
Map<String, String> baggageMap = new HashMap<>();
Baggage.current().forEach((key, baggageEntry) ->
baggageMap.put(key, baggageEntry.getValue()));
return baggageMap;
}
}
@Data
@NoArgsConstructor
@AllArgsConstructor
@Builder
public class PropagationHeaders {
private Map<String, String> headers;
public static PropagationHeaders create() {
return PropagationHeaders.builder()
.headers(new HashMap<>())
.build();
}
public void addHeader(String key, String value) {
headers.put(key, value);
}
public String getHeader(String key) {
return headers.get(key);
}
}
HTTP Context Propagation
1. HTTP Client Propagation
@Component
@Slf4j
public class TracePropagatingHttpClient {
private final WebClient webClient;
private final Tracer tracer;
private final TextMapPropagator propagator;
public TracePropagatingHttpClient(WebClient.Builder webClientBuilder,
Tracer tracer,
TextMapPropagator propagator) {
this.tracer = tracer;
this.propagator = propagator;
this.webClient = webClientBuilder.build();
}
/**
* Make HTTP request with automatic trace context propagation
*/
public <T> Mono<T> getWithTracePropagation(String url, Class<T> responseType) {
return Mono.deferContextual(contextView -> {
Span span = tracer.spanBuilder("http-client-call")
.setSpanKind(SpanKind.CLIENT)
.startSpan();
try (Scope scope = span.makeCurrent()) {
// Add semantic attributes
span.setAttribute(SemanticAttributes.HTTP_METHOD, "GET");
span.setAttribute(SemanticAttributes.HTTP_URL, url);
// Create headers with trace context
PropagationHeaders headers = injectTraceContext();
// Make the HTTP call
return webClient.get()
.uri(url)
.headers(httpHeaders -> headers.getHeaders().forEach(httpHeaders::add))
.retrieve()
.bodyToMono(responseType)
.doOnSuccess(response -> {
span.setStatus(StatusCode.OK);
span.end();
})
.doOnError(error -> {
span.recordException(error);
span.setStatus(StatusCode.ERROR);
span.end();
});
}
});
}
/**
* Make POST request with trace propagation
*/
public <T, R> Mono<T> postWithTracePropagation(String url, R body, Class<T> responseType) {
return Mono.deferContextual(contextView -> {
Span span = tracer.spanBuilder("http-client-post")
.setSpanKind(SpanKind.CLIENT)
.startSpan();
try (Scope scope = span.makeCurrent()) {
span.setAttribute(SemanticAttributes.HTTP_METHOD, "POST");
span.setAttribute(SemanticAttributes.HTTP_URL, url);
PropagationHeaders headers = injectTraceContext();
return webClient.post()
.uri(url)
.headers(httpHeaders -> headers.getHeaders().forEach(httpHeaders::add))
.bodyValue(body)
.retrieve()
.bodyToMono(responseType)
.doOnSuccess(response -> {
span.setStatus(StatusCode.OK);
span.end();
})
.doOnError(error -> {
span.recordException(error);
span.setStatus(StatusCode.ERROR);
span.end();
});
}
});
}
/**
* Inject trace context into headers
*/
private PropagationHeaders injectTraceContext() {
PropagationHeaders headers = PropagationHeaders.create();
propagator.inject(Context.current(), headers, (carrier, key, value) -> {
if (carrier instanceof PropagationHeaders) {
((PropagationHeaders) carrier).addHeader(key, value);
}
});
// Add custom baggage
Baggage.current().toBuilder()
.put("client.timestamp", Instant.now().toString())
.put("client.service", "trace-context-demo")
.build()
.makeCurrent();
log.debug("Injected trace context into headers: {}", headers.getHeaders());
return headers;
}
/**
* Extract trace context from response headers
*/
public void extractTraceContextFromResponse(ClientHttpResponse response) {
try {
HttpHeaders headers = response.getHeaders();
Map<String, String> carrier = new HashMap<>();
headers.forEach((key, values) -> {
if (!values.isEmpty()) {
carrier.put(key, values.get(0));
}
});
Context context = propagator.extract(Context.current(), carrier, (carrierMap, key) ->
carrierMap.get(key));
// Update the current context with extracted trace context
if (context != Context.current()) {
context.makeCurrent();
}
log.debug("Extracted trace context from response headers");
} catch (Exception e) {
log.warn("Failed to extract trace context from response", e);
}
}
}
2. HTTP Server Propagation
@Component
@Slf4j
public class TraceExtractingFilter implements Filter {
private final Tracer tracer;
private final TextMapPropagator propagator;
public TraceExtractingFilter(Tracer tracer, TextMapPropagator propagator) {
this.tracer = tracer;
this.propagator = propagator;
}
@Override
public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain)
throws IOException, ServletException {
HttpServletRequest httpRequest = (HttpServletRequest) request;
HttpServletResponse httpResponse = (HttpServletResponse) response;
// Extract trace context from incoming headers
Context context = extractTraceContext(httpRequest);
// Create server span
Span serverSpan = createServerSpan(httpRequest, context);
try (Scope scope = serverSpan.makeCurrent()) {
// Add baggage to context
addRequestBaggage(httpRequest);
// Continue with the filter chain
chain.doFilter(request, response);
// Set span status based on response
setSpanStatus(serverSpan, httpResponse);
} catch (Exception e) {
serverSpan.recordException(e);
serverSpan.setStatus(StatusCode.ERROR, e.getMessage());
throw e;
} finally {
serverSpan.end();
}
}
private Context extractTraceContext(HttpServletRequest request) {
Map<String, String> carrier = new HashMap<>();
Enumeration<String> headerNames = request.getHeaderNames();
while (headerNames.hasMoreElements()) {
String headerName = headerNames.nextElement();
String headerValue = request.getHeader(headerName);
carrier.put(headerName, headerValue);
}
Context context = propagator.extract(Context.current(), carrier, (carrierMap, key) ->
carrierMap.get(key));
log.debug("Extracted trace context from request: {}",
context.get(SpanContextKey.KEY).getTraceId());
return context;
}
private Span createServerSpan(HttpServletRequest request, Context context) {
String spanName = request.getMethod() + " " + request.getRequestURI();
SpanBuilder spanBuilder = tracer.spanBuilder(spanName)
.setParent(context)
.setSpanKind(SpanKind.SERVER);
Span span = spanBuilder.startSpan();
// Set semantic attributes
span.setAttribute(SemanticAttributes.HTTP_METHOD, request.getMethod());
span.setAttribute(SemanticAttributes.HTTP_SCHEME, request.getScheme());
span.setAttribute(SemanticAttributes.HTTP_HOST, request.getServerName() + ":" + request.getServerPort());
span.setAttribute(SemanticAttributes.HTTP_TARGET, request.getRequestURI());
span.setAttribute(SemanticAttributes.HTTP_USER_AGENT, getHeaderSafe(request, "User-Agent"));
span.setAttribute(SemanticAttributes.HTTP_ROUTE, request.getRequestURI());
return span;
}
private void addRequestBaggage(HttpServletRequest request) {
Baggage baggage = Baggage.current();
BaggageBuilder baggageBuilder = baggage.toBuilder();
// Add request-specific baggage
baggageBuilder.put("request.client_ip", getClientIp(request));
baggageBuilder.put("request.user_agent", getHeaderSafe(request, "User-Agent"));
baggageBuilder.put("request.referer", getHeaderSafe(request, "Referer"));
baggageBuilder.build().makeCurrent();
}
private void setSpanStatus(Span span, HttpServletResponse response) {
int statusCode = response.getStatus();
span.setAttribute(SemanticAttributes.HTTP_STATUS_CODE, statusCode);
if (statusCode >= 400) {
span.setStatus(StatusCode.ERROR);
} else {
span.setStatus(StatusCode.OK);
}
}
private String getHeaderSafe(HttpServletRequest request, String headerName) {
String value = request.getHeader(headerName);
return value != null ? value : "";
}
private String getClientIp(HttpServletRequest request) {
String xForwardedFor = request.getHeader("X-Forwarded-For");
if (xForwardedFor != null && !xForwardedFor.isEmpty()) {
return xForwardedFor.split(",")[0].trim();
}
return request.getRemoteAddr();
}
}
// Register the filter
@Configuration
public class FilterConfig {
@Bean
public FilterRegistrationBean<TraceExtractingFilter> traceFilter(Tracer tracer,
TextMapPropagator propagator) {
FilterRegistrationBean<TraceExtractingFilter> registrationBean = new FilterRegistrationBean<>();
registrationBean.setFilter(new TraceExtractingFilter(tracer, propagator));
registrationBean.addUrlPatterns("/*");
registrationBean.setOrder(1);
return registrationBean;
}
}
Messaging Context Propagation
1. RabbitMQ Propagation
@Component
@Slf4j
public class TracePropagatingMessageService {
private final Tracer tracer;
private final TextMapPropagator propagator;
private final AmqpTemplate amqpTemplate;
private final ObjectMapper objectMapper;
public TracePropagatingMessageService(Tracer tracer,
TextMapPropagator propagator,
AmqpTemplate amqpTemplate,
ObjectMapper objectMapper) {
this.tracer = tracer;
this.propagator = propagator;
this.amqpTemplate = amqpTemplate;
this.objectMapper = objectMapper;
}
/**
* Send message with trace context propagation
*/
public void sendMessageWithTrace(String exchange, String routingKey, Object message) {
Span span = tracer.spanBuilder("message-publish")
.setSpanKind(SpanKind.PRODUCER)
.startSpan();
try (Scope scope = span.makeCurrent()) {
// Add message attributes to span
span.setAttribute("messaging.system", "rabbitmq");
span.setAttribute("messaging.destination", exchange);
span.setAttribute("messaging.destination_kind", "topic");
span.setAttribute("messaging.rabbitmq.routing_key", routingKey);
// Create message with trace context
Message amqpMessage = createMessageWithTraceContext(message);
// Send message
amqpTemplate.send(exchange, routingKey, amqpMessage);
span.setStatus(StatusCode.OK);
log.debug("Sent message with trace context to {}:{}", exchange, routingKey);
} catch (Exception e) {
span.recordException(e);
span.setStatus(StatusCode.ERROR);
throw new MessageSendException("Failed to send message with trace context", e);
} finally {
span.end();
}
}
/**
* Create AMQP message with injected trace context
*/
private Message createMessageWithTraceContext(Object payload) throws Exception {
MessageProperties properties = new MessageProperties();
// Inject trace context into message headers
propagator.inject(Context.current(), properties, (carrier, key, value) -> {
if (carrier instanceof MessageProperties) {
((MessageProperties) carrier).setHeader(key, value);
}
});
// Add custom baggage
Baggage.current().toBuilder()
.put("message.timestamp", Instant.now().toString())
.put("message.payload_type", payload.getClass().getSimpleName())
.build()
.makeCurrent();
// Add additional tracing headers
properties.setHeader("x-trace-service", "trace-context-demo");
properties.setHeader("x-trace-version", "1.0");
byte[] body = objectMapper.writeValueAsBytes(payload);
return new Message(body, properties);
}
/**
* Extract trace context from received message
*/
public Context extractTraceContext(Message message) {
MessageProperties properties = message.getMessageProperties();
Map<String, String> carrier = new HashMap<>();
if (properties.getHeaders() != null) {
properties.getHeaders().forEach((key, value) -> {
if (value instanceof String) {
carrier.put(key, (String) value);
}
});
}
Context context = propagator.extract(Context.current(), carrier, (carrierMap, key) ->
carrierMap.get(key));
log.debug("Extracted trace context from message: {}",
context.get(SpanContextKey.KEY).getTraceId());
return context;
}
}
@Component
@Slf4j
public class TraceAwareMessageListener {
private final Tracer tracer;
private final TextMapPropagator propagator;
public TraceAwareMessageListener(Tracer tracer, TextMapPropagator propagator) {
this.tracer = tracer;
this.propagator = propagator;
}
/**
* Process message with trace context extraction
*/
public <T> void processMessageWithTrace(Message message, Class<T> payloadType,
Function<T, Boolean> processor) {
// Extract trace context from message
Context context = extractTraceContext(message);
// Create consumer span
Span span = createConsumerSpan(message, context, payloadType);
try (Scope scope = span.makeCurrent()) {
// Extract payload
T payload = extractPayload(message, payloadType);
// Add message baggage
addMessageBaggage(message, payload);
// Process the message
boolean success = processor.apply(payload);
if (success) {
span.setStatus(StatusCode.OK);
log.debug("Successfully processed message with trace: {}",
span.getSpanContext().getTraceId());
} else {
span.setStatus(StatusCode.ERROR, "Message processing failed");
}
} catch (Exception e) {
span.recordException(e);
span.setStatus(StatusCode.ERROR, e.getMessage());
throw new MessageProcessingException("Failed to process message", e);
} finally {
span.end();
}
}
private <T> Span createConsumerSpan(Message message, Context context, Class<T> payloadType) {
MessageProperties properties = message.getMessageProperties();
String spanName = "message.process " + properties.getConsumerQueue();
SpanBuilder spanBuilder = tracer.spanBuilder(spanName)
.setParent(context)
.setSpanKind(SpanKind.CONSUMER);
Span span = spanBuilder.startSpan();
// Set messaging attributes
span.setAttribute("messaging.system", "rabbitmq");
span.setAttribute("messaging.destination", properties.getReceivedExchange());
span.setAttribute("messaging.destination_kind", "topic");
span.setAttribute("messaging.rabbitmq.routing_key", properties.getReceivedRoutingKey());
span.setAttribute("messaging.message.payload_size", message.getBody().length);
span.setAttribute("messaging.message.payload_type", payloadType.getSimpleName());
return span;
}
private <T> T extractPayload(Message message, Class<T> payloadType) {
try {
return new ObjectMapper().readValue(message.getBody(), payloadType);
} catch (Exception e) {
throw new MessageProcessingException("Failed to extract message payload", e);
}
}
private void addMessageBaggage(Message message, Object payload) {
MessageProperties properties = message.getMessageProperties();
BaggageBuilder baggageBuilder = Baggage.current().toBuilder();
baggageBuilder.put("message.queue", properties.getConsumerQueue());
baggageBuilder.put("message.exchange", properties.getReceivedExchange());
baggageBuilder.put("message.routing_key", properties.getReceivedRoutingKey());
baggageBuilder.put("message.delivery_tag", properties.getDeliveryTag().toString());
baggageBuilder.put("message.payload_type", payload.getClass().getSimpleName());
baggageBuilder.build().makeCurrent();
}
private Context extractTraceContext(Message message) {
MessageProperties properties = message.getMessageProperties();
Map<String, String> carrier = new HashMap<>();
if (properties.getHeaders() != null) {
properties.getHeaders().forEach((key, value) -> {
if (value instanceof String) {
carrier.put(key, (String) value);
}
});
}
return propagator.extract(Context.current(), carrier, (carrierMap, key) ->
carrierMap.get(key));
}
}
Custom Context Propagation
1. Thread Pool Propagation
@Component
@Slf4j
public class TraceAwareExecutorService {
private final Tracer tracer;
private final TextMapPropagator propagator;
public TraceAwareExecutorService(Tracer tracer, TextMapPropagator propagator) {
this.tracer = tracer;
this.propagator = propagator;
}
/**
* Create trace-aware executor service
*/
public ExecutorService createTraceAwareExecutor(int corePoolSize, int maxPoolSize, String threadNamePrefix) {
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(corePoolSize);
executor.setMaxPoolSize(maxPoolSize);
executor.setThreadNamePrefix(threadNamePrefix);
executor.setTaskDecorator(new TraceContextTaskDecorator(tracer, propagator));
executor.initialize();
return executor.getThreadPoolExecutor();
}
/**
* Execute runnable with trace context propagation
*/
public void executeWithTrace(Runnable task) {
Context currentContext = Context.current();
Map<String, String> traceContext = captureTraceContext();
Runnable tracedTask = () -> {
// Restore trace context in the new thread
Context restoredContext = restoreTraceContext(traceContext);
try (Scope scope = restoredContext.makeCurrent()) {
task.run();
} catch (Exception e) {
log.error("Error in traced task execution", e);
throw e;
}
};
// Use a trace-aware executor or the common pool
ForkJoinPool.commonPool().execute(tracedTask);
}
/**
* Submit callable with trace context propagation
*/
public <T> CompletableFuture<T> submitWithTrace(Callable<T> task) {
Map<String, String> traceContext = captureTraceContext();
return CompletableFuture.supplyAsync(() -> {
Context restoredContext = restoreTraceContext(traceContext);
try (Scope scope = restoredContext.makeCurrent()) {
return task.call();
} catch (Exception e) {
throw new CompletionException(e);
}
});
}
/**
* Capture current trace context as a map
*/
private Map<String, String> captureTraceContext() {
Map<String, String> contextMap = new HashMap<>();
// Capture W3C Trace Context
propagator.inject(Context.current(), contextMap, (carrier, key, value) ->
carrier.put(key, value));
// Capture baggage
Baggage.current().forEach((key, baggageEntry) ->
contextMap.put("baggage-" + key, baggageEntry.getValue()));
log.debug("Captured trace context: {}", contextMap.keySet());
return contextMap;
}
/**
* Restore trace context from captured map
*/
private Context restoreTraceContext(Map<String, String> traceContext) {
// Extract W3C Trace Context
Context context = propagator.extract(Context.current(), traceContext, (carrier, key) ->
carrier.get(key));
// Restore baggage
BaggageBuilder baggageBuilder = Baggage.builder();
traceContext.forEach((key, value) -> {
if (key.startsWith("baggage-")) {
String baggageKey = key.substring(8); // Remove "baggage-" prefix
baggageBuilder.put(baggageKey, value);
}
});
Baggage baggage = baggageBuilder.build();
return context.with(baggage);
}
/**
* Task decorator for Spring's ThreadPoolTaskExecutor
*/
public static class TraceContextTaskDecorator implements TaskDecorator {
private final Tracer tracer;
private final TextMapPropagator propagator;
public TraceContextTaskDecorator(Tracer tracer, TextMapPropagator propagator) {
this.tracer = tracer;
this.propagator = propagator;
}
@Override
public Runnable decorate(Runnable runnable) {
Map<String, String> parentTraceContext = captureParentTraceContext();
return () -> {
Context restoredContext = restoreTraceContext(parentTraceContext);
try (Scope scope = restoredContext.makeCurrent()) {
runnable.run();
}
};
}
private Map<String, String> captureParentTraceContext() {
Map<String, String> contextMap = new HashMap<>();
propagator.inject(Context.current(), contextMap, (carrier, key, value) ->
carrier.put(key, value));
return contextMap;
}
private Context restoreTraceContext(Map<String, String> traceContext) {
return propagator.extract(Context.current(), traceContext, (carrier, key) ->
carrier.get(key));
}
}
}
2. Database Context Propagation
@Component
@Slf4j
public class TraceAwareJdbcTemplate {
private final JdbcTemplate jdbcTemplate;
private final Tracer tracer;
public TraceAwareJdbcTemplate(JdbcTemplate jdbcTemplate, Tracer tracer) {
this.jdbcTemplate = jdbcTemplate;
this.tracer = tracer;
}
/**
* Execute query with trace context
*/
public <T> List<T> queryWithTrace(String sql, Object[] args, RowMapper<T> rowMapper) {
Span span = tracer.spanBuilder("database.query")
.setSpanKind(SpanKind.CLIENT)
.startSpan();
try (Scope scope = span.makeCurrent()) {
// Add database attributes
span.setAttribute(SemanticAttributes.DB_SYSTEM, "postgresql");
span.setAttribute(SemanticAttributes.DB_STATEMENT, sql);
span.setAttribute(SemanticAttributes.DB_OPERATION, extractDbOperation(sql));
// Add trace context to SQL comment
String tracedSql = addTraceContextToSql(sql);
// Execute query
List<T> result = jdbcTemplate.query(tracedSql, args, rowMapper);
span.setAttribute("db.result.count", result.size());
span.setStatus(StatusCode.OK);
return result;
} catch (Exception e) {
span.recordException(e);
span.setStatus(StatusCode.ERROR);
throw new DataAccessException("Database query failed", e) {};
} finally {
span.end();
}
}
/**
* Execute update with trace context
*/
public int updateWithTrace(String sql, Object... args) {
Span span = tracer.spanBuilder("database.update")
.setSpanKind(SpanKind.CLIENT)
.startSpan();
try (Scope scope = span.makeCurrent()) {
span.setAttribute(SemanticAttributes.DB_SYSTEM, "postgresql");
span.setAttribute(SemanticAttributes.DB_STATEMENT, sql);
span.setAttribute(SemanticAttributes.DB_OPERATION, extractDbOperation(sql));
String tracedSql = addTraceContextToSql(sql);
int affectedRows = jdbcTemplate.update(tracedSql, args);
span.setAttribute("db.affected.rows", affectedRows);
span.setStatus(StatusCode.OK);
return affectedRows;
} catch (Exception e) {
span.recordException(e);
span.setStatus(StatusCode.ERROR);
throw new DataAccessException("Database update failed", e) {};
} finally {
span.end();
}
}
/**
* Add trace context as SQL comment
*/
private String addTraceContextToSql(String sql) {
Span currentSpan = Span.current();
if (!currentSpan.getSpanContext().isValid()) {
return sql;
}
String traceId = currentSpan.getSpanContext().getTraceId();
String spanId = currentSpan.getSpanContext().getSpanId();
// Add trace context as SQL comment (PostgreSQL style)
String traceComment = String.format("/* trace_id=%s, span_id=%s */ ", traceId, spanId);
return traceComment + sql;
}
/**
* Extract database operation from SQL
*/
private String extractDbOperation(String sql) {
if (sql == null) return "unknown";
String normalizedSql = sql.trim().toLowerCase();
if (normalizedSql.startsWith("select")) return "SELECT";
if (normalizedSql.startsWith("insert")) return "INSERT";
if (normalizedSql.startsWith("update")) return "UPDATE";
if (normalizedSql.startsWith("delete")) return "DELETE";
if (normalizedSql.startsWith("create")) return "CREATE";
if (normalizedSql.startsWith("drop")) return "DROP";
if (normalizedSql.startsWith("alter")) return "ALTER";
return "OTHER";
}
}
// JPA/Hibernate interceptor for trace context
@Component
@Slf4j
public class TraceContextJpaInterceptor {
private final Tracer tracer;
public TraceContextJpaInterceptor(Tracer tracer) {
this.tracer = tracer;
}
@EventListener
public void handleJpaEvent(PreInsertEvent event) {
addTraceContextToEntity(event.getEntity());
}
@EventListener
public void handleJpaEvent(PreUpdateEvent event) {
addTraceContextToEntity(event.getEntity());
}
private void addTraceContextToEntity(Object entity) {
if (entity instanceof TraceableEntity) {
TraceableEntity traceableEntity = (TraceableEntity) entity;
Span currentSpan = Span.current();
if (currentSpan.getSpanContext().isValid()) {
traceableEntity.setTraceId(currentSpan.getSpanContext().getTraceId());
traceableEntity.setSpanId(currentSpan.getSpanContext().getSpanId());
traceableEntity.setLastTracedAt(Instant.now());
}
}
}
}
// Marker interface for traceable entities
public interface TraceableEntity {
void setTraceId(String traceId);
void setSpanId(String spanId);
void setLastTracedAt(Instant timestamp);
}
Advanced Propagation Patterns
1. Cross-Service Correlation
@Component
@Slf4j
public class CorrelationIdPropagator {
private static final String CORRELATION_ID_HEADER = "X-Correlation-ID";
private static final String CORRELATION_ID_BAGGAGE_KEY = "correlation.id";
private final Tracer tracer;
private final TextMapPropagator propagator;
public CorrelationIdPropagator(Tracer tracer, TextMapPropagator propagator) {
this.tracer = tracer;
this.propagator = propagator;
}
/**
* Get or generate correlation ID
*/
public String getCorrelationId() {
// First try to get from baggage
String correlationId = Baggage.current().getEntryValue(CORRELATION_ID_BAGGAGE_KEY);
if (correlationId == null || correlationId.isEmpty()) {
// Generate new correlation ID
correlationId = generateCorrelationId();
// Store in baggage
Baggage.current().toBuilder()
.put(CORRELATION_ID_BAGGAGE_KEY, correlationId)
.build()
.makeCurrent();
}
return correlationId;
}
/**
* Set correlation ID in current context
*/
public void setCorrelationId(String correlationId) {
Baggage.current().toBuilder()
.put(CORRELATION_ID_BAGGAGE_KEY, correlationId)
.build()
.makeCurrent();
}
/**
* Inject correlation ID into headers
*/
public void injectCorrelationId(Map<String, String> headers) {
String correlationId = getCorrelationId();
headers.put(CORRELATION_ID_HEADER, correlationId);
// Also inject into trace context
propagator.inject(Context.current(), headers, (carrier, key, value) ->
carrier.put(key, value));
}
/**
* Extract correlation ID from headers
*/
public String extractCorrelationId(Map<String, String> headers) {
String correlationId = headers.get(CORRELATION_ID_HEADER);
if (correlationId != null && !correlationId.isEmpty()) {
setCorrelationId(correlationId);
// Also extract trace context
Context context = propagator.extract(Context.current(), headers, (carrier, key) ->
carrier.get(key));
context.makeCurrent();
} else {
// Generate new correlation ID if not present
correlationId = getCorrelationId();
}
return correlationId;
}
/**
* Create span with correlation ID attributes
*/
public Span createCorrelatedSpan(String name) {
String correlationId = getCorrelationId();
Span span = tracer.spanBuilder(name)
.setAttribute("correlation.id", correlationId)
.startSpan();
// Add correlation ID to span attributes
span.setAttribute("correlation.id", correlationId);
return span;
}
private String generateCorrelationId() {
return "corr-" + UUID.randomUUID().toString();
}
}
2. Baggage Management
@Component
@Slf4j
public class BaggageManager {
/**
* Add key-value pair to current baggage
*/
public void addToBaggage(String key, String value) {
Baggage.current().toBuilder()
.put(key, value)
.build()
.makeCurrent();
log.debug("Added to baggage: {} = {}", key, value);
}
/**
* Get value from current baggage
*/
public String getFromBaggage(String key) {
return Baggage.current().getEntryValue(key);
}
/**
* Remove key from current baggage
*/
public void removeFromBaggage(String key) {
Baggage.current().toBuilder()
.remove(key)
.build()
.makeCurrent();
log.debug("Removed from baggage: {}", key);
}
/**
* Get all baggage entries
*/
public Map<String, String> getAllBaggage() {
Map<String, String> baggageMap = new HashMap<>();
Baggage.current().forEach((key, baggageEntry) ->
baggageMap.put(key, baggageEntry.getValue()));
return baggageMap;
}
/**
* Clear all baggage
*/
public void clearBaggage() {
Baggage.empty().makeCurrent();
log.debug("Cleared all baggage");
}
/**
* Create baggage with common context fields
*/
public void initializeCommonBaggage(String serviceName, String environment) {
Baggage baggage = Baggage.builder()
.put("service.name", serviceName)
.put("service.environment", environment)
.put("service.instance.id", getInstanceId())
.put("startup.timestamp", Instant.now().toString())
.build();
baggage.makeCurrent();
log.debug("Initialized common baggage for service: {}", serviceName);
}
/**
* Add user context to baggage
*/
public void addUserContext(String userId, String sessionId, String userAgent) {
Baggage.current().toBuilder()
.put("user.id", userId)
.put("user.session.id", sessionId)
.put("user.agent", userAgent)
.build()
.makeCurrent();
log.debug("Added user context to baggage: userId={}", userId);
}
/**
* Add business context to baggage
*/
public void addBusinessContext(String tenantId, String businessUnit, String transactionType) {
Baggage.current().toBuilder()
.put("tenant.id", tenantId)
.put("business.unit", businessUnit)
.put("transaction.type", transactionType)
.build()
.makeCurrent();
log.debug("Added business context to baggage: tenantId={}", tenantId);
}
private String getInstanceId() {
try {
return InetAddress.getLocalHost().getHostName() + "-" +
ManagementFactory.getRuntimeMXBean().getName().split("@")[0];
} catch (Exception e) {
return "unknown-" + UUID.randomUUID().toString().substring(0, 8);
}
}
}
REST Controller with Trace Propagation
@RestController
@RequestMapping("/api/trace")
@Slf4j
public class TraceDemoController {
private final Tracer tracer;
private final TracePropagatingHttpClient httpClient;
private final TracePropagatingMessageService messageService;
private final CorrelationIdPropagator correlationPropagator;
private final BaggageManager baggageManager;
public TraceDemoController(Tracer tracer,
TracePropagatingHttpClient httpClient,
TracePropagatingMessageService messageService,
CorrelationIdPropagator correlationPropagator,
BaggageManager baggageManager) {
this.tracer = tracer;
this.httpClient = httpClient;
this.messageService = messageService;
this.correlationPropagator = correlationPropagator;
this.baggageManager = baggageManager;
}
@GetMapping("/demo")
public ResponseEntity<TraceDemoResponse> traceDemo(HttpServletRequest request) {
// Extract correlation ID from headers
String correlationId = extractCorrelationId(request);
// Create span for this request
Span span = correlationPropagator.createCorrelatedSpan("trace-demo");
try (Scope scope = span.makeCurrent()) {
// Add request details to baggage
baggageManager.addToBaggage("request.path", request.getRequestURI());
baggageManager.addToBaggage("request.method", request.getMethod());
// Simulate some work
simulateWork();
// Make external HTTP call with trace propagation
String externalResponse = httpClient.getWithTracePropagation(
"https://httpbin.org/uuid", String.class).block();
// Send message with trace propagation
messageService.sendMessageWithTrace(
"demo-exchange",
"demo.routing.key",
new DemoMessage("Hello from trace demo", correlationId)
);
TraceDemoResponse response = TraceDemoResponse.builder()
.correlationId(correlationId)
.traceId(span.getSpanContext().getTraceId())
.spanId(span.getSpanContext().getSpanId())
.message("Trace context propagated successfully")
.baggage(baggageManager.getAllBaggage())
.externalCallResult(externalResponse)
.build();
span.setStatus(StatusCode.OK);
return ResponseEntity.ok(response);
} catch (Exception e) {
span.recordException(e);
span.setStatus(StatusCode.ERROR);
throw e;
} finally {
span.end();
}
}
@PostMapping("/propagate")
public ResponseEntity<PropagationResult> testPropagation(@RequestBody PropagationRequest request) {
Span span = tracer.spanBuilder("propagation-test")
.setSpanKind(SpanKind.SERVER)
.startSpan();
try (Scope scope = span.makeCurrent()) {
// Set correlation ID from request or generate new
if (request.getCorrelationId() != null) {
correlationPropagator.setCorrelationId(request.getCorrelationId());
}
// Add custom baggage from request
if (request.getCustomBaggage() != null) {
request.getCustomBaggage().forEach(baggageManager::addToBaggage);
}
// Simulate multi-step processing with trace propagation
List<String> stepResults = processWithPropagation(request.getSteps());
PropagationResult result = PropagationResult.builder()
.correlationId(correlationPropagator.getCorrelationId())
.traceId(span.getSpanContext().getTraceId())
.stepsExecuted(stepResults.size())
.stepResults(stepResults)
.currentBaggage(baggageManager.getAllBaggage())
.build();
span.setStatus(StatusCode.OK);
return ResponseEntity.ok(result);
} finally {
span.end();
}
}
private String extractCorrelationId(HttpServletRequest request) {
Map<String, String> headers = new HashMap<>();
Enumeration<String> headerNames = request.getHeaderNames();
while (headerNames.hasMoreElements()) {
String headerName = headerNames.nextElement();
headers.put(headerName, request.getHeader(headerName));
}
return correlationPropagator.extractCorrelationId(headers);
}
private void simulateWork() {
try {
// Simulate processing time
Thread.sleep(100 + new Random().nextInt(200));
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
private List<String> processWithPropagation(List<String> steps) {
TraceAwareExecutorService executorService = new TraceAwareExecutorService(tracer, null);
List<CompletableFuture<String>> futures = new ArrayList<>();
for (String step : steps) {
CompletableFuture<String> future = executorService.submitWithTrace(() -> {
// Simulate step processing
Thread.sleep(50);
return "Processed: " + step + " (trace: " +
Span.current().getSpanContext().getTraceId() + ")";
});
futures.add(future);
}
// Wait for all steps to complete
return futures.stream()
.map(CompletableFuture::join)
.collect(Collectors.toList());
}
@Data
@Builder
public static class TraceDemoResponse {
private String correlationId;
private String traceId;
private String spanId;
private String message;
private Map<String, String> baggage;
private String externalCallResult;
}
@Data
@Builder
public static class PropagationRequest {
private String correlationId;
private Map<String, String> customBaggage;
private List<String> steps;
}
@Data
@Builder
public static class PropagationResult {
private String correlationId;
private String traceId;
private int stepsExecuted;
private List<String> stepResults;
private Map<String, String> currentBaggage;
}
@Data
@AllArgsConstructor
public static class DemoMessage {
private String content;
private String correlationId;
}
}
Testing Trace Propagation
1. Propagation Test Suite
@SpringBootTest
@ActiveProfiles("test")
class TracePropagationTest {
@Autowired
private Tracer tracer;
@Autowired
private TextMapPropagator propagator;
@Autowired
private CorrelationIdPropagator correlationPropagator;
@MockBean
private WebClient webClient;
@Test
void testTraceContextPropagation() {
// Given
Span span = tracer.spanBuilder("test-span").startSpan();
try (Scope scope = span.makeCurrent()) {
// When - inject trace context
Map<String, String> carrier = new HashMap<>();
propagator.inject(Context.current(), carrier, (c, key, value) -> c.put(key, value));
// Then - verify trace context headers
assertNotNull(carrier.get("traceparent"));
assertNotNull(carrier.get("tracestate"));
// When - extract trace context
Context extractedContext = propagator.extract(Context.current(), carrier, (c, key) -> c.get(key));
SpanContext extractedSpanContext = Span.fromContext(extractedContext).getSpanContext();
// Then - verify extracted context matches original
assertEquals(span.getSpanContext().getTraceId(), extractedSpanContext.getTraceId());
assertEquals(span.getSpanContext().getTraceFlags(), extractedSpanContext.getTraceFlags());
} finally {
span.end();
}
}
@Test
void testCorrelationIdPropagation() {
// Given
String testCorrelationId = "test-corr-123";
// When
correlationPropagator.setCorrelationId(testCorrelationId);
Map<String, String> headers = new HashMap<>();
correlationPropagator.injectCorrelationId(headers);
// Then
assertEquals(testCorrelationId, headers.get("X-Correlation-ID"));
assertEquals(testCorrelationId, correlationPropagator.getCorrelationId());
// When - extract from headers
String extractedCorrelationId = correlationPropagator.extractCorrelationId(headers);
// Then
assertEquals(testCorrelationId, extractedCorrelationId);
}
@Test
void testBaggagePropagation() {
// Given
BaggageManager baggageManager = new BaggageManager();
baggageManager.addToBaggage("user.id", "test-user-123");
baggageManager.addToBaggage("tenant.id", "test-tenant");
// When
Map<String, String> baggage = baggageManager.getAllBaggage();
// Then
assertEquals("test-user-123", baggage.get("user.id"));
assertEquals("test-tenant", baggage.get("tenant.id"));
// When - clear baggage
baggageManager.clearBaggage();
// Then
assertTrue(baggageManager.getAllBaggage().isEmpty());
}
@Test
void testThreadPoolPropagation() throws Exception {
// Given
TraceAwareExecutorService executorService = new TraceAwareExecutorService(tracer, propagator);
Span parentSpan = tracer.spanBuilder("parent-span").startSpan();
try (Scope scope = parentSpan.makeCurrent()) {
String expectedTraceId = parentSpan.getSpanContext().getTraceId();
// When - execute task in different thread
CompletableFuture<String> future = executorService.submitWithTrace(() -> {
Span currentSpan = Span.current();
return "Trace ID in thread: " + currentSpan.getSpanContext().getTraceId();
});
String result = future.get(5, TimeUnit.SECONDS);
// Then - verify trace context was propagated
assertTrue(result.contains(expectedTraceId));
} finally {
parentSpan.end();
}
}
}
Monitoring and Observability
1. Trace Context Metrics
@Component
@Slf4j
public class TraceContextMetrics {
private final MeterRegistry meterRegistry;
private final Counter propagationSuccessCounter;
private final Counter propagationFailureCounter;
private final Timer propagationTimer;
public TraceContextMetrics(MeterRegistry meterRegistry) {
this.meterRegistry = meterRegistry;
this.propagationSuccessCounter = meterRegistry.counter("trace.propagation.success");
this.propagationFailureCounter = meterRegistry.counter("trace.propagation.failure");
this.propagationTimer = meterRegistry.timer("trace.propagation.duration");
}
public void recordPropagationSuccess(String propagationType, Duration duration) {
propagationSuccessCounter.increment();
Tags tags = Tags.of(
Tag.of("propagation.type", propagationType)
);
meterRegistry.counter("trace.propagation.success.detailed", tags).increment();
meterRegistry.timer("trace.propagation.duration.detailed", tags).record(duration);
log.debug("Trace propagation successful - type: {}, duration: {}ms",
propagationType, duration.toMillis());
}
public void recordPropagationFailure(String propagationType, String errorType) {
propagationFailureCounter.increment();
Tags tags = Tags.of(
Tag.of("propagation.type", propagationType),
Tag.of("error.type", errorType)
);
meterRegistry.counter("trace.propagation.failure.detailed", tags).increment();
log.warn("Trace propagation failed - type: {}, error: {}", propagationType, errorType);
}
public void recordBaggageOperation(String operation, String key) {
Tags tags = Tags.of(
Tag.of("operation", operation),
Tag.of("key", key)
);
meterRegistry.counter("trace.baggage.operations", tags).increment();
}
}
Conclusion
Trace context propagation is crucial for observability in distributed systems. The OpenTelemetry standard provides a robust foundation for implementing trace propagation across various communication channels.
Key Implementation Patterns:
- HTTP Propagation: Automatic context injection/extraction for REST APIs
- Messaging Propagation: Trace context propagation through message brokers
- Thread Pool Propagation: Context preservation across asynchronous boundaries
- Database Propagation: SQL comments and entity tracing
Best Practices:
- Standard Compliance: Use W3C Trace Context for interoperability
- Baggage Management: Use baggage for non-trace context information
- Performance: Minimize baggage size and propagation overhead
- Security: Be cautious with sensitive information in trace context
- Testing: Comprehensive testing of propagation across service boundaries
Use Cases:
- Microservices Architectures: End-to-end request tracing
- Async Processing: Correlation of related asynchronous operations
- Batch Processing: Tracing of batch job execution flows
- Third-Party Integrations: Context propagation to external services
By implementing proper trace context propagation, you gain complete visibility into your distributed system's behavior, enabling effective debugging, performance optimization, and operational monitoring.