Tempo tracing refers to implementing distributed tracing in Java applications, inspired by Grafana Tempo's approach to tracing. This involves capturing, processing, and visualizing request flows across microservices to identify performance bottlenecks, debug issues, and understand system behavior. Java provides robust tools for implementing distributed tracing that integrates with modern observability platforms.
Understanding Tempo Tracing Architecture
Key Components:
- Trace Context Propagation across service boundaries
- Span Creation for operations and method calls
- Trace Collection and export to backend systems
- Trace ID Generation and sampling strategies
- Integration with Logging and metrics
Core Implementation Patterns
1. Project Setup and Dependencies
Configure tracing dependencies and instrumentation.
Maven Configuration:
<dependencies> <!-- OpenTelemetry SDK --> <dependency> <groupId>io.opentelemetry</groupId> <artifactId>opentelemetry-api</artifactId> <version>1.31.0</version> </dependency> <dependency> <groupId>io.opentelemetry</groupId> <artifactId>opentelemetry-sdk</artifactId> <version>1.31.0</version> </dependency> <dependency> <groupId>io.opentelemetry</groupId> <artifactId>opentelemetry-exporter-otlp</artifactId> <version>1.31.0</version> </dependency> <!-- Micrometer Tracing --> <dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-tracing</artifactId> <version>1.1.4</version> </dependency> <dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-tracing-bridge-otel</artifactId> <version>1.1.4</version> </dependency> <!-- Spring Boot Starter --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-actuator</artifactId> <version>3.1.0</version> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-aop</artifactId> <version>3.1.0</version> </dependency> <!-- HTTP Client --> <dependency> <groupId>org.apache.httpcomponents.client5</groupId> <artifactId>httpclient5</artifactId> <version>5.2.1</version> </dependency> <!-- JSON Processing --> <dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-databind</artifactId> <version>2.15.2</version> </dependency> </dependencies>
2. Tracing Configuration and Setup
Configure OpenTelemetry and tracing components.
Tracing Configuration:
@Configuration
@EnableAspectJAutoProxy
public class TracingConfig {
@Bean
public OpenTelemetry openTelemetry() {
return OpenTelemetrySdk.builder()
.setTracerProvider(
SdkTracerProvider.builder()
.addSpanProcessor(BatchSpanProcessor.builder(
OtlpSpanExporter.builder()
.setEndpoint("http://localhost:4317")
.build()
).build())
.setSampler(Sampler.alwaysOn())
.build()
)
.setPropagators(
ContextPropagators.create(
TextMapPropagator.composite(
W3CTraceContextPropagator.getInstance(),
W3CBaggagePropagator.getInstance()
)
)
)
.build();
}
@Bean
public Tracer tracer(OpenTelemetry openTelemetry) {
return openTelemetry.getTracer("com.example.tempo", "1.0.0");
}
@Bean
public MeterRegistry meterRegistry() {
return new SimpleMeterRegistry();
}
@Bean
public ObservationRegistry observationRegistry(MeterRegistry meterRegistry, Tracer tracer) {
ObservationRegistry registry = ObservationRegistry.create();
registry.observationConfig()
.observationHandler(new DefaultMeterObservationHandler(meterRegistry))
.observationHandler(new TracingObservationHandler(tracer));
return registry;
}
}
@Component
public class TracingContext {
private final Tracer tracer;
private final ContextPropagators propagators;
public TracingContext(OpenTelemetry openTelemetry) {
this.tracer = openTelemetry.getTracer("tempo-tracing");
this.propagators = openTelemetry.getPropagators();
}
public Span startSpan(String name) {
return tracer.spanBuilder(name).startSpan();
}
public Span startSpan(String name, SpanContext parentContext) {
return tracer.spanBuilder(name)
.setParent(Context.current().with(Span.wrap(parentContext)))
.startSpan();
}
public void injectContext(Map<String, String> carrier) {
TextMapSetter<Map<String, String>> setter = (carrier, key, value) -> {
if (carrier != null) {
carrier.put(key, value);
}
};
propagators.getTextMapPropagator().inject(Context.current(), carrier, setter);
}
public Context extractContext(Map<String, String> carrier) {
TextMapGetter<Map<String, String>> getter = new TextMapGetter<Map<String, String>>() {
@Override
public String get(Map<String, String> carrier, String key) {
return carrier != null ? carrier.get(key) : null;
}
@Override
public Iterable<String> keys(Map<String, String> carrier) {
return carrier != null ? carrier.keySet() : Collections.emptySet();
}
};
return propagators.getTextMapPropagator().extract(Context.current(), carrier, getter);
}
}
3. Domain Models for Tracing
Create comprehensive models for tracing data.
Core Domain Models:
@Data
public class TraceContext {
private String traceId;
private String spanId;
private String traceFlags;
private String traceState;
private boolean isRemote;
private boolean isValid;
private Map<String, String> baggage;
public static TraceContext fromCurrent() {
Span currentSpan = Span.current();
if (!currentSpan.getSpanContext().isValid()) {
return createInvalid();
}
TraceContext context = new TraceContext();
context.setTraceId(currentSpan.getSpanContext().getTraceId());
context.setSpanId(currentSpan.getSpanContext().getSpanId());
context.setTraceFlags(currentSpan.getSpanContext().getTraceFlags().asHex());
context.setTraceState(currentSpan.getSpanContext().getTraceState().toString());
context.setValid(true);
context.setBaggage(new HashMap<>());
return context;
}
public static TraceContext createInvalid() {
TraceContext context = new TraceContext();
context.setValid(false);
context.setBaggage(new HashMap<>());
return context;
}
public SpanContext toSpanContext() {
if (!isValid) {
return SpanContext.getInvalid();
}
return SpanContext.createFromRemoteParent(
traceId,
spanId,
TraceFlags.fromHex(traceFlags, 0),
TraceState.builder().build()
);
}
}
@Data
public class SpanData {
private String name;
private String traceId;
private String spanId;
private String parentSpanId;
private long startTime;
private long endTime;
private SpanKind kind;
private SpanStatus status;
private Map<String, Object> attributes;
private List<SpanEvent> events;
private List<SpanLink> links;
public Duration getDuration() {
return Duration.ofNanos(endTime - startTime);
}
public boolean isError() {
return status != null && status == SpanStatus.ERROR;
}
}
@Data
public class SpanEvent {
private String name;
private long timestamp;
private Map<String, Object> attributes;
}
@Data
public class SpanLink {
private String traceId;
private String spanId;
private Map<String, Object> attributes;
}
@Data
public class TraceRequest {
private String serviceName;
private String operationName;
private Map<String, String> headers;
private Map<String, Object> attributes;
private boolean forceSampling = false;
private String correlationId;
public Map<String, String> getTracingHeaders() {
Map<String, String> tracingHeaders = new HashMap<>();
if (headers != null) {
tracingHeaders.putAll(headers);
}
tracingHeaders.put("x-correlation-id", correlationId);
return tracingHeaders;
}
}
@Data
public class TraceResponse {
private String traceId;
private String spanId;
private boolean sampled;
private Duration duration;
private SpanStatus status;
private Map<String, Object> metrics;
private List<String> warnings;
public boolean isSuccess() {
return status == SpanStatus.UNSET || status == SpanStatus.OK;
}
}
public enum SpanStatus {
UNSET, OK, ERROR
}
public enum SpanKind {
INTERNAL, SERVER, CLIENT, PRODUCER, CONSUMER
}
@Data
public class TraceQuery {
private String traceId;
private String serviceName;
private String operationName;
private Duration minDuration;
private Duration maxDuration;
private LocalDateTime startTime;
private LocalDateTime endTime;
private Map<String, String> tags;
private int limit = 100;
public boolean isValid() {
return traceId != null || (serviceName != null && operationName != null);
}
}
@Data
public class TraceResult {
private String traceId;
private List<SpanData> spans;
private Duration totalDuration;
private String rootService;
private boolean hasErrors;
private int totalSpans;
private Map<String, Object> summary;
public Optional<SpanData> getRootSpan() {
return spans.stream()
.filter(span -> span.getParentSpanId() == null || span.getParentSpanId().isEmpty())
.findFirst();
}
public Map<String, Long> getServiceDurations() {
return spans.stream()
.collect(Collectors.groupingBy(
span -> getServiceFromAttributes(span),
Collectors.summingLong(span -> span.getDuration().toMillis())
));
}
private String getServiceFromAttributes(SpanData span) {
if (span.getAttributes() != null && span.getAttributes().containsKey("service.name")) {
return span.getAttributes().get("service.name").toString();
}
return "unknown";
}
}
4. Core Tracing Service
Implement the main tracing functionality.
Tracing Service:
@Service
@Slf4j
public class TempoTracingService {
private final Tracer tracer;
private final TracingContext tracingContext;
private final TraceStorageService storageService;
private final TraceSampler traceSampler;
public TempoTracingService(Tracer tracer, TracingContext tracingContext,
TraceStorageService storageService, TraceSampler traceSampler) {
this.tracer = tracer;
this.tracingContext = tracingContext;
this.storageService = storageService;
this.traceSampler = traceSampler;
}
public <T> T traceOperation(String operationName, TraceableOperation<T> operation) {
return traceOperation(operationName, Collections.emptyMap(), operation);
}
public <T> T traceOperation(String operationName, Map<String, Object> attributes,
TraceableOperation<T> operation) {
Span span = tracer.spanBuilder(operationName).startSpan();
try (Scope scope = span.makeCurrent()) {
// Set attributes
attributes.forEach((key, value) ->
span.setAttribute(key, value.toString()));
span.setAttribute("thread.name", Thread.currentThread().getName());
span.setAttribute("start.time", Instant.now().toString());
// Execute the operation
T result = operation.execute();
span.setStatus(StatusCode.OK);
return result;
} catch (Exception e) {
span.setStatus(StatusCode.ERROR, e.getMessage());
span.recordException(e);
throw e;
} finally {
span.end();
storageService.storeSpan(convertToSpanData(span));
}
}
public Span startSpan(String name, Map<String, Object> attributes) {
Span span = tracer.spanBuilder(name).startSpan();
attributes.forEach((key, value) -> span.setAttribute(key, value.toString()));
return span;
}
public void endSpan(Span span, SpanStatus status) {
if (span != null) {
if (status == SpanStatus.ERROR) {
span.setStatus(StatusCode.ERROR);
} else if (status == SpanStatus.OK) {
span.setStatus(StatusCode.OK);
}
span.end();
storageService.storeSpan(convertToSpanData(span));
}
}
public TraceContext createChildContext(String operationName) {
Span currentSpan = Span.current();
if (!currentSpan.getSpanContext().isValid()) {
return TraceContext.createInvalid();
}
Span childSpan = tracer.spanBuilder(operationName)
.setParent(Context.current().with(currentSpan))
.startSpan();
TraceContext context = TraceContext.fromCurrent();
childSpan.end(); // End immediately since we just needed the context
return context;
}
public Map<String, String> propagateHeaders() {
Map<String, String> headers = new HashMap<>();
tracingContext.injectContext(headers);
headers.put("x-tempo-sampled", String.valueOf(traceSampler.shouldSample()));
return headers;
}
public Context extractContextFromHeaders(Map<String, String> headers) {
return tracingContext.extractContext(headers);
}
public void recordEvent(String eventName, Map<String, Object> attributes) {
Span currentSpan = Span.current();
if (currentSpan.getSpanContext().isValid()) {
AttributesBuilder attrBuilder = Attributes.builder();
attributes.forEach((key, value) ->
attrBuilder.put(key, value.toString()));
currentSpan.addEvent(eventName, attrBuilder.build());
}
}
public void addBaggage(String key, String value) {
// Baggage would be handled through OpenTelemetry Baggage API
Baggage.current().toBuilder()
.put(key, value)
.build()
.makeCurrent();
}
public String getBaggage(String key) {
return Baggage.current().getEntryValue(key);
}
private SpanData convertToSpanData(Span span) {
// Convert OpenTelemetry Span to our domain model
SpanData spanData = new SpanData();
spanData.setName(span.getSpanContext().getTraceId());
spanData.setSpanId(span.getSpanContext().getSpanId());
spanData.setTraceId(span.getSpanContext().getTraceId());
// ... additional conversion logic
return spanData;
}
@FunctionalInterface
public interface TraceableOperation<T> {
T execute();
}
}
@Component
@Slf4j
public class TraceSampler {
private final double samplingRate;
private final Random random;
public TraceSampler(@Value("${tempo.tracing.sampling-rate:0.1}") double samplingRate) {
this.samplingRate = samplingRate;
this.random = new Random();
}
public boolean shouldSample() {
return random.nextDouble() < samplingRate;
}
public boolean shouldSample(String traceId) {
// Consistent sampling based on traceId
int hash = traceId.hashCode();
double sampleValue = (double) Math.abs(hash) / Integer.MAX_VALUE;
return sampleValue < samplingRate;
}
public boolean shouldSample(String traceId, String operationName) {
// Sample important operations more frequently
if (isImportantOperation(operationName)) {
return true;
}
return shouldSample(traceId);
}
private boolean isImportantOperation(String operationName) {
return operationName != null && (
operationName.contains("error") ||
operationName.contains("critical") ||
operationName.contains("payment") ||
operationName.contains("auth")
);
}
}
5. HTTP Request Tracing
Implement HTTP request/response tracing.
HTTP Tracing Interceptor:
@Component
@Slf4j
public class HttpTracingInterceptor {
private final TempoTracingService tracingService;
private final ObjectMapper objectMapper;
public HttpTracingInterceptor(TempoTracingService tracingService, ObjectMapper objectMapper) {
this.tracingService = tracingService;
this.objectMapper = objectMapper;
}
public <T> T executeWithTracing(HttpRequest request, HttpClient httpClient,
Class<T> responseType) {
String operationName = String.format("HTTP %s %s",
request.getMethod(), request.getURI().getPath());
return tracingService.traceOperation(operationName,
createSpanAttributes(request),
() -> executeHttpRequest(request, httpClient, responseType));
}
public void interceptRequest(HttpRequest request) {
Map<String, String> tracingHeaders = tracingService.propagateHeaders();
tracingHeaders.forEach(request::setHeader);
// Add additional tracing headers
request.setHeader("x-request-id", UUID.randomUUID().toString());
request.setHeader("x-timestamp", Instant.now().toString());
}
public void interceptResponse(HttpResponse response, Span span) {
if (response != null && span != null) {
span.setAttribute("http.status_code", response.getCode());
span.setAttribute("http.response_size",
response.getHeaders().getHeader("content-length"));
if (response.getCode() >= 400) {
span.setStatus(StatusCode.ERROR);
span.setAttribute("error", true);
}
}
}
private <T> T executeHttpRequest(HttpRequest request, HttpClient httpClient,
Class<T> responseType) {
try {
interceptRequest(request);
HttpResponse response = httpClient.execute(request);
Span currentSpan = Span.current();
interceptResponse(response, currentSpan);
if (response.getEntity() != null) {
String responseBody = EntityUtils.toString(response.getEntity());
return objectMapper.readValue(responseBody, responseType);
}
return null;
} catch (Exception e) {
Span.current().recordException(e);
Span.current().setStatus(StatusCode.ERROR, e.getMessage());
throw new RuntimeException("HTTP request failed", e);
}
}
private Map<String, Object> createSpanAttributes(HttpRequest request) {
Map<String, Object> attributes = new HashMap<>();
attributes.put("http.method", request.getMethod());
attributes.put("http.url", request.getURI().toString());
attributes.put("http.target", request.getURI().getPath());
attributes.put("http.host", request.getURI().getHost());
attributes.put("http.scheme", request.getURI().getScheme());
// Add headers as attributes (sanitized)
request.getHeaders().getHeaders().forEach((header, values) -> {
if (!isSensitiveHeader(header)) {
attributes.put("http.header." + header, String.join(",", values));
}
});
return attributes;
}
private boolean isSensitiveHeader(String headerName) {
String lowerHeader = headerName.toLowerCase();
return lowerHeader.contains("auth") ||
lowerHeader.contains("token") ||
lowerHeader.contains("password") ||
lowerHeader.contains("cookie");
}
}
@Aspect
@Component
@Slf4j
public class WebRequestTracingAspect {
private final TempoTracingService tracingService;
public WebRequestTracingAspect(TempoTracingService tracingService) {
this.tracingService = tracingService;
}
@Around("@annotation(org.springframework.web.bind.annotation.RequestMapping) || " +
"@annotation(org.springframework.web.bind.annotation.GetMapping) || " +
"@annotation(org.springframework.web.bind.annotation.PostMapping) || " +
"@annotation(org.springframework.web.bind.annotation.PutMapping) || " +
"@annotation(org.springframework.web.bind.annotation.DeleteMapping)")
public Object traceWebRequest(ProceedingJoinPoint joinPoint) throws Throwable {
String operationName = createOperationName(joinPoint);
Map<String, Object> attributes = createRequestAttributes(joinPoint);
return tracingService.traceOperation(operationName, attributes,
() -> proceedWithTracing(joinPoint));
}
private Object proceedWithTracing(ProceedingJoinPoint joinPoint) throws Throwable {
try {
Object result = joinPoint.proceed();
recordSuccessAttributes(result);
return result;
} catch (Exception e) {
recordErrorAttributes(e);
throw e;
}
}
private String createOperationName(ProceedingJoinPoint joinPoint) {
Method method = ((MethodSignature) joinPoint.getSignature()).getMethod();
String className = method.getDeclaringClass().getSimpleName();
String methodName = method.getName();
RequestMapping classMapping = method.getDeclaringClass()
.getAnnotation(RequestMapping.class);
String basePath = classMapping != null && classMapping.value().length > 0 ?
classMapping.value()[0] : "";
return String.format("HTTP %s%s.%s", basePath, className, methodName);
}
private Map<String, Object> createRequestAttributes(ProceedingJoinPoint joinPoint) {
Map<String, Object> attributes = new HashMap<>();
// Extract method information
Method method = ((MethodSignature) joinPoint.getSignature()).getMethod();
attributes.put("method.name", method.getName());
attributes.put("class.name", method.getDeclaringClass().getName());
// Extract request information if available
Object[] args = joinPoint.getArgs();
for (Object arg : args) {
if (arg instanceof HttpServletRequest) {
extractHttpRequestAttributes((HttpServletRequest) arg, attributes);
}
}
return attributes;
}
private void extractHttpRequestAttributes(HttpServletRequest request,
Map<String, Object> attributes) {
attributes.put("http.method", request.getMethod());
attributes.put("http.url", request.getRequestURL().toString());
attributes.put("http.uri", request.getRequestURI());
attributes.put("http.query", request.getQueryString());
attributes.put("http.remote_addr", request.getRemoteAddr());
attributes.put("http.user_agent", request.getHeader("User-Agent"));
// Add headers (sanitized)
Collections.list(request.getHeaderNames()).forEach(headerName -> {
if (!isSensitiveHeader(headerName)) {
attributes.put("http.header." + headerName, request.getHeader(headerName));
}
});
}
private void recordSuccessAttributes(Object result) {
if (result instanceof ResponseEntity) {
ResponseEntity<?> response = (ResponseEntity<?>) result;
Span.current().setAttribute("http.status_code", response.getStatusCodeValue());
}
}
private void recordErrorAttributes(Exception e) {
Span.current().recordException(e);
Span.current().setAttribute("error.type", e.getClass().getName());
Span.current().setAttribute("error.message", e.getMessage());
}
private boolean isSensitiveHeader(String headerName) {
String lowerHeader = headerName.toLowerCase();
return lowerHeader.contains("auth") ||
lowerHeader.contains("token") ||
lowerHeader.contains("password") ||
lowerHeader.contains("cookie");
}
}
6. Database Query Tracing
Implement database operation tracing.
Database Tracing:
@Aspect
@Component
@Slf4j
public class DatabaseTracingAspect {
private final TempoTracingService tracingService;
public DatabaseTracingAspect(TempoTracingService tracingService) {
this.tracingService = tracingService;
}
@Around("execution(* org.springframework.data.repository.Repository+.*(..))")
public Object traceRepositoryMethod(ProceedingJoinPoint joinPoint) throws Throwable {
String operationName = createRepositoryOperationName(joinPoint);
Map<String, Object> attributes = createRepositoryAttributes(joinPoint);
return tracingService.traceOperation(operationName, attributes,
() -> proceedWithDatabaseTracing(joinPoint));
}
@Around("execution(* javax.sql.DataSource.getConnection(..))")
public Object traceConnection(ProceedingJoinPoint joinPoint) throws Throwable {
return tracingService.traceOperation("DataSource.getConnection",
Collections.emptyMap(), joinPoint::proceed);
}
private Object proceedWithDatabaseTracing(ProceedingJoinPoint joinPoint) throws Throwable {
long startTime = System.nanoTime();
try {
Object result = joinPoint.proceed();
long duration = System.nanoTime() - startTime;
Span.current().setAttribute("db.duration_ns", duration);
recordResultAttributes(result);
return result;
} catch (Exception e) {
long duration = System.nanoTime() - startTime;
Span.current().setAttribute("db.duration_ns", duration);
recordErrorAttributes(e);
throw e;
}
}
private String createRepositoryOperationName(ProceedingJoinPoint joinPoint) {
Method method = ((MethodSignature) joinPoint.getSignature()).getMethod();
String repositoryName = method.getDeclaringClass().getSimpleName();
String methodName = method.getName();
return String.format("DB %s.%s", repositoryName, methodName);
}
private Map<String, Object> createRepositoryAttributes(ProceedingJoinPoint joinPoint) {
Map<String, Object> attributes = new HashMap<>();
Method method = ((MethodSignature) joinPoint.getSignature()).getMethod();
attributes.put("db.operation", method.getName());
attributes.put("db.repository", method.getDeclaringClass().getSimpleName());
// Add query parameters (limited to avoid too much data)
Object[] args = joinPoint.getArgs();
if (args.length > 0) {
attributes.put("db.parameter_count", args.length);
for (int i = 0; i < Math.min(args.length, 5); i++) {
attributes.put("db.param." + i, safeToString(args[i]));
}
}
return attributes;
}
private void recordResultAttributes(Object result) {
if (result instanceof List) {
List<?> listResult = (List<?>) result;
Span.current().setAttribute("db.result_count", listResult.size());
} else if (result != null) {
Span.current().setAttribute("db.result_type", result.getClass().getSimpleName());
}
}
private void recordErrorAttributes(Exception e) {
Span.current().recordException(e);
Span.current().setAttribute("db.error", true);
Span.current().setAttribute("db.error_type", e.getClass().getSimpleName());
}
private String safeToString(Object obj) {
if (obj == null) return "null";
try {
return obj.toString();
} catch (Exception e) {
return "[unable to stringify]";
}
}
}
@Component
public class JdbcTracingWrapper {
private final TempoTracingService tracingService;
public JdbcTracingWrapper(TempoTracingService tracingService) {
this.tracingService = tracingService;
}
public Connection wrapConnection(Connection connection, String dataSourceName) {
return (Connection) Proxy.newProxyInstance(
connection.getClass().getClassLoader(),
new Class[]{Connection.class},
new TracingInvocationHandler(connection, dataSourceName, tracingService)
);
}
private static class TracingInvocationHandler implements InvocationHandler {
private final Connection delegate;
private final String dataSourceName;
private final TempoTracingService tracingService;
public TracingInvocationHandler(Connection delegate, String dataSourceName,
TempoTracingService tracingService) {
this.delegate = delegate;
this.dataSourceName = dataSourceName;
this.tracingService = tracingService;
}
@Override
public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
String methodName = method.getName();
if ("prepareStatement".equals(methodName) || "createStatement".equals(methodName)) {
return traceStatementCreation(method, args);
} else if ("close".equals(methodName)) {
return traceClose(method, args);
}
return method.invoke(delegate, args);
}
private Object traceStatementCreation(Method method, Object[] args) throws Throwable {
String operationName = String.format("DB Connection.%s", method.getName());
return tracingService.traceOperation(operationName,
Map.of(
"db.connection.operation", method.getName(),
"db.data_source", dataSourceName
),
() -> method.invoke(delegate, args));
}
private Object traceClose(Method method, Object[] args) throws Throwable {
String operationName = "DB Connection.close";
return tracingService.traceOperation(operationName,
Map.of("db.data_source", dataSourceName),
() -> method.invoke(delegate, args));
}
}
}
7. Trace Storage and Query Service
Implement trace storage and query capabilities.
Trace Storage Service:
@Service
@Slf4j
public class TraceStorageService {
private final Map<String, List<SpanData>> traceStorage = new ConcurrentHashMap<>();
private final List<TraceResult> traceCache = new CopyOnWriteArrayList<>();
private final long maxStorageDurationHours = 24;
public void storeSpan(SpanData spanData) {
String traceId = spanData.getTraceId();
traceStorage.compute(traceId, (key, existingSpans) -> {
if (existingSpans == null) {
existingSpans = new CopyOnWriteArrayList<>();
}
existingSpans.add(spanData);
return existingSpans;
});
cleanupOldTraces();
}
public Optional<TraceResult> getTrace(String traceId) {
List<SpanData> spans = traceStorage.get(traceId);
if (spans == null || spans.isEmpty()) {
return Optional.empty();
}
TraceResult result = buildTraceResult(traceId, spans);
return Optional.of(result);
}
public List<TraceResult> searchTraces(TraceQuery query) {
return traceStorage.entrySet().stream()
.filter(entry -> matchesQuery(entry.getKey(), entry.getValue(), query))
.map(entry -> buildTraceResult(entry.getKey(), entry.getValue()))
.limit(query.getLimit())
.collect(Collectors.toList());
}
public Map<String, Object> getTraceStatistics(Duration timeRange) {
Instant cutoff = Instant.now().minus(timeRange);
List<SpanData> recentSpans = traceStorage.values().stream()
.flatMap(List::stream)
.filter(span -> span.getStartTime() >= cutoff.toEpochMilli())
.collect(Collectors.toList());
Map<String, Object> stats = new HashMap<>();
stats.put("total_traces", traceStorage.size());
stats.put("total_spans", recentSpans.size());
stats.put("time_range", timeRange.toString());
// Calculate average duration
Double averageDuration = recentSpans.stream()
.mapToLong(span -> span.getEndTime() - span.getStartTime())
.average()
.orElse(0.0);
stats.put("average_duration_ms", averageDuration / 1_000_000);
// Error rate
long errorCount = recentSpans.stream()
.filter(SpanData::isError)
.count();
stats.put("error_rate", recentSpans.isEmpty() ? 0 : (double) errorCount / recentSpans.size());
// Top operations
Map<String, Long> operationCounts = recentSpans.stream()
.collect(Collectors.groupingBy(
SpanData::getName,
Collectors.counting()
));
stats.put("top_operations", operationCounts.entrySet().stream()
.sorted(Map.Entry.<String, Long>comparingByValue().reversed())
.limit(10)
.collect(Collectors.toList()));
return stats;
}
private boolean matchesQuery(String traceId, List<SpanData> spans, TraceQuery query) {
if (query.getTraceId() != null && !query.getTraceId().equals(traceId)) {
return false;
}
if (query.getServiceName() != null || query.getOperationName() != null) {
boolean matchesService = spans.stream()
.anyMatch(span -> matchesServiceAndOperation(span, query));
if (!matchesService) return false;
}
if (query.getMinDuration() != null || query.getMaxDuration() != null) {
Duration traceDuration = calculateTraceDuration(spans);
if (query.getMinDuration() != null && traceDuration.compareTo(query.getMinDuration()) < 0) {
return false;
}
if (query.getMaxDuration() != null && traceDuration.compareTo(query.getMaxDuration()) > 0) {
return false;
}
}
if (query.getTags() != null && !query.getTags().isEmpty()) {
boolean matchesTags = spans.stream()
.anyMatch(span -> matchesTags(span, query.getTags()));
if (!matchesTags) return false;
}
return true;
}
private boolean matchesServiceAndOperation(SpanData span, TraceQuery query) {
String serviceName = span.getAttributes() != null ?
(String) span.getAttributes().get("service.name") : null;
String operationName = span.getName();
boolean serviceMatch = query.getServiceName() == null ||
query.getServiceName().equals(serviceName);
boolean operationMatch = query.getOperationName() == null ||
span.getName().contains(query.getOperationName());
return serviceMatch && operationMatch;
}
private boolean matchesTags(SpanData span, Map<String, String> tags) {
if (span.getAttributes() == null) return false;
return tags.entrySet().stream()
.allMatch(entry -> {
Object value = span.getAttributes().get(entry.getKey());
return value != null && value.toString().equals(entry.getValue());
});
}
private Duration calculateTraceDuration(List<SpanData> spans) {
long start = spans.stream()
.mapToLong(SpanData::getStartTime)
.min()
.orElse(0);
long end = spans.stream()
.mapToLong(SpanData::getEndTime)
.max()
.orElse(0);
return Duration.ofNanos(end - start);
}
private TraceResult buildTraceResult(String traceId, List<SpanData> spans) {
TraceResult result = new TraceResult();
result.setTraceId(traceId);
result.setSpans(spans);
result.setTotalSpans(spans.size());
result.setTotalDuration(calculateTraceDuration(spans));
result.setHasErrors(spans.stream().anyMatch(SpanData::isError));
result.setSummary(calculateTraceSummary(spans));
// Find root service
spans.stream()
.filter(span -> span.getParentSpanId() == null)
.findFirst()
.ifPresent(rootSpan -> {
String serviceName = rootSpan.getAttributes() != null ?
(String) rootSpan.getAttributes().get("service.name") : "unknown";
result.setRootService(serviceName);
});
return result;
}
private Map<String, Object> calculateTraceSummary(List<SpanData> spans) {
Map<String, Object> summary = new HashMap<>();
summary.put("span_count", spans.size());
summary.put("error_count", spans.stream().filter(SpanData::isError).count());
summary.put("service_count", spans.stream()
.map(span -> span.getAttributes() != null ?
span.getAttributes().get("service.name") : "unknown")
.distinct()
.count());
// Duration statistics
DoubleSummaryStatistics durationStats = spans.stream()
.mapToDouble(span -> (span.getEndTime() - span.getStartTime()) / 1_000_000.0)
.summaryStatistics();
summary.put("min_duration_ms", durationStats.getMin());
summary.put("max_duration_ms", durationStats.getMax());
summary.put("avg_duration_ms", durationStats.getAverage());
return summary;
}
private void cleanupOldTraces() {
long cutoff = Instant.now().minus(maxStorageDurationHours, ChronoUnit.HOURS)
.toEpochMilli();
traceStorage.entrySet().removeIf(entry ->
entry.getValue().stream()
.mapToLong(SpanData::getStartTime)
.max()
.orElse(Long.MAX_VALUE) < cutoff
);
}
}
8. REST API for Trace Management
Expose tracing functionality as web services.
Trace Controller:
@RestController
@RequestMapping("/api/tracing")
@Slf4j
public class TraceController {
private final TempoTracingService tracingService;
private final TraceStorageService storageService;
public TraceController(TempoTracingService tracingService,
TraceStorageService storageService) {
this.tracingService = tracingService;
this.storageService = storageService;
}
@PostMapping("/trace")
public ResponseEntity<TraceResponse> startTrace(@RequestBody TraceRequest request) {
try {
Span span = tracingService.startSpan(request.getOperationName(),
request.getAttributes());
TraceResponse response = new TraceResponse();
response.setTraceId(span.getSpanContext().getTraceId());
response.setSpanId(span.getSpanContext().getSpanId());
response.setSampled(span.getSpanContext().isSampled());
return ResponseEntity.ok(response);
} catch (Exception e) {
log.error("Failed to start trace", e);
return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).build();
}
}
@GetMapping("/trace/{traceId}")
public ResponseEntity<TraceResult> getTrace(@PathVariable String traceId) {
Optional<TraceResult> trace = storageService.getTrace(traceId);
return trace.map(ResponseEntity::ok)
.orElse(ResponseEntity.notFound().build());
}
@PostMapping("/search")
public ResponseEntity<List<TraceResult>> searchTraces(@RequestBody TraceQuery query) {
if (!query.isValid()) {
return ResponseEntity.badRequest().build();
}
List<TraceResult> results = storageService.searchTraces(query);
return ResponseEntity.ok(results);
}
@GetMapping("/statistics")
public ResponseEntity<Map<String, Object>> getStatistics(
@RequestParam(defaultValue = "PT1H") Duration timeRange) {
Map<String, Object> stats = storageService.getTraceStatistics(timeRange);
return ResponseEntity.ok(stats);
}
@GetMapping("/context/current")
public ResponseEntity<TraceContext> getCurrentContext() {
TraceContext context = TraceContext.fromCurrent();
return ResponseEntity.ok(context);
}
@PostMapping("/context/propagate")
public ResponseEntity<Map<String, String>> propagateContext() {
Map<String, String> headers = tracingService.propagateHeaders();
return ResponseEntity.ok(headers);
}
@PostMapping("/custom-span")
public ResponseEntity<String> createCustomSpan(@RequestBody CustomSpanRequest request) {
try {
tracingService.traceOperation(request.getOperationName(),
request.getAttributes(),
() -> {
// Custom operation logic would go here
log.info("Executing custom operation: {}", request.getOperationName());
return null;
});
return ResponseEntity.ok("Span created successfully");
} catch (Exception e) {
return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR)
.body("Failed to create span: " + e.getMessage());
}
}
}
@Data
class CustomSpanRequest {
private String operationName;
private Map<String, Object> attributes;
private Map<String, String> baggage;
}
Best Practices for Tempo Tracing
- Sampling Strategy: Implement appropriate sampling to balance visibility and performance
- Context Propagation: Ensure trace context is properly propagated across service boundaries
- Attribute Management: Be mindful of attribute cardinality and sensitive data
- Performance Impact: Monitor tracing overhead and adjust sampling rates accordingly
- Error Handling: Implement robust error handling for tracing infrastructure failures
- Storage Management: Implement trace retention policies and cleanup mechanisms
- Security: Sanitize sensitive data from spans and attributes
Conclusion: Comprehensive Distributed Tracing
Tempo tracing in Java provides a powerful foundation for understanding system behavior across distributed architectures. By implementing comprehensive tracing with context propagation, span creation, and trace analysis, developers can gain deep insights into application performance, identify bottlenecks, and debug complex issues in microservices environments.
This implementation demonstrates that effective tracing isn't just about collecting data—it's about creating a cohesive observability story that connects logs, metrics, and traces to provide actionable insights for maintaining and improving system reliability and performance.
Java Observability, Logging Intelligence & AI-Driven Monitoring (APM, Tracing, Logs & Anomaly Detection)
https://macronepal.com/blog/beyond-metrics-observing-serverless-and-traditional-java-applications-with-thundra-apm/
Explains using Thundra APM to observe both serverless and traditional Java applications by combining tracing, metrics, and logs into a unified observability platform for faster debugging and performance insights.
https://macronepal.com/blog/dynatrace-oneagent-in-java-2/
Explains Dynatrace OneAgent for Java, which automatically instruments JVM applications to capture metrics, traces, and logs, enabling full-stack monitoring and root-cause analysis with minimal configuration.
https://macronepal.com/blog/lightstep-java-sdk-distributed-tracing-and-observability-implementation/
Explains Lightstep Java SDK for distributed tracing, helping developers track requests across microservices and identify latency issues using OpenTelemetry-based observability.
https://macronepal.com/blog/honeycomb-io-beeline-for-java-complete-guide-2/
Explains Honeycomb Beeline for Java, which provides high-cardinality observability and deep query capabilities to understand complex system behavior and debug distributed systems efficiently.
https://macronepal.com/blog/lumigo-for-serverless-in-java-complete-distributed-tracing-guide-2/
Explains Lumigo for Java serverless applications, offering automatic distributed tracing, log correlation, and error tracking to simplify debugging in cloud-native environments. (Lumigo Docs)
https://macronepal.com/blog/from-noise-to-signals-implementing-log-anomaly-detection-in-java-applications/
Explains how to detect anomalies in Java logs using behavioral patterns and machine learning techniques to separate meaningful incidents from noisy log data and improve incident response.
https://macronepal.com/blog/ai-powered-log-analysis-in-java-from-reactive-debugging-to-proactive-insights/
Explains AI-driven log analysis for Java applications, shifting from manual debugging to predictive insights that identify issues early and improve system reliability using intelligent log processing.
https://macronepal.com/blog/titliel-java-logging-best-practices/
Explains best practices for Java logging, focusing on structured logs, proper log levels, performance optimization, and ensuring logs are useful for debugging and observability systems.
https://macronepal.com/blog/seeking-a-loguru-for-java-the-quest-for-elegant-and-simple-logging/
Explains the search for simpler, more elegant logging frameworks in Java, comparing modern logging approaches that aim to reduce complexity while improving readability and developer experience.