In modern microservices architectures, a single user request can traverse dozens of services, making debugging and performance monitoring incredibly challenging. Distributed tracing provides the solution by giving you complete visibility into the flow of requests across service boundaries. This article explores how to implement distributed tracing in Java using Spring Cloud Sleuth for trace instrumentation and Zipkin for visualization and analysis.
The Challenge: Debugging in Microservices
Consider a simple e-commerce request flow:
User Request → API Gateway → Auth Service → Product Service → Order Service → Payment Service → Notification Service
Without distributed tracing:
- No correlation between logs across services
- Impossible to track the entire request journey
- Difficult to identify performance bottlenecks
- Debugging requires manual log correlation across multiple systems
With distributed tracing:
- Complete visibility of the request flow
- Correlation IDs that span all services
- Performance metrics for each service call
- Visual representation of service dependencies
Core Concepts: Trace, Span, and Annotations
Trace: A complete request journey from start to end
- Contains a unique Trace ID
- Represents the entire workflow
Span: A single operation within a trace
- Contains a unique Span ID
- Has a parent span (except the root span)
- Represents work done by a single service
Annotations: Key events within a span
cs(Client Sent): When a request is initiatedsr(Server Received): When a request is receivedss(Server Sent): When a response is sentcr(Client Received): When a response is received
Visual Representation:
Trace: 5aaf3e (entire request) ├── Span A: API Gateway (root) │ ├── cs: Gateway starts request │ └── cr: Gateway receives response ├── Span B: Auth Service (child of A) │ ├── sr: Auth receives request │ └── ss: Auth sends response └── Span C: Product Service (child of A) ├── sr: Product receives request └── ss: Product sends response
Spring Cloud Sleuth: Automatic Instrumentation
Spring Cloud Sleuth automatically instruments your Spring applications to add trace and span IDs to logs and propagate them across service boundaries.
Key Features:
- Automatic context propagation via HTTP headers
- Integration with Spring ecosystem (Web, Feign, Messaging, etc.)
- Log correlation with trace/span IDs
- Multiple sampler strategies for controlling trace volume
Project Setup and Dependencies
Maven Dependencies:
<!-- Spring Boot Starter --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <!-- Spring Cloud Sleuth --> <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-sleuth</artifactId> <version>3.1.9</version> </dependency> <!-- Zipkin Integration (optional) --> <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-sleuth-zipkin</artifactId> <version>3.1.9</version> </dependency> <!-- For WebClient support --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-webflux</artifactId> </dependency> <!-- For Feign Client support --> <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-openfeign</artifactId> </dependency>
Gradle Dependencies:
dependencies {
implementation 'org.springframework.boot:spring-boot-starter-web'
implementation 'org.springframework.cloud:spring-cloud-starter-sleuth:3.1.9'
implementation 'org.springframework.cloud:spring-cloud-sleuth-zipkin:3.1.9'
implementation 'org.springframework.boot:spring-boot-starter-webflux'
}
Basic Sleuth Configuration
application.yml:
spring:
application:
name: product-service # Service name appears in traces
sleuth:
sampler:
probability: 1.0 # Sample 100% of requests (1.0 = 100%)
# Optional: Custom configuration
web:
enabled: true
feign:
enabled: true
messaging:
enabled: true
# Zipkin configuration (if using Zipkin)
zipkin:
base-url: http://localhost:9411
sender:
type: web # Options: web, rabbit, kafka
logging:
pattern:
level: "%5p [${spring.application.name:},%X{traceId:-},%X{spanId:-}]"
Sampler Configuration Options:
@Configuration
public class SleuthConfig {
// Sample all requests (for development)
@Bean
public Sampler alwaysSampler() {
return Sampler.ALWAYS_SAMPLE;
}
// Sample based on probability (for production)
@Bean
public Sampler probabilitySampler() {
return ProbabilityBasedSampler.create(0.5f); // 50% sampling
}
// Custom sampler based on request attributes
@Bean
public Sampler customSampler() {
return new Sampler() {
@Override
public boolean isSampled(TraceContext traceContext) {
// Custom sampling logic
return Math.random() < 0.1; // 10% sampling
}
};
}
}
Service Implementation Examples
Product Service:
@SpringBootApplication
@RestController
@Slf4j
public class ProductServiceApplication {
public static void main(String[] args) {
SpringApplication.run(ProductServiceApplication.class, args);
}
@GetMapping("/products/{id}")
public Product getProduct(@PathVariable String id) {
log.info("Fetching product with ID: {}", id);
// Simulate business logic
if ("999".equals(id)) {
log.warn("Product {} not found", id);
throw new ProductNotFoundException("Product not found: " + id);
}
// Simulate database call
Product product = findProductInDatabase(id);
log.debug("Found product: {}", product.getName());
return product;
}
private Product findProductInDatabase(String id) {
// Simulate database latency
try {
Thread.sleep(100);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
return new Product(id, "Sample Product", 29.99);
}
}
class Product {
private String id;
private String name;
private double price;
// Constructors, getters, setters
public Product(String id, String name, double price) {
this.id = id;
this.name = name;
this.price = price;
}
// Getters and setters...
}
Order Service (with Feign Client):
@SpringBootApplication
@RestController
@EnableFeignClients
@Slf4j
public class OrderServiceApplication {
@Autowired
private ProductServiceClient productServiceClient;
@Autowired
private WebClient.Builder webClientBuilder;
public static void main(String[] args) {
SpringApplication.run(OrderServiceApplication.class, args);
}
@PostMapping("/orders")
public Order createOrder(@RequestBody OrderRequest request) {
log.info("Creating order for user: {}", request.getUserId());
// Method 1: Using Feign Client (automatically propagates trace headers)
Product product = productServiceClient.getProduct(request.getProductId());
log.debug("Retrieved product: {}", product.getName());
// Method 2: Using WebClient (manual header propagation)
Product product2 = webClientBuilder.build()
.get()
.uri("http://localhost:8081/products/{id}", request.getProductId())
.retrieve()
.bodyToMono(Product.class)
.block();
// Create order logic
Order order = new Order(UUID.randomUUID().toString(), request.getUserId(), product);
log.info("Order created: {}", order.getId());
return order;
}
}
// Feign Client interface
@FeignClient(name = "product-service", url = "http://localhost:8081")
interface ProductServiceClient {
@GetMapping("/products/{id}")
Product getProduct(@PathVariable String id);
}
class OrderRequest {
private String userId;
private String productId;
// Getters and setters...
}
class Order {
private String id;
private String userId;
private Product product;
private LocalDateTime createdAt;
public Order(String id, String userId, Product product) {
this.id = id;
this.userId = userId;
this.product = product;
this.createdAt = LocalDateTime.now();
}
// Getters and setters...
}
Manual Tracing with @NewSpan and @ContinueSpan
For custom business operations, you can manually create spans:
@Service
@Slf4j
public class OrderProcessingService {
private final Tracer tracer;
public OrderProcessingService(Tracer tracer) {
this.tracer = tracer;
}
// Automatically creates a new span
@NewSpan("process-payment")
public PaymentResult processPayment(Order order) {
log.info("Processing payment for order: {}", order.getId());
// Custom business logic
boolean success = chargeCreditCard(order);
// Add custom tags to the span
tracer.currentSpan().tag("order.amount", String.valueOf(order.getTotalAmount()));
tracer.currentSpan().tag("payment.success", String.valueOf(success));
if (!success) {
tracer.currentSpan().tag("error", true);
log.error("Payment failed for order: {}", order.getId());
}
return new PaymentResult(success, success ? "Payment processed" : "Payment failed");
}
// Continues existing span with custom name
@ContinueSpan(log = "validate-inventory")
public boolean validateInventory(Order order) {
log.debug("Validating inventory for product: {}", order.getProductId());
// Manual span creation for specific operations
Span inventorySpan = tracer.nextSpan().name("check-stock-level").start();
try (SpanInScope ws = tracer.withSpan(inventorySpan)) {
// Complex inventory checking logic
int stockLevel = checkStockLevel(order.getProductId());
inventorySpan.tag("stock.level", String.valueOf(stockLevel));
return stockLevel >= order.getQuantity();
} finally {
inventorySpan.end();
}
}
// Manual tracing without annotations
public void shipOrder(Order order) {
Span shippingSpan = tracer.nextSpan().name("ship-order").start();
try (SpanInScope ws = tracer.withSpan(shippingSpan)) {
log.info("Shipping order: {}", order.getId());
// Add baggage (custom context that propagates to downstream services)
shippingSpan.tag("shipping.carrier", "UPS");
shippingSpan.tag("order.weight", "2.5kg");
// Simulate shipping logic
arrangeShipping(order);
} catch (Exception e) {
shippingSpan.error(e); // Mark span as error
log.error("Shipping failed for order: {}", order.getId(), e);
throw e;
} finally {
shippingSpan.end();
}
}
private boolean chargeCreditCard(Order order) {
// Simulate payment processing
return Math.random() > 0.1; // 90% success rate
}
private int checkStockLevel(String productId) {
// Simulate stock check
return (int) (Math.random() * 100);
}
private void arrangeShipping(Order order) {
// Simulate shipping arrangement
try {
Thread.sleep(200);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
}
Setting Up Zipkin for Visualization
Running Zipkin with Docker:
# Quick start with Docker docker run -d -p 9411:9411 --name zipkin openzipkin/zipkin # With persistence (MySQL) docker run -d -p 9411:9411 \ -e STORAGE_TYPE=mysql \ -e MYSQL_HOST=localhost \ -e MYSQL_TCP_PORT=3306 \ -e MYSQL_DB=zipkin \ -e MYSQL_USER=zipkin \ -e MYSQL_PASS=zipkin \ --name zipkin openzipkin/zipkin # With RabbitMQ integration docker run -d -p 9411:9411 \ -e RABBIT_URI=amqp://localhost \ --name zipkin openzipkin/zipkin
Alternative: Using Java
# Download and run curl -sSL https://zipkin.io/quickstart.sh | bash -s java -jar zipkin.jar
Zipkin Configuration:
spring: zipkin: base-url: http://localhost:9411 sender: type: web # Options: web, rabbit, kafka # Optional: Service name mapping service: name: product-service # Connection settings connect-timeout: 5s read-timeout: 10s # Alternative: Using message broker (better for production) rabbitmq: host: localhost port: 5672 username: guest password: guest # For Kafka transport spring: zipkin: sender: type: kafka kafka: bootstrap-servers: localhost:9092
Analyzing Traces in Zipkin
Key Zipkin Features:
- Service Dependency Diagram: Visual map of service interactions
- Trace Timeline: Detailed view of request flow with timing
- Span Details: Deep dive into individual operations
- Error Identification: Quickly spot failed requests
- Performance Analysis: Identify slow services and bottlenecks
Example Trace Analysis:
Order Creation Request (Trace ID: 7d3e4a) ├── API Gateway (15ms) ├── Auth Service (8ms) ├── Product Service (45ms) ← BOTTLENECK! │ └── Database Query (40ms) ├── Inventory Service (12ms) └── Payment Service (25ms)
Custom Zipkin Queries:
serviceName:product-service and duration>100ms- Slow product service callshttp.path:/products/999 and error- Failed product requestsspanName:process-payment- Custom business spans
Advanced Configuration and Best Practices
Production Configuration:
spring:
sleuth:
sampler:
probability: 0.1 # Sample 10% in production
# Baggage configuration
baggage:
remote-fields: version,user-id,country-code
correlation:
enabled: true
fields: user-id,country-code
# Propagation options
propagation:
type: B3 # Options: B3, W3C (Trace Context)
zipkin:
base-url: ${ZIPKIN_URL:http://zipkin-prod:9411}
sender:
type: kafka # Use message broker in production
logging:
level:
org.springframework.cloud.sleuth: INFO
brave: WARN
Custom Trace Configuration:
@Configuration
@Slf4j
public class TracingConfiguration {
@Bean
public CurrentTraceContext currentTraceContext() {
return CurrentTraceContext.newBuilder()
.addScopeDecorator(new MDCScopeDecorator())
.build();
}
@Bean
public Tracing tracing(Sampler sampler) {
return Tracing.newBuilder()
.localServiceName("order-service")
.sampler(sampler)
.currentTraceContext(currentTraceContext())
.addSpanHandler(new LoggingSpanHandler())
.build();
}
// Custom span handler for additional processing
@Bean
public SpanHandler customSpanHandler() {
return new SpanHandler() {
@Override
public boolean end(TraceContext context, MutableSpan span, Cause cause) {
// Add custom logic before span is exported
if (span.error() != null) {
log.warn("Span {} ended with error: {}", span.name(), span.error());
}
// Add custom tags to all spans
span.tag("deployment.env", System.getenv("DEPLOYMENT_ENV"));
span.tag("application.version", "1.0.0");
return true; // Process this span
}
};
}
}
// Custom span handler for logging
class LoggingSpanHandler extends SpanHandler {
private static final Logger log = LoggerFactory.getLogger(LoggingSpanHandler.class);
@Override
public boolean end(TraceContext context, MutableSpan span, Cause cause) {
log.debug("Span completed: {} (Duration: {}ms)",
span.name(), span.finishTimestamp() - span.startTimestamp());
return true;
}
}
Testing Distributed Tracing
Unit Test Example:
@SpringBootTest
@AutoConfigureTestDatabase
class OrderServiceTracingTest {
@Autowired
private TestRestTemplate restTemplate;
@Autowired
private Tracer tracer;
@Test
void whenCreateOrder_thenTraceIdPropagated() {
// Given
OrderRequest request = new OrderRequest("user-123", "product-456");
// When
ResponseEntity<Order> response = restTemplate.postForEntity(
"/orders", request, Order.class);
// Then - verify response contains trace headers
assertThat(response.getHeaders())
.containsKey("X-B3-TraceId")
.containsKey("X-B3-SpanId");
}
@Test
void whenManualSpanCreated_thenAppearsInTracer() {
// Given
Span span = tracer.nextSpan().name("test-operation").start();
// When
try (SpanInScope ws = tracer.withSpan(span)) {
// Perform operation
log.info("Testing manual span");
// Then
assertThat(tracer.currentSpan()).isNotNull();
assertThat(tracer.currentSpan().context().spanId())
.isEqualTo(span.context().spanId());
} finally {
span.end();
}
}
}
Integration Test:
@Testcontainers
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
class DistributedTracingIntegrationTest {
@Container
static GenericContainer<?> zipkin = new GenericContainer<>("openzipkin/zipkin")
.withExposedPorts(9411);
@Test
void whenRequestFlowsThroughServices_thenCompleteTraceInZipkin() throws Exception {
// Given
String traceId = performDistributedRequest();
// When - wait for trace to appear in Zipkin
Thread.sleep(2000);
// Then - verify trace exists in Zipkin
String zipkinUrl = String.format("http://%s:%d/api/v2/trace/%s",
zipkin.getHost(), zipkin.getFirstMappedPort(), traceId);
ResponseEntity<String> response = restTemplate.getForEntity(zipkinUrl, String.class);
assertThat(response.getStatusCode()).isEqualTo(HttpStatus.OK);
assertThat(response.getBody()).contains(traceId);
}
private String performDistributedRequest() {
// Implementation that triggers multi-service call
return "mock-trace-id";
}
}
Troubleshooting Common Issues
Problem: Traces not appearing in Zipkin
# Solution: Check configuration spring: sleuth: enabled: true sampler: probability: 1.0 # Ensure sampling is enabled zipkin: base-url: http://localhost:9411 # Correct Zipkin URL enabled: true
Problem: Headers not propagating
// Solution: Manual propagation for non-Sleuth clients
@Bean
public WebClient webClient(Tracer tracer) {
return WebClient.builder()
.filter((request, next) -> {
Span currentSpan = tracer.currentSpan();
if (currentSpan != null) {
// Manually inject headers
request.headers().set("X-B3-TraceId", currentSpan.context().traceIdString());
request.headers().set("X-B3-SpanId", currentSpan.context().spanIdString());
}
return next.exchange(request);
})
.build();
}
Best Practices for Production
- Sampling Strategy: Use probability sampling (0.01-0.1) in production
- Storage Backend: Use scalable storage (Elasticsearch, Cassandra) for Zipkin
- Transport: Prefer message brokers (Kafka, RabbitMQ) over direct HTTP
- Error Handling: Implement fallback mechanisms for tracing failures
- Security: Secure Zipkin endpoint and consider data sensitivity
- Monitoring: Monitor tracing system health and performance
# Production-ready configuration spring: sleuth: sampler: probability: 0.05 # 5% sampling rate zipkin: sender: type: kafka kafka: topic: zipkin kafka: bootstrap-servers: kafka-cluster:9092 management: endpoints: web: exposure: include: health,metrics,prometheus
Conclusion
Spring Cloud Sleuth and Zipkin provide a powerful combination for distributed tracing in Java microservices:
Key Benefits:
- End-to-End Visibility: Track requests across service boundaries
- Performance Insights: Identify bottlenecks and optimize performance
- Debugging Efficiency: Quickly pinpoint failures in complex workflows
- Dependency Analysis: Understand service relationships and impacts
Implementation Strategy:
- Start with automatic Sleuth instrumentation
- Add Zipkin for visualization and analysis
- Implement manual spans for critical business operations
- Configure appropriate sampling for production
- Monitor and optimize based on actual usage patterns
By implementing distributed tracing, you transform from blind debugging to systematic observability, making your microservices architecture truly manageable and maintainable at scale.