In a distributed system, failures are inevitable. While Resilience4j provides powerful patterns like Circuit Breakers, Rate Limiters, and Retries to make your application resilient, you need visibility into how these patterns are performing. Prometheus, combined with Grafana, offers the perfect observability stack to monitor these resilience patterns in real-time.
This article provides a comprehensive guide to exposing Resilience4j metrics to Prometheus, creating actionable dashboards, and leveraging these metrics for operational excellence.
Why Monitor Resilience4j Patterns?
Each Resilience4j pattern provides crucial metrics that indicate the health of your service interactions:
- Circuit Breaker: Failure rates, state transitions, call counts
- Rate Limiter: Available permissions, wait times
- Retry: Retry attempts, successful/failed retries
- Bulkhead: Concurrent calls, queue depths
- Time Limiter: Timeout occurrences, duration metrics
Without proper monitoring, you're operating blind to how your resilience patterns are performing under load.
Architecture Overview
Java Application → Resilience4j → Micrometer Metrics → Prometheus Registry → /actuator/prometheus ↓ Prometheus Server (scrapes metrics) ↓ Grafana Dashboard (visualizes metrics)
Implementation Guide
1. Dependencies Setup
First, add the necessary dependencies to your pom.xml:
<dependencies> <!-- Spring Boot Starter --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-actuator</artifactId> </dependency> <!-- Resilience4j Core --> <dependency> <groupId>io.github.resilience4j</groupId> <artifactId>resilience4j-spring-boot3</artifactId> <version>2.1.0</version> </dependency> <dependency> <groupId>io.github.resilience4j</groupId> <artifactId>resilience4j-micrometer</artifactId> <version>2.1.0</version> </dependency> <!-- Micrometer Prometheus --> <dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-registry-prometheus</artifactId> </dependency> <!-- AspectJ for Resilience4j Annotations --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-aop</artifactId> </dependency> </dependencies>
2. Application Configuration
Configure your application.yml to expose metrics:
spring: application: name: resilience4j-demo management: endpoints: web: exposure: include: health,info,prometheus,metrics endpoint: prometheus: enabled: true health: show-details: always metrics: export: prometheus: enabled: true distribution: percentiles-histogram: http.server.requests: true tags: application: ${spring.application.name} resilience4j: circuitbreaker: instances: backendService: register-health-indicator: true sliding-window-size: 10 failure-rate-threshold: 50 wait-duration-in-open-state: 10s permitted-number-of-calls-in-half-open-state: 3 ratelimiter: instances: pricingService: limit-for-period: 5 limit-refresh-period: 1s timeout-duration: 0 retry: instances: externalApi: max-attempts: 3 wait-duration: 500ms bulkhead: instances: databaseService: max-concurrent-calls: 5 max-wait-duration: 100ms logging: level: io.github.resilience4j: DEBUG 3. Service Implementation with Resilience4j
Create a service that uses various Resilience4j patterns:
@Service public class BackendService { private static final Logger logger = LoggerFactory.getLogger(BackendService.class); private final Random random = new Random(); @CircuitBreaker(name = "backendService", fallbackMethod = "fallback") @RateLimiter(name = "pricingService") @Bulkhead(name = "databaseService") @Retry(name = "externalApi", fallbackMethod = "retryFallback") @TimeLimiter(name = "backendService") public CompletableFuture<String> processRequest(String input) { return CompletableFuture.supplyAsync(() -> { // Simulate various scenarios simulatePotentialFailures(); logger.info("Successfully processed request: {}", input); return "Processed: " + input; }); } private void simulatePotentialFailures() { int scenario = random.nextInt(100); if (scenario < 10) { // 10% slow response try { Thread.sleep(2000); } catch (InterruptedException e) { Thread.currentThread().interrupt(); } } else if (scenario < 25) { // 15% failure rate throw new RuntimeException("Simulated backend failure"); } else if (scenario < 30) { // 5% timeout scenario try { Thread.sleep(5000); } catch (InterruptedException e) { Thread.currentThread().interrupt(); } } // 70% success case } // Fallback methods public CompletableFuture<String> fallback(String input, Exception e) { logger.warn("Using fallback for input: {}, due to: {}", input, e.getMessage()); return CompletableFuture.completedFuture("Fallback response for: " + input); } public CompletableFuture<String> retryFallback(String input, Exception e) { logger.error("All retries exhausted for input: {}", input); return CompletableFuture.completedFuture("Retry exhausted for: " + input); } } 4. REST Controller
Create a controller to expose the service:
@RestController @RequestMapping("/api") public class DemoController { @Autowired private BackendService backendService; @GetMapping("/process/{input}") public CompletableFuture<ResponseEntity<Map<String, String>>> processInput( @PathVariable String input) { return backendService.processRequest(input) .thenApply(result -> ResponseEntity.ok(Map.of( "status", "success", "result", result, "timestamp", Instant.now().toString() ))) .exceptionally(throwable -> ResponseEntity.status(503).body(Map.of( "status", "error", "message", "Service unavailable", "timestamp", Instant.now().toString() ))); } @GetMapping("/batch-process") public CompletableFuture<ResponseEntity<Map<String, Object>>> batchProcess() { List<CompletableFuture<String>> futures = new ArrayList<>(); // Simulate concurrent requests to test bulkhead and rate limiter for (int i = 0; i < 10; i++) { futures.add(backendService.processRequest("item-" + i)); } return CompletableFuture.allOf(futures.toArray(new CompletableFuture[0])) .thenApply(v -> { List<String> results = futures.stream() .map(CompletableFuture::join) .collect(Collectors.toList()); return ResponseEntity.ok(Map.of( "processed", results.size(), "results", results, "timestamp", Instant.now().toString() )); }); } } 5. Custom Metrics Configuration (Optional)
For more advanced metric collection, create a custom metrics configuration:
@Configuration public class ResilienceMetricsConfig { @Autowired private MeterRegistry meterRegistry; @PostConstruct public void init() { // Custom metrics for business-specific resilience patterns Gauge.builder("resilience4j.custom.circuit_breaker.health") .description("Custom circuit breaker health indicator") .tags("application", "resilience4j-demo") .register(meterRegistry, this, config -> { // Custom health calculation logic return calculateCircuitBreakerHealth(); }); } private double calculateCircuitBreakerHealth() { // Implement custom health logic return 1.0; // Placeholder } } Prometheus Metrics Explained
When you access /actuator/prometheus, you'll see metrics like:
Circuit Breaker Metrics
# Circuit Breaker State (0=CLOSED, 1=OPEN, 2=HALF_OPEN, 3=DISABLED, 4=FORCED_OPEN) resilience4j_circuitbreaker_state{name="backendService",} 0.0 # Circuit Breaker Calls resilience4j_circuitbreaker_calls{name="backendService",kind="successful",} 42.0 resilience4j_circuitbreaker_calls{name="backendService",kind="failed",} 8.0 resilience4j_circuitbreaker_calls{name="backendService",kind="ignored",} 2.0 # Failure Rate resilience4j_circuitbreaker_failure_rate{name="backendService",} 16.0 # Slow Call Rate resilience4j_circuitbreaker_slow_call_rate{name="backendService",} 5.0 Rate Limiter Metrics
# Available Permissions resilience4j_ratelimiter_available_permissions{name="pricingService",} 3.0 # Waiting Threads resilience4j_ratelimiter_waiting_threads{name="pricingService",} 0.0 Bulkhead Metrics
# Concurrent Calls resilience4j_bulkhead_concurrent_calls{name="databaseService",} 2.0 # Available Capacity resilience4j_bulkhead_available_capacity{name="databaseService",} 3.0 Retry Metrics
# Retry Attempts resilience4j_retry_calls{name="externalApi",kind="successful_without_retry",} 35.0 resilience4j_retry_calls{name="externalApi",kind="successful_with_retry",} 7.0 resilience4j_retry_calls{name="externalApi",kind="failed_without_retry",} 3.0 resilience4j_retry_calls{name="externalApi",kind="failed_with_retry",} 2.0 Prometheus Configuration
Create a prometheus.yml to scrape your application:
global: scrape_interval: 15s evaluation_interval: 15s rule_files: # - "first_rules.yml" scrape_configs: - job_name: 'resilience4j-demo' metrics_path: '/actuator/prometheus' scrape_interval: 5s static_configs: - targets: ['localhost:8080'] labels: application: 'resilience4j-demo' environment: 'development' - job_name: 'prometheus' static_configs: - targets: ['localhost:9090']
Grafana Dashboard Queries
Create a comprehensive dashboard with these key queries:
1. Circuit Breaker Status
# Circuit Breaker State resilience4j_circuitbreaker_state{name="backendService"} # Failure Rate resilience4j_circuitbreaker_failure_rate{name="backendService"} # Call Volume sum by (kind) (rate(resilience4j_circuitbreaker_calls{name="backendService"}[5m])) 2. Rate Limiter Utilization
# Available Permissions Over Time resilience4j_ratelimiter_available_permissions{name="pricingService"} # Permission Utilization Rate (5 - resiliance4j_ratelimiter_available_permissions{name="pricingService"}) / 5 * 100 3. Bulkhead Concurrency
# Concurrent Calls resilience4j_bulkhead_concurrent_calls{name="databaseService"} # Capacity Utilization resilience4j_bulkhead_concurrent_calls{name="databaseService"} / 5 * 100 4. Retry Effectiveness
# Retry Success Rate sum(rate(resilience4j_retry_calls{name="externalApi",kind=~"successful.*"}[5m])) / sum(rate(resilience4j_retry_calls{name="externalApi"}[5m])) * 100 Alerting Rules
Create Prometheus alerting rules for critical scenarios:
groups: - name: resilience4j_alerts rules: - alert: CircuitBreakerOpen expr: resilience4j_circuitbreaker_state{name="backendService"} == 1 for: 1m labels: severity: critical annotations: summary: "Circuit breaker for {{ $labels.name }} is OPEN" description: "The circuit breaker has been open for more than 1 minute. Service degradation detected." - alert: HighFailureRate expr: resilience4j_circuitbreaker_failure_rate{name="backendService"} > 30 for: 2m labels: severity: warning annotations: summary: "High failure rate for {{ $labels.name }}" description: "Failure rate is {{ $value }}%, approaching circuit breaker threshold." - alert: RateLimiterExhausted expr: resilience4j_ratelimiter_available_permissions{name="pricingService"} == 0 for: 30s labels: severity: warning annotations: summary: "Rate limiter exhausted for {{ $labels.name }}" description: "No available permissions in rate limiter." Best Practices
- Use Meaningful Names: Choose descriptive circuit breaker and rate limiter names
- Set Appropriate thresholds: Align failure rate thresholds with your SLOs
- Monitor State Transitions: Alert on frequent OPEN/CLOSED transitions (flapping)
- Correlate with Business Metrics: Link resilience metrics to business KPIs
- Test Resilience: Use chaos engineering to validate your monitoring
- Set Up Dashboards: Create team-specific dashboards for different services
Conclusion
Integrating Resilience4j metrics with Prometheus provides crucial visibility into your application's resilience patterns. By monitoring circuit breaker states, rate limiter utilization, retry effectiveness, and bulkhead concurrency, you can:
- Detect service degradation before it affects users
- Validate your resilience configuration matches real-world conditions
- Make data-driven decisions about timeout and retry configurations
- Create actionable alerts for operational teams
The combination of Resilience4j's powerful patterns and Prometheus's robust monitoring creates a foundation for building truly observable and resilient microservices. With proper dashboards and alerting, you can confidently deploy circuit breakers and other resilience patterns, knowing you'll be alerted when they're needed most.