Alertmanager Webhook Integration in Java: Building Custom Alert Handlers

Alertmanager webhooks allow you to extend Prometheus alerting by sending alert notifications to your custom applications. This enables powerful integrations with chat platforms, ticketing systems, and custom automation workflows.

Core Concepts

What is Alertmanager Webhook?

HTTP endpoint that receives POST requests from Alertmanager
Receives alert notifications in JSON format
Enables custom processing and routing of alerts
Supports both firing and resolved alerts

Key Use Cases:

Send alerts to Slack, Microsoft Teams, Discord
Create JIRA tickets automatically
Trigger runbooks or automation scripts
Custom filtering and routing logic
Alert enrichment and deduplication

Dependencies and Setup

Maven Dependencies

<properties> <spring-boot.version>3.1.0</spring-boot.version> <jackson.version>2.15.2</jackson.version> </properties> <dependencies> <!-- Spring Boot --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> <version>${spring-boot.version}</version> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-validation</artifactId> <version>${spring-boot.version}</version> </dependency> <!-- JSON Processing --> <dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-databind</artifactId> <version>${jackson.version}</version> </dependency> <!-- HTTP Client --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-webflux</artifactId> <version>${spring-boot.version}</version> </dependency> <!-- Testing --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-test</artifactId> <version>${spring-boot.version}</version> <scope>test</scope> </dependency> </dependencies>

Alertmanager Configuration

# alertmanager.yml global: smtp_smarthost: 'localhost:1025' smtp_from: '[email protected]' route: group_by: ['alertname', 'cluster'] group_wait: 10s group_interval: 10s repeat_interval: 1h receiver: 'webhook-receiver' routes: - match: severity: critical receiver: 'critical-webhook' receivers: - name: 'webhook-receiver' webhook_configs: - url: 'http://localhost:8080/api/v1/alerts' send_resolved: true max_alerts: 10 - name: 'critical-webhook' webhook_configs: - url: 'http://localhost:8080/api/v1/alerts/critical' send_resolved: true

Core Implementation

1. Alertmanager Webhook Data Models

@Data @JsonIgnoreProperties(ignoreUnknown = true) public class AlertmanagerWebhook { @JsonProperty("version") private String version; @JsonProperty("groupKey") private String groupKey; @JsonProperty("truncatedAlerts") private Integer truncatedAlerts; @JsonProperty("status") private String status; // "firing" or "resolved" @JsonProperty("receiver") private String receiver; @JsonProperty("groupLabels") private Map<String, String> groupLabels; @JsonProperty("commonLabels") private Map<String, String> commonLabels; @JsonProperty("commonAnnotations") private Map<String, String> commonAnnotations; @JsonProperty("externalURL") private String externalURL; @JsonProperty("alerts") private List<Alert> alerts; public boolean isFiring() { return "firing".equals(status); } public boolean isResolved() { return "resolved".equals(status); } } @Data @JsonIgnoreProperties(ignoreUnknown = true) public class Alert { @JsonProperty("status") private String status; // "firing" or "resolved" @JsonProperty("labels") private Map<String, String> labels; @JsonProperty("annotations") private Map<String, String> annotations; @JsonProperty("startsAt") private String startsAt; @JsonProperty("endsAt") private String endsAt; @JsonProperty("generatorURL") private String generatorURL; @JsonProperty("fingerprint") private String fingerprint; // Helper methods public String getAlertName() { return labels != null ? labels.get("alertname") : null; } public String getSeverity() { return labels != null ? labels.get("severity") : null; } public String getSummary() { return annotations != null ? annotations.get("summary") : null; } public String getDescription() { return annotations != null ? annotations.get("description") : null; } public boolean isCritical() { return "critical".equals(getSeverity()); } public boolean isWarning() { return "warning".equals(getSeverity()); } }

2. Webhook Controller

@RestController @RequestMapping("/api/v1/alerts") @Slf4j public class AlertmanagerWebhookController { private final AlertProcessingService alertProcessingService; private final AlertMetricsService metricsService; public AlertmanagerWebhookController(AlertProcessingService alertProcessingService, AlertMetricsService metricsService) { this.alertProcessingService = alertProcessingService; this.metricsService = metricsService; } @PostMapping public ResponseEntity<Void> handleWebhook(@Valid @RequestBody AlertmanagerWebhook webhook) { log.info("Received Alertmanager webhook: status={}, alerts={}, receiver={}", webhook.getStatus(), webhook.getAlerts().size(), webhook.getReceiver()); // Process alerts asynchronously CompletableFuture.runAsync(() -> processWebhook(webhook)); // Return 200 immediately to acknowledge receipt return ResponseEntity.ok().build(); } @PostMapping("/critical") public ResponseEntity<Void> handleCriticalAlerts(@Valid @RequestBody AlertmanagerWebhook webhook) { log.warn("Received CRITICAL alerts: status={}, alerts={}", webhook.getStatus(), webhook.getAlerts().size()); metricsService.incrementCriticalAlerts(webhook.getAlerts().size()); // Process critical alerts with high priority CompletableFuture.runAsync(() -> processCriticalWebhook(webhook)); return ResponseEntity.ok().build(); } private void processWebhook(AlertmanagerWebhook webhook) { try { alertProcessingService.processAlerts(webhook); metricsService.incrementProcessedAlerts(webhook.getAlerts().size()); } catch (Exception e) { log.error("Failed to process webhook: {}", webhook.getGroupKey(), e); metricsService.incrementFailedAlerts(webhook.getAlerts().size()); } } private void processCriticalWebhook(AlertmanagerWebhook webhook) { try { alertProcessingService.processCriticalAlerts(webhook); metricsService.incrementProcessedCriticalAlerts(webhook.getAlerts().size()); } catch (Exception e) { log.error("Failed to process critical webhook: {}", webhook.getGroupKey(), e); metricsService.incrementFailedCriticalAlerts(webhook.getAlerts().size()); } } }

3. Alert Processing Service

@Service @Slf4j public class AlertProcessingService { private final List<AlertHandler> alertHandlers; private final AlertDeduplicationService deduplicationService; private final WebClient webClient; public AlertProcessingService(List<AlertHandler> alertHandlers, AlertDeduplicationService deduplicationService, WebClient.Builder webClientBuilder) { this.alertHandlers = alertHandlers; this.deduplicationService = deduplicationService; this.webClient = webClientBuilder.build(); } public void processAlerts(AlertmanagerWebhook webhook) { List<Alert> alerts = webhook.getAlerts(); // Filter out duplicates List<Alert> uniqueAlerts = deduplicationService.filterDuplicates(alerts); if (uniqueAlerts.isEmpty()) { log.debug("All alerts were duplicates, skipping processing"); return; } // Process each alert through all handlers for (Alert alert : uniqueAlerts) { processAlert(alert, webhook.isFiring()); } log.info("Processed {} unique alerts ({} duplicates filtered)", uniqueAlerts.size(), alerts.size() - uniqueAlerts.size()); } public void processCriticalAlerts(AlertmanagerWebhook webhook) { List<Alert> criticalAlerts = webhook.getAlerts().stream() .filter(Alert::isCritical) .collect(Collectors.toList()); if (criticalAlerts.isEmpty()) { return; } log.warn("Processing {} critical alerts", criticalAlerts.size()); for (Alert alert : criticalAlerts) { processCriticalAlert(alert, webhook.isFiring()); } } private void processAlert(Alert alert, boolean isFiring) { for (AlertHandler handler : alertHandlers) { try { if (handler.supports(alert)) { if (isFiring) { handler.handleFiringAlert(alert); } else { handler.handleResolvedAlert(alert); } } } catch (Exception e) { log.error("Handler {} failed to process alert: {}", handler.getClass().getSimpleName(), alert.getFingerprint(), e); } } } private void processCriticalAlert(Alert alert, boolean isFiring) { // Critical alerts get special treatment if (isFiring) { triggerPagerDuty(alert); createJiraTicket(alert); notifyOnCallEngineers(alert); } else { resolvePagerDuty(alert); closeJiraTicket(alert); } // Also process through normal handlers processAlert(alert, isFiring); } private void triggerPagerDuty(Alert alert) { // Implement PagerDuty integration log.warn("Triggering PagerDuty for critical alert: {}", alert.getAlertName()); } private void createJiraTicket(Alert alert) { // Implement JIRA integration log.warn("Creating JIRA ticket for critical alert: {}", alert.getAlertName()); } private void notifyOnCallEngineers(Alert alert) { // Implement on-call notification log.warn("Notifying on-call engineers for critical alert: {}", alert.getAlertName()); } private void resolvePagerDuty(Alert alert) { log.info("Resolving PagerDuty for alert: {}", alert.getAlertName()); } private void closeJiraTicket(Alert alert) { log.info("Closing JIRA ticket for alert: {}", alert.getAlertName()); } }

4. Alert Handler Interface and Implementations

public interface AlertHandler { boolean supports(Alert alert); void handleFiringAlert(Alert alert); void handleResolvedAlert(Alert alert); } @Service @Slf4j public class SlackAlertHandler implements AlertHandler { private final WebClient webClient; private final SlackConfig slackConfig; public SlackAlertHandler(WebClient.Builder webClientBuilder, SlackConfig slackConfig) { this.webClient = webClientBuilder.build(); this.slackConfig = slackConfig; } @Override public boolean supports(Alert alert) { // Handle all alerts except those with "silence" label return !"true".equals(alert.getLabels().get("silence")); } @Override public void handleFiringAlert(Alert alert) { String message = buildSlackMessage(alert, "🔥 FIRING"); sendSlackMessage(message, getChannelForSeverity(alert.getSeverity())); log.info("Sent Slack notification for firing alert: {}", alert.getAlertName()); } @Override public void handleResolvedAlert(Alert alert) { String message = buildSlackMessage(alert, "✅ RESOLVED"); sendSlackMessage(message, getChannelForSeverity(alert.getSeverity())); log.info("Sent Slack notification for resolved alert: {}", alert.getAlertName()); } private String buildSlackMessage(Alert alert, String status) { return String.format(""" { "blocks": [ { "type": "header", "text": { "type": "plain_text", "text": "%s: %s" } }, { "type": "section", "fields": [ { "type": "mrkdwn", "text": "*Severity:*\\n%s" }, { "type": "mrkdwn", "text": "*Status:*\\n%s" } ] }, { "type": "section", "text": { "type": "mrkdwn", "text": "*Description:*\\n%s" } }, { "type": "actions", "elements": [ { "type": "button", "text": { "type": "plain_text", "text": "View in Prometheus" }, "url": "%s" } ] } ] } """, status, alert.getAlertName(), alert.getSeverity(), status, alert.getDescription(), alert.getGeneratorURL()); } private void sendSlackMessage(String message, String channel) { try { webClient.post() .uri(slackConfig.getWebhookUrl()) .contentType(MediaType.APPLICATION_JSON) .bodyValue(message) .retrieve() .bodyToMono(String.class) .block(); } catch (Exception e) { log.error("Failed to send Slack message", e); } } private String getChannelForSeverity(String severity) { return switch (severity) { case "critical" -> slackConfig.getCriticalChannel(); case "warning" -> slackConfig.getWarningChannel(); default -> slackConfig.getDefaultChannel(); }; } } @Service @Slf4j public class DatabaseAlertHandler implements AlertHandler { private final AlertRepository alertRepository; public DatabaseAlertHandler(AlertRepository alertRepository) { this.alertRepository = alertRepository; } @Override public boolean supports(Alert alert) { return true; // Store all alerts in database } @Override public void handleFiringAlert(Alert alert) { AlertEntity entity = convertToEntity(alert, AlertStatus.FIRING); alertRepository.save(entity); log.debug("Stored firing alert in database: {}", alert.getFingerprint()); } @Override public void handleResolvedAlert(Alert alert) { AlertEntity entity = convertToEntity(alert, AlertStatus.RESOLVED); alertRepository.save(entity); log.debug("Stored resolved alert in database: {}", alert.getFingerprint()); } private AlertEntity convertToEntity(Alert alert, AlertStatus status) { AlertEntity entity = new AlertEntity(); entity.setFingerprint(alert.getFingerprint()); entity.setAlertName(alert.getAlertName()); entity.setSeverity(alert.getSeverity()); entity.setStatus(status); entity.setSummary(alert.getSummary()); entity.setDescription(alert.getDescription()); entity.setLabels(alert.getLabels()); entity.setAnnotations(alert.getAnnotations()); entity.setStartsAt(parseDate(alert.getStartsAt())); entity.setEndsAt(parseDate(alert.getEndsAt())); entity.setGeneratorUrl(alert.getGeneratorURL()); entity.setCreatedAt(Instant.now()); return entity; } private Instant parseDate(String dateString) { try { return Instant.parse(dateString); } catch (Exception e) { return Instant.now(); } } }

5. Deduplication Service

@Service @Slf4j public class AlertDeduplicationService { private final Cache<String, Instant> alertCache; public AlertDeduplicationService() { this.alertCache = Caffeine.newBuilder() .expireAfterWrite(1, TimeUnit.HOURS) .maximumSize(10000) .build(); } public List<Alert> filterDuplicates(List<Alert> alerts) { List<Alert> uniqueAlerts = new ArrayList<>(); Instant now = Instant.now(); for (Alert alert : alerts) { String fingerprint = alert.getFingerprint(); if (fingerprint == null) { // No fingerprint, can't deduplicate uniqueAlerts.add(alert); continue; } Instant lastSeen = alertCache.getIfPresent(fingerprint); if (lastSeen == null) { // New alert alertCache.put(fingerprint, now); uniqueAlerts.add(alert); } else { // Duplicate alert log.debug("Filtered duplicate alert: {}", fingerprint); } } return uniqueAlerts; } public void clearCache() { alertCache.invalidateAll(); log.info("Alert deduplication cache cleared"); } }

6. Metrics and Monitoring

@Service public class AlertMetricsService { private final MeterRegistry meterRegistry; private final Counter processedAlerts; private final Counter failedAlerts; private final Counter criticalAlerts; public AlertMetricsService(MeterRegistry meterRegistry) { this.meterRegistry = meterRegistry; this.processedAlerts = Counter.builder("alerts.processed") .description("Total number of processed alerts") .register(meterRegistry); this.failedAlerts = Counter.builder("alerts.failed") .description("Total number of failed alert processing attempts") .register(meterRegistry); this.criticalAlerts = Counter.builder("alerts.critical") .description("Total number of critical alerts") .register(meterRegistry); } public void incrementProcessedAlerts(int count) { processedAlerts.increment(count); } public void incrementFailedAlerts(int count) { failedAlerts.increment(count); } public void incrementCriticalAlerts(int count) { criticalAlerts.increment(count); } public void incrementProcessedCriticalAlerts(int count) { criticalAlerts.increment(count); } public void incrementFailedCriticalAlerts(int count) { failedAlerts.increment(count); } }

7. Configuration Classes

@Configuration @ConfigurationProperties(prefix = "slack") @Data public class SlackConfig { private String webhookUrl; private String defaultChannel; private String warningChannel; private String criticalChannel; } @Configuration @ConfigurationProperties(prefix = "alertmanager") @Data public class AlertmanagerConfig { private int maxAlertsPerRequest = 100; private int requestTimeoutSeconds = 30; private boolean enableDeduplication = true; private List<String> ignoredLabels = List.of("ignore"); }

8. Database Entities

@Entity @Table(name = "alerts") @Data public class AlertEntity { @Id private String fingerprint; @Column(nullable = false) private String alertName; @Column(nullable = false) private String severity; @Enumerated(EnumType.STRING) @Column(nullable = false) private AlertStatus status; private String summary; private String description; @Column(columnDefinition = "JSON") @Convert(converter = HashMapConverter.class) private Map<String, String> labels; @Column(columnDefinition = "JSON") @Convert(converter = HashMapConverter.class) private Map<String, String> annotations; private Instant startsAt; private Instant endsAt; private String generatorUrl; private Instant createdAt; } public enum AlertStatus { FIRING, RESOLVED } @Converter public class HashMapConverter implements AttributeConverter<Map<String, String>, String> { private final ObjectMapper objectMapper = new ObjectMapper(); @Override public String convertToDatabaseColumn(Map<String, String> attribute) { try { return objectMapper.writeValueAsString(attribute); } catch (JsonProcessingException e) { return "{}"; } } @Override public Map<String, String> convertToEntityAttribute(String dbData) { try { return objectMapper.readValue(dbData, new TypeReference<Map<String, String>>() {}); } catch (Exception e) { return new HashMap<>(); } } }

Advanced Features

1. Alert Enrichment

@Service public class AlertEnrichmentService { private final WebClient webClient; public Alert enrichAlert(Alert alert) { // Add additional context to alerts Map<String, String> enrichedAnnotations = new HashMap<>(alert.getAnnotations()); // Add runbook URL if available String runbookUrl = getRunbookUrl(alert.getAlertName()); if (runbookUrl != null) { enrichedAnnotations.put("runbook", runbookUrl); } // Add service owner information String serviceOwner = getServiceOwner(alert.getLabels().get("service")); if (serviceOwner != null) { enrichedAnnotations.put("owner", serviceOwner); } Alert enrichedAlert = new Alert(); // Copy all fields and set enriched annotations // Implementation details... return enrichedAlert; } private String getRunbookUrl(String alertName) { // Lookup runbook URL from configuration or CMDB return null; // Implementation } private String getServiceOwner(String serviceName) { // Lookup service owner from service registry return null; // Implementation } }

2. Rate Limiting

@Service public class AlertRateLimitService { private final Cache<String, RateLimitCounter> rateLimitCache; public AlertRateLimitService() { this.rateLimitCache = Caffeine.newBuilder() .expireAfterWrite(1, TimeUnit.HOURS) .maximumSize(1000) .build(); } public boolean isRateLimited(String alertKey, int maxAlertsPerMinute) { RateLimitCounter counter = rateLimitCache.get(alertKey, k -> new RateLimitCounter()); return counter.incrementAndCheck(maxAlertsPerMinute); } private static class RateLimitCounter { private final AtomicInteger count = new AtomicInteger(0); private volatile long lastReset = System.currentTimeMillis(); public boolean incrementAndCheck(int maxPerMinute) { resetIfNeeded(); int current = count.incrementAndGet(); return current > maxPerMinute; } private void resetIfNeeded() { long now = System.currentTimeMillis(); if (now - lastReset > 60000) { // 1 minute count.set(0); lastReset = now; } } } }

Testing

1. Unit Tests

@ExtendWith(MockitoExtension.class) class AlertProcessingServiceTest { @Mock private AlertHandler slackHandler; @Mock private AlertHandler databaseHandler; @InjectMocks private AlertProcessingService alertProcessingService; @Test void shouldProcessAlertsThroughAllHandlers() { // Given Alert alert = createTestAlert(); AlertmanagerWebhook webhook = createTestWebhook(List.of(alert)); when(slackHandler.supports(any())).thenReturn(true); when(databaseHandler.supports(any())).thenReturn(true); // When alertProcessingService.processAlerts(webhook); // Then verify(slackHandler).handleFiringAlert(alert); verify(databaseHandler).handleFiringAlert(alert); } private Alert createTestAlert() { Alert alert = new Alert(); alert.setStatus("firing"); alert.setFingerprint("test-fingerprint"); // Set other properties... return alert; } private AlertmanagerWebhook createTestWebhook(List<Alert> alerts) { AlertmanagerWebhook webhook = new AlertmanagerWebhook(); webhook.setStatus("firing"); webhook.setAlerts(alerts); return webhook; } }

2. Integration Test

@SpringBootTest @AutoConfigureTestDatabase class AlertmanagerWebhookControllerIntegrationTest { @Autowired private TestRestTemplate restTemplate; @Autowired private AlertRepository alertRepository; @Test void shouldHandleWebhookAndStoreAlert() { // Given AlertmanagerWebhook webhook = createTestWebhook(); // When ResponseEntity<Void> response = restTemplate.postForEntity( "/api/v1/alerts", webhook, Void.class); // Then assertThat(response.getStatusCode()).isEqualTo(HttpStatus.OK); // Wait for async processing await().atMost(5, TimeUnit.SECONDS) .until(() -> !alertRepository.findAll().isEmpty()); List<AlertEntity> storedAlerts = alertRepository.findAll(); assertThat(storedAlerts).hasSize(1); } }

Best Practices

Idempotency: Ensure alert processing is idempotent
Async Processing: Process alerts asynchronously to avoid timeouts
Rate Limiting: Implement rate limiting to prevent overload
Deduplication: Filter duplicate alerts to reduce noise
Monitoring: Monitor your webhook handler's performance
Error Handling: Implement robust error handling and retries
Security: Validate incoming requests and consider authentication

@Component public class WebhookSecurityFilter implements Filter { @Value("${webhook.secret-token:}") private String secretToken; @Override public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException { if (!secretToken.isEmpty()) { HttpServletRequest httpRequest = (HttpServletRequest) request; String authHeader = httpRequest.getHeader("Authorization"); if (!isValidToken(authHeader)) { ((HttpServletResponse) response).sendError(HttpStatus.UNAUTHORIZED.value()); return; } } chain.doFilter(request, response); } private boolean isValidToken(String authHeader) { return authHeader != null && authHeader.equals("Bearer " + secretToken); } }

Conclusion

Alertmanager webhook integration in Java enables:

Custom alert routing to various destinations
Alert enrichment with additional context
Advanced processing logic and filtering
Integration with existing systems and workflows
Reliable alert handling with deduplication and rate limiting

By implementing the patterns shown above, you can build a robust, scalable alert handling system that integrates seamlessly with your Prometheus monitoring stack and provides intelligent alert routing and processing capabilities.