Alertmanager Webhook Integration in Java: Building Custom Alert Handlers

Alertmanager webhooks allow you to extend Prometheus alerting by sending alert notifications to your custom applications. This enables powerful integrations with chat platforms, ticketing systems, and custom automation workflows.

Core Concepts

What is Alertmanager Webhook?

HTTP endpoint that receives POST requests from Alertmanager
Receives alert notifications in JSON format
Enables custom processing and routing of alerts
Supports both firing and resolved alerts

Key Use Cases:

Send alerts to Slack, Microsoft Teams, Discord
Create JIRA tickets automatically
Trigger runbooks or automation scripts
Custom filtering and routing logic
Alert enrichment and deduplication

Dependencies and Setup

Maven Dependencies

<properties>
<spring-boot.version>3.1.0</spring-boot.version>
<jackson.version>2.15.2</jackson.version>
</properties>
<dependencies>
<!-- Spring Boot -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
<version>${spring-boot.version}</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-validation</artifactId>
<version>${spring-boot.version}</version>
</dependency>
<!-- JSON Processing -->
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>${jackson.version}</version>
</dependency>
<!-- HTTP Client -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-webflux</artifactId>
<version>${spring-boot.version}</version>
</dependency>
<!-- Testing -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<version>${spring-boot.version}</version>
<scope>test</scope>
</dependency>
</dependencies>

Alertmanager Configuration

# alertmanager.yml
global:
smtp_smarthost: 'localhost:1025'
smtp_from: '[email protected]'
route:
group_by: ['alertname', 'cluster']
group_wait: 10s
group_interval: 10s
repeat_interval: 1h
receiver: 'webhook-receiver'
routes:
- match:
severity: critical
receiver: 'critical-webhook'
receivers:
- name: 'webhook-receiver'
webhook_configs:
- url: 'http://localhost:8080/api/v1/alerts'
send_resolved: true
max_alerts: 10
- name: 'critical-webhook'
webhook_configs:
- url: 'http://localhost:8080/api/v1/alerts/critical'
send_resolved: true

Core Implementation

1. Alertmanager Webhook Data Models

@Data
@JsonIgnoreProperties(ignoreUnknown = true)
public class AlertmanagerWebhook {
@JsonProperty("version")
private String version;
@JsonProperty("groupKey")
private String groupKey;
@JsonProperty("truncatedAlerts")
private Integer truncatedAlerts;
@JsonProperty("status")
private String status; // "firing" or "resolved"
@JsonProperty("receiver")
private String receiver;
@JsonProperty("groupLabels")
private Map<String, String> groupLabels;
@JsonProperty("commonLabels")
private Map<String, String> commonLabels;
@JsonProperty("commonAnnotations")
private Map<String, String> commonAnnotations;
@JsonProperty("externalURL")
private String externalURL;
@JsonProperty("alerts")
private List<Alert> alerts;
public boolean isFiring() {
return "firing".equals(status);
}
public boolean isResolved() {
return "resolved".equals(status);
}
}
@Data
@JsonIgnoreProperties(ignoreUnknown = true)
public class Alert {
@JsonProperty("status")
private String status; // "firing" or "resolved"
@JsonProperty("labels")
private Map<String, String> labels;
@JsonProperty("annotations")
private Map<String, String> annotations;
@JsonProperty("startsAt")
private String startsAt;
@JsonProperty("endsAt")
private String endsAt;
@JsonProperty("generatorURL")
private String generatorURL;
@JsonProperty("fingerprint")
private String fingerprint;
// Helper methods
public String getAlertName() {
return labels != null ? labels.get("alertname") : null;
}
public String getSeverity() {
return labels != null ? labels.get("severity") : null;
}
public String getSummary() {
return annotations != null ? annotations.get("summary") : null;
}
public String getDescription() {
return annotations != null ? annotations.get("description") : null;
}
public boolean isCritical() {
return "critical".equals(getSeverity());
}
public boolean isWarning() {
return "warning".equals(getSeverity());
}
}

2. Webhook Controller

@RestController
@RequestMapping("/api/v1/alerts")
@Slf4j
public class AlertmanagerWebhookController {
private final AlertProcessingService alertProcessingService;
private final AlertMetricsService metricsService;
public AlertmanagerWebhookController(AlertProcessingService alertProcessingService,
AlertMetricsService metricsService) {
this.alertProcessingService = alertProcessingService;
this.metricsService = metricsService;
}
@PostMapping
public ResponseEntity<Void> handleWebhook(@Valid @RequestBody AlertmanagerWebhook webhook) {
log.info("Received Alertmanager webhook: status={}, alerts={}, receiver={}",
webhook.getStatus(), webhook.getAlerts().size(), webhook.getReceiver());
// Process alerts asynchronously
CompletableFuture.runAsync(() -> processWebhook(webhook));
// Return 200 immediately to acknowledge receipt
return ResponseEntity.ok().build();
}
@PostMapping("/critical")
public ResponseEntity<Void> handleCriticalAlerts(@Valid @RequestBody AlertmanagerWebhook webhook) {
log.warn("Received CRITICAL alerts: status={}, alerts={}",
webhook.getStatus(), webhook.getAlerts().size());
metricsService.incrementCriticalAlerts(webhook.getAlerts().size());
// Process critical alerts with high priority
CompletableFuture.runAsync(() -> processCriticalWebhook(webhook));
return ResponseEntity.ok().build();
}
private void processWebhook(AlertmanagerWebhook webhook) {
try {
alertProcessingService.processAlerts(webhook);
metricsService.incrementProcessedAlerts(webhook.getAlerts().size());
} catch (Exception e) {
log.error("Failed to process webhook: {}", webhook.getGroupKey(), e);
metricsService.incrementFailedAlerts(webhook.getAlerts().size());
}
}
private void processCriticalWebhook(AlertmanagerWebhook webhook) {
try {
alertProcessingService.processCriticalAlerts(webhook);
metricsService.incrementProcessedCriticalAlerts(webhook.getAlerts().size());
} catch (Exception e) {
log.error("Failed to process critical webhook: {}", webhook.getGroupKey(), e);
metricsService.incrementFailedCriticalAlerts(webhook.getAlerts().size());
}
}
}

3. Alert Processing Service

@Service
@Slf4j
public class AlertProcessingService {
private final List<AlertHandler> alertHandlers;
private final AlertDeduplicationService deduplicationService;
private final WebClient webClient;
public AlertProcessingService(List<AlertHandler> alertHandlers,
AlertDeduplicationService deduplicationService,
WebClient.Builder webClientBuilder) {
this.alertHandlers = alertHandlers;
this.deduplicationService = deduplicationService;
this.webClient = webClientBuilder.build();
}
public void processAlerts(AlertmanagerWebhook webhook) {
List<Alert> alerts = webhook.getAlerts();
// Filter out duplicates
List<Alert> uniqueAlerts = deduplicationService.filterDuplicates(alerts);
if (uniqueAlerts.isEmpty()) {
log.debug("All alerts were duplicates, skipping processing");
return;
}
// Process each alert through all handlers
for (Alert alert : uniqueAlerts) {
processAlert(alert, webhook.isFiring());
}
log.info("Processed {} unique alerts ({} duplicates filtered)",
uniqueAlerts.size(), alerts.size() - uniqueAlerts.size());
}
public void processCriticalAlerts(AlertmanagerWebhook webhook) {
List<Alert> criticalAlerts = webhook.getAlerts().stream()
.filter(Alert::isCritical)
.collect(Collectors.toList());
if (criticalAlerts.isEmpty()) {
return;
}
log.warn("Processing {} critical alerts", criticalAlerts.size());
for (Alert alert : criticalAlerts) {
processCriticalAlert(alert, webhook.isFiring());
}
}
private void processAlert(Alert alert, boolean isFiring) {
for (AlertHandler handler : alertHandlers) {
try {
if (handler.supports(alert)) {
if (isFiring) {
handler.handleFiringAlert(alert);
} else {
handler.handleResolvedAlert(alert);
}
}
} catch (Exception e) {
log.error("Handler {} failed to process alert: {}",
handler.getClass().getSimpleName(), alert.getFingerprint(), e);
}
}
}
private void processCriticalAlert(Alert alert, boolean isFiring) {
// Critical alerts get special treatment
if (isFiring) {
triggerPagerDuty(alert);
createJiraTicket(alert);
notifyOnCallEngineers(alert);
} else {
resolvePagerDuty(alert);
closeJiraTicket(alert);
}
// Also process through normal handlers
processAlert(alert, isFiring);
}
private void triggerPagerDuty(Alert alert) {
// Implement PagerDuty integration
log.warn("Triggering PagerDuty for critical alert: {}", alert.getAlertName());
}
private void createJiraTicket(Alert alert) {
// Implement JIRA integration
log.warn("Creating JIRA ticket for critical alert: {}", alert.getAlertName());
}
private void notifyOnCallEngineers(Alert alert) {
// Implement on-call notification
log.warn("Notifying on-call engineers for critical alert: {}", alert.getAlertName());
}
private void resolvePagerDuty(Alert alert) {
log.info("Resolving PagerDuty for alert: {}", alert.getAlertName());
}
private void closeJiraTicket(Alert alert) {
log.info("Closing JIRA ticket for alert: {}", alert.getAlertName());
}
}

4. Alert Handler Interface and Implementations

public interface AlertHandler {
boolean supports(Alert alert);
void handleFiringAlert(Alert alert);
void handleResolvedAlert(Alert alert);
}
@Service
@Slf4j
public class SlackAlertHandler implements AlertHandler {
private final WebClient webClient;
private final SlackConfig slackConfig;
public SlackAlertHandler(WebClient.Builder webClientBuilder, SlackConfig slackConfig) {
this.webClient = webClientBuilder.build();
this.slackConfig = slackConfig;
}
@Override
public boolean supports(Alert alert) {
// Handle all alerts except those with "silence" label
return !"true".equals(alert.getLabels().get("silence"));
}
@Override
public void handleFiringAlert(Alert alert) {
String message = buildSlackMessage(alert, "🔥 FIRING");
sendSlackMessage(message, getChannelForSeverity(alert.getSeverity()));
log.info("Sent Slack notification for firing alert: {}", alert.getAlertName());
}
@Override
public void handleResolvedAlert(Alert alert) {
String message = buildSlackMessage(alert, "✅ RESOLVED");
sendSlackMessage(message, getChannelForSeverity(alert.getSeverity()));
log.info("Sent Slack notification for resolved alert: {}", alert.getAlertName());
}
private String buildSlackMessage(Alert alert, String status) {
return String.format("""
{
"blocks": [
{
"type": "header",
"text": {
"type": "plain_text",
"text": "%s: %s"
}
},
{
"type": "section",
"fields": [
{
"type": "mrkdwn",
"text": "*Severity:*\\n%s"
},
{
"type": "mrkdwn",
"text": "*Status:*\\n%s"
}
]
},
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "*Description:*\\n%s"
}
},
{
"type": "actions",
"elements": [
{
"type": "button",
"text": {
"type": "plain_text",
"text": "View in Prometheus"
},
"url": "%s"
}
]
}
]
}
""", status, alert.getAlertName(), alert.getSeverity(), status,
alert.getDescription(), alert.getGeneratorURL());
}
private void sendSlackMessage(String message, String channel) {
try {
webClient.post()
.uri(slackConfig.getWebhookUrl())
.contentType(MediaType.APPLICATION_JSON)
.bodyValue(message)
.retrieve()
.bodyToMono(String.class)
.block();
} catch (Exception e) {
log.error("Failed to send Slack message", e);
}
}
private String getChannelForSeverity(String severity) {
return switch (severity) {
case "critical" -> slackConfig.getCriticalChannel();
case "warning" -> slackConfig.getWarningChannel();
default -> slackConfig.getDefaultChannel();
};
}
}
@Service
@Slf4j
public class DatabaseAlertHandler implements AlertHandler {
private final AlertRepository alertRepository;
public DatabaseAlertHandler(AlertRepository alertRepository) {
this.alertRepository = alertRepository;
}
@Override
public boolean supports(Alert alert) {
return true; // Store all alerts in database
}
@Override
public void handleFiringAlert(Alert alert) {
AlertEntity entity = convertToEntity(alert, AlertStatus.FIRING);
alertRepository.save(entity);
log.debug("Stored firing alert in database: {}", alert.getFingerprint());
}
@Override
public void handleResolvedAlert(Alert alert) {
AlertEntity entity = convertToEntity(alert, AlertStatus.RESOLVED);
alertRepository.save(entity);
log.debug("Stored resolved alert in database: {}", alert.getFingerprint());
}
private AlertEntity convertToEntity(Alert alert, AlertStatus status) {
AlertEntity entity = new AlertEntity();
entity.setFingerprint(alert.getFingerprint());
entity.setAlertName(alert.getAlertName());
entity.setSeverity(alert.getSeverity());
entity.setStatus(status);
entity.setSummary(alert.getSummary());
entity.setDescription(alert.getDescription());
entity.setLabels(alert.getLabels());
entity.setAnnotations(alert.getAnnotations());
entity.setStartsAt(parseDate(alert.getStartsAt()));
entity.setEndsAt(parseDate(alert.getEndsAt()));
entity.setGeneratorUrl(alert.getGeneratorURL());
entity.setCreatedAt(Instant.now());
return entity;
}
private Instant parseDate(String dateString) {
try {
return Instant.parse(dateString);
} catch (Exception e) {
return Instant.now();
}
}
}

5. Deduplication Service

@Service
@Slf4j
public class AlertDeduplicationService {
private final Cache<String, Instant> alertCache;
public AlertDeduplicationService() {
this.alertCache = Caffeine.newBuilder()
.expireAfterWrite(1, TimeUnit.HOURS)
.maximumSize(10000)
.build();
}
public List<Alert> filterDuplicates(List<Alert> alerts) {
List<Alert> uniqueAlerts = new ArrayList<>();
Instant now = Instant.now();
for (Alert alert : alerts) {
String fingerprint = alert.getFingerprint();
if (fingerprint == null) {
// No fingerprint, can't deduplicate
uniqueAlerts.add(alert);
continue;
}
Instant lastSeen = alertCache.getIfPresent(fingerprint);
if (lastSeen == null) {
// New alert
alertCache.put(fingerprint, now);
uniqueAlerts.add(alert);
} else {
// Duplicate alert
log.debug("Filtered duplicate alert: {}", fingerprint);
}
}
return uniqueAlerts;
}
public void clearCache() {
alertCache.invalidateAll();
log.info("Alert deduplication cache cleared");
}
}

6. Metrics and Monitoring

@Service
public class AlertMetricsService {
private final MeterRegistry meterRegistry;
private final Counter processedAlerts;
private final Counter failedAlerts;
private final Counter criticalAlerts;
public AlertMetricsService(MeterRegistry meterRegistry) {
this.meterRegistry = meterRegistry;
this.processedAlerts = Counter.builder("alerts.processed")
.description("Total number of processed alerts")
.register(meterRegistry);
this.failedAlerts = Counter.builder("alerts.failed")
.description("Total number of failed alert processing attempts")
.register(meterRegistry);
this.criticalAlerts = Counter.builder("alerts.critical")
.description("Total number of critical alerts")
.register(meterRegistry);
}
public void incrementProcessedAlerts(int count) {
processedAlerts.increment(count);
}
public void incrementFailedAlerts(int count) {
failedAlerts.increment(count);
}
public void incrementCriticalAlerts(int count) {
criticalAlerts.increment(count);
}
public void incrementProcessedCriticalAlerts(int count) {
criticalAlerts.increment(count);
}
public void incrementFailedCriticalAlerts(int count) {
failedAlerts.increment(count);
}
}

7. Configuration Classes

@Configuration
@ConfigurationProperties(prefix = "slack")
@Data
public class SlackConfig {
private String webhookUrl;
private String defaultChannel;
private String warningChannel;
private String criticalChannel;
}
@Configuration
@ConfigurationProperties(prefix = "alertmanager")
@Data
public class AlertmanagerConfig {
private int maxAlertsPerRequest = 100;
private int requestTimeoutSeconds = 30;
private boolean enableDeduplication = true;
private List<String> ignoredLabels = List.of("ignore");
}

8. Database Entities

@Entity
@Table(name = "alerts")
@Data
public class AlertEntity {
@Id
private String fingerprint;
@Column(nullable = false)
private String alertName;
@Column(nullable = false)
private String severity;
@Enumerated(EnumType.STRING)
@Column(nullable = false)
private AlertStatus status;
private String summary;
private String description;
@Column(columnDefinition = "JSON")
@Convert(converter = HashMapConverter.class)
private Map<String, String> labels;
@Column(columnDefinition = "JSON")
@Convert(converter = HashMapConverter.class)
private Map<String, String> annotations;
private Instant startsAt;
private Instant endsAt;
private String generatorUrl;
private Instant createdAt;
}
public enum AlertStatus {
FIRING, RESOLVED
}
@Converter
public class HashMapConverter implements AttributeConverter<Map<String, String>, String> {
private final ObjectMapper objectMapper = new ObjectMapper();
@Override
public String convertToDatabaseColumn(Map<String, String> attribute) {
try {
return objectMapper.writeValueAsString(attribute);
} catch (JsonProcessingException e) {
return "{}";
}
}
@Override
public Map<String, String> convertToEntityAttribute(String dbData) {
try {
return objectMapper.readValue(dbData, 
new TypeReference<Map<String, String>>() {});
} catch (Exception e) {
return new HashMap<>();
}
}
}

Advanced Features

1. Alert Enrichment

@Service
public class AlertEnrichmentService {
private final WebClient webClient;
public Alert enrichAlert(Alert alert) {
// Add additional context to alerts
Map<String, String> enrichedAnnotations = new HashMap<>(alert.getAnnotations());
// Add runbook URL if available
String runbookUrl = getRunbookUrl(alert.getAlertName());
if (runbookUrl != null) {
enrichedAnnotations.put("runbook", runbookUrl);
}
// Add service owner information
String serviceOwner = getServiceOwner(alert.getLabels().get("service"));
if (serviceOwner != null) {
enrichedAnnotations.put("owner", serviceOwner);
}
Alert enrichedAlert = new Alert();
// Copy all fields and set enriched annotations
// Implementation details...
return enrichedAlert;
}
private String getRunbookUrl(String alertName) {
// Lookup runbook URL from configuration or CMDB
return null; // Implementation
}
private String getServiceOwner(String serviceName) {
// Lookup service owner from service registry
return null; // Implementation
}
}

2. Rate Limiting

@Service
public class AlertRateLimitService {
private final Cache<String, RateLimitCounter> rateLimitCache;
public AlertRateLimitService() {
this.rateLimitCache = Caffeine.newBuilder()
.expireAfterWrite(1, TimeUnit.HOURS)
.maximumSize(1000)
.build();
}
public boolean isRateLimited(String alertKey, int maxAlertsPerMinute) {
RateLimitCounter counter = rateLimitCache.get(alertKey, 
k -> new RateLimitCounter());
return counter.incrementAndCheck(maxAlertsPerMinute);
}
private static class RateLimitCounter {
private final AtomicInteger count = new AtomicInteger(0);
private volatile long lastReset = System.currentTimeMillis();
public boolean incrementAndCheck(int maxPerMinute) {
resetIfNeeded();
int current = count.incrementAndGet();
return current > maxPerMinute;
}
private void resetIfNeeded() {
long now = System.currentTimeMillis();
if (now - lastReset > 60000) { // 1 minute
count.set(0);
lastReset = now;
}
}
}
}

Testing

1. Unit Tests

@ExtendWith(MockitoExtension.class)
class AlertProcessingServiceTest {
@Mock
private AlertHandler slackHandler;
@Mock
private AlertHandler databaseHandler;
@InjectMocks
private AlertProcessingService alertProcessingService;
@Test
void shouldProcessAlertsThroughAllHandlers() {
// Given
Alert alert = createTestAlert();
AlertmanagerWebhook webhook = createTestWebhook(List.of(alert));
when(slackHandler.supports(any())).thenReturn(true);
when(databaseHandler.supports(any())).thenReturn(true);
// When
alertProcessingService.processAlerts(webhook);
// Then
verify(slackHandler).handleFiringAlert(alert);
verify(databaseHandler).handleFiringAlert(alert);
}
private Alert createTestAlert() {
Alert alert = new Alert();
alert.setStatus("firing");
alert.setFingerprint("test-fingerprint");
// Set other properties...
return alert;
}
private AlertmanagerWebhook createTestWebhook(List<Alert> alerts) {
AlertmanagerWebhook webhook = new AlertmanagerWebhook();
webhook.setStatus("firing");
webhook.setAlerts(alerts);
return webhook;
}
}

2. Integration Test

@SpringBootTest
@AutoConfigureTestDatabase
class AlertmanagerWebhookControllerIntegrationTest {
@Autowired
private TestRestTemplate restTemplate;
@Autowired
private AlertRepository alertRepository;
@Test
void shouldHandleWebhookAndStoreAlert() {
// Given
AlertmanagerWebhook webhook = createTestWebhook();
// When
ResponseEntity<Void> response = restTemplate.postForEntity(
"/api/v1/alerts", webhook, Void.class);
// Then
assertThat(response.getStatusCode()).isEqualTo(HttpStatus.OK);
// Wait for async processing
await().atMost(5, TimeUnit.SECONDS)
.until(() -> !alertRepository.findAll().isEmpty());
List<AlertEntity> storedAlerts = alertRepository.findAll();
assertThat(storedAlerts).hasSize(1);
}
}

Best Practices

Idempotency: Ensure alert processing is idempotent
Async Processing: Process alerts asynchronously to avoid timeouts
Rate Limiting: Implement rate limiting to prevent overload
Deduplication: Filter duplicate alerts to reduce noise
Monitoring: Monitor your webhook handler's performance
Error Handling: Implement robust error handling and retries
Security: Validate incoming requests and consider authentication

@Component
public class WebhookSecurityFilter implements Filter {
@Value("${webhook.secret-token:}")
private String secretToken;
@Override
public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain)
throws IOException, ServletException {
if (!secretToken.isEmpty()) {
HttpServletRequest httpRequest = (HttpServletRequest) request;
String authHeader = httpRequest.getHeader("Authorization");
if (!isValidToken(authHeader)) {
((HttpServletResponse) response).sendError(HttpStatus.UNAUTHORIZED.value());
return;
}
}
chain.doFilter(request, response);
}
private boolean isValidToken(String authHeader) {
return authHeader != null && authHeader.equals("Bearer " + secretToken);
}
}

Conclusion

Alertmanager webhook integration in Java enables:

Custom alert routing to various destinations
Alert enrichment with additional context
Advanced processing logic and filtering
Integration with existing systems and workflows
Reliable alert handling with deduplication and rate limiting

By implementing the patterns shown above, you can build a robust, scalable alert handling system that integrates seamlessly with your Prometheus monitoring stack and provides intelligent alert routing and processing capabilities.