Article
ECS (Elastic Common Schema) is a standardized schema for log data that ensures consistency across different services and platforms. When using the Elastic Stack, ECS-formatted logs provide better searchability, correlation, and analysis capabilities. This guide covers implementing ECS logging in Java applications.
Why ECS Logging?
- Standardization: Consistent field names across all services
- Better Analytics: Enables correlation between different data sources
- Elasticsearch Optimization: Better performance and mapping
- Tooling Support: Works seamlessly with Kibana, Logstash, and Beats
- Future-Proof: Vendor-neutral schema
Project Setup and Dependencies
Maven Dependencies:
<properties>
<log4j2.version>2.20.0</log4j2.version>
<logback.version>1.4.11</logback.version>
<jackson.version>2.15.2</jackson.version>
<ecs-logging.version>1.5.0</ecs-logging.version>
</properties>
<dependencies>
<!-- ECS Logging Core -->
<dependency>
<groupId>co.elastic.logging</groupId>
<artifactId>ecs-logging-core</artifactId>
<version>${ecs-logging.version}</version>
</dependency>
<!-- Logback with ECS -->
<dependency>
<groupId>co.elastic.logging</groupId>
<artifactId>ecs-logging-logback</artifactId>
<version>${ecs-logging.version}</version>
</dependency>
<!-- Log4j2 with ECS -->
<dependency>
<groupId>co.elastic.logging</groupId>
<artifactId>ecs-logging-log4j2</artifactId>
<version>${ecs-logging.version}</version>
</dependency>
<!-- Jackson for JSON -->
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>${jackson.version}</version>
</dependency>
<!-- SLF4J API -->
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>2.0.7</version>
</dependency>
</dependencies>
1. ECS Logging with Logback
logback-spring.xml Configuration:
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<!-- ECS JSON Layout -->
<appender name="ECS_JSON" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="co.elastic.logging.logback.EcsEncoder">
<!-- Service information -->
<serviceName>order-service</serviceName>
<serviceVersion>1.0.0</serviceVersion>
<serviceEnvironment>production</serviceEnvironment>
<!-- Additional fields -->
<serviceNodeName>${HOSTNAME}</serviceNodeName>
<eventDataset>order-service.log</eventDataset>
<!-- Include stack traces -->
<includeStacktrace>true</includeStacktrace>
<!-- Custom fields -->
<additionalFields>
{"app.team":"platform-team","app.component":"orders"}
</additionalFields>
</encoder>
</appender>
<!-- File appender with ECS -->
<appender name="ECS_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>logs/order-service-ecs.json</file>
<encoder class="co.elastic.logging.logback.EcsEncoder">
<serviceName>order-service</serviceName>
<serviceVersion>1.0.0</serviceVersion>
<serviceEnvironment>production</serviceEnvironment>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>logs/order-service-ecs.%d{yyyy-MM-dd}.json</fileNamePattern>
<maxHistory>30</maxHistory>
</rollingPolicy>
</appender>
<!-- Async appender for better performance -->
<appender name="ASYNC_ECS" class="ch.qos.logback.classic.AsyncAppender">
<appender-ref ref="ECS_JSON" />
<queueSize>1024</queueSize>
<discardingThreshold>0</discardingThreshold>
<includeCallerData>true</includeCallerData>
</appender>
<root level="INFO">
<appender-ref ref="ASYNC_ECS" />
<appender-ref ref="ECS_FILE" />
</root>
<!-- Application-specific logger -->
<logger name="com.example.orders" level="DEBUG" additivity="false">
<appender-ref ref="ASYNC_ECS" />
</logger>
</configuration>
2. ECS Logging with Log4j2
log4j2-ecs.xml Configuration:
<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="WARN">
<Appenders>
<!-- ECS JSON Console Appender -->
<Console name="ECSConsole" target="SYSTEM_OUT">
<EcsLayout serviceName="order-service"
serviceVersion="1.0.0"
serviceEnvironment="${sys:ENVIRONMENT:-development}"
serviceNodeName="${hostName}"
eventDataset="order-service.log"
includeMarkers="true"
includeOrigin="true">
<KeyValuePair key="app.team" value="platform-team"/>
<KeyValuePair key="app.component" value="orders"/>
</EcsLayout>
</Console>
<!-- ECS JSON File Appender -->
<File name="ECSFile" fileName="logs/order-service-ecs.json">
<EcsLayout serviceName="order-service"
serviceVersion="1.0.0"
serviceEnvironment="${sys:ENVIRONMENT:-development}"/>
</File>
<!-- Async Appender -->
<Async name="AsyncEcs">
<AppenderRef ref="ECSConsole"/>
</Async>
</Appenders>
<Loggers>
<Root level="info">
<AppenderRef ref="AsyncEcs"/>
<AppenderRef ref="ECSFile"/>
</Root>
</Loggers>
</Configuration>
3. Custom ECS Logger Implementation
ECS Log Event Builder:
package com.example.ecs.logging;
import com.fasterxml.jackson.core.JsonGenerator;
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.SerializerProvider;
import com.fasterxml.jackson.databind.annotation.JsonSerialize;
import com.fasterxml.jackson.databind.ser.std.StdSerializer;
import java.io.IOException;
import java.time.Instant;
import java.util.*;
@JsonSerialize(using = EcsLogEvent.EcsLogEventSerializer.class)
public class EcsLogEvent {
private final Instant timestamp;
private final String logLevel;
private final String message;
private final String loggerName;
private final String threadName;
private final Map<String, Object> ecsFields;
private final Map<String, Object> customFields;
private Throwable exception;
private EcsLogEvent(Builder builder) {
this.timestamp = builder.timestamp != null ? builder.timestamp : Instant.now();
this.logLevel = builder.logLevel;
this.message = builder.message;
this.loggerName = builder.loggerName;
this.threadName = builder.threadName != null ? builder.threadName : Thread.currentThread().getName();
this.ecsFields = new HashMap<>(builder.ecsFields);
this.customFields = new HashMap<>(builder.customFields);
this.exception = builder.exception;
// Set ECS required fields
this.ecsFields.putIfAbsent("@timestamp", timestamp);
this.ecsFields.putIfAbsent("log.level", logLevel);
this.ecsFields.putIfAbsent("message", message);
}
public static class Builder {
private Instant timestamp;
private String logLevel;
private String message;
private String loggerName;
private String threadName;
private Map<String, Object> ecsFields = new HashMap<>();
private Map<String, Object> customFields = new HashMap<>();
private Throwable exception;
public Builder(String logLevel, String message) {
this.logLevel = logLevel;
this.message = message;
}
public Builder timestamp(Instant timestamp) {
this.timestamp = timestamp;
return this;
}
public Builder loggerName(String loggerName) {
this.loggerName = loggerName;
return this;
}
public Builder threadName(String threadName) {
this.threadName = threadName;
return this;
}
public Builder exception(Throwable exception) {
this.exception = exception;
return this;
}
// ECS Field methods
public Builder serviceName(String serviceName) {
this.ecsFields.put("service.name", serviceName);
return this;
}
public Builder serviceVersion(String serviceVersion) {
this.ecsFields.put("service.version", serviceVersion);
return this;
}
public Builder serviceEnvironment(String environment) {
this.ecsFields.put("service.environment", environment);
return this;
}
public Builder serviceNodeName(String nodeName) {
this.ecsFields.put("service.node.name", nodeName);
return this;
}
public Builder eventDataset(String dataset) {
this.ecsFields.put("event.dataset", dataset);
return this;
}
public Builder traceId(String traceId) {
this.ecsFields.put("trace.id", traceId);
return this;
}
public Builder transactionId(String transactionId) {
this.ecsFields.put("transaction.id", transactionId);
return this;
}
public Builder httpRequest(String method, String url) {
Map<String, Object> http = new HashMap<>();
http.put("request.method", method);
if (url != null) {
// Extract path from URL
try {
java.net.URI uri = new java.net.URI(url);
http.put("request.path", uri.getPath());
} catch (Exception e) {
http.put("request.path", url);
}
}
this.ecsFields.put("http", http);
return this;
}
public Builder httpResponse(int statusCode, long responseTime) {
Map<String, Object> http = (Map<String, Object>) this.ecsFields.getOrDefault("http", new HashMap<>());
http.put("response.status_code", statusCode);
http.put("response.body.bytes", responseTime);
this.ecsFields.put("http", http);
return this;
}
public Builder user(String userId, String username) {
Map<String, Object> user = new HashMap<>();
if (userId != null) user.put("id", userId);
if (username != null) user.put("name", username);
this.ecsFields.put("user", user);
return this;
}
// Custom field methods
public Builder customField(String key, Object value) {
this.customFields.put(key, value);
return this;
}
public Builder businessContext(String orderId, String customerId, String action) {
Map<String, Object> business = new HashMap<>();
if (orderId != null) business.put("order.id", orderId);
if (customerId != null) business.put("customer.id", customerId);
if (action != null) business.put("action", action);
this.customFields.put("business", business);
return this;
}
public EcsLogEvent build() {
return new EcsLogEvent(this);
}
}
// Custom serializer for ECS format
public static class EcsLogEventSerializer extends StdSerializer<EcsLogEvent> {
private final ObjectMapper objectMapper = new ObjectMapper();
public EcsLogEventSerializer() {
this(null);
}
public EcsLogEventSerializer(Class<EcsLogEvent> t) {
super(t);
}
@Override
public void serialize(EcsLogEvent value, JsonGenerator gen, SerializerProvider provider)
throws IOException {
gen.writeStartObject();
// Write ECS fields first
for (Map.Entry<String, Object> entry : value.ecsFields.entrySet()) {
gen.writeObjectField(entry.getKey(), entry.getValue());
}
// Write error/exception information
if (value.exception != null) {
gen.writeObjectField("error.type", value.exception.getClass().getName());
gen.writeObjectField("error.message", value.exception.getMessage());
gen.writeObjectField("error.stack_trace", getStackTrace(value.exception));
}
// Write log-specific fields
gen.writeObjectField("log.logger", value.loggerName);
// Write custom fields under 'labels' or directly
if (!value.customFields.isEmpty()) {
gen.writeObjectField("labels", value.customFields);
}
gen.writeEndObject();
}
private String getStackTrace(Throwable throwable) {
if (throwable == null) return null;
StringWriter sw = new StringWriter();
throwable.printStackTrace(new PrintWriter(sw));
return sw.toString();
}
}
// Getters
public Instant getTimestamp() { return timestamp; }
public String getLogLevel() { return logLevel; }
public String getMessage() { return message; }
public Map<String, Object> getEcsFields() { return Collections.unmodifiableMap(ecsFields); }
public Map<String, Object> getCustomFields() { return Collections.unmodifiableMap(customFields); }
}
4. ECS Logger Utility
ECS Logger Service:
package com.example.ecs.logging;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.slf4j.MDC;
import org.springframework.stereotype.Service;
import java.time.Instant;
import java.util.HashMap;
import java.util.Map;
@Service
public class EcsLogger {
private static final ObjectMapper objectMapper = new ObjectMapper();
private final String serviceName;
private final String serviceVersion;
private final String serviceEnvironment;
public EcsLogger() {
this.serviceName = System.getProperty("service.name", "unknown-service");
this.serviceVersion = System.getProperty("service.version", "1.0.0");
this.serviceEnvironment = System.getProperty("service.environment", "development");
}
public void info(String message, Map<String, Object> customFields) {
log("INFO", message, customFields, null);
}
public void error(String message, Map<String, Object> customFields, Throwable exception) {
log("ERROR", message, customFields, exception);
}
public void warn(String message, Map<String, Object> customFields) {
log("WARN", message, customFields, null);
}
public void debug(String message, Map<String, Object> customFields) {
log("DEBUG", message, customFields, null);
}
// Structured logging methods
public void httpRequest(String method, String path, String userAgent, String clientIp) {
Map<String, Object> fields = new HashMap<>();
Map<String, Object> http = new HashMap<>();
http.put("request.method", method);
http.put("request.path", path);
if (userAgent != null) http.put("request.headers.user_agent", userAgent);
Map<String, Object> client = new HashMap<>();
if (clientIp != null) client.put("ip", clientIp);
fields.put("http", http);
if (!client.isEmpty()) fields.put("client", client);
info("HTTP request: " + method + " " + path, fields);
}
public void httpResponse(String method, String path, int statusCode, long durationMs) {
Map<String, Object> fields = new HashMap<>();
Map<String, Object> http = new HashMap<>();
http.put("request.method", method);
http.put("request.path", path);
http.put("response.status_code", statusCode);
http.put("event.duration", durationMs * 1_000_000); // Convert to nanoseconds
fields.put("http", http);
fields.put("event.action", "http_response");
String level = statusCode >= 400 ? "WARN" : "INFO";
log(level, "HTTP response: " + statusCode + " for " + method + " " + path, fields, null);
}
public void businessEvent(String eventType, String entityId, String action, Map<String, Object> details) {
Map<String, Object> fields = new HashMap<>();
fields.put("event.action", action);
fields.put("event.category", "business");
Map<String, Object> business = new HashMap<>();
business.put("event.type", eventType);
business.put("entity.id", entityId);
if (details != null) business.putAll(details);
fields.put("business", business);
info("Business event: " + eventType + " - " + action, fields);
}
private void log(String level, String message, Map<String, Object> customFields, Throwable exception) {
try {
EcsLogEvent logEvent = new EcsLogEvent.Builder(level, message)
.timestamp(Instant.now())
.serviceName(serviceName)
.serviceVersion(serviceVersion)
.serviceEnvironment(serviceEnvironment)
.serviceNodeName(System.getenv("HOSTNAME"))
.eventDataset(serviceName + ".log")
.loggerName(getCallerClassName())
.exception(exception)
.traceId(MDC.get("traceId"))
.transactionId(MDC.get("transactionId"))
.build();
// Add custom fields
if (customFields != null) {
customFields.forEach((key, value) ->
logEvent.getCustomFields().put(key, value));
}
// Convert to JSON and log using SLF4J
String jsonLog = objectMapper.writeValueAsString(logEvent);
Logger slf4jLogger = LoggerFactory.getLogger(getCallerClassName());
switch (level) {
case "ERROR":
slf4jLogger.error(jsonLog);
break;
case "WARN":
slf4jLogger.warn(jsonLog);
break;
case "DEBUG":
slf4jLogger.debug(jsonLog);
break;
default:
slf4jLogger.info(jsonLog);
}
} catch (JsonProcessingException e) {
// Fallback to regular logging
LoggerFactory.getLogger(EcsLogger.class)
.error("Failed to serialize ECS log event", e);
}
}
private String getCallerClassName() {
StackTraceElement[] stElements = Thread.currentThread().getStackTrace();
for (int i = 1; i < stElements.length; i++) {
StackTraceElement ste = stElements[i];
if (!ste.getClassName().equals(EcsLogger.class.getName()) &&
!ste.getClassName().equals(Thread.class.getName())) {
return ste.getClassName();
}
}
return "unknown";
}
}
5. Spring Boot Configuration
ECS Logging Auto-Configuration:
package com.example.ecs.config;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.web.filter.CommonsRequestLoggingFilter;
import javax.servlet.Filter;
@Configuration
@EnableConfigurationProperties(EcsLoggingProperties.class)
public class EcsLoggingAutoConfiguration {
@Bean
public EcsLogger ecsLogger() {
return new EcsLogger();
}
@Bean
public Filter requestLoggingFilter() {
CommonsRequestLoggingFilter filter = new CommonsRequestLoggingFilter() {
@Override
protected void beforeRequest(javax.servlet.http.HttpServletRequest request, String message) {
// MDC setup for tracing
String traceId = request.getHeader("X-Trace-Id");
if (traceId != null) {
MDC.put("traceId", traceId);
}
}
@Override
protected void afterRequest(javax.servlet.http.HttpServletRequest request, String message) {
MDC.clear();
}
};
filter.setIncludeQueryString(true);
filter.setIncludePayload(false);
filter.setIncludeHeaders(true);
return filter;
}
}
@ConfigurationProperties(prefix = "ecs.logging")
class EcsLoggingProperties {
private String serviceName = "unknown-service";
private String serviceVersion = "1.0.0";
private String serviceEnvironment = "development";
private boolean enabled = true;
// Getters and setters
}
6. Spring Boot Web Integration
Request/Response Logging Interceptor:
package com.example.ecs.web;
import com.example.ecs.logging.EcsLogger;
import org.slf4j.MDC;
import org.springframework.stereotype.Component;
import org.springframework.web.servlet.HandlerInterceptor;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.util.UUID;
@Component
public class EcsLoggingInterceptor implements HandlerInterceptor {
private final EcsLogger ecsLogger;
public EcsLoggingInterceptor(EcsLogger ecsLogger) {
this.ecsLogger = ecsLogger;
}
@Override
public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) {
// Generate or extract trace ID
String traceId = extractTraceId(request);
MDC.put("traceId", traceId);
MDC.put("transactionId", UUID.randomUUID().toString());
// Log request
ecsLogger.httpRequest(
request.getMethod(),
request.getRequestURI(),
request.getHeader("User-Agent"),
getClientIp(request)
);
request.setAttribute("startTime", System.currentTimeMillis());
return true;
}
@Override
public void afterCompletion(HttpServletRequest request, HttpServletResponse response,
Object handler, Exception ex) {
Long startTime = (Long) request.getAttribute("startTime");
long duration = startTime != null ? System.currentTimeMillis() - startTime : 0;
// Log response
ecsLogger.httpResponse(
request.getMethod(),
request.getRequestURI(),
response.getStatus(),
duration
);
// Log errors
if (ex != null) {
ecsLogger.error("Request processing failed", Map.of(
"http.request.method", request.getMethod(),
"http.request.path", request.getRequestURI()
), ex);
}
MDC.clear();
}
private String extractTraceId(HttpServletRequest request) {
String traceId = request.getHeader("X-Trace-Id");
if (traceId == null) {
traceId = request.getHeader("X-Request-Id");
}
return traceId != null ? traceId : UUID.randomUUID().toString();
}
private String getClientIp(HttpServletRequest request) {
String xForwardedFor = request.getHeader("X-Forwarded-For");
if (xForwardedFor != null && !xForwardedFor.isEmpty()) {
return xForwardedFor.split(",")[0].trim();
}
return request.getRemoteAddr();
}
}
7. Application Usage Examples
Service Class with ECS Logging:
package com.example.orders.service;
import com.example.ecs.logging.EcsLogger;
import org.springframework.stereotype.Service;
import java.util.Map;
@Service
public class OrderService {
private final EcsLogger ecsLogger;
public OrderService(EcsLogger ecsLogger) {
this.ecsLogger = ecsLogger;
}
public Order createOrder(CreateOrderRequest request) {
try {
ecsLogger.businessEvent("order_creation", null, "start", Map.of(
"customer.id", request.getCustomerId(),
"order.total", request.getTotalAmount()
));
// Business logic
Order order = processOrder(request);
ecsLogger.businessEvent("order_creation", order.getId(), "success", Map.of(
"order.status", order.getStatus(),
"order.items.count", order.getItems().size()
));
return order;
} catch (Exception e) {
ecsLogger.error("Order creation failed", Map.of(
"customer.id", request.getCustomerId(),
"order.total", request.getTotalAmount()
), e);
throw e;
}
}
public void processPayment(String orderId, Payment payment) {
ecsLogger.info("Processing payment", Map.of(
"order.id", orderId,
"payment.amount", payment.getAmount(),
"payment.method", payment.getMethod(),
"business.operation", "payment_processing"
));
// Payment processing logic
}
}
8. Testing ECS Logging
Test Configuration:
package com.example.ecs.test;
import com.example.ecs.logging.EcsLogger;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.junit.jupiter.api.Test;
import org.springframework.boot.test.context.SpringBootTest;
import java.io.ByteArrayOutputStream;
import java.io.PrintStream;
import java.util.Map;
import static org.assertj.core.api.Assertions.assertThat;
@SpringBootTest
class EcsLoggingTest {
private final ObjectMapper objectMapper = new ObjectMapper();
@Test
void testEcsLogStructure() throws Exception {
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
PrintStream originalOut = System.out;
System.setOut(new PrintStream(outputStream));
EcsLogger logger = new EcsLogger();
logger.info("Test message", Map.of("test.field", "value"));
System.setOut(originalOut);
String logLine = outputStream.toString().trim();
JsonNode logJson = objectMapper.readTree(logLine);
// Verify ECS structure
assertThat(logJson.has("@timestamp")).isTrue();
assertThat(logJson.has("log.level")).isTrue();
assertThat(logJson.has("message")).isTrue();
assertThat(logJson.has("service.name")).isTrue();
assertThat(logJson.get("message").asText()).isEqualTo("Test message");
assertThat(logJson.get("labels").get("test.field").asText()).isEqualTo("value");
}
}
9. Docker and Kubernetes Configuration
Dockerfile with ECS Logging:
FROM openjdk:17-jre-slim # Set ECS logging properties ENV SERVICE_NAME=order-service ENV SERVICE_VERSION=1.0.0 ENV SERVICE_ENVIRONMENT=production # Copy application COPY target/order-service.jar /app/ COPY src/main/resources/logback-spring.xml /app/config/ WORKDIR /app CMD ["java", "-jar", "order-service.jar", "--logging.config=config/logback-spring.xml"]
Kubernetes ConfigMap for Logging:
apiVersion: v1
kind: ConfigMap
metadata:
name: ecs-logging-config
data:
logback-spring.xml: |
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<appender name="ECS_JSON" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="co.elastic.logging.logback.EcsEncoder">
<serviceName>${SERVICE_NAME}</serviceName>
<serviceVersion>${SERVICE_VERSION}</serviceVersion>
<serviceEnvironment>${SERVICE_ENVIRONMENT}</serviceEnvironment>
</encoder>
</appender>
<root level="INFO">
<appender-ref ref="ECS_JSON" />
</root>
</configuration>
10. Best Practices
Field Naming Conventions:
- Use ECS field names when available
- Prefix custom fields with domain (e.g.,
business.,app.) - Use snake_case for all field names
Performance Considerations:
- Use async appenders in production
- Avoid logging large objects in hot paths
- Use structured logging judiciously
Security:
- Don't log sensitive information (passwords, tokens, PII)
- Use field-level redaction if needed
public Map<String, Object> sanitizeFields(Map<String, Object> fields) {
Map<String, Object> sanitized = new HashMap<>(fields);
sanitized.keySet().removeIf(key ->
key.contains("password") || key.contains("token") || key.contains("secret"));
return sanitized;
}
Conclusion
ECS logging in Java provides a standardized, powerful way to structure your log data for better observability. Key benefits include:
- Consistent Schema: Standard field names across all services
- Better Analytics: Enables powerful Kibana visualizations and correlations
- Operational Excellence: Improved debugging and monitoring capabilities
- Future-Proof: Vendor-neutral approach
By implementing ECS logging with the patterns shown above, you'll create maintainable, searchable, and analyzable logs that work seamlessly with the Elastic Stack and other observability tools.