Article

ECS (Elastic Common Schema) is a standardized schema for log data that ensures consistency across different services and platforms. When using the Elastic Stack, ECS-formatted logs provide better searchability, correlation, and analysis capabilities. This guide covers implementing ECS logging in Java applications.

Why ECS Logging?

• Standardization: Consistent field names across all services
• Better Analytics: Enables correlation between different data sources
• Elasticsearch Optimization: Better performance and mapping
• Tooling Support: Works seamlessly with Kibana, Logstash, and Beats
• Future-Proof: Vendor-neutral schema

Project Setup and Dependencies

Maven Dependencies:

<properties> <log4j2.version>2.20.0</log4j2.version> <logback.version>1.4.11</logback.version> <jackson.version>2.15.2</jackson.version> <ecs-logging.version>1.5.0</ecs-logging.version> </properties> <dependencies> <!-- ECS Logging Core --> <dependency> <groupId>co.elastic.logging</groupId> <artifactId>ecs-logging-core</artifactId> <version>${ecs-logging.version}</version> </dependency> <!-- Logback with ECS --> <dependency> <groupId>co.elastic.logging</groupId> <artifactId>ecs-logging-logback</artifactId> <version>${ecs-logging.version}</version> </dependency> <!-- Log4j2 with ECS --> <dependency> <groupId>co.elastic.logging</groupId> <artifactId>ecs-logging-log4j2</artifactId> <version>${ecs-logging.version}</version> </dependency> <!-- Jackson for JSON --> <dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-databind</artifactId> <version>${jackson.version}</version> </dependency> <!-- SLF4J API --> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-api</artifactId> <version>2.0.7</version> </dependency> </dependencies>

1. ECS Logging with Logback

logback-spring.xml Configuration:

<?xml version="1.0" encoding="UTF-8"?> <configuration> <!-- ECS JSON Layout --> <appender name="ECS_JSON" class="ch.qos.logback.core.ConsoleAppender"> <encoder class="co.elastic.logging.logback.EcsEncoder"> <!-- Service information --> <serviceName>order-service</serviceName> <serviceVersion>1.0.0</serviceVersion> <serviceEnvironment>production</serviceEnvironment> <!-- Additional fields --> <serviceNodeName>${HOSTNAME}</serviceNodeName> <eventDataset>order-service.log</eventDataset> <!-- Include stack traces --> <includeStacktrace>true</includeStacktrace> <!-- Custom fields --> <additionalFields> {"app.team":"platform-team","app.component":"orders"} </additionalFields> </encoder> </appender> <!-- File appender with ECS --> <appender name="ECS_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender"> <file>logs/order-service-ecs.json</file> <encoder class="co.elastic.logging.logback.EcsEncoder"> <serviceName>order-service</serviceName> <serviceVersion>1.0.0</serviceVersion> <serviceEnvironment>production</serviceEnvironment> </encoder> <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"> <fileNamePattern>logs/order-service-ecs.%d{yyyy-MM-dd}.json</fileNamePattern> <maxHistory>30</maxHistory> </rollingPolicy> </appender> <!-- Async appender for better performance --> <appender name="ASYNC_ECS" class="ch.qos.logback.classic.AsyncAppender"> <appender-ref ref="ECS_JSON" /> <queueSize>1024</queueSize> <discardingThreshold>0</discardingThreshold> <includeCallerData>true</includeCallerData> </appender> <root level="INFO"> <appender-ref ref="ASYNC_ECS" /> <appender-ref ref="ECS_FILE" /> </root> <!-- Application-specific logger --> <logger name="com.example.orders" level="DEBUG" additivity="false"> <appender-ref ref="ASYNC_ECS" /> </logger> </configuration>

2. ECS Logging with Log4j2

log4j2-ecs.xml Configuration:

<?xml version="1.0" encoding="UTF-8"?> <Configuration status="WARN"> <Appenders> <!-- ECS JSON Console Appender --> <Console name="ECSConsole" target="SYSTEM_OUT"> <EcsLayout serviceName="order-service" serviceVersion="1.0.0" serviceEnvironment="${sys:ENVIRONMENT:-development}" serviceNodeName="${hostName}" eventDataset="order-service.log" includeMarkers="true" includeOrigin="true"> <KeyValuePair key="app.team" value="platform-team"/> <KeyValuePair key="app.component" value="orders"/> </EcsLayout> </Console> <!-- ECS JSON File Appender --> <File name="ECSFile" fileName="logs/order-service-ecs.json"> <EcsLayout serviceName="order-service" serviceVersion="1.0.0" serviceEnvironment="${sys:ENVIRONMENT:-development}"/> </File> <!-- Async Appender --> <Async name="AsyncEcs"> <AppenderRef ref="ECSConsole"/> </Async> </Appenders> <Loggers> <Root level="info"> <AppenderRef ref="AsyncEcs"/> <AppenderRef ref="ECSFile"/> </Root> </Loggers> </Configuration>

3. Custom ECS Logger Implementation

ECS Log Event Builder:

package com.example.ecs.logging; import com.fasterxml.jackson.core.JsonGenerator; import com.fasterxml.jackson.databind.ObjectMapper; import com.fasterxml.jackson.databind.SerializerProvider; import com.fasterxml.jackson.databind.annotation.JsonSerialize; import com.fasterxml.jackson.databind.ser.std.StdSerializer; import java.io.IOException; import java.time.Instant; import java.util.*; @JsonSerialize(using = EcsLogEvent.EcsLogEventSerializer.class) public class EcsLogEvent { private final Instant timestamp; private final String logLevel; private final String message; private final String loggerName; private final String threadName; private final Map<String, Object> ecsFields; private final Map<String, Object> customFields; private Throwable exception; private EcsLogEvent(Builder builder) { this.timestamp = builder.timestamp != null ? builder.timestamp : Instant.now(); this.logLevel = builder.logLevel; this.message = builder.message; this.loggerName = builder.loggerName; this.threadName = builder.threadName != null ? builder.threadName : Thread.currentThread().getName(); this.ecsFields = new HashMap<>(builder.ecsFields); this.customFields = new HashMap<>(builder.customFields); this.exception = builder.exception; // Set ECS required fields this.ecsFields.putIfAbsent("@timestamp", timestamp); this.ecsFields.putIfAbsent("log.level", logLevel); this.ecsFields.putIfAbsent("message", message); } public static class Builder { private Instant timestamp; private String logLevel; private String message; private String loggerName; private String threadName; private Map<String, Object> ecsFields = new HashMap<>(); private Map<String, Object> customFields = new HashMap<>(); private Throwable exception; public Builder(String logLevel, String message) { this.logLevel = logLevel; this.message = message; } public Builder timestamp(Instant timestamp) { this.timestamp = timestamp; return this; } public Builder loggerName(String loggerName) { this.loggerName = loggerName; return this; } public Builder threadName(String threadName) { this.threadName = threadName; return this; } public Builder exception(Throwable exception) { this.exception = exception; return this; } // ECS Field methods public Builder serviceName(String serviceName) { this.ecsFields.put("service.name", serviceName); return this; } public Builder serviceVersion(String serviceVersion) { this.ecsFields.put("service.version", serviceVersion); return this; } public Builder serviceEnvironment(String environment) { this.ecsFields.put("service.environment", environment); return this; } public Builder serviceNodeName(String nodeName) { this.ecsFields.put("service.node.name", nodeName); return this; } public Builder eventDataset(String dataset) { this.ecsFields.put("event.dataset", dataset); return this; } public Builder traceId(String traceId) { this.ecsFields.put("trace.id", traceId); return this; } public Builder transactionId(String transactionId) { this.ecsFields.put("transaction.id", transactionId); return this; } public Builder httpRequest(String method, String url) { Map<String, Object> http = new HashMap<>(); http.put("request.method", method); if (url != null) { // Extract path from URL try { java.net.URI uri = new java.net.URI(url); http.put("request.path", uri.getPath()); } catch (Exception e) { http.put("request.path", url); } } this.ecsFields.put("http", http); return this; } public Builder httpResponse(int statusCode, long responseTime) { Map<String, Object> http = (Map<String, Object>) this.ecsFields.getOrDefault("http", new HashMap<>()); http.put("response.status_code", statusCode); http.put("response.body.bytes", responseTime); this.ecsFields.put("http", http); return this; } public Builder user(String userId, String username) { Map<String, Object> user = new HashMap<>(); if (userId != null) user.put("id", userId); if (username != null) user.put("name", username); this.ecsFields.put("user", user); return this; } // Custom field methods public Builder customField(String key, Object value) { this.customFields.put(key, value); return this; } public Builder businessContext(String orderId, String customerId, String action) { Map<String, Object> business = new HashMap<>(); if (orderId != null) business.put("order.id", orderId); if (customerId != null) business.put("customer.id", customerId); if (action != null) business.put("action", action); this.customFields.put("business", business); return this; } public EcsLogEvent build() { return new EcsLogEvent(this); } } // Custom serializer for ECS format public static class EcsLogEventSerializer extends StdSerializer<EcsLogEvent> { private final ObjectMapper objectMapper = new ObjectMapper(); public EcsLogEventSerializer() { this(null); } public EcsLogEventSerializer(Class<EcsLogEvent> t) { super(t); } @Override public void serialize(EcsLogEvent value, JsonGenerator gen, SerializerProvider provider) throws IOException { gen.writeStartObject(); // Write ECS fields first for (Map.Entry<String, Object> entry : value.ecsFields.entrySet()) { gen.writeObjectField(entry.getKey(), entry.getValue()); } // Write error/exception information if (value.exception != null) { gen.writeObjectField("error.type", value.exception.getClass().getName()); gen.writeObjectField("error.message", value.exception.getMessage()); gen.writeObjectField("error.stack_trace", getStackTrace(value.exception)); } // Write log-specific fields gen.writeObjectField("log.logger", value.loggerName); // Write custom fields under 'labels' or directly if (!value.customFields.isEmpty()) { gen.writeObjectField("labels", value.customFields); } gen.writeEndObject(); } private String getStackTrace(Throwable throwable) { if (throwable == null) return null; StringWriter sw = new StringWriter(); throwable.printStackTrace(new PrintWriter(sw)); return sw.toString(); } } // Getters public Instant getTimestamp() { return timestamp; } public String getLogLevel() { return logLevel; } public String getMessage() { return message; } public Map<String, Object> getEcsFields() { return Collections.unmodifiableMap(ecsFields); } public Map<String, Object> getCustomFields() { return Collections.unmodifiableMap(customFields); } }

4. ECS Logger Utility

ECS Logger Service:

package com.example.ecs.logging; import com.fasterxml.jackson.core.JsonProcessingException; import com.fasterxml.jackson.databind.ObjectMapper; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.slf4j.MDC; import org.springframework.stereotype.Service; import java.time.Instant; import java.util.HashMap; import java.util.Map; @Service public class EcsLogger { private static final ObjectMapper objectMapper = new ObjectMapper(); private final String serviceName; private final String serviceVersion; private final String serviceEnvironment; public EcsLogger() { this.serviceName = System.getProperty("service.name", "unknown-service"); this.serviceVersion = System.getProperty("service.version", "1.0.0"); this.serviceEnvironment = System.getProperty("service.environment", "development"); } public void info(String message, Map<String, Object> customFields) { log("INFO", message, customFields, null); } public void error(String message, Map<String, Object> customFields, Throwable exception) { log("ERROR", message, customFields, exception); } public void warn(String message, Map<String, Object> customFields) { log("WARN", message, customFields, null); } public void debug(String message, Map<String, Object> customFields) { log("DEBUG", message, customFields, null); } // Structured logging methods public void httpRequest(String method, String path, String userAgent, String clientIp) { Map<String, Object> fields = new HashMap<>(); Map<String, Object> http = new HashMap<>(); http.put("request.method", method); http.put("request.path", path); if (userAgent != null) http.put("request.headers.user_agent", userAgent); Map<String, Object> client = new HashMap<>(); if (clientIp != null) client.put("ip", clientIp); fields.put("http", http); if (!client.isEmpty()) fields.put("client", client); info("HTTP request: " + method + " " + path, fields); } public void httpResponse(String method, String path, int statusCode, long durationMs) { Map<String, Object> fields = new HashMap<>(); Map<String, Object> http = new HashMap<>(); http.put("request.method", method); http.put("request.path", path); http.put("response.status_code", statusCode); http.put("event.duration", durationMs * 1_000_000); // Convert to nanoseconds fields.put("http", http); fields.put("event.action", "http_response"); String level = statusCode >= 400 ? "WARN" : "INFO"; log(level, "HTTP response: " + statusCode + " for " + method + " " + path, fields, null); } public void businessEvent(String eventType, String entityId, String action, Map<String, Object> details) { Map<String, Object> fields = new HashMap<>(); fields.put("event.action", action); fields.put("event.category", "business"); Map<String, Object> business = new HashMap<>(); business.put("event.type", eventType); business.put("entity.id", entityId); if (details != null) business.putAll(details); fields.put("business", business); info("Business event: " + eventType + " - " + action, fields); } private void log(String level, String message, Map<String, Object> customFields, Throwable exception) { try { EcsLogEvent logEvent = new EcsLogEvent.Builder(level, message) .timestamp(Instant.now()) .serviceName(serviceName) .serviceVersion(serviceVersion) .serviceEnvironment(serviceEnvironment) .serviceNodeName(System.getenv("HOSTNAME")) .eventDataset(serviceName + ".log") .loggerName(getCallerClassName()) .exception(exception) .traceId(MDC.get("traceId")) .transactionId(MDC.get("transactionId")) .build(); // Add custom fields if (customFields != null) { customFields.forEach((key, value) -> logEvent.getCustomFields().put(key, value)); } // Convert to JSON and log using SLF4J String jsonLog = objectMapper.writeValueAsString(logEvent); Logger slf4jLogger = LoggerFactory.getLogger(getCallerClassName()); switch (level) { case "ERROR": slf4jLogger.error(jsonLog); break; case "WARN": slf4jLogger.warn(jsonLog); break; case "DEBUG": slf4jLogger.debug(jsonLog); break; default: slf4jLogger.info(jsonLog); } } catch (JsonProcessingException e) { // Fallback to regular logging LoggerFactory.getLogger(EcsLogger.class) .error("Failed to serialize ECS log event", e); } } private String getCallerClassName() { StackTraceElement[] stElements = Thread.currentThread().getStackTrace(); for (int i = 1; i < stElements.length; i++) { StackTraceElement ste = stElements[i]; if (!ste.getClassName().equals(EcsLogger.class.getName()) && !ste.getClassName().equals(Thread.class.getName())) { return ste.getClassName(); } } return "unknown"; } }

5. Spring Boot Configuration

ECS Logging Auto-Configuration:

package com.example.ecs.config; import org.springframework.boot.context.properties.EnableConfigurationProperties; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.web.filter.CommonsRequestLoggingFilter; import javax.servlet.Filter; @Configuration @EnableConfigurationProperties(EcsLoggingProperties.class) public class EcsLoggingAutoConfiguration { @Bean public EcsLogger ecsLogger() { return new EcsLogger(); } @Bean public Filter requestLoggingFilter() { CommonsRequestLoggingFilter filter = new CommonsRequestLoggingFilter() { @Override protected void beforeRequest(javax.servlet.http.HttpServletRequest request, String message) { // MDC setup for tracing String traceId = request.getHeader("X-Trace-Id"); if (traceId != null) { MDC.put("traceId", traceId); } } @Override protected void afterRequest(javax.servlet.http.HttpServletRequest request, String message) { MDC.clear(); } }; filter.setIncludeQueryString(true); filter.setIncludePayload(false); filter.setIncludeHeaders(true); return filter; } } @ConfigurationProperties(prefix = "ecs.logging") class EcsLoggingProperties { private String serviceName = "unknown-service"; private String serviceVersion = "1.0.0"; private String serviceEnvironment = "development"; private boolean enabled = true; // Getters and setters }

6. Spring Boot Web Integration

Request/Response Logging Interceptor:

package com.example.ecs.web; import com.example.ecs.logging.EcsLogger; import org.slf4j.MDC; import org.springframework.stereotype.Component; import org.springframework.web.servlet.HandlerInterceptor; import javax.servlet.http.HttpServletRequest; import javax.servlet.http.HttpServletResponse; import java.util.UUID; @Component public class EcsLoggingInterceptor implements HandlerInterceptor { private final EcsLogger ecsLogger; public EcsLoggingInterceptor(EcsLogger ecsLogger) { this.ecsLogger = ecsLogger; } @Override public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) { // Generate or extract trace ID String traceId = extractTraceId(request); MDC.put("traceId", traceId); MDC.put("transactionId", UUID.randomUUID().toString()); // Log request ecsLogger.httpRequest( request.getMethod(), request.getRequestURI(), request.getHeader("User-Agent"), getClientIp(request) ); request.setAttribute("startTime", System.currentTimeMillis()); return true; } @Override public void afterCompletion(HttpServletRequest request, HttpServletResponse response, Object handler, Exception ex) { Long startTime = (Long) request.getAttribute("startTime"); long duration = startTime != null ? System.currentTimeMillis() - startTime : 0; // Log response ecsLogger.httpResponse( request.getMethod(), request.getRequestURI(), response.getStatus(), duration ); // Log errors if (ex != null) { ecsLogger.error("Request processing failed", Map.of( "http.request.method", request.getMethod(), "http.request.path", request.getRequestURI() ), ex); } MDC.clear(); } private String extractTraceId(HttpServletRequest request) { String traceId = request.getHeader("X-Trace-Id"); if (traceId == null) { traceId = request.getHeader("X-Request-Id"); } return traceId != null ? traceId : UUID.randomUUID().toString(); } private String getClientIp(HttpServletRequest request) { String xForwardedFor = request.getHeader("X-Forwarded-For"); if (xForwardedFor != null && !xForwardedFor.isEmpty()) { return xForwardedFor.split(",")[0].trim(); } return request.getRemoteAddr(); } }

7. Application Usage Examples

Service Class with ECS Logging:

package com.example.orders.service; import com.example.ecs.logging.EcsLogger; import org.springframework.stereotype.Service; import java.util.Map; @Service public class OrderService { private final EcsLogger ecsLogger; public OrderService(EcsLogger ecsLogger) { this.ecsLogger = ecsLogger; } public Order createOrder(CreateOrderRequest request) { try { ecsLogger.businessEvent("order_creation", null, "start", Map.of( "customer.id", request.getCustomerId(), "order.total", request.getTotalAmount() )); // Business logic Order order = processOrder(request); ecsLogger.businessEvent("order_creation", order.getId(), "success", Map.of( "order.status", order.getStatus(), "order.items.count", order.getItems().size() )); return order; } catch (Exception e) { ecsLogger.error("Order creation failed", Map.of( "customer.id", request.getCustomerId(), "order.total", request.getTotalAmount() ), e); throw e; } } public void processPayment(String orderId, Payment payment) { ecsLogger.info("Processing payment", Map.of( "order.id", orderId, "payment.amount", payment.getAmount(), "payment.method", payment.getMethod(), "business.operation", "payment_processing" )); // Payment processing logic } }

8. Testing ECS Logging

Test Configuration:

package com.example.ecs.test; import com.example.ecs.logging.EcsLogger; import com.fasterxml.jackson.databind.JsonNode; import com.fasterxml.jackson.databind.ObjectMapper; import org.junit.jupiter.api.Test; import org.springframework.boot.test.context.SpringBootTest; import java.io.ByteArrayOutputStream; import java.io.PrintStream; import java.util.Map; import static org.assertj.core.api.Assertions.assertThat; @SpringBootTest class EcsLoggingTest { private final ObjectMapper objectMapper = new ObjectMapper(); @Test void testEcsLogStructure() throws Exception { ByteArrayOutputStream outputStream = new ByteArrayOutputStream(); PrintStream originalOut = System.out; System.setOut(new PrintStream(outputStream)); EcsLogger logger = new EcsLogger(); logger.info("Test message", Map.of("test.field", "value")); System.setOut(originalOut); String logLine = outputStream.toString().trim(); JsonNode logJson = objectMapper.readTree(logLine); // Verify ECS structure assertThat(logJson.has("@timestamp")).isTrue(); assertThat(logJson.has("log.level")).isTrue(); assertThat(logJson.has("message")).isTrue(); assertThat(logJson.has("service.name")).isTrue(); assertThat(logJson.get("message").asText()).isEqualTo("Test message"); assertThat(logJson.get("labels").get("test.field").asText()).isEqualTo("value"); } }

9. Docker and Kubernetes Configuration

Dockerfile with ECS Logging:

FROM openjdk:17-jre-slim # Set ECS logging properties ENV SERVICE_NAME=order-service ENV SERVICE_VERSION=1.0.0 ENV SERVICE_ENVIRONMENT=production # Copy application COPY target/order-service.jar /app/ COPY src/main/resources/logback-spring.xml /app/config/ WORKDIR /app CMD ["java", "-jar", "order-service.jar", "--logging.config=config/logback-spring.xml"]

Kubernetes ConfigMap for Logging:

apiVersion: v1 kind: ConfigMap metadata: name: ecs-logging-config data: logback-spring.xml: | <?xml version="1.0" encoding="UTF-8"?> <configuration> <appender name="ECS_JSON" class="ch.qos.logback.core.ConsoleAppender"> <encoder class="co.elastic.logging.logback.EcsEncoder"> <serviceName>${SERVICE_NAME}</serviceName> <serviceVersion>${SERVICE_VERSION}</serviceVersion> <serviceEnvironment>${SERVICE_ENVIRONMENT}</serviceEnvironment> </encoder> </appender> <root level="INFO"> <appender-ref ref="ECS_JSON" /> </root> </configuration>

10. Best Practices

Field Naming Conventions:

• Use ECS field names when available
• Prefix custom fields with domain (e.g., business., app.)
• Use snake_case for all field names

Performance Considerations:

• Use async appenders in production
• Avoid logging large objects in hot paths
• Use structured logging judiciously

Security:

• Don't log sensitive information (passwords, tokens, PII)
• Use field-level redaction if needed

public Map<String, Object> sanitizeFields(Map<String, Object> fields) { Map<String, Object> sanitized = new HashMap<>(fields); sanitized.keySet().removeIf(key -> key.contains("password") || key.contains("token") || key.contains("secret")); return sanitized; }

Conclusion

ECS logging in Java provides a standardized, powerful way to structure your log data for better observability. Key benefits include:

1. Consistent Schema: Standard field names across all services
2. Better Analytics: Enables powerful Kibana visualizations and correlations
3. Operational Excellence: Improved debugging and monitoring capabilities
4. Future-Proof: Vendor-neutral approach

By implementing ECS logging with the patterns shown above, you'll create maintainable, searchable, and analyzable logs that work seamlessly with the Elastic Stack and other observability tools.