ECS Logging in Java: Structured Logging with Elastic Common Schema

ECS (Elastic Common Schema) provides a standardized way to structure your log data, making it easier to analyze, visualize, and correlate logs across different services and platforms.

Understanding ECS Logging

What is ECS?

A common schema for logs, metrics, and security events
Standardized field names and data types
Enables correlation across different data sources
Maintained by Elastic

Benefits of ECS Logging:

Consistency: Uniform field names across applications
Interoperability: Works seamlessly with Elastic Stack
Searchability: Powerful filtering and aggregation
Compliance: Standardized security and audit logging

Dependencies and Setup

Maven Dependencies

<properties> <logback.version>1.4.11</logback.version> <logstash-logback-encoder.version>7.4</logstash-logback-encoder.version> <ecs-logging-java.version>1.5.0</ecs-logging-java.version> <spring-boot.version>3.1.0</spring-boot.version> </properties> <dependencies> <!-- ECS Logging Core --> <dependency> <groupId>co.elastic.logging</groupId> <artifactId>ecs-logging-core</artifactId> <version>${ecs-logging-java.version}</version> </dependency> <!-- Logback ECS Encoder --> <dependency> <groupId>co.elastic.logging</groupId> <artifactId>ecs-logging-logback</artifactId> <version>${ecs-logging-java.version}</version> </dependency> <!-- Spring Boot Starter --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> <version>${spring-boot.version}</version> </dependency> <!-- Logstash Logback Encoder (alternative) --> <dependency> <groupId>net.logstash.logback</groupId> <artifactId>logstash-logback-encoder</artifactId> <version>${logstash-logback-encoder.version}</version> </dependency> </dependencies>

ECS Logback Configuration

1. Basic ECS Logback Configuration

<!-- src/main/resources/logback-spring.xml --> <configuration> <!-- ECS JSON Console Appender --> <appender name="ECS_JSON_CONSOLE" class="ch.qos.logback.core.ConsoleAppender"> <encoder class="co.elastic.logging.logback.EcsEncoder"> <serviceName>user-service</serviceName> <serviceVersion>1.0.0</serviceVersion> <serviceEnvironment>${LOG_ENV:-development}</serviceEnvironment> <includeMarkers>true</includeMarkers> <includeMdc>true</includeMdc> <stackTraceAsArray>true</stackTraceAsArray> </encoder> </appender> <!-- ECS JSON File Appender --> <appender name="ECS_JSON_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender"> <file>logs/application-ecs.json</file> <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy"> <fileNamePattern>logs/application-ecs.%d{yyyy-MM-dd}.json</fileNamePattern> <maxHistory>7</maxHistory> </rollingPolicy> <encoder class="co.elastic.logging.logback.EcsEncoder"> <serviceName>user-service</serviceName> <serviceVersion>1.0.0</serviceVersion> <serviceEnvironment>${LOG_ENV:-development}</serviceEnvironment> <includeMarkers>true</includeMarkers> <includeMdc>true</includeMdc> <stackTraceAsArray>true</stackTraceAsArray> </encoder> </appender> <!-- Async Appender for Production --> <appender name="ASYNC_ECS" class="ch.qos.logback.classic.AsyncAppender"> <queueSize>1024</queueSize> <discardingThreshold>0</discardingThreshold> <appender-ref ref="ECS_JSON_FILE" /> </appender> <root level="INFO"> <appender-ref ref="ECS_JSON_CONSOLE" /> <appender-ref ref="ASYNC_ECS" /> </root> <!-- Application-specific loggers --> <logger name="com.example.userservice" level="DEBUG" additivity="false"> <appender-ref ref="ECS_JSON_CONSOLE" /> </logger> </configuration>

2. Advanced ECS Configuration with Custom Fields

<configuration> <appender name="ECS_CUSTOM" class="ch.qos.logback.core.ConsoleAppender"> <encoder class="co.elastic.logging.logback.EcsEncoder"> <serviceName>${SERVICE_NAME:-unknown-service}</serviceName> <serviceVersion>${SERVICE_VERSION:-1.0.0}</serviceVersion> <serviceEnvironment>${ENVIRONMENT:-development}</serviceEnvironment> <serviceNodeName>${HOSTNAME:-localhost}</serviceNodeName> <eventDataset>${SERVICE_NAME:-unknown}.log</eventDataset> <includeMarkers>true</includeMarkers> <includeMdc>true</includeMdc> <includeOrigin>true</includeOrigin> <stackTraceAsArray>true</stackTraceAsArray> <!-- Custom additional fields --> <additionalField> <key>deployment.region</key> <value>${DEPLOYMENT_REGION:-us-east-1}</value> </additionalField> <additionalField> <key>team.name</key> <value>user-platform-team</value> </additionalField> </encoder> </appender> <root level="INFO"> <appender-ref ref="ECS_CUSTOM" /> </root> </configuration>

ECS Logging Implementation

1. ECS Logging Utility Class

import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.slf4j.MDC; import org.slf4j.Marker; import org.slf4j.MarkerFactory; import java.time.Instant; import java.util.HashMap; import java.util.Map; @Component public class EcsLogger { private final Logger logger; private static final Marker AUDIT_MARKER = MarkerFactory.getMarker("AUDIT"); private static final Marker SECURITY_MARKER = MarkerFactory.getMarker("SECURITY"); private static final Marker PERFORMANCE_MARKER = MarkerFactory.getMarker("PERFORMANCE"); public EcsLogger(Class<?> clazz) { this.logger = LoggerFactory.getLogger(clazz); } public static EcsLogger getLogger(Class<?> clazz) { return new EcsLogger(clazz); } // Basic structured logging methods public void info(String message, Map<String, Object> fields) { withMDC(fields, () -> logger.info(message)); } public void error(String message, Map<String, Object> fields, Throwable throwable) { withMDC(fields, () -> logger.error(message, throwable)); } public void warn(String message, Map<String, Object> fields) { withMDC(fields, () -> logger.warn(message)); } public void debug(String message, Map<String, Object> fields) { withMDC(fields, () -> logger.debug(message)); } // ECS-specific logging methods public void logHttpRequest(HttpRequestEvent event) { Map<String, Object> fields = new HashMap<>(); fields.put("event.action", "http_request"); fields.put("http.request.method", event.getMethod()); fields.put("url.path", event.getPath()); fields.put("http.request.headers.user_agent", event.getUserAgent()); fields.put("http.response.status_code", event.getStatusCode()); fields.put("event.duration", event.getDuration()); fields.put("source.ip", event.getSourceIp()); if (event.getStatusCode() >= 400) { fields.put("event.outcome", "failure"); warn("HTTP request completed with error", fields); } else { fields.put("event.outcome", "success"); info("HTTP request completed successfully", fields); } } public void logDatabaseQuery(DatabaseEvent event) { Map<String, Object> fields = new HashMap<>(); fields.put("event.action", "database_query"); fields.put("db.system", event.getDatabaseSystem()); fields.put("db.name", event.getDatabaseName()); fields.put("db.operation", event.getOperation()); fields.put("event.duration", event.getDuration()); fields.put("db.rows_affected", event.getRowsAffected()); if (event.isSuccess()) { fields.put("event.outcome", "success"); debug("Database query executed", fields); } else { fields.put("event.outcome", "failure"); error("Database query failed", fields, event.getError()); } } public void logSecurityEvent(SecurityEvent event) { Map<String, Object> fields = new HashMap<>(); fields.put("event.action", event.getAction()); fields.put("event.category", "authentication"); fields.put("user.name", event.getUsername()); fields.put("source.ip", event.getSourceIp()); fields.put("event.outcome", event.isSuccess() ? "success" : "failure"); if (!event.isSuccess()) { fields.put("error.message", event.getErrorMessage()); } withMDC(fields, () -> logger.info(SECURITY_MARKER, "Security event: {}", event.getAction())); } public void logBusinessEvent(BusinessEvent event) { Map<String, Object> fields = new HashMap<>(); fields.put("event.action", event.getAction()); fields.put("event.category", "business"); fields.put("user.id", event.getUserId()); fields.put("order.id", event.getOrderId()); fields.put("transaction.amount", event.getAmount()); fields.put("event.outcome", event.isSuccess() ? "success" : "failure"); info("Business event: " + event.getAction(), fields); } public void logPerformanceMetric(String metricName, long duration, Map<String, Object> tags) { Map<String, Object> fields = new HashMap<>(); fields.put("event.action", "performance_measurement"); fields.put("metric.name", metricName); fields.put("event.duration", duration); fields.putAll(tags); withMDC(fields, () -> logger.info(PERFORMANCE_MARKER, "Performance metric: {}", metricName)); } // Audit logging public void logAuditEvent(AuditEvent event) { Map<String, Object> fields = new HashMap<>(); fields.put("event.action", event.getAction()); fields.put("event.category", "audit"); fields.put("user.name", event.getUsername()); fields.put("user.roles", String.join(",", event.getRoles())); fields.put("resource.name", event.getResourceName()); fields.put("event.outcome", event.isSuccess() ? "success" : "failure"); withMDC(fields, () -> logger.info(AUDIT_MARKER, "Audit event: {}", event.getAction())); } private void withMDC(Map<String, Object> fields, Runnable loggingOperation) { try { // Set MDC fields fields.forEach((key, value) -> MDC.put(key, String.valueOf(value))); // Add timestamp if not present if (!fields.containsKey("@timestamp")) { MDC.put("@timestamp", Instant.now().toString()); } // Execute the actual logging loggingOperation.run(); } finally { // Clear only the fields we set fields.keySet().forEach(MDC::remove); } } }

2. ECS Event Models

// HTTP Request Event public class HttpRequestEvent { private String method; private String path; private String userAgent; private int statusCode; private long duration; private String sourceIp; // constructors, getters, setters public HttpRequestEvent(String method, String path, String userAgent, int statusCode, long duration, String sourceIp) { this.method = method; this.path = path; this.userAgent = userAgent; this.statusCode = statusCode; this.duration = duration; this.sourceIp = sourceIp; } // getters... } // Database Event public class DatabaseEvent { private String databaseSystem; private String databaseName; private String operation; private long duration; private Integer rowsAffected; private boolean success; private Throwable error; // constructors, getters, setters } // Security Event public class SecurityEvent { private String action; private String username; private String sourceIp; private boolean success; private String errorMessage; // constructors, getters, setters } // Business Event public class BusinessEvent { private String action; private String userId; private String orderId; private BigDecimal amount; private boolean success; // constructors, getters, setters } // Audit Event public class AuditEvent { private String action; private String username; private List<String> roles; private String resourceName; private boolean success; // constructors, getters, setters }

Spring Boot Integration

1. ECS Logging Configuration

@Configuration public class EcsLoggingConfig { @Bean @Profile("!test") public EcsEncoder ecsEncoder() { EcsEncoder encoder = new EcsEncoder(); encoder.setServiceName("user-service"); encoder.setServiceVersion("1.0.0"); encoder.setServiceEnvironment(getEnvironment()); encoder.setIncludeMarkers(true); encoder.setIncludeMdc(true); encoder.setIncludeOrigin(true); return encoder; } private String getEnvironment() { return System.getenv().getOrDefault("ENVIRONMENT", System.getProperty("spring.profiles.active", "development")); } @Bean public EcsLogger ecsLogger() { return EcsLogger.getLogger(EcsLoggingConfig.class); } }

2. HTTP Request/Response Logging Filter

@Component public class EcsHttpLoggingFilter implements Filter { private final EcsLogger logger = EcsLogger.getLogger(EcsHttpLoggingFilter.class); @Override public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) throws IOException, ServletException { HttpServletRequest httpRequest = (HttpServletRequest) request; HttpServletResponse httpResponse = (HttpServletResponse) response; long startTime = System.currentTimeMillis(); try { // Set trace context in MDC setupMdcContext(httpRequest); chain.doFilter(request, response); } finally { long duration = System.currentTimeMillis() - startTime; logHttpEvent(httpRequest, httpResponse, duration); MDC.clear(); } } private void setupMdcContext(HttpServletRequest request) { // Trace context MDC.put("trace.id", getHeader(request, "X-Trace-ID")); MDC.put("span.id", getHeader(request, "X-Span-ID")); // User context MDC.put("user.id", getHeader(request, "X-User-ID")); MDC.put("user.roles", getHeader(request, "X-User-Roles")); // HTTP context MDC.put("http.request.id", request.getHeader("X-Request-ID")); MDC.put("source.ip", getClientIpAddress(request)); } private void logHttpEvent(HttpServletRequest request, HttpServletResponse response, long duration) { HttpRequestEvent event = new HttpRequestEvent( request.getMethod(), request.getRequestURI(), request.getHeader("User-Agent"), response.getStatus(), duration, getClientIpAddress(request) ); logger.logHttpRequest(event); } private String getClientIpAddress(HttpServletRequest request) { String xForwardedFor = request.getHeader("X-Forwarded-For"); if (xForwardedFor != null && !xForwardedFor.isEmpty()) { return xForwardedFor.split(",")[0].trim(); } return request.getRemoteAddr(); } private String getHeader(HttpServletRequest request, String headerName) { String value = request.getHeader(headerName); return value != null ? value : ""; } }

3. Spring AOP for Method Logging

@Aspect @Component public class EcsMethodLoggingAspect { private final EcsLogger logger = EcsLogger.getLogger(EcsMethodLoggingAspect.class); @Around("@annotation(ecsLogged)") public Object logMethodExecution(ProceedingJoinPoint joinPoint, EcsLogged ecsLogged) throws Throwable { String methodName = joinPoint.getSignature().toShortString(); String operation = ecsLogged.value().isEmpty() ? methodName : ecsLogged.value(); Map<String, Object> fields = new HashMap<>(); fields.put("event.action", "method_execution"); fields.put("service.method", methodName); fields.put("event.category", "application"); long startTime = System.currentTimeMillis(); try { Object result = joinPoint.proceed(); long duration = System.currentTimeMillis() - startTime; fields.put("event.duration", duration); fields.put("event.outcome", "success"); logger.debug("Method executed successfully: " + operation, fields); return result; } catch (Exception e) { long duration = System.currentTimeMillis() - startTime; fields.put("event.duration", duration); fields.put("event.outcome", "failure"); fields.put("error.message", e.getMessage()); fields.put("error.type", e.getClass().getSimpleName()); logger.error("Method execution failed: " + operation, fields, e); throw e; } } } @Retention(RetentionPolicy.RUNTIME) @Target(ElementType.METHOD) public @interface EcsLogged { String value() default ""; LogLevel level() default LogLevel.DEBUG; } enum LogLevel { DEBUG, INFO, WARN, ERROR }

4. Service Implementation with ECS Logging

@Service public class UserService { private final EcsLogger logger = EcsLogger.getLogger(UserService.class); private final UserRepository userRepository; public UserService(UserRepository userRepository) { this.userRepository = userRepository; } @EcsLogged("find_user_by_id") public User findUserById(String userId) { Map<String, Object> fields = new HashMap<>(); fields.put("user.id", userId); fields.put("db.operation", "find_by_id"); logger.debug("Finding user by ID", fields); try { User user = userRepository.findById(userId) .orElseThrow(() -> new UserNotFoundException("User not found: " + userId)); fields.put("event.outcome", "success"); fields.put("user.found", true); logger.debug("User found successfully", fields); return user; } catch (UserNotFoundException e) { fields.put("event.outcome", "failure"); fields.put("user.found", false); fields.put("error.message", e.getMessage()); logger.warn("User not found", fields); throw e; } } @EcsLogged("create_user") public User createUser(CreateUserRequest request) { Map<String, Object> fields = new HashMap<>(); fields.put("user.email", request.getEmail()); fields.put("user.role", request.getRole()); fields.put("event.action", "user_creation"); logger.info("Creating new user", fields); try { User user = new User(request.getEmail(), request.getName(), request.getRole()); User savedUser = userRepository.save(user); // Log business event BusinessEvent event = new BusinessEvent( "user_created", savedUser.getId(), null, null, true ); logger.logBusinessEvent(event); fields.put("user.id", savedUser.getId()); fields.put("event.outcome", "success"); logger.info("User created successfully", fields); return savedUser; } catch (Exception e) { fields.put("event.outcome", "failure"); fields.put("error.message", e.getMessage()); logger.error("Failed to create user", fields, e); throw e; } } public void processUserLogin(String userId, String sourceIp) { SecurityEvent event = new SecurityEvent( "user_login", userId, sourceIp, true, null ); logger.logSecurityEvent(event); // Additional login processing... } }

5. REST Controller with ECS Logging

@RestController @RequestMapping("/api/users") public class UserController { private final EcsLogger logger = EcsLogger.getLogger(UserController.class); private final UserService userService; public UserController(UserService userService) { this.userService = userService; } @GetMapping("/{userId}") public ResponseEntity<User> getUser(@PathVariable String userId, HttpServletRequest request) { Map<String, Object> fields = new HashMap<>(); fields.put("user.id", userId); fields.put("http.route", "/api/users/{userId}"); fields.put("source.ip", getClientIpAddress(request)); logger.info("Processing GET user request", fields); try { User user = userService.findUserById(userId); fields.put("event.outcome", "success"); fields.put("http.response.status_code", 200); logger.info("User retrieved successfully", fields); return ResponseEntity.ok(user); } catch (UserNotFoundException e) { fields.put("event.outcome", "failure"); fields.put("http.response.status_code", 404); fields.put("error.message", e.getMessage()); logger.warn("User not found", fields); return ResponseEntity.notFound().build(); } } @PostMapping public ResponseEntity<User> createUser(@RequestBody CreateUserRequest request, HttpServletRequest httpRequest) { Map<String, Object> fields = new HashMap<>(); fields.put("user.email", request.getEmail()); fields.put("source.ip", getClientIpAddress(httpRequest)); logger.info("Processing user creation request", fields); try { User user = userService.createUser(request); fields.put("user.id", user.getId()); fields.put("event.outcome", "success"); fields.put("http.response.status_code", 201); logger.info("User created successfully", fields); return ResponseEntity.status(HttpStatus.CREATED).body(user); } catch (Exception e) { fields.put("event.outcome", "failure"); fields.put("http.response.status_code", 400); fields.put("error.message", e.getMessage()); logger.error("User creation failed", fields, e); return ResponseEntity.badRequest().build(); } } private String getClientIpAddress(HttpServletRequest request) { // Implementation as shown earlier return "unknown"; } }

Custom ECS Appender for Advanced Use Cases

public class CustomEcsAppender extends AppenderBase<ILoggingEvent> { private String serviceName; private String serviceVersion; private String serviceEnvironment; private boolean includeMdc = true; private final ObjectMapper objectMapper = new ObjectMapper(); @Override protected void append(ILoggingEvent event) { try { Map<String, Object> ecsLog = createEcsLog(event); String jsonLog = objectMapper.writeValueAsString(ecsLog); // Write to console or send to external system System.out.println(jsonLog); } catch (Exception e) { addError("Failed to create ECS log", e); } } private Map<String, Object> createEcsLog(ILoggingEvent event) { Map<String, Object> log = new HashMap<>(); // Timestamp log.put("@timestamp", Instant.ofEpochMilli(event.getTimeStamp()).toString()); // Log level and message log.put("log.level", event.getLevel().toString()); log.put("message", event.getFormattedMessage()); // Service context log.put("service.name", serviceName); log.put("service.version", serviceVersion); log.put("service.environment", serviceEnvironment); // Event context log.put("event.module", "java-application"); log.put("event.dataset", serviceName + ".log"); // Logger context log.put("log.logger", event.getLoggerName()); // Add MDC context if (includeMdc && event.getMDCPropertyMap() != null) { log.putAll(event.getMDCPropertyMap()); } // Add error information if present if (event.getThrowableProxy() != null) { log.put("error.type", event.getThrowableProxy().getClassName()); log.put("error.message", event.getThrowableProxy().getMessage()); log.put("error.stack_trace", Arrays.asList(event.getThrowableProxy().getStackTraceElementProxyArray())); } return log; } // Getters and setters for configuration public void setServiceName(String serviceName) { this.serviceName = serviceName; } public void setServiceVersion(String serviceVersion) { this.serviceVersion = serviceVersion; } public void setServiceEnvironment(String serviceEnvironment) { this.serviceEnvironment = serviceEnvironment; } public void setIncludeMdc(boolean includeMdc) { this.includeMdc = includeMdc; } }

Logback Configuration for Custom Appender

<configuration> <appender name="CUSTOM_ECS" class="com.example.logging.CustomEcsAppender"> <serviceName>user-service</serviceName> <serviceVersion>1.0.0</serviceVersion> <serviceEnvironment>production</serviceEnvironment> <includeMdc>true</includeMdc> </appender> <root level="INFO"> <appender-ref ref="CUSTOM_ECS" /> </root> </configuration>

Testing ECS Logging

1. Unit Test for ECS Logger

@ExtendWith(MockitoExtension.class) class EcsLoggerTest { @Test void shouldLogWithStructuredFields() { // Given Logger mockLogger = mock(Logger.class); EcsLogger ecsLogger = new EcsLogger(mockLogger); Map<String, Object> fields = new HashMap<>(); fields.put("user.id", "123"); fields.put("event.action", "test_action"); // When ecsLogger.info("Test message", fields); // Then verify(mockLogger).info("Test message"); } @Test void shouldLogHttpRequestEvent() { // Given EcsLogger ecsLogger = EcsLogger.getLogger(EcsLoggerTest.class); HttpRequestEvent event = new HttpRequestEvent( "GET", "/api/users", "test-agent", 200, 150L, "192.168.1.1" ); // When/Then - This should execute without errors ecsLogger.logHttpRequest(event); } }

2. Integration Test

@SpringBootTest @ActiveProfiles("test") class EcsLoggingIntegrationTest { @Autowired private UserService userService; @Test void shouldLogUserCreationWithEcsFields() { // Given CreateUserRequest request = new CreateUserRequest("[email protected]", "Test User", "USER"); // When User user = userService.createUser(request); // Then - Verify through log inspection or mock verification assertThat(user).isNotNull(); } }

Best Practices for ECS Logging

Standard Field Names: Always use ECS standard field names
Meaningful Messages: Log messages should be human-readable
Structured Data: Put variable data in fields, not in message templates
Performance: Use async appenders in production
Sensitive Data: Never log passwords, tokens, or PII
Consistent Level: Use appropriate log levels consistently

// Good practice - structured logging logger.info("User login successful", Map.of( "user.id", userId, "event.action", "user_login", "source.ip", ipAddress )); // Bad practice - unstructured logging logger.info("User " + userId + " logged in from " + ipAddress);

Conclusion

ECS logging in Java provides:

Standardized structure for all log events
Excellent Elastic Stack integration
Powerful search and analytics capabilities
Cross-service correlation through standardized fields
Comprehensive audit trails for security and compliance

By implementing ECS logging as shown above, you can significantly improve your application's observability, make debugging easier, and gain valuable insights into your system's behavior through structured, searchable logs.