ECS (Elastic Common Schema) provides a standardized way to structure your log data, making it easier to analyze, visualize, and correlate logs across different services and platforms.
Understanding ECS Logging
What is ECS?
- A common schema for logs, metrics, and security events
- Standardized field names and data types
- Enables correlation across different data sources
- Maintained by Elastic
Benefits of ECS Logging:
- Consistency: Uniform field names across applications
- Interoperability: Works seamlessly with Elastic Stack
- Searchability: Powerful filtering and aggregation
- Compliance: Standardized security and audit logging
Dependencies and Setup
Maven Dependencies
<properties>
<logback.version>1.4.11</logback.version>
<logstash-logback-encoder.version>7.4</logstash-logback-encoder.version>
<ecs-logging-java.version>1.5.0</ecs-logging-java.version>
<spring-boot.version>3.1.0</spring-boot.version>
</properties>
<dependencies>
<!-- ECS Logging Core -->
<dependency>
<groupId>co.elastic.logging</groupId>
<artifactId>ecs-logging-core</artifactId>
<version>${ecs-logging-java.version}</version>
</dependency>
<!-- Logback ECS Encoder -->
<dependency>
<groupId>co.elastic.logging</groupId>
<artifactId>ecs-logging-logback</artifactId>
<version>${ecs-logging-java.version}</version>
</dependency>
<!-- Spring Boot Starter -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
<version>${spring-boot.version}</version>
</dependency>
<!-- Logstash Logback Encoder (alternative) -->
<dependency>
<groupId>net.logstash.logback</groupId>
<artifactId>logstash-logback-encoder</artifactId>
<version>${logstash-logback-encoder.version}</version>
</dependency>
</dependencies>
ECS Logback Configuration
1. Basic ECS Logback Configuration
<!-- src/main/resources/logback-spring.xml -->
<configuration>
<!-- ECS JSON Console Appender -->
<appender name="ECS_JSON_CONSOLE" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="co.elastic.logging.logback.EcsEncoder">
<serviceName>user-service</serviceName>
<serviceVersion>1.0.0</serviceVersion>
<serviceEnvironment>${LOG_ENV:-development}</serviceEnvironment>
<includeMarkers>true</includeMarkers>
<includeMdc>true</includeMdc>
<stackTraceAsArray>true</stackTraceAsArray>
</encoder>
</appender>
<!-- ECS JSON File Appender -->
<appender name="ECS_JSON_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>logs/application-ecs.json</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>logs/application-ecs.%d{yyyy-MM-dd}.json</fileNamePattern>
<maxHistory>7</maxHistory>
</rollingPolicy>
<encoder class="co.elastic.logging.logback.EcsEncoder">
<serviceName>user-service</serviceName>
<serviceVersion>1.0.0</serviceVersion>
<serviceEnvironment>${LOG_ENV:-development}</serviceEnvironment>
<includeMarkers>true</includeMarkers>
<includeMdc>true</includeMdc>
<stackTraceAsArray>true</stackTraceAsArray>
</encoder>
</appender>
<!-- Async Appender for Production -->
<appender name="ASYNC_ECS" class="ch.qos.logback.classic.AsyncAppender">
<queueSize>1024</queueSize>
<discardingThreshold>0</discardingThreshold>
<appender-ref ref="ECS_JSON_FILE" />
</appender>
<root level="INFO">
<appender-ref ref="ECS_JSON_CONSOLE" />
<appender-ref ref="ASYNC_ECS" />
</root>
<!-- Application-specific loggers -->
<logger name="com.example.userservice" level="DEBUG" additivity="false">
<appender-ref ref="ECS_JSON_CONSOLE" />
</logger>
</configuration>
2. Advanced ECS Configuration with Custom Fields
<configuration>
<appender name="ECS_CUSTOM" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="co.elastic.logging.logback.EcsEncoder">
<serviceName>${SERVICE_NAME:-unknown-service}</serviceName>
<serviceVersion>${SERVICE_VERSION:-1.0.0}</serviceVersion>
<serviceEnvironment>${ENVIRONMENT:-development}</serviceEnvironment>
<serviceNodeName>${HOSTNAME:-localhost}</serviceNodeName>
<eventDataset>${SERVICE_NAME:-unknown}.log</eventDataset>
<includeMarkers>true</includeMarkers>
<includeMdc>true</includeMdc>
<includeOrigin>true</includeOrigin>
<stackTraceAsArray>true</stackTraceAsArray>
<!-- Custom additional fields -->
<additionalField>
<key>deployment.region</key>
<value>${DEPLOYMENT_REGION:-us-east-1}</value>
</additionalField>
<additionalField>
<key>team.name</key>
<value>user-platform-team</value>
</additionalField>
</encoder>
</appender>
<root level="INFO">
<appender-ref ref="ECS_CUSTOM" />
</root>
</configuration>
ECS Logging Implementation
1. ECS Logging Utility Class
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.slf4j.MDC;
import org.slf4j.Marker;
import org.slf4j.MarkerFactory;
import java.time.Instant;
import java.util.HashMap;
import java.util.Map;
@Component
public class EcsLogger {
private final Logger logger;
private static final Marker AUDIT_MARKER = MarkerFactory.getMarker("AUDIT");
private static final Marker SECURITY_MARKER = MarkerFactory.getMarker("SECURITY");
private static final Marker PERFORMANCE_MARKER = MarkerFactory.getMarker("PERFORMANCE");
public EcsLogger(Class<?> clazz) {
this.logger = LoggerFactory.getLogger(clazz);
}
public static EcsLogger getLogger(Class<?> clazz) {
return new EcsLogger(clazz);
}
// Basic structured logging methods
public void info(String message, Map<String, Object> fields) {
withMDC(fields, () -> logger.info(message));
}
public void error(String message, Map<String, Object> fields, Throwable throwable) {
withMDC(fields, () -> logger.error(message, throwable));
}
public void warn(String message, Map<String, Object> fields) {
withMDC(fields, () -> logger.warn(message));
}
public void debug(String message, Map<String, Object> fields) {
withMDC(fields, () -> logger.debug(message));
}
// ECS-specific logging methods
public void logHttpRequest(HttpRequestEvent event) {
Map<String, Object> fields = new HashMap<>();
fields.put("event.action", "http_request");
fields.put("http.request.method", event.getMethod());
fields.put("url.path", event.getPath());
fields.put("http.request.headers.user_agent", event.getUserAgent());
fields.put("http.response.status_code", event.getStatusCode());
fields.put("event.duration", event.getDuration());
fields.put("source.ip", event.getSourceIp());
if (event.getStatusCode() >= 400) {
fields.put("event.outcome", "failure");
warn("HTTP request completed with error", fields);
} else {
fields.put("event.outcome", "success");
info("HTTP request completed successfully", fields);
}
}
public void logDatabaseQuery(DatabaseEvent event) {
Map<String, Object> fields = new HashMap<>();
fields.put("event.action", "database_query");
fields.put("db.system", event.getDatabaseSystem());
fields.put("db.name", event.getDatabaseName());
fields.put("db.operation", event.getOperation());
fields.put("event.duration", event.getDuration());
fields.put("db.rows_affected", event.getRowsAffected());
if (event.isSuccess()) {
fields.put("event.outcome", "success");
debug("Database query executed", fields);
} else {
fields.put("event.outcome", "failure");
error("Database query failed", fields, event.getError());
}
}
public void logSecurityEvent(SecurityEvent event) {
Map<String, Object> fields = new HashMap<>();
fields.put("event.action", event.getAction());
fields.put("event.category", "authentication");
fields.put("user.name", event.getUsername());
fields.put("source.ip", event.getSourceIp());
fields.put("event.outcome", event.isSuccess() ? "success" : "failure");
if (!event.isSuccess()) {
fields.put("error.message", event.getErrorMessage());
}
withMDC(fields, () -> logger.info(SECURITY_MARKER, "Security event: {}", event.getAction()));
}
public void logBusinessEvent(BusinessEvent event) {
Map<String, Object> fields = new HashMap<>();
fields.put("event.action", event.getAction());
fields.put("event.category", "business");
fields.put("user.id", event.getUserId());
fields.put("order.id", event.getOrderId());
fields.put("transaction.amount", event.getAmount());
fields.put("event.outcome", event.isSuccess() ? "success" : "failure");
info("Business event: " + event.getAction(), fields);
}
public void logPerformanceMetric(String metricName, long duration, Map<String, Object> tags) {
Map<String, Object> fields = new HashMap<>();
fields.put("event.action", "performance_measurement");
fields.put("metric.name", metricName);
fields.put("event.duration", duration);
fields.putAll(tags);
withMDC(fields, () -> logger.info(PERFORMANCE_MARKER, "Performance metric: {}", metricName));
}
// Audit logging
public void logAuditEvent(AuditEvent event) {
Map<String, Object> fields = new HashMap<>();
fields.put("event.action", event.getAction());
fields.put("event.category", "audit");
fields.put("user.name", event.getUsername());
fields.put("user.roles", String.join(",", event.getRoles()));
fields.put("resource.name", event.getResourceName());
fields.put("event.outcome", event.isSuccess() ? "success" : "failure");
withMDC(fields, () -> logger.info(AUDIT_MARKER, "Audit event: {}", event.getAction()));
}
private void withMDC(Map<String, Object> fields, Runnable loggingOperation) {
try {
// Set MDC fields
fields.forEach((key, value) -> MDC.put(key, String.valueOf(value)));
// Add timestamp if not present
if (!fields.containsKey("@timestamp")) {
MDC.put("@timestamp", Instant.now().toString());
}
// Execute the actual logging
loggingOperation.run();
} finally {
// Clear only the fields we set
fields.keySet().forEach(MDC::remove);
}
}
}
2. ECS Event Models
// HTTP Request Event
public class HttpRequestEvent {
private String method;
private String path;
private String userAgent;
private int statusCode;
private long duration;
private String sourceIp;
// constructors, getters, setters
public HttpRequestEvent(String method, String path, String userAgent,
int statusCode, long duration, String sourceIp) {
this.method = method;
this.path = path;
this.userAgent = userAgent;
this.statusCode = statusCode;
this.duration = duration;
this.sourceIp = sourceIp;
}
// getters...
}
// Database Event
public class DatabaseEvent {
private String databaseSystem;
private String databaseName;
private String operation;
private long duration;
private Integer rowsAffected;
private boolean success;
private Throwable error;
// constructors, getters, setters
}
// Security Event
public class SecurityEvent {
private String action;
private String username;
private String sourceIp;
private boolean success;
private String errorMessage;
// constructors, getters, setters
}
// Business Event
public class BusinessEvent {
private String action;
private String userId;
private String orderId;
private BigDecimal amount;
private boolean success;
// constructors, getters, setters
}
// Audit Event
public class AuditEvent {
private String action;
private String username;
private List<String> roles;
private String resourceName;
private boolean success;
// constructors, getters, setters
}
Spring Boot Integration
1. ECS Logging Configuration
@Configuration
public class EcsLoggingConfig {
@Bean
@Profile("!test")
public EcsEncoder ecsEncoder() {
EcsEncoder encoder = new EcsEncoder();
encoder.setServiceName("user-service");
encoder.setServiceVersion("1.0.0");
encoder.setServiceEnvironment(getEnvironment());
encoder.setIncludeMarkers(true);
encoder.setIncludeMdc(true);
encoder.setIncludeOrigin(true);
return encoder;
}
private String getEnvironment() {
return System.getenv().getOrDefault("ENVIRONMENT",
System.getProperty("spring.profiles.active", "development"));
}
@Bean
public EcsLogger ecsLogger() {
return EcsLogger.getLogger(EcsLoggingConfig.class);
}
}
2. HTTP Request/Response Logging Filter
@Component
public class EcsHttpLoggingFilter implements Filter {
private final EcsLogger logger = EcsLogger.getLogger(EcsHttpLoggingFilter.class);
@Override
public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain)
throws IOException, ServletException {
HttpServletRequest httpRequest = (HttpServletRequest) request;
HttpServletResponse httpResponse = (HttpServletResponse) response;
long startTime = System.currentTimeMillis();
try {
// Set trace context in MDC
setupMdcContext(httpRequest);
chain.doFilter(request, response);
} finally {
long duration = System.currentTimeMillis() - startTime;
logHttpEvent(httpRequest, httpResponse, duration);
MDC.clear();
}
}
private void setupMdcContext(HttpServletRequest request) {
// Trace context
MDC.put("trace.id", getHeader(request, "X-Trace-ID"));
MDC.put("span.id", getHeader(request, "X-Span-ID"));
// User context
MDC.put("user.id", getHeader(request, "X-User-ID"));
MDC.put("user.roles", getHeader(request, "X-User-Roles"));
// HTTP context
MDC.put("http.request.id", request.getHeader("X-Request-ID"));
MDC.put("source.ip", getClientIpAddress(request));
}
private void logHttpEvent(HttpServletRequest request, HttpServletResponse response, long duration) {
HttpRequestEvent event = new HttpRequestEvent(
request.getMethod(),
request.getRequestURI(),
request.getHeader("User-Agent"),
response.getStatus(),
duration,
getClientIpAddress(request)
);
logger.logHttpRequest(event);
}
private String getClientIpAddress(HttpServletRequest request) {
String xForwardedFor = request.getHeader("X-Forwarded-For");
if (xForwardedFor != null && !xForwardedFor.isEmpty()) {
return xForwardedFor.split(",")[0].trim();
}
return request.getRemoteAddr();
}
private String getHeader(HttpServletRequest request, String headerName) {
String value = request.getHeader(headerName);
return value != null ? value : "";
}
}
3. Spring AOP for Method Logging
@Aspect
@Component
public class EcsMethodLoggingAspect {
private final EcsLogger logger = EcsLogger.getLogger(EcsMethodLoggingAspect.class);
@Around("@annotation(ecsLogged)")
public Object logMethodExecution(ProceedingJoinPoint joinPoint, EcsLogged ecsLogged) throws Throwable {
String methodName = joinPoint.getSignature().toShortString();
String operation = ecsLogged.value().isEmpty() ? methodName : ecsLogged.value();
Map<String, Object> fields = new HashMap<>();
fields.put("event.action", "method_execution");
fields.put("service.method", methodName);
fields.put("event.category", "application");
long startTime = System.currentTimeMillis();
try {
Object result = joinPoint.proceed();
long duration = System.currentTimeMillis() - startTime;
fields.put("event.duration", duration);
fields.put("event.outcome", "success");
logger.debug("Method executed successfully: " + operation, fields);
return result;
} catch (Exception e) {
long duration = System.currentTimeMillis() - startTime;
fields.put("event.duration", duration);
fields.put("event.outcome", "failure");
fields.put("error.message", e.getMessage());
fields.put("error.type", e.getClass().getSimpleName());
logger.error("Method execution failed: " + operation, fields, e);
throw e;
}
}
}
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.METHOD)
public @interface EcsLogged {
String value() default "";
LogLevel level() default LogLevel.DEBUG;
}
enum LogLevel {
DEBUG, INFO, WARN, ERROR
}
4. Service Implementation with ECS Logging
@Service
public class UserService {
private final EcsLogger logger = EcsLogger.getLogger(UserService.class);
private final UserRepository userRepository;
public UserService(UserRepository userRepository) {
this.userRepository = userRepository;
}
@EcsLogged("find_user_by_id")
public User findUserById(String userId) {
Map<String, Object> fields = new HashMap<>();
fields.put("user.id", userId);
fields.put("db.operation", "find_by_id");
logger.debug("Finding user by ID", fields);
try {
User user = userRepository.findById(userId)
.orElseThrow(() -> new UserNotFoundException("User not found: " + userId));
fields.put("event.outcome", "success");
fields.put("user.found", true);
logger.debug("User found successfully", fields);
return user;
} catch (UserNotFoundException e) {
fields.put("event.outcome", "failure");
fields.put("user.found", false);
fields.put("error.message", e.getMessage());
logger.warn("User not found", fields);
throw e;
}
}
@EcsLogged("create_user")
public User createUser(CreateUserRequest request) {
Map<String, Object> fields = new HashMap<>();
fields.put("user.email", request.getEmail());
fields.put("user.role", request.getRole());
fields.put("event.action", "user_creation");
logger.info("Creating new user", fields);
try {
User user = new User(request.getEmail(), request.getName(), request.getRole());
User savedUser = userRepository.save(user);
// Log business event
BusinessEvent event = new BusinessEvent(
"user_created",
savedUser.getId(),
null,
null,
true
);
logger.logBusinessEvent(event);
fields.put("user.id", savedUser.getId());
fields.put("event.outcome", "success");
logger.info("User created successfully", fields);
return savedUser;
} catch (Exception e) {
fields.put("event.outcome", "failure");
fields.put("error.message", e.getMessage());
logger.error("Failed to create user", fields, e);
throw e;
}
}
public void processUserLogin(String userId, String sourceIp) {
SecurityEvent event = new SecurityEvent(
"user_login",
userId,
sourceIp,
true,
null
);
logger.logSecurityEvent(event);
// Additional login processing...
}
}
5. REST Controller with ECS Logging
@RestController
@RequestMapping("/api/users")
public class UserController {
private final EcsLogger logger = EcsLogger.getLogger(UserController.class);
private final UserService userService;
public UserController(UserService userService) {
this.userService = userService;
}
@GetMapping("/{userId}")
public ResponseEntity<User> getUser(@PathVariable String userId,
HttpServletRequest request) {
Map<String, Object> fields = new HashMap<>();
fields.put("user.id", userId);
fields.put("http.route", "/api/users/{userId}");
fields.put("source.ip", getClientIpAddress(request));
logger.info("Processing GET user request", fields);
try {
User user = userService.findUserById(userId);
fields.put("event.outcome", "success");
fields.put("http.response.status_code", 200);
logger.info("User retrieved successfully", fields);
return ResponseEntity.ok(user);
} catch (UserNotFoundException e) {
fields.put("event.outcome", "failure");
fields.put("http.response.status_code", 404);
fields.put("error.message", e.getMessage());
logger.warn("User not found", fields);
return ResponseEntity.notFound().build();
}
}
@PostMapping
public ResponseEntity<User> createUser(@RequestBody CreateUserRequest request,
HttpServletRequest httpRequest) {
Map<String, Object> fields = new HashMap<>();
fields.put("user.email", request.getEmail());
fields.put("source.ip", getClientIpAddress(httpRequest));
logger.info("Processing user creation request", fields);
try {
User user = userService.createUser(request);
fields.put("user.id", user.getId());
fields.put("event.outcome", "success");
fields.put("http.response.status_code", 201);
logger.info("User created successfully", fields);
return ResponseEntity.status(HttpStatus.CREATED).body(user);
} catch (Exception e) {
fields.put("event.outcome", "failure");
fields.put("http.response.status_code", 400);
fields.put("error.message", e.getMessage());
logger.error("User creation failed", fields, e);
return ResponseEntity.badRequest().build();
}
}
private String getClientIpAddress(HttpServletRequest request) {
// Implementation as shown earlier
return "unknown";
}
}
Custom ECS Appender for Advanced Use Cases
public class CustomEcsAppender extends AppenderBase<ILoggingEvent> {
private String serviceName;
private String serviceVersion;
private String serviceEnvironment;
private boolean includeMdc = true;
private final ObjectMapper objectMapper = new ObjectMapper();
@Override
protected void append(ILoggingEvent event) {
try {
Map<String, Object> ecsLog = createEcsLog(event);
String jsonLog = objectMapper.writeValueAsString(ecsLog);
// Write to console or send to external system
System.out.println(jsonLog);
} catch (Exception e) {
addError("Failed to create ECS log", e);
}
}
private Map<String, Object> createEcsLog(ILoggingEvent event) {
Map<String, Object> log = new HashMap<>();
// Timestamp
log.put("@timestamp", Instant.ofEpochMilli(event.getTimeStamp()).toString());
// Log level and message
log.put("log.level", event.getLevel().toString());
log.put("message", event.getFormattedMessage());
// Service context
log.put("service.name", serviceName);
log.put("service.version", serviceVersion);
log.put("service.environment", serviceEnvironment);
// Event context
log.put("event.module", "java-application");
log.put("event.dataset", serviceName + ".log");
// Logger context
log.put("log.logger", event.getLoggerName());
// Add MDC context
if (includeMdc && event.getMDCPropertyMap() != null) {
log.putAll(event.getMDCPropertyMap());
}
// Add error information if present
if (event.getThrowableProxy() != null) {
log.put("error.type", event.getThrowableProxy().getClassName());
log.put("error.message", event.getThrowableProxy().getMessage());
log.put("error.stack_trace", Arrays.asList(event.getThrowableProxy().getStackTraceElementProxyArray()));
}
return log;
}
// Getters and setters for configuration
public void setServiceName(String serviceName) { this.serviceName = serviceName; }
public void setServiceVersion(String serviceVersion) { this.serviceVersion = serviceVersion; }
public void setServiceEnvironment(String serviceEnvironment) { this.serviceEnvironment = serviceEnvironment; }
public void setIncludeMdc(boolean includeMdc) { this.includeMdc = includeMdc; }
}
Logback Configuration for Custom Appender
<configuration> <appender name="CUSTOM_ECS" class="com.example.logging.CustomEcsAppender"> <serviceName>user-service</serviceName> <serviceVersion>1.0.0</serviceVersion> <serviceEnvironment>production</serviceEnvironment> <includeMdc>true</includeMdc> </appender> <root level="INFO"> <appender-ref ref="CUSTOM_ECS" /> </root> </configuration>
Testing ECS Logging
1. Unit Test for ECS Logger
@ExtendWith(MockitoExtension.class)
class EcsLoggerTest {
@Test
void shouldLogWithStructuredFields() {
// Given
Logger mockLogger = mock(Logger.class);
EcsLogger ecsLogger = new EcsLogger(mockLogger);
Map<String, Object> fields = new HashMap<>();
fields.put("user.id", "123");
fields.put("event.action", "test_action");
// When
ecsLogger.info("Test message", fields);
// Then
verify(mockLogger).info("Test message");
}
@Test
void shouldLogHttpRequestEvent() {
// Given
EcsLogger ecsLogger = EcsLogger.getLogger(EcsLoggerTest.class);
HttpRequestEvent event = new HttpRequestEvent(
"GET", "/api/users", "test-agent", 200, 150L, "192.168.1.1"
);
// When/Then - This should execute without errors
ecsLogger.logHttpRequest(event);
}
}
2. Integration Test
@SpringBootTest
@ActiveProfiles("test")
class EcsLoggingIntegrationTest {
@Autowired
private UserService userService;
@Test
void shouldLogUserCreationWithEcsFields() {
// Given
CreateUserRequest request = new CreateUserRequest("[email protected]", "Test User", "USER");
// When
User user = userService.createUser(request);
// Then - Verify through log inspection or mock verification
assertThat(user).isNotNull();
}
}
Best Practices for ECS Logging
- Standard Field Names: Always use ECS standard field names
- Meaningful Messages: Log messages should be human-readable
- Structured Data: Put variable data in fields, not in message templates
- Performance: Use async appenders in production
- Sensitive Data: Never log passwords, tokens, or PII
- Consistent Level: Use appropriate log levels consistently
// Good practice - structured logging
logger.info("User login successful", Map.of(
"user.id", userId,
"event.action", "user_login",
"source.ip", ipAddress
));
// Bad practice - unstructured logging
logger.info("User " + userId + " logged in from " + ipAddress);
Conclusion
ECS logging in Java provides:
- Standardized structure for all log events
- Excellent Elastic Stack integration
- Powerful search and analytics capabilities
- Cross-service correlation through standardized fields
- Comprehensive audit trails for security and compliance
By implementing ECS logging as shown above, you can significantly improve your application's observability, make debugging easier, and gain valuable insights into your system's behavior through structured, searchable logs.