Log sampling is a critical technique for production systems where high-volume logging can impact performance, storage costs, and signal-to-noise ratio. Java provides sophisticated approaches to implement intelligent log sampling that captures meaningful data while controlling volume. This guide covers practical patterns for implementing effective log sampling strategies in production Java applications.
Understanding Log Sampling Strategies
Common Sampling Approaches:
- Rate-based sampling (Nth occurrence)
- Probabilistic sampling (random percentage)
- Dynamic sampling (adaptive based on conditions)
- Structured sampling (based on log content/context)
- Head-based vs tail-based sampling
Core Implementation Patterns
1. Project Setup and Dependencies
Configure logging dependencies with sampling support.
Maven Configuration:
<dependencies> <!-- Logback with Sampling Support --> <dependency> <groupId>ch.qos.logback</groupId> <artifactId>logback-classic</artifactId> <version>1.4.11</version> </dependency> <!-- Logstash Logback Encoder --> <dependency> <groupId>net.logstash.logback</groupId> <artifactId>logstash-logback-encoder</artifactId> <version>7.4</version> </dependency> <!-- Micrometer for Metrics --> <dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-core</artifactId> <version>1.11.5</version> </dependency> <!-- Configuration --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter</artifactId> <version>3.1.0</version> </dependency> <!-- Cache --> <dependency> <groupId>com.github.ben-manes.caffeine</groupId> <artifactId>caffeine</artifactId> <version>3.1.6</version> </dependency> </dependencies>
2. Domain Models for Log Sampling
Create comprehensive models for sampling configuration and state.
Core Domain Models:
@Data
public class SamplingConfig {
private String strategy = "PROBABILISTIC";
private double rate = 0.1; // 10% for probabilistic
private int interval = 100; // For rate-based
private Map<String, Double> keyedRates = new HashMap<>();
private boolean enabled = true;
private int windowSize = 1000;
private double burstThreshold = 5.0;
private Map<String, Object> customRules = new HashMap<>();
public enum Strategy {
PROBABILISTIC, // Random sampling
RATE_BASED, // Every Nth occurrence
ADAPTIVE, // Dynamic based on conditions
CONTENT_BASED, // Based on log message content
BURST_DETECTION, // Sample more during bursts
KEY_BASED // Different rates for different log keys
}
}
@Data
public class LogEvent {
private String loggerName;
private String level;
private String message;
private String threadName;
private long timestamp;
private Map<String, String> mdc;
private Throwable throwable;
private String traceId;
private String spanId;
private Map<String, Object> customFields;
public String getLogKey() {
return loggerName + ":" + level;
}
public boolean isError() {
return "ERROR".equals(level) || "WARN".equals(level);
}
public boolean containsKeyword(String keyword) {
return message != null && message.toLowerCase().contains(keyword.toLowerCase());
}
}
@Data
public class SamplingDecision {
private boolean shouldLog;
private String reason;
private double sampleRate;
private String strategy;
private Map<String, Object> metadata;
public static SamplingDecision sample() {
return new SamplingDecision(true, "sampled", 1.0, "DEFAULT");
}
public static SamplingDecision skip() {
return new SamplingDecision(false, "sampled", 0.0, "DEFAULT");
}
public SamplingDecision(boolean shouldLog, String reason, double sampleRate, String strategy) {
this.shouldLog = shouldLog;
this.reason = reason;
this.sampleRate = sampleRate;
this.strategy = strategy;
this.metadata = new HashMap<>();
}
}
@Data
public class SamplingMetrics {
private String samplerName;
private long totalEvents;
private long sampledEvents;
private double effectiveRate;
private long windowStart;
private long windowEnd;
private Map<String, Long> decisionCounts;
private double currentLoad;
public SamplingMetrics(String samplerName) {
this.samplerName = samplerName;
this.decisionCounts = new HashMap<>();
this.windowStart = System.currentTimeMillis();
}
public void recordDecision(boolean sampled, String reason) {
totalEvents++;
if (sampled) {
sampledEvents++;
}
decisionCounts.merge(reason, 1L, Long::sum);
effectiveRate = totalEvents > 0 ? (double) sampledEvents / totalEvents : 0;
}
public void resetWindow() {
windowStart = System.currentTimeMillis();
totalEvents = 0;
sampledEvents = 0;
decisionCounts.clear();
}
}
@Data
public class BurstDetection {
private final int windowSize;
private final double threshold;
private final Deque<Long> eventTimestamps = new LinkedList<>();
private long lastBurstDetected = 0;
public BurstDetection(int windowSize, double threshold) {
this.windowSize = windowSize;
this.threshold = threshold;
}
public boolean isBurst() {
long now = System.currentTimeMillis();
long windowStart = now - (windowSize * 1000L);
// Remove old events
while (!eventTimestamps.isEmpty() && eventTimestamps.peekFirst() < windowStart) {
eventTimestamps.pollFirst();
}
double eventsPerSecond = (double) eventTimestamps.size() / windowSize;
return eventsPerSecond > threshold;
}
public void recordEvent() {
eventTimestamps.addLast(System.currentTimeMillis());
if (isBurst()) {
lastBurstDetected = System.currentTimeMillis();
}
}
public boolean isInBurstCoolDown() {
return System.currentTimeMillis() - lastBurstDetected < 30000; // 30 second cooldown
}
}
3. Core Sampling Strategies
Implement various sampling strategies.
Sampling Strategy Interface:
public interface SamplingStrategy {
SamplingDecision shouldSample(LogEvent event);
String getName();
void configure(SamplingConfig config);
SamplingMetrics getMetrics();
}
@Component
@Slf4j
public class ProbabilisticSampler implements SamplingStrategy {
private final Random random = new Random();
private double sampleRate;
private final SamplingMetrics metrics;
public ProbabilisticSampler() {
this.metrics = new SamplingMetrics("probabilistic");
this.sampleRate = 0.1; // Default 10%
}
@Override
public SamplingDecision shouldSample(LogEvent event) {
metrics.recordDecision(false, "evaluated");
// Always sample errors
if (event.isError()) {
metrics.recordDecision(true, "error");
return new SamplingDecision(true, "error", 1.0, getName());
}
// Sample based on probability
boolean shouldSample = random.nextDouble() < sampleRate;
String reason = shouldSample ? "probabilistic" : "sampled_out";
metrics.recordDecision(shouldSample, reason);
return new SamplingDecision(shouldSample, reason, sampleRate, getName());
}
@Override
public String getName() {
return "PROBABILISTIC";
}
@Override
public void configure(SamplingConfig config) {
this.sampleRate = config.getRate();
log.info("Configured probabilistic sampler with rate: {}", sampleRate);
}
@Override
public SamplingMetrics getMetrics() {
return metrics;
}
}
@Component
@Slf4j
public class RateBasedSampler implements SamplingStrategy {
private final AtomicLong counter = new AtomicLong(0);
private int sampleInterval;
private final SamplingMetrics metrics;
public RateBasedSampler() {
this.metrics = new SamplingMetrics("rate_based");
this.sampleInterval = 100; // Default: sample every 100th event
}
@Override
public SamplingDecision shouldSample(LogEvent event) {
metrics.recordDecision(false, "evaluated");
// Always sample errors
if (event.isError()) {
metrics.recordDecision(true, "error");
return new SamplingDecision(true, "error", 1.0, getName());
}
long count = counter.incrementAndGet();
boolean shouldSample = count % sampleInterval == 0;
String reason = shouldSample ? "rate_based" : "sampled_out";
metrics.recordDecision(shouldSample, reason);
return new SamplingDecision(shouldSample, reason, 1.0 / sampleInterval, getName());
}
@Override
public String getName() {
return "RATE_BASED";
}
@Override
public void configure(SamplingConfig config) {
this.sampleInterval = config.getInterval();
log.info("Configured rate-based sampler with interval: {}", sampleInterval);
}
@Override
public SamplingMetrics getMetrics() {
return metrics;
}
}
@Component
@Slf4j
public class AdaptiveSampler implements SamplingStrategy {
private double baseRate;
private double currentRate;
private final BurstDetection burstDetection;
private final SamplingMetrics metrics;
private long lastAdjustment = System.currentTimeMillis();
public AdaptiveSampler() {
this.metrics = new SamplingMetrics("adaptive");
this.baseRate = 0.1;
this.currentRate = 0.1;
this.burstDetection = new BurstDetection(10, 5.0); // 10s window, 5 events/sec threshold
}
@Override
public SamplingDecision shouldSample(LogEvent event) {
metrics.recordDecision(false, "evaluated");
burstDetection.recordEvent();
// Adjust rate based on conditions
adjustSamplingRate();
// Always sample errors and important keywords
if (shouldAlwaysSample(event)) {
metrics.recordDecision(true, "important");
return new SamplingDecision(true, "important", 1.0, getName());
}
// Sample based on current adaptive rate
boolean shouldSample = Math.random() < currentRate;
String reason = shouldSample ? "adaptive" : "sampled_out";
metrics.recordDecision(shouldSample, reason);
SamplingDecision decision = new SamplingDecision(shouldSample, reason, currentRate, getName());
decision.getMetadata().put("current_rate", currentRate);
decision.getMetadata().put("base_rate", baseRate);
return decision;
}
private boolean shouldAlwaysSample(LogEvent event) {
if (event.isError()) return true;
// Sample important business events
String message = event.getMessage();
if (message != null) {
return message.contains("transaction") ||
message.contains("payment") ||
message.contains("audit") ||
message.contains("security");
}
return false;
}
private void adjustSamplingRate() {
long now = System.currentTimeMillis();
if (now - lastAdjustment < 5000) { // Adjust every 5 seconds
return;
}
double currentLoad = calculateCurrentLoad();
if (burstDetection.isBurst()) {
// Reduce sampling during bursts to control volume
currentRate = Math.max(baseRate * 0.1, 0.01);
log.debug("Burst detected, reducing sampling rate to: {}", currentRate);
} else if (currentLoad > 0.8) {
// High load, reduce sampling
currentRate = Math.max(baseRate * 0.5, 0.05);
} else if (currentLoad < 0.2) {
// Low load, increase sampling
currentRate = Math.min(baseRate * 2.0, 1.0);
} else {
// Normal load, use base rate
currentRate = baseRate;
}
lastAdjustment = now;
}
private double calculateCurrentLoad() {
// Simplified load calculation based on recent sampling rate
double targetEventsPerSecond = 1000; // Target volume
double actualEventsPerSecond = metrics.getTotalEvents() / 10.0; // Last 10 seconds
return Math.min(actualEventsPerSecond / targetEventsPerSecond, 1.0);
}
@Override
public String getName() {
return "ADAPTIVE";
}
@Override
public void configure(SamplingConfig config) {
this.baseRate = config.getRate();
this.currentRate = config.getRate();
log.info("Configured adaptive sampler with base rate: {}", baseRate);
}
@Override
public SamplingMetrics getMetrics() {
return metrics;
}
}
@Component
@Slf4j
public class KeyBasedSampler implements SamplingStrategy {
private final Map<String, Double> keyRates;
private final Double defaultRate;
private final SamplingMetrics metrics;
public KeyBasedSampler() {
this.metrics = new SamplingMetrics("key_based");
this.keyRates = new HashMap<>();
this.defaultRate = 0.1;
// Default rates for common loggers
keyRates.put("com.example.service.PaymentService:INFO", 1.0); // Always log payments
keyRates.put("com.example.controller.AuthController:DEBUG", 0.01); // Rarely log auth debug
}
@Override
public SamplingDecision shouldSample(LogEvent event) {
metrics.recordDecision(false, "evaluated");
String logKey = event.getLogKey();
double rate = keyRates.getOrDefault(logKey, defaultRate);
boolean shouldSample = Math.random() < rate;
String reason = shouldSample ? "key_based" : "sampled_out";
metrics.recordDecision(shouldSample, reason);
SamplingDecision decision = new SamplingDecision(shouldSample, reason, rate, getName());
decision.getMetadata().put("log_key", logKey);
decision.getMetadata().put("key_rate", rate);
return decision;
}
@Override
public String getName() {
return "KEY_BASED";
}
@Override
public void configure(SamplingConfig config) {
keyRates.clear();
if (config.getKeyedRates() != null) {
keyRates.putAll(config.getKeyedRates());
}
log.info("Configured key-based sampler with {} key rates", keyRates.size());
}
@Override
public SamplingMetrics getMetrics() {
return metrics;
}
}
4. Sampling Manager and Orchestration
Coordinate multiple sampling strategies.
Sampling Manager:
@Service
@Slf4j
public class SamplingManager {
private final Map<String, SamplingStrategy> strategies;
private SamplingStrategy activeStrategy;
private SamplingConfig currentConfig;
private final MeterRegistry meterRegistry;
private final Map<String, SamplingMetrics> historicalMetrics;
public SamplingManager(List<SamplingStrategy> strategyList, MeterRegistry meterRegistry) {
this.strategies = strategyList.stream()
.collect(Collectors.toMap(SamplingStrategy::getName, Function.identity()));
this.meterRegistry = meterRegistry;
this.historicalMetrics = new ConcurrentHashMap<>();
this.activeStrategy = strategies.get("PROBABILISTIC");
this.currentConfig = new SamplingConfig();
// Initialize metrics collection
startMetricsCollection();
}
public SamplingDecision shouldSample(LogEvent event) {
if (!currentConfig.isEnabled()) {
return SamplingDecision.sample();
}
SamplingDecision decision = activeStrategy.shouldSample(event);
// Record metrics
recordSamplingMetrics(decision);
return decision;
}
public void configureSampling(SamplingConfig config) {
this.currentConfig = config;
SamplingStrategy strategy = strategies.get(config.getStrategy());
if (strategy != null) {
strategy.configure(config);
this.activeStrategy = strategy;
log.info("Switched to sampling strategy: {}", config.getStrategy());
} else {
log.warn("Unknown sampling strategy: {}, keeping current: {}",
config.getStrategy(), activeStrategy.getName());
}
}
public Map<String, Object> getSamplingStatus() {
Map<String, Object> status = new HashMap<>();
status.put("active_strategy", activeStrategy.getName());
status.put("config", currentConfig);
status.put("enabled", currentConfig.isEnabled());
// Add metrics from all strategies
Map<String, Object> strategyMetrics = new HashMap<>();
strategies.forEach((name, strategy) -> {
strategyMetrics.put(name, strategy.getMetrics());
});
status.put("strategy_metrics", strategyMetrics);
return status;
}
public void setSamplingRate(double rate) {
currentConfig.setRate(rate);
activeStrategy.configure(currentConfig);
log.info("Updated sampling rate to: {}", rate);
}
public void setKeyedRate(String key, double rate) {
currentConfig.getKeyedRates().put(key, rate);
if (activeStrategy instanceof KeyBasedSampler) {
activeStrategy.configure(currentConfig);
}
log.info("Updated keyed rate for {} to: {}", key, rate);
}
private void recordSamplingMetrics(SamplingDecision decision) {
// Record to Micrometer metrics
Counter.builder("log.sampling.decisions")
.tag("strategy", decision.getStrategy())
.tag("decision", decision.shouldLog() ? "sampled" : "dropped")
.tag("reason", decision.getReason())
.register(meterRegistry)
.increment();
// Record sampling rate
Gauge.builder("log.sampling.rate",
() -> activeStrategy.getMetrics().getEffectiveRate())
.tag("strategy", activeStrategy.getName())
.register(meterRegistry);
}
private void startMetricsCollection() {
ScheduledExecutorService scheduler = Executors.newSingleThreadScheduledExecutor();
scheduler.scheduleAtFixedRate(() -> {
try {
collectHistoricalMetrics();
} catch (Exception e) {
log.warn("Failed to collect sampling metrics", e);
}
}, 0, 60, TimeUnit.SECONDS); // Collect every minute
}
private void collectHistoricalMetrics() {
String timestamp = Instant.now().toString();
strategies.forEach((name, strategy) -> {
String key = name + "_" + timestamp;
// Create a copy of current metrics
SamplingMetrics current = strategy.getMetrics();
SamplingMetrics historical = new SamplingMetrics(current.getSamplerName());
historical.setTotalEvents(current.getTotalEvents());
historical.setSampledEvents(current.getSampledEvents());
historical.setEffectiveRate(current.getEffectiveRate());
historical.setDecisionCounts(new HashMap<>(current.getDecisionCounts()));
historicalMetrics.put(key, historical);
});
// Keep only last hour of metrics
long oneHourAgo = System.currentTimeMillis() - 3600000;
historicalMetrics.keySet().removeIf(key -> {
String[] parts = key.split("_");
if (parts.length > 1) {
try {
Instant metricTime = Instant.parse(parts[1]);
return metricTime.toEpochMilli() < oneHourAgo;
} catch (Exception e) {
return true;
}
}
return true;
});
}
public Map<String, SamplingMetrics> getHistoricalMetrics() {
return new HashMap<>(historicalMetrics);
}
}
5. Logback Sampling Appender
Implement custom Logback appender with sampling.
Custom Sampling Appender:
public class SamplingAppender extends AppenderBase<ILoggingEvent> {
private SamplingManager samplingManager;
private LogEventConverter logEventConverter;
private boolean initialized = false;
@Override
public void start() {
if (samplingManager == null) {
// Try to get from Spring context or create default
initializeFromContext();
}
if (samplingManager != null) {
initialized = true;
super.start();
} else {
addError("SamplingManager not configured for SamplingAppender");
}
}
@Override
protected void append(ILoggingEvent loggingEvent) {
if (!initialized || samplingManager == null) {
// Fallback to always log if not initialized
fallbackAppend(loggingEvent);
return;
}
try {
LogEvent logEvent = logEventConverter.convert(loggingEvent);
SamplingDecision decision = samplingManager.shouldSample(logEvent);
if (decision.shouldLog()) {
logWithSamplingMetadata(loggingEvent, decision);
}
// Else: skip logging based on sampling decision
} catch (Exception e) {
// On error, log without sampling
addError("Sampling failed, falling back to regular logging", e);
fallbackAppend(loggingEvent);
}
}
private void logWithSamplingMetadata(ILoggingEvent originalEvent, SamplingDecision decision) {
// Create a new event with sampling metadata
Map<String, String> mdc = originalEvent.getMDCPropertyMap() != null ?
new HashMap<>(originalEvent.getMDCPropertyMap()) : new HashMap<>();
// Add sampling metadata to MDC
mdc.put("sampling_strategy", decision.getStrategy());
mdc.put("sampling_rate", String.valueOf(decision.getSampleRate()));
mdc.put("sampling_reason", decision.getReason());
LoggingEventWithMdc sampledEvent = new LoggingEventWithMdc(
originalEvent, mdc);
// Delegate to actual appenders
getContext().getLogger(originalEvent.getLoggerName())
.callAppenders(sampledEvent);
}
private void fallbackAppend(ILoggingEvent loggingEvent) {
// Delegate to actual appenders without sampling
getContext().getLogger(loggingEvent.getLoggerName())
.callAppenders(loggingEvent);
}
private void initializeFromContext() {
try {
// Try to get from Spring context
ApplicationContext context = getSpringApplicationContext();
if (context != null) {
this.samplingManager = context.getBean(SamplingManager.class);
this.logEventConverter = context.getBean(LogEventConverter.class);
}
} catch (Exception e) {
addWarn("Could not initialize from Spring context", e);
}
// Fallback to default initialization
if (samplingManager == null) {
this.samplingManager = createDefaultSamplingManager();
this.logEventConverter = new LogEventConverter();
}
}
private ApplicationContext getSpringApplicationContext() {
try {
// This would need to be adapted based on your application setup
return ApplicationContextProvider.getApplicationContext();
} catch (Exception e) {
return null;
}
}
private SamplingManager createDefaultSamplingManager() {
MeterRegistry meterRegistry = new SimpleMeterRegistry();
List<SamplingStrategy> strategies = Arrays.asList(
new ProbabilisticSampler(),
new RateBasedSampler(),
new AdaptiveSampler(),
new KeyBasedSampler()
);
return new SamplingManager(strategies, meterRegistry);
}
// Setter for Spring configuration
public void setSamplingManager(SamplingManager samplingManager) {
this.samplingManager = samplingManager;
}
public void setLogEventConverter(LogEventConverter logEventConverter) {
this.logEventConverter = logEventConverter;
}
}
@Component
public class LogEventConverter {
public LogEvent convert(ILoggingEvent loggingEvent) {
LogEvent event = new LogEvent();
event.setLoggerName(loggingEvent.getLoggerName());
event.setLevel(loggingEvent.getLevel().toString());
event.setMessage(loggingEvent.getFormattedMessage());
event.setThreadName(loggingEvent.getThreadName());
event.setTimestamp(loggingEvent.getTimeStamp());
event.setMdc(copyMdc(loggingEvent));
event.setThrowable(loggingEvent.getThrowableProxy() != null ?
deserializeThrowable(loggingEvent.getThrowableProxy()) : null);
// Extract tracing information
if (loggingEvent.getMDCPropertyMap() != null) {
event.setTraceId(loggingEvent.getMDCPropertyMap().get("traceId"));
event.setSpanId(loggingEvent.getMDCPropertyMap().get("spanId"));
}
return event;
}
private Map<String, String> copyMdc(ILoggingEvent event) {
if (event.getMDCPropertyMap() == null) {
return new HashMap<>();
}
return new HashMap<>(event.getMDCPropertyMap());
}
private Throwable deserializeThrowable(IThrowableProxy throwableProxy) {
// Simplified - in practice you'd want to reconstruct the throwable
return new RuntimeException(throwableProxy.getClassName() + ": " +
throwableProxy.getMessage());
}
}
// Custom logging event that allows MDC modification
class LoggingEventWithMdc implements ILoggingEvent {
private final ILoggingEvent delegate;
private final Map<String, String> mdc;
public LoggingEventWithMdc(ILoggingEvent delegate, Map<String, String> mdc) {
this.delegate = delegate;
this.mdc = mdc;
}
@Override
public String getThreadName() {
return delegate.getThreadName();
}
@Override
public Level getLevel() {
return delegate.getLevel();
}
@Override
public String getMessage() {
return delegate.getMessage();
}
@Override
public Object[] getArgumentArray() {
return delegate.getArgumentArray();
}
@Override
public String getFormattedMessage() {
return delegate.getFormattedMessage();
}
@Override
public String getLoggerName() {
return delegate.getLoggerName();
}
@Override
public LoggerContextVO getLoggerContextVO() {
return delegate.getLoggerContextVO();
}
@Override
public IThrowableProxy getThrowableProxy() {
return delegate.getThrowableProxy();
}
@Override
public StackTraceElement[] getCallerData() {
return delegate.getCallerData();
}
@Override
public boolean hasCallerData() {
return delegate.hasCallerData();
}
@Override
public long getTimeStamp() {
return delegate.getTimeStamp();
}
@Override
public long getNanoseconds() {
return delegate.getNanoseconds();
}
@Override
public Map<String, String> getMDCPropertyMap() {
return mdc;
}
@Override
public String getMDCProperty(String key) {
return mdc.get(key);
}
@Override
public void prepareForDeferredProcessing() {
delegate.prepareForDeferredProcessing();
}
}
6. Dynamic Configuration Management
Manage sampling configuration at runtime.
Configuration Service:
@Service
@Slf4j
public class SamplingConfigurationService {
private final SamplingManager samplingManager;
private final Map<String, SamplingConfig> configTemplates;
private final String configFilePath;
public SamplingConfigurationService(SamplingManager samplingManager,
@Value("${app.sampling.config-file:sampling-config.json}")
String configFilePath) {
this.samplingManager = samplingManager;
this.configFilePath = configFilePath;
this.configTemplates = loadConfigTemplates();
// Load initial configuration
loadInitialConfiguration();
}
public void updateConfiguration(SamplingConfig newConfig) {
try {
samplingManager.configureSampling(newConfig);
saveConfiguration(newConfig);
log.info("Updated sampling configuration: strategy={}, rate={}",
newConfig.getStrategy(), newConfig.getRate());
} catch (Exception e) {
log.error("Failed to update sampling configuration", e);
throw new RuntimeException("Configuration update failed", e);
}
}
public void applyTemplate(String templateName) {
SamplingConfig template = configTemplates.get(templateName);
if (template != null) {
updateConfiguration(template);
} else {
throw new IllegalArgumentException("Unknown template: " + templateName);
}
}
public void setSamplingRate(double rate) {
SamplingConfig current = getCurrentConfig();
current.setRate(rate);
updateConfiguration(current);
}
public void enableSampling() {
SamplingConfig config = getCurrentConfig();
config.setEnabled(true);
updateConfiguration(config);
}
public void disableSampling() {
SamplingConfig config = getCurrentConfig();
config.setEnabled(false);
updateConfiguration(config);
}
public SamplingConfig getCurrentConfig() {
Map<String, Object> status = samplingManager.getSamplingStatus();
return (SamplingConfig) status.get("config");
}
public Map<String, Object> getSamplingStatus() {
return samplingManager.getSamplingStatus();
}
public void reloadConfiguration() {
loadInitialConfiguration();
}
private void loadInitialConfiguration() {
try {
Path path = Paths.get(configFilePath);
if (Files.exists(path)) {
ObjectMapper mapper = new ObjectMapper();
SamplingConfig config = mapper.readValue(path.toFile(), SamplingConfig.class);
samplingManager.configureSampling(config);
log.info("Loaded sampling configuration from: {}", configFilePath);
} else {
// Use default configuration
SamplingConfig defaultConfig = new SamplingConfig();
samplingManager.configureSampling(defaultConfig);
log.info("Using default sampling configuration");
}
} catch (Exception e) {
log.error("Failed to load sampling configuration, using defaults", e);
SamplingConfig defaultConfig = new SamplingConfig();
samplingManager.configureSampling(defaultConfig);
}
}
private void saveConfiguration(SamplingConfig config) {
try {
ObjectMapper mapper = new ObjectMapper();
mapper.writerWithDefaultPrettyPrinter()
.writeValue(Paths.get(configFilePath).toFile(), config);
} catch (Exception e) {
log.error("Failed to save sampling configuration", e);
}
}
private Map<String, SamplingConfig> loadConfigTemplates() {
Map<String, SamplingConfig> templates = new HashMap<>();
// Development template - sample everything
SamplingConfig devConfig = new SamplingConfig();
devConfig.setStrategy("PROBABILISTIC");
devConfig.setRate(1.0);
devConfig.setEnabled(true);
templates.put("development", devConfig);
// Production template - sample 10%
SamplingConfig prodConfig = new SamplingConfig();
prodConfig.setStrategy("ADAPTIVE");
prodConfig.setRate(0.1);
prodConfig.setEnabled(true);
templates.put("production", prodConfig);
// Debug template - sample more during debugging
SamplingConfig debugConfig = new SamplingConfig();
debugConfig.setStrategy("KEY_BASED");
debugConfig.setEnabled(true);
debugConfig.setKeyedRates(Map.of(
"com.example:DEBUG", 0.5,
"com.example:INFO", 0.2,
"com.example:ERROR", 1.0
));
templates.put("debug", debugConfig);
return templates;
}
@Scheduled(fixedRate = 300000) // Every 5 minutes
public void adaptiveConfigurationAdjustment() {
if (!isAdaptiveStrategyActive()) {
return;
}
Map<String, Object> status = samplingManager.getSamplingStatus();
Map<String, Object> metrics = (Map<String, Object>) status.get("strategy_metrics");
Map<String, Object> adaptiveMetrics = (Map<String, Object>) metrics.get("ADAPTIVE");
if (adaptiveMetrics != null) {
double effectiveRate = (Double) adaptiveMetrics.get("effectiveRate");
SamplingConfig currentConfig = getCurrentConfig();
// Adjust if we're significantly off target
double targetRate = currentConfig.getRate();
if (Math.abs(effectiveRate - targetRate) > 0.2) {
log.info("Adjusting sampling configuration. Effective rate: {}, Target: {}",
effectiveRate, targetRate);
// Could implement more sophisticated adjustment logic here
}
}
}
private boolean isAdaptiveStrategyActive() {
SamplingConfig config = getCurrentConfig();
return "ADAPTIVE".equals(config.getStrategy());
}
}
7. REST API for Sampling Management
Expose sampling configuration as web services.
Sampling Controller:
@RestController
@RequestMapping("/api/sampling")
@Slf4j
public class SamplingController {
private final SamplingConfigurationService configService;
private final SamplingManager samplingManager;
public SamplingController(SamplingConfigurationService configService,
SamplingManager samplingManager) {
this.configService = configService;
this.samplingManager = samplingManager;
}
@GetMapping("/status")
public ResponseEntity<Map<String, Object>> getSamplingStatus() {
return ResponseEntity.ok(configService.getSamplingStatus());
}
@PostMapping("/configure")
public ResponseEntity<String> configureSampling(@RequestBody SamplingConfig config) {
try {
configService.updateConfiguration(config);
return ResponseEntity.ok("Sampling configuration updated successfully");
} catch (Exception e) {
return ResponseEntity.badRequest()
.body("Failed to update configuration: " + e.getMessage());
}
}
@PostMapping("/template/{templateName}")
public ResponseEntity<String> applyTemplate(@PathVariable String templateName) {
try {
configService.applyTemplate(templateName);
return ResponseEntity.ok("Applied template: " + templateName);
} catch (Exception e) {
return ResponseEntity.badRequest()
.body("Failed to apply template: " + e.getMessage());
}
}
@PostMapping("/rate")
public ResponseEntity<String> setSamplingRate(@RequestParam double rate) {
try {
configService.setSamplingRate(rate);
return ResponseEntity.ok("Sampling rate set to: " + rate);
} catch (Exception e) {
return ResponseEntity.badRequest()
.body("Failed to set sampling rate: " + e.getMessage());
}
}
@PostMapping("/enable")
public ResponseEntity<String> enableSampling() {
configService.enableSampling();
return ResponseEntity.ok("Sampling enabled");
}
@PostMapping("/disable")
public ResponseEntity<String> disableSampling() {
configService.disableSampling();
return ResponseEntity.ok("Sampling disabled");
}
@GetMapping("/metrics")
public ResponseEntity<Map<String, SamplingMetrics>> getMetrics() {
return ResponseEntity.ok(samplingManager.getHistoricalMetrics());
}
@PostMapping("/keyed-rate")
public ResponseEntity<String> setKeyedRate(
@RequestParam String key,
@RequestParam double rate) {
try {
samplingManager.setKeyedRate(key, rate);
return ResponseEntity.ok("Set rate for " + key + " to: " + rate);
} catch (Exception e) {
return ResponseEntity.badRequest()
.body("Failed to set keyed rate: " + e.getMessage());
}
}
@PostMapping("/reload")
public ResponseEntity<String> reloadConfiguration() {
configService.reloadConfiguration();
return ResponseEntity.ok("Configuration reloaded");
}
}
8. Logback Configuration
Configure Logback with sampling appender.
logback-spring.xml:
<?xml version="1.0" encoding="UTF-8"?> <configuration> <!-- Spring Profile-based configuration --> <springProfile name="dev"> <include resource="console-appender.xml" /> </springProfile> <springProfile name="prod"> <include resource="file-appender.xml" /> </springProfile> <!-- Sampling Appender Configuration --> <appender name="SAMPLING" class="com.example.logging.SamplingAppender"> <!-- These will be injected by Spring --> </appender> <!-- Async Appender for Better Performance --> <appender name="ASYNC_SAMPLING" class="ch.qos.logback.classic.AsyncAppender"> <appender-ref ref="SAMPLING" /> <queueSize>10000</queueSize> <discardingThreshold>0</discardingThreshold> <includeCallerData>false</includeCallerData> <neverBlock>true</neverBlock> </appender> <!-- Root Logger with Sampling --> <root level="INFO"> <appender-ref ref="ASYNC_SAMPLING" /> </root> <!-- Specific logger configurations with different sampling rates --> <logger name="com.example.service.PaymentService" level="DEBUG" additivity="false"> <appender-ref ref="ASYNC_SAMPLING" /> </logger> <logger name="com.example.security" level="WARN" additivity="false"> <appender-ref ref="ASYNC_SAMPLING" /> </logger> <!-- Always log errors regardless of sampling --> <turboFilter class="ch.qos.logback.classic.turbo.DuplicateMessageFilter"> <AllowedRepetitions>0</AllowedRepetitions> </turboFilter> </configuration>
Best Practices for Production Log Sampling
- Start Conservative: Begin with higher sampling rates and adjust based on volume
- Monitor Effectiveness: Track sampling rates and adjust strategies based on actual usage
- Preserve Errors: Always sample error-level logs and important business events
- Consider Costs: Balance storage costs against debugging needs
- Dynamic Configuration: Enable runtime configuration changes without redeployment
- Test Thoroughly: Verify sampling behavior doesn't break log-based monitoring
- Document Strategy: Clearly document sampling strategy for troubleshooting
Conclusion: Intelligent Log Volume Management
Log sampling in Java production environments is essential for maintaining system performance while preserving observability. By implementing sophisticated sampling strategies that adapt to runtime conditions, prioritize important events, and provide operational control, organizations can achieve the optimal balance between log volume and debugging capability.
This implementation demonstrates that effective log sampling isn't about simply reducing volume—it's about making intelligent decisions about which logs provide the most value, ensuring that when issues occur, the right information is available while maintaining system performance and controlling costs.