Log sampling is a critical technique for production systems where high-volume logging can impact performance, storage costs, and signal-to-noise ratio. Java provides sophisticated approaches to implement intelligent log sampling that captures meaningful data while controlling volume. This guide covers practical patterns for implementing effective log sampling strategies in production Java applications.

Understanding Log Sampling Strategies

Common Sampling Approaches:

• Rate-based sampling (Nth occurrence)
• Probabilistic sampling (random percentage)
• Dynamic sampling (adaptive based on conditions)
• Structured sampling (based on log content/context)
• Head-based vs tail-based sampling

Core Implementation Patterns

1. Project Setup and Dependencies

Configure logging dependencies with sampling support.

Maven Configuration:

<dependencies> <!-- Logback with Sampling Support --> <dependency> <groupId>ch.qos.logback</groupId> <artifactId>logback-classic</artifactId> <version>1.4.11</version> </dependency> <!-- Logstash Logback Encoder --> <dependency> <groupId>net.logstash.logback</groupId> <artifactId>logstash-logback-encoder</artifactId> <version>7.4</version> </dependency> <!-- Micrometer for Metrics --> <dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-core</artifactId> <version>1.11.5</version> </dependency> <!-- Configuration --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter</artifactId> <version>3.1.0</version> </dependency> <!-- Cache --> <dependency> <groupId>com.github.ben-manes.caffeine</groupId> <artifactId>caffeine</artifactId> <version>3.1.6</version> </dependency> </dependencies>

2. Domain Models for Log Sampling

Create comprehensive models for sampling configuration and state.

Core Domain Models:

@Data public class SamplingConfig { private String strategy = "PROBABILISTIC"; private double rate = 0.1; // 10% for probabilistic private int interval = 100; // For rate-based private Map<String, Double> keyedRates = new HashMap<>(); private boolean enabled = true; private int windowSize = 1000; private double burstThreshold = 5.0; private Map<String, Object> customRules = new HashMap<>(); public enum Strategy { PROBABILISTIC, // Random sampling RATE_BASED, // Every Nth occurrence ADAPTIVE, // Dynamic based on conditions CONTENT_BASED, // Based on log message content BURST_DETECTION, // Sample more during bursts KEY_BASED // Different rates for different log keys } } @Data public class LogEvent { private String loggerName; private String level; private String message; private String threadName; private long timestamp; private Map<String, String> mdc; private Throwable throwable; private String traceId; private String spanId; private Map<String, Object> customFields; public String getLogKey() { return loggerName + ":" + level; } public boolean isError() { return "ERROR".equals(level) || "WARN".equals(level); } public boolean containsKeyword(String keyword) { return message != null && message.toLowerCase().contains(keyword.toLowerCase()); } } @Data public class SamplingDecision { private boolean shouldLog; private String reason; private double sampleRate; private String strategy; private Map<String, Object> metadata; public static SamplingDecision sample() { return new SamplingDecision(true, "sampled", 1.0, "DEFAULT"); } public static SamplingDecision skip() { return new SamplingDecision(false, "sampled", 0.0, "DEFAULT"); } public SamplingDecision(boolean shouldLog, String reason, double sampleRate, String strategy) { this.shouldLog = shouldLog; this.reason = reason; this.sampleRate = sampleRate; this.strategy = strategy; this.metadata = new HashMap<>(); } } @Data public class SamplingMetrics { private String samplerName; private long totalEvents; private long sampledEvents; private double effectiveRate; private long windowStart; private long windowEnd; private Map<String, Long> decisionCounts; private double currentLoad; public SamplingMetrics(String samplerName) { this.samplerName = samplerName; this.decisionCounts = new HashMap<>(); this.windowStart = System.currentTimeMillis(); } public void recordDecision(boolean sampled, String reason) { totalEvents++; if (sampled) { sampledEvents++; } decisionCounts.merge(reason, 1L, Long::sum); effectiveRate = totalEvents > 0 ? (double) sampledEvents / totalEvents : 0; } public void resetWindow() { windowStart = System.currentTimeMillis(); totalEvents = 0; sampledEvents = 0; decisionCounts.clear(); } } @Data public class BurstDetection { private final int windowSize; private final double threshold; private final Deque<Long> eventTimestamps = new LinkedList<>(); private long lastBurstDetected = 0; public BurstDetection(int windowSize, double threshold) { this.windowSize = windowSize; this.threshold = threshold; } public boolean isBurst() { long now = System.currentTimeMillis(); long windowStart = now - (windowSize * 1000L); // Remove old events while (!eventTimestamps.isEmpty() && eventTimestamps.peekFirst() < windowStart) { eventTimestamps.pollFirst(); } double eventsPerSecond = (double) eventTimestamps.size() / windowSize; return eventsPerSecond > threshold; } public void recordEvent() { eventTimestamps.addLast(System.currentTimeMillis()); if (isBurst()) { lastBurstDetected = System.currentTimeMillis(); } } public boolean isInBurstCoolDown() { return System.currentTimeMillis() - lastBurstDetected < 30000; // 30 second cooldown } }

3. Core Sampling Strategies

Implement various sampling strategies.

Sampling Strategy Interface:

public interface SamplingStrategy { SamplingDecision shouldSample(LogEvent event); String getName(); void configure(SamplingConfig config); SamplingMetrics getMetrics(); } @Component @Slf4j public class ProbabilisticSampler implements SamplingStrategy { private final Random random = new Random(); private double sampleRate; private final SamplingMetrics metrics; public ProbabilisticSampler() { this.metrics = new SamplingMetrics("probabilistic"); this.sampleRate = 0.1; // Default 10% } @Override public SamplingDecision shouldSample(LogEvent event) { metrics.recordDecision(false, "evaluated"); // Always sample errors if (event.isError()) { metrics.recordDecision(true, "error"); return new SamplingDecision(true, "error", 1.0, getName()); } // Sample based on probability boolean shouldSample = random.nextDouble() < sampleRate; String reason = shouldSample ? "probabilistic" : "sampled_out"; metrics.recordDecision(shouldSample, reason); return new SamplingDecision(shouldSample, reason, sampleRate, getName()); } @Override public String getName() { return "PROBABILISTIC"; } @Override public void configure(SamplingConfig config) { this.sampleRate = config.getRate(); log.info("Configured probabilistic sampler with rate: {}", sampleRate); } @Override public SamplingMetrics getMetrics() { return metrics; } } @Component @Slf4j public class RateBasedSampler implements SamplingStrategy { private final AtomicLong counter = new AtomicLong(0); private int sampleInterval; private final SamplingMetrics metrics; public RateBasedSampler() { this.metrics = new SamplingMetrics("rate_based"); this.sampleInterval = 100; // Default: sample every 100th event } @Override public SamplingDecision shouldSample(LogEvent event) { metrics.recordDecision(false, "evaluated"); // Always sample errors if (event.isError()) { metrics.recordDecision(true, "error"); return new SamplingDecision(true, "error", 1.0, getName()); } long count = counter.incrementAndGet(); boolean shouldSample = count % sampleInterval == 0; String reason = shouldSample ? "rate_based" : "sampled_out"; metrics.recordDecision(shouldSample, reason); return new SamplingDecision(shouldSample, reason, 1.0 / sampleInterval, getName()); } @Override public String getName() { return "RATE_BASED"; } @Override public void configure(SamplingConfig config) { this.sampleInterval = config.getInterval(); log.info("Configured rate-based sampler with interval: {}", sampleInterval); } @Override public SamplingMetrics getMetrics() { return metrics; } } @Component @Slf4j public class AdaptiveSampler implements SamplingStrategy { private double baseRate; private double currentRate; private final BurstDetection burstDetection; private final SamplingMetrics metrics; private long lastAdjustment = System.currentTimeMillis(); public AdaptiveSampler() { this.metrics = new SamplingMetrics("adaptive"); this.baseRate = 0.1; this.currentRate = 0.1; this.burstDetection = new BurstDetection(10, 5.0); // 10s window, 5 events/sec threshold } @Override public SamplingDecision shouldSample(LogEvent event) { metrics.recordDecision(false, "evaluated"); burstDetection.recordEvent(); // Adjust rate based on conditions adjustSamplingRate(); // Always sample errors and important keywords if (shouldAlwaysSample(event)) { metrics.recordDecision(true, "important"); return new SamplingDecision(true, "important", 1.0, getName()); } // Sample based on current adaptive rate boolean shouldSample = Math.random() < currentRate; String reason = shouldSample ? "adaptive" : "sampled_out"; metrics.recordDecision(shouldSample, reason); SamplingDecision decision = new SamplingDecision(shouldSample, reason, currentRate, getName()); decision.getMetadata().put("current_rate", currentRate); decision.getMetadata().put("base_rate", baseRate); return decision; } private boolean shouldAlwaysSample(LogEvent event) { if (event.isError()) return true; // Sample important business events String message = event.getMessage(); if (message != null) { return message.contains("transaction") || message.contains("payment") || message.contains("audit") || message.contains("security"); } return false; } private void adjustSamplingRate() { long now = System.currentTimeMillis(); if (now - lastAdjustment < 5000) { // Adjust every 5 seconds return; } double currentLoad = calculateCurrentLoad(); if (burstDetection.isBurst()) { // Reduce sampling during bursts to control volume currentRate = Math.max(baseRate * 0.1, 0.01); log.debug("Burst detected, reducing sampling rate to: {}", currentRate); } else if (currentLoad > 0.8) { // High load, reduce sampling currentRate = Math.max(baseRate * 0.5, 0.05); } else if (currentLoad < 0.2) { // Low load, increase sampling currentRate = Math.min(baseRate * 2.0, 1.0); } else { // Normal load, use base rate currentRate = baseRate; } lastAdjustment = now; } private double calculateCurrentLoad() { // Simplified load calculation based on recent sampling rate double targetEventsPerSecond = 1000; // Target volume double actualEventsPerSecond = metrics.getTotalEvents() / 10.0; // Last 10 seconds return Math.min(actualEventsPerSecond / targetEventsPerSecond, 1.0); } @Override public String getName() { return "ADAPTIVE"; } @Override public void configure(SamplingConfig config) { this.baseRate = config.getRate(); this.currentRate = config.getRate(); log.info("Configured adaptive sampler with base rate: {}", baseRate); } @Override public SamplingMetrics getMetrics() { return metrics; } } @Component @Slf4j public class KeyBasedSampler implements SamplingStrategy { private final Map<String, Double> keyRates; private final Double defaultRate; private final SamplingMetrics metrics; public KeyBasedSampler() { this.metrics = new SamplingMetrics("key_based"); this.keyRates = new HashMap<>(); this.defaultRate = 0.1; // Default rates for common loggers keyRates.put("com.example.service.PaymentService:INFO", 1.0); // Always log payments keyRates.put("com.example.controller.AuthController:DEBUG", 0.01); // Rarely log auth debug } @Override public SamplingDecision shouldSample(LogEvent event) { metrics.recordDecision(false, "evaluated"); String logKey = event.getLogKey(); double rate = keyRates.getOrDefault(logKey, defaultRate); boolean shouldSample = Math.random() < rate; String reason = shouldSample ? "key_based" : "sampled_out"; metrics.recordDecision(shouldSample, reason); SamplingDecision decision = new SamplingDecision(shouldSample, reason, rate, getName()); decision.getMetadata().put("log_key", logKey); decision.getMetadata().put("key_rate", rate); return decision; } @Override public String getName() { return "KEY_BASED"; } @Override public void configure(SamplingConfig config) { keyRates.clear(); if (config.getKeyedRates() != null) { keyRates.putAll(config.getKeyedRates()); } log.info("Configured key-based sampler with {} key rates", keyRates.size()); } @Override public SamplingMetrics getMetrics() { return metrics; } }

4. Sampling Manager and Orchestration

Coordinate multiple sampling strategies.

Sampling Manager:

@Service @Slf4j public class SamplingManager { private final Map<String, SamplingStrategy> strategies; private SamplingStrategy activeStrategy; private SamplingConfig currentConfig; private final MeterRegistry meterRegistry; private final Map<String, SamplingMetrics> historicalMetrics; public SamplingManager(List<SamplingStrategy> strategyList, MeterRegistry meterRegistry) { this.strategies = strategyList.stream() .collect(Collectors.toMap(SamplingStrategy::getName, Function.identity())); this.meterRegistry = meterRegistry; this.historicalMetrics = new ConcurrentHashMap<>(); this.activeStrategy = strategies.get("PROBABILISTIC"); this.currentConfig = new SamplingConfig(); // Initialize metrics collection startMetricsCollection(); } public SamplingDecision shouldSample(LogEvent event) { if (!currentConfig.isEnabled()) { return SamplingDecision.sample(); } SamplingDecision decision = activeStrategy.shouldSample(event); // Record metrics recordSamplingMetrics(decision); return decision; } public void configureSampling(SamplingConfig config) { this.currentConfig = config; SamplingStrategy strategy = strategies.get(config.getStrategy()); if (strategy != null) { strategy.configure(config); this.activeStrategy = strategy; log.info("Switched to sampling strategy: {}", config.getStrategy()); } else { log.warn("Unknown sampling strategy: {}, keeping current: {}", config.getStrategy(), activeStrategy.getName()); } } public Map<String, Object> getSamplingStatus() { Map<String, Object> status = new HashMap<>(); status.put("active_strategy", activeStrategy.getName()); status.put("config", currentConfig); status.put("enabled", currentConfig.isEnabled()); // Add metrics from all strategies Map<String, Object> strategyMetrics = new HashMap<>(); strategies.forEach((name, strategy) -> { strategyMetrics.put(name, strategy.getMetrics()); }); status.put("strategy_metrics", strategyMetrics); return status; } public void setSamplingRate(double rate) { currentConfig.setRate(rate); activeStrategy.configure(currentConfig); log.info("Updated sampling rate to: {}", rate); } public void setKeyedRate(String key, double rate) { currentConfig.getKeyedRates().put(key, rate); if (activeStrategy instanceof KeyBasedSampler) { activeStrategy.configure(currentConfig); } log.info("Updated keyed rate for {} to: {}", key, rate); } private void recordSamplingMetrics(SamplingDecision decision) { // Record to Micrometer metrics Counter.builder("log.sampling.decisions") .tag("strategy", decision.getStrategy()) .tag("decision", decision.shouldLog() ? "sampled" : "dropped") .tag("reason", decision.getReason()) .register(meterRegistry) .increment(); // Record sampling rate Gauge.builder("log.sampling.rate", () -> activeStrategy.getMetrics().getEffectiveRate()) .tag("strategy", activeStrategy.getName()) .register(meterRegistry); } private void startMetricsCollection() { ScheduledExecutorService scheduler = Executors.newSingleThreadScheduledExecutor(); scheduler.scheduleAtFixedRate(() -> { try { collectHistoricalMetrics(); } catch (Exception e) { log.warn("Failed to collect sampling metrics", e); } }, 0, 60, TimeUnit.SECONDS); // Collect every minute } private void collectHistoricalMetrics() { String timestamp = Instant.now().toString(); strategies.forEach((name, strategy) -> { String key = name + "_" + timestamp; // Create a copy of current metrics SamplingMetrics current = strategy.getMetrics(); SamplingMetrics historical = new SamplingMetrics(current.getSamplerName()); historical.setTotalEvents(current.getTotalEvents()); historical.setSampledEvents(current.getSampledEvents()); historical.setEffectiveRate(current.getEffectiveRate()); historical.setDecisionCounts(new HashMap<>(current.getDecisionCounts())); historicalMetrics.put(key, historical); }); // Keep only last hour of metrics long oneHourAgo = System.currentTimeMillis() - 3600000; historicalMetrics.keySet().removeIf(key -> { String[] parts = key.split("_"); if (parts.length > 1) { try { Instant metricTime = Instant.parse(parts[1]); return metricTime.toEpochMilli() < oneHourAgo; } catch (Exception e) { return true; } } return true; }); } public Map<String, SamplingMetrics> getHistoricalMetrics() { return new HashMap<>(historicalMetrics); } }

5. Logback Sampling Appender

Implement custom Logback appender with sampling.

Custom Sampling Appender:

public class SamplingAppender extends AppenderBase<ILoggingEvent> { private SamplingManager samplingManager; private LogEventConverter logEventConverter; private boolean initialized = false; @Override public void start() { if (samplingManager == null) { // Try to get from Spring context or create default initializeFromContext(); } if (samplingManager != null) { initialized = true; super.start(); } else { addError("SamplingManager not configured for SamplingAppender"); } } @Override protected void append(ILoggingEvent loggingEvent) { if (!initialized || samplingManager == null) { // Fallback to always log if not initialized fallbackAppend(loggingEvent); return; } try { LogEvent logEvent = logEventConverter.convert(loggingEvent); SamplingDecision decision = samplingManager.shouldSample(logEvent); if (decision.shouldLog()) { logWithSamplingMetadata(loggingEvent, decision); } // Else: skip logging based on sampling decision } catch (Exception e) { // On error, log without sampling addError("Sampling failed, falling back to regular logging", e); fallbackAppend(loggingEvent); } } private void logWithSamplingMetadata(ILoggingEvent originalEvent, SamplingDecision decision) { // Create a new event with sampling metadata Map<String, String> mdc = originalEvent.getMDCPropertyMap() != null ? new HashMap<>(originalEvent.getMDCPropertyMap()) : new HashMap<>(); // Add sampling metadata to MDC mdc.put("sampling_strategy", decision.getStrategy()); mdc.put("sampling_rate", String.valueOf(decision.getSampleRate())); mdc.put("sampling_reason", decision.getReason()); LoggingEventWithMdc sampledEvent = new LoggingEventWithMdc( originalEvent, mdc); // Delegate to actual appenders getContext().getLogger(originalEvent.getLoggerName()) .callAppenders(sampledEvent); } private void fallbackAppend(ILoggingEvent loggingEvent) { // Delegate to actual appenders without sampling getContext().getLogger(loggingEvent.getLoggerName()) .callAppenders(loggingEvent); } private void initializeFromContext() { try { // Try to get from Spring context ApplicationContext context = getSpringApplicationContext(); if (context != null) { this.samplingManager = context.getBean(SamplingManager.class); this.logEventConverter = context.getBean(LogEventConverter.class); } } catch (Exception e) { addWarn("Could not initialize from Spring context", e); } // Fallback to default initialization if (samplingManager == null) { this.samplingManager = createDefaultSamplingManager(); this.logEventConverter = new LogEventConverter(); } } private ApplicationContext getSpringApplicationContext() { try { // This would need to be adapted based on your application setup return ApplicationContextProvider.getApplicationContext(); } catch (Exception e) { return null; } } private SamplingManager createDefaultSamplingManager() { MeterRegistry meterRegistry = new SimpleMeterRegistry(); List<SamplingStrategy> strategies = Arrays.asList( new ProbabilisticSampler(), new RateBasedSampler(), new AdaptiveSampler(), new KeyBasedSampler() ); return new SamplingManager(strategies, meterRegistry); } // Setter for Spring configuration public void setSamplingManager(SamplingManager samplingManager) { this.samplingManager = samplingManager; } public void setLogEventConverter(LogEventConverter logEventConverter) { this.logEventConverter = logEventConverter; } } @Component public class LogEventConverter { public LogEvent convert(ILoggingEvent loggingEvent) { LogEvent event = new LogEvent(); event.setLoggerName(loggingEvent.getLoggerName()); event.setLevel(loggingEvent.getLevel().toString()); event.setMessage(loggingEvent.getFormattedMessage()); event.setThreadName(loggingEvent.getThreadName()); event.setTimestamp(loggingEvent.getTimeStamp()); event.setMdc(copyMdc(loggingEvent)); event.setThrowable(loggingEvent.getThrowableProxy() != null ? deserializeThrowable(loggingEvent.getThrowableProxy()) : null); // Extract tracing information if (loggingEvent.getMDCPropertyMap() != null) { event.setTraceId(loggingEvent.getMDCPropertyMap().get("traceId")); event.setSpanId(loggingEvent.getMDCPropertyMap().get("spanId")); } return event; } private Map<String, String> copyMdc(ILoggingEvent event) { if (event.getMDCPropertyMap() == null) { return new HashMap<>(); } return new HashMap<>(event.getMDCPropertyMap()); } private Throwable deserializeThrowable(IThrowableProxy throwableProxy) { // Simplified - in practice you'd want to reconstruct the throwable return new RuntimeException(throwableProxy.getClassName() + ": " + throwableProxy.getMessage()); } } // Custom logging event that allows MDC modification class LoggingEventWithMdc implements ILoggingEvent { private final ILoggingEvent delegate; private final Map<String, String> mdc; public LoggingEventWithMdc(ILoggingEvent delegate, Map<String, String> mdc) { this.delegate = delegate; this.mdc = mdc; } @Override public String getThreadName() { return delegate.getThreadName(); } @Override public Level getLevel() { return delegate.getLevel(); } @Override public String getMessage() { return delegate.getMessage(); } @Override public Object[] getArgumentArray() { return delegate.getArgumentArray(); } @Override public String getFormattedMessage() { return delegate.getFormattedMessage(); } @Override public String getLoggerName() { return delegate.getLoggerName(); } @Override public LoggerContextVO getLoggerContextVO() { return delegate.getLoggerContextVO(); } @Override public IThrowableProxy getThrowableProxy() { return delegate.getThrowableProxy(); } @Override public StackTraceElement[] getCallerData() { return delegate.getCallerData(); } @Override public boolean hasCallerData() { return delegate.hasCallerData(); } @Override public long getTimeStamp() { return delegate.getTimeStamp(); } @Override public long getNanoseconds() { return delegate.getNanoseconds(); } @Override public Map<String, String> getMDCPropertyMap() { return mdc; } @Override public String getMDCProperty(String key) { return mdc.get(key); } @Override public void prepareForDeferredProcessing() { delegate.prepareForDeferredProcessing(); } }

6. Dynamic Configuration Management

Manage sampling configuration at runtime.

Configuration Service:

@Service @Slf4j public class SamplingConfigurationService { private final SamplingManager samplingManager; private final Map<String, SamplingConfig> configTemplates; private final String configFilePath; public SamplingConfigurationService(SamplingManager samplingManager, @Value("${app.sampling.config-file:sampling-config.json}") String configFilePath) { this.samplingManager = samplingManager; this.configFilePath = configFilePath; this.configTemplates = loadConfigTemplates(); // Load initial configuration loadInitialConfiguration(); } public void updateConfiguration(SamplingConfig newConfig) { try { samplingManager.configureSampling(newConfig); saveConfiguration(newConfig); log.info("Updated sampling configuration: strategy={}, rate={}", newConfig.getStrategy(), newConfig.getRate()); } catch (Exception e) { log.error("Failed to update sampling configuration", e); throw new RuntimeException("Configuration update failed", e); } } public void applyTemplate(String templateName) { SamplingConfig template = configTemplates.get(templateName); if (template != null) { updateConfiguration(template); } else { throw new IllegalArgumentException("Unknown template: " + templateName); } } public void setSamplingRate(double rate) { SamplingConfig current = getCurrentConfig(); current.setRate(rate); updateConfiguration(current); } public void enableSampling() { SamplingConfig config = getCurrentConfig(); config.setEnabled(true); updateConfiguration(config); } public void disableSampling() { SamplingConfig config = getCurrentConfig(); config.setEnabled(false); updateConfiguration(config); } public SamplingConfig getCurrentConfig() { Map<String, Object> status = samplingManager.getSamplingStatus(); return (SamplingConfig) status.get("config"); } public Map<String, Object> getSamplingStatus() { return samplingManager.getSamplingStatus(); } public void reloadConfiguration() { loadInitialConfiguration(); } private void loadInitialConfiguration() { try { Path path = Paths.get(configFilePath); if (Files.exists(path)) { ObjectMapper mapper = new ObjectMapper(); SamplingConfig config = mapper.readValue(path.toFile(), SamplingConfig.class); samplingManager.configureSampling(config); log.info("Loaded sampling configuration from: {}", configFilePath); } else { // Use default configuration SamplingConfig defaultConfig = new SamplingConfig(); samplingManager.configureSampling(defaultConfig); log.info("Using default sampling configuration"); } } catch (Exception e) { log.error("Failed to load sampling configuration, using defaults", e); SamplingConfig defaultConfig = new SamplingConfig(); samplingManager.configureSampling(defaultConfig); } } private void saveConfiguration(SamplingConfig config) { try { ObjectMapper mapper = new ObjectMapper(); mapper.writerWithDefaultPrettyPrinter() .writeValue(Paths.get(configFilePath).toFile(), config); } catch (Exception e) { log.error("Failed to save sampling configuration", e); } } private Map<String, SamplingConfig> loadConfigTemplates() { Map<String, SamplingConfig> templates = new HashMap<>(); // Development template - sample everything SamplingConfig devConfig = new SamplingConfig(); devConfig.setStrategy("PROBABILISTIC"); devConfig.setRate(1.0); devConfig.setEnabled(true); templates.put("development", devConfig); // Production template - sample 10% SamplingConfig prodConfig = new SamplingConfig(); prodConfig.setStrategy("ADAPTIVE"); prodConfig.setRate(0.1); prodConfig.setEnabled(true); templates.put("production", prodConfig); // Debug template - sample more during debugging SamplingConfig debugConfig = new SamplingConfig(); debugConfig.setStrategy("KEY_BASED"); debugConfig.setEnabled(true); debugConfig.setKeyedRates(Map.of( "com.example:DEBUG", 0.5, "com.example:INFO", 0.2, "com.example:ERROR", 1.0 )); templates.put("debug", debugConfig); return templates; } @Scheduled(fixedRate = 300000) // Every 5 minutes public void adaptiveConfigurationAdjustment() { if (!isAdaptiveStrategyActive()) { return; } Map<String, Object> status = samplingManager.getSamplingStatus(); Map<String, Object> metrics = (Map<String, Object>) status.get("strategy_metrics"); Map<String, Object> adaptiveMetrics = (Map<String, Object>) metrics.get("ADAPTIVE"); if (adaptiveMetrics != null) { double effectiveRate = (Double) adaptiveMetrics.get("effectiveRate"); SamplingConfig currentConfig = getCurrentConfig(); // Adjust if we're significantly off target double targetRate = currentConfig.getRate(); if (Math.abs(effectiveRate - targetRate) > 0.2) { log.info("Adjusting sampling configuration. Effective rate: {}, Target: {}", effectiveRate, targetRate); // Could implement more sophisticated adjustment logic here } } } private boolean isAdaptiveStrategyActive() { SamplingConfig config = getCurrentConfig(); return "ADAPTIVE".equals(config.getStrategy()); } }

7. REST API for Sampling Management

Expose sampling configuration as web services.

Sampling Controller:

@RestController @RequestMapping("/api/sampling") @Slf4j public class SamplingController { private final SamplingConfigurationService configService; private final SamplingManager samplingManager; public SamplingController(SamplingConfigurationService configService, SamplingManager samplingManager) { this.configService = configService; this.samplingManager = samplingManager; } @GetMapping("/status") public ResponseEntity<Map<String, Object>> getSamplingStatus() { return ResponseEntity.ok(configService.getSamplingStatus()); } @PostMapping("/configure") public ResponseEntity<String> configureSampling(@RequestBody SamplingConfig config) { try { configService.updateConfiguration(config); return ResponseEntity.ok("Sampling configuration updated successfully"); } catch (Exception e) { return ResponseEntity.badRequest() .body("Failed to update configuration: " + e.getMessage()); } } @PostMapping("/template/{templateName}") public ResponseEntity<String> applyTemplate(@PathVariable String templateName) { try { configService.applyTemplate(templateName); return ResponseEntity.ok("Applied template: " + templateName); } catch (Exception e) { return ResponseEntity.badRequest() .body("Failed to apply template: " + e.getMessage()); } } @PostMapping("/rate") public ResponseEntity<String> setSamplingRate(@RequestParam double rate) { try { configService.setSamplingRate(rate); return ResponseEntity.ok("Sampling rate set to: " + rate); } catch (Exception e) { return ResponseEntity.badRequest() .body("Failed to set sampling rate: " + e.getMessage()); } } @PostMapping("/enable") public ResponseEntity<String> enableSampling() { configService.enableSampling(); return ResponseEntity.ok("Sampling enabled"); } @PostMapping("/disable") public ResponseEntity<String> disableSampling() { configService.disableSampling(); return ResponseEntity.ok("Sampling disabled"); } @GetMapping("/metrics") public ResponseEntity<Map<String, SamplingMetrics>> getMetrics() { return ResponseEntity.ok(samplingManager.getHistoricalMetrics()); } @PostMapping("/keyed-rate") public ResponseEntity<String> setKeyedRate( @RequestParam String key, @RequestParam double rate) { try { samplingManager.setKeyedRate(key, rate); return ResponseEntity.ok("Set rate for " + key + " to: " + rate); } catch (Exception e) { return ResponseEntity.badRequest() .body("Failed to set keyed rate: " + e.getMessage()); } } @PostMapping("/reload") public ResponseEntity<String> reloadConfiguration() { configService.reloadConfiguration(); return ResponseEntity.ok("Configuration reloaded"); } }

8. Logback Configuration

Configure Logback with sampling appender.

logback-spring.xml:

<?xml version="1.0" encoding="UTF-8"?> <configuration> <!-- Spring Profile-based configuration --> <springProfile name="dev"> <include resource="console-appender.xml" /> </springProfile> <springProfile name="prod"> <include resource="file-appender.xml" /> </springProfile> <!-- Sampling Appender Configuration --> <appender name="SAMPLING" class="com.example.logging.SamplingAppender"> <!-- These will be injected by Spring --> </appender> <!-- Async Appender for Better Performance --> <appender name="ASYNC_SAMPLING" class="ch.qos.logback.classic.AsyncAppender"> <appender-ref ref="SAMPLING" /> <queueSize>10000</queueSize> <discardingThreshold>0</discardingThreshold> <includeCallerData>false</includeCallerData> <neverBlock>true</neverBlock> </appender> <!-- Root Logger with Sampling --> <root level="INFO"> <appender-ref ref="ASYNC_SAMPLING" /> </root> <!-- Specific logger configurations with different sampling rates --> <logger name="com.example.service.PaymentService" level="DEBUG" additivity="false"> <appender-ref ref="ASYNC_SAMPLING" /> </logger> <logger name="com.example.security" level="WARN" additivity="false"> <appender-ref ref="ASYNC_SAMPLING" /> </logger> <!-- Always log errors regardless of sampling --> <turboFilter class="ch.qos.logback.classic.turbo.DuplicateMessageFilter"> <AllowedRepetitions>0</AllowedRepetitions> </turboFilter> </configuration>

Best Practices for Production Log Sampling

1. Start Conservative: Begin with higher sampling rates and adjust based on volume
2. Monitor Effectiveness: Track sampling rates and adjust strategies based on actual usage
3. Preserve Errors: Always sample error-level logs and important business events
4. Consider Costs: Balance storage costs against debugging needs
5. Dynamic Configuration: Enable runtime configuration changes without redeployment
6. Test Thoroughly: Verify sampling behavior doesn't break log-based monitoring
7. Document Strategy: Clearly document sampling strategy for troubleshooting

Conclusion: Intelligent Log Volume Management

Log sampling in Java production environments is essential for maintaining system performance while preserving observability. By implementing sophisticated sampling strategies that adapt to runtime conditions, prioritize important events, and provide operational control, organizations can achieve the optimal balance between log volume and debugging capability.

This implementation demonstrates that effective log sampling isn't about simply reducing volume—it's about making intelligent decisions about which logs provide the most value, ensuring that when issues occur, the right information is available while maintaining system performance and controlling costs.