Process Anomaly Detection in Java: Comprehensive Behavioral Monitoring and Threat Detection

Process anomaly detection involves monitoring system processes, their behavior, and resource usage to identify suspicious activities, malware, and security threats in real-time.

Setup and Dependencies

1. Maven Dependencies

<!-- pom.xml --> <properties> <oshi.version>6.4.6</oshi.version> <opencsv.version>5.8</opencsv.version> <smile.version>3.0.1</smile.version> <spring-boot.version>3.1.0</spring-boot.version> </properties> <dependencies> <!-- System Monitoring --> <dependency> <groupId>com.github.oshi</groupId> <artifactId>oshi-core</artifactId> <version>${oshi.version}</version> </dependency> <!-- Machine Learning --> <dependency> <groupId>com.github.haifengl</groupId> <artifactId>smile-core</artifactId> <version>${smile.version}</version> </dependency> <!-- CSV Processing --> <dependency> <groupId>com.opencsv</groupId> <artifactId>opencsv</artifactId> <version>${opencsv.version}</version> </dependency> <!-- Spring Boot --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> <version>${spring-boot.version}</version> </dependency> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-actuator</artifactId> <version>${spring-boot.version}</version> </dependency> <!-- JSON Processing --> <dependency> <groupId>com.fasterxml.jackson.core</groupId> <artifactId>jackson-databind</artifactId> <version>2.15.2</version> </dependency> </dependencies>

2. Configuration Properties

# application.yml process: monitoring: enabled: true scan-interval: 5000 # 5 seconds history-size: 1000 anomaly-threshold: 0.85 max-cpu-usage: 80.0 max-memory-usage: 90.0 max-file-handles: 10000 detection: rules: - name: "high_cpu_usage" threshold: 90.0 duration: 30000 # 30 seconds - name: "suspicious_process" patterns: ["cryptominer", "malware", "backdoor"] - name: "unusual_process_tree" max-depth: 5 alert-on-orphan: true logging: level: com.example.security.process: DEBUG

Core Process Monitoring

1. Process Monitor Service

package com.example.security.process; import oshi.SystemInfo; import oshi.hardware.HardwareAbstractionLayer; import oshi.software.os.OSProcess; import oshi.software.os.OperatingSystem; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.scheduling.annotation.Scheduled; import org.springframework.stereotype.Service; import java.util.*; import java.util.concurrent.ConcurrentHashMap; import java.util.concurrent.CopyOnWriteArrayList; import java.util.stream.Collectors; @Service public class ProcessMonitorService { private static final Logger logger = LoggerFactory.getLogger(ProcessMonitorService.class); private final SystemInfo systemInfo; private final HardwareAbstractionLayer hardware; private final OperatingSystem operatingSystem; private final Map<Integer, ProcessStats> processHistory; private final List<ProcessAnomaly> detectedAnomalies; private final AnomalyDetectionEngine detectionEngine; public ProcessMonitorService(AnomalyDetectionEngine detectionEngine) { this.systemInfo = new SystemInfo(); this.hardware = systemInfo.getHardware(); this.operatingSystem = systemInfo.getOperatingSystem(); this.processHistory = new ConcurrentHashMap<>(); this.detectedAnomalies = new CopyOnWriteArrayList<>(); this.detectionEngine = detectionEngine; } @Scheduled(fixedRateString = "${process.monitoring.scan-interval:5000}") public void monitorProcesses() { try { List<OSProcess> processes = Arrays.stream(operatingSystem.getProcesses( OperatingSystem.ProcessFiltering.ALL_PROCESSES, OperatingSystem.ProcessSorting.CPU_DESC, 100 )).collect(Collectors.toList()); // Update process statistics updateProcessStats(processes); // Detect anomalies detectAnomalies(processes); // Clean up old entries cleanupOldEntries(); } catch (Exception e) { logger.error("Process monitoring failed", e); } } public ProcessSnapshot getCurrentSnapshot() { List<OSProcess> processes = Arrays.stream(operatingSystem.getProcesses( OperatingSystem.ProcessFiltering.ALL_PROCESSES, OperatingSystem.ProcessSorting.CPU_DESC, 50 )).collect(Collectors.toList()); return new ProcessSnapshot(processes, System.currentTimeMillis()); } public List<ProcessStats> getProcessHistory(int pid) { return processHistory.entrySet().stream() .filter(entry -> entry.getKey() == pid) .map(Map.Entry::getValue) .collect(Collectors.toList()); } public List<ProcessAnomaly> getDetectedAnomalies() { return new ArrayList<>(detectedAnomalies); } public SystemHealth getSystemHealth() { SystemHealth health = new SystemHealth(); health.setTimestamp(System.currentTimeMillis()); health.setCpuUsage(hardware.getProcessor().getSystemCpuLoad(1000)); health.setMemoryUsage(getMemoryUsage()); health.setProcessCount(operatingSystem.getProcessCount()); health.setThreadCount(operatingSystem.getThreadCount()); return health; } private void updateProcessStats(List<OSProcess> processes) { long currentTime = System.currentTimeMillis(); for (OSProcess process : processes) { int pid = process.getProcessID(); ProcessStats stats = processHistory.computeIfAbsent(pid, k -> new ProcessStats()); stats.setPid(pid); stats.setName(process.getName()); stats.setCommandLine(process.getCommandLine()); stats.setCpuUsage(process.getProcessCpuLoadCumulative()); stats.setMemoryUsage(process.getResidentSetSize()); stats.setThreadCount(process.getThreadCount()); stats.setOpenFiles(process.getOpenFiles()); stats.setUser(process.getUser()); stats.setStartTime(process.getStartTime()); stats.setTimestamp(currentTime); // Add to history stats.addToHistory(stats.copy()); } } private void detectAnomalies(List<OSProcess> processes) { for (OSProcess process : processes) { int pid = process.getProcessID(); ProcessStats stats = processHistory.get(pid); if (stats != null) { // Check for various types of anomalies checkResourceAnomalies(stats, process); checkBehavioralAnomalies(stats); checkSuspiciousPatterns(stats); } } } private void checkResourceAnomalies(ProcessStats stats, OSProcess process) { // CPU usage anomaly if (process.getProcessCpuLoadCumulative() > 90.0) { ProcessAnomaly anomaly = new ProcessAnomaly(); anomaly.setPid(stats.getPid()); anomaly.setProcessName(stats.getName()); anomaly.setAnomalyType(AnomalyType.HIGH_CPU_USAGE); anomaly.setSeverity(Severity.HIGH); anomaly.setDescription(String.format("Process %s (PID: %d) using %.2f%% CPU", stats.getName(), stats.getPid(), process.getProcessCpuLoadCumulative())); anomaly.setTimestamp(System.currentTimeMillis()); anomaly.setMetrics(Map.of("cpu_usage", process.getProcessCpuLoadCumulative())); detectedAnomalies.add(anomaly); logger.warn("High CPU usage detected: {}", anomaly.getDescription()); } // Memory usage anomaly long maxMemory = Runtime.getRuntime().maxMemory(); if (process.getResidentSetSize() > maxMemory * 0.8) { ProcessAnomaly anomaly = new ProcessAnomaly(); anomaly.setPid(stats.getPid()); anomaly.setProcessName(stats.getName()); anomaly.setAnomalyType(AnomalyType.HIGH_MEMORY_USAGE); anomaly.setSeverity(Severity.HIGH); anomaly.setDescription(String.format("Process %s (PID: %d) using %d bytes memory", stats.getName(), stats.getPid(), process.getResidentSetSize())); anomaly.setTimestamp(System.currentTimeMillis()); anomaly.setMetrics(Map.of("memory_usage", process.getResidentSetSize())); detectedAnomalies.add(anomaly); logger.warn("High memory usage detected: {}", anomaly.getDescription()); } // File handle anomaly if (process.getOpenFiles() > 10000) { ProcessAnomaly anomaly = new ProcessAnomaly(); anomaly.setPid(stats.getPid()); anomaly.setProcessName(stats.getName()); anomaly.setAnomalyType(AnomalyType.HIGH_FILE_HANDLES); anomaly.setSeverity(Severity.MEDIUM); anomaly.setDescription(String.format("Process %s (PID: %d) has %d open files", stats.getName(), stats.getPid(), process.getOpenFiles())); anomaly.setTimestamp(System.currentTimeMillis()); anomaly.setMetrics(Map.of("open_files", process.getOpenFiles())); detectedAnomalies.add(anomaly); logger.warn("High file handle count detected: {}", anomaly.getDescription()); } } private void checkBehavioralAnomalies(ProcessStats stats) { // Use machine learning to detect behavioral anomalies double anomalyScore = detectionEngine.calculateAnomalyScore(stats); if (anomalyScore > 0.85) { ProcessAnomaly anomaly = new ProcessAnomaly(); anomaly.setPid(stats.getPid()); anomaly.setProcessName(stats.getName()); anomaly.setAnomalyType(AnomalyType.BEHAVIORAL_ANOMALY); anomaly.setSeverity(Severity.HIGH); anomaly.setDescription(String.format("Behavioral anomaly detected in process %s (PID: %d). Score: %.2f", stats.getName(), stats.getPid(), anomalyScore)); anomaly.setTimestamp(System.currentTimeMillis()); anomaly.setMetrics(Map.of("anomaly_score", anomalyScore)); detectedAnomalies.add(anomaly); logger.warn("Behavioral anomaly detected: {}", anomaly.getDescription()); } } private void checkSuspiciousPatterns(ProcessStats stats) { // Check for suspicious process names or command lines List<String> suspiciousPatterns = Arrays.asList( "minerd", "cpuminer", "xmrig", "ccminer", "backdoor", "reverse_shell", "keylogger", "ransomware" ); for (String pattern : suspiciousPatterns) { if (stats.getName().toLowerCase().contains(pattern) || (stats.getCommandLine() != null && stats.getCommandLine().toLowerCase().contains(pattern))) { ProcessAnomaly anomaly = new ProcessAnomaly(); anomaly.setPid(stats.getPid()); anomaly.setProcessName(stats.getName()); anomaly.setAnomalyType(AnomalyType.SUSPICIOUS_PROCESS); anomaly.setSeverity(Severity.CRITICAL); anomaly.setDescription(String.format("Suspicious process detected: %s (PID: %d) matches pattern: %s", stats.getName(), stats.getPid(), pattern)); anomaly.setTimestamp(System.currentTimeMillis()); anomaly.setMetrics(Map.of("suspicious_pattern", pattern)); detectedAnomalies.add(anomaly); logger.error("Suspicious process detected: {}", anomaly.getDescription()); break; } } } private void cleanupOldEntries() { long cutoffTime = System.currentTimeMillis() - (5 * 60 * 1000); // 5 minutes processHistory.entrySet().removeIf(entry -> entry.getValue().getTimestamp() < cutoffTime ); // Keep only recent anomalies detectedAnomalies.removeIf(anomaly -> anomaly.getTimestamp() < (System.currentTimeMillis() - (30 * 60 * 1000)) // 30 minutes ); } private double getMemoryUsage() { long totalMemory = hardware.getMemory().getTotal(); long availableMemory = hardware.getMemory().getAvailable(); return ((double) (totalMemory - availableMemory) / totalMemory) * 100; } // DTO classes public static class ProcessStats { private int pid; private String name; private String commandLine; private double cpuUsage; private long memoryUsage; private int threadCount; private int openFiles; private String user; private long startTime; private long timestamp; private List<ProcessStats> history = new ArrayList<>(); public ProcessStats copy() { ProcessStats copy = new ProcessStats(); copy.pid = this.pid; copy.name = this.name; copy.commandLine = this.commandLine; copy.cpuUsage = this.cpuUsage; copy.memoryUsage = this.memoryUsage; copy.threadCount = this.threadCount; copy.openFiles = this.openFiles; copy.user = this.user; copy.startTime = this.startTime; copy.timestamp = this.timestamp; return copy; } public void addToHistory(ProcessStats stats) { if (history.size() > 100) { // Keep last 100 entries history.remove(0); } history.add(stats); } // Getters and setters public int getPid() { return pid; } public void setPid(int pid) { this.pid = pid; } public String getName() { return name; } public void setName(String name) { this.name = name; } public String getCommandLine() { return commandLine; } public void setCommandLine(String commandLine) { this.commandLine = commandLine; } public double getCpuUsage() { return cpuUsage; } public void setCpuUsage(double cpuUsage) { this.cpuUsage = cpuUsage; } public long getMemoryUsage() { return memoryUsage; } public void setMemoryUsage(long memoryUsage) { this.memoryUsage = memoryUsage; } public int getThreadCount() { return threadCount; } public void setThreadCount(int threadCount) { this.threadCount = threadCount; } public int getOpenFiles() { return openFiles; } public void setOpenFiles(int openFiles) { this.openFiles = openFiles; } public String getUser() { return user; } public void setUser(String user) { this.user = user; } public long getStartTime() { return startTime; } public void setStartTime(long startTime) { this.startTime = startTime; } public long getTimestamp() { return timestamp; } public void setTimestamp(long timestamp) { this.timestamp = timestamp; } public List<ProcessStats> getHistory() { return history; } public void setHistory(List<ProcessStats> history) { this.history = history; } } public static class ProcessSnapshot { private List<OSProcess> processes; private long timestamp; public ProcessSnapshot(List<OSProcess> processes, long timestamp) { this.processes = processes; this.timestamp = timestamp; } // Getters public List<OSProcess> getProcesses() { return processes; } public long getTimestamp() { return timestamp; } } public static class SystemHealth { private long timestamp; private double cpuUsage; private double memoryUsage; private int processCount; private int threadCount; // Getters and setters public long getTimestamp() { return timestamp; } public void setTimestamp(long timestamp) { this.timestamp = timestamp; } public double getCpuUsage() { return cpuUsage; } public void setCpuUsage(double cpuUsage) { this.cpuUsage = cpuUsage; } public double getMemoryUsage() { return memoryUsage; } public void setMemoryUsage(double memoryUsage) { this.memoryUsage = memoryUsage; } public int getProcessCount() { return processCount; } public void setProcessCount(int processCount) { this.processCount = processCount; } public int getThreadCount() { return threadCount; } public void setThreadCount(int threadCount) { this.threadCount = threadCount; } } }

2. Anomaly Detection Engine

package com.example.security.process; import smile.anomaly.*; import smile.data.DataFrame; import smile.data.type.StructType; import smile.data.vector.DoubleVector; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.stereotype.Service; import java.util.*; import java.util.concurrent.ConcurrentHashMap; @Service public class AnomalyDetectionEngine { private static final Logger logger = LoggerFactory.getLogger(AnomalyDetectionEngine.class); private final Map<Integer, ProcessBehaviorProfile> processProfiles; private LOF<double[]> lofDetector; private boolean isTrained = false; public AnomalyDetectionEngine() { this.processProfiles = new ConcurrentHashMap<>(); initializeDetector(); } public double calculateAnomalyScore(ProcessMonitorService.ProcessStats stats) { try { if (!isTrained) { return 0.0; // Not enough data yet } double[] features = extractFeatures(stats); return lofDetector.score(features); } catch (Exception e) { logger.error("Anomaly score calculation failed for PID: {}", stats.getPid(), e); return 0.0; } } public void updateModel(ProcessMonitorService.ProcessStats stats) { try { // Update process profile ProcessBehaviorProfile profile = processProfiles.computeIfAbsent( stats.getPid(), k -> new ProcessBehaviorProfile(stats.getPid(), stats.getName())); profile.update(stats); // Train model if we have enough data if (processProfiles.size() >= 10) { trainModel(); } } catch (Exception e) { logger.error("Model update failed for PID: {}", stats.getPid(), e); } } public BehavioralAnomaly detectBehavioralChange(ProcessMonitorService.ProcessStats currentStats) { ProcessBehaviorProfile profile = processProfiles.get(currentStats.getPid()); if (profile == null) { return null; } BehavioralAnomaly anomaly = new BehavioralAnomaly(); anomaly.setPid(currentStats.getPid()); anomaly.setProcessName(currentStats.getName()); anomaly.setTimestamp(System.currentTimeMillis()); // Check CPU usage pattern double cpuDeviation = profile.checkCpuDeviation(currentStats.getCpuUsage()); if (cpuDeviation > 2.0) { // 2 standard deviations anomaly.getDeviations().put("cpu_usage", cpuDeviation); } // Check memory usage pattern double memoryDeviation = profile.checkMemoryDeviation(currentStats.getMemoryUsage()); if (memoryDeviation > 2.0) { anomaly.getDeviations().put("memory_usage", memoryDeviation); } // Check thread count pattern double threadDeviation = profile.checkThreadDeviation(currentStats.getThreadCount()); if (threadDeviation > 2.0) { anomaly.getDeviations().put("thread_count", threadDeviation); } if (!anomaly.getDeviations().isEmpty()) { anomaly.setOverallScore(calculateOverallAnomalyScore(anomaly.getDeviations())); return anomaly; } return null; } private void initializeDetector() { try { // Initialize with default parameters this.lofDetector = LOF.fit(new double[0][0], 20); // k=20 neighbors } catch (Exception e) { logger.error("LOF detector initialization failed", e); } } private void trainModel() { try { List<double[]> trainingData = new ArrayList<>(); for (ProcessBehaviorProfile profile : processProfiles.values()) { if (profile.getStatsHistory().size() >= 5) { double[] features = profile.getAverageFeatures(); trainingData.add(features); } } if (trainingData.size() >= 10) { double[][] data = trainingData.toArray(new double[0][0]); this.lofDetector = LOF.fit(data, 20); this.isTrained = true; logger.info("Anomaly detection model trained with {} samples", trainingData.size()); } } catch (Exception e) { logger.error("Model training failed", e); } } private double[] extractFeatures(ProcessMonitorService.ProcessStats stats) { return new double[] { stats.getCpuUsage(), Math.log(stats.getMemoryUsage() + 1), // Log transform for memory stats.getThreadCount(), stats.getOpenFiles(), normalizeTimestamp(stats.getTimestamp()) }; } private double normalizeTimestamp(long timestamp) { // Normalize timestamp to 0-1 range based on current time long currentTime = System.currentTimeMillis(); long dayInMillis = 24 * 60 * 60 * 1000L; return (double) (timestamp % dayInMillis) / dayInMillis; } private double calculateOverallAnomalyScore(Map<String, Double> deviations) { return deviations.values().stream() .mapToDouble(Double::doubleValue) .average() .orElse(0.0); } public static class ProcessBehaviorProfile { private final int pid; private final String processName; private final List<ProcessMonitorService.ProcessStats> statsHistory; private final double[] cpuStats = new double[2]; // mean, stdDev private final double[] memoryStats = new double[2]; private final double[] threadStats = new double[2]; public ProcessBehaviorProfile(int pid, String processName) { this.pid = pid; this.processName = processName; this.statsHistory = new ArrayList<>(); } public void update(ProcessMonitorService.ProcessStats stats) { statsHistory.add(stats); // Keep only recent history if (statsHistory.size() > 100) { statsHistory.remove(0); } // Update statistics updateStatistics(); } public double checkCpuDeviation(double currentCpu) { if (statsHistory.size() < 5) return 0.0; return Math.abs(currentCpu - cpuStats[0]) / (cpuStats[1] + 0.001); // Avoid division by zero } public double checkMemoryDeviation(double currentMemory) { if (statsHistory.size() < 5) return 0.0; return Math.abs(Math.log(currentMemory + 1) - memoryStats[0]) / (memoryStats[1] + 0.001); } public double checkThreadDeviation(int currentThreads) { if (statsHistory.size() < 5) return 0.0; return Math.abs(currentThreads - threadStats[0]) / (threadStats[1] + 0.001); } public double[] getAverageFeatures() { if (statsHistory.isEmpty()) { return new double[5]; } double avgCpu = statsHistory.stream() .mapToDouble(ProcessMonitorService.ProcessStats::getCpuUsage) .average().orElse(0.0); double avgMemory = statsHistory.stream() .mapToDouble(stats -> Math.log(stats.getMemoryUsage() + 1)) .average().orElse(0.0); double avgThreads = statsHistory.stream() .mapToInt(ProcessMonitorService.ProcessStats::getThreadCount) .average().orElse(0.0); double avgFiles = statsHistory.stream() .mapToInt(ProcessMonitorService.ProcessStats::getOpenFiles) .average().orElse(0.0); return new double[] {avgCpu, avgMemory, avgThreads, avgFiles, 0.5}; // Default timestamp } private void updateStatistics() { if (statsHistory.size() >= 5) { // CPU statistics double[] cpuValues = statsHistory.stream() .mapToDouble(ProcessMonitorService.ProcessStats::getCpuUsage) .toArray(); cpuStats[0] = calculateMean(cpuValues); cpuStats[1] = calculateStdDev(cpuValues, cpuStats[0]); // Memory statistics (log transformed) double[] memoryValues = statsHistory.stream() .mapToDouble(stats -> Math.log(stats.getMemoryUsage() + 1)) .toArray(); memoryStats[0] = calculateMean(memoryValues); memoryStats[1] = calculateStdDev(memoryValues, memoryStats[0]); // Thread statistics double[] threadValues = statsHistory.stream() .mapToInt(ProcessMonitorService.ProcessStats::getThreadCount) .asDoubleStream().toArray(); threadStats[0] = calculateMean(threadValues); threadStats[1] = calculateStdDev(threadValues, threadStats[0]); } } private double calculateMean(double[] values) { return Arrays.stream(values).average().orElse(0.0); } private double calculateStdDev(double[] values, double mean) { double variance = Arrays.stream(values) .map(v -> Math.pow(v - mean, 2)) .average().orElse(0.0); return Math.sqrt(variance); } // Getters public int getPid() { return pid; } public String getProcessName() { return processName; } public List<ProcessMonitorService.ProcessStats> getStatsHistory() { return statsHistory; } } public static class BehavioralAnomaly { private int pid; private String processName; private long timestamp; private double overallScore; private Map<String, Double> deviations = new HashMap<>(); // Getters and setters public int getPid() { return pid; } public void setPid(int pid) { this.pid = pid; } public String getProcessName() { return processName; } public void setProcessName(String processName) { this.processName = processName; } public long getTimestamp() { return timestamp; } public void setTimestamp(long timestamp) { this.timestamp = timestamp; } public double getOverallScore() { return overallScore; } public void setOverallScore(double overallScore) { this.overallScore = overallScore; } public Map<String, Double> getDeviations() { return deviations; } public void setDeviations(Map<String, Double> deviations) { this.deviations = deviations; } } }

3. Process Anomaly Types and Models

package com.example.security.process; import java.util.Map; public class ProcessAnomaly { public enum AnomalyType { HIGH_CPU_USAGE, HIGH_MEMORY_USAGE, HIGH_FILE_HANDLES, SUSPICIOUS_PROCESS, BEHAVIORAL_ANOMALY, UNUSUAL_PROCESS_TREE, PROCESS_INJECTION, PRIVILEGE_ESCALATION } public enum Severity { LOW, MEDIUM, HIGH, CRITICAL } private int pid; private String processName; private AnomalyType anomalyType; private Severity severity; private String description; private long timestamp; private Map<String, Object> metrics; private String recommendation; // Getters and setters public int getPid() { return pid; } public void setPid(int pid) { this.pid = pid; } public String getProcessName() { return processName; } public void setProcessName(String processName) { this.processName = processName; } public AnomalyType getAnomalyType() { return anomalyType; } public void setAnomalyType(AnomalyType anomalyType) { this.anomalyType = anomalyType; } public Severity getSeverity() { return severity; } public void setSeverity(Severity severity) { this.severity = severity; } public String getDescription() { return description; } public void setDescription(String description) { this.description = description; } public long getTimestamp() { return timestamp; } public void setTimestamp(long timestamp) { this.timestamp = timestamp; } public Map<String, Object> getMetrics() { return metrics; } public void setMetrics(Map<String, Object> metrics) { this.metrics = metrics; } public String getRecommendation() { return recommendation; } public void setRecommendation(String recommendation) { this.recommendation = recommendation; } }

4. Alerting and Notification Service

package com.example.security.process; import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.springframework.beans.factory.annotation.Value; import org.springframework.mail.SimpleMailMessage; import org.springframework.mail.javamail.JavaMailSender; import org.springframework.stereotype.Service; import org.springframework.web.client.RestTemplate; import java.util.HashMap; import java.util.Map; @Service public class ProcessAlertService { private static final Logger logger = LoggerFactory.getLogger(ProcessAlertService.class); private final JavaMailSender mailSender; private final RestTemplate restTemplate; @Value("${security.alerts.email.recipient:[email protected]}") private String alertEmailRecipient; @Value("${security.alerts.slack.webhook:}") private String slackWebhookUrl; public ProcessAlertService(JavaMailSender mailSender, RestTemplate restTemplate) { this.mailSender = mailSender; this.restTemplate = restTemplate; } public void sendAlert(ProcessAnomaly anomaly) { switch (anomaly.getSeverity()) { case CRITICAL: sendCriticalAlert(anomaly); break; case HIGH: sendHighAlert(anomaly); break; case MEDIUM: sendMediumAlert(anomaly); break; case LOW: sendLowAlert(anomaly); break; } logAnomaly(anomaly); } private void sendCriticalAlert(ProcessAnomaly anomaly) { String subject = "🚨 CRITICAL Process Anomaly Detected"; String message = buildAlertMessage(anomaly); sendEmailAlert(subject, message); sendSlackAlert(subject, message); sendPagerDutyAlert(anomaly); logger.error("CRITICAL process anomaly: {}", anomaly.getDescription()); } private void sendHighAlert(ProcessAnomaly anomaly) { String subject = "⚠️ HIGH Process Anomaly Detected"; String message = buildAlertMessage(anomaly); sendEmailAlert(subject, message); sendSlackAlert(subject, message); logger.warn("HIGH process anomaly: {}", anomaly.getDescription()); } private void sendMediumAlert(ProcessAnomaly anomaly) { String subject = "🔔 Medium Process Anomaly Detected"; String message = buildAlertMessage(anomaly); sendSlackAlert(subject, message); logger.info("MEDIUM process anomaly: {}", anomaly.getDescription()); } private void sendLowAlert(ProcessAnomaly anomaly) { // Just log low severity anomalies logger.info("LOW process anomaly: {}", anomaly.getDescription()); } private void sendEmailAlert(String subject, String message) { try { SimpleMailMessage mailMessage = new SimpleMailMessage(); mailMessage.setTo(alertEmailRecipient); mailMessage.setSubject("[Process Security] " + subject); mailMessage.setText(message); mailSender.send(mailMessage); logger.info("Process alert email sent: {}", subject); } catch (Exception e) { logger.error("Failed to send process alert email", e); } } private void sendSlackAlert(String subject, String message) { if (slackWebhookUrl == null || slackWebhookUrl.isEmpty()) { return; } try { Map<String, Object> slackMessage = new HashMap<>(); slackMessage.put("text", "🔍 *Process Anomaly Alert*\n*" + subject + "*\n" + message); restTemplate.postForEntity(slackWebhookUrl, slackMessage, String.class); logger.info("Process alert sent to Slack: {}", subject); } catch (Exception e) { logger.error("Failed to send process alert to Slack", e); } } private void sendPagerDutyAlert(ProcessAnomaly anomaly) { // Implementation for PagerDuty integration // This would require additional PagerDuty dependencies and configuration logger.info("PagerDuty alert would be sent for critical anomaly: {}", anomaly.getDescription()); } private String buildAlertMessage(ProcessAnomaly anomaly) { StringBuilder message = new StringBuilder(); message.append("Process Anomaly Details:\n"); message.append("------------------------\n"); message.append("Process: ").append(anomaly.getProcessName()).append(" (PID: ").append(anomaly.getPid()).append(")\n"); message.append("Type: ").append(anomaly.getAnomalyType()).append("\n"); message.append("Severity: ").append(anomaly.getSeverity()).append("\n"); message.append("Description: ").append(anomaly.getDescription()).append("\n"); message.append("Timestamp: ").append(new java.util.Date(anomaly.getTimestamp())).append("\n"); if (anomaly.getMetrics() != null && !anomaly.getMetrics().isEmpty()) { message.append("Metrics:\n"); anomaly.getMetrics().forEach((key, value) -> message.append(" - ").append(key).append(": ").append(value).append("\n")); } if (anomaly.getRecommendation() != null) { message.append("Recommendation: ").append(anomaly.getRecommendation()).append("\n"); } return message.toString(); } private void logAnomaly(ProcessAnomaly anomaly) { // Structured logging for SIEM integration Map<String, Object> logData = new HashMap<>(); logData.put("event_type", "process_anomaly"); logData.put("pid", anomaly.getPid()); logData.put("process_name", anomaly.getProcessName()); logData.put("anomaly_type", anomaly.getAnomalyType()); logData.put("severity", anomaly.getSeverity()); logData.put("description", anomaly.getDescription()); logData.put("timestamp", anomaly.getTimestamp()); logData.put("metrics", anomaly.getMetrics()); logger.info("Process anomaly detected: {}", logData); } }

5. Security Dashboard Controller

package com.example.security.process; import org.springframework.security.access.prepost.PreAuthorize; import org.springframework.web.bind.annotation.*; import java.util.List; @RestController @RequestMapping("/api/security/process") @PreAuthorize("hasRole('SECURITY_TEAM')") public class ProcessSecurityController { private final ProcessMonitorService processMonitor; private final ProcessAlertService alertService; private final AnomalyDetectionEngine detectionEngine; public ProcessSecurityController(ProcessMonitorService processMonitor, ProcessAlertService alertService, AnomalyDetectionEngine detectionEngine) { this.processMonitor = processMonitor; this.alertService = alertService; this.detectionEngine = detectionEngine; } @GetMapping("/snapshot") public ProcessMonitorService.ProcessSnapshot getCurrentSnapshot() { return processMonitor.getCurrentSnapshot(); } @GetMapping("/health") public ProcessMonitorService.SystemHealth getSystemHealth() { return processMonitor.getSystemHealth(); } @GetMapping("/anomalies") public List<ProcessAnomaly> getDetectedAnomalies() { return processMonitor.getDetectedAnomalies(); } @GetMapping("/history/{pid}") public List<ProcessMonitorService.ProcessStats> getProcessHistory(@PathVariable int pid) { return processMonitor.getProcessHistory(pid); } @PostMapping("/scan/trigger") public String triggerManualScan() { processMonitor.monitorProcesses(); return "Manual process scan triggered"; } @GetMapping("/suspicious") public List<ProcessAnomaly> getSuspiciousProcesses() { return processMonitor.getDetectedAnomalies().stream() .filter(anomaly -> anomaly.getSeverity() == ProcessAnomaly.Severity.CRITICAL || anomaly.getSeverity() == ProcessAnomaly.Severity.HIGH) .collect(java.util.stream.Collectors.toList()); } @PostMapping("/analyze/{pid}") public AnomalyDetectionEngine.BehavioralAnomaly analyzeProcessBehavior(@PathVariable int pid) { List<ProcessMonitorService.ProcessStats> history = processMonitor.getProcessHistory(pid); if (history.isEmpty()) { return null; } ProcessMonitorService.ProcessStats latest = history.get(history.size() - 1); return detectionEngine.detectBehavioralChange(latest); } }

Advanced Detection Rules

1. Process Tree Analysis

package com.example.security.process; import oshi.software.os.OSProcess; import oshi.software.os.OperatingSystem; import java.util.*; import java.util.stream.Collectors; @Service public class ProcessTreeAnalyzer { public ProcessTree analyzeProcessTree(int rootPid, OperatingSystem os) { ProcessTree tree = new ProcessTree(); tree.setRootPid(rootPid); Map<Integer, ProcessTreeNode> nodes = new HashMap<>(); buildTree(rootPid, nodes, os, 0); tree.setNodes(new ArrayList<>(nodes.values())); analyzeTreeStructure(tree); return tree; } public List<ProcessTreeAnomaly> detectTreeAnomalies(ProcessTree tree) { List<ProcessTreeAnomaly> anomalies = new ArrayList<>(); // Check for orphan processes detectOrphanProcesses(tree, anomalies); // Check for unusual parent-child relationships detectUnusualRelationships(tree, anomalies); // Check for process injection patterns detectProcessInjection(tree, anomalies); return anomalies; } private void buildTree(int pid, Map<Integer, ProcessTreeNode> nodes, OperatingSystem os, int depth) { if (depth > 10) return; // Prevent infinite recursion OSProcess process = os.getProcess(pid); if (process == null) return; ProcessTreeNode node = new ProcessTreeNode(); node.setPid(pid); node.setName(process.getName()); node.setCommandLine(process.getCommandLine()); node.setDepth(depth); node.setParentPid(process.getParentProcessID()); nodes.put(pid, node); // Recursively build tree for children int[] childPids = os.getChildProcesses(pid, OperatingSystem.ProcessFiltering.ALL_PROCESSES, 0); for (int childPid : childPids) { if (!nodes.containsKey(childPid)) { buildTree(childPid, nodes, os, depth + 1); } } } private void detectOrphanProcesses(ProcessTree tree, List<ProcessTreeAnomaly> anomalies) { for (ProcessTreeNode node : tree.getNodes()) { if (node.getParentPid() != 0 && !tree.getNodes().stream() .anyMatch(n -> n.getPid() == node.getParentPid())) { ProcessTreeAnomaly anomaly = new ProcessTreeAnomaly(); anomaly.setType(ProcessTreeAnomaly.Type.ORPHAN_PROCESS); anomaly.setPid(node.getPid()); anomaly.setProcessName(node.getName()); anomaly.setDescription("Orphan process detected: " + node.getName() + " (PID: " + node.getPid() + ")"); anomaly.setSeverity(ProcessAnomaly.Severity.HIGH); anomalies.add(anomaly); } } } private void detectUnusualRelationships(ProcessTree tree, List<ProcessTreeAnomaly> anomalies) { // Implementation for detecting unusual parent-child relationships // e.g., browser spawning command shell, system process spawning user process } private void detectProcessInjection(ProcessTree tree, List<ProcessTreeAnomaly> anomalies) { // Implementation for detecting process injection patterns } private void analyzeTreeStructure(ProcessTree tree) { tree.setMaxDepth(tree.getNodes().stream() .mapToInt(ProcessTreeNode::getDepth) .max().orElse(0)); tree.setTotalProcesses(tree.getNodes().size()); } public static class ProcessTree { private int rootPid; private List<ProcessTreeNode> nodes; private int maxDepth; private int totalProcesses; // Getters and setters public int getRootPid() { return rootPid; } public void setRootPid(int rootPid) { this.rootPid = rootPid; } public List<ProcessTreeNode> getNodes() { return nodes; } public void setNodes(List<ProcessTreeNode> nodes) { this.nodes = nodes; } public int getMaxDepth() { return maxDepth; } public void setMaxDepth(int maxDepth) { this.maxDepth = maxDepth; } public int getTotalProcesses() { return totalProcesses; } public void setTotalProcesses(int totalProcesses) { this.totalProcesses = totalProcesses; } } public static class ProcessTreeNode { private int pid; private String name; private String commandLine; private int depth; private int parentPid; // Getters and setters public int getPid() { return pid; } public void setPid(int pid) { this.pid = pid; } public String getName() { return name; } public void setName(String name) { this.name = name; } public String getCommandLine() { return commandLine; } public void setCommandLine(String commandLine) { this.commandLine = commandLine; } public int getDepth() { return depth; } public void setDepth(int depth) { this.depth = depth; } public int getParentPid() { return parentPid; } public void setParentPid(int parentPid) { this.parentPid = parentPid; } } public static class ProcessTreeAnomaly { public enum Type { ORPHAN_PROCESS, UNUSUAL_PARENT_CHILD, PROCESS_INJECTION, PRIVILEGE_ESCALATION } private Type type; private int pid; private String processName; private String description; private ProcessAnomaly.Severity severity; // Getters and setters public Type getType() { return type; } public void setType(Type type) { this.type = type; } public int getPid() { return pid; } public void setPid(int pid) { this.pid = pid; } public String getProcessName() { return processName; } public void setProcessName(String processName) { this.processName = processName; } public String getDescription() { return description; } public void setDescription(String description) { this.description = description; } public ProcessAnomaly.Severity getSeverity() { return severity; } public void setSeverity(ProcessAnomaly.Severity severity) { this.severity = severity; } } }

Best Practices

Monitoring Strategy:

Continuous real-time monitoring
Behavioral baseline establishment
Multi-layered detection approach
Resource usage optimization

Detection Techniques:

Statistical anomaly detection
Machine learning-based detection
Signature-based detection
Behavioral pattern analysis

Alert Management:

Severity-based alerting
Alert correlation
False positive reduction
Automated response actions

Performance Considerations:

Efficient resource usage
Scalable architecture
Minimal performance impact
Optimized data collection

# Example usage for testing curl -X GET http://localhost:8080/api/security/process/snapshot curl -X GET http://localhost:8080/api/security/process/anomalies curl -X POST http://localhost:8080/api/security/process/scan/trigger

Conclusion

Process Anomaly Detection in Java provides:

Real-time process monitoring and behavioral analysis
Machine learning-based anomaly detection
Multi-layered detection strategies
Comprehensive alerting and notification system
Process tree analysis for advanced threat detection

By implementing the patterns and techniques shown above, you can establish robust process monitoring, detect suspicious activities in real-time, identify advanced threats through behavioral analysis, and maintain comprehensive visibility into your system's process ecosystem for enhanced security posture.