AI-Powered Anomaly Detection in Java

Anomaly detection is a critical capability in modern applications, enabling identification of unusual patterns, fraud detection, system monitoring, and quality control. Java provides robust libraries and frameworks for implementing AI-powered anomaly detection systems.

Overview of Anomaly Detection Approaches

Statistical Methods - Z-score, IQR, Gaussian distribution
Machine Learning - Isolation Forest, One-Class SVM, Autoencoders
Time Series Analysis - Seasonal decomposition, ARIMA, LSTM
Clustering-Based - K-means, DBSCAN
Deep Learning - Autoencoders, LSTM networks

Core Dependencies

<dependencies> <!-- Deep Learning --> <dependency> <groupId>org.deeplearning4j</groupId> <artifactId>deeplearning4j-core</artifactId> <version>1.0.0-M2.1</version> </dependency> <dependency> <groupId>org.nd4j</groupId> <artifactId>nd4j-native-platform</artifactId> <version>1.0.0-M2.1</version> </dependency> <!-- Machine Learning --> <dependency> <groupId>com.github.haifengl</groupId> <artifactId>smile-core</artifactId> <version>3.0.1</version> </dependency> <!-- Time Series --> <dependency> <groupId>com.github.haifengl</groupId> <artifactId>smile-ts</artifactId> <version>3.0.1</version> </dependency> <!-- Statistical Computing --> <dependency> <groupId>org.apache.commons</groupId> <artifactId>commons-math3</artifactId> <version>3.6.1</version> </dependency> <!-- Data Processing --> <dependency> <groupId>org.apache.commons</groupId> <artifactId>commons-csv</artifactId> <version>1.10.0</version> </dependency> </dependencies>

Statistical Anomaly Detection

Example 1: Z-Score and IQR Methods

import org.apache.commons.math3.stat.descriptive.DescriptiveStatistics; import java.util.*; import java.util.stream.Collectors; public class StatisticalAnomalyDetector { // Z-Score based anomaly detection public static Map<String, Object> detectZScoreAnomalies(double[] data, double threshold) { DescriptiveStatistics stats = new DescriptiveStatistics(data); double mean = stats.getMean(); double stdDev = stats.getStandardDeviation(); List<Integer> anomalies = new ArrayList<>(); List<Double> zScores = new ArrayList<>(); for (int i = 0; i < data.length; i++) { double zScore = Math.abs((data[i] - mean) / stdDev); zScores.add(zScore); if (zScore > threshold) { anomalies.add(i); } } Map<String, Object> result = new HashMap<>(); result.put("anomalies", anomalies); result.put("zScores", zScores); result.put("mean", mean); result.put("stdDev", stdDev); result.put("threshold", threshold); return result; } // Interquartile Range (IQR) method public static Map<String, Object> detectIQRAnomalies(double[] data) { Arrays.sort(data); DescriptiveStatistics stats = new DescriptiveStatistics(data); double q1 = stats.getPercentile(25); double q3 = stats.getPercentile(75); double iqr = q3 - q1; double lowerBound = q1 - 1.5 * iqr; double upperBound = q3 + 1.5 * iqr; List<Integer> anomalies = new ArrayList<>(); List<Double> values = new ArrayList<>(); for (int i = 0; i < data.length; i++) { values.add(data[i]); if (data[i] < lowerBound || data[i] > upperBound) { anomalies.add(i); } } Map<String, Object> result = new HashMap<>(); result.put("anomalies", anomalies); result.put("values", values); result.put("q1", q1); result.put("q3", q3); result.put("iqr", iqr); result.put("lowerBound", lowerBound); result.put("upperBound", upperBound); return result; } public static void main(String[] args) { // Sample data with some anomalies double[] data = {10.2, 10.5, 10.3, 10.1, 50.8, 10.4, 10.2, 9.8, 10.6, -5.2, 10.3, 10.7}; System.out.println("=== Z-Score Anomaly Detection ==="); Map<String, Object> zScoreResult = detectZScoreAnomalies(data, 2.0); System.out.println("Mean: " + zScoreResult.get("mean")); System.out.println("Std Dev: " + zScoreResult.get("stdDev")); System.out.println("Anomalies at indices: " + zScoreResult.get("anomalies")); System.out.println("\n=== IQR Anomaly Detection ==="); Map<String, Object> iqrResult = detectIQRAnomalies(data); System.out.println("Q1: " + iqrResult.get("q1")); System.out.println("Q3: " + iqrResult.get("q3")); System.out.println("IQR: " + iqrResult.get("iqr")); System.out.println("Bounds: [" + iqrResult.get("lowerBound") + ", " + iqrResult.get("upperBound") + "]"); System.out.println("Anomalies at indices: " + iqrResult.get("anomalies")); } }

Machine Learning-Based Anomaly Detection

Example 2: Isolation Forest with Smile Library

import smile.anomaly.IsolationForest; import smile.data.DataFrame; import smile.data.measure.NominalScale; import smile.data.type.DataTypes; import smile.data.type.StructField; import smile.data.type.StructType; import smile.io.Read; import java.util.Arrays; import java.util.ArrayList; import java.util.List; public class MLAnomalyDetector { public static class AnomalyResult { private final int index; private final double score; private final boolean isAnomaly; private final double[] features; public AnomalyResult(int index, double score, boolean isAnomaly, double[] features) { this.index = index; this.score = score; this.isAnomaly = isAnomaly; this.features = features; } // Getters public int getIndex() { return index; } public double getScore() { return score; } public boolean isAnomaly() { return isAnomaly; } public double[] getFeatures() { return features; } } public static List<AnomalyResult> detectWithIsolationForest(double[][] data, double contamination) { // Train Isolation Forest IsolationForest forest = IsolationForest.fit(data, 100, 256); // Calculate anomaly scores double[] scores = forest.score(data); // Determine threshold based on contamination rate double[] sortedScores = Arrays.copyOf(scores, scores.length); Arrays.sort(sortedScores); int thresholdIndex = (int) (sortedScores.length * (1 - contamination)); double threshold = sortedScores[thresholdIndex]; // Identify anomalies List<AnomalyResult> results = new ArrayList<>(); for (int i = 0; i < data.length; i++) { boolean isAnomaly = scores[i] > threshold; results.add(new AnomalyResult(i, scores[i], isAnomaly, data[i])); } return results; } // Generate sample multivariate data public static double[][] generateSampleData(int samples, int features, int anomalies) { Random random = new Random(42); double[][] data = new double[samples][features]; // Generate normal data (multivariate Gaussian) for (int i = 0; i < samples - anomalies; i++) { for (int j = 0; j < features; j++) { data[i][j] = random.nextGaussian() * 10 + 50; // Mean=50, Std=10 } } // Generate anomalies for (int i = samples - anomalies; i < samples; i++) { for (int j = 0; j < features; j++) { // Anomalies are far from normal distribution data[i][j] = random.nextGaussian() * 30 + 100; // Mean=100, Std=30 } } return data; } public static void main(String[] args) { // Generate sample data: 1000 normal points, 50 anomalies double[][] data = generateSampleData(1050, 3, 50); System.out.println("=== Isolation Forest Anomaly Detection ==="); List<AnomalyResult> results = detectWithIsolationForest(data, 0.05); long anomalyCount = results.stream().filter(AnomalyResult::isAnomaly).count(); System.out.println("Total samples: " + data.length); System.out.println("Detected anomalies: " + anomalyCount); // Print top 10 anomalies System.out.println("\nTop 10 anomalies:"); results.stream() .filter(AnomalyResult::isAnomaly) .sorted((a, b) -> Double.compare(b.getScore(), a.getScore())) .limit(10) .forEach(result -> System.out.printf( "Index: %d, Score: %.4f, Features: %s%n", result.getIndex(), result.getScore(), Arrays.toString(result.getFeatures()) )); } }

Deep Learning Anomaly Detection

Example 3: Autoencoder for Anomaly Detection

import org.deeplearning4j.nn.api.OptimizationAlgorithm; import org.deeplearning4j.nn.conf.NeuralNetConfiguration; import org.deeplearning4j.nn.conf.layers.DenseLayer; import org.deeplearning4j.nn.conf.layers.OutputLayer; import org.deeplearning4j.nn.weights.WeightInit; import org.deeplearning4j.datasets.iterator.impl.ListDataSetIterator; import org.nd4j.linalg.activations.Activation; import org.nd4j.linalg.api.ndarray.INDArray; import org.nd4j.linalg.dataset.DataSet; import org.nd4j.linalg.dataset.api.iterator.DataSetIterator; import org.nd4j.linalg.factory.Nd4j; import org.nd4j.linalg.learning.config.Adam; import org.nd4j.linalg.lossfunctions.LossFunctions; import java.util.*; public class DeepLearningAnomalyDetector { private org.deeplearning4j.nn.multilayer.MultiLayerNetwork autoencoder; private double reconstructionThreshold; public void buildAutoencoder(int inputSize, int encodingDim) { var conf = new NeuralNetConfiguration.Builder() .seed(12345) .optimizationAlgo(OptimizationAlgorithm.STOCHASTIC_GRADIENT_DESCENT) .updater(new Adam(0.001)) .list() .layer(new DenseLayer.Builder() .nIn(inputSize) .nOut(encodingDim * 2) .activation(Activation.RELU) .weightInit(WeightInit.XAVIER) .build()) .layer(new DenseLayer.Builder() .nIn(encodingDim * 2) .nOut(encodingDim) .activation(Activation.RELU) .weightInit(WeightInit.XAVIER) .build()) .layer(new DenseLayer.Builder() .nIn(encodingDim) .nOut(encodingDim * 2) .activation(Activation.RELU) .weightInit(WeightInit.XAVIER) .build()) .layer(new OutputLayer.Builder() .nIn(encodingDim * 2) .nOut(inputSize) .activation(Activation.IDENTITY) .lossFunction(LossFunctions.LossFunction.MSE) .weightInit(WeightInit.XAVIER) .build()) .build(); autoencoder = new org.deeplearning4j.nn.multilayer.MultiLayerNetwork(conf); autoencoder.init(); } public void train(double[][] trainingData, int epochs, int batchSize) { List<DataSet> dataSets = new ArrayList<>(); for (double[] sample : trainingData) { INDArray features = Nd4j.create(sample); DataSet dataSet = new DataSet(features, features); // Autoencoder: input = output dataSets.add(dataSet); } DataSetIterator iterator = new ListDataSetIterator(dataSets, batchSize); for (int i = 0; i < epochs; i++) { iterator.reset(); autoencoder.fit(iterator); if (i % 10 == 0) { double loss = autoencoder.score(); System.out.printf("Epoch %d, Loss: %.6f%n", i, loss); } } // Calculate reconstruction threshold (95th percentile of training reconstruction errors) calculateReconstructionThreshold(trainingData); } private void calculateReconstructionThreshold(double[][] trainingData) { List<Double> errors = new ArrayList<>(); for (double[] sample : trainingData) { double error = calculateReconstructionError(sample); errors.add(error); } Collections.sort(errors); int thresholdIndex = (int) (errors.size() * 0.95); reconstructionThreshold = errors.get(thresholdIndex); System.out.printf("Reconstruction threshold (95th percentile): %.6f%n", reconstructionThreshold); } public double calculateReconstructionError(double[] sample) { INDArray input = Nd4j.create(sample); INDArray output = autoencoder.output(input); INDArray error = output.sub(input); return error.norm2Number().doubleValue(); } public boolean isAnomaly(double[] sample) { double error = calculateReconstructionError(sample); return error > reconstructionThreshold; } public static void main(String[] args) { DeepLearningAnomalyDetector detector = new DeepLearningAnomalyDetector(); // Generate training data (normal patterns) double[][] trainingData = new double[1000][10]; Random random = new Random(42); for (int i = 0; i < trainingData.length; i++) { for (int j = 0; j < trainingData[i].length; j++) { trainingData[i][j] = random.nextGaussian() * 2 + 10; // Normal pattern } } System.out.println("=== Training Autoencoder ==="); detector.buildAutoencoder(10, 3); detector.train(trainingData, 100, 32); // Test with normal and anomalous data System.out.println("\n=== Testing Anomaly Detection ==="); // Normal sample (similar to training) double[] normalSample = new double[10]; for (int i = 0; i < normalSample.length; i++) { normalSample[i] = random.nextGaussian() * 2 + 10; } // Anomalous sample (different distribution) double[] anomalousSample = new double[10]; for (int i = 0; i < anomalousSample.length; i++) { anomalousSample[i] = random.nextGaussian() * 10 + 50; // Very different } double normalError = detector.calculateReconstructionError(normalSample); double anomalousError = detector.calculateReconstructionError(anomalousSample); System.out.printf("Normal sample error: %.6f, Anomaly: %s%n", normalError, detector.isAnomaly(normalSample)); System.out.printf("Anomalous sample error: %.6f, Anomaly: %s%n", anomalousError, detector.isAnomaly(anomalousSample)); } }

Real-Time Streaming Anomaly Detection

Example 4: Real-Time Time Series Anomaly Detection

import java.util.*; import java.util.concurrent.ConcurrentLinkedQueue; import java.util.concurrent.Executors; import java.util.concurrent.ScheduledExecutorService; import java.util.concurrent.TimeUnit; public class StreamingAnomalyDetector { private final ConcurrentLinkedQueue<Double> dataStream; private final int windowSize; private final double zScoreThreshold; private final List<AnomalyListener> listeners; public StreamingAnomalyDetector(int windowSize, double zScoreThreshold) { this.dataStream = new ConcurrentLinkedQueue<>(); this.windowSize = windowSize; this.zScoreThreshold = zScoreThreshold; this.listeners = new ArrayList<>(); } public interface AnomalyListener { void onAnomalyDetected(double value, double zScore, long timestamp); } public void addDataPoint(double value) { dataStream.offer(value); // Maintain window size while (dataStream.size() > windowSize) { dataStream.poll(); } // Check for anomaly if we have enough data if (dataStream.size() >= windowSize) { checkForAnomalies(value); } } private void checkForAnomalies(double latestValue) { Double[] windowArray = dataStream.toArray(new Double[0]); double[] window = Arrays.stream(windowArray) .mapToDouble(Double::doubleValue) .toArray(); double mean = calculateMean(window); double stdDev = calculateStdDev(window, mean); if (stdDev > 0) { double zScore = Math.abs((latestValue - mean) / stdDev); if (zScore > zScoreThreshold) { notifyListeners(latestValue, zScore, System.currentTimeMillis()); } } } private double calculateMean(double[] data) { return Arrays.stream(data).average().orElse(0.0); } private double calculateStdDev(double[] data, double mean) { double variance = Arrays.stream(data) .map(val -> Math.pow(val - mean, 2)) .average() .orElse(0.0); return Math.sqrt(variance); } public void addListener(AnomalyListener listener) { listeners.add(listener); } private void notifyListeners(double value, double zScore, long timestamp) { for (AnomalyListener listener : listeners) { listener.onAnomalyDetected(value, zScore, timestamp); } } public static void main(String[] args) throws InterruptedException { StreamingAnomalyDetector detector = new StreamingAnomalyDetector(50, 2.5); // Add console listener detector.addListener((value, zScore, timestamp) -> { System.out.printf("[ANOMALY] Value: %.2f, Z-Score: %.2f, Time: %tT%n", value, zScore, timestamp); }); // Simulate data stream ScheduledExecutorService executor = Executors.newScheduledThreadPool(1); Random random = new Random(); executor.scheduleAtFixedRate(() -> { // Mostly normal data with occasional spikes double value = random.nextGaussian() * 10 + 100; // 5% chance of anomaly if (random.nextDouble() < 0.05) { value += random.nextGaussian() * 50 + 100; // Large spike } detector.addDataPoint(value); }, 0, 100, TimeUnit.MILLISECONDS); // Run for 30 seconds Thread.sleep(30000); executor.shutdown(); } }

Advanced: Ensemble Anomaly Detection

Example 5: Combining Multiple Detectors

import java.util.*; public class EnsembleAnomalyDetector { private final List<AnomalyDetector> detectors; private final double votingThreshold; public interface AnomalyDetector { boolean isAnomaly(double[] data); String getName(); } public EnsembleAnomalyDetector(double votingThreshold) { this.detectors = new ArrayList<>(); this.votingThreshold = votingThreshold; } public void addDetector(AnomalyDetector detector) { detectors.add(detector); } public EnsembleResult detect(double[] data) { int anomalyVotes = 0; Map<String, Boolean> detectorResults = new HashMap<>(); for (AnomalyDetector detector : detectors) { boolean isAnomaly = detector.isAnomaly(data); detectorResults.put(detector.getName(), isAnomaly); if (isAnomaly) { anomalyVotes++; } } double anomalyRatio = (double) anomalyVotes / detectors.size(); boolean finalDecision = anomalyRatio >= votingThreshold; return new EnsembleResult(data, finalDecision, anomalyRatio, detectorResults); } public static class EnsembleResult { private final double[] data; private final boolean isAnomaly; private final double confidence; private final Map<String, Boolean> detectorResults; public EnsembleResult(double[] data, boolean isAnomaly, double confidence, Map<String, Boolean> detectorResults) { this.data = data; this.isAnomaly = isAnomaly; this.confidence = confidence; this.detectorResults = detectorResults; } // Getters public double[] getData() { return data; } public boolean isAnomaly() { return isAnomaly; } public double getConfidence() { return confidence; } public Map<String, Boolean> getDetectorResults() { return detectorResults; } } public static void main(String[] args) { EnsembleAnomalyDetector ensemble = new EnsembleAnomalyDetector(0.6); // Add different detectors ensemble.addDetector(new ZScoreDetector(2.0)); ensemble.addDetector(new IQRDetector()); ensemble.addDetector(new IsolationForestDetector(0.05)); // Test data double[][] testData = { {10.1, 20.2, 30.3}, // Normal {100.5, 200.8, 300.9}, // Likely anomaly {15.2, 25.1, 35.3} // Normal }; System.out.println("=== Ensemble Anomaly Detection ==="); for (double[] data : testData) { EnsembleResult result = ensemble.detect(data); System.out.printf("Data: %s%n", Arrays.toString(data)); System.out.printf("Final Decision: %s (Confidence: %.2f)%n", result.isAnomaly() ? "ANOMALY" : "NORMAL", result.getConfidence()); System.out.printf("Detector Results: %s%n%n", result.getDetectorResults()); } } // Example detector implementations static class ZScoreDetector implements AnomalyDetector { private final double threshold; public ZScoreDetector(double threshold) { this.threshold = threshold; } @Override public boolean isAnomaly(double[] data) { // Simplified implementation double mean = Arrays.stream(data).average().orElse(0.0); double stdDev = Math.sqrt( Arrays.stream(data).map(v -> Math.pow(v - mean, 2)).average().orElse(0.0) ); for (double value : data) { double zScore = Math.abs((value - mean) / stdDev); if (zScore > threshold) return true; } return false; } @Override public String getName() { return "ZScoreDetector"; } } static class IQRDetector implements AnomalyDetector { @Override public boolean isAnomaly(double[] data) { // Simplified IQR implementation Arrays.sort(data); double q1 = data[data.length / 4]; double q3 = data[3 * data.length / 4]; double iqr = q3 - q1; double lowerBound = q1 - 1.5 * iqr; double upperBound = q3 + 1.5 * iqr; for (double value : data) { if (value < lowerBound || value > upperBound) return true; } return false; } @Override public String getName() { return "IQRDetector"; } } static class IsolationForestDetector implements AnomalyDetector { private final double contamination; public IsolationForestDetector(double contamination) { this.contamination = contamination; } @Override public boolean isAnomaly(double[] data) { // Simplified isolation forest logic // In real implementation, you'd use a trained model double randomnessScore = Arrays.stream(data).map(Math::abs).sum(); return randomnessScore > 100; // Simplified threshold } @Override public String getName() { return "IsolationForestDetector"; } } }

Key Considerations for Production

Performance Optimization:

Use incremental learning for streaming data
Implement model versioning and A/B testing
Cache frequently used models

Monitoring and Alerting:

Track false positive/negative rates
Monitor model drift and performance degradation
Implement alerting for critical anomalies

Scalability:

Use distributed computing for large datasets
Implement batch processing for historical data
Consider cloud-based AI services for complex models

Conclusion

AI-powered anomaly detection in Java offers:

Key Advantages:

Real-time Detection - Immediate identification of issues
Multiple Algorithms - Choice of statistical, ML, or deep learning approaches
Scalability - Handles large volumes of data
Customizability - Tailor to specific domain requirements

Use Cases:

Fraud detection in financial transactions
Network intrusion detection
Industrial equipment monitoring
Healthcare patient monitoring
E-commerce fraud prevention

Best Practices:

Start with simple statistical methods
Gradually incorporate ML for complex patterns
Validate models with domain experts
Implement continuous monitoring and retraining

Java's rich ecosystem of AI/ML libraries makes it a powerful platform for building sophisticated anomaly detection systems that can scale from simple statistical approaches to complex deep learning models.

Next Steps: Explore integration with cloud AI services, implement real-time dashboards for anomaly visualization, and consider hybrid approaches combining multiple detection methods for improved accuracy.