Implementing Prometheus Remote Write in Java: A Complete Guide

Table of Contents

Article

The Prometheus Remote Write protocol allows you to send metrics from various sources directly to Prometheus-compatible backends. While Prometheus typically pulls metrics, Remote Write enables pushing - crucial for ephemeral workloads, serverless functions, or custom applications that can't be scraped.

This article covers how to implement a Prometheus Remote Write client in Java, including protocol buffers, authentication, and best practices.

Understanding the Protocol

Prometheus Remote Write uses:

Protocol Buffers: Binary format for efficient transmission
HTTP POST: Typically to /api/v1/write endpoint
Snappy Compression: For reducing payload size
Various Authentication Methods: Basic Auth, Bearer Tokens, mTLS

Project Setup

Maven Dependencies:

<properties>
<protobuf.version>3.24.4</protobuf.version>
</properties>
<dependencies>
<!-- Prometheus Remote Write Protocol -->
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>prometheus-remote-storage</artifactId>
<version>0.4.0</version>
</dependency>
<!-- HTTP Client -->
<dependency>
<groupId>org.apache.httpcomponents.client5</groupId>
<artifactId>httpclient5</artifactId>
<version>5.2.1</version>
</dependency>
<!-- Snappy Compression -->
<dependency>
<groupId>org.xerial.snappy</groupId>
<artifactId>snappy-java</artifactId>
<version>1.1.10.5</version>
</dependency>
<!-- Protobuf -->
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>${protobuf.version}</version>
</dependency>
</dependencies>

Core Implementation

1. Basic Remote Write Client

package com.example.prometheus.remote;
import org.apache.hc.client5.http.classic.methods.HttpPost;
import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
import org.apache.hc.client5.http.impl.classic.CloseableHttpResponse;
import org.apache.hc.client5.http.impl.classic.HttpClients;
import org.apache.hc.core5.http.ContentType;
import org.apache.hc.core5.http.io.entity.ByteArrayEntity;
import org.xerial.snappy.Snappy;
import prometheus.Remote;
import prometheus.Types;
import java.io.IOException;
import java.time.Instant;
import java.util.ArrayList;
import java.util.List;
public class PrometheusRemoteWriteClient {
private final String remoteWriteUrl;
private final CloseableHttpClient httpClient;
private final String username;
private final String password;
public PrometheusRemoteWriteClient(String remoteWriteUrl) {
this(remoteWriteUrl, null, null);
}
public PrometheusRemoteWriteClient(String remoteWriteUrl, String username, String password) {
this.remoteWriteUrl = remoteWriteUrl;
this.username = username;
this.password = password;
this.httpClient = HttpClients.createDefault();
}
public void sendMetrics(List<Metric> metrics) throws IOException {
Remote.WriteRequest writeRequest = buildWriteRequest(metrics);
byte[] compressedPayload = compressWriteRequest(writeRequest);
HttpPost httpPost = new HttpPost(remoteWriteUrl);
httpPost.setEntity(new ByteArrayEntity(compressedPayload, 
ContentType.APPLICATION_OCTET_STREAM));
httpPost.setHeader("Content-Encoding", "snappy");
httpPost.setHeader("X-Prometheus-Remote-Write-Version", "0.1.0");
// Add authentication if provided
if (username != null && password != null) {
// In production, use proper credential provider
String auth = username + ":" + password;
String encodedAuth = java.util.Base64.getEncoder().encodeToString(auth.getBytes());
httpPost.setHeader("Authorization", "Basic " + encodedAuth);
}
try (CloseableHttpResponse response = httpClient.execute(httpPost)) {
int statusCode = response.getCode();
if (statusCode < 200 || statusCode >= 300) {
throw new IOException("Remote write failed with status: " + statusCode);
}
System.out.println("Successfully sent " + metrics.size() + " metrics");
}
}
private Remote.WriteRequest buildWriteRequest(List<Metric> metrics) {
Remote.WriteRequest.Builder requestBuilder = Remote.WriteRequest.newBuilder();
for (Metric metric : metrics) {
Types.TimeSeries.Builder timeSeriesBuilder = Types.TimeSeries.newBuilder();
// Add labels
for (Label label : metric.getLabels()) {
timeSeriesBuilder.addLabels(Types.Label.newBuilder()
.setName(label.getName())
.setValue(label.getValue())
.build());
}
// Add mandatory __name__ label
timeSeriesBuilder.addLabels(Types.Label.newBuilder()
.setName("__name__")
.setValue(metric.getName())
.build());
// Add sample (data point)
Types.Sample sample = Types.Sample.newBuilder()
.setValue(metric.getValue())
.setTimestamp(metric.getTimestamp())
.build();
timeSeriesBuilder.addSamples(sample);
requestBuilder.addTimeseries(timeSeriesBuilder.build());
}
return requestBuilder.build();
}
private byte[] compressWriteRequest(Remote.WriteRequest writeRequest) throws IOException {
byte[] serialized = writeRequest.toByteArray();
return Snappy.compress(serialized);
}
public void close() throws IOException {
httpClient.close();
}
}

2. Supporting Data Classes

package com.example.prometheus.remote;
import java.time.Instant;
import java.util.ArrayList;
import java.util.List;
public class Metric {
private final String name;
private final double value;
private final long timestamp;
private final List<Label> labels;
public Metric(String name, double value) {
this(name, value, Instant.now().toEpochMilli(), new ArrayList<>());
}
public Metric(String name, double value, long timestamp) {
this(name, value, timestamp, new ArrayList<>());
}
public Metric(String name, double value, long timestamp, List<Label> labels) {
this.name = name;
this.value = value;
this.timestamp = timestamp;
this.labels = new ArrayList<>(labels);
}
// Getters
public String getName() { return name; }
public double getValue() { return value; }
public long getTimestamp() { return timestamp; }
public List<Label> getLabels() { return labels; }
// Builder-style methods for convenience
public Metric withLabel(String name, String value) {
this.labels.add(new Label(name, value));
return this;
}
public Metric withTimestamp(long timestamp) {
return new Metric(this.name, this.value, timestamp, this.labels);
}
}
class Label {
private final String name;
private final String value;
public Label(String name, String value) {
this.name = name;
this.value = value;
}
public String getName() { return name; }
public String getValue() { return value; }
}

3. Advanced Client with Retry and Headers

package com.example.prometheus.remote;
import org.apache.hc.client5.http.classic.methods.HttpPost;
import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
import org.apache.hc.client5.http.impl.classic.HttpClients;
import org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager;
import org.apache.hc.core5.http.io.entity.ByteArrayEntity;
import org.apache.hc.core5.util.Timeout;
import java.io.IOException;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.TimeUnit;
public class AdvancedRemoteWriteClient {
private final String remoteWriteUrl;
private final CloseableHttpClient httpClient;
private final Map<String, String> customHeaders;
private final int maxRetries;
public static class Builder {
private String remoteWriteUrl;
private String bearerToken;
private Map<String, String> headers = new HashMap<>();
private int maxRetries = 3;
private int timeoutSeconds = 30;
public Builder(String remoteWriteUrl) {
this.remoteWriteUrl = remoteWriteUrl;
}
public Builder withBearerToken(String token) {
this.bearerToken = token;
return this;
}
public Builder withHeader(String name, String value) {
this.headers.put(name, value);
return this;
}
public Builder withMaxRetries(int retries) {
this.maxRetries = retries;
return this;
}
public Builder withTimeout(int seconds) {
this.timeoutSeconds = seconds;
return this;
}
public AdvancedRemoteWriteClient build() {
return new AdvancedRemoteWriteClient(this);
}
}
private AdvancedRemoteWriteClient(Builder builder) {
this.remoteWriteUrl = builder.remoteWriteUrl;
this.maxRetries = builder.maxRetries;
this.customHeaders = new HashMap<>(builder.headers);
if (builder.bearerToken != null) {
this.customHeaders.put("Authorization", "Bearer " + builder.bearerToken);
}
PoolingHttpClientConnectionManager connectionManager = 
new PoolingHttpClientConnectionManager();
connectionManager.setMaxTotal(100);
connectionManager.setDefaultMaxPerRoute(20);
this.httpClient = HttpClients.custom()
.setConnectionManager(connectionManager)
.setDefaultRequestConfig(org.apache.hc.client5.http.config.RequestConfig.custom()
.setResponseTimeout(Timeout.of(builder.timeoutSeconds, TimeUnit.SECONDS))
.build())
.build();
}
public void sendMetricsWithRetry(List<Metric> metrics) throws IOException {
IOException lastException = null;
for (int attempt = 1; attempt <= maxRetries; attempt++) {
try {
sendMetrics(metrics);
return; // Success
} catch (IOException e) {
lastException = e;
System.err.println("Attempt " + attempt + " failed: " + e.getMessage());
if (attempt < maxRetries) {
try {
Thread.sleep(1000L * attempt); // Exponential backoff base
} catch (InterruptedException ie) {
Thread.currentThread().interrupt();
throw new IOException("Interrupted during retry", ie);
}
}
}
}
throw new IOException("All " + maxRetries + " attempts failed", lastException);
}
private void sendMetrics(List<Metric> metrics) throws IOException {
PrometheusRemoteWriteClient basicClient = new PrometheusRemoteWriteClient(remoteWriteUrl);
// Reflection or adapter pattern would be better in production
// This is simplified for demonstration
basicClient.sendMetrics(metrics);
}
public void close() throws IOException {
httpClient.close();
}
}

4. Usage Examples

package com.example.prometheus.remote;
import java.time.Instant;
import java.util.Arrays;
import java.util.List;
public class RemoteWriteExample {
public static void main(String[] args) {
// Basic usage
PrometheusRemoteWriteClient client = 
new PrometheusRemoteWriteClient("http://localhost:9090/api/v1/write");
try {
List<Metric> metrics = Arrays.asList(
new Metric("http_requests_total", 42.0)
.withLabel("method", "GET")
.withLabel("handler", "/api/users")
.withLabel("status", "200"),
new Metric("http_request_duration_seconds", 0.125)
.withLabel("method", "GET")
.withLabel("handler", "/api/users"),
new Metric("application_memory_usage_bytes", 256000000.0)
.withLabel("area", "heap")
);
client.sendMetrics(metrics);
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
client.close();
} catch (Exception e) {
e.printStackTrace();
}
}
// Advanced usage with authentication and retry
AdvancedRemoteWriteClient advancedClient = 
new AdvancedRemoteWriteClient.Builder("https://prometheus.example.com/api/v1/write")
.withBearerToken("your-token-here")
.withHeader("X-Tenant-ID", "tenant-123")
.withMaxRetries(5)
.withTimeout(10)
.build();
try {
// Create metrics with specific timestamps for backfilling
long timestamp = Instant.now().minusSeconds(3600).toEpochMilli();
Metric historicalMetric = new Metric(
"temperature_celsius", 
23.5, 
timestamp
).withLabel("sensor", "sensor-01");
advancedClient.sendMetricsWithRetry(Arrays.asList(historicalMetric));
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
advancedClient.close();
} catch (Exception e) {
e.printStackTrace();
}
}
}
}

Best Practices and Considerations

1. Metric Design

Use meaningful metric names
Leverage labels effectively (but don't overuse - cardinality explosion!)
Follow Prometheus naming conventions

2. Performance Optimization

Batch metrics to reduce HTTP overhead
Reuse HTTP client connections
Consider async sending for high-volume applications

3. Error Handling

Implement proper retry logic with backoff
Log failed metrics for debugging
Monitor failed remote write attempts

4. Security

Use HTTPS in production
Secure credentials properly (avoid hardcoding)
Consider network policies and firewalls

5. Monitoring the Remote Write Client

// Instrument your client itself
public class InstrumentedRemoteWriteClient {
private final Counter sentMetrics = Counter.build()
.name("remote_write_metrics_sent_total")
.help("Total metrics sent via remote write")
.register();
private final Counter failedWrites = Counter.build()
.name("remote_write_failures_total")
.help("Total remote write failures")
.register();
private final Histogram requestDuration = Histogram.build()
.name("remote_write_request_duration_seconds")
.help("Remote write request duration")
.register();
// Use these in your send methods...
}

Common Pitfalls

High Cardinality: Too many label combinations can overwhelm Prometheus
Timestamp Issues: Future timestamps or out-of-order data
Memory Leaks: Not properly closing HTTP clients
Network Timeouts: Not setting appropriate timeouts
Authentication Errors: Misconfigured credentials or tokens

Conclusion

Implementing Prometheus Remote Write in Java gives you flexibility to push metrics from any Java application. The protocol's efficiency with Protocol Buffers and Snappy compression makes it suitable for high-volume scenarios. By following the patterns shown here and incorporating proper error handling, batching, and monitoring, you can build a robust metrics pipeline that integrates seamlessly with your Prometheus ecosystem.

Remember that while Remote Write is powerful, it's often better to use the pull model when possible, as it's Prometheus's native operational pattern and provides better reliability and discovery characteristics.

Java Observability, Logging Intelligence & AI-Driven Monitoring (APM, Tracing, Logs & Anomaly Detection)

https://macronepal.com/blog/beyond-metrics-observing-serverless-and-traditional-java-applications-with-thundra-apm/
Explains using Thundra APM to observe both serverless and traditional Java applications by combining tracing, metrics, and logs into a unified observability platform for faster debugging and performance insights.

https://macronepal.com/blog/dynatrace-oneagent-in-java-2/
Explains Dynatrace OneAgent for Java, which automatically instruments JVM applications to capture metrics, traces, and logs, enabling full-stack monitoring and root-cause analysis with minimal configuration.

https://macronepal.com/blog/lightstep-java-sdk-distributed-tracing-and-observability-implementation/
Explains Lightstep Java SDK for distributed tracing, helping developers track requests across microservices and identify latency issues using OpenTelemetry-based observability.

https://macronepal.com/blog/honeycomb-io-beeline-for-java-complete-guide-2/
Explains Honeycomb Beeline for Java, which provides high-cardinality observability and deep query capabilities to understand complex system behavior and debug distributed systems efficiently.

https://macronepal.com/blog/lumigo-for-serverless-in-java-complete-distributed-tracing-guide-2/
Explains Lumigo for Java serverless applications, offering automatic distributed tracing, log correlation, and error tracking to simplify debugging in cloud-native environments. (Lumigo Docs)

https://macronepal.com/blog/from-noise-to-signals-implementing-log-anomaly-detection-in-java-applications/
Explains how to detect anomalies in Java logs using behavioral patterns and machine learning techniques to separate meaningful incidents from noisy log data and improve incident response.

https://macronepal.com/blog/ai-powered-log-analysis-in-java-from-reactive-debugging-to-proactive-insights/
Explains AI-driven log analysis for Java applications, shifting from manual debugging to predictive insights that identify issues early and improve system reliability using intelligent log processing.

https://macronepal.com/blog/titliel-java-logging-best-practices/
Explains best practices for Java logging, focusing on structured logs, proper log levels, performance optimization, and ensuring logs are useful for debugging and observability systems.

https://macronepal.com/blog/seeking-a-loguru-for-java-the-quest-for-elegant-and-simple-logging/
Explains the search for simpler, more elegant logging frameworks in Java, comparing modern logging approaches that aim to reduce complexity while improving readability and developer experience.