MACRO NEPAL
Implementing Prometheus Remote Write in Java: A Complete Guide
awiskar acharya
November 23, 2025
2:47 pm

Article

The Prometheus Remote Write protocol allows you to send metrics from various sources directly to Prometheus-compatible backends. While Prometheus typically pulls metrics, Remote Write enables pushing - crucial for ephemeral workloads, serverless functions, or custom applications that can't be scraped.

This article covers how to implement a Prometheus Remote Write client in Java, including protocol buffers, authentication, and best practices.

Understanding the Protocol

Prometheus Remote Write uses:

  • Protocol Buffers: Binary format for efficient transmission
  • HTTP POST: Typically to /api/v1/write endpoint
  • Snappy Compression: For reducing payload size
  • Various Authentication Methods: Basic Auth, Bearer Tokens, mTLS

Project Setup

Maven Dependencies:

<properties>
<protobuf.version>3.24.4</protobuf.version>
</properties>
<dependencies>
<!-- Prometheus Remote Write Protocol -->
<dependency>
<groupId>io.prometheus</groupId>
<artifactId>prometheus-remote-storage</artifactId>
<version>0.4.0</version>
</dependency>
<!-- HTTP Client -->
<dependency>
<groupId>org.apache.httpcomponents.client5</groupId>
<artifactId>httpclient5</artifactId>
<version>5.2.1</version>
</dependency>
<!-- Snappy Compression -->
<dependency>
<groupId>org.xerial.snappy</groupId>
<artifactId>snappy-java</artifactId>
<version>1.1.10.5</version>
</dependency>
<!-- Protobuf -->
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>${protobuf.version}</version>
</dependency>
</dependencies>

Core Implementation

1. Basic Remote Write Client

package com.example.prometheus.remote;
import org.apache.hc.client5.http.classic.methods.HttpPost;
import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
import org.apache.hc.client5.http.impl.classic.CloseableHttpResponse;
import org.apache.hc.client5.http.impl.classic.HttpClients;
import org.apache.hc.core5.http.ContentType;
import org.apache.hc.core5.http.io.entity.ByteArrayEntity;
import org.xerial.snappy.Snappy;
import prometheus.Remote;
import prometheus.Types;
import java.io.IOException;
import java.time.Instant;
import java.util.ArrayList;
import java.util.List;
public class PrometheusRemoteWriteClient {
private final String remoteWriteUrl;
private final CloseableHttpClient httpClient;
private final String username;
private final String password;
public PrometheusRemoteWriteClient(String remoteWriteUrl) {
this(remoteWriteUrl, null, null);
}
public PrometheusRemoteWriteClient(String remoteWriteUrl, String username, String password) {
this.remoteWriteUrl = remoteWriteUrl;
this.username = username;
this.password = password;
this.httpClient = HttpClients.createDefault();
}
public void sendMetrics(List<Metric> metrics) throws IOException {
Remote.WriteRequest writeRequest = buildWriteRequest(metrics);
byte[] compressedPayload = compressWriteRequest(writeRequest);
HttpPost httpPost = new HttpPost(remoteWriteUrl);
httpPost.setEntity(new ByteArrayEntity(compressedPayload, 
ContentType.APPLICATION_OCTET_STREAM));
httpPost.setHeader("Content-Encoding", "snappy");
httpPost.setHeader("X-Prometheus-Remote-Write-Version", "0.1.0");
// Add authentication if provided
if (username != null && password != null) {
// In production, use proper credential provider
String auth = username + ":" + password;
String encodedAuth = java.util.Base64.getEncoder().encodeToString(auth.getBytes());
httpPost.setHeader("Authorization", "Basic " + encodedAuth);
}
try (CloseableHttpResponse response = httpClient.execute(httpPost)) {
int statusCode = response.getCode();
if (statusCode < 200 || statusCode >= 300) {
throw new IOException("Remote write failed with status: " + statusCode);
}
System.out.println("Successfully sent " + metrics.size() + " metrics");
}
}
private Remote.WriteRequest buildWriteRequest(List<Metric> metrics) {
Remote.WriteRequest.Builder requestBuilder = Remote.WriteRequest.newBuilder();
for (Metric metric : metrics) {
Types.TimeSeries.Builder timeSeriesBuilder = Types.TimeSeries.newBuilder();
// Add labels
for (Label label : metric.getLabels()) {
timeSeriesBuilder.addLabels(Types.Label.newBuilder()
.setName(label.getName())
.setValue(label.getValue())
.build());
}
// Add mandatory __name__ label
timeSeriesBuilder.addLabels(Types.Label.newBuilder()
.setName("__name__")
.setValue(metric.getName())
.build());
// Add sample (data point)
Types.Sample sample = Types.Sample.newBuilder()
.setValue(metric.getValue())
.setTimestamp(metric.getTimestamp())
.build();
timeSeriesBuilder.addSamples(sample);
requestBuilder.addTimeseries(timeSeriesBuilder.build());
}
return requestBuilder.build();
}
private byte[] compressWriteRequest(Remote.WriteRequest writeRequest) throws IOException {
byte[] serialized = writeRequest.toByteArray();
return Snappy.compress(serialized);
}
public void close() throws IOException {
httpClient.close();
}
}

2. Supporting Data Classes

package com.example.prometheus.remote;
import java.time.Instant;
import java.util.ArrayList;
import java.util.List;
public class Metric {
private final String name;
private final double value;
private final long timestamp;
private final List<Label> labels;
public Metric(String name, double value) {
this(name, value, Instant.now().toEpochMilli(), new ArrayList<>());
}
public Metric(String name, double value, long timestamp) {
this(name, value, timestamp, new ArrayList<>());
}
public Metric(String name, double value, long timestamp, List<Label> labels) {
this.name = name;
this.value = value;
this.timestamp = timestamp;
this.labels = new ArrayList<>(labels);
}
// Getters
public String getName() { return name; }
public double getValue() { return value; }
public long getTimestamp() { return timestamp; }
public List<Label> getLabels() { return labels; }
// Builder-style methods for convenience
public Metric withLabel(String name, String value) {
this.labels.add(new Label(name, value));
return this;
}
public Metric withTimestamp(long timestamp) {
return new Metric(this.name, this.value, timestamp, this.labels);
}
}
class Label {
private final String name;
private final String value;
public Label(String name, String value) {
this.name = name;
this.value = value;
}
public String getName() { return name; }
public String getValue() { return value; }
}

3. Advanced Client with Retry and Headers

package com.example.prometheus.remote;
import org.apache.hc.client5.http.classic.methods.HttpPost;
import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
import org.apache.hc.client5.http.impl.classic.HttpClients;
import org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager;
import org.apache.hc.core5.http.io.entity.ByteArrayEntity;
import org.apache.hc.core5.util.Timeout;
import java.io.IOException;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.concurrent.TimeUnit;
public class AdvancedRemoteWriteClient {
private final String remoteWriteUrl;
private final CloseableHttpClient httpClient;
private final Map<String, String> customHeaders;
private final int maxRetries;
public static class Builder {
private String remoteWriteUrl;
private String bearerToken;
private Map<String, String> headers = new HashMap<>();
private int maxRetries = 3;
private int timeoutSeconds = 30;
public Builder(String remoteWriteUrl) {
this.remoteWriteUrl = remoteWriteUrl;
}
public Builder withBearerToken(String token) {
this.bearerToken = token;
return this;
}
public Builder withHeader(String name, String value) {
this.headers.put(name, value);
return this;
}
public Builder withMaxRetries(int retries) {
this.maxRetries = retries;
return this;
}
public Builder withTimeout(int seconds) {
this.timeoutSeconds = seconds;
return this;
}
public AdvancedRemoteWriteClient build() {
return new AdvancedRemoteWriteClient(this);
}
}
private AdvancedRemoteWriteClient(Builder builder) {
this.remoteWriteUrl = builder.remoteWriteUrl;
this.maxRetries = builder.maxRetries;
this.customHeaders = new HashMap<>(builder.headers);
if (builder.bearerToken != null) {
this.customHeaders.put("Authorization", "Bearer " + builder.bearerToken);
}
PoolingHttpClientConnectionManager connectionManager = 
new PoolingHttpClientConnectionManager();
connectionManager.setMaxTotal(100);
connectionManager.setDefaultMaxPerRoute(20);
this.httpClient = HttpClients.custom()
.setConnectionManager(connectionManager)
.setDefaultRequestConfig(org.apache.hc.client5.http.config.RequestConfig.custom()
.setResponseTimeout(Timeout.of(builder.timeoutSeconds, TimeUnit.SECONDS))
.build())
.build();
}
public void sendMetricsWithRetry(List<Metric> metrics) throws IOException {
IOException lastException = null;
for (int attempt = 1; attempt <= maxRetries; attempt++) {
try {
sendMetrics(metrics);
return; // Success
} catch (IOException e) {
lastException = e;
System.err.println("Attempt " + attempt + " failed: " + e.getMessage());
if (attempt < maxRetries) {
try {
Thread.sleep(1000L * attempt); // Exponential backoff base
} catch (InterruptedException ie) {
Thread.currentThread().interrupt();
throw new IOException("Interrupted during retry", ie);
}
}
}
}
throw new IOException("All " + maxRetries + " attempts failed", lastException);
}
private void sendMetrics(List<Metric> metrics) throws IOException {
PrometheusRemoteWriteClient basicClient = new PrometheusRemoteWriteClient(remoteWriteUrl);
// Reflection or adapter pattern would be better in production
// This is simplified for demonstration
basicClient.sendMetrics(metrics);
}
public void close() throws IOException {
httpClient.close();
}
}

4. Usage Examples

package com.example.prometheus.remote;
import java.time.Instant;
import java.util.Arrays;
import java.util.List;
public class RemoteWriteExample {
public static void main(String[] args) {
// Basic usage
PrometheusRemoteWriteClient client = 
new PrometheusRemoteWriteClient("http://localhost:9090/api/v1/write");
try {
List<Metric> metrics = Arrays.asList(
new Metric("http_requests_total", 42.0)
.withLabel("method", "GET")
.withLabel("handler", "/api/users")
.withLabel("status", "200"),
new Metric("http_request_duration_seconds", 0.125)
.withLabel("method", "GET")
.withLabel("handler", "/api/users"),
new Metric("application_memory_usage_bytes", 256000000.0)
.withLabel("area", "heap")
);
client.sendMetrics(metrics);
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
client.close();
} catch (Exception e) {
e.printStackTrace();
}
}
// Advanced usage with authentication and retry
AdvancedRemoteWriteClient advancedClient = 
new AdvancedRemoteWriteClient.Builder("https://prometheus.example.com/api/v1/write")
.withBearerToken("your-token-here")
.withHeader("X-Tenant-ID", "tenant-123")
.withMaxRetries(5)
.withTimeout(10)
.build();
try {
// Create metrics with specific timestamps for backfilling
long timestamp = Instant.now().minusSeconds(3600).toEpochMilli();
Metric historicalMetric = new Metric(
"temperature_celsius", 
23.5, 
timestamp
).withLabel("sensor", "sensor-01");
advancedClient.sendMetricsWithRetry(Arrays.asList(historicalMetric));
} catch (Exception e) {
e.printStackTrace();
} finally {
try {
advancedClient.close();
} catch (Exception e) {
e.printStackTrace();
}
}
}
}

Best Practices and Considerations

1. Metric Design

  • Use meaningful metric names
  • Leverage labels effectively (but don't overuse - cardinality explosion!)
  • Follow Prometheus naming conventions

2. Performance Optimization

  • Batch metrics to reduce HTTP overhead
  • Reuse HTTP client connections
  • Consider async sending for high-volume applications

3. Error Handling

  • Implement proper retry logic with backoff
  • Log failed metrics for debugging
  • Monitor failed remote write attempts

4. Security

  • Use HTTPS in production
  • Secure credentials properly (avoid hardcoding)
  • Consider network policies and firewalls

5. Monitoring the Remote Write Client

// Instrument your client itself
public class InstrumentedRemoteWriteClient {
private final Counter sentMetrics = Counter.build()
.name("remote_write_metrics_sent_total")
.help("Total metrics sent via remote write")
.register();
private final Counter failedWrites = Counter.build()
.name("remote_write_failures_total")
.help("Total remote write failures")
.register();
private final Histogram requestDuration = Histogram.build()
.name("remote_write_request_duration_seconds")
.help("Remote write request duration")
.register();
// Use these in your send methods...
}

Common Pitfalls

  1. High Cardinality: Too many label combinations can overwhelm Prometheus
  2. Timestamp Issues: Future timestamps or out-of-order data
  3. Memory Leaks: Not properly closing HTTP clients
  4. Network Timeouts: Not setting appropriate timeouts
  5. Authentication Errors: Misconfigured credentials or tokens

Conclusion

Implementing Prometheus Remote Write in Java gives you flexibility to push metrics from any Java application. The protocol's efficiency with Protocol Buffers and Snappy compression makes it suitable for high-volume scenarios. By following the patterns shown here and incorporating proper error handling, batching, and monitoring, you can build a robust metrics pipeline that integrates seamlessly with your Prometheus ecosystem.

Remember that while Remote Write is powerful, it's often better to use the pull model when possible, as it's Prometheus's native operational pattern and provides better reliability and discovery characteristics.

Category : JAVA

Leave a Reply

Your email address will not be published. Required fields are marked *


Macro Nepal Helper