Introduction to OpenTelemetry Metrics
OpenTelemetry Metrics is a vendor-neutral observability framework that provides tools and APIs for collecting and exporting metric data from applications. It enables monitoring of application performance, business metrics, and system health through a standardized approach.
System Architecture Overview
OpenTelemetry Metrics Pipeline ├── Application Instrumentation │ ├ - Counter │ ├ - UpDownCounter │ ├ - Histogram │ ├ - Gauge │ └ - Observable Instruments ├── SDK Configuration │ ├ - Meter Provider │ ├ - Metric Readers │ └ - Views (Filtering & Aggregation) ├── Exporters │ ├ - Prometheus │ ├ - OTLP │ ├ - Jaeger │ └ - Console └── Backend Systems ├ - Prometheus ├ - Grafana ├ - Datadog └ - New Relic
Core Implementation
1. Maven Dependencies
<properties>
<opentelemetry.version>1.32.0</opentelemetry.version>
<opentelemetry.alpha.version>1.32.0-alpha</opentelemetry.alpha.version>
</properties>
<dependencies>
<!-- OpenTelemetry API -->
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-api</artifactId>
<version>${opentelemetry.version}</version>
</dependency>
<!-- OpenTelemetry SDK -->
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-sdk</artifactId>
<version>${opentelemetry.version}</version>
</dependency>
<!-- OpenTelemetry Metrics API & SDK -->
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-api-metrics</artifactId>
<version>${opentelemetry.alpha.version}</version>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-sdk-metrics</artifactId>
<version>${opentelemetry.version}</version>
</dependency>
<!-- Exporters -->
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-exporter-otlp</artifactId>
<version>${opentelemetry.version}</version>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-exporter-prometheus</artifactId>
<version>${opentelemetry.alpha.version}</version>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-exporter-logging</artifactId>
<version>${opentelemetry.version}</version>
</dependency>
<!-- Spring Boot Integration -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
<version>2.7.0</version>
</dependency>
<dependency>
<groupId>io.opentelemetry.instrumentation</groupId>
<artifactId>opentelemetry-spring-boot-starter</artifactId>
<version>2.0.0-alpha</version>
</dependency>
</dependencies>
2. Configuration Setup
import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.metrics.Meter;
import io.opentelemetry.api.metrics.MeterProvider;
import io.opentelemetry.exporter.otlp.metrics.OtlpGrpcMetricExporter;
import io.opentelemetry.exporter.prometheus.PrometheusHttpServer;
import io.opentelemetry.sdk.metrics.SdkMeterProvider;
import io.opentelemetry.sdk.metrics.export.PeriodicMetricReader;
import io.opentelemetry.sdk.resources.Resource;
import io.opentelemetry.semconv.ResourceAttributes;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import java.time.Duration;
@Configuration
public class OpenTelemetryConfig {
@Bean
public SdkMeterProvider meterProvider() {
Resource resource = Resource.getDefault()
.merge(Resource.builder()
.put(ResourceAttributes.SERVICE_NAME, "my-service")
.put(ResourceAttributes.SERVICE_VERSION, "1.0.0")
.put(ResourceAttributes.DEPLOYMENT_ENVIRONMENT, "production")
.build());
// OTLP Exporter (for Jaeger, Tempo, etc.)
OtlpGrpcMetricExporter otlpExporter = OtlpGrpcMetricExporter.builder()
.setEndpoint("http://localhost:4317")
.setTimeout(Duration.ofSeconds(10))
.build();
// Periodic metric reader that exports every 30 seconds
PeriodicMetricReader periodicMetricReader = PeriodicMetricReader.builder(otlpExporter)
.setInterval(Duration.ofSeconds(30))
.build();
// Prometheus Exporter (exposes /metrics endpoint)
PrometheusHttpServer prometheusExporter = PrometheusHttpServer.builder()
.setPort(9464)
.build();
return SdkMeterProvider.builder()
.setResource(resource)
.registerMetricReader(periodicMetricReader)
.registerMetricReader(prometheusExporter)
.build();
}
@Bean
public Meter meter(SdkMeterProvider meterProvider) {
return meterProvider.get("my-instrumentation-name", "1.0.0");
}
}
3. Metric Instrumentation Service
import io.opentelemetry.api.metrics.DoubleCounter;
import io.opentelemetry.api.metrics.DoubleHistogram;
import io.opentelemetry.api.metrics.LongCounter;
import io.opentelemetry.api.metrics.LongUpDownCounter;
import io.opentelemetry.api.metrics.Meter;
import io.opentelemetry.api.metrics.ObservableDoubleGauge;
import io.opentelemetry.api.common.AttributeKey;
import io.opentelemetry.api.common.Attributes;
import org.springframework.stereotype.Service;
import javax.annotation.PostConstruct;
import java.lang.management.ManagementFactory;
import java.lang.management.MemoryMXBean;
import java.util.concurrent.atomic.AtomicLong;
import java.util.function.Supplier;
@Service
public class MetricService {
private final Meter meter;
// Metric instruments
private LongCounter httpRequestsCounter;
private DoubleHistogram httpRequestDuration;
private LongUpDownCounter activeUsersCounter;
private ObservableDoubleGauge memoryUsageGauge;
private LongCounter businessEventsCounter;
private DoubleHistogram orderValueHistogram;
// Application state for metrics
private final AtomicLong activeUsers = new AtomicLong(0);
private final AtomicLong totalOrders = new AtomicLong(0);
private final AtomicDouble revenue = new AtomicDouble(0.0);
// Attribute keys
private static final AttributeKey<String> HTTP_METHOD = AttributeKey.stringKey("http.method");
private static final AttributeKey<String> HTTP_STATUS = AttributeKey.stringKey("http.status");
private static final AttributeKey<String> ENDPOINT = AttributeKey.stringKey("endpoint");
private static final AttributeKey<String> EVENT_TYPE = AttributeKey.stringKey("event.type");
private static final AttributeKey<String> CUSTOMER_TIER = AttributeKey.stringKey("customer.tier");
public MetricService(Meter meter) {
this.meter = meter;
}
@PostConstruct
public void initMetrics() {
initializeCounters();
initializeHistograms();
initializeGauges();
initializeUpDownCounters();
}
private void initializeCounters() {
// HTTP Requests Counter
httpRequestsCounter = meter.counterBuilder("http.requests.total")
.setDescription("Total number of HTTP requests")
.setUnit("1")
.build();
// Business Events Counter
businessEventsCounter = meter.counterBuilder("business.events.total")
.setDescription("Total business events")
.setUnit("1")
.build();
}
private void initializeHistograms() {
// HTTP Request Duration Histogram
httpRequestDuration = meter.histogramBuilder("http.request.duration")
.setDescription("HTTP request duration in seconds")
.setUnit("s")
.build();
// Order Value Histogram
orderValueHistogram = meter.histogramBuilder("order.value")
.setDescription("Distribution of order values")
.setUnit("USD")
.build();
}
private void initializeGauges() {
// Memory Usage Gauge
MemoryMXBean memoryMXBean = ManagementFactory.getMemoryMXBean();
meter.gaugeBuilder("jvm.memory.usage")
.setDescription("JVM memory usage")
.setUnit("bytes")
.buildWithCallback(measurement -> {
measurement.record(
memoryMXBean.getHeapMemoryUsage().getUsed(),
Attributes.of(AttributeKey.stringKey("type"), "heap")
);
measurement.record(
memoryMXBean.getNonHeapMemoryUsage().getUsed(),
Attributes.of(AttributeKey.stringKey("type"), "non_heap")
);
});
// Custom Business Gauge
meter.gaugeBuilder("app.active.orders")
.setDescription("Number of active orders")
.setUnit("1")
.buildWithCallback(measurement -> {
measurement.record(totalOrders.get());
});
// Revenue Gauge
meter.gaugeBuilder("app.total.revenue")
.setDescription("Total revenue")
.setUnit("USD")
.buildWithCallback(measurement -> {
measurement.record(revenue.get());
});
}
private void initializeUpDownCounters() {
// Active Users Counter
activeUsersCounter = meter.upDownCounterBuilder("app.active.users")
.setDescription("Number of active users")
.setUnit("1")
.build();
}
// Business methods for recording metrics
public void recordHttpRequest(String method, String endpoint, String status, long durationMs) {
Attributes attributes = Attributes.builder()
.put(HTTP_METHOD, method)
.put(ENDPOINT, endpoint)
.put(HTTP_STATUS, status)
.build();
httpRequestsCounter.add(1, attributes);
httpRequestDuration.record(durationMs / 1000.0, attributes);
}
public void recordUserLogin() {
activeUsersCounter.add(1);
activeUsers.incrementAndGet();
}
public void recordUserLogout() {
activeUsersCounter.add(-1);
activeUsers.decrementAndGet();
}
public void recordOrder(double amount, String customerTier) {
totalOrders.incrementAndGet();
revenue.addAndGet(amount);
Attributes attributes = Attributes.builder()
.put(CUSTOMER_TIER, customerTier)
.build();
businessEventsCounter.add(1, attributes);
orderValueHistogram.record(amount, attributes);
}
public void recordBusinessEvent(String eventType) {
Attributes attributes = Attributes.builder()
.put(EVENT_TYPE, eventType)
.build();
businessEventsCounter.add(1, attributes);
}
// Getters for metric values (useful for testing)
public long getActiveUsers() {
return activeUsers.get();
}
public long getTotalOrders() {
return totalOrders.get();
}
public double getTotalRevenue() {
return revenue.get();
}
}
// Utility class for atomic double operations
class AtomicDouble {
private final AtomicLong value = new AtomicLong(0);
public void addAndGet(double delta) {
long current;
double currentDouble;
long newLong;
do {
current = value.get();
currentDouble = Double.longBitsToDouble(current);
newLong = Double.doubleToLongBits(currentDouble + delta);
} while (!value.compareAndSet(current, newLong));
}
public double get() {
return Double.longBitsToDouble(value.get());
}
}
4. Spring Boot Integration
import org.springframework.web.filter.OncePerRequestFilter;
import javax.servlet.FilterChain;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.time.Duration;
import java.time.Instant;
@Component
public class MetricsFilter extends OncePerRequestFilter {
private final MetricService metricService;
public MetricsFilter(MetricService metricService) {
this.metricService = metricService;
}
@Override
protected void doFilterInternal(HttpServletRequest request,
HttpServletResponse response,
FilterChain filterChain) throws ServletException, IOException {
Instant start = Instant.now();
try {
filterChain.doFilter(request, response);
} finally {
Instant end = Instant.now();
long duration = Duration.between(start, end).toMillis();
String method = request.getMethod();
String endpoint = request.getRequestURI();
String status = String.valueOf(response.getStatus());
metricService.recordHttpRequest(method, endpoint, status, duration);
}
}
}
5. REST Controller with Metrics
import org.springframework.web.bind.annotation.*;
import java.util.concurrent.ThreadLocalRandom;
@RestController
@RequestMapping("/api")
public class OrderController {
private final MetricService metricService;
private final OrderRepository orderRepository;
public OrderController(MetricService metricService, OrderRepository orderRepository) {
this.metricService = metricService;
this.orderRepository = orderRepository;
}
@PostMapping("/orders")
public Order createOrder(@RequestBody OrderRequest request) {
// Simulate business logic
metricService.recordBusinessEvent("order_created");
// Record order metrics
double orderAmount = request.getAmount();
String customerTier = determineCustomerTier(request.getCustomerId());
metricService.recordOrder(orderAmount, customerTier);
return orderRepository.save(convertToOrder(request));
}
@GetMapping("/users/{userId}/login")
public String userLogin(@PathVariable String userId) {
metricService.recordUserLogin();
metricService.recordBusinessEvent("user_login");
return "User logged in successfully";
}
@GetMapping("/users/{userId}/logout")
public String userLogout(@PathVariable String userId) {
metricService.recordUserLogout();
metricService.recordBusinessEvent("user_logout");
return "User logged out successfully";
}
@GetMapping("/products/{productId}")
public Product getProduct(@PathVariable String productId) {
metricService.recordBusinessEvent("product_viewed");
// Simulate some processing time
try {
Thread.sleep(ThreadLocalRandom.current().nextInt(50, 200));
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
return productRepository.findById(productId);
}
private String determineCustomerTier(String customerId) {
// Simplified tier determination
return customerId.hashCode() % 3 == 0 ? "premium" : "standard";
}
}
6. Custom Metric Annotations
import java.lang.annotation.*;
@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
public @interface WithMetrics {
String name();
String description() default "";
String unit() default "1";
}
@Aspect
@Component
public class MetricsAspect {
private final MetricService metricService;
private final Meter meter;
public MetricsAspect(MetricService metricService, Meter meter) {
this.metricService = metricService;
this.meter = meter;
}
@Around("@annotation(withMetrics)")
public Object measureMethodExecution(ProceedingJoinPoint joinPoint, WithMetrics withMetrics) throws Throwable {
String metricName = withMetrics.name();
LongCounter counter = meter.counterBuilder(metricName)
.setDescription(withMetrics.description())
.setUnit(withMetrics.unit())
.build();
Instant start = Instant.now();
try {
Object result = joinPoint.proceed();
counter.add(1, Attributes.of(AttributeKey.stringKey("status"), "success"));
return result;
} catch (Exception e) {
counter.add(1, Attributes.of(AttributeKey.stringKey("status"), "error"));
throw e;
} finally {
long duration = Duration.between(start, Instant.now()).toMillis();
// Record duration if needed
}
}
}
// Usage example
@Service
public class PaymentService {
@WithMetrics(name = "payment.processed", description = "Number of processed payments")
public PaymentResult processPayment(PaymentRequest request) {
// Payment processing logic
return paymentResult;
}
}
7. Advanced Metric Views and Filtering
import io.opentelemetry.sdk.metrics.View;
import io.opentelemetry.sdk.metrics.Aggregation;
import io.opentelemetry.sdk.metrics.InstrumentSelector;
import io.opentelemetry.sdk.metrics.ViewBuilder;
@Configuration
public class MetricViewsConfig {
@Bean
public View httpRequestDurationView() {
return View.builder()
.setName("http.request.duration.custom")
.setDescription("Custom view for HTTP request duration")
.setAggregation(Aggregation.explicitBucketHistogram(
Arrays.asList(10.0, 25.0, 50.0, 100.0, 250.0, 500.0, 1000.0, 2500.0, 5000.0)
))
.build();
}
@Bean
public View businessEventsView() {
return View.builder()
.setName("business.events.filtered")
.setDescription("Business events excluding debug events")
.setAttributeFilter(attributes -> {
String eventType = attributes.get(AttributeKey.stringKey("event.type"));
return !"debug".equals(eventType);
})
.build();
}
@Bean
public SdkMeterProvider configuredMeterProvider() {
InstrumentSelector httpSelector = InstrumentSelector.builder()
.setType(InstrumentType.HISTOGRAM)
.setName("http.request.duration")
.build();
return SdkMeterProvider.builder()
.registerView(httpSelector, httpRequestDurationView())
.registerMetricReader(PeriodicMetricReader.builder(
OtlpGrpcMetricExporter.builder().build()
).build())
.build();
}
}
8. Testing Metrics
import io.opentelemetry.sdk.metrics.testing.InMemoryMetricReader;
import io.opentelemetry.sdk.testing.exporter.InMemoryMetricExporter;
import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import static org.assertj.core.api.Assertions.assertThat;
@SpringBootTest
class MetricServiceTest {
@Autowired
private MetricService metricService;
@Autowired
private OrderController orderController;
private InMemoryMetricReader metricReader;
@BeforeEach
void setUp() {
metricReader = InMemoryMetricReader.create();
}
@Test
void testHttpRequestMetrics() {
// Simulate HTTP request
metricService.recordHttpRequest("GET", "/api/orders", "200", 150L);
// Verify metrics were recorded
// In a real test, you would collect and assert on the metrics
assertThat(metricService.getActiveUsers()).isEqualTo(0);
}
@Test
void testOrderMetrics() {
// Record an order
metricService.recordOrder(99.99, "premium");
assertThat(metricService.getTotalOrders()).isEqualTo(1);
assertThat(metricService.getTotalRevenue()).isEqualTo(99.99);
}
@Test
void testUserSessionMetrics() {
metricService.recordUserLogin();
assertThat(metricService.getActiveUsers()).isEqualTo(1);
metricService.recordUserLogout();
assertThat(metricService.getActiveUsers()).isEqualTo(0);
}
}
9. Spring Boot Actuator Integration
# application.yml management: endpoints: web: exposure: include: metrics,prometheus,health,info endpoint: metrics: enabled: true prometheus: enabled: true metrics: export: prometheus: enabled: true distribution: percentiles-histogram: http.server.requests: true percentiles: http.server.requests: 0.5, 0.75, 0.95, 0.99 tags: application: my-service environment: production
10. Docker Compose for Testing
# docker-compose.yml version: '3.8' services: prometheus: image: prom/prometheus:latest ports: - "9090:9090" volumes: - ./prometheus.yml:/etc/prometheus/prometheus.yml grafana: image: grafana/grafana:latest ports: - "3000:3000" environment: - GF_SECURITY_ADMIN_PASSWORD=admin volumes: - ./grafana/dashboards:/var/lib/grafana/dashboards jaeger: image: jaegertracing/all-in-one:latest ports: - "16686:16686" - "4317:4317"
# prometheus.yml global: scrape_interval: 15s scrape_configs: - job_name: 'my-service' static_configs: - targets: ['host.docker.internal:9464'] metrics_path: '/metrics' scrape_interval: 5s
Best Practices
1. Metric Naming Convention
// Good - clear, descriptive names
meter.counterBuilder("http.server.requests.total")
meter.histogramBuilder("http.server.request.duration.seconds")
// Avoid - ambiguous names
meter.counterBuilder("requests")
meter.histogramBuilder("duration")
2. Attribute Cardinality
// Good - low cardinality attributes Attributes.of( HTTP_METHOD, "GET", HTTP_STATUS, "200", SERVICE_NAME, "order-service" ) // Avoid - high cardinality attributes Attributes.of( USER_ID, "user-12345", // High cardinality! REQUEST_ID, "req-abcde" // Very high cardinality! )
3. Resource Management
@PreDestroy
public void cleanup() {
if (meterProvider != null) {
meterProvider.close();
}
}
4. Error Handling
public void safeRecordMetric(Runnable metricOperation) {
try {
metricOperation.run();
} catch (Exception e) {
logger.warn("Failed to record metric", e);
// Don't let metric failures break application logic
}
}
Conclusion
This comprehensive OpenTelemetry Metrics implementation provides:
- Multiple metric types (Counters, Histograms, Gauges, UpDownCounters)
- Flexible exporters (OTLP, Prometheus, Console)
- Spring Boot integration with filters and aspects
- Custom metric annotations for easy instrumentation
- Advanced features like views and filtering
- Testing support with in-memory exporters
- Production-ready configuration
Key benefits:
- Vendor-neutral instrumentation
- Rich context with attributes
- Performance efficient with minimal overhead
- Standardized across different languages and frameworks
- Extensible with custom metrics and exporters
This setup enables comprehensive monitoring of application performance, business metrics, and system health, providing valuable insights for debugging, optimization, and business intelligence.
Java Observability, Logging Intelligence & AI-Driven Monitoring (APM, Tracing, Logs & Anomaly Detection)
https://macronepal.com/blog/beyond-metrics-observing-serverless-and-traditional-java-applications-with-thundra-apm/
Explains using Thundra APM to observe both serverless and traditional Java applications by combining tracing, metrics, and logs into a unified observability platform for faster debugging and performance insights.
https://macronepal.com/blog/dynatrace-oneagent-in-java-2/
Explains Dynatrace OneAgent for Java, which automatically instruments JVM applications to capture metrics, traces, and logs, enabling full-stack monitoring and root-cause analysis with minimal configuration.
https://macronepal.com/blog/lightstep-java-sdk-distributed-tracing-and-observability-implementation/
Explains Lightstep Java SDK for distributed tracing, helping developers track requests across microservices and identify latency issues using OpenTelemetry-based observability.
https://macronepal.com/blog/honeycomb-io-beeline-for-java-complete-guide-2/
Explains Honeycomb Beeline for Java, which provides high-cardinality observability and deep query capabilities to understand complex system behavior and debug distributed systems efficiently.
https://macronepal.com/blog/lumigo-for-serverless-in-java-complete-distributed-tracing-guide-2/
Explains Lumigo for Java serverless applications, offering automatic distributed tracing, log correlation, and error tracking to simplify debugging in cloud-native environments. (Lumigo Docs)
https://macronepal.com/blog/from-noise-to-signals-implementing-log-anomaly-detection-in-java-applications/
Explains how to detect anomalies in Java logs using behavioral patterns and machine learning techniques to separate meaningful incidents from noisy log data and improve incident response.
https://macronepal.com/blog/ai-powered-log-analysis-in-java-from-reactive-debugging-to-proactive-insights/
Explains AI-driven log analysis for Java applications, shifting from manual debugging to predictive insights that identify issues early and improve system reliability using intelligent log processing.
https://macronepal.com/blog/titliel-java-logging-best-practices/
Explains best practices for Java logging, focusing on structured logs, proper log levels, performance optimization, and ensuring logs are useful for debugging and observability systems.
https://macronepal.com/blog/seeking-a-loguru-for-java-the-quest-for-elegant-and-simple-logging/
Explains the search for simpler, more elegant logging frameworks in Java, comparing modern logging approaches that aim to reduce complexity while improving readability and developer experience.