Grafana Agent for Metrics in Java: Comprehensive Monitoring Setup

Grafana Agent is a telemetry collector that efficiently sends metrics, logs, and trace data to compatible backends. It's designed to be compatible with Prometheus, Loki, and Tempo, making it an ideal choice for modern observability stacks.

Core Concepts

What is Grafana Agent?

A lightweight, batteries-included telemetry collector
Supports metrics (Prometheus), logs (Loki), and traces (Tempo)
Efficiently handles multi-tenancy and data shipping
Reduces resource usage compared to running multiple collectors

Key Benefits:

Unified collection for metrics, logs, and traces
Native integration with Grafana Cloud and Grafana Stack
Resource efficient with built-in filtering and relabeling
Kubernetes-native with easy discovery and scraping

Architecture Overview

Java Application → JMX/Micrometer → Grafana Agent → Prometheus/Grafana Cloud ↓ Custom Metrics → HTTP Endpoint → Grafana Agent

Dependencies and Setup

Maven Dependencies

<properties> <micrometer.version>1.11.5</micrometer.version> <spring-boot.version>3.1.0</spring-boot.version> <prometheus.version>0.16.0</prometheus.version> </properties> <dependencies> <!-- Micrometer Core --> <dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-core</artifactId> <version>${micrometer.version}</version> </dependency> <!-- Micrometer Prometheus Registry --> <dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-registry-prometheus</artifactId> <version>${micrometer.version}</version> </dependency> <!-- Spring Boot Actuator --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-actuator</artifactId> <version>${spring-boot.version}</version> </dependency> <!-- JMX Exporter for non-Spring applications --> <dependency> <groupId>io.prometheus.jmx</groupId> <artifactId>jmx_prometheus_javaagent</artifactId> <version>${prometheus.version}</version> </dependency> </dependencies>

Spring Boot Configuration

# application.yml management: endpoints: web: exposure: include: health,info,metrics,prometheus endpoint: prometheus: enabled: true health: enabled: true metrics: enabled: true metrics: export: prometheus: enabled: true distribution: percentiles-histogram: http.server.requests: true tags: application: user-service environment: ${ENVIRONMENT:dev} version: ${APP_VERSION:1.0.0} prometheus: metrics: export: enabled: true server: port: 8080 # Custom metrics configuration app: metrics: prefix: myapp

Grafana Agent Configuration

1. Basic Grafana Agent Config

# grafana-agent.yaml server: log_level: info http_listen_port: 12345 metrics: global: scrape_interval: 15s remote_write: - url: https://prometheus-us-central1.grafana.net/api/prom/push basic_auth: username: YOUR_USERNAME password: YOUR_API_KEY configs: - name: java-applications scrape_configs: # Scrape Spring Boot Actuator endpoints - job_name: 'spring-boot-apps' scrape_interval: 15s metrics_path: '/actuator/prometheus' static_configs: - targets: ['host.docker.internal:8080'] labels: application: 'user-service' environment: 'development' relabel_configs: - source_labels: [__address__] target_label: instance - source_labels: [__meta_kubernetes_pod_name] target_label: pod # Scrape JMX metrics - job_name: 'jmx-apps' static_configs: - targets: ['host.docker.internal:12346'] metrics_path: '/metrics' remote_write: - url: https://prometheus-us-central1.grafana.net/api/prom/push basic_auth: username: YOUR_USERNAME password: YOUR_API_KEY logs: configs: - name: java-logs positions: filename: /tmp/positions.yaml scrape_configs: - job_name: java-app-logs static_configs: - targets: [localhost] labels: job: user-service-logs __path__: /var/log/java-app/*.log clients: - url: https://logs-prod-us-central1.grafana.net/loki/api/v1/push basic_auth: username: YOUR_USERNAME password: YOUR_API_KEY traces: configs: - name: java-traces receivers: otlp: protocols: grpc: http: remote_write: - endpoint: tempo-us-central1.grafana.net:443 basic_auth: username: YOUR_USERNAME password: YOUR_API_KEY

2. Docker Compose Setup

# docker-compose.yml version: '3.8' services: grafana-agent: image: grafana/agent:latest container_name: grafana-agent ports: - "12345:12345" # Agent admin interface - "12346:12346" # JMX metrics endpoint volumes: - ./grafana-agent.yaml:/etc/agent-config/agent.yaml - /var/log/java-app:/var/log/java-app command: - --config.file=/etc/agent-config/agent.yaml - --metrics.wal-directory=/tmp/grafana-agent/wal restart: unless-stopped java-application: image: my-java-app:latest container_name: java-app ports: - "8080:8080" environment: - JAVA_OPTS=-javaagent:/app/jmx_prometheus_javaagent-0.16.0.jar=12346:/app/jmx-config.yaml volumes: - ./jmx-config.yaml:/app/jmx-config.yaml - ./jmx_prometheus_javaagent.jar:/app/jmx_prometheus_javaagent.jar depends_on: - grafana-agent

3. JMX Exporter Configuration

# jmx-config.yaml lowercaseOutputName: true lowercaseOutputLabelNames: true rules: - pattern: "java.lang<type=Memory><>(.*):" name: "jvm_memory_$1" - pattern: "java.lang<type=GarbageCollector, name=(.*)><>(.*):" name: "jvm_gc_$2" labels: gc: "$1" - pattern: "java.lang<type=OperatingSystem><>(.*):" name: "jvm_os_$1" - pattern: "java.lang<type=Threading><>(.*):" name: "jvm_threads_$1" - pattern: "com.myapp<name=(.*), type=(.*)><>(.*):" name: "myapp_$1_$3" labels: type: "$2" # Custom application MBeans - pattern: "org.springframework.boot<type=Tomcat, name=global><>(.*):" name: "spring_tomcat_$1" - pattern: "org.apache.tomcat<type=ThreadPool, name=\"http-nio-8080\"><>(.*):" name: "tomcat_thread_pool_$1"

Java Application Implementation

1. Spring Boot Metrics Configuration

@Configuration public class MetricsConfig { @Bean public MeterRegistryCustomizer<MeterRegistry> metricsCommonTags() { return registry -> { registry.config().commonTags( "application", "user-service", "environment", System.getenv().getOrDefault("ENVIRONMENT", "dev"), "version", System.getenv().getOrDefault("APP_VERSION", "1.0.0"), "region", System.getenv().getOrDefault("AWS_REGION", "local") ); }; } @Bean public TimedAspect timedAspect(MeterRegistry registry) { return new TimedAspect(registry); } @Bean public CountedAspect countedAspect(MeterRegistry registry) { return new CountedAspect(registry); } }

2. Custom Metrics Service

@Service public class ApplicationMetrics { private final MeterRegistry meterRegistry; // Counters private final Counter userRegistrationCounter; private final Counter loginAttemptCounter; private final Counter apiRequestCounter; // Gauges private final Gauge activeUsersGauge; private final Gauge cacheSizeGauge; // Timers private final Timer databaseQueryTimer; private final Timer externalApiCallTimer; // Distribution summaries private final DistributionSummary requestSizeSummary; // Custom metrics storage for gauges private final Map<String, Double> gaugeValues = new ConcurrentHashMap<>(); public ApplicationMetrics(MeterRegistry meterRegistry) { this.meterRegistry = meterRegistry; // Initialize counters this.userRegistrationCounter = Counter.builder("app.user.registrations") .description("Total number of user registrations") .tag("type", "success") .register(meterRegistry); this.loginAttemptCounter = Counter.builder("app.user.login.attempts") .description("Total login attempts") .baseUnit("attempts") .register(meterRegistry); this.apiRequestCounter = Counter.builder("app.api.requests") .description("API requests by endpoint and status") .register(meterRegistry); // Initialize gauges this.activeUsersGauge = Gauge.builder("app.users.active") .description("Number of currently active users") .register(meterRegistry); this.cacheSizeGauge = Gauge.builder("app.cache.size") .description("Current cache size in elements") .register(meterRegistry); // Initialize timers this.databaseQueryTimer = Timer.builder("app.database.queries") .description("Database query execution time") .publishPercentiles(0.5, 0.95, 0.99) .publishPercentileHistogram() .register(meterRegistry); this.externalApiCallTimer = Timer.builder("app.external.api.calls") .description("External API call execution time") .publishPercentiles(0.5, 0.95, 0.99) .register(meterRegistry); // Initialize distribution summary this.requestSizeSummary = DistributionSummary.builder("app.request.size") .description("Incoming request size in bytes") .baseUnit("bytes") .register(meterRegistry); } // Counter methods public void incrementUserRegistration(String type) { userRegistrationCounter.increment(); } public void incrementLoginAttempt(boolean success) { loginAttemptCounter.increment(); if (!success) { Counter.builder("app.user.login.failures") .register(meterRegistry) .increment(); } } public void incrementApiRequest(String endpoint, String method, int statusCode) { apiRequestCounter.increment(); // Also record with tags for better filtering Counter.builder("app.api.requests.detailed") .tag("endpoint", endpoint) .tag("method", method) .tag("status", String.valueOf(statusCode)) .register(meterRegistry) .increment(); } // Gauge methods public void setActiveUsers(int count) { // Gauges are updated by keeping a reference to the value gaugeValues.put("activeUsers", (double) count); activeUsersGauge.set(count); } public void setCacheSize(int size) { gaugeValues.put("cacheSize", (double) size); cacheSizeGauge.set(size); } // Timer methods public Timer.Sample startDatabaseQueryTimer() { return Timer.start(meterRegistry); } public void stopDatabaseQueryTimer(Timer.Sample sample, String queryType) { sample.stop(Timer.builder("app.database.queries.detailed") .tag("query_type", queryType) .register(meterRegistry)); } public void recordExternalApiCall(Runnable operation, String service) { externalApiCallTimer.record(operation); // Also record with service tag Timer.builder("app.external.api.calls.detailed") .tag("service", service) .register(meterRegistry) .record(() -> { try { operation.run(); } catch (Exception e) { // Record failure Counter.builder("app.external.api.failures") .tag("service", service) .register(meterRegistry) .increment(); throw e; } }); } // Distribution summary methods public void recordRequestSize(long bytes) { requestSizeSummary.record(bytes); } // Business-specific metrics public void recordOrderValue(double amount, String currency) { Counter.builder("app.orders.total_value") .tag("currency", currency) .baseUnit(currency) .register(meterRegistry) .increment(amount); DistributionSummary.builder("app.orders.amount") .tag("currency", currency) .baseUnit(currency) .register(meterRegistry) .record(amount); } }

3. Metrics Aspect for Automatic Instrumentation

@Aspect @Component public class MetricsAspect { private final ApplicationMetrics metrics; public MetricsAspect(ApplicationMetrics metrics) { this.metrics = metrics; } @Around("@annotation(monitored)") public Object monitorMethod(ProceedingJoinPoint joinPoint, Monitored monitored) throws Throwable { String methodName = joinPoint.getSignature().getName(); String className = joinPoint.getTarget().getClass().getSimpleName(); Timer.Sample sample = Timer.start(metrics.getMeterRegistry()); try { Object result = joinPoint.proceed(); sample.stop(Timer.builder("app.method.execution") .tag("class", className) .tag("method", methodName) .tag("result", "success") .register(metrics.getMeterRegistry())); return result; } catch (Exception e) { sample.stop(Timer.builder("app.method.execution") .tag("class", className) .tag("method", methodName) .tag("result", "error") .tag("exception", e.getClass().getSimpleName()) .register(metrics.getMeterRegistry())); throw e; } } } @Retention(RetentionPolicy.RUNTIME) @Target(ElementType.METHOD) public @interface Monitored { String value() default ""; }

4. REST Controller with Metrics

@RestController @RequestMapping("/api/users") public class UserController { private static final Logger logger = LoggerFactory.getLogger(UserController.class); private final UserService userService; private final ApplicationMetrics metrics; public UserController(UserService userService, ApplicationMetrics metrics) { this.userService = userService; this.metrics = metrics; } @PostMapping("/register") public ResponseEntity<User> registerUser(@RequestBody UserRegistrationRequest request) { Timer.Sample sample = metrics.startDatabaseQueryTimer(); try { User user = userService.registerUser(request); metrics.incrementUserRegistration("success"); metrics.incrementApiRequest("/api/users/register", "POST", 201); return ResponseEntity.status(HttpStatus.CREATED).body(user); } catch (Exception e) { metrics.incrementUserRegistration("failed"); metrics.incrementApiRequest("/api/users/register", "POST", 500); throw e; } finally { metrics.stopDatabaseQueryTimer(sample, "insert_user"); } } @PostMapping("/login") public ResponseEntity<LoginResponse> login(@RequestBody LoginRequest request) { metrics.incrementLoginAttempt(true); try { LoginResponse response = userService.authenticate(request); metrics.incrementApiRequest("/api/users/login", "POST", 200); return ResponseEntity.ok(response); } catch (AuthenticationException e) { metrics.incrementLoginAttempt(false); metrics.incrementApiRequest("/api/users/login", "POST", 401); return ResponseEntity.status(HttpStatus.UNAUTHORIZED).build(); } } @GetMapping("/{userId}") @Monitored public ResponseEntity<User> getUser(@PathVariable String userId) { metrics.incrementApiRequest("/api/users/{id}", "GET", 200); User user = userService.getUserById(userId); return ResponseEntity.ok(user); } @GetMapping("/active") public ResponseEntity<Integer> getActiveUsers() { int activeUsers = userService.getActiveUsersCount(); metrics.setActiveUsers(activeUsers); return ResponseEntity.ok(activeUsers); } }

5. Database Service with Metrics

@Service public class UserService { private final UserRepository userRepository; private final ApplicationMetrics metrics; private final Cache<String, User> userCache; public UserService(UserRepository userRepository, ApplicationMetrics metrics) { this.userRepository = userRepository; this.metrics = metrics; this.userCache = Caffeine.newBuilder() .maximumSize(1000) .expireAfterWrite(10, TimeUnit.MINUTES) .build(); } @Monitored public User registerUser(UserRegistrationRequest request) { // Record request size if applicable metrics.recordRequestSize(request.toString().getBytes().length); return metrics.recordExternalApiCall(() -> { User user = new User(request.getEmail(), request.getName()); return userRepository.save(user); }, "database"); } public User getUserById(String userId) { // Check cache first User user = userCache.getIfPresent(userId); if (user != null) { return user; } // Cache miss - query database Timer.Sample sample = metrics.startDatabaseQueryTimer(); try { user = userRepository.findById(userId).orElseThrow(); userCache.put(userId, user); metrics.setCacheSize(userCache.estimatedSize()); return user; } finally { metrics.stopDatabaseQueryTimer(sample, "select_user"); } } public int getActiveUsersCount() { Timer.Sample sample = metrics.startDatabaseQueryTimer(); try { return userRepository.countActiveUsers(); } finally { metrics.stopDatabaseQueryTimer(sample, "count_users"); } } }

6. Health Check with Metrics

@Component public class DatabaseHealthIndicator implements HealthIndicator { private final UserRepository userRepository; private final ApplicationMetrics metrics; public DatabaseHealthIndicator(UserRepository userRepository, ApplicationMetrics metrics) { this.userRepository = userRepository; this.metrics = metrics; } @Override public Health health() { try { // Simple query to check database connectivity userRepository.count(); metrics.incrementApiRequest("/actuator/health", "GET", 200); return Health.up() .withDetail("database", "connected") .withDetail("timestamp", Instant.now()) .build(); } catch (Exception e) { metrics.incrementApiRequest("/actuator/health", "GET", 503); return Health.down() .withDetail("database", "disconnected") .withDetail("error", e.getMessage()) .withDetail("timestamp", Instant.now()) .build(); } } }

Advanced Configuration

1. Kubernetes Deployment with Grafana Agent

# k8s-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: user-service labels: app: user-service spec: replicas: 3 selector: matchLabels: app: user-service template: metadata: labels: app: user-service annotations: prometheus.io/scrape: "true" prometheus.io/path: "/actuator/prometheus" prometheus.io/port: "8080" spec: containers: - name: user-service image: my-java-app:latest ports: - containerPort: 8080 env: - name: ENVIRONMENT value: "production" - name: APP_VERSION value: "2.1.0" resources: requests: memory: "512Mi" cpu: "250m" limits: memory: "1Gi" cpu: "500m" livenessProbe: httpGet: path: /actuator/health port: 8080 initialDelaySeconds: 30 periodSeconds: 10 readinessProbe: httpGet: path: /actuator/health/readiness port: 8080 initialDelaySeconds: 5 periodSeconds: 5 --- # Grafana Agent as DaemonSet apiVersion: apps/v1 kind: DaemonSet metadata: name: grafana-agent spec: selector: matchLabels: app: grafana-agent template: metadata: labels: app: grafana-agent spec: containers: - name: grafana-agent image: grafana/agent:latest args: - --config.file=/etc/agent-config/agent.yaml - --metrics.wal-directory=/tmp/grafana-agent/wal ports: - containerPort: 12345 name: http-metrics volumeMounts: - name: agent-config mountPath: /etc/agent-config - name: wal-storage mountPath: /tmp/grafana-agent resources: requests: memory: "64Mi" cpu: "50m" limits: memory: "128Mi" cpu: "100m" volumes: - name: agent-config configMap: name: grafana-agent-config - name: wal-storage emptyDir: {}

2. Advanced Grafana Agent Config for Kubernetes

# grafana-agent-k8s.yaml metrics: global: scrape_interval: 30s configs: - name: kubernetes scrape_configs: - job_name: 'kubernetes-pods' kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: true - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port] action: replace regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 target_label: __address__ - action: labelmap regex: __meta_kubernetes_pod_label_(.+) - source_labels: [__meta_kubernetes_namespace] action: replace target_label: kubernetes_namespace - source_labels: [__meta_kubernetes_pod_name] action: replace target_label: kubernetes_pod_name remote_write: - url: https://prometheus-us-central1.grafana.net/api/prom/push basic_auth: username: YOUR_USERNAME password: YOUR_API_KEY

Testing and Validation

1. Metrics Endpoint Test

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT) class MetricsIntegrationTest { @Autowired private TestRestTemplate restTemplate; @Test void testPrometheusEndpoint() { ResponseEntity<String> response = restTemplate.getForEntity("/actuator/prometheus", String.class); assertThat(response.getStatusCode()).isEqualTo(HttpStatus.OK); assertThat(response.getBody()).contains("jvm_memory_used", "http_server_requests_seconds"); } @Test void testCustomMetrics() { // Trigger some business logic restTemplate.postForEntity("/api/users/register", new UserRegistrationRequest("[email protected]", "Test User"), User.class); // Check if metrics are exposed ResponseEntity<String> metricsResponse = restTemplate.getForEntity("/actuator/prometheus", String.class); assertThat(metricsResponse.getBody()).contains("app_user_registrations"); } }

2. Manual Metrics Verification

# Check Spring Boot actuator endpoint curl http://localhost:8080/actuator/prometheus | grep app_ # Check JMX exporter endpoint (if enabled) curl http://localhost:12346/metrics # Check Grafana Agent status curl http://localhost:12345/agent/api/v1/metrics/targets

Best Practices

Use Meaningful Metric Names: Follow naming conventions (app_component_metric)
Add Relevant Tags/Labels: Include environment, version, instance information
Avoid High Cardinality: Don't use unique values like user IDs as labels
Set Appropriate Scrape Intervals: Balance between granularity and resource usage
Monitor Agent Health: Set up alerts for Grafana Agent failures
Use Histograms for Latency: For better percentile calculations
Clean Up Unused Metrics: Remove deprecated metrics to reduce noise

// Example of good metric naming and tagging Counter.builder("app_http_requests_total") .description("Total HTTP requests") .tag("method", "GET") .tag("status", "200") .tag("endpoint", "/api/users") .register(registry);

Conclusion

Grafana Agent provides a powerful and efficient way to collect and forward metrics from Java applications:

Unified observability with metrics, logs, and traces
Kubernetes-native discovery and scraping
Resource-efficient compared to multiple agents
Easy integration with Grafana Cloud and self-managed Grafana

By combining Spring Boot Actuator, Micrometer, and Grafana Agent, you can build a comprehensive monitoring solution that provides deep insights into your Java applications' performance and health. The setup scales from development to production and integrates seamlessly with modern cloud-native infrastructure.

Pyroscope Profiling in Java
Explains how to use Pyroscope for continuous profiling in Java applications, helping developers analyze CPU and memory usage patterns to improve performance and identify bottlenecks.
https://macronepal.com/blog/pyroscope-profiling-in-java/

OpenTelemetry Metrics in Java: Comprehensive Guide
Provides a complete guide to collecting and exporting metrics in Java using OpenTelemetry, including counters, histograms, gauges, and integration with monitoring tools. (MACRO NEPAL)
https://macronepal.com/blog/opentelemetry-metrics-in-java-comprehensive-guide/

OTLP Exporter in Java: Complete Guide for OpenTelemetry
Explains how to configure OTLP exporters in Java to send telemetry data such as traces, metrics, and logs to monitoring systems using HTTP or gRPC protocols. (MACRO NEPAL)
https://macronepal.com/blog/otlp-exporter-in-java-complete-guide-for-opentelemetry/

Thanos Integration in Java: Global View of Metrics
Explains how to integrate Thanos with Java monitoring systems to create a scalable global metrics view across multiple Prometheus instances.

https://macronepal.com/blog/thanos-integration-in-java-global-view-of-metrics

Time Series with InfluxDB in Java: Complete Guide (Version 2)
Explains how to manage time-series data using InfluxDB in Java applications, including storing, querying, and analyzing metrics data.

https://macronepal.com/blog/time-series-with-influxdb-in-java-complete-guide-2

Time Series with InfluxDB in Java: Complete Guide
Provides an overview of integrating InfluxDB with Java for time-series data handling, including monitoring applications and managing performance metrics.

https://macronepal.com/blog/time-series-with-influxdb-in-java-complete-guide

Implementing Prometheus Remote Write in Java (Version 2)
Explains how to configure Java applications to send metrics data to Prometheus-compatible systems using the remote write feature for scalable monitoring.

https://macronepal.com/blog/implementing-prometheus-remote-write-in-java-a-complete-guide-2

Implementing Prometheus Remote Write in Java: Complete Guide
Provides instructions for sending metrics from Java services to Prometheus servers, enabling centralized monitoring and real-time analytics.

https://macronepal.com/blog/implementing-prometheus-remote-write-in-java-a-complete-guide

Building a TileServer GL in Java: Vector and Raster Tile Server
Explains how to build a TileServer GL in Java for serving vector and raster map tiles, useful for geographic visualization and mapping applications.

https://macronepal.com/blog/building-a-tileserver-gl-in-java-vector-and-raster-tile-server

Indoor Mapping in Java
Explains how to create indoor mapping systems in Java, including navigation inside buildings, spatial data handling, and visualization techniques.