Grafana Agent is a telemetry collector that efficiently sends metrics, logs, and trace data to compatible backends. It's designed to be compatible with Prometheus, Loki, and Tempo, making it an ideal choice for modern observability stacks.
Core Concepts
What is Grafana Agent?
- A lightweight, batteries-included telemetry collector
- Supports metrics (Prometheus), logs (Loki), and traces (Tempo)
- Efficiently handles multi-tenancy and data shipping
- Reduces resource usage compared to running multiple collectors
Key Benefits:
- Unified collection for metrics, logs, and traces
- Native integration with Grafana Cloud and Grafana Stack
- Resource efficient with built-in filtering and relabeling
- Kubernetes-native with easy discovery and scraping
Architecture Overview
Java Application → JMX/Micrometer → Grafana Agent → Prometheus/Grafana Cloud ↓ Custom Metrics → HTTP Endpoint → Grafana Agent
Dependencies and Setup
Maven Dependencies
<properties>
<micrometer.version>1.11.5</micrometer.version>
<spring-boot.version>3.1.0</spring-boot.version>
<prometheus.version>0.16.0</prometheus.version>
</properties>
<dependencies>
<!-- Micrometer Core -->
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-core</artifactId>
<version>${micrometer.version}</version>
</dependency>
<!-- Micrometer Prometheus Registry -->
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
<version>${micrometer.version}</version>
</dependency>
<!-- Spring Boot Actuator -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
<version>${spring-boot.version}</version>
</dependency>
<!-- JMX Exporter for non-Spring applications -->
<dependency>
<groupId>io.prometheus.jmx</groupId>
<artifactId>jmx_prometheus_javaagent</artifactId>
<version>${prometheus.version}</version>
</dependency>
</dependencies>
Spring Boot Configuration
# application.yml
management:
endpoints:
web:
exposure:
include: health,info,metrics,prometheus
endpoint:
prometheus:
enabled: true
health:
enabled: true
metrics:
enabled: true
metrics:
export:
prometheus:
enabled: true
distribution:
percentiles-histogram:
http.server.requests: true
tags:
application: user-service
environment: ${ENVIRONMENT:dev}
version: ${APP_VERSION:1.0.0}
prometheus:
metrics:
export:
enabled: true
server:
port: 8080
# Custom metrics configuration
app:
metrics:
prefix: myapp
Grafana Agent Configuration
1. Basic Grafana Agent Config
# grafana-agent.yaml server: log_level: info http_listen_port: 12345 metrics: global: scrape_interval: 15s remote_write: - url: https://prometheus-us-central1.grafana.net/api/prom/push basic_auth: username: YOUR_USERNAME password: YOUR_API_KEY configs: - name: java-applications scrape_configs: # Scrape Spring Boot Actuator endpoints - job_name: 'spring-boot-apps' scrape_interval: 15s metrics_path: '/actuator/prometheus' static_configs: - targets: ['host.docker.internal:8080'] labels: application: 'user-service' environment: 'development' relabel_configs: - source_labels: [__address__] target_label: instance - source_labels: [__meta_kubernetes_pod_name] target_label: pod # Scrape JMX metrics - job_name: 'jmx-apps' static_configs: - targets: ['host.docker.internal:12346'] metrics_path: '/metrics' remote_write: - url: https://prometheus-us-central1.grafana.net/api/prom/push basic_auth: username: YOUR_USERNAME password: YOUR_API_KEY logs: configs: - name: java-logs positions: filename: /tmp/positions.yaml scrape_configs: - job_name: java-app-logs static_configs: - targets: [localhost] labels: job: user-service-logs __path__: /var/log/java-app/*.log clients: - url: https://logs-prod-us-central1.grafana.net/loki/api/v1/push basic_auth: username: YOUR_USERNAME password: YOUR_API_KEY traces: configs: - name: java-traces receivers: otlp: protocols: grpc: http: remote_write: - endpoint: tempo-us-central1.grafana.net:443 basic_auth: username: YOUR_USERNAME password: YOUR_API_KEY
2. Docker Compose Setup
# docker-compose.yml version: '3.8' services: grafana-agent: image: grafana/agent:latest container_name: grafana-agent ports: - "12345:12345" # Agent admin interface - "12346:12346" # JMX metrics endpoint volumes: - ./grafana-agent.yaml:/etc/agent-config/agent.yaml - /var/log/java-app:/var/log/java-app command: - --config.file=/etc/agent-config/agent.yaml - --metrics.wal-directory=/tmp/grafana-agent/wal restart: unless-stopped java-application: image: my-java-app:latest container_name: java-app ports: - "8080:8080" environment: - JAVA_OPTS=-javaagent:/app/jmx_prometheus_javaagent-0.16.0.jar=12346:/app/jmx-config.yaml volumes: - ./jmx-config.yaml:/app/jmx-config.yaml - ./jmx_prometheus_javaagent.jar:/app/jmx_prometheus_javaagent.jar depends_on: - grafana-agent
3. JMX Exporter Configuration
# jmx-config.yaml lowercaseOutputName: true lowercaseOutputLabelNames: true rules: - pattern: "java.lang<type=Memory><>(.*):" name: "jvm_memory_$1" - pattern: "java.lang<type=GarbageCollector, name=(.*)><>(.*):" name: "jvm_gc_$2" labels: gc: "$1" - pattern: "java.lang<type=OperatingSystem><>(.*):" name: "jvm_os_$1" - pattern: "java.lang<type=Threading><>(.*):" name: "jvm_threads_$1" - pattern: "com.myapp<name=(.*), type=(.*)><>(.*):" name: "myapp_$1_$3" labels: type: "$2" # Custom application MBeans - pattern: "org.springframework.boot<type=Tomcat, name=global><>(.*):" name: "spring_tomcat_$1" - pattern: "org.apache.tomcat<type=ThreadPool, name=\"http-nio-8080\"><>(.*):" name: "tomcat_thread_pool_$1"
Java Application Implementation
1. Spring Boot Metrics Configuration
@Configuration
public class MetricsConfig {
@Bean
public MeterRegistryCustomizer<MeterRegistry> metricsCommonTags() {
return registry -> {
registry.config().commonTags(
"application", "user-service",
"environment", System.getenv().getOrDefault("ENVIRONMENT", "dev"),
"version", System.getenv().getOrDefault("APP_VERSION", "1.0.0"),
"region", System.getenv().getOrDefault("AWS_REGION", "local")
);
};
}
@Bean
public TimedAspect timedAspect(MeterRegistry registry) {
return new TimedAspect(registry);
}
@Bean
public CountedAspect countedAspect(MeterRegistry registry) {
return new CountedAspect(registry);
}
}
2. Custom Metrics Service
@Service
public class ApplicationMetrics {
private final MeterRegistry meterRegistry;
// Counters
private final Counter userRegistrationCounter;
private final Counter loginAttemptCounter;
private final Counter apiRequestCounter;
// Gauges
private final Gauge activeUsersGauge;
private final Gauge cacheSizeGauge;
// Timers
private final Timer databaseQueryTimer;
private final Timer externalApiCallTimer;
// Distribution summaries
private final DistributionSummary requestSizeSummary;
// Custom metrics storage for gauges
private final Map<String, Double> gaugeValues = new ConcurrentHashMap<>();
public ApplicationMetrics(MeterRegistry meterRegistry) {
this.meterRegistry = meterRegistry;
// Initialize counters
this.userRegistrationCounter = Counter.builder("app.user.registrations")
.description("Total number of user registrations")
.tag("type", "success")
.register(meterRegistry);
this.loginAttemptCounter = Counter.builder("app.user.login.attempts")
.description("Total login attempts")
.baseUnit("attempts")
.register(meterRegistry);
this.apiRequestCounter = Counter.builder("app.api.requests")
.description("API requests by endpoint and status")
.register(meterRegistry);
// Initialize gauges
this.activeUsersGauge = Gauge.builder("app.users.active")
.description("Number of currently active users")
.register(meterRegistry);
this.cacheSizeGauge = Gauge.builder("app.cache.size")
.description("Current cache size in elements")
.register(meterRegistry);
// Initialize timers
this.databaseQueryTimer = Timer.builder("app.database.queries")
.description("Database query execution time")
.publishPercentiles(0.5, 0.95, 0.99)
.publishPercentileHistogram()
.register(meterRegistry);
this.externalApiCallTimer = Timer.builder("app.external.api.calls")
.description("External API call execution time")
.publishPercentiles(0.5, 0.95, 0.99)
.register(meterRegistry);
// Initialize distribution summary
this.requestSizeSummary = DistributionSummary.builder("app.request.size")
.description("Incoming request size in bytes")
.baseUnit("bytes")
.register(meterRegistry);
}
// Counter methods
public void incrementUserRegistration(String type) {
userRegistrationCounter.increment();
}
public void incrementLoginAttempt(boolean success) {
loginAttemptCounter.increment();
if (!success) {
Counter.builder("app.user.login.failures")
.register(meterRegistry)
.increment();
}
}
public void incrementApiRequest(String endpoint, String method, int statusCode) {
apiRequestCounter.increment();
// Also record with tags for better filtering
Counter.builder("app.api.requests.detailed")
.tag("endpoint", endpoint)
.tag("method", method)
.tag("status", String.valueOf(statusCode))
.register(meterRegistry)
.increment();
}
// Gauge methods
public void setActiveUsers(int count) {
// Gauges are updated by keeping a reference to the value
gaugeValues.put("activeUsers", (double) count);
activeUsersGauge.set(count);
}
public void setCacheSize(int size) {
gaugeValues.put("cacheSize", (double) size);
cacheSizeGauge.set(size);
}
// Timer methods
public Timer.Sample startDatabaseQueryTimer() {
return Timer.start(meterRegistry);
}
public void stopDatabaseQueryTimer(Timer.Sample sample, String queryType) {
sample.stop(Timer.builder("app.database.queries.detailed")
.tag("query_type", queryType)
.register(meterRegistry));
}
public void recordExternalApiCall(Runnable operation, String service) {
externalApiCallTimer.record(operation);
// Also record with service tag
Timer.builder("app.external.api.calls.detailed")
.tag("service", service)
.register(meterRegistry)
.record(() -> {
try {
operation.run();
} catch (Exception e) {
// Record failure
Counter.builder("app.external.api.failures")
.tag("service", service)
.register(meterRegistry)
.increment();
throw e;
}
});
}
// Distribution summary methods
public void recordRequestSize(long bytes) {
requestSizeSummary.record(bytes);
}
// Business-specific metrics
public void recordOrderValue(double amount, String currency) {
Counter.builder("app.orders.total_value")
.tag("currency", currency)
.baseUnit(currency)
.register(meterRegistry)
.increment(amount);
DistributionSummary.builder("app.orders.amount")
.tag("currency", currency)
.baseUnit(currency)
.register(meterRegistry)
.record(amount);
}
}
3. Metrics Aspect for Automatic Instrumentation
@Aspect
@Component
public class MetricsAspect {
private final ApplicationMetrics metrics;
public MetricsAspect(ApplicationMetrics metrics) {
this.metrics = metrics;
}
@Around("@annotation(monitored)")
public Object monitorMethod(ProceedingJoinPoint joinPoint, Monitored monitored) throws Throwable {
String methodName = joinPoint.getSignature().getName();
String className = joinPoint.getTarget().getClass().getSimpleName();
Timer.Sample sample = Timer.start(metrics.getMeterRegistry());
try {
Object result = joinPoint.proceed();
sample.stop(Timer.builder("app.method.execution")
.tag("class", className)
.tag("method", methodName)
.tag("result", "success")
.register(metrics.getMeterRegistry()));
return result;
} catch (Exception e) {
sample.stop(Timer.builder("app.method.execution")
.tag("class", className)
.tag("method", methodName)
.tag("result", "error")
.tag("exception", e.getClass().getSimpleName())
.register(metrics.getMeterRegistry()));
throw e;
}
}
}
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.METHOD)
public @interface Monitored {
String value() default "";
}
4. REST Controller with Metrics
@RestController
@RequestMapping("/api/users")
public class UserController {
private static final Logger logger = LoggerFactory.getLogger(UserController.class);
private final UserService userService;
private final ApplicationMetrics metrics;
public UserController(UserService userService, ApplicationMetrics metrics) {
this.userService = userService;
this.metrics = metrics;
}
@PostMapping("/register")
public ResponseEntity<User> registerUser(@RequestBody UserRegistrationRequest request) {
Timer.Sample sample = metrics.startDatabaseQueryTimer();
try {
User user = userService.registerUser(request);
metrics.incrementUserRegistration("success");
metrics.incrementApiRequest("/api/users/register", "POST", 201);
return ResponseEntity.status(HttpStatus.CREATED).body(user);
} catch (Exception e) {
metrics.incrementUserRegistration("failed");
metrics.incrementApiRequest("/api/users/register", "POST", 500);
throw e;
} finally {
metrics.stopDatabaseQueryTimer(sample, "insert_user");
}
}
@PostMapping("/login")
public ResponseEntity<LoginResponse> login(@RequestBody LoginRequest request) {
metrics.incrementLoginAttempt(true);
try {
LoginResponse response = userService.authenticate(request);
metrics.incrementApiRequest("/api/users/login", "POST", 200);
return ResponseEntity.ok(response);
} catch (AuthenticationException e) {
metrics.incrementLoginAttempt(false);
metrics.incrementApiRequest("/api/users/login", "POST", 401);
return ResponseEntity.status(HttpStatus.UNAUTHORIZED).build();
}
}
@GetMapping("/{userId}")
@Monitored
public ResponseEntity<User> getUser(@PathVariable String userId) {
metrics.incrementApiRequest("/api/users/{id}", "GET", 200);
User user = userService.getUserById(userId);
return ResponseEntity.ok(user);
}
@GetMapping("/active")
public ResponseEntity<Integer> getActiveUsers() {
int activeUsers = userService.getActiveUsersCount();
metrics.setActiveUsers(activeUsers);
return ResponseEntity.ok(activeUsers);
}
}
5. Database Service with Metrics
@Service
public class UserService {
private final UserRepository userRepository;
private final ApplicationMetrics metrics;
private final Cache<String, User> userCache;
public UserService(UserRepository userRepository, ApplicationMetrics metrics) {
this.userRepository = userRepository;
this.metrics = metrics;
this.userCache = Caffeine.newBuilder()
.maximumSize(1000)
.expireAfterWrite(10, TimeUnit.MINUTES)
.build();
}
@Monitored
public User registerUser(UserRegistrationRequest request) {
// Record request size if applicable
metrics.recordRequestSize(request.toString().getBytes().length);
return metrics.recordExternalApiCall(() -> {
User user = new User(request.getEmail(), request.getName());
return userRepository.save(user);
}, "database");
}
public User getUserById(String userId) {
// Check cache first
User user = userCache.getIfPresent(userId);
if (user != null) {
return user;
}
// Cache miss - query database
Timer.Sample sample = metrics.startDatabaseQueryTimer();
try {
user = userRepository.findById(userId).orElseThrow();
userCache.put(userId, user);
metrics.setCacheSize(userCache.estimatedSize());
return user;
} finally {
metrics.stopDatabaseQueryTimer(sample, "select_user");
}
}
public int getActiveUsersCount() {
Timer.Sample sample = metrics.startDatabaseQueryTimer();
try {
return userRepository.countActiveUsers();
} finally {
metrics.stopDatabaseQueryTimer(sample, "count_users");
}
}
}
6. Health Check with Metrics
@Component
public class DatabaseHealthIndicator implements HealthIndicator {
private final UserRepository userRepository;
private final ApplicationMetrics metrics;
public DatabaseHealthIndicator(UserRepository userRepository, ApplicationMetrics metrics) {
this.userRepository = userRepository;
this.metrics = metrics;
}
@Override
public Health health() {
try {
// Simple query to check database connectivity
userRepository.count();
metrics.incrementApiRequest("/actuator/health", "GET", 200);
return Health.up()
.withDetail("database", "connected")
.withDetail("timestamp", Instant.now())
.build();
} catch (Exception e) {
metrics.incrementApiRequest("/actuator/health", "GET", 503);
return Health.down()
.withDetail("database", "disconnected")
.withDetail("error", e.getMessage())
.withDetail("timestamp", Instant.now())
.build();
}
}
}
Advanced Configuration
1. Kubernetes Deployment with Grafana Agent
# k8s-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: user-service
labels:
app: user-service
spec:
replicas: 3
selector:
matchLabels:
app: user-service
template:
metadata:
labels:
app: user-service
annotations:
prometheus.io/scrape: "true"
prometheus.io/path: "/actuator/prometheus"
prometheus.io/port: "8080"
spec:
containers:
- name: user-service
image: my-java-app:latest
ports:
- containerPort: 8080
env:
- name: ENVIRONMENT
value: "production"
- name: APP_VERSION
value: "2.1.0"
resources:
requests:
memory: "512Mi"
cpu: "250m"
limits:
memory: "1Gi"
cpu: "500m"
livenessProbe:
httpGet:
path: /actuator/health
port: 8080
initialDelaySeconds: 30
periodSeconds: 10
readinessProbe:
httpGet:
path: /actuator/health/readiness
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
---
# Grafana Agent as DaemonSet
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: grafana-agent
spec:
selector:
matchLabels:
app: grafana-agent
template:
metadata:
labels:
app: grafana-agent
spec:
containers:
- name: grafana-agent
image: grafana/agent:latest
args:
- --config.file=/etc/agent-config/agent.yaml
- --metrics.wal-directory=/tmp/grafana-agent/wal
ports:
- containerPort: 12345
name: http-metrics
volumeMounts:
- name: agent-config
mountPath: /etc/agent-config
- name: wal-storage
mountPath: /tmp/grafana-agent
resources:
requests:
memory: "64Mi"
cpu: "50m"
limits:
memory: "128Mi"
cpu: "100m"
volumes:
- name: agent-config
configMap:
name: grafana-agent-config
- name: wal-storage
emptyDir: {}
2. Advanced Grafana Agent Config for Kubernetes
# grafana-agent-k8s.yaml metrics: global: scrape_interval: 30s configs: - name: kubernetes scrape_configs: - job_name: 'kubernetes-pods' kubernetes_sd_configs: - role: pod relabel_configs: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: true - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port] action: replace regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 target_label: __address__ - action: labelmap regex: __meta_kubernetes_pod_label_(.+) - source_labels: [__meta_kubernetes_namespace] action: replace target_label: kubernetes_namespace - source_labels: [__meta_kubernetes_pod_name] action: replace target_label: kubernetes_pod_name remote_write: - url: https://prometheus-us-central1.grafana.net/api/prom/push basic_auth: username: YOUR_USERNAME password: YOUR_API_KEY
Testing and Validation
1. Metrics Endpoint Test
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
class MetricsIntegrationTest {
@Autowired
private TestRestTemplate restTemplate;
@Test
void testPrometheusEndpoint() {
ResponseEntity<String> response = restTemplate.getForEntity("/actuator/prometheus", String.class);
assertThat(response.getStatusCode()).isEqualTo(HttpStatus.OK);
assertThat(response.getBody()).contains("jvm_memory_used", "http_server_requests_seconds");
}
@Test
void testCustomMetrics() {
// Trigger some business logic
restTemplate.postForEntity("/api/users/register",
new UserRegistrationRequest("[email protected]", "Test User"), User.class);
// Check if metrics are exposed
ResponseEntity<String> metricsResponse = restTemplate.getForEntity("/actuator/prometheus", String.class);
assertThat(metricsResponse.getBody()).contains("app_user_registrations");
}
}
2. Manual Metrics Verification
# Check Spring Boot actuator endpoint curl http://localhost:8080/actuator/prometheus | grep app_ # Check JMX exporter endpoint (if enabled) curl http://localhost:12346/metrics # Check Grafana Agent status curl http://localhost:12345/agent/api/v1/metrics/targets
Best Practices
- Use Meaningful Metric Names: Follow naming conventions (
app_component_metric) - Add Relevant Tags/Labels: Include environment, version, instance information
- Avoid High Cardinality: Don't use unique values like user IDs as labels
- Set Appropriate Scrape Intervals: Balance between granularity and resource usage
- Monitor Agent Health: Set up alerts for Grafana Agent failures
- Use Histograms for Latency: For better percentile calculations
- Clean Up Unused Metrics: Remove deprecated metrics to reduce noise
// Example of good metric naming and tagging
Counter.builder("app_http_requests_total")
.description("Total HTTP requests")
.tag("method", "GET")
.tag("status", "200")
.tag("endpoint", "/api/users")
.register(registry);
Conclusion
Grafana Agent provides a powerful and efficient way to collect and forward metrics from Java applications:
- Unified observability with metrics, logs, and traces
- Kubernetes-native discovery and scraping
- Resource-efficient compared to multiple agents
- Easy integration with Grafana Cloud and self-managed Grafana
By combining Spring Boot Actuator, Micrometer, and Grafana Agent, you can build a comprehensive monitoring solution that provides deep insights into your Java applications' performance and health. The setup scales from development to production and integrates seamlessly with modern cloud-native infrastructure.
Pyroscope Profiling in Java
Explains how to use Pyroscope for continuous profiling in Java applications, helping developers analyze CPU and memory usage patterns to improve performance and identify bottlenecks.
https://macronepal.com/blog/pyroscope-profiling-in-java/
OpenTelemetry Metrics in Java: Comprehensive Guide
Provides a complete guide to collecting and exporting metrics in Java using OpenTelemetry, including counters, histograms, gauges, and integration with monitoring tools. (MACRO NEPAL)
https://macronepal.com/blog/opentelemetry-metrics-in-java-comprehensive-guide/
OTLP Exporter in Java: Complete Guide for OpenTelemetry
Explains how to configure OTLP exporters in Java to send telemetry data such as traces, metrics, and logs to monitoring systems using HTTP or gRPC protocols. (MACRO NEPAL)
https://macronepal.com/blog/otlp-exporter-in-java-complete-guide-for-opentelemetry/
Thanos Integration in Java: Global View of Metrics
Explains how to integrate Thanos with Java monitoring systems to create a scalable global metrics view across multiple Prometheus instances.
https://macronepal.com/blog/thanos-integration-in-java-global-view-of-metrics
Time Series with InfluxDB in Java: Complete Guide (Version 2)
Explains how to manage time-series data using InfluxDB in Java applications, including storing, querying, and analyzing metrics data.
https://macronepal.com/blog/time-series-with-influxdb-in-java-complete-guide-2
Time Series with InfluxDB in Java: Complete Guide
Provides an overview of integrating InfluxDB with Java for time-series data handling, including monitoring applications and managing performance metrics.
https://macronepal.com/blog/time-series-with-influxdb-in-java-complete-guide
Implementing Prometheus Remote Write in Java (Version 2)
Explains how to configure Java applications to send metrics data to Prometheus-compatible systems using the remote write feature for scalable monitoring.
https://macronepal.com/blog/implementing-prometheus-remote-write-in-java-a-complete-guide-2
Implementing Prometheus Remote Write in Java: Complete Guide
Provides instructions for sending metrics from Java services to Prometheus servers, enabling centralized monitoring and real-time analytics.
https://macronepal.com/blog/implementing-prometheus-remote-write-in-java-a-complete-guide
Building a TileServer GL in Java: Vector and Raster Tile Server
Explains how to build a TileServer GL in Java for serving vector and raster map tiles, useful for geographic visualization and mapping applications.
https://macronepal.com/blog/building-a-tileserver-gl-in-java-vector-and-raster-tile-server
Indoor Mapping in Java
Explains how to create indoor mapping systems in Java, including navigation inside buildings, spatial data handling, and visualization techniques.