Complete Observability: Integrating Pinpoint APM in Java Applications

In today's distributed microservices architecture, understanding application performance and troubleshooting issues across multiple services has become increasingly complex. Pinpoint APM is an open-source Application Performance Management (APM) tool specifically designed for Java applications that provides deep insights into your system's behavior, performance bottlenecks, and dependencies.

This comprehensive guide explores Pinpoint APM—from basic concepts to advanced integration techniques for Java applications.

What is Pinpoint APM?

Pinpoint is a distributed tracing system developed by Naver to monitor and analyze large-scale distributed systems. It helps developers understand the topology and interactions between various components while providing detailed performance metrics.

Key Features:

Distributed Tracing: Track requests across multiple microservices
Application Topology: Visualize service dependencies and interactions
Real-time Monitoring: Live view of application performance
Code-level Visibility: See exact method calls and execution times
Minimal Code Changes: Mostly works through Java agent instrumentation

Pinpoint Architecture Overview

Pinpoint consists of three main components:

Pinpoint Agent: Attaches to Java applications and collects data
Collector: Receives and stores data from agents
Web UI: Provides visualization and analysis interface

Java App + Agent → Pinpoint Collector → HBase → Pinpoint Web UI

Installation and Setup

Step 1: Download Pinpoint

# Download latest release from GitHub wget https://github.com/pinpoint-apm/pinpoint/releases/download/v2.5.2/pinpoint-agent-2.5.2.tar.gz tar -xzf pinpoint-agent-2.5.2.tar.gz

Step 2: Start Pinpoint Components with Docker (Recommended)

# docker-compose.yml version: '3.8' services: pinpoint-hbase: image: pinpointdocker/pinpoint-hbase:2.5.2 container_name: pinpoint-hbase ports: - "2181:2181" - "16010:16010" pinpoint-collector: image: pinpointdocker/pinpoint-collector:2.5.2 container_name: pinpoint-collector ports: - "9991-9996:9991-9996" environment: - HBASE_HOST=pinpoint-hbase depends_on: - pinpoint-hbase pinpoint-web: image: pinpointdocker/pinpoint-web:2.5.2 container_name: pinpoint-web ports: - "8080:8080" environment: - HBASE_HOST=pinpoint-hbase - COLLECTOR_HOST=pinpoint-collector depends_on: - pinpoint-hbase - pinpoint-collector

docker-compose up -d

Java Application Integration

Method 1: Java Agent (Recommended)

Using Command Line:

java -javaagent:/path/to/pinpoint-agent/pinpoint-bootstrap-2.5.2.jar \ -Dpinpoint.applicationName=MyWebApplication \ -Dpinpoint.agentId=web-app-01 \ -jar my-application.jar

Using Spring Boot Maven Plugin:

<plugin> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-maven-plugin</artifactId> <configuration> <jvmArguments> -javaagent:./pinpoint-agent/pinpoint-bootstrap-2.5.2.jar -Dpinpoint.applicationName=MySpringBootApp -Dpinpoint.agentId=spring-app-01 </jvmArguments> </configuration> </plugin>

Using Application Properties:

# application.properties pinpoint.applicationName=MySpringBootApp pinpoint.agentId=spring-app-01

Method 2: Docker Integration

FROM openjdk:11-jre-slim # Copy application jar COPY target/my-app.jar /app/my-app.jar # Copy Pinpoint agent COPY pinpoint-agent /app/pinpoint-agent # Run with Pinpoint agent CMD ["java", "-javaagent:/app/pinpoint-agent/pinpoint-bootstrap-2.5.2.jar", \ "-Dpinpoint.applicationName=MyDockerApp", \ "-Dpinpoint.agentId=docker-app-01", \ "-jar", "/app/my-app.jar"]

Configuration Deep Dive

Agent Configuration File: pinpoint.config

# Basic Identification profiler.collector.ip=192.168.1.100 profiler.collector.tcp.port=9994 profiler.collector.stat.port=9995 profiler.collector.span.port=9996 # Application Info profiler.applicationservertype=TOMCAT profiler.container.type=SPRING_BOOT # Sampling and Performance profiler.sampling.rate=1 # 1=100% sampling, 10=10% sampling profiler.io.buffering.enable=true # Instrumentation Configuration profiler.instrument.engine=ASM profiler.include= profiler.exclude= # Logging profiler.log4j.enable=true profiler.logback.enable=true

Custom Agent Configuration:

# Custom packages to monitor profiler.include=com.mycompany.myapp.service,com.mycompany.myapp.controller # Exclude specific methods/packages profiler.exclude=com.mycompany.myapp.util.InternalUtility # JVM monitoring profiler.jvm.stat.collect.interval=1000 profiler.jvm.gc.stat.collect.interval=1000

Spring Boot Integration Example

1. Basic Spring Boot Application:

@SpringBootApplication @RestController public class OrderServiceApplication { @Autowired private OrderRepository orderRepository; @GetMapping("/orders/{id}") public ResponseEntity<Order> getOrder(@PathVariable Long id) { Order order = orderRepository.findById(id) .orElseThrow(() -> new OrderNotFoundException("Order not found")); return ResponseEntity.ok(order); } @PostMapping("/orders") public ResponseEntity<Order> createOrder(@RequestBody Order order) { Order saved = orderRepository.save(order); return ResponseEntity.status(HttpStatus.CREATED).body(saved); } public static void main(String[] args) { SpringApplication.run(OrderServiceApplication.class, args); } }

2. Repository with Custom Tracing:

@Repository public class OrderRepository { private static final Logger logger = LoggerFactory.getLogger(OrderRepository.class); @PersistenceContext private EntityManager entityManager; public Optional<Order> findById(Long id) { // This method will be automatically traced by Pinpoint logger.info("Finding order by ID: {}", id); return Optional.ofNullable(entityManager.find(Order.class, id)); } }

Custom Method Tracing

Using Annotations for Custom Tracing:

@Service public class PaymentService { private final PaymentGateway paymentGateway; public PaymentService(PaymentGateway paymentGateway) { this.paymentGateway = paymentGateway; } @Trace(transactionId = "processPayment") public PaymentResult processPayment(PaymentRequest request) { // This method will appear as a separate transaction in Pinpoint validatePayment(request); PaymentResult result = paymentGateway.charge(request); logPayment(result); return result; } @Trace private void validatePayment(PaymentRequest request) { if (request.getAmount() == null || request.getAmount().compareTo(BigDecimal.ZERO) <= 0) { throw new InvalidPaymentException("Invalid payment amount"); } } @Trace private void logPayment(PaymentResult result) { // Custom logging that will be traced logger.info("Payment processed: {}", result.getTransactionId()); } }

Advanced Configuration Scenarios

1. Multiple Environment Configuration:

#!/bin/bash # start-with-pinpoint.sh ENVIRONMENT=${1:-dev} APP_NAME="my-application" AGENT_ID="${HOSTNAME}-${ENVIRONMENT}" case $ENVIRONMENT in "prod") COLLECTOR_HOST="pinpoint-collector.prod.company.com" SAMPLING_RATE="10" # 10% sampling in production ;; "staging") COLLECTOR_HOST="pinpoint-collector.staging.company.com" SAMPLING_RATE="50" ;; *) COLLECTOR_HOST="localhost" SAMPLING_RATE="100" # 100% sampling in dev ;; esac java -javaagent:/opt/pinpoint-agent/pinpoint-bootstrap.jar \ -Dpinpoint.applicationName=${APP_NAME} \ -Dpinpoint.agentId=${AGENT_ID} \ -Dprofiler.collector.ip=${COLLECTOR_HOST} \ -Dprofiler.sampling.rate=${SAMPLING_RATE} \ -jar /app/my-application.jar

2. Kubernetes Integration:

apiVersion: apps/v1 kind: Deployment metadata: name: order-service spec: replicas: 3 template: spec: containers: - name: order-service image: mycompany/order-service:1.0.0 env: - name: PINPOINT_APPLICATION_NAME value: "order-service" - name: PINPOINT_AGENT_ID valueFrom: fieldRef: fieldPath: metadata.name ports: - containerPort: 8080 volumeMounts: - name: pinpoint-agent mountPath: /pinpoint-agent command: ["java"] args: - "-javaagent:/pinpoint-agent/pinpoint-bootstrap.jar" - "-Dpinpoint.applicationName=$(PINPOINT_APPLICATION_NAME)" - "-Dpinpoint.agentId=$(PINPOINT_AGENT_ID)" - "-jar" - "/app/app.jar" volumes: - name: pinpoint-agent configMap: name: pinpoint-agent-config

Monitoring and Custom Metrics

Custom Metric Collection:

@Service public class OrderMetricsService { private final Timer orderProcessingTimer; private final Counter successfulOrdersCounter; private final Counter failedOrdersCounter; public OrderMetricsService() { this.orderProcessingTimer = new Timer(); this.successfulOrdersCounter = new Counter(); this.failedOrdersCounter = new Counter(); } public void recordOrderProcessingTime(long durationMs) { orderProcessingTimer.add(durationMs); } public void incrementSuccessfulOrders() { successfulOrdersCounter.increment(); } public void incrementFailedOrders() { failedOrdersCounter.increment(); } @Trace public void processOrderWithMetrics(Order order) { long startTime = System.currentTimeMillis(); try { // Business logic processOrder(order); incrementSuccessfulOrders(); } catch (Exception e) { incrementFailedOrders(); throw e; } finally { long duration = System.currentTimeMillis() - startTime; recordOrderProcessingTime(duration); } } }

Troubleshooting Common Issues

1. Agent Not Connecting:

# Check agent logs tail -f /path/to/pinpoint-agent/log/pinpoint.log # Verify network connectivity telnet pinpoint-collector 9994 # Check application logs for agent initialization

2. High Overhead Performance Impact:

# Reduce sampling rate profiler.sampling.rate=10 # Exclude heavy methods profiler.exclude=com.mycompany.heavy.ProcessingService # Disable specific plugins profiler.plugin.disable=spring-data-r2dbc

3. Memory Issues:

# Adjust buffer sizes profiler.io.buffering.buffersize=10 profiler.io.buffering.enable=false

Best Practices for Production

Use Meaningful Names: Choose descriptive application and agent IDs
Adjust Sampling Rates: Use lower sampling rates (1-20%) in production
Monitor Agent Performance: Keep an eye on application overhead
Secure Communication: Use internal networks for collector communication
Regular Updates: Keep Pinpoint components updated
Capacity Planning: Ensure HBase cluster can handle your data volume
Custom Instrumentation: Add tracing for critical business methods

# Production configuration example profiler.sampling.rate=5 profiler.io.buffering.enable=true profiler.jvm.stat.collect.interval=5000 profiler.include=com.company.business.critical

Integration with Other Tools

With Spring Boot Actuator:

@Configuration public class MonitoringConfig { @Bean public MeterRegistryCustomizer<MeterRegistry> metricsCommonTags() { return registry -> registry.config().commonTags( "application", "order-service", "region", System.getenv("REGION") ); } }

With Logging Correlation:

<!-- logback-spring.xml --> <configuration> <appender name="JSON" class="ch.qos.logback.core.ConsoleAppender"> <encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder"> <providers> <mdc/> <context/> <logLevel/> <loggerName/> <pattern> <pattern> { "timestamp": "%date{ISO8601}", "traceId": "%X{Pinpoint-TraceId}", "spanId": "%X{Pinpoint-SpanId}", "level": "%level", "logger": "%logger", "message": "%message", "stackTrace": "%exception" } </pattern> </pattern> </providers> </encoder> </appender> <root level="INFO"> <appender-ref ref="JSON" /> </root> </configuration>

Conclusion

Pinpoint APM provides comprehensive observability for Java applications with minimal performance overhead. By following this integration guide, you can:

Gain Deep Insights: Understand exactly how your application behaves in production
Troubleshoot Effectively: Quickly identify performance bottlenecks and errors
Visualize Dependencies: See how microservices interact with each other
Monitor Business Transactions: Track critical business processes end-to-end
Reduce MTTR: Decrease mean time to resolution for production issues

The key to successful Pinpoint integration is starting with basic configuration and gradually adding custom instrumentation as needed. With proper setup, Pinpoint becomes an invaluable tool for maintaining the health and performance of your Java applications in production environments.