In today's distributed microservices architecture, understanding application performance and troubleshooting issues across multiple services has become increasingly complex. Pinpoint APM is an open-source Application Performance Management (APM) tool specifically designed for Java applications that provides deep insights into your system's behavior, performance bottlenecks, and dependencies.
This comprehensive guide explores Pinpoint APM—from basic concepts to advanced integration techniques for Java applications.
What is Pinpoint APM?
Pinpoint is a distributed tracing system developed by Naver to monitor and analyze large-scale distributed systems. It helps developers understand the topology and interactions between various components while providing detailed performance metrics.
Key Features:
- Distributed Tracing: Track requests across multiple microservices
- Application Topology: Visualize service dependencies and interactions
- Real-time Monitoring: Live view of application performance
- Code-level Visibility: See exact method calls and execution times
- Minimal Code Changes: Mostly works through Java agent instrumentation
Pinpoint Architecture Overview
Pinpoint consists of three main components:
- Pinpoint Agent: Attaches to Java applications and collects data
- Collector: Receives and stores data from agents
- Web UI: Provides visualization and analysis interface
Java App + Agent → Pinpoint Collector → HBase → Pinpoint Web UI
Installation and Setup
Step 1: Download Pinpoint
# Download latest release from GitHub wget https://github.com/pinpoint-apm/pinpoint/releases/download/v2.5.2/pinpoint-agent-2.5.2.tar.gz tar -xzf pinpoint-agent-2.5.2.tar.gz
Step 2: Start Pinpoint Components with Docker (Recommended)
# docker-compose.yml version: '3.8' services: pinpoint-hbase: image: pinpointdocker/pinpoint-hbase:2.5.2 container_name: pinpoint-hbase ports: - "2181:2181" - "16010:16010" pinpoint-collector: image: pinpointdocker/pinpoint-collector:2.5.2 container_name: pinpoint-collector ports: - "9991-9996:9991-9996" environment: - HBASE_HOST=pinpoint-hbase depends_on: - pinpoint-hbase pinpoint-web: image: pinpointdocker/pinpoint-web:2.5.2 container_name: pinpoint-web ports: - "8080:8080" environment: - HBASE_HOST=pinpoint-hbase - COLLECTOR_HOST=pinpoint-collector depends_on: - pinpoint-hbase - pinpoint-collector
docker-compose up -d
Java Application Integration
Method 1: Java Agent (Recommended)
Using Command Line:
java -javaagent:/path/to/pinpoint-agent/pinpoint-bootstrap-2.5.2.jar \ -Dpinpoint.applicationName=MyWebApplication \ -Dpinpoint.agentId=web-app-01 \ -jar my-application.jar
Using Spring Boot Maven Plugin:
<plugin> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-maven-plugin</artifactId> <configuration> <jvmArguments> -javaagent:./pinpoint-agent/pinpoint-bootstrap-2.5.2.jar -Dpinpoint.applicationName=MySpringBootApp -Dpinpoint.agentId=spring-app-01 </jvmArguments> </configuration> </plugin>
Using Application Properties:
# application.properties pinpoint.applicationName=MySpringBootApp pinpoint.agentId=spring-app-01
Method 2: Docker Integration
FROM openjdk:11-jre-slim # Copy application jar COPY target/my-app.jar /app/my-app.jar # Copy Pinpoint agent COPY pinpoint-agent /app/pinpoint-agent # Run with Pinpoint agent CMD ["java", "-javaagent:/app/pinpoint-agent/pinpoint-bootstrap-2.5.2.jar", \ "-Dpinpoint.applicationName=MyDockerApp", \ "-Dpinpoint.agentId=docker-app-01", \ "-jar", "/app/my-app.jar"]
Configuration Deep Dive
Agent Configuration File: pinpoint.config
# Basic Identification profiler.collector.ip=192.168.1.100 profiler.collector.tcp.port=9994 profiler.collector.stat.port=9995 profiler.collector.span.port=9996 # Application Info profiler.applicationservertype=TOMCAT profiler.container.type=SPRING_BOOT # Sampling and Performance profiler.sampling.rate=1 # 1=100% sampling, 10=10% sampling profiler.io.buffering.enable=true # Instrumentation Configuration profiler.instrument.engine=ASM profiler.include= profiler.exclude= # Logging profiler.log4j.enable=true profiler.logback.enable=true
Custom Agent Configuration:
# Custom packages to monitor profiler.include=com.mycompany.myapp.service,com.mycompany.myapp.controller # Exclude specific methods/packages profiler.exclude=com.mycompany.myapp.util.InternalUtility # JVM monitoring profiler.jvm.stat.collect.interval=1000 profiler.jvm.gc.stat.collect.interval=1000
Spring Boot Integration Example
1. Basic Spring Boot Application:
@SpringBootApplication
@RestController
public class OrderServiceApplication {
@Autowired
private OrderRepository orderRepository;
@GetMapping("/orders/{id}")
public ResponseEntity<Order> getOrder(@PathVariable Long id) {
Order order = orderRepository.findById(id)
.orElseThrow(() -> new OrderNotFoundException("Order not found"));
return ResponseEntity.ok(order);
}
@PostMapping("/orders")
public ResponseEntity<Order> createOrder(@RequestBody Order order) {
Order saved = orderRepository.save(order);
return ResponseEntity.status(HttpStatus.CREATED).body(saved);
}
public static void main(String[] args) {
SpringApplication.run(OrderServiceApplication.class, args);
}
}
2. Repository with Custom Tracing:
@Repository
public class OrderRepository {
private static final Logger logger = LoggerFactory.getLogger(OrderRepository.class);
@PersistenceContext
private EntityManager entityManager;
public Optional<Order> findById(Long id) {
// This method will be automatically traced by Pinpoint
logger.info("Finding order by ID: {}", id);
return Optional.ofNullable(entityManager.find(Order.class, id));
}
}
Custom Method Tracing
Using Annotations for Custom Tracing:
@Service
public class PaymentService {
private final PaymentGateway paymentGateway;
public PaymentService(PaymentGateway paymentGateway) {
this.paymentGateway = paymentGateway;
}
@Trace(transactionId = "processPayment")
public PaymentResult processPayment(PaymentRequest request) {
// This method will appear as a separate transaction in Pinpoint
validatePayment(request);
PaymentResult result = paymentGateway.charge(request);
logPayment(result);
return result;
}
@Trace
private void validatePayment(PaymentRequest request) {
if (request.getAmount() == null || request.getAmount().compareTo(BigDecimal.ZERO) <= 0) {
throw new InvalidPaymentException("Invalid payment amount");
}
}
@Trace
private void logPayment(PaymentResult result) {
// Custom logging that will be traced
logger.info("Payment processed: {}", result.getTransactionId());
}
}
Advanced Configuration Scenarios
1. Multiple Environment Configuration:
#!/bin/bash
# start-with-pinpoint.sh
ENVIRONMENT=${1:-dev}
APP_NAME="my-application"
AGENT_ID="${HOSTNAME}-${ENVIRONMENT}"
case $ENVIRONMENT in
"prod")
COLLECTOR_HOST="pinpoint-collector.prod.company.com"
SAMPLING_RATE="10" # 10% sampling in production
;;
"staging")
COLLECTOR_HOST="pinpoint-collector.staging.company.com"
SAMPLING_RATE="50"
;;
*)
COLLECTOR_HOST="localhost"
SAMPLING_RATE="100" # 100% sampling in dev
;;
esac
java -javaagent:/opt/pinpoint-agent/pinpoint-bootstrap.jar \
-Dpinpoint.applicationName=${APP_NAME} \
-Dpinpoint.agentId=${AGENT_ID} \
-Dprofiler.collector.ip=${COLLECTOR_HOST} \
-Dprofiler.sampling.rate=${SAMPLING_RATE} \
-jar /app/my-application.jar
2. Kubernetes Integration:
apiVersion: apps/v1 kind: Deployment metadata: name: order-service spec: replicas: 3 template: spec: containers: - name: order-service image: mycompany/order-service:1.0.0 env: - name: PINPOINT_APPLICATION_NAME value: "order-service" - name: PINPOINT_AGENT_ID valueFrom: fieldRef: fieldPath: metadata.name ports: - containerPort: 8080 volumeMounts: - name: pinpoint-agent mountPath: /pinpoint-agent command: ["java"] args: - "-javaagent:/pinpoint-agent/pinpoint-bootstrap.jar" - "-Dpinpoint.applicationName=$(PINPOINT_APPLICATION_NAME)" - "-Dpinpoint.agentId=$(PINPOINT_AGENT_ID)" - "-jar" - "/app/app.jar" volumes: - name: pinpoint-agent configMap: name: pinpoint-agent-config
Monitoring and Custom Metrics
Custom Metric Collection:
@Service
public class OrderMetricsService {
private final Timer orderProcessingTimer;
private final Counter successfulOrdersCounter;
private final Counter failedOrdersCounter;
public OrderMetricsService() {
this.orderProcessingTimer = new Timer();
this.successfulOrdersCounter = new Counter();
this.failedOrdersCounter = new Counter();
}
public void recordOrderProcessingTime(long durationMs) {
orderProcessingTimer.add(durationMs);
}
public void incrementSuccessfulOrders() {
successfulOrdersCounter.increment();
}
public void incrementFailedOrders() {
failedOrdersCounter.increment();
}
@Trace
public void processOrderWithMetrics(Order order) {
long startTime = System.currentTimeMillis();
try {
// Business logic
processOrder(order);
incrementSuccessfulOrders();
} catch (Exception e) {
incrementFailedOrders();
throw e;
} finally {
long duration = System.currentTimeMillis() - startTime;
recordOrderProcessingTime(duration);
}
}
}
Troubleshooting Common Issues
1. Agent Not Connecting:
# Check agent logs tail -f /path/to/pinpoint-agent/log/pinpoint.log # Verify network connectivity telnet pinpoint-collector 9994 # Check application logs for agent initialization
2. High Overhead Performance Impact:
# Reduce sampling rate profiler.sampling.rate=10 # Exclude heavy methods profiler.exclude=com.mycompany.heavy.ProcessingService # Disable specific plugins profiler.plugin.disable=spring-data-r2dbc
3. Memory Issues:
# Adjust buffer sizes profiler.io.buffering.buffersize=10 profiler.io.buffering.enable=false
Best Practices for Production
- Use Meaningful Names: Choose descriptive application and agent IDs
- Adjust Sampling Rates: Use lower sampling rates (1-20%) in production
- Monitor Agent Performance: Keep an eye on application overhead
- Secure Communication: Use internal networks for collector communication
- Regular Updates: Keep Pinpoint components updated
- Capacity Planning: Ensure HBase cluster can handle your data volume
- Custom Instrumentation: Add tracing for critical business methods
# Production configuration example profiler.sampling.rate=5 profiler.io.buffering.enable=true profiler.jvm.stat.collect.interval=5000 profiler.include=com.company.business.critical
Integration with Other Tools
With Spring Boot Actuator:
@Configuration
public class MonitoringConfig {
@Bean
public MeterRegistryCustomizer<MeterRegistry> metricsCommonTags() {
return registry -> registry.config().commonTags(
"application", "order-service",
"region", System.getenv("REGION")
);
}
}
With Logging Correlation:
<!-- logback-spring.xml -->
<configuration>
<appender name="JSON" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
<providers>
<mdc/>
<context/>
<logLevel/>
<loggerName/>
<pattern>
<pattern>
{
"timestamp": "%date{ISO8601}",
"traceId": "%X{Pinpoint-TraceId}",
"spanId": "%X{Pinpoint-SpanId}",
"level": "%level",
"logger": "%logger",
"message": "%message",
"stackTrace": "%exception"
}
</pattern>
</pattern>
</providers>
</encoder>
</appender>
<root level="INFO">
<appender-ref ref="JSON" />
</root>
</configuration>
Conclusion
Pinpoint APM provides comprehensive observability for Java applications with minimal performance overhead. By following this integration guide, you can:
- Gain Deep Insights: Understand exactly how your application behaves in production
- Troubleshoot Effectively: Quickly identify performance bottlenecks and errors
- Visualize Dependencies: See how microservices interact with each other
- Monitor Business Transactions: Track critical business processes end-to-end
- Reduce MTTR: Decrease mean time to resolution for production issues
The key to successful Pinpoint integration is starting with basic configuration and gradually adding custom instrumentation as needed. With proper setup, Pinpoint becomes an invaluable tool for maintaining the health and performance of your Java applications in production environments.