Flame graphs have revolutionized how developers analyze performance in Java applications. When combined with Async-Profiler, they create a powerful toolkit for identifying bottlenecks and optimizing code. This comprehensive guide explores how to generate and interpret flame graphs to uncover performance insights.
What are Flame Graphs?
Flame graphs are visual representations of stack traces that show the frequency of code execution across your application. Created by Brendan Gregg, they provide an intuitive way to understand where your application spends most of its CPU time or where memory allocation occurs.
Key Characteristics:
- Width represents the frequency of stack traces
- Height shows the depth of call stacks
- Color typically indicates stack type (Java, C++, Kernel)
- Each rectangle represents a stack frame
Setting Up Async-Profiler
Installation
# Download latest async-profiler wget https://github.com/async-profiler/async-profiler/releases/download/v2.9/async-profiler-2.9-linux-x64.tar.gz tar -xzf async-profiler-2.9-linux-x64.tar.gz cd async-profiler-2.9-linux-x64
Basic Usage
# Profile CPU for 60 seconds and generate flame graph ./profiler.sh -d 60 -f /tmp/flamegraph.svg <pid> # Profile allocations ./profiler.sh -d 60 -e alloc -f /tmp/alloc-flamegraph.svg <pid>
Generating Flame Graphs: Common Scenarios
1. CPU Profiling
# Simple CPU profiling ./profiler.sh -d 30 -f cpu-profile.svg <pid> # With more detailed output ./profiler.sh -d 60 --chunksize 10m -f detailed-cpu.svg <pid> # Include kernel stacks ./profiler.sh -d 30 -e cpu -i 1ms --all-kernel -f cpu-kernel.svg <pid> # Include user and kernel stacks ./profiler.sh -d 30 -e cpu --all-user --all-kernel -f full-stacks.svg <pid>
2. Memory Allocation Profiling
# Allocation profiling ./profiler.sh -d 60 -e alloc -f alloc-flamegraph.svg <pid> # Live object monitoring ./profiler.sh -d 60 -e live -f live-objects.svg <pid> # Heap usage profiling ./profiler.sh -d 60 -e heap -f heap-usage.svg <pid>
3. Advanced Profiling Scenarios
# Profile with specific event ./profiler.sh -d 30 -e lock -f lock-contention.svg <pid> # Wall-clock profiling (measures elapsed time) ./profiler.sh -d 60 -e wall -t -f wall-clock.svg <pid> # Profile with custom interval ./profiler.sh -d 120 -i 10ms -f custom-interval.svg <pid>
Java-Specific Integration
JVM Attach Modes
# Attach to running JVM ./profiler.sh start <pid> ./profiler.sh stop <pid> -f profile.svg # Profile from application start java -agentpath:/path/to/libasyncProfiler.so=start,event=cpu,file=startup.svg \ -jar myapp.jar
Containerized Environments
# Profile Java app in Docker docker exec <container> /profiler.sh -d 30 -f /tmp/profile.svg <pid> # Copy profile out of container docker cp <container>:/tmp/profile.svg ./profile.svg
Interpreting Flame Graphs
Reading the Visualization
[ java.lang.Thread.run() ] ← Top (shallow) [ java.util.concurrent.ThreadPoolExecutor$Worker.run() ] [ com.example.Service.process() ] [ com.example.Database.query() ] ← Bottom (deep) [ mysql.driver.executeQuery() ]
Common Patterns and What They Mean
1. Wide Top Frames
▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆ [java.util.HashMap.get() ]
Interpretation: This method is called frequently from many different callers. May indicate a hotspot.
2. Tall Narrow Stacks
[ Top ] [ ] [ ... ] ← Deep call chain [ ] [Bottom]
Interpretation: Deep call stack with relatively low individual frequency.
3. Plateaus
[ com.example.Validator.validate() ] ← Wide plateau [ com.example.Validator.checkRule() ] [ com.example.Validator.verify() ]
Interpretation: A single method dominating execution time across multiple call paths.
Real-World Examples
Identifying CPU Bottlenecks
# Generate profile for web application ./profiler.sh -d 180 -f /tmp/webapp-cpu.svg $(pgrep -f tomcat)
Common Findings:
- JSON/XML serialization dominating
- Inefficient database query processing
- Cryptographic operations
- Regular expression compilation
Memory Allocation Analysis
# Profile allocations in microservice ./profiler.sh -d 120 -e alloc -f /tmp/alloc-profile.svg $(jps | grep MyService | cut -d' ' -f1)
Common Findings:
- Excessive string creation in loops
- Unnecessary object creation in hot paths
- Inefficient collection usage
- Memory leaks in cached data
Advanced Features and Integration
Programmatic Control
// Start profiling programmatically
AsyncProfiler.getInstance().start("cpu", 1000000);
// Critical section
performExpensiveOperation();
// Stop and get results
String svg = AsyncProfiler.getInstance().stop();
Files.write(Paths.get("profile.svg"), svg.getBytes());
Continuous Profiling
# Continuous profiling with rolling outputs while true; do ./profiler.sh -d 300 -f /tmp/profiles/profile-$(date +%s).svg <pid> sleep 10 done
Integration with APM Tools
# Generate profile and upload to monitoring system ./profiler.sh -d 60 -f /tmp/current.svg <pid> curl -X POST -F "profile=@/tmp/current.svg" \ https://monitoring.company.com/api/profiles
Best Practices
1. Profile Under Realistic Load
# Profile during load test ./profiler.sh -d 300 -f production-load.svg <pid>
2. Use Appropriate Duration
- Short runs (30-60s): For immediate issues
- Medium runs (2-5min): For typical performance analysis
- Long runs (10-30min): For intermittent issues
3. Compare Before/After
# Before optimization ./profiler.sh -d 60 -f before-optimization.svg <pid> # After optimization ./profiler.sh -d 60 -f after-optimization.svg <pid>
4. Container-Aware Profiling
# Dockerfile example
FROM openjdk:11-jre
COPY async-profiler /opt/async-profiler
ENV PATH="/opt/async-profiler:${PATH}"
Troubleshooting Common Issues
Permission Problems
# Fix permissions for container profiling docker run --cap-add=ALL --security-opt seccomp=unconfined ...
Missing Symbols
# Include debug symbols ./profiler.sh --include-klass <classname> -d 60 -f debug.svg <pid>
High Overhead
# Use lower frequency for reduced overhead ./profiler.sh -d 60 -i 10ms -f low-overhead.svg <pid>
Conclusion
Flame graphs generated with Async-Profiler provide unparalleled visibility into Java application performance. By mastering their generation and interpretation, developers can:
- Quickly identify performance bottlenecks
- Optimize resource-intensive code paths
- Validate performance improvements
- Understand complex call stack relationships
The combination of easy generation, intuitive visualization, and detailed stack trace information makes flame graphs an essential tool in every Java developer's performance optimization toolkit.
Remember that the most valuable insights often come from comparing flame graphs across different loads, versions, or configurations, revealing the true impact of your code changes on application performance.