Core Dump Analysis with jcmd in Java

Introduction

Core dump analysis is crucial for diagnosing complex Java application issues like crashes, memory leaks, and performance problems. The jcmd tool, introduced in JDK 7, provides comprehensive diagnostic capabilities for live JVM processes and core dump analysis.

jcmd Fundamentals

Basic jcmd Syntax and Usage

# List all Java processes
jcmd -l
# Display available commands for a specific process
jcmd <pid> help
# Get basic process information
jcmd <pid> VM.version
jcmd <pid> VM.command_line
jcmd <pid> VM.flags
# Thread information
jcmd <pid> Thread.print
jcmd <pid> VM.threads

Core Dump Generation

Creating Core Dumps

# Generate core dump for running Java process
jcmd <pid> GC.heap_dump /path/to/heapdump.hprof
jcmd <pid> VM.native_memory summary
jcmd <pid> VM.native_memory baseline
jcmd <pid> VM.native_memory summary.diff
# On Unix/Linux systems - generate system core dump
ulimit -c unlimited
kill -SIGABRT <pid>
# JVM options for core dump configuration
java -XX:+HeapDumpOnOutOfMemoryError \
-XX:HeapDumpPath=/path/to/dumps \
-XX:OnError="gcore %p" \
-jar application.jar
# Generate multiple diagnostic files
jcmd <pid> PerfCounter.print
jcmd <pid> VM.system_properties
jcmd <pid> VM.uptime

Comprehensive jcmd Diagnostic Commands

Memory Analysis Commands

# Heap histogram
jcmd <pid> GC.class_histogram
# Detailed heap histogram
jcmd <pid> GC.class_histogram -all
# Run garbage collection
jcmd <pid> GC.run
# Get heap usage
jcmd <pid> GC.heap_info
# Native memory tracking
jcmd <pid> VM.native_memory detail
jcmd <pid> VM.native_memory scale=MB

Thread and Performance Analysis

# Thread dump with additional information
jcmd <pid> Thread.print -l
# JMX data
jcmd <pid> ManagementAgent.status
jcmd <pid> ManagementAgent.start_local
# Performance counters
jcmd <pid> PerfCounter.print
# Compiler information
jcmd <pid> Compiler.queue
jcmd <pid> Compiler.directives_print

Advanced Core Dump Analysis

Heap Dump Analysis

// Example Java application that might cause memory issues
public class MemoryLeakExample {
private static final List<byte[]> LEAKING_LIST = new ArrayList<>();
private static final Map<String, String> CACHE = new ConcurrentHashMap<>();
public static void main(String[] args) {
// Simulate memory leak
ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(1);
scheduler.scheduleAtFixedRate(() -> {
// Intentionally leak memory
LEAKING_LIST.add(new byte[1024 * 1024]); // 1MB chunks
CACHE.put(UUID.randomUUID().toString(), "Some cached data");
System.out.println("Allocated: " + (LEAKING_LIST.size()) + " MB");
}, 0, 1, TimeUnit.SECONDS);
// Keep application running
try {
Thread.sleep(300000); // 5 minutes
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
}

Analysis Script for Core Dumps

#!/bin/bash
# comprehensive_dump_analysis.sh
PID=$1
DUMP_DIR="/tmp/java_dumps_$(date +%Y%m%d_%H%M%S)"
mkdir -p $DUMP_DIR
echo "Starting comprehensive analysis for PID: $PID"
echo "Output directory: $DUMP_DIR"
# Basic JVM information
jcmd $PID VM.version > $DUMP_DIR/vm_version.txt
jcmd $PID VM.command_line > $DUMP_DIR/vm_command_line.txt
jcmd $PID VM.flags > $DUMP_DIR/vm_flags.txt
# System properties
jcmd $PID VM.system_properties > $DUMP_DIR/system_properties.txt
# Memory information
jcmd $PID GC.heap_info > $DUMP_DIR/heap_info.txt
jcmd $PID VM.native_memory summary > $DUMP_DIR/native_memory_summary.txt
# Thread analysis
jcmd $PID Thread.print -l > $DUMP_DIR/thread_dump.txt
jcmd $PID VM.threads > $DUMP_DIR/vm_threads.txt
# Class histogram
jcmd $PID GC.class_histogram -all > $DUMP_DIR/class_histogram.txt
# Performance counters
jcmd $PID PerfCounter.print > $DUMP_DIR/perf_counters.txt
# Generate heap dump
jcmd $PID GC.heap_dump $DUMP_DIR/heap_dump.hprof
echo "Analysis complete. Files available in: $DUMP_DIR"

Native Memory Tracking

Enabling and Using NMT

# Start JVM with Native Memory Tracking
java -XX:NativeMemoryTracking=detail \
-XX:+UnlockDiagnosticVMOptions \
-XX:+PrintNMTStatistics \
-jar application.jar
# Monitor native memory
jcmd <pid> VM.native_memory baseline
jcmd <pid> VM.native_memory summary.diff
jcmd <pid> VM.native_memory detail.diff
jcmd <pid> VM.native_memory scale=KB
# Example output analysis
echo "=== Native Memory Tracking Summary ==="
jcmd <pid> VM.native_memory summary

NMT Analysis Script

#!/bin/bash
# nmt_monitor.sh
PID=$1
INTERVAL=${2:-10}
echo "Monitoring Native Memory for PID: $PID every ${INTERVAL}s"
echo "Time,Total,Java_Heap,Class,Thread,Code,GC,Compiler,Internal,Symbol,Native_Memory_Tracking"
while true; do
TIMESTAMP=$(date +%H:%M:%S)
jcmd $PID VM.native_memory summary | awk -v ts="$TIMESTAMP" '
/Total: / {total=$2}
/Java Heap \(reserved=/ {heap=$3}
/Class \(reserved=/ {class=$3}
/Thread \(reserved=/ {thread=$3}
/Code \(reserved=/ {code=$3}
/GC \(reserved=/ {gc=$3}
/Compiler \(reserved=/ {compiler=$3}
/Internal \(reserved=/ {internal=$3}
/Symbol \(reserved=/ {symbol=$3}
/Native Memory Tracking \(reserved=/ {nmt=$4}
END {
printf "%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s\n", 
ts, total, heap, class, thread, code, gc, compiler, internal, symbol, nmt
}'
sleep $INTERVAL
done

Thread Dump Analysis

Advanced Thread Analysis

#!/bin/bash
# thread_analysis.sh
PID=$1
COUNT=${2:-5}
INTERVAL=${3:-10}
echo "Capturing $COUNT thread dumps for PID: $PID with ${INTERVAL}s interval"
for i in $(seq 1 $COUNT); do
echo "=== Thread Dump $i at $(date) ===" >> thread_dumps_$PID.txt
jcmd $PID Thread.print -l >> thread_dumps_$PID.txt
echo -e "\n\n" >> thread_dumps_$PID.txt
if [ $i -lt $COUNT ]; then
sleep $INTERVAL
fi
done
echo "Thread dumps saved to thread_dumps_$PID.txt"
# Analyze common thread issues
echo "=== Thread State Analysis ===" > thread_analysis_$PID.txt
grep "java.lang.Thread.State:" thread_dumps_$PID.txt | sort | uniq -c >> thread_analysis_$PID.txt
echo "=== Waiting Threads ===" >> thread_analysis_$PID.txt
grep -A 2 "WAITING" thread_dumps_$PID.txt >> thread_analysis_$PID.txt
echo "=== Blocked Threads ===" >> thread_analysis_$PID.txt
grep -A 2 "BLOCKED" thread_dumps_$PID.txt >> thread_analysis_$PID.txt

Performance Counter Analysis

Monitoring Performance Metrics

#!/bin/bash
# performance_monitor.sh
PID=$1
DURATION=${2:-300}
INTERVAL=${3:-5}
END_TIME=$((SECONDS + DURATION))
OUTPUT_FILE="performance_${PID}_$(date +%Y%m%d_%H%M%S).csv"
echo "Monitoring performance for PID: $PID for ${DURATION}s"
echo "Timestamp,HeapUsed,HeapCommitted,GC_Time,Loaded_Classes,Active_Threads" > $OUTPUT_FILE
while [ $SECONDS -lt $END_TIME ]; do
TIMESTAMP=$(date +%H:%M:%S)
jcmd $PID PerfCounter.print | awk -v ts="$TIMESTAMP" '
/java.heap.used/ {heap_used=$2}
/java.heap.committed/ {heap_committed=$2}
/sun.gc.collector.0.time/ {gc_time=$2}
/java.cls.loadedClasses/ {loaded_classes=$2}
/java.threads.live/ {active_threads=$2}
END {
printf "%s,%s,%s,%s,%s,%s\n", 
ts, heap_used, heap_committed, gc_time, loaded_classes, active_threads
}' >> $OUTPUT_FILE
sleep $INTERVAL
done
echo "Performance data saved to: $OUTPUT_FILE"

Memory Leak Detection

Heap Analysis Automation

#!/bin/bash
# memory_leak_detector.sh
PID=$1
BASELINE_DIR="/tmp/memory_baseline_$PID"
if [ ! -d "$BASELINE_DIR" ]; then
echo "Creating baseline..."
mkdir -p $BASELINE_DIR
jcmd $PID GC.heap_dump $BASELINE_DIR/baseline.hprof
jcmd $PID GC.class_histogram -all > $BASELINE_DIR/baseline_histogram.txt
echo "Baseline created in: $BASELINE_DIR"
else
echo "Comparing with baseline..."
CURRENT_DIR="/tmp/memory_current_$PID"
mkdir -p $CURRENT_DIR
jcmd $PID GC.heap_dump $CURRENT_DIR/current.hprof
jcmd $PID GC.class_histogram -all > $CURRENT_DIR/current_histogram.txt
# Simple diff analysis
echo "=== Class Histogram Differences ===" > $CURRENT_DIR/comparison.txt
diff $BASELINE_DIR/baseline_histogram.txt $CURRENT_DIR/current_histogram.txt >> $CURRENT_DIR/comparison.txt
echo "Analysis complete. Check: $CURRENT_DIR/comparison.txt"
fi

JVM Troubleshooting Scenarios

OutOfMemoryError Analysis

#!/bin/bash
# oom_analysis.sh
PID=$1
echo "=== OutOfMemoryError Diagnostic ==="
echo "Timestamp: $(date)"
# Basic JVM info
echo -e "\n=== JVM Version ==="
jcmd $PID VM.version
echo -e "\n=== JVM Flags ==="
jcmd $PID VM.flags | grep -i heap
echo -e "\n=== Heap Usage ==="
jcmd $PID GC.heap_info
echo -e "\n=== Class Histogram (Top 20) ==="
jcmd $PID GC.class_histogram | head -20
echo -e "\n=== Native Memory ==="
jcmd $PID VM.native_memory summary
echo -e "\n=== GC Statistics ==="
jcmd $PID GC.run
jcmd $PID GC.heap_info
echo -e "\n=== Generating Heap Dump ==="
DUMP_FILE="/tmp/oom_analysis_${PID}_$(date +%Y%m%d_%H%M%S).hprof"
jcmd $PID GC.heap_dump $DUMP_FILE
echo "Heap dump saved to: $DUMP_FILE"

Deadlock Detection

#!/bin/bash
# deadlock_detector.sh
PID=$1
echo "Checking for deadlocks in PID: $PID"
# Thread dump analysis for deadlocks
jcmd $PID Thread.print -l | awk '
BEGIN { deadlock_found=0; in_deadlock=0 }
/Found.*deadlock/ { 
deadlock_found=1; 
in_deadlock=1;
print "=== DEADLOCK DETECTED ==="
next
}
/Found [0-9]+ deadlock/ { 
deadlock_found=1; 
in_deadlock=1;
print "=== DEADLOCK DETECTED ==="
next
}
in_deadlock && /^$/ { in_deadlock=0 }
in_deadlock { print }
END { 
if (!deadlock_found) {
print "No deadlocks found"
}
}'

Automated Monitoring System

Comprehensive Monitoring Script

#!/bin/bash
# jvm_health_monitor.sh
PID=$1
MONITOR_DIR="/tmp/jvm_monitor_${PID}_$(date +%Y%m%d_%H%M%S)"
mkdir -p $MONITOR_DIR
echo "Starting JVM health monitor for PID: $PID"
echo "Output directory: $MONITOR_DIR"
# Continuous monitoring loop
while true; do
TIMESTAMP=$(date +%Y%m%d_%H%M%S)
# Basic health checks
{
echo "=== Health Check at $(date) ==="
echo "=== Thread States ==="
jcmd $PID Thread.print -l | grep "java.lang.Thread.State:" | sort | uniq -c
echo -e "\n=== Memory Usage ==="
jcmd $PID GC.heap_info
echo -e "\n=== GC Activity ==="
jcmd $PID PerfCounter.print | grep -E "(gc.|GC.)"
echo -e "\n=== Loaded Classes ==="
jcmd $PID PerfCounter.print | grep "loadedClasses"
} > "$MONITOR_DIR/health_check_${TIMESTAMP}.txt"
# Check for critical conditions
THREAD_COUNT=$(jcmd $PID PerfCounter.print | grep "java.threads.live" | awk '{print $2}')
HEAP_USED=$(jcmd $PID GC.heap_info | grep "used" | awk '{print $3}' | sed 's/K//')
# Alert conditions
if [ $THREAD_COUNT -gt 1000 ]; then
echo "WARNING: High thread count: $THREAD_COUNT" >> "$MONITOR_DIR/alerts.txt"
fi
if [ $HEAP_USED -gt 4000000 ]; then  # 4GB threshold
echo "WARNING: High heap usage: $HEAP_USED K" >> "$MONITOR_DIR/alerts.txt"
# Generate emergency heap dump
jcmd $PID GC.heap_dump "$MONITOR_DIR/emergency_dump_${TIMESTAMP}.hprof"
fi
sleep 60  # Check every minute
done

Analysis Tools Integration

Integrating with Other Tools

#!/bin/bash
# integrated_analysis.sh
PID=$1
echo "=== Integrated JVM Analysis ==="
# Use jcmd with other JDK tools
echo -e "\n=== jstat GC Analysis ==="
jstat -gc $PID 1000 5
echo -e "\n=== jstack Thread Analysis ==="
jstack $PID > /tmp/jstack_$PID.txt
echo -e "\n=== jmap Heap Summary ==="
jmap -heap $PID
# Generate comprehensive report
{
echo "Comprehensive JVM Analysis Report"
echo "Generated: $(date)"
echo "PID: $PID"
echo -e "\n--- jcmd VM Information ---"
jcmd $PID VM.version
jcmd $PID VM.flags
echo -e "\n--- Memory Analysis ---"
jcmd $PID GC.heap_info
jcmd $PID VM.native_memory summary
echo -e "\n--- Thread Analysis ---"
jcmd $PID Thread.print -l | head -50
} > /tmp/jvm_analysis_report_$PID.txt
echo "Report generated: /tmp/jvm_analysis_report_$PID.txt"

Best Practices for Production

Production Monitoring Setup

#!/bin/bash
# production_monitor_setup.sh
# Set up core dump configuration
echo "Configuring system for core dumps..."
ulimit -c unlimited
echo "core.%p" > /proc/sys/kernel/core_pattern
# JVM options for production monitoring
JVM_OPTS="
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/var/log/java/heapdumps
-XX:ErrorFile=/var/log/java/hs_err_pid%p.log
-XX:NativeMemoryTracking=detail
-XX:+UnlockDiagnosticVMOptions
-XX:+PrintGC
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-Xloggc:/var/log/java/gc.log
"
echo "Add these JVM options for production monitoring:"
echo "$JVM_OPTS"
# Create monitoring directory
mkdir -p /var/log/java/heapdumps
chmod 755 /var/log/java/heapdumps
echo "Production monitoring setup complete"

Conclusion

Core dump analysis with jcmd provides powerful insights into JVM behavior and application issues. Key takeaways:

jcmd offers comprehensive diagnostics for live processes and core dumps
Native Memory Tracking helps identify memory issues outside the Java heap
Regular monitoring can prevent production incidents
Automated scripts enable proactive issue detection
Integration with other JDK tools provides complete visibility

By mastering these techniques, developers and operations teams can effectively diagnose and resolve complex Java application issues in production environments.