Core Dump Analysis with jcmd in Java

Introduction

Core dump analysis is crucial for diagnosing complex Java application issues like crashes, memory leaks, and performance problems. The jcmd tool, introduced in JDK 7, provides comprehensive diagnostic capabilities for live JVM processes and core dump analysis.

jcmd Fundamentals

Basic jcmd Syntax and Usage

# List all Java processes jcmd -l # Display available commands for a specific process jcmd <pid> help # Get basic process information jcmd <pid> VM.version jcmd <pid> VM.command_line jcmd <pid> VM.flags # Thread information jcmd <pid> Thread.print jcmd <pid> VM.threads

Core Dump Generation

Creating Core Dumps

# Generate core dump for running Java process jcmd <pid> GC.heap_dump /path/to/heapdump.hprof jcmd <pid> VM.native_memory summary jcmd <pid> VM.native_memory baseline jcmd <pid> VM.native_memory summary.diff # On Unix/Linux systems - generate system core dump ulimit -c unlimited kill -SIGABRT <pid> # JVM options for core dump configuration java -XX:+HeapDumpOnOutOfMemoryError \ -XX:HeapDumpPath=/path/to/dumps \ -XX:OnError="gcore %p" \ -jar application.jar # Generate multiple diagnostic files jcmd <pid> PerfCounter.print jcmd <pid> VM.system_properties jcmd <pid> VM.uptime

Comprehensive jcmd Diagnostic Commands

Memory Analysis Commands

# Heap histogram jcmd <pid> GC.class_histogram # Detailed heap histogram jcmd <pid> GC.class_histogram -all # Run garbage collection jcmd <pid> GC.run # Get heap usage jcmd <pid> GC.heap_info # Native memory tracking jcmd <pid> VM.native_memory detail jcmd <pid> VM.native_memory scale=MB

Thread and Performance Analysis

# Thread dump with additional information jcmd <pid> Thread.print -l # JMX data jcmd <pid> ManagementAgent.status jcmd <pid> ManagementAgent.start_local # Performance counters jcmd <pid> PerfCounter.print # Compiler information jcmd <pid> Compiler.queue jcmd <pid> Compiler.directives_print

Advanced Core Dump Analysis

Heap Dump Analysis

// Example Java application that might cause memory issues public class MemoryLeakExample { private static final List<byte[]> LEAKING_LIST = new ArrayList<>(); private static final Map<String, String> CACHE = new ConcurrentHashMap<>(); public static void main(String[] args) { // Simulate memory leak ScheduledExecutorService scheduler = Executors.newScheduledThreadPool(1); scheduler.scheduleAtFixedRate(() -> { // Intentionally leak memory LEAKING_LIST.add(new byte[1024 * 1024]); // 1MB chunks CACHE.put(UUID.randomUUID().toString(), "Some cached data"); System.out.println("Allocated: " + (LEAKING_LIST.size()) + " MB"); }, 0, 1, TimeUnit.SECONDS); // Keep application running try { Thread.sleep(300000); // 5 minutes } catch (InterruptedException e) { Thread.currentThread().interrupt(); } } }

Analysis Script for Core Dumps

#!/bin/bash # comprehensive_dump_analysis.sh PID=$1 DUMP_DIR="/tmp/java_dumps_$(date +%Y%m%d_%H%M%S)" mkdir -p $DUMP_DIR echo "Starting comprehensive analysis for PID: $PID" echo "Output directory: $DUMP_DIR" # Basic JVM information jcmd $PID VM.version > $DUMP_DIR/vm_version.txt jcmd $PID VM.command_line > $DUMP_DIR/vm_command_line.txt jcmd $PID VM.flags > $DUMP_DIR/vm_flags.txt # System properties jcmd $PID VM.system_properties > $DUMP_DIR/system_properties.txt # Memory information jcmd $PID GC.heap_info > $DUMP_DIR/heap_info.txt jcmd $PID VM.native_memory summary > $DUMP_DIR/native_memory_summary.txt # Thread analysis jcmd $PID Thread.print -l > $DUMP_DIR/thread_dump.txt jcmd $PID VM.threads > $DUMP_DIR/vm_threads.txt # Class histogram jcmd $PID GC.class_histogram -all > $DUMP_DIR/class_histogram.txt # Performance counters jcmd $PID PerfCounter.print > $DUMP_DIR/perf_counters.txt # Generate heap dump jcmd $PID GC.heap_dump $DUMP_DIR/heap_dump.hprof echo "Analysis complete. Files available in: $DUMP_DIR"

Native Memory Tracking

Enabling and Using NMT

# Start JVM with Native Memory Tracking java -XX:NativeMemoryTracking=detail \ -XX:+UnlockDiagnosticVMOptions \ -XX:+PrintNMTStatistics \ -jar application.jar # Monitor native memory jcmd <pid> VM.native_memory baseline jcmd <pid> VM.native_memory summary.diff jcmd <pid> VM.native_memory detail.diff jcmd <pid> VM.native_memory scale=KB # Example output analysis echo "=== Native Memory Tracking Summary ===" jcmd <pid> VM.native_memory summary

NMT Analysis Script

#!/bin/bash # nmt_monitor.sh PID=$1 INTERVAL=${2:-10} echo "Monitoring Native Memory for PID: $PID every ${INTERVAL}s" echo "Time,Total,Java_Heap,Class,Thread,Code,GC,Compiler,Internal,Symbol,Native_Memory_Tracking" while true; do TIMESTAMP=$(date +%H:%M:%S) jcmd $PID VM.native_memory summary | awk -v ts="$TIMESTAMP" ' /Total: / {total=$2} /Java Heap \(reserved=/ {heap=$3} /Class \(reserved=/ {class=$3} /Thread \(reserved=/ {thread=$3} /Code \(reserved=/ {code=$3} /GC \(reserved=/ {gc=$3} /Compiler \(reserved=/ {compiler=$3} /Internal \(reserved=/ {internal=$3} /Symbol \(reserved=/ {symbol=$3} /Native Memory Tracking \(reserved=/ {nmt=$4} END { printf "%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s\n", ts, total, heap, class, thread, code, gc, compiler, internal, symbol, nmt }' sleep $INTERVAL done

Thread Dump Analysis

Advanced Thread Analysis

#!/bin/bash # thread_analysis.sh PID=$1 COUNT=${2:-5} INTERVAL=${3:-10} echo "Capturing $COUNT thread dumps for PID: $PID with ${INTERVAL}s interval" for i in $(seq 1 $COUNT); do echo "=== Thread Dump $i at $(date) ===" >> thread_dumps_$PID.txt jcmd $PID Thread.print -l >> thread_dumps_$PID.txt echo -e "\n\n" >> thread_dumps_$PID.txt if [ $i -lt $COUNT ]; then sleep $INTERVAL fi done echo "Thread dumps saved to thread_dumps_$PID.txt" # Analyze common thread issues echo "=== Thread State Analysis ===" > thread_analysis_$PID.txt grep "java.lang.Thread.State:" thread_dumps_$PID.txt | sort | uniq -c >> thread_analysis_$PID.txt echo "=== Waiting Threads ===" >> thread_analysis_$PID.txt grep -A 2 "WAITING" thread_dumps_$PID.txt >> thread_analysis_$PID.txt echo "=== Blocked Threads ===" >> thread_analysis_$PID.txt grep -A 2 "BLOCKED" thread_dumps_$PID.txt >> thread_analysis_$PID.txt

Performance Counter Analysis

Monitoring Performance Metrics

#!/bin/bash # performance_monitor.sh PID=$1 DURATION=${2:-300} INTERVAL=${3:-5} END_TIME=$((SECONDS + DURATION)) OUTPUT_FILE="performance_${PID}_$(date +%Y%m%d_%H%M%S).csv" echo "Monitoring performance for PID: $PID for ${DURATION}s" echo "Timestamp,HeapUsed,HeapCommitted,GC_Time,Loaded_Classes,Active_Threads" > $OUTPUT_FILE while [ $SECONDS -lt $END_TIME ]; do TIMESTAMP=$(date +%H:%M:%S) jcmd $PID PerfCounter.print | awk -v ts="$TIMESTAMP" ' /java.heap.used/ {heap_used=$2} /java.heap.committed/ {heap_committed=$2} /sun.gc.collector.0.time/ {gc_time=$2} /java.cls.loadedClasses/ {loaded_classes=$2} /java.threads.live/ {active_threads=$2} END { printf "%s,%s,%s,%s,%s,%s\n", ts, heap_used, heap_committed, gc_time, loaded_classes, active_threads }' >> $OUTPUT_FILE sleep $INTERVAL done echo "Performance data saved to: $OUTPUT_FILE"

Memory Leak Detection

Heap Analysis Automation

#!/bin/bash # memory_leak_detector.sh PID=$1 BASELINE_DIR="/tmp/memory_baseline_$PID" if [ ! -d "$BASELINE_DIR" ]; then echo "Creating baseline..." mkdir -p $BASELINE_DIR jcmd $PID GC.heap_dump $BASELINE_DIR/baseline.hprof jcmd $PID GC.class_histogram -all > $BASELINE_DIR/baseline_histogram.txt echo "Baseline created in: $BASELINE_DIR" else echo "Comparing with baseline..." CURRENT_DIR="/tmp/memory_current_$PID" mkdir -p $CURRENT_DIR jcmd $PID GC.heap_dump $CURRENT_DIR/current.hprof jcmd $PID GC.class_histogram -all > $CURRENT_DIR/current_histogram.txt # Simple diff analysis echo "=== Class Histogram Differences ===" > $CURRENT_DIR/comparison.txt diff $BASELINE_DIR/baseline_histogram.txt $CURRENT_DIR/current_histogram.txt >> $CURRENT_DIR/comparison.txt echo "Analysis complete. Check: $CURRENT_DIR/comparison.txt" fi

JVM Troubleshooting Scenarios

OutOfMemoryError Analysis

#!/bin/bash # oom_analysis.sh PID=$1 echo "=== OutOfMemoryError Diagnostic ===" echo "Timestamp: $(date)" # Basic JVM info echo -e "\n=== JVM Version ===" jcmd $PID VM.version echo -e "\n=== JVM Flags ===" jcmd $PID VM.flags | grep -i heap echo -e "\n=== Heap Usage ===" jcmd $PID GC.heap_info echo -e "\n=== Class Histogram (Top 20) ===" jcmd $PID GC.class_histogram | head -20 echo -e "\n=== Native Memory ===" jcmd $PID VM.native_memory summary echo -e "\n=== GC Statistics ===" jcmd $PID GC.run jcmd $PID GC.heap_info echo -e "\n=== Generating Heap Dump ===" DUMP_FILE="/tmp/oom_analysis_${PID}_$(date +%Y%m%d_%H%M%S).hprof" jcmd $PID GC.heap_dump $DUMP_FILE echo "Heap dump saved to: $DUMP_FILE"

Deadlock Detection

#!/bin/bash # deadlock_detector.sh PID=$1 echo "Checking for deadlocks in PID: $PID" # Thread dump analysis for deadlocks jcmd $PID Thread.print -l | awk ' BEGIN { deadlock_found=0; in_deadlock=0 } /Found.*deadlock/ { deadlock_found=1; in_deadlock=1; print "=== DEADLOCK DETECTED ===" next } /Found [0-9]+ deadlock/ { deadlock_found=1; in_deadlock=1; print "=== DEADLOCK DETECTED ===" next } in_deadlock && /^$/ { in_deadlock=0 } in_deadlock { print } END { if (!deadlock_found) { print "No deadlocks found" } }'

Automated Monitoring System

Comprehensive Monitoring Script

#!/bin/bash # jvm_health_monitor.sh PID=$1 MONITOR_DIR="/tmp/jvm_monitor_${PID}_$(date +%Y%m%d_%H%M%S)" mkdir -p $MONITOR_DIR echo "Starting JVM health monitor for PID: $PID" echo "Output directory: $MONITOR_DIR" # Continuous monitoring loop while true; do TIMESTAMP=$(date +%Y%m%d_%H%M%S) # Basic health checks { echo "=== Health Check at $(date) ===" echo "=== Thread States ===" jcmd $PID Thread.print -l | grep "java.lang.Thread.State:" | sort | uniq -c echo -e "\n=== Memory Usage ===" jcmd $PID GC.heap_info echo -e "\n=== GC Activity ===" jcmd $PID PerfCounter.print | grep -E "(gc.|GC.)" echo -e "\n=== Loaded Classes ===" jcmd $PID PerfCounter.print | grep "loadedClasses" } > "$MONITOR_DIR/health_check_${TIMESTAMP}.txt" # Check for critical conditions THREAD_COUNT=$(jcmd $PID PerfCounter.print | grep "java.threads.live" | awk '{print $2}') HEAP_USED=$(jcmd $PID GC.heap_info | grep "used" | awk '{print $3}' | sed 's/K//') # Alert conditions if [ $THREAD_COUNT -gt 1000 ]; then echo "WARNING: High thread count: $THREAD_COUNT" >> "$MONITOR_DIR/alerts.txt" fi if [ $HEAP_USED -gt 4000000 ]; then # 4GB threshold echo "WARNING: High heap usage: $HEAP_USED K" >> "$MONITOR_DIR/alerts.txt" # Generate emergency heap dump jcmd $PID GC.heap_dump "$MONITOR_DIR/emergency_dump_${TIMESTAMP}.hprof" fi sleep 60 # Check every minute done

Analysis Tools Integration

Integrating with Other Tools

#!/bin/bash # integrated_analysis.sh PID=$1 echo "=== Integrated JVM Analysis ===" # Use jcmd with other JDK tools echo -e "\n=== jstat GC Analysis ===" jstat -gc $PID 1000 5 echo -e "\n=== jstack Thread Analysis ===" jstack $PID > /tmp/jstack_$PID.txt echo -e "\n=== jmap Heap Summary ===" jmap -heap $PID # Generate comprehensive report { echo "Comprehensive JVM Analysis Report" echo "Generated: $(date)" echo "PID: $PID" echo -e "\n--- jcmd VM Information ---" jcmd $PID VM.version jcmd $PID VM.flags echo -e "\n--- Memory Analysis ---" jcmd $PID GC.heap_info jcmd $PID VM.native_memory summary echo -e "\n--- Thread Analysis ---" jcmd $PID Thread.print -l | head -50 } > /tmp/jvm_analysis_report_$PID.txt echo "Report generated: /tmp/jvm_analysis_report_$PID.txt"

Best Practices for Production

Production Monitoring Setup

#!/bin/bash # production_monitor_setup.sh # Set up core dump configuration echo "Configuring system for core dumps..." ulimit -c unlimited echo "core.%p" > /proc/sys/kernel/core_pattern # JVM options for production monitoring JVM_OPTS=" -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/log/java/heapdumps -XX:ErrorFile=/var/log/java/hs_err_pid%p.log -XX:NativeMemoryTracking=detail -XX:+UnlockDiagnosticVMOptions -XX:+PrintGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:/var/log/java/gc.log " echo "Add these JVM options for production monitoring:" echo "$JVM_OPTS" # Create monitoring directory mkdir -p /var/log/java/heapdumps chmod 755 /var/log/java/heapdumps echo "Production monitoring setup complete"

Conclusion

Core dump analysis with jcmd provides powerful insights into JVM behavior and application issues. Key takeaways:

jcmd offers comprehensive diagnostics for live processes and core dumps
Native Memory Tracking helps identify memory issues outside the Java heap
Regular monitoring can prevent production incidents
Automated scripts enable proactive issue detection
Integration with other JDK tools provides complete visibility

By mastering these techniques, developers and operations teams can effectively diagnose and resolve complex Java application issues in production environments.