Thread Dump Analysis with jstack: Scripting and Automation Guide

Thread dumps are essential for diagnosing Java application issues like deadlocks, high CPU usage, memory problems, and performance bottlenecks. The jstack tool is Java's built-in utility for capturing thread information from running JVM processes.

What is jstack?

jstack is a command-line tool that comes with the JDK that prints Java thread stack traces for a specified Java process. It helps developers understand what threads are doing at a specific moment in time.

Location: $JAVA_HOME/bin/jstack

Basic jstack Usage

Syntax

jstack [options] <pid> jstack [options] <executable> <core> jstack [options] [server_id@]<remote server IP or hostname>

Common Options

-F : Force thread dump (use when jstack -l <pid> doesn't respond)
-l : Long listing - prints additional information about locks
-m : Mixed mode - prints both Java and native frames

Manual Thread Dump Examples

Example 1: Basic Thread Dump

# Find Java process ID jps -l # Take thread dump jstack 1234 > thread_dump.txt # Take thread dump with lock information jstack -l 1234 > thread_dump_detailed.txt

Example 2: Force Thread Dump (for unresponsive processes)

jstack -F 1234 > forced_thread_dump.txt

Automated Thread Dump Scripts

Script 1: Basic Automated Thread Dump Collector

#!/bin/bash # thread_dump_collector.sh # Configuration JAVA_PID=$1 DUMP_COUNT=${2:-5} # Number of dumps to collect INTERVAL=${3:-10} # Seconds between dumps OUTPUT_DIR="./thread_dumps" # Validate input if [ -z "$JAVA_PID" ]; then echo "Usage: $0 <java_pid> [dump_count] [interval_seconds]" echo "Available Java processes:" jps -l exit 1 fi # Create output directory mkdir -p "$OUTPUT_DIR" echo "Starting thread dump collection for PID: $JAVA_PID" echo "Collecting $DUMP_COUNT dumps with $INTERVAL second intervals" for ((i=1; i<=DUMP_COUNT; i++)) do TIMESTAMP=$(date +"%Y%m%d_%H%M%S") FILENAME="$OUTPUT_DIR/thread_dump_${JAVA_PID}_${TIMESTAMP}_${i}.txt" echo "Taking thread dump $i at $TIMESTAMP -> $FILENAME" jstack -l $JAVA_PID > "$FILENAME" # Verify dump was successful if [ $? -eq 0 ] && [ -s "$FILENAME" ]; then echo "✓ Thread dump $i completed successfully" else echo "✗ Failed to take thread dump $i" fi # Wait between dumps (except after the last one) if [ $i -lt $DUMP_COUNT ]; then sleep $INTERVAL fi done echo "Thread dump collection completed. Files saved in: $OUTPUT_DIR"

Script 2: Advanced Monitoring with CPU Threshold

#!/bin/bash # monitor_threads_with_threshold.sh JAVA_PID=$1 CPU_THRESHOLD=${2:-80} # Default 80% CPU threshold CHECK_INTERVAL=${3:-30} # Check every 30 seconds OUTPUT_DIR="./high_cpu_dumps" # Function to get process CPU usage get_cpu_usage() { local pid=$1 ps -p $pid -o %cpu --no-headers | awk '{print int($1)}' } # Function to take comprehensive thread dump take_comprehensive_dump() { local pid=$1 local reason=$2 local timestamp=$(date +"%Y%m%d_%H%M%S") local filename="$OUTPUT_DIR/thread_dump_${pid}_${timestamp}_${reason}.txt" echo "=== Taking comprehensive thread dump ===" > "$filename" echo "Timestamp: $(date)" >> "$filename" echo "Reason: $reason" >> "$filename" echo "PID: $pid" >> "$filename" echo "CPU Usage: $(get_cpu_usage $pid)%" >> "$filename" echo "========================================" >> "$filename" echo "" >> "$filename" # Thread dump with locks jstack -l $pid >> "$filename" 2>/dev/null # Additional system information echo "" >> "$filename" echo "=== System Information ===" >> "$filename" top -b -n 1 -p $pid >> "$filename" 2>/dev/null echo "$filename" } # Validate input if [ -z "$JAVA_PID" ]; then echo "Usage: $0 <java_pid> [cpu_threshold] [check_interval]" jps -l exit 1 fi # Create output directory mkdir -p "$OUTPUT_DIR" echo "Starting thread monitoring for PID: $JAVA_PID" echo "CPU Threshold: ${CPU_THRESHOLD}%" echo "Check Interval: ${CHECK_INTERVAL}s" echo "Output Directory: $OUTPUT_DIR" while true; do CPU_USAGE=$(get_cpu_usage $JAVA_PID) TIMESTAMP=$(date +"%Y-%m-%d %H:%M:%S") echo "[$TIMESTAMP] PID: $JAVA_PID, CPU: ${CPU_USAGE}%" if [ "$CPU_USAGE" -ge "$CPU_THRESHOLD" ]; then echo "⚠️ High CPU usage detected! Taking thread dump..." DUMP_FILE=$(take_comprehensive_dump $JAVA_PID "high_cpu_${CPU_USAGE}percent") echo "✓ Thread dump saved: $DUMP_FILE" # Also take 3 quick successive dumps to see thread progression for i in {1..3}; do sleep 2 jstack -l $JAVA_PID > "${DUMP_FILE%.txt}_quick_${i}.txt" 2>/dev/null done echo "✓ Additional quick dumps taken" fi sleep $CHECK_INTERVAL done

Script 3: Deadlock Detection and Auto-Dump

#!/bin/bash # deadlock_detector.sh JAVA_PID=$1 CHECK_INTERVAL=${2:-60} OUTPUT_DIR="./deadlock_dumps" # Function to check for deadlocks check_deadlocks() { local pid=$1 local dump_file=$(mktemp) jstack -l $pid > "$dump_file" 2>/dev/null # Check for deadlocks in the thread dump if grep -q "Found one Java-level deadlock:" "$dump_file" || \ grep -q "Found .* deadlock" "$dump_file"; then echo "DEADLOCK" else echo "CLEAN" fi rm -f "$dump_file" } # Function to take deadlock dump take_deadlock_dump() { local pid=$1 local timestamp=$(date +"%Y%m%d_%H%M%S") local filename="$OUTPUT_DIR/deadlock_dump_${pid}_${timestamp}.txt" echo "🚨 DEADLOCK DETECTED! Taking detailed dump..." > "$filename" echo "Timestamp: $(date)" >> "$filename" echo "PID: $pid" >> "$filename" echo "" >> "$filename" # Thread dump with detailed lock information jstack -l -F $pid >> "$filename" 2>/dev/null echo "$filename" } # Validate input if [ -z "$JAVA_PID" ]; then echo "Usage: $0 <java_pid> [check_interval]" jps -l exit 1 fi mkdir -p "$OUTPUT_DIR" echo "Starting deadlock monitoring for PID: $JAVA_PID" echo "Check Interval: ${CHECK_INTERVAL}s" while true; do STATUS=$(check_deadlocks $JAVA_PID) TIMESTAMP=$(date +"%Y-%m-%d %H:%M:%S") if [ "$STATUS" = "DEADLOCK" ]; then echo "[$TIMESTAMP] 🚨 DEADLOCK DETECTED in PID: $JAVA_PID" DUMP_FILE=$(take_deadlock_dump $JAVA_PID) echo "✓ Deadlock dump saved: $DUMP_FILE" # Optional: Send notification # mail -s "Deadlock Detected in $JAVA_PID" [email protected] < "$DUMP_FILE" # Wait longer after detecting deadlock sleep 300 else echo "[$TIMESTAMP] ✓ No deadlocks found" fi sleep $CHECK_INTERVAL done

Windows Batch Script Examples

Script 4: Windows Thread Dump Collector

@echo off REM thread_dump_windows.bat set JAVA_PID=%1 set DUMP_COUNT=%2 set INTERVAL=%3 if "%JAVA_PID%"=="" ( echo Usage: %0 ^<java_pid^> ^[dump_count^] ^[interval_seconds^] jps -l goto :exit ) if "%DUMP_COUNT%"=="" set DUMP_COUNT=5 if "%INTERVAL%"=="" set INTERVAL=10 set OUTPUT_DIR=thread_dumps if not exist "%OUTPUT_DIR%" mkdir "%OUTPUT_DIR%" echo Starting thread dump collection for PID: %JAVA_PID% echo Collecting %DUMP_COUNT% dumps with %INTERVAL% second intervals for /l %%i in (1,1,%DUMP_COUNT%) do ( for /f "tokens=2 delims==" %%t in ('wmic OS Get localdatetime /value') do set TIMESTAMP=%%t set FILENAME=%OUTPUT_DIR%\thread_dump_%JAVA_PID%_%TIMESTAMP:~0,8%_%TIMESTAMP:~8,6%_%%i.txt echo Taking thread dump %%i at %TIMESTAMP% jstack -l %JAVA_PID% > "%FILENAME%" if errorlevel 1 ( echo Failed to take thread dump %%i ) else ( echo ✓ Thread dump %%i completed successfully ) if %%i lss %DUMP_COUNT% ( timeout /t %INTERVAL% /nobreak > nul ) ) echo Thread dump collection completed :exit

Analysis Helper Scripts

Script 5: Thread Dump Analyzer

#!/bin/bash # analyze_thread_dump.sh DUMP_FILE=$1 if [ -z "$DUMP_FILE" ] || [ ! -f "$DUMP_FILE" ]; then echo "Usage: $0 <thread_dump_file>" exit 1 fi echo "=== Thread Dump Analysis: $DUMP_FILE ===" echo "Analysis Time: $(date)" echo "" # Count total threads TOTAL_THREADS=$(grep -c "^\".*\" " "$DUMP_FILE") echo "Total Threads: $TOTAL_THREADS" # Count threads by state echo "" echo "Thread States:" grep "^\".*\" " "$DUMP_FILE" | awk '{print $2}' | sort | uniq -c | sort -nr # Look for deadlocks echo "" echo "Deadlock Analysis:" if grep -q "Found one Java-level deadlock:" "$DUMP_FILE"; then echo "🚨 JAVA-LEVEL DEADLOCK DETECTED!" awk '/Found one Java-level deadlock:/,/^$/' "$DUMP_FILE" elif grep -q "deadlock" "$DUMP_FILE"; then echo "⚠️ Possible deadlock references found" grep -i "deadlock" "$DUMP_FILE" else echo "✓ No deadlocks detected" fi # Analyze thread pool patterns echo "" echo "Common Thread Patterns:" echo "Pool threads: $(grep -c "pool-" "$DUMP_FILE")" echo "HTTP threads: $(grep -c "http-" "$DUMP_FILE")" echo "Timer threads: $(grep -c "timer-" "$DUMP_FILE")" echo "GC threads: $(grep -c "GC" "$DUMP_FILE")" # Show top waiting threads echo "" echo "Top Waiting Threads (first 10):" grep -B2 "WAITING" "$DUMP_FILE" | grep "^\".*\" " | head -10 # Check for blocked threads BLOCKED_COUNT=$(grep -c "BLOCKED" "$DUMP_FILE") if [ "$BLOCKED_COUNT" -gt 0 ]; then echo "" echo "⚠️ Blocked Threads Found: $BLOCKED_COUNT" grep -B1 "BLOCKED" "$DUMP_FILE" | grep "^\".*\" " fi

Script 6: Compare Multiple Thread Dumps

#!/bin/bash # compare_thread_dumps.sh DUMP_DIR=$1 PATTERN=${2:-"thread_dump_*.txt"} if [ -z "$DUMP_DIR" ] || [ ! -d "$DUMP_DIR" ]; then echo "Usage: $0 <dump_directory> [file_pattern]" exit 1 fi echo "=== Thread Dump Comparison ===" echo "Directory: $DUMP_DIR" echo "Pattern: $PATTERN" echo "" # Get list of dump files DUMP_FILES=($(find "$DUMP_DIR" -name "$PATTERN" | sort)) if [ ${#DUMP_FILES[@]} -eq 0 ]; then echo "No thread dump files found matching pattern: $PATTERN" exit 1 fi echo "Found ${#DUMP_FILES[@]} thread dump files" echo "" # Create comparison report REPORT_FILE="thread_dump_comparison_$(date +%Y%m%d_%H%M%S).txt" { echo "Thread Dump Comparison Report" echo "Generated: $(date)" echo "======================================" echo "" for file in "${DUMP_FILES[@]}"; do echo "File: $(basename "$file")" echo "Timestamp: $(stat -c %y "$file" 2>/dev/null || stat -f %Sm "$file")" echo "Total Threads: $(grep -c "^\".*\" " "$file")" echo "Thread States:" grep "^\".*\" " "$file" | awk '{print $2}' | sort | uniq -c | sort -nr if grep -q "Found one Java-level deadlock:" "$file"; then echo "STATUS: 🚨 DEADLOCK DETECTED" else echo "STATUS: ✓ No deadlocks" fi echo "--------------------------------------" echo "" done } > "$REPORT_FILE" echo "Comparison report generated: $REPORT_FILE"

Advanced Script: Production Monitoring

Script 7: Production-Ready Thread Monitoring

#!/bin/bash # production_thread_monitor.sh # Configuration CONFIG_FILE="./thread_monitor.conf" LOG_FILE="./thread_monitor.log" PID_FILE="./thread_monitor.pid" # Load configuration load_config() { if [ -f "$CONFIG_FILE" ]; then source "$CONFIG_FILE" else # Default configuration JAVA_PID="" MONITOR_INTERVAL=60 CPU_THRESHOLD=85 COLLECT_DUMPS_ON_THRESHOLD=true MAX_DUMPS_PER_DAY=50 RETENTION_DAYS=7 ALERT_EMAIL="" OUTPUT_BASE_DIR="/var/log/thread_dumps" fi } # Logging function log() { echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" >> "$LOG_FILE" echo "$1" } # Cleanup old dumps cleanup_old_dumps() { find "$OUTPUT_BASE_DIR" -name "thread_dump_*.txt" -mtime +$RETENTION_DAYS -delete log "Cleaned up dumps older than $RETENTION_DAYS days" } # Send alert send_alert() { local subject=$1 local message=$2 log "ALERT: $subject - $message" if [ -n "$ALERT_EMAIL" ]; then echo "$message" | mail -s "$subject" "$ALERT_EMAIL" fi } # Main monitoring loop monitor() { local today=$(date +%Y%m%d) local output_dir="$OUTPUT_BASE_DIR/$today" local dumps_today=0 mkdir -p "$output_dir" while true; do # Check if process is still running if ! ps -p "$JAVA_PID" > /dev/null; then log "Process $JAVA_PID is not running. Exiting." exit 1 fi # Check CPU usage cpu_usage=$(ps -p "$JAVA_PID" -o %cpu --no-headers | awk '{print int($1)}') if [ "$cpu_usage" -ge "$CPU_THRESHOLD" ] && [ "$dumps_today" -lt "$MAX_DUMPS_PER_DAY" ]; then log "High CPU usage detected: ${cpu_usage}%" timestamp=$(date +"%H%M%S") dump_file="$output_dir/thread_dump_${JAVA_PID}_${timestamp}_cpu${cpu_usage}.txt" # Take thread dump jstack -l "$JAVA_PID" > "$dump_file" 2>/dev/null if [ $? -eq 0 ]; then dumps_today=$((dumps_today + 1)) log "Thread dump saved: $(basename "$dump_file") (${dumps_today}/${MAX_DUMPS_PER_DAY} today)" # Analyze for immediate issues if grep -q "Found one Java-level deadlock:" "$dump_file"; then send_alert "DEADLOCK_DETECTED" "Deadlock found in process $JAVA_PID. Check $dump_file" fi else log "Failed to take thread dump" fi fi # Rotate date if needed current_date=$(date +%Y%m%d) if [ "$current_date" != "$today" ]; then today="$current_date" output_dir="$OUTPUT_BASE_DIR/$today" dumps_today=0 mkdir -p "$output_dir" cleanup_old_dumps fi sleep "$MONITOR_INTERVAL" done } # Main execution main() { case "${1:-}" in start) load_config if [ -z "$JAVA_PID" ]; then log "JAVA_PID not configured. Please set in $CONFIG_FILE" exit 1 fi if [ -f "$PID_FILE" ]; then if ps -p $(cat "$PID_FILE") > /dev/null; then log "Monitor is already running (PID: $(cat "$PID_FILE"))" exit 1 else rm -f "$PID_FILE" fi fi echo $$ > "$PID_FILE" log "Starting thread monitor for PID: $JAVA_PID" trap "rm -f '$PID_FILE'; log 'Monitor stopped'" EXIT monitor ;; stop) if [ -f "$PID_FILE" ]; then kill $(cat "$PID_FILE") 2>/dev/null rm -f "$PID_FILE" log "Monitor stopped" else log "Monitor is not running" fi ;; status) if [ -f "$PID_FILE" ] && ps -p $(cat "$PID_FILE") > /dev/null; then log "Monitor is running (PID: $(cat "$PID_FILE"))" else log "Monitor is not running" fi ;; *) echo "Usage: $0 {start|stop|status}" exit 1 ;; esac } main "$@"

Best Practices for Thread Dump Analysis

Take Multiple Dumps: Capture 3-5 dumps at 10-second intervals to see thread progression
Monitor During Issues: Capture dumps during performance degradation
Use Consistent Naming: Include timestamp, PID, and reason in filenames
Automate Cleanup: Implement retention policies to manage disk space
Correlate with Metrics: Combine with application logs and system metrics
Security: Secure dump files as they may contain sensitive information

Common Thread Dump Analysis Tools

VisualVM: GUI-based analysis
fastThread: Online thread dump analyzer
IBM Thread and Monitor Dump Analyzer: Advanced analysis
YourKit: Commercial profiler with thread analysis

These scripts provide a comprehensive toolkit for automated thread dump collection and analysis, helping you quickly identify and resolve threading issues in production environments.