Thread Dump Analysis with jstack: Scripting and Automation Guide

Thread dumps are essential for diagnosing Java application issues like deadlocks, high CPU usage, memory problems, and performance bottlenecks. The jstack tool is Java's built-in utility for capturing thread information from running JVM processes.

What is jstack?

jstack is a command-line tool that comes with the JDK that prints Java thread stack traces for a specified Java process. It helps developers understand what threads are doing at a specific moment in time.

Location: $JAVA_HOME/bin/jstack

Basic jstack Usage

Syntax

jstack [options] <pid>
jstack [options] <executable> <core>
jstack [options] [server_id@]<remote server IP or hostname>

Common Options

-F : Force thread dump (use when jstack -l <pid> doesn't respond)
-l : Long listing - prints additional information about locks
-m : Mixed mode - prints both Java and native frames

Manual Thread Dump Examples

Example 1: Basic Thread Dump

# Find Java process ID
jps -l
# Take thread dump
jstack 1234 > thread_dump.txt
# Take thread dump with lock information
jstack -l 1234 > thread_dump_detailed.txt

Example 2: Force Thread Dump (for unresponsive processes)

jstack -F 1234 > forced_thread_dump.txt

Automated Thread Dump Scripts

Script 1: Basic Automated Thread Dump Collector

#!/bin/bash
# thread_dump_collector.sh
# Configuration
JAVA_PID=$1
DUMP_COUNT=${2:-5}          # Number of dumps to collect
INTERVAL=${3:-10}           # Seconds between dumps
OUTPUT_DIR="./thread_dumps"
# Validate input
if [ -z "$JAVA_PID" ]; then
echo "Usage: $0 <java_pid> [dump_count] [interval_seconds]"
echo "Available Java processes:"
jps -l
exit 1
fi
# Create output directory
mkdir -p "$OUTPUT_DIR"
echo "Starting thread dump collection for PID: $JAVA_PID"
echo "Collecting $DUMP_COUNT dumps with $INTERVAL second intervals"
for ((i=1; i<=DUMP_COUNT; i++))
do
TIMESTAMP=$(date +"%Y%m%d_%H%M%S")
FILENAME="$OUTPUT_DIR/thread_dump_${JAVA_PID}_${TIMESTAMP}_${i}.txt"
echo "Taking thread dump $i at $TIMESTAMP -> $FILENAME"
jstack -l $JAVA_PID > "$FILENAME"
# Verify dump was successful
if [ $? -eq 0 ] && [ -s "$FILENAME" ]; then
echo "✓ Thread dump $i completed successfully"
else
echo "✗ Failed to take thread dump $i"
fi
# Wait between dumps (except after the last one)
if [ $i -lt $DUMP_COUNT ]; then
sleep $INTERVAL
fi
done
echo "Thread dump collection completed. Files saved in: $OUTPUT_DIR"

Script 2: Advanced Monitoring with CPU Threshold

#!/bin/bash
# monitor_threads_with_threshold.sh
JAVA_PID=$1
CPU_THRESHOLD=${2:-80}      # Default 80% CPU threshold
CHECK_INTERVAL=${3:-30}     # Check every 30 seconds
OUTPUT_DIR="./high_cpu_dumps"
# Function to get process CPU usage
get_cpu_usage() {
local pid=$1
ps -p $pid -o %cpu --no-headers | awk '{print int($1)}'
}
# Function to take comprehensive thread dump
take_comprehensive_dump() {
local pid=$1
local reason=$2
local timestamp=$(date +"%Y%m%d_%H%M%S")
local filename="$OUTPUT_DIR/thread_dump_${pid}_${timestamp}_${reason}.txt"
echo "=== Taking comprehensive thread dump ===" > "$filename"
echo "Timestamp: $(date)" >> "$filename"
echo "Reason: $reason" >> "$filename"
echo "PID: $pid" >> "$filename"
echo "CPU Usage: $(get_cpu_usage $pid)%" >> "$filename"
echo "========================================" >> "$filename"
echo "" >> "$filename"
# Thread dump with locks
jstack -l $pid >> "$filename" 2>/dev/null
# Additional system information
echo "" >> "$filename"
echo "=== System Information ===" >> "$filename"
top -b -n 1 -p $pid >> "$filename" 2>/dev/null
echo "$filename"
}
# Validate input
if [ -z "$JAVA_PID" ]; then
echo "Usage: $0 <java_pid> [cpu_threshold] [check_interval]"
jps -l
exit 1
fi
# Create output directory
mkdir -p "$OUTPUT_DIR"
echo "Starting thread monitoring for PID: $JAVA_PID"
echo "CPU Threshold: ${CPU_THRESHOLD}%"
echo "Check Interval: ${CHECK_INTERVAL}s"
echo "Output Directory: $OUTPUT_DIR"
while true; do
CPU_USAGE=$(get_cpu_usage $JAVA_PID)
TIMESTAMP=$(date +"%Y-%m-%d %H:%M:%S")
echo "[$TIMESTAMP] PID: $JAVA_PID, CPU: ${CPU_USAGE}%"
if [ "$CPU_USAGE" -ge "$CPU_THRESHOLD" ]; then
echo "⚠️  High CPU usage detected! Taking thread dump..."
DUMP_FILE=$(take_comprehensive_dump $JAVA_PID "high_cpu_${CPU_USAGE}percent")
echo "✓ Thread dump saved: $DUMP_FILE"
# Also take 3 quick successive dumps to see thread progression
for i in {1..3}; do
sleep 2
jstack -l $JAVA_PID > "${DUMP_FILE%.txt}_quick_${i}.txt" 2>/dev/null
done
echo "✓ Additional quick dumps taken"
fi
sleep $CHECK_INTERVAL
done

Script 3: Deadlock Detection and Auto-Dump

#!/bin/bash
# deadlock_detector.sh
JAVA_PID=$1
CHECK_INTERVAL=${2:-60}
OUTPUT_DIR="./deadlock_dumps"
# Function to check for deadlocks
check_deadlocks() {
local pid=$1
local dump_file=$(mktemp)
jstack -l $pid > "$dump_file" 2>/dev/null
# Check for deadlocks in the thread dump
if grep -q "Found one Java-level deadlock:" "$dump_file" || \
grep -q "Found .* deadlock" "$dump_file"; then
echo "DEADLOCK"
else
echo "CLEAN"
fi
rm -f "$dump_file"
}
# Function to take deadlock dump
take_deadlock_dump() {
local pid=$1
local timestamp=$(date +"%Y%m%d_%H%M%S")
local filename="$OUTPUT_DIR/deadlock_dump_${pid}_${timestamp}.txt"
echo "🚨 DEADLOCK DETECTED! Taking detailed dump..." > "$filename"
echo "Timestamp: $(date)" >> "$filename"
echo "PID: $pid" >> "$filename"
echo "" >> "$filename"
# Thread dump with detailed lock information
jstack -l -F $pid >> "$filename" 2>/dev/null
echo "$filename"
}
# Validate input
if [ -z "$JAVA_PID" ]; then
echo "Usage: $0 <java_pid> [check_interval]"
jps -l
exit 1
fi
mkdir -p "$OUTPUT_DIR"
echo "Starting deadlock monitoring for PID: $JAVA_PID"
echo "Check Interval: ${CHECK_INTERVAL}s"
while true; do
STATUS=$(check_deadlocks $JAVA_PID)
TIMESTAMP=$(date +"%Y-%m-%d %H:%M:%S")
if [ "$STATUS" = "DEADLOCK" ]; then
echo "[$TIMESTAMP] 🚨 DEADLOCK DETECTED in PID: $JAVA_PID"
DUMP_FILE=$(take_deadlock_dump $JAVA_PID)
echo "✓ Deadlock dump saved: $DUMP_FILE"
# Optional: Send notification
# mail -s "Deadlock Detected in $JAVA_PID" [email protected] < "$DUMP_FILE"
# Wait longer after detecting deadlock
sleep 300
else
echo "[$TIMESTAMP] ✓ No deadlocks found"
fi
sleep $CHECK_INTERVAL
done

Windows Batch Script Examples

Script 4: Windows Thread Dump Collector

@echo off
REM thread_dump_windows.bat
set JAVA_PID=%1
set DUMP_COUNT=%2
set INTERVAL=%3
if "%JAVA_PID%"=="" (
echo Usage: %0 ^<java_pid^> ^[dump_count^] ^[interval_seconds^]
jps -l
goto :exit
)
if "%DUMP_COUNT%"=="" set DUMP_COUNT=5
if "%INTERVAL%"=="" set INTERVAL=10
set OUTPUT_DIR=thread_dumps
if not exist "%OUTPUT_DIR%" mkdir "%OUTPUT_DIR%"
echo Starting thread dump collection for PID: %JAVA_PID%
echo Collecting %DUMP_COUNT% dumps with %INTERVAL% second intervals
for /l %%i in (1,1,%DUMP_COUNT%) do (
for /f "tokens=2 delims==" %%t in ('wmic OS Get localdatetime /value') do set TIMESTAMP=%%t
set FILENAME=%OUTPUT_DIR%\thread_dump_%JAVA_PID%_%TIMESTAMP:~0,8%_%TIMESTAMP:~8,6%_%%i.txt
echo Taking thread dump %%i at %TIMESTAMP%
jstack -l %JAVA_PID% > "%FILENAME%"
if errorlevel 1 (
echo Failed to take thread dump %%i
) else (
echo ✓ Thread dump %%i completed successfully
)
if %%i lss %DUMP_COUNT% (
timeout /t %INTERVAL% /nobreak > nul
)
)
echo Thread dump collection completed
:exit

Analysis Helper Scripts

Script 5: Thread Dump Analyzer

#!/bin/bash
# analyze_thread_dump.sh
DUMP_FILE=$1
if [ -z "$DUMP_FILE" ] || [ ! -f "$DUMP_FILE" ]; then
echo "Usage: $0 <thread_dump_file>"
exit 1
fi
echo "=== Thread Dump Analysis: $DUMP_FILE ==="
echo "Analysis Time: $(date)"
echo ""
# Count total threads
TOTAL_THREADS=$(grep -c "^\".*\" " "$DUMP_FILE")
echo "Total Threads: $TOTAL_THREADS"
# Count threads by state
echo ""
echo "Thread States:"
grep "^\".*\" " "$DUMP_FILE" | awk '{print $2}' | sort | uniq -c | sort -nr
# Look for deadlocks
echo ""
echo "Deadlock Analysis:"
if grep -q "Found one Java-level deadlock:" "$DUMP_FILE"; then
echo "🚨 JAVA-LEVEL DEADLOCK DETECTED!"
awk '/Found one Java-level deadlock:/,/^$/' "$DUMP_FILE"
elif grep -q "deadlock" "$DUMP_FILE"; then
echo "⚠️  Possible deadlock references found"
grep -i "deadlock" "$DUMP_FILE"
else
echo "✓ No deadlocks detected"
fi
# Analyze thread pool patterns
echo ""
echo "Common Thread Patterns:"
echo "Pool threads: $(grep -c "pool-" "$DUMP_FILE")"
echo "HTTP threads: $(grep -c "http-" "$DUMP_FILE")"
echo "Timer threads: $(grep -c "timer-" "$DUMP_FILE")"
echo "GC threads: $(grep -c "GC" "$DUMP_FILE")"
# Show top waiting threads
echo ""
echo "Top Waiting Threads (first 10):"
grep -B2 "WAITING" "$DUMP_FILE" | grep "^\".*\" " | head -10
# Check for blocked threads
BLOCKED_COUNT=$(grep -c "BLOCKED" "$DUMP_FILE")
if [ "$BLOCKED_COUNT" -gt 0 ]; then
echo ""
echo "⚠️  Blocked Threads Found: $BLOCKED_COUNT"
grep -B1 "BLOCKED" "$DUMP_FILE" | grep "^\".*\" "
fi

Script 6: Compare Multiple Thread Dumps

#!/bin/bash
# compare_thread_dumps.sh
DUMP_DIR=$1
PATTERN=${2:-"thread_dump_*.txt"}
if [ -z "$DUMP_DIR" ] || [ ! -d "$DUMP_DIR" ]; then
echo "Usage: $0 <dump_directory> [file_pattern]"
exit 1
fi
echo "=== Thread Dump Comparison ==="
echo "Directory: $DUMP_DIR"
echo "Pattern: $PATTERN"
echo ""
# Get list of dump files
DUMP_FILES=($(find "$DUMP_DIR" -name "$PATTERN" | sort))
if [ ${#DUMP_FILES[@]} -eq 0 ]; then
echo "No thread dump files found matching pattern: $PATTERN"
exit 1
fi
echo "Found ${#DUMP_FILES[@]} thread dump files"
echo ""
# Create comparison report
REPORT_FILE="thread_dump_comparison_$(date +%Y%m%d_%H%M%S).txt"
{
echo "Thread Dump Comparison Report"
echo "Generated: $(date)"
echo "======================================"
echo ""
for file in "${DUMP_FILES[@]}"; do
echo "File: $(basename "$file")"
echo "Timestamp: $(stat -c %y "$file" 2>/dev/null || stat -f %Sm "$file")"
echo "Total Threads: $(grep -c "^\".*\" " "$file")"
echo "Thread States:"
grep "^\".*\" " "$file" | awk '{print $2}' | sort | uniq -c | sort -nr
if grep -q "Found one Java-level deadlock:" "$file"; then
echo "STATUS: 🚨 DEADLOCK DETECTED"
else
echo "STATUS: ✓ No deadlocks"
fi
echo "--------------------------------------"
echo ""
done
} > "$REPORT_FILE"
echo "Comparison report generated: $REPORT_FILE"

Advanced Script: Production Monitoring

Script 7: Production-Ready Thread Monitoring

#!/bin/bash
# production_thread_monitor.sh
# Configuration
CONFIG_FILE="./thread_monitor.conf"
LOG_FILE="./thread_monitor.log"
PID_FILE="./thread_monitor.pid"
# Load configuration
load_config() {
if [ -f "$CONFIG_FILE" ]; then
source "$CONFIG_FILE"
else
# Default configuration
JAVA_PID=""
MONITOR_INTERVAL=60
CPU_THRESHOLD=85
COLLECT_DUMPS_ON_THRESHOLD=true
MAX_DUMPS_PER_DAY=50
RETENTION_DAYS=7
ALERT_EMAIL=""
OUTPUT_BASE_DIR="/var/log/thread_dumps"
fi
}
# Logging function
log() {
echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" >> "$LOG_FILE"
echo "$1"
}
# Cleanup old dumps
cleanup_old_dumps() {
find "$OUTPUT_BASE_DIR" -name "thread_dump_*.txt" -mtime +$RETENTION_DAYS -delete
log "Cleaned up dumps older than $RETENTION_DAYS days"
}
# Send alert
send_alert() {
local subject=$1
local message=$2
log "ALERT: $subject - $message"
if [ -n "$ALERT_EMAIL" ]; then
echo "$message" | mail -s "$subject" "$ALERT_EMAIL"
fi
}
# Main monitoring loop
monitor() {
local today=$(date +%Y%m%d)
local output_dir="$OUTPUT_BASE_DIR/$today"
local dumps_today=0
mkdir -p "$output_dir"
while true; do
# Check if process is still running
if ! ps -p "$JAVA_PID" > /dev/null; then
log "Process $JAVA_PID is not running. Exiting."
exit 1
fi
# Check CPU usage
cpu_usage=$(ps -p "$JAVA_PID" -o %cpu --no-headers | awk '{print int($1)}')
if [ "$cpu_usage" -ge "$CPU_THRESHOLD" ] && [ "$dumps_today" -lt "$MAX_DUMPS_PER_DAY" ]; then
log "High CPU usage detected: ${cpu_usage}%"
timestamp=$(date +"%H%M%S")
dump_file="$output_dir/thread_dump_${JAVA_PID}_${timestamp}_cpu${cpu_usage}.txt"
# Take thread dump
jstack -l "$JAVA_PID" > "$dump_file" 2>/dev/null
if [ $? -eq 0 ]; then
dumps_today=$((dumps_today + 1))
log "Thread dump saved: $(basename "$dump_file") (${dumps_today}/${MAX_DUMPS_PER_DAY} today)"
# Analyze for immediate issues
if grep -q "Found one Java-level deadlock:" "$dump_file"; then
send_alert "DEADLOCK_DETECTED" "Deadlock found in process $JAVA_PID. Check $dump_file"
fi
else
log "Failed to take thread dump"
fi
fi
# Rotate date if needed
current_date=$(date +%Y%m%d)
if [ "$current_date" != "$today" ]; then
today="$current_date"
output_dir="$OUTPUT_BASE_DIR/$today"
dumps_today=0
mkdir -p "$output_dir"
cleanup_old_dumps
fi
sleep "$MONITOR_INTERVAL"
done
}
# Main execution
main() {
case "${1:-}" in
start)
load_config
if [ -z "$JAVA_PID" ]; then
log "JAVA_PID not configured. Please set in $CONFIG_FILE"
exit 1
fi
if [ -f "$PID_FILE" ]; then
if ps -p $(cat "$PID_FILE") > /dev/null; then
log "Monitor is already running (PID: $(cat "$PID_FILE"))"
exit 1
else
rm -f "$PID_FILE"
fi
fi
echo $$ > "$PID_FILE"
log "Starting thread monitor for PID: $JAVA_PID"
trap "rm -f '$PID_FILE'; log 'Monitor stopped'" EXIT
monitor
;;
stop)
if [ -f "$PID_FILE" ]; then
kill $(cat "$PID_FILE") 2>/dev/null
rm -f "$PID_FILE"
log "Monitor stopped"
else
log "Monitor is not running"
fi
;;
status)
if [ -f "$PID_FILE" ] && ps -p $(cat "$PID_FILE") > /dev/null; then
log "Monitor is running (PID: $(cat "$PID_FILE"))"
else
log "Monitor is not running"
fi
;;
*)
echo "Usage: $0 {start|stop|status}"
exit 1
;;
esac
}
main "$@"

Best Practices for Thread Dump Analysis

Take Multiple Dumps: Capture 3-5 dumps at 10-second intervals to see thread progression
Monitor During Issues: Capture dumps during performance degradation
Use Consistent Naming: Include timestamp, PID, and reason in filenames
Automate Cleanup: Implement retention policies to manage disk space
Correlate with Metrics: Combine with application logs and system metrics
Security: Secure dump files as they may contain sensitive information

Common Thread Dump Analysis Tools

VisualVM: GUI-based analysis
fastThread: Online thread dump analyzer
IBM Thread and Monitor Dump Analyzer: Advanced analysis
YourKit: Commercial profiler with thread analysis

These scripts provide a comprehensive toolkit for automated thread dump collection and analysis, helping you quickly identify and resolve threading issues in production environments.