Introduction to scp
The scp (Secure Copy) command is a network file transfer tool that uses SSH (Secure Shell) for secure data transfer between local and remote hosts. It provides encrypted authentication and data transfer, making it a reliable choice for moving files across networks securely.
Key Concepts
- SSH-based: Uses the same authentication and security as SSH
- Encryption: All data transferred is encrypted
- Authentication: Supports password, key-based, and other SSH authentication methods
- Recursive Copy: Can copy entire directory structures
- Preserves Attributes: Can maintain file permissions and timestamps
1. Basic Syntax and Usage
Basic Syntax
# Basic syntax scp [options] source_file target_file # Local to remote scp file.txt user@remote_host:/path/to/destination/ # Remote to local scp user@remote_host:/path/to/file.txt /local/destination/ # Between two remote hosts scp user1@host1:/path/file.txt user2@host2:/path/destination/ # Examples # Copy local file to remote home directory scp document.pdf [email protected]:~/Documents/ # Copy remote file to current local directory scp [email protected]:/var/log/syslog . # Copy with different username scp [email protected]:~/data.txt .
Common Use Cases
# Copy single file scp backup.tar.gz admin@backup-server:/backups/ # Copy with different remote filename scp local.txt user@host:remote.txt # Copy multiple files scp file1.txt file2.txt file3.txt user@host:/destination/ # Copy with wildcards scp *.log user@host:/logs/ # Copy entire directory recursively scp -r documents/ user@host:/backup/ # Preserve file attributes (permissions, timestamps) scp -p important.conf user@host:/etc/
2. Command Options
-P (Port) Option
# Specify SSH port (note: uppercase P) scp -P 2222 file.txt user@host:/destination/ # Common alternative SSH port scp -P 2222 backup.tar.gz [email protected]:/backups/ # Multiple files with custom port scp -P 2222 *.log user@host:/var/log/ # Script with variable port PORT=2222 scp -P $PORT file.txt user@host:~ # Check port connectivity first check_port() { local host="$1" local port="${2:-22}" nc -zv "$host" "$port" 2>&1 | grep -q succeeded } if check_port server.com 2222; then scp -P 2222 file.txt [email protected]: else echo "Port 2222 not accessible" fi
-r (Recursive) Option
# Copy entire directory recursively
scp -r projects/ user@host:/backup/
# Copy directory contents (not the directory itself)
scp -r projects/* user@host:/backup/projects/
# Copy multiple directories
scp -r dir1 dir2 dir3 user@host:/destination/
# Preserve symlinks when copying recursively
scp -r -l projects/ user@host:/backup/
# Exclude certain files while copying
copy_with_exclude() {
local source="$1"
local dest="$2"
local exclude="${3:-.git}"
tar --exclude="$exclude" -cf - "$source" | \
ssh user@host "tar -xf - -C $dest"
}
# Progress for recursive copy
scp -rv big_directory/ user@host:/destination/
-p (Preserve) Option
# Preserve modification times, access times, and modes
scp -p important.conf user@host:/etc/
# Preserve when copying backups
scp -rp /var/backups/ user@backup-server:/backups/
# Verify preservation
copy_with_attributes() {
local src="$1"
local dest="$2"
# Get original attributes
orig_perms=$(stat -c %a "$src")
orig_mtime=$(stat -c %y "$src")
# Copy with preservation
scp -p "$src" "$dest"
# Verify
new_perms=$(stat -c %a "$dest")
if [ "$orig_perms" = "$new_perms" ]; then
echo "✓ Permissions preserved"
else
echo "✗ Permissions changed"
fi
}
-C (Compression) Option
# Enable compression (useful for slow connections)
scp -C large_file.tar.gz user@host:/backup/
# Compress directory transfer
scp -Cr documents/ user@host:/backup/
# Measure compression benefit
test_compression() {
local file="$1"
local host="$2"
echo "Without compression:"
time scp "$file" "$host":~/test/
echo "With compression:"
time scp -C "$file" "$host":~/test/
}
# Adaptive compression based on file type
smart_copy() {
local file="$1"
local dest="$2"
# Don't compress already compressed files
if [[ "$file" =~ \.(gz|bz2|zip|jpg|png)$ ]]; then
scp "$file" "$dest"
else
scp -C "$file" "$dest"
fi
}
-l (Limit) Option
# Limit bandwidth (in Kbit/s)
scp -l 1000 bigfile.iso user@host:/downloads/ # Limit to 1000 Kbit/s
# Throttle transfer
scp -l 500 -r videos/ user@host:/media/
# Dynamic rate limiting
rate_limited_copy() {
local file="$1"
local dest="$2"
local max_rate="${3:-1000}" # Default 1Mbit/s
# Check current network usage
current_load=$(ifstat 1 1 | tail -1 | awk '{print $1}')
if [ "$(echo "$current_load > 50" | bc)" -eq 1 ]; then
rate=$((max_rate / 2))
else
rate=$max_rate
fi
scp -l "$rate" "$file" "$dest"
}
-i (Identity) Option
# Use specific SSH private key
scp -i ~/.ssh/id_rsa_custom file.txt user@host:/destination/
# Multiple key files
scp -i ~/.ssh/key1 -i ~/.ssh/key2 file.txt user@host:
# Key selection based on host
copy_with_key() {
local host="$1"
local file="$2"
local dest="$3"
case "$host" in
*.github.com)
key="~/.ssh/github_rsa"
;;
*.company.com)
key="~/.ssh/company_rsa"
;;
*)
key="~/.ssh/id_rsa"
;;
esac
scp -i "$key" "$file" "user@$host:$dest"
}
-v (Verbose) Option
# Verbose output for debugging
scp -v file.txt user@host:/destination/
# Debug connection issues
debug_scp() {
local src="$1"
local dest="$2"
scp -vvv "$src" "$dest" 2>&1 | tee scp_debug.log
echo "Debug log saved to scp_debug.log"
}
# Trace authentication process
trace_auth() {
scp -v file.txt user@host: 2>&1 | grep -E "Authentications|Authentication"
}
# Monitor connection phases
monitor_phases() {
scp -v file.txt user@host: 2>&1 | \
grep -E "Connecting|Connection established|Authenticated|Starting transfer" | \
while read line; do
echo "$(date +%H:%M:%S) - $line"
done
}
3. Practical Examples
Backup Scripts
#!/bin/bash
# Simple backup script
backup_files() {
local source_dir="$1"
local remote_host="$2"
local remote_dir="$3"
local timestamp=$(date +%Y%m%d_%H%M%S)
local backup_name="backup_${timestamp}.tar.gz"
echo "Creating backup archive..."
tar -czf "/tmp/$backup_name" -C "$source_dir" .
echo "Copying to remote host..."
scp "/tmp/$backup_name" "${remote_host}:${remote_dir}/"
if [ $? -eq 0 ]; then
echo "Backup successful: $backup_name"
rm "/tmp/$backup_name"
else
echo "Backup failed"
return 1
fi
}
# Incremental backup
incremental_backup() {
local source="$1"
local remote="$2"
local last_backup="/tmp/last_backup_time"
# Get last backup time
if [ -f "$last_backup" ]; then
last_time=$(cat "$last_backup")
else
last_time="1970-01-01"
fi
# Find files modified since last backup
find "$source" -type f -newermt "$last_time" -print0 | while IFS= read -r -d '' file; do
rel_path="${file#$source/}"
echo "Backing up: $rel_path"
scp "$file" "${remote}:backup/$rel_path"
done
date +%Y-%m-%d > "$last_backup"
}
# Encrypted backup
encrypted_backup() {
local source="$1"
local remote="$2"
local passphrase="$3"
tar -czf - "$source" | \
openssl enc -aes-256-cbc -salt -k "$passphrase" | \
ssh user@"$remote" "cat > backup_$(date +%Y%m%d).enc"
echo "Encrypted backup complete"
}
File Synchronization
#!/bin/bash
# Sync directory with remote
sync_directory() {
local local_dir="$1"
local remote_dir="$2"
# First, copy new/modified files
find "$local_dir" -type f -mtime -1 -exec scp {} "$remote_dir"/ \;
# Then, handle deletions (requires list of remote files)
ssh "${remote_dir%%:*}" "ls ${remote_dir#*:}" > /tmp/remote_files.txt
while read file; do
if [ ! -f "$local_dir/$file" ]; then
echo "File $file exists only on remote"
fi
done < /tmp/remote_files.txt
}
# Bi-directional sync
bidirectional_sync() {
local dir1="$1"
local dir2="$2"
# Get latest files from each side
echo "Syncing $dir1 -> $dir2"
scp -r "$dir1"/* "$dir2"/ 2>/dev/null
echo "Syncing $dir2 -> $dir1"
scp -r "$dir2"/* "$dir1"/ 2>/dev/null
}
# Sync with checksum verification
sync_with_checksum() {
local src="$1"
local dest="$2"
# Calculate local checksums
find "$src" -type f -exec md5sum {} \; > /tmp/local.md5
# Copy files
scp -r "$src"/* "$dest"/
# Calculate remote checksums
ssh "${dest%%:*}" "cd ${dest#*:} && md5sum *" > /tmp/remote.md5
# Compare
diff /tmp/local.md5 /tmp/remote.md5
}
Deployment Scripts
#!/bin/bash
# Deploy web application
deploy_app() {
local app_dir="$1"
local server="$2"
local deploy_dir="${3:-/var/www/html}"
echo "Packaging application..."
tar -czf /tmp/deploy.tar.gz -C "$app_dir" --exclude=.git --exclude=node_modules .
echo "Deploying to $server..."
scp /tmp/deploy.tar.gz "deploy@$server:/tmp/"
ssh "deploy@$server" "
cd $deploy_dir &&
sudo rm -rf * &&
sudo tar -xzf /tmp/deploy.tar.gz &&
sudo chown -R www-data:www-data . &&
sudo systemctl reload nginx
"
rm /tmp/deploy.tar.gz
echo "Deployment complete"
}
# Rollback function
rollback_deploy() {
local server="$1"
local backup_dir="${2:-/backups}"
# List available backups
ssh "deploy@$server" "ls -lt $backup_dir | head -20"
echo "Enter backup date (YYYYMMDD):"
read backup_date
ssh "deploy@$server" "
cd /var/www/html &&
sudo rm -rf * &&
sudo tar -xzf $backup_dir/backup_$backup_date.tar.gz &&
sudo systemctl reload nginx
"
}
4. Scripting with scp
Connection Testing
#!/bin/bash
# Test SCP connectivity
test_scp_connection() {
local host="$1"
local user="${2:-$USER}"
local port="${3:-22}"
local test_file="/tmp/scp_test_$$"
# Create test file
echo "test" > "$test_file"
# Try upload
if scp -P "$port" -q "$test_file" "$user@$host":/tmp/ 2>/dev/null; then
echo "✓ Upload successful"
# Try download
if scp -P "$port" -q "$user@$host:/tmp/$(basename "$test_file")" /dev/null 2>/dev/null; then
echo "✓ Download successful"
ssh "$user@$host" "rm /tmp/$(basename "$test_file")"
rm "$test_file"
return 0
fi
fi
echo "✗ SCP connection failed"
rm "$test_file"
return 1
}
# Check SCP prerequisites
check_scp_prereqs() {
local host="$1"
# Check SSH connectivity
if ! ssh -o ConnectTimeout=5 -q "$host" exit 2>/dev/null; then
echo "SSH connection failed"
return 1
fi
# Check if SCP is available
if ! ssh "$host" "which scp" >/dev/null 2>&1; then
echo "SCP not available on remote host"
return 1
fi
# Check disk space
space=$(ssh "$host" "df -h /tmp | tail -1 | awk '{print \$4}'")
echo "Remote /tmp has $space free"
return 0
}
Batch Processing
#!/bin/bash
# Copy multiple files with progress
batch_copy() {
local source_dir="$1"
local dest="$2"
local pattern="${3:-*}"
local log="/tmp/scp_batch_$(date +%s).log"
total=$(ls "$source_dir"/$pattern 2>/dev/null | wc -l)
count=0
for file in "$source_dir"/$pattern; do
if [ -f "$file" ]; then
count=$((count + 1))
echo "[$count/$total] Copying $(basename "$file")..."
if scp -q "$file" "$dest/" >> "$log" 2>&1; then
echo " ✓ Success"
else
echo " ✗ Failed" >> "$log"
fi
fi
done
echo "Batch copy complete. Log saved to $log"
}
# Parallel SCP transfers
parallel_scp() {
local files=("$@")
local dest="${files[-1]}"
unset 'files[-1]'
for file in "${files[@]}"; do
(
echo "Starting transfer of $(basename "$file")"
scp "$file" "$dest/"
echo "Completed $(basename "$file")"
) &
done
wait
echo "All transfers complete"
}
# Retry logic
scp_with_retry() {
local src="$1"
local dest="$2"
local max_attempts="${3:-3}"
local attempt=1
while [ $attempt -le $max_attempts ]; do
echo "Attempt $attempt of $max_attempts"
if scp "$src" "$dest"; then
echo "Transfer successful"
return 0
fi
attempt=$((attempt + 1))
[ $attempt -le $max_attempts ] && sleep 5
done
echo "Transfer failed after $max_attempts attempts"
return 1
}
Monitoring and Logging
#!/bin/bash
# Monitor SCP transfers
monitor_scp() {
local logfile="/var/log/scp_monitor.log"
# Monitor SCP processes
while true; do
scp_pids=$(pgrep -f "scp .*")
if [ -n "$scp_pids" ]; then
for pid in $scp_pids; do
# Get process info
cmd=$(ps -p "$pid" -o cmd=)
start=$(ps -p "$pid" -o lstart=)
# Get network stats (requires lsof)
if command -v lsof >/dev/null; then
conn=$(lsof -p "$pid" 2>/dev/null | grep TCP | head -1)
fi
echo "$(date): Active SCP: $cmd" >> "$logfile"
done
fi
sleep 10
done
}
# Log all SCP transfers
log_scp() {
local src="$1"
local dest="$2"
local log="/var/log/scp_transfers.log"
{
echo "=== SCP Transfer ==="
echo "Timestamp: $(date)"
echo "User: $(whoami)"
echo "Source: $src"
echo "Destination: $dest"
echo "Size: $(du -h "$src" 2>/dev/null | cut -f1)"
echo "==================="
} >> "$log"
scp "$src" "$dest"
if [ $? -eq 0 ]; then
echo "Status: SUCCESS" >> "$log"
else
echo "Status: FAILED" >> "$log"
fi
}
5. Advanced Techniques
Compression and Archiving
#!/bin/bash
# Compress on the fly
compress_and_send() {
local dir="$1"
local dest="$2"
tar -czf - "$dir" | ssh user@"$dest" "cat > archive_$(date +%Y%m%d).tar.gz"
}
# Receive and decompress
receive_and_extract() {
local remote="$1"
local remote_file="$2"
local local_dir="$3"
ssh user@"$remote" "cat $remote_file" | tar -xzf - -C "$local_dir"
}
# Split large files for transfer
split_and_send() {
local file="$1"
local dest="$2"
local chunk_size="${3:-100M}"
local tmp_dir="/tmp/split_$$"
mkdir -p "$tmp_dir"
# Split file
split -b "$chunk_size" "$file" "$tmp_dir/part_"
# Send parts
scp -r "$tmp_dir"/* "$dest/parts/"
# Reassemble command
echo "To reassemble: cat parts/* > $file" | ssh "$dest" "cat > reassemble.sh"
rm -rf "$tmp_dir"
}
# Resume interrupted transfer
resume_transfer() {
local file="$1"
local dest="$2"
# Check if partial transfer exists
if ssh "${dest%%:*}" "test -f ${dest#*:}.part"; then
remote_size=$(ssh "${dest%%:*}" "stat -c%s ${dest#*:}.part")
local_size=$(stat -c%s "$file")
if [ "$local_size" -gt "$remote_size" ]; then
echo "Resuming transfer..."
tail -c +$((remote_size + 1)) "$file" | \
ssh "${dest%%:*}" "cat >> ${dest#*:}.part"
# Rename when complete
ssh "${dest%%:*}" "mv ${dest#*:}.part ${dest#*:}"
fi
else
scp "$file" "$dest"
fi
}
Using Different Shells
#!/bin/bash
# Remote command execution with scp
scp_with_remote_cmd() {
local file="$1"
local remote="$2"
local remote_cmd="$3"
# Copy file
scp "$file" "$remote:/tmp/"
# Execute remote command
ssh "$remote" "$remote_cmd /tmp/$(basename "$file")"
}
# Check file integrity after transfer
verify_transfer() {
local src="$1"
local dest="$2"
# Calculate local checksum
local_sum=$(md5sum "$src" | cut -d' ' -f1)
# Copy file
scp "$src" "$dest"
# Calculate remote checksum
remote_sum=$(ssh "${dest%%:*}" "md5sum ${dest#*:}" | cut -d' ' -f1)
if [ "$local_sum" = "$remote_sum" ]; then
echo "✓ Transfer verified"
return 0
else
echo "✗ Checksum mismatch"
return 1
fi
}
# Atomic file replacement
atomic_replace() {
local new_file="$1"
local remote_target="$2"
local temp_file="${remote_target}.new"
# Copy to temp location
scp "$new_file" "$remote_target_host:$temp_file"
# Atomic rename
ssh "$remote_target_host" "mv $temp_file $remote_target"
}
6. Error Handling and Robustness
Comprehensive Error Handling
#!/bin/bash
# Robust SCP function
robust_scp() {
local src="$1"
local dest="$2"
local max_attempts="${3:-3}"
local timeout="${4:-30}"
# Validate source
if [ ! -e "$src" ]; then
echo "Error: Source does not exist: $src" >&2
return 1
fi
if [ ! -r "$src" ]; then
echo "Error: Source not readable: $src" >&2
return 1
fi
# Parse destination
dest_host="${dest%%:*}"
dest_path="${dest#*:}"
if [ "$dest_host" = "$dest" ]; then
# Local destination
dest_dir=$(dirname "$dest_path")
if [ ! -d "$dest_dir" ]; then
mkdir -p "$dest_dir"
fi
else
# Remote destination - check connectivity
if ! ssh -o ConnectTimeout=5 -o BatchMode=yes "$dest_host" exit 2>/dev/null; then
echo "Error: Cannot connect to $dest_host" >&2
return 1
fi
# Check remote directory exists
if ! ssh "$dest_host" "mkdir -p $(dirname "$dest_path")" 2>/dev/null; then
echo "Error: Cannot create remote directory" >&2
return 1
fi
fi
# Attempt transfer with retry
local attempt=1
while [ $attempt -le $max_attempts ]; do
echo "Attempt $attempt of $max_attempts"
if timeout "$timeout" scp -q "$src" "$dest" 2>/dev/null; then
echo "Transfer successful"
return 0
fi
echo "Transfer failed, retrying in 5 seconds..."
sleep 5
attempt=$((attempt + 1))
done
echo "Transfer failed after $max_attempts attempts" >&2
return 1
}
# Error classification
classify_scp_error() {
local exit_code=$1
local output="$2"
case $exit_code in
0)
echo "SUCCESS"
;;
1)
if echo "$output" | grep -q "Permission denied"; then
echo "AUTHENTICATION_FAILED"
elif echo "$output" | grep -q "No such file"; then
echo "FILE_NOT_FOUND"
else
echo "GENERAL_ERROR"
fi
;;
2)
echo "CONNECTION_FAILED"
;;
3)
echo "PROTOCOL_ERROR"
;;
*)
echo "UNKNOWN_ERROR"
;;
esac
}
Recovery Strategies
#!/bin/bash
# Resume failed transfer
resume_failed() {
local src="$1"
local dest="$2"
local state_file="/tmp/scp_resume_$$"
# Check if partial transfer exists
if ssh "${dest%%:*}" "test -f ${dest#*:}.part" 2>/dev/null; then
remote_size=$(ssh "${dest%%:*}" "stat -c%s ${dest#*:}.part")
local_size=$(stat -c%s "$src")
if [ "$local_size" -gt "$remote_size" ]; then
echo "Resuming from byte $remote_size"
dd if="$src" bs=1024 skip=$((remote_size / 1024)) 2>/dev/null | \
ssh "${dest%%:*}" "dd of=${dest#*:}.part bs=1024 seek=$((remote_size / 1024)) 2>/dev/null"
# Verify final size
final_size=$(ssh "${dest%%:*}" "stat -c%s ${dest#*:}.part")
if [ "$final_size" -eq "$local_size" ]; then
ssh "${dest%%:*}" "mv ${dest#*:}.part ${dest#*:}"
echo "Transfer completed"
fi
fi
else
# Start fresh transfer with part file
scp "$src" "${dest}.part" && \
ssh "${dest%%:*}" "mv ${dest#*:}.part ${dest#*:}"
fi
}
# Create recovery script
create_recovery_script() {
local files=("$@")
local recovery_script="/tmp/recover_scp_$$.sh"
{
echo "#!/bin/bash"
echo "# Recovery script for SCP transfers"
echo "# Generated: $(date)"
echo
echo "FAILED=()"
for file in "${files[@]}"; do
echo "if [ ! -f \"$file\" ]; then"
echo " echo \"Recovering $file...\""
echo " scp user@backup-server:backup/$(basename "$file") ."
echo " [ \$? -eq 0 ] || FAILED+=(\"$file\")"
echo "fi"
echo
done
echo "if [ \${#FAILED[@]} -gt 0 ]; then"
echo " echo \"Failed to recover: \${FAILED[@]}\""
echo "else"
echo " echo \"All files recovered successfully\""
echo "fi"
} > "$recovery_script"
chmod +x "$recovery_script"
echo "Recovery script created: $recovery_script"
}
7. Performance Optimization
Bandwidth Optimization
#!/bin/bash
# Optimize SCP for different networks
optimize_scp() {
local file="$1"
local dest="$2"
local network_type="${3:-auto}" # auto, lan, wifi, wan
case "$network_type" in
lan)
# Fast LAN - no compression, large window
scp -o IPQoS=throughput -o TCPKeepAlive=yes "$file" "$dest"
;;
wifi)
# WiFi - some compression, moderate window
scp -C -o IPQoS=reliability "$file" "$dest"
;;
wan)
# WAN/WAN - maximum compression, small window
scp -C -c aes128-ctr -o CompressionLevel=9 "$file" "$dest"
;;
*)
# Auto-detect
# Check RTT to destination
rtt=$(ping -c 1 "${dest%%:*}" 2>/dev/null | grep time= | cut -d= -f4)
if [ -n "$rtt" ] && [ "$(echo "$rtt < 5" | bc)" -eq 1 ]; then
optimize_scp "$file" "$dest" "lan"
elif [ -n "$rtt" ] && [ "$(echo "$rtt < 50" | bc)" -eq 1 ]; then
optimize_scp "$file" "$dest" "wifi"
else
optimize_scp "$file" "$dest" "wan"
fi
;;
esac
}
# Parallel chunked transfer
parallel_chunk() {
local file="$1"
local dest="$2"
local chunks="${3:-4}"
local tmp_dir="/tmp/scp_parallel_$$"
mkdir -p "$tmp_dir"
# Get file size
total=$(stat -c%s "$file")
chunk_size=$((total / chunks))
# Split and transfer in parallel
for ((i=0; i<chunks; i++)); do
start=$((i * chunk_size))
if [ $i -eq $((chunks - 1)) ]; then
end=$total
else
end=$(( (i + 1) * chunk_size ))
fi
(
dd if="$file" bs=1024 skip=$((start / 1024)) count=$(((end - start) / 1024)) 2>/dev/null | \
ssh "${dest%%:*}" "dd of=${dest#*:}.part$i bs=1024 seek=$((start / 1024)) 2>/dev/null"
) &
done
wait
# Reassemble
ssh "${dest%%:*}" "cat ${dest#*:}.part* > ${dest#*:} && rm ${dest#*:}.part*"
rm -rf "$tmp_dir"
}
Caching and Optimization
#!/bin/bash
# Cache SCP connections
setup_scp_cache() {
local host="$1"
# Setup persistent connection for multiple transfers
ssh -o ControlMaster=auto -o ControlPath=/tmp/ssh-%r@%h:%p -o ControlPersist=10m "$host" exit
# Use cached connection for SCP
scp_with_cache() {
scp -o ControlPath=/tmp/ssh-%r@%h:%p "$@"
}
}
# Batch transfer with connection reuse
batch_transfer() {
local host="$1"
shift
local files=("$@")
# Setup master connection
ssh -fN -o ControlMaster=yes -o ControlPath=/tmp/ssh-%r@%h:%p "$host"
# Transfer all files using same connection
for file in "${files[@]}"; do
scp -o ControlPath=/tmp/ssh-%r@%h:%p "$file" "$host:"
done
# Close master connection
ssh -O exit -o ControlPath=/tmp/ssh-%r@%h:%p "$host"
}
8. Security Considerations
Secure Practices
#!/bin/bash
# Secure SCP wrapper
secure_scp() {
local src="$1"
local dest="$2"
# Verify host key
ssh-keygen -F "${dest%%:*}" >/dev/null || {
echo "Warning: Host not known"
ssh-keyscan -H "${dest%%:*}" 2>/dev/null
}
# Use strong cipher
scp -c aes256-ctr -o MACs=hmac-sha2-512 "$src" "$dest"
}
# Encrypt before transfer
encrypt_before_scp() {
local file="$1"
local dest="$2"
local password="$3"
# Encrypt
openssl enc -aes-256-cbc -salt -in "$file" -out "$file.enc" -k "$password"
# Transfer
scp "$file.enc" "$dest"
# Cleanup
rm "$file.enc"
echo "To decrypt: openssl enc -d -aes-256-cbc -in $file.enc -out $file -k password"
}
# Sign and verify
sign_and_send() {
local file="$1"
local dest="$2"
local key="$HOME/.ssh/id_rsa"
# Create signature
openssl dgst -sha256 -sign "$key" -out "$file.sig" "$file"
# Send both file and signature
scp "$file" "$file.sig" "$dest"
# Remote verification command
echo "To verify: openssl dgst -sha256 -verify <(ssh-keygen -y -f $key.pub) -signature $file.sig $file"
}
Access Control
#!/bin/bash
# Restricted SCP for specific users
restricted_scp() {
local user="$1"
local src="$2"
local dest="$3"
# Check user permissions
if ! grep -q "^$user:" /etc/passwd; then
echo "User $user does not exist"
return 1
fi
# Check allowed directories
allowed_dirs=("/home/$user" "/backup/$user" "/tmp")
dest_dir=$(dirname "$dest")
allowed=0
for dir in "${allowed_dirs[@]}"; do
if [[ "$dest_dir" == "$dir"* ]]; then
allowed=1
break
fi
done
if [ $allowed -eq 0 ]; then
echo "Destination not allowed"
return 1
fi
scp "$src" "$dest"
}
# SCP with IP whitelist
whitelist_scp() {
local src="$1"
local dest="$2"
local client_ip="${SSH_CLIENT%% *}"
# Check if IP is whitelisted
if ! grep -q "^$client_ip$" /etc/scp_whitelist 2>/dev/null; then
logger -t scp "Blocked SCP attempt from $client_ip"
echo "Access denied"
return 1
fi
scp "$src" "$dest"
}
9. Integration with Other Tools
With rsync for better sync
#!/bin/bash
# Hybrid scp/rsync approach
smart_sync() {
local src="$1"
local dest="$2"
# Check if rsync is available on both ends
if command -v rsync >/dev/null && ssh "${dest%%:*}" "command -v rsync" >/dev/null; then
echo "Using rsync for efficient sync"
rsync -avz -e ssh "$src" "$dest"
else
echo "Using scp (rsync not available)"
scp -r "$src" "$dest"
fi
}
# Scp with tar for better performance
tar_scp() {
local src="$1"
local dest="$2"
tar -cf - "$src" | ssh "${dest%%:*}" "tar -xf - -C ${dest#*:}"
}
# Scp with find for selective copy
find_scp() {
local src_dir="$1"
local dest="$2"
local pattern="$3"
find "$src_dir" -name "$pattern" -type f -exec scp {} "$dest" \;
}
With monitoring tools
#!/bin/bash
# Monitor with pv
scp_with_progress() {
local src="$1"
local dest="$2"
# Get file size
size=$(stat -c%s "$src")
# Copy with progress bar
pv -s "$size" "$src" | ssh "${dest%%:*}" "cat > ${dest#*:}"
}
# Monitor with custom progress
scp_progress() {
local src="$1"
local dest="$2"
scp "$src" "$dest" &
pid=$!
while kill -0 $pid 2>/dev/null; do
# Get process size (simplified)
if [ -f "$dest" ]; then
transferred=$(stat -c%s "$dest" 2>/dev/null || echo 0)
total=$(stat -c%s "$src")
percent=$((transferred * 100 / total))
echo -ne "Progress: $percent% ($transferred/$total bytes)\r"
fi
sleep 1
done
echo
}
# Log to systemd journal
scp_journal() {
local src="$1"
local dest="$2"
logger -t scp "Starting transfer: $src -> $dest"
if scp "$src" "$dest"; then
logger -t scp "Transfer complete: $src -> $dest"
else
logger -t scp "Transfer failed: $src -> $dest"
fi
}
10. Real-World Applications
Automated Backup System
#!/bin/bash
# Comprehensive backup system
backup_system() {
local config="/etc/backup.conf"
local log="/var/log/backup.log"
# Load configuration
source "$config"
{
echo "=== Backup Started: $(date) ==="
for item in "${BACKUP_ITEMS[@]}"; do
echo "Backing up: $item"
# Create backup name
backup_name="$(basename "$item")_$(date +%Y%m%d).tar.gz"
# Compress
tar -czf "/tmp/$backup_name" "$item"
# Transfer
if scp "/tmp/$backup_name" "${BACKUP_SERVER}:${BACKUP_DIR}/"; then
echo "✓ $item backed up successfully"
rm "/tmp/$backup_name"
# Rotate old backups
ssh "${BACKUP_SERVER}" "cd ${BACKUP_DIR} && ls -t $(basename "$item")_* | tail -n +${BACKUP_ROTATE} | xargs rm -f"
else
echo "✗ Failed to backup $item"
fi
done
echo "=== Backup Complete: $(date) ==="
} | tee -a "$log"
}
# Example config file
cat > /etc/backup.conf << EOF
BACKUP_SERVER="user@backup-server"
BACKUP_DIR="/backups"
BACKUP_ROTATE=7
BACKUP_ITEMS=(
"/etc"
"/home"
"/var/www"
"/var/log"
)
EOF
Remote Deployment System
#!/bin/bash
# Multi-server deployment
deploy_to_servers() {
local app_name="$1"
local version="$2"
local servers=("${@:3}")
local deploy_log="/tmp/deploy_${app_name}_${version}.log"
echo "Deploying $appName version $version to ${#servers[@]} servers"
# Create deployment package
package_name="${app_name}_${version}.tar.gz"
tar -czf "/tmp/$package_name" -C "/builds/$appName" .
# Deploy to each server
for server in "${servers[@]}"; do
echo -n "Deploying to $server... "
{
# Copy package
scp "/tmp/$package_name" "$server:/tmp/"
# Deploy on remote
ssh "$server" "
cd /opt/apps &&
sudo tar -xzf /tmp/$package_name &&
sudo chown -R app:app $app_name &&
sudo systemctl restart $app_name
"
} >> "$deploy_log" 2>&1
if [ $? -eq 0 ]; then
echo "✓"
else
echo "✗"
fi
done
rm "/tmp/$package_name"
echo "Deployment log: $deploy_log"
}
# Rolling deployment
rolling_deploy() {
local servers=("$@")
for server in "${servers[@]}"; do
echo "Deploying to $server..."
# Take out of load balancer
ssh "$server" "sudo systemctl stop app"
# Deploy
scp app.tar.gz "$server:/tmp/"
ssh "$server" "
cd /opt/app &&
sudo tar -xzf /tmp/app.tar.gz &&
sudo systemctl start app
"
# Wait for health check
sleep 10
# Verify
if curl -f "http://$server:8080/health"; then
echo "✓ $server deployed successfully"
else
echo "✗ $server deployment failed - rolling back"
# Rollback logic here
break
fi
done
}
Log Collection System
#!/bin/bash
# Collect logs from multiple servers
collect_logs() {
local date=$(date +%Y%m%d)
local log_dir="/logs/collect_$date"
mkdir -p "$log_dir"
# Read server list
while read server; do
echo "Collecting logs from $server"
# Collect syslog
scp "$server:/var/log/syslog" "$log_dir/${server}_syslog" &
# Collect application logs
scp "$server:/var/log/app/*.log" "$log_dir/${server}_app/" &
# Collect auth logs
scp "$server:/var/log/auth.log" "$log_dir/${server}_auth.log" &
# Wait for background jobs to complete
wait
echo "✓ Logs collected from $server"
done < /etc/log_collector/servers.txt
# Create archive
tar -czf "/backups/logs_$date.tar.gz" "$log_dir"
rm -rf "$log_dir"
echo "Log collection complete: /backups/logs_$date.tar.gz"
}
# Real-time log aggregation
aggregate_logs() {
local servers=("$@")
local logfile="/var/log/aggregated.log"
while true; do
for server in "${servers[@]}"; do
# Get last 10 lines from each server
ssh "$server" "tail -10 /var/log/app.log" | \
sed "s/^/[$server] /" >> "$logfile"
done
sleep 5
done
}
11. Best Practices and Tips
Security Best Practices
#!/bin/bash # 1. Use key-based authentication instead of passwords ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 ssh-copy-id user@remote-server # 2. Use specific ciphers scp -c [email protected] file.txt user@host: # 3. Limit SCP in sshd_config # In /etc/ssh/sshd_config: # Subsystem sftp internal-sftp # Match User backup # ChrootDirectory /backup # ForceCommand internal-sftp # X11Forwarding no # AllowTcpForwarding no # 4. Use bastion hosts for complex networks scp -o ProxyJump=bastion-server user@internal-server:/file.txt . # 5. Log all transfers # In /etc/ssh/sshd_config: # Subsystem sftp /usr/lib/openssh/sftp-server -l INFO
Performance Tips
#!/bin/bash # 1. Use compression for text files scp -C large_text_file.log user@host: # 2. Don't compress already compressed files if [[ ! "$file" =~ \.(gz|bz2|zip|jpg|png)$ ]]; then scp -C "$file" user@host: else scp "$file" user@host: fi # 3. Use appropriate cipher for speed # Fastest: aes128-ctr scp -c aes128-ctr file.txt user@host: # 4. Increase window size for high latency scp -o IPQoS=throughput -o TCPKeepAlive=yes file.txt user@host: # 5. Use ControlMaster for multiple files ssh -fN -o ControlMaster=yes -o ControlPath=/tmp/ssh-%r@%h:%p user@host scp -o ControlPath=/tmp/ssh-%r@%h:%p file1.txt user@host: scp -o ControlPath=/tmp/ssh-%r@%h:%p file2.txt user@host: ssh -O exit -o ControlPath=/tmp/ssh-%r@%h:%p user@host
Scripting Best Practices
#!/bin/bash
# 1. Always check return codes
if ! scp file.txt user@host:/destination/; then
echo "SCP failed"
exit 1
fi
# 2. Use timeout to prevent hanging
timeout 60 scp largefile.txt user@host: || echo "SCP timed out"
# 3. Quote variables to handle spaces
scp "$file" "user@host:$dest"
# 4. Use arrays for multiple files
files=("file1.txt" "file with spaces.txt" "file3.txt")
scp "${files[@]}" user@host:
# 5. Create temporary files safely
temp_file=$(mktemp)
scp "user@host:file.txt" "$temp_file"
# 6. Clean up temp files
trap 'rm -f "$temp_file"' EXIT
# 7. Use batch mode in scripts
scp -o BatchMode=yes file.txt user@host:
# 8. Set connection timeout
scp -o ConnectTimeout=10 file.txt user@host:
Common Pitfalls and Solutions
#!/bin/bash # Pitfall 1: Host key verification # Solution: Add host key automatically ssh-keyscan -H hostname >> ~/.ssh/known_hosts # Pitfall 2: Permission denied # Solution: Check permissions chmod 600 ~/.ssh/id_rsa chmod 700 ~/.ssh # Pitfall 3: Path expansion issues # Solution: Use full paths or proper quoting scp "user@host:'/path/with spaces/file.txt'" . # Pitfall 4: Symlinks not preserved # Solution: Use tar to preserve links tar -chf - symlink_dir | ssh user@host "tar -xf - -C /dest" # Pitfall 5: Interrupted transfers # Solution: Use rsync or implement resume rsync -avP -e ssh file.txt user@host: # Pitfall 6: Firewall blocking # Solution: Test connectivity first nc -zv host 22 && scp file.txt user@host:
12. Command Summary and Cheat Sheet
Quick Reference
# Basic commands scp file.txt user@host: # Copy to remote home scp user@host:file.txt . # Copy from remote to local scp -r dir/ user@host:backup/ # Copy directory recursively scp -p file.txt user@host: # Preserve attributes scp -C file.txt user@host: # Enable compression scp -P 2222 file.txt user@host: # Use specific port # Advanced options scp -l 1000 file.txt user@host: # Limit bandwidth scp -c aes256-ctr file.txt user@host: # Use specific cipher scp -o ConnectTimeout=10 file.txt user@host: # Set timeout scp -o BatchMode=yes file.txt user@host: # Batch mode (no password) scp -v file.txt user@host: # Verbose output scp -q file.txt user@host: # Quiet mode # Multiple files scp file1.txt file2.txt file3.txt user@host: scp *.log user@host:/logs/ # With different usernames scp file.txt username@host:/path/ # Between remote hosts scp user1@host1:file.txt user2@host2:
Common Options
| Option | Description | Example |
|---|---|---|
-P | Port number | scp -P 2222 file.txt host: |
-p | Preserve attributes | scp -p file.txt host: |
-r | Recursive | scp -r dir/ host: |
-C | Enable compression | scp -C bigfile.tar host: |
-l | Limit bandwidth | scp -l 1000 file.txt host: |
-i | Identity file | scp -i ~/.ssh/key file.txt host: |
-c | Cipher specification | scp -c aes256-ctr file.txt host: |
-o | SSH options | scp -o ConnectTimeout=10 file.txt host: |
-v | Verbose | scp -v file.txt host: |
-q | Quiet | scp -q file.txt host: |
Exit Codes
# scp exit codes and meanings # 0 = Success # 1 = General error (file not found, permission denied) # 2 = Connection failed # 3 = Protocol error # 4 = No matching cipher/MAC scp file.txt user@host: case $? in 0) echo "Success" ;; 1) echo "File error or permission denied" ;; 2) echo "Connection failed" ;; 3) echo "Protocol error" ;; 4) echo "Cipher mismatch" ;; esac
Conclusion
The scp command is an essential tool for secure file transfer:
Key Takeaways
- Security: All transfers are encrypted using SSH
- Simplicity: Simple syntax similar to cp command
- Flexibility: Supports single files, directories, and multiple files
- Integration: Works seamlessly with SSH authentication
- Scripting: Easy to incorporate into automation scripts
Best Practices Summary
| Scenario | Recommendation |
|---|---|
| Single file | scp file.txt user@host: |
| Directory | scp -r dir/ user@host: |
| Large files | Use -C compression and -l rate limiting |
| Many small files | Use tar + ssh or rsync |
| Automated scripts | Use key-based auth and -o BatchMode=yes |
| Debugging | Use -v for verbose output |
| Security | Use strong ciphers and key-based auth |
| Reliability | Implement retry logic and error checking |
When to Use Alternatives
- rsync: Better for syncing directories, only transfers differences
- sftp: Interactive file transfers, better for manual operations
- ftp: Unencrypted, only for internal/trusted networks
- curl: HTTP/HTTPS transfers, API interactions
- nc: Simple unencrypted transfers, debugging
The scp command remains a reliable choice for secure file transfers, especially when simplicity and security are paramount.