Introduction to sed
sed (Stream Editor) is a powerful text processing tool in Unix/Linux that performs basic text transformations on an input stream (a file or input from a pipeline). It's designed for efficient, non-interactive editing of text files using simple programming constructs.
Key Concepts
- Stream Processing: Processes text line by line
- Pattern Matching: Uses regular expressions for pattern matching
- In-Place Editing: Can modify files directly
- Scripting: Supports complex editing scripts
- Pipeline Integration: Works seamlessly with other Unix commands
1. Basic Syntax and Structure
Command Structure
# Basic syntax sed [options] 'command' file.txt sed [options] -f script.sed file.txt # Common patterns sed 's/pattern/replacement/' file.txt # Substitute sed '/pattern/d' file.txt # Delete sed -n '/pattern/p' file.txt # Print matching lines sed -i 's/old/new/g' file.txt # In-place edit # Example: Simple substitution echo "hello world" | sed 's/world/universe/' # Output: hello universe # Multiple commands sed -e 's/foo/bar/' -e 's/baz/qux/' file.txt # Using semicolons sed 's/foo/bar/; s/baz/qux/' file.txt
Command Options
# Common options sed -n 'p' file.txt # Suppress automatic printing (-n) sed -i 's/old/new/' file.txt # Edit files in-place (-i) sed -i.bak 's/old/new/' file # Create backup before editing sed -e 'command' file.txt # Add multiple commands (-e) sed -f script.sed file.txt # Read commands from file (-f) sed -r 'pattern' file.txt # Use extended regex (-r or -E) # Example with backup sed -i.bak 's/error/warning/' config.txt # Creates config.txt.bak before modifying config.txt # Multiple expressions sed -e 's/foo/bar/' -e 's/[0-9]//g' data.txt
2. Basic Operations
Substitution (s Command)
# Basic substitution sed 's/old/new/' file.txt # Replace first occurrence per line sed 's/old/new/g' file.txt # Replace all occurrences (global) sed 's/old/new/2' file.txt # Replace second occurrence only sed 's/old/new/gi' file.txt # Case-insensitive global replace # Using different delimiters sed 's|/usr/local|/opt|' paths.txt # Use | as delimiter sed 's:/home:/users:' passwd # Use : as delimiter sed 's_old_new_' file.txt # Use _ as delimiter # Examples echo "foo foo foo" | sed 's/foo/bar/' # bar foo foo echo "foo foo foo" | sed 's/foo/bar/2' # foo bar foo echo "foo foo foo" | sed 's/foo/bar/g' # bar bar bar # Capture groups and backreferences echo "abc123" | sed 's/\([a-z]*\)\([0-9]*\)/\2\1/' # 123abc echo "John Doe" | sed 's/\(.*\) \(.*\)/\2, \1/' # Doe, John # Ampersand as matched pattern echo "hello world" | sed 's/.*/(&)/' # (hello world) echo "123-456-7890" | sed 's/[0-9]\+/(&)/g' # (123)-(456)-(7890)
Delete (d Command)
# Delete lines sed '3d' file.txt # Delete line 3 sed '5,10d' file.txt # Delete lines 5-10 sed '/pattern/d' file.txt # Delete lines matching pattern sed '/^#/d' file.txt # Delete comments (lines starting with #) sed '/^$/d' file.txt # Delete empty lines sed '/start/,/end/d' file.txt # Delete range between patterns # Complex deletions sed '1,5d' data.txt # Delete first 5 lines sed '$d' file.txt # Delete last line sed '/^[[:space:]]*$/d' file # Delete lines with only whitespace sed '/ERROR/,+3d' log.txt # Delete line with ERROR and next 3 lines # Examples cat > test.txt << EOF Line 1 Line 2 Line 3 Line 4 Line 5 EOF sed '2,4d' test.txt # Output: Line 1, Line 5 sed '/2/,/4/d' test.txt # Delete lines containing 2 through 4
Print (p Command)
# Print specific lines
sed -n '5p' file.txt # Print line 5
sed -n '10,20p' file.txt # Print lines 10-20
sed -n '/pattern/p' file.txt # Print lines matching pattern
sed -n '/start/,/end/p' file # Print range between patterns
# Print with line numbers
sed -n '/error/{=;p}' log.txt # Print line number and matching line
sed -n '/error/{=;p}' log.txt | sed 'N;s/\n/ /' # Format: "number line"
# Multiple print conditions
sed -n '/ERROR/p; /WARNING/p' log.txt # Print ERROR or WARNING lines
sed -n '1,5p; /important/p' file.txt # Print first 5 lines and important lines
# Examples
cat data.txt | sed -n '/^[0-9]/p' # Print lines starting with number
sed -n '/[A-Z]\{5,\}/p' file.txt # Print lines with 5+ uppercase letters
3. Address Ranges and Patterns
Line Addressing
# Single line addresses sed '5 s/old/new/' file.txt # Replace on line 5 only sed '1,10 s/old/new/g' file.txt # Replace on lines 1-10 sed '$ s/old/new/' file.txt # Replace on last line only # Pattern addresses sed '/^Start/,/^End/ s/old/new/' file.txt # Replace within range sed '/section/,/endsection/ d' file.txt # Delete section sed '/BEGIN/,/END/!d' file.txt # Keep only between BEGIN and END # Address negation sed '/pattern/!s/old/new/' file.txt # Replace where pattern doesn't match sed '5,10!d' file.txt # Delete all except lines 5-10 # Step addressing (GNU sed) sed '0~2 s/old/new/' file.txt # Replace on even lines sed '1~2 s/old/new/' file.txt # Replace on odd lines sed '0~5d' file.txt # Delete every 5th line # Examples sed '/debug/,/enddebug/ s/^/# /' config.c # Comment out debug section sed '/^[[:space:]]*$/d' file.txt # Delete empty lines
Regular Expressions in sed
# Basic regex (BRE)
sed 's/^[[:space:]]*//' file.txt # Remove leading whitespace
sed 's/[[:space:]]*$//' file.txt # Remove trailing whitespace
sed 's/[[:alpha:]]\+/WORD/g' file.txt # Replace words with WORD
# Extended regex with -r or -E
sed -r 's/[0-9]{3}-[0-9]{4}/(xxx)/' phones.txt
sed -E 's/[[:alnum:]_]+@[[:alnum:]_]+\.[a-z]{2,3}/[EMAIL]/' emails.txt
# Common regex patterns
sed -n '/^#/!p' config.txt # Print non-comment lines
sed '/^$/d' file.txt # Remove empty lines
sed 's/[[:punct:]]//g' file.txt # Remove punctuation
sed 's/[[:upper:]]/\L&/g' file.txt # Convert to lowercase
sed 's/[[:lower:]]/\U&/g' file.txt # Convert to uppercase
# Complex regex examples
sed -E 's/\b([0-9]{1,3}\.){3}[0-9]{1,3}\b/IP/' file # Replace IP addresses
sed -E 's/^([^:]+):.*/\1/' passwd # Extract usernames
sed -E 's/<[^>]+>//g' html.html # Remove HTML tags
4. Advanced Commands
Transform (y Command)
# Character translation sed 'y/abc/xyz/' file.txt # a->x, b->y, c->z sed 'y/[A-Z]/[a-z]/' file.txt # Convert uppercase to lowercase sed 'y/ \t/__/' file.txt # Convert spaces/tabs to underscores # Examples echo "hello world" | sed 'y/helo/HELO/' # HELLO wOrLd echo "123-456-7890" | sed 'y/-/ /' # 123 456 7890 # Note: y does character-by-character translation, not patterns sed 'y/123456789/987654321/' numbers.txt # Reverse digits
Hold Space and Pattern Space
# Hold space commands
# h - copy pattern space to hold space
# H - append pattern space to hold space
# g - copy hold space to pattern space
# G - append hold space to pattern space
# x - exchange pattern and hold spaces
# Reverse lines of a file (tac equivalent)
sed -n '1!G;h;$p' file.txt
# Double-space a file
sed 'G' file.txt
# Join lines
sed 'N;s/\n/ /' file.txt # Join first two lines
sed 'N;N;s/\n/ /g' file # Join every three lines
# Advanced examples
# Print even lines
sed -n 'n;p' file.txt
# Print odd lines
sed -n 'p;n' file.txt
# Remove duplicate consecutive lines
sed '$!N; /^\(.*\)\n\1$/!P; D' file.txt
# Extract text between markers
sed -n '/START/{:a;n;/END/q;p;ba}' file.txt
Branching and Conditional Operations
# Branching commands
# :label - define a label
# b label - branch to label
# t label - branch if substitution succeeded
# Example: Multiple replacements
sed ':a;s/foo/bar/;ta' file.txt # Repeat substitution until no more changes
# Conditional processing
sed '/error/{
s/error/ERROR/
s/warning/WARNING/
}' log.txt
# Complex example - convert to title case
sed 's/.*/\L&/; s/[a-z]*/\u&/g' file.txt
# Loop until no more changes
sed ':start; s/[0-9][0-9]/&/; t start' numbers.txt
5. In-Place Editing
Basic In-Place Operations
# Edit file in-place (GNU sed) sed -i 's/old/new/g' file.txt # Create backup before editing sed -i.bak 's/old/new/g' file.txt # Multiple files sed -i 's/old/new/g' file1.txt file2.txt file3.txt # With backup extension sed -i'.original' 's/old/new/g' config.txt # Examples # Replace all instances of "foo" with "bar" in all .txt files sed -i 's/foo/bar/g' *.txt # Remove trailing whitespace from multiple files sed -i 's/[[:space:]]*$//' *.c *.h # Add header to files sed -i '1i # This file was modified' *.txt
Safe In-Place Editing
#!/bin/bash
# Safe in-place edit with backup and verification
safe_sed() {
local file="$1"
local backup="${file}.bak.$(date +%Y%m%d-%H%M%S)"
shift
# Create backup
cp "$file" "$backup"
# Perform edit
sed -i "$@" "$file"
# Ask for verification
echo "Changes made. Check $file"
echo "Backup saved at $backup"
echo -n "Keep changes? (y/n): "
read answer
if [ "$answer" != "y" ]; then
cp "$backup" "$file"
echo "Restored from backup"
fi
}
# Edit with diff preview
preview_sed() {
local file="$1"
shift
cp "$file" /tmp/sed_preview
sed -i /tmp/sed_preview "$@"
echo "=== Changes ==="
diff -u "$file" /tmp/sed_preview
echo "==============="
echo -n "Apply changes? (y/n): "
read answer
if [ "$answer" = "y" ]; then
cp /tmp/sed_preview "$file"
fi
rm /tmp/sed_preview
}
6. Multi-line Operations
Working with Multiple Lines
# Join lines
sed ':a;N;$!ba;s/\n/ /g' file.txt # Join all lines
sed 'N;N;s/\n/ /g' file.txt # Join three lines at a time
# Process paragraphs
sed -n '/^$/{p;d}; H; $ {x;s/\n/ /g;p}' file.txt # Join paragraph lines
# Multi-line pattern matching
sed '/start/,/end/ {
/start/d
/end/d
s/.*/CHANGED/
}' file.txt
# Examples
# Convert CSV to single line
sed ':a;N;$!ba;s/\n/,/g' data.csv
# Wrap text at column 80
sed 's/.\{80\}/&\n/g' longlines.txt
# Remove HTML tags across lines
sed ':a;N;$!ba;s/<[^>]*>//g' html.txt
Complex Multi-line Examples
#!/bin/bash
# Extract function definitions from C file
extract_functions() {
sed -n '
/^[a-zA-Z_][a-zA-Z0-9_]* *(.*)/ {
:func
N
/\n[{}]/!b func
p
}
' "$1"
}
# Join lines ending with backslash
join_continuations() {
sed ':a;/\\$/N;s/\\\n//;ta' "$1"
}
# Wrap long lines
wrap_lines() {
local width="${1:-80}"
shift
sed "s/.\{$width\}/&\n/g" "$@"
}
# Number paragraphs
number_paragraphs() {
sed '/^$/ {
x
s/^$/Paragraph 0/
s/Paragraph \([0-9]*\)/Paragraph \1/
s/.*/&/
x
}
/^[^ ]/ {
H
x
s/^\(.*\)\n\(.*\)/\2 \1/
s/Paragraph \([0-9]*\)/\1. /
s/\([0-9]*\. \)\(.*\)/\1\2/
s/^[0-9]*\. //
}' "$1"
}
7. Practical Examples
Text Processing
#!/bin/bash
# Remove comments and blank lines
clean_config() {
sed -e 's/#.*$//' -e '/^$/d' "$1"
}
# Extract email addresses
extract_emails() {
sed -n 's/.*\([a-zA-Z0-9._%+-]\+@[a-zA-Z0-9.-]\+\.[a-zA-Z]\{2,\}\).*/\1/p' "$1"
}
# Convert to title case
title_case() {
sed 's/.*/\L&/; s/[a-z]*/\u&/g' "$1"
}
# Add line numbers
add_line_numbers() {
sed = "$1" | sed 'N;s/\n/ /'
}
# Format JSON (simple)
format_json() {
sed 's/[{,]/&\n /g; s/}/&\n/g' "$1"
}
# URL decode
urldecode() {
sed 's/%/\\x/g' "$1" | while read -r line; do
echo -e "$line"
done
}
# Examples usage
# clean_config /etc/config.ini
# extract_emails contacts.txt
# title_case names.txt
File and Directory Operations
#!/bin/bash
# Rename files in directory (dry run)
rename_files() {
local pattern="$1"
local replacement="$2"
for file in *; do
newname=$(echo "$file" | sed "s/$pattern/$replacement/")
if [ "$newname" != "$file" ]; then
echo "mv '$file' '$newname'"
fi
done
}
# Batch rename with sed
batch_rename() {
local pattern="$1"
local replacement="$2"
shift 2
for file in "$@"; do
newname=$(echo "$file" | sed "s/$pattern/$replacement/")
if [ "$newname" != "$file" ]; then
mv -v "$file" "$newname"
fi
done
}
# Update file extensions
change_extension() {
local old_ext="$1"
local new_ext="$2"
for file in *."$old_ext"; do
[ -f "$file" ] || continue
base=$(echo "$file" | sed "s/\.$old_ext$//")
mv -v "$file" "$base.$new_ext"
done
}
Log Processing
#!/bin/bash
# Filter log by date range
filter_log_by_date() {
local start_date="$1"
local end_date="$2"
local logfile="$3"
sed -n "/^$start_date/,/^$end_date/p" "$logfile"
}
# Extract error messages with context
extract_errors() {
local logfile="$1"
local context="${2:-2}"
sed -n "
/ERROR/ {
=
N
N
p
}
" "$logfile" | sed 'N;s/\n/ /g'
}
# Summarize log levels
log_summary() {
sed -n 's/.*\(ERROR\|WARN\|INFO\).*/\1/p' "$1" | sort | uniq -c
}
# Remove timestamp prefix
remove_timestamp() {
sed 's/^\[[0-9-]* [0-9:]*\] //' "$1"
}
# Format log for better readability
format_log() {
sed 's/\[\([^]]*\)\]/\n[\1]\n /g' "$1" | sed '/^$/d'
}
CSV Processing
#!/bin/bash
# Extract specific column
extract_column() {
local col="$1"
local file="$2"
sed "s/[^,]*\(,[^,]*\)\{$((col-1))\},//" "$file"
}
# Change delimiter
change_delimiter() {
local old="$1"
local new="$2"
local file="$3"
sed "s/$old/$new/g" "$file"
}
# Remove header
remove_header() {
sed '1d' "$1"
}
# Add header
add_header() {
local header="$1"
local file="$2"
sed "1i $header" "$file"
}
# Filter rows by value in column
filter_by_column() {
local col="$1"
local value="$2"
local file="$3"
sed -n "/\([^,]*,\?\)\{$((col-1))\}$value/p" "$file"
}
# Examples
# extract_column 3 data.csv
# change_delimiter ',' '|' data.csv
8. sed Scripts
Creating sed Scripts
#!/bin/bash # Create a sed script file cat > cleanup.sed << 'EOF' # Remove comments s/#.*$// # Remove empty lines /^$/d # Trim whitespace s/^[[:space:]]*// s/[[:space:]]*$// # Normalize spaces s/[[:space:]]\+/ /g # Convert to lowercase s/[A-Z]/\L&/g EOF # Use the script sed -f cleanup.sed input.txt > output.txt # Another example: markdown to plain text cat > md2txt.sed << 'EOF' # Remove headers s/^#\+ //g # Remove emphasis s/\*\*\([^*]*\)\*\*/\1/g s/\*\([^*]*\)\*/\1/g # Remove links s/\[\([^]]*\)\]([^)]*)/\1/g # Remove code blocks /^```/,/^```/d # Remove images s/!\[[^]]*\]([^)]*)//g # Convert lists s/^[*-] /• /g EOF
Complex sed Script Example
#!/bin/bash
# HTML to text converter
cat > html2text.sed << 'EOF'
#!/usr/bin/sed -f
# Remove script and style sections
/<script/,/<\/script>/d
/<style/,/<\/style>/d
# Remove HTML tags
s/<[^>]*>//g
# Replace common entities
s/ / /g
s/</</g
s/>/>/g
s/&/\&/g
s/"/"/g
# Handle multiple blank lines
/^$/{
N
/^\n$/d
}
# Trim lines
s/^[[:space:]]*//
s/[[:space:]]*$//
# Join lines that don't end with punctuation
:join
N
/\n[^A-Z]/s/\n/ /
t join
# Add paragraph breaks
s/\. /. \n\n/g
EOF
# Make executable and use
chmod +x html2text.sed
./html2text.sed webpage.html > webpage.txt
9. Performance Optimization
Efficient sed Usage
#!/bin/bash # Use single sed command instead of multiple # Bad: multiple processes cat file.txt | sed 's/foo/bar/' | sed 's/baz/qux/' > output.txt # Good: single sed with multiple commands sed 's/foo/bar/; s/baz/qux/' file.txt > output.txt # Use sed -e for multiple expressions sed -e 's/foo/bar/' -e 's/baz/qux/' file.txt # Avoid unnecessary pipes # Bad: useless use of cat cat file.txt | sed 's/foo/bar/' # Good: sed reads file directly sed 's/foo/bar/' file.txt # Use address ranges to limit processing sed '1000,2000 s/foo/bar/g' largefile.txt # Only process relevant section # Exit early when done sed '/stop/q' file.txt # Stop processing after finding "stop" # Performance comparison time sed 's/foo/bar/g' largefile.txt > /dev/null time sed -n 's/foo/bar/gp' largefile.txt > /dev/null # Usually slower
Handling Large Files
#!/bin/bash
# Process large files in chunks
process_large_file() {
local file="$1"
local chunk_size=10000
split -l $chunk_size "$file" chunk_
for chunk in chunk_*; do
sed -i 's/foo/bar/g' "$chunk" &
# Process in parallel
[ $(jobs | wc -l) -ge 4 ] && wait
done
wait
cat chunk_* > "${file}.processed"
rm chunk_*
}
# Stream processing for very large files
stream_process() {
while IFS= read -r line; do
echo "$line" | sed 's/foo/bar/'
done < "$1"
}
# Use sed with buffering
sed -u 's/foo/bar/' largefile.txt # Unbuffered output
# Monitor progress with pv
pv largefile.txt | sed 's/foo/bar/' > output.txt
10. Error Handling and Debugging
Debugging sed Scripts
#!/bin/bash
# Debug with line numbering
sed -n '/pattern/{
=
p
}' file.txt
# Trace sed execution (GNU sed)
sed --debug 's/foo/bar/' file.txt
# Validate sed script
validate_sed() {
local script="$1"
local testfile="/tmp/sed_test.$$"
echo "test line" > "$testfile"
if sed -f "$script" "$testfile" >/dev/null 2>&1; then
echo "Script syntax OK"
else
echo "Script has errors"
sed -f "$script" "$testfile" 2>&1
fi
rm "$testfile"
}
# Dry run (show what would be changed)
sed_dry_run() {
local file="$1"
shift
diff -u "$file" <(sed "$@" "$file")
}
# Verbose mode for complex scripts
sed_verbose() {
local script="$1"
shift
echo "=== Input ==="
cat "$1"
echo "=== Processing ==="
sed -n "$script" "$1"
echo "=== Output ==="
sed "$script" "$1"
}
Common Error Patterns
#!/bin/bash # Error: Unescaped delimiter in pattern # sed 's/foo/bar/baz' file.txt # Error # Correct: Use different delimiter or escape sed 's|foo/bar|baz|' file.txt sed 's/foo\/bar/baz/' file.txt # Error: Missing closing quote # sed 's/foo/bar file.txt # Error # Error: Invalid range # sed '10,5d' file.txt # Error (start > end) # Error: Using non-GNU extensions on non-GNU sed # sed -i 's/foo/bar/' file.txt # Works on GNU, may fail on BSD # Fix for cross-platform if sed --version 2>/dev/null | grep -q GNU; then sed -i 's/foo/bar/' file.txt else sed -i '' 's/foo/bar/' file.txt # BSD sed fi
11. Integration with Other Commands
Pipeline Integration
#!/bin/bash
# With grep
grep "ERROR" log.txt | sed 's/^/*** /' # Highlight errors
# With awk
awk '{print $1}' data.txt | sed 's/[0-9]/X/g' # Mask numbers
# With sort
sed 's/^[[:space:]]*//' file.txt | sort -u # Sort unique lines
# With find
find . -name "*.txt" -exec sed -i 's/foo/bar/g' {} \;
# With xargs
grep -l "pattern" *.txt | xargs sed -i 's/old/new/g'
# Complex pipeline
ps aux |
sed -n '/^USER/d; s/ */ /g; p' |
sort -k3 -rn |
head -10
# Monitor and process
tail -f log.txt | sed -u 's/ERROR/\x1b[31m&\x1b[0m/'
Shell Functions with sed
#!/bin/bash
# Create reusable sed functions
trim() {
sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//'
}
lowercase() {
sed 's/[A-Z]/\L&/g'
}
uppercase() {
sed 's/[a-z]/\U&/g'
}
slugify() {
sed -e 's/[^[:alnum:]]/-/g' -e 's/--*/-/g' -e 's/^-//' -e 's/-$//' | lowercase
}
strip_ansi() {
sed 's/\x1b\[[0-9;]*m//g'
}
# Usage
echo " Hello World " | trim # "Hello World"
echo "Hello World" | lowercase # "hello world"
echo "Hello World" | uppercase # "HELLO WORLD"
echo "My File Name.txt" | slugify # "my-file-name-txt"
12. Advanced Patterns and Techniques
Text Transformations
#!/bin/bash
# Rot13 encoding
rot13() {
sed 'y/ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz/NOPQRSTUVWXYZABCDEFGHIJKLMnopqrstuvwxyzabcdefghijklm/'
}
# Base64-like encoding (simple)
simple_encode() {
sed 's/./\&#\1;/g' | sed 's/[^ ]/ /g'
}
# Word wrap
word_wrap() {
local width="${1:-80}"
shift
sed "s/.\{$width\}/&\n/g" "$@"
}
# Justify text
justify() {
local width="${1:-80}"
shift
sed -e ":a" -e "s/^.\{1,$((width-1))\}$/& /" -e "ta" "$@"
}
# Reverse lines (tac emulation)
reverse_lines() {
sed -n '1!G;h;$p' "$1"
}
# Reverse characters on each line
reverse_chars() {
sed '/\n/!G;s/\(.\)\(.*\n\)/&\2\1/;//D;s/.//' "$1"
}
Code Generation and Transformation
#!/bin/bash
# Add line numbers to code
number_code() {
sed = "$1" | sed 'N;s/^/ /; s/\n/ /'
}
# Comment out code blocks
comment_block() {
local start="$1"
local end="$2"
local file="$3"
sed "/$start/,/$end/ s/^/# /" "$file"
}
# Uncomment code blocks
uncomment_block() {
local start="$1"
local end="$2"
local file="$3"
sed "/$start/,/$end/ s/^# //" "$file"
}
# Convert indentation
spaces_to_tabs() {
sed 's/^\( *\)/\1/' | sed 's/ /\t/g' # Simple version
}
tabs_to_spaces() {
local count="${1:-4}"
sed "s/\t/$(printf '%*s' $count)/g"
}
# Extract functions from source
extract_functions() {
sed -n '/^[a-zA-Z_][a-zA-Z0-9_]* *(.*)/,/^}/p' "$1"
}
13. sed One-Liners
Common One-Liners
# File operations sed -i '1i Header line' file.txt # Insert line at top sed -i '$a Footer line' file.txt # Append line at bottom sed '2r insert.txt' file.txt # Insert file at line 2 sed -n '2,$p' file.txt # Print from line 2 to end sed -n '1!G;h;$p' file.txt # Reverse lines (tac) # Text transformations sed 's/[[:punct:]]//g' file.txt # Remove punctuation sed 's/[0-9]//g' file.txt # Remove digits sed 's/[[:alpha:]]//g' file.txt # Remove letters sed 's/\(.*\)/\L\1/' file.txt # Lowercase sed 's/\(.*\)/\U\1/' file.txt # Uppercase sed 's/^./\u&/' file.txt # Capitalize first letter # Whitespace handling sed 's/^[[:space:]]*//' file.txt # Remove leading spaces sed 's/[[:space:]]*$//' file.txt # Remove trailing spaces sed 's/[[:space:]]\+/ /g' file.txt # Normalize spaces sed '/^$/d' file.txt # Remove empty lines sed '/./,$!d' file.txt # Remove leading blank lines sed -n '/[^[:space:]]/p' file.txt # Remove blank lines # Line manipulation sed 'n;d' file.txt # Print odd lines sed -n 'n;p' file.txt # Print even lines sed '$!N;$!D' file.txt # Print last 2 lines sed 'N;s/\n/ /' file.txt # Join lines sed ':a;N;$!ba;s/\n/ /g' file.txt # Join all lines # Pattern matching sed -n '/pattern/p' file.txt # Print matching lines sed '/pattern/d' file.txt # Delete matching lines sed '/pattern/!d' file.txt # Delete non-matching sed -n '/start/,/end/p' file.txt # Print range sed '/start/,/end/d' file.txt # Delete range # Substitution sed 's/foo/bar/' file.txt # First occurrence sed 's/foo/bar/2' file.txt # Second occurrence sed 's/foo/bar/g' file.txt # All occurrences sed 's/foo/bar/gi' file.txt # Case-insensitive sed 's/\(.*\)/[\1]/' file.txt # Wrap line in brackets
Practical One-Liners
# System administration df -h | sed '1d' | sort -k5 -rn # Sort filesystems by usage ps aux | sed '1d' | sort -k3 -rn # Sort processes by CPU ifconfig | sed -n '/inet /p' # Show IP addresses last | sed -n '/still/p' # Show logged-in users # Text processing sed -n '1,5p' /etc/passwd # First 5 lines of passwd sed -n '/^root/=' /etc/passwd # Line number of root sed 's/:.*//' /etc/passwd | sort # List all users sed 's/.*:\(.*\):.*/\1/' /etc/passwd | sort -u # List all shells # File conversion sed 's/\r$//' dosfile.txt > unixfile.txt # DOS to Unix sed 's/$/\r/' unixfile.txt > dosfile.txt # Unix to DOS sed 's/\t/ /g' file.txt # Tabs to spaces sed 's/ /\t/g' file.txt # Spaces to tabs # HTML/XML sed 's/<[^>]*>//g' file.html # Remove HTML tags sed -n 's/.*<title>\(.*\)<\/title>.*/\1/p' page # Extract title sed 's/&/\&/g; s/</</g; s/>/>/g' # Decode HTML entities # CSV sed 's/,/|/g' file.csv # Change delimiter sed 's/"\([^"]*\)"/\1/g' file.csv # Remove quotes sed '1d' file.csv # Remove header
14. sed vs Other Tools
Comparison with awk, grep, cut
#!/bin/bash
# sed vs grep
grep "pattern" file.txt # Find lines with pattern
sed -n '/pattern/p' file.txt # Same as above
grep -v "pattern" file.txt # Lines without pattern
sed -n '/pattern/!p' file.txt # Same as above
# sed vs awk
awk '{print $2}' file.txt # Print second field
sed 's/[^[:space:]]*[[:space:]]*\([^[:space:]]*\).*/\1/' file.txt # Complex
# sed vs cut
cut -d: -f1 /etc/passwd # First field
sed 's/:.*//' /etc/passwd # Same as above
# sed vs tr
tr 'A-Z' 'a-z' < file.txt # Lowercase
sed 's/[A-Z]/\L&/g' file.txt # Same as above
# Performance comparison
time grep "pattern" largefile.txt > /dev/null
time sed -n '/pattern/p' largefile.txt > /dev/null
# grep is usually faster for simple pattern matching
When to Use sed
# sed is best for: # 1. Simple substitutions sed 's/old/new/g' file.txt # 2. Line-based operations sed '1,10d' file.txt # 3. Stream editing in pipelines command | sed 's/^/ /' | another_command # 4. Simple scripts sed -f script.sed file.txt # 5. In-place editing sed -i 's/foo/bar/g' *.txt # Use awk for: # - Field-based processing # - Complex calculations # - Data reporting # Use perl for: # - Very complex regex # - Multi-line patterns # - One-liners with advanced features
15. Best Practices and Tips
sed Best Practices
#!/bin/bash
# 1. Quote your sed commands
# Good
sed 's/foo/bar/' file.txt
# Bad (if file has special characters)
sed s/foo/bar/ file.txt
# 2. Use different delimiters for paths
sed 's|/usr/local|/opt|' paths.txt
# 3. Test with -n first
sed -n 's/pattern/replacement/p' file.txt # Preview changes
# 4. Create backups before in-place edits
sed -i.bak 's/foo/bar/g' important.txt
# 5. Use addresses to limit scope
sed '100,200 s/foo/bar/g' largefile.txt
# 6. Comment complex scripts
sed '
# Remove comments
s/#.*$//
# Remove empty lines
/^$/d
' file.txt
# 7. Use extended regex for readability
sed -E 's/[0-9]{3}-[0-9]{4}/(xxx)/' phones.txt
# 8. Handle errors gracefully
if ! sed -i 's/foo/bar/' file.txt 2>/dev/null; then
echo "sed failed"
fi
Performance Tips
#!/bin/bash # 1. Use single sed instead of multiple # Slow sed 's/foo/bar/' file.txt | sed 's/baz/qux/' > out.txt # Fast sed 's/foo/bar/; s/baz/qux/' file.txt > out.txt # 2. Use addresses to limit processing sed '1000,2000 s/foo/bar/' largefile.txt # 3. Avoid unnecessary pipes # Slow cat file.txt | sed 's/foo/bar/' > out.txt # Fast sed 's/foo/bar/' file.txt > out.txt # 4. Use -n when you don't need all output sed -n '/pattern/p' file.txt # 5. For huge files, consider splitting split -l 100000 largefile.txt part for f in part*; do sed -i 's/foo/bar/' "$f" & done wait cat part* > processed.txt # 6. Use -u for unbuffered output in pipelines tail -f log.txt | sed -u 's/ERROR/!!!/' | grep -v "DEBUG"
Common Pitfalls and Solutions
#!/bin/bash
# Pitfall 1: Special characters in pattern
# Wrong
sed 's/$10/amount/' file.txt # $ is special
# Correct
sed 's/\$10/amount/' file.txt
# Pitfall 2: Using / in patterns
# Wrong
sed 's/usr/local/opt/' file.txt # / in pattern
# Correct
sed 's|/usr/local|/opt|' file.txt
# Pitfall 3: In-place on symlinks (GNU sed)
sed -i 's/foo/bar/' symlink # Replaces symlink with regular file
# Better
sed -i 's/foo/bar/' "$(readlink -f symlink)"
# Pitfall 4: Portability between GNU and BSD sed
# GNU sed
sed -i 's/foo/bar/' file.txt
# BSD sed (macOS)
sed -i '' 's/foo/bar/' file.txt
# Portable version
if sed --version 2>/dev/null | grep -q GNU; then
sed -i 's/foo/bar/' file.txt
else
sed -i '' 's/foo/bar/' file.txt
fi
# Pitfall 5: Newline handling
# Won't match across lines
sed '/start/,/end/ s/foo/bar/' file.txt
# Use N for multi-line
sed '/start/{N;s/foo\nbar/bar\nfoo/}' file.txt
Conclusion
sed is an incredibly powerful stream editor that every Unix/Linux user should master:
Key Takeaways
- Stream Processing: sed processes text line by line efficiently
- Pattern Matching: Powerful regex support for text manipulation
- In-Place Editing: Can modify files directly with
-ioption - Scripting: Complex transformations can be scripted
- Pipeline Friendly: Works seamlessly with other Unix commands
- Portability: Available on virtually all Unix-like systems
Command Summary
| Command | Description |
|---|---|
s/pattern/replacement/ | Substitute |
/pattern/d | Delete matching lines |
/pattern/p | Print matching lines |
5,10d | Delete lines 5-10 |
y/abc/xyz/ | Translate characters |
= | Print line numbers |
r file | Read file |
w file | Write to file |
q | Quit |
Best Practices Checklist
- [ ] Quote sed commands to prevent shell interpretation
- [ ] Test with
-nbefore actual execution - [ ] Use different delimiters for paths
- [ ] Create backups with
-i.bak - [ ] Use addresses to limit scope
- [ ] Comment complex scripts
- [ ] Handle errors gracefully
- [ ] Consider portability between GNU and BSD sed
- [ ] Use extended regex (
-Eor-r) for complex patterns - [ ] Avoid useless use of cat
Common Use Cases
- Text substitution in files and streams
- Filtering lines by patterns
- Data extraction from structured text
- Code generation and transformation
- Log processing and analysis
- File format conversion
- Batch renaming of files
- Configuration file updates
The sed command's simplicity and power make it an indispensable tool for text processing tasks. While it has a learning curve, mastering sed will significantly enhance your productivity in the Unix/Linux environment.