Complete Guide to Bash sed Command (Stream Editor)

Table of Contents

Introduction to `sed`

sed (Stream Editor) is a powerful text processing tool in Unix/Linux that performs basic text transformations on an input stream (a file or input from a pipeline). It's designed for efficient, non-interactive editing of text files using simple programming constructs.

Key Concepts

Stream Processing: Processes text line by line
Pattern Matching: Uses regular expressions for pattern matching
In-Place Editing: Can modify files directly
Scripting: Supports complex editing scripts
Pipeline Integration: Works seamlessly with other Unix commands

1. Basic Syntax and Structure

Command Structure

# Basic syntax
sed [options] 'command' file.txt
sed [options] -f script.sed file.txt
# Common patterns
sed 's/pattern/replacement/' file.txt     # Substitute
sed '/pattern/d' file.txt                  # Delete
sed -n '/pattern/p' file.txt                # Print matching lines
sed -i 's/old/new/g' file.txt               # In-place edit
# Example: Simple substitution
echo "hello world" | sed 's/world/universe/'
# Output: hello universe
# Multiple commands
sed -e 's/foo/bar/' -e 's/baz/qux/' file.txt
# Using semicolons
sed 's/foo/bar/; s/baz/qux/' file.txt

Command Options

# Common options
sed -n 'p' file.txt           # Suppress automatic printing (-n)
sed -i 's/old/new/' file.txt  # Edit files in-place (-i)
sed -i.bak 's/old/new/' file  # Create backup before editing
sed -e 'command' file.txt     # Add multiple commands (-e)
sed -f script.sed file.txt     # Read commands from file (-f)
sed -r 'pattern' file.txt      # Use extended regex (-r or -E)
# Example with backup
sed -i.bak 's/error/warning/' config.txt
# Creates config.txt.bak before modifying config.txt
# Multiple expressions
sed -e 's/foo/bar/' -e 's/[0-9]//g' data.txt

2. Basic Operations

Substitution (s Command)

# Basic substitution
sed 's/old/new/' file.txt      # Replace first occurrence per line
sed 's/old/new/g' file.txt     # Replace all occurrences (global)
sed 's/old/new/2' file.txt     # Replace second occurrence only
sed 's/old/new/gi' file.txt    # Case-insensitive global replace
# Using different delimiters
sed 's|/usr/local|/opt|' paths.txt    # Use | as delimiter
sed 's:/home:/users:' passwd           # Use : as delimiter
sed 's_old_new_' file.txt              # Use _ as delimiter
# Examples
echo "foo foo foo" | sed 's/foo/bar/'     # bar foo foo
echo "foo foo foo" | sed 's/foo/bar/2'    # foo bar foo
echo "foo foo foo" | sed 's/foo/bar/g'    # bar bar bar
# Capture groups and backreferences
echo "abc123" | sed 's/\([a-z]*\)\([0-9]*\)/\2\1/'  # 123abc
echo "John Doe" | sed 's/\(.*\) \(.*\)/\2, \1/'     # Doe, John
# Ampersand as matched pattern
echo "hello world" | sed 's/.*/(&)/'        # (hello world)
echo "123-456-7890" | sed 's/[0-9]\+/(&)/g' # (123)-(456)-(7890)

Delete (d Command)

# Delete lines
sed '3d' file.txt              # Delete line 3
sed '5,10d' file.txt           # Delete lines 5-10
sed '/pattern/d' file.txt      # Delete lines matching pattern
sed '/^#/d' file.txt           # Delete comments (lines starting with #)
sed '/^$/d' file.txt           # Delete empty lines
sed '/start/,/end/d' file.txt  # Delete range between patterns
# Complex deletions
sed '1,5d' data.txt            # Delete first 5 lines
sed '$d' file.txt              # Delete last line
sed '/^[[:space:]]*$/d' file   # Delete lines with only whitespace
sed '/ERROR/,+3d' log.txt      # Delete line with ERROR and next 3 lines
# Examples
cat > test.txt << EOF
Line 1
Line 2
Line 3
Line 4
Line 5
EOF
sed '2,4d' test.txt            # Output: Line 1, Line 5
sed '/2/,/4/d' test.txt        # Delete lines containing 2 through 4

Print (p Command)

# Print specific lines
sed -n '5p' file.txt           # Print line 5
sed -n '10,20p' file.txt       # Print lines 10-20
sed -n '/pattern/p' file.txt   # Print lines matching pattern
sed -n '/start/,/end/p' file   # Print range between patterns
# Print with line numbers
sed -n '/error/{=;p}' log.txt  # Print line number and matching line
sed -n '/error/{=;p}' log.txt | sed 'N;s/\n/ /'  # Format: "number line"
# Multiple print conditions
sed -n '/ERROR/p; /WARNING/p' log.txt  # Print ERROR or WARNING lines
sed -n '1,5p; /important/p' file.txt   # Print first 5 lines and important lines
# Examples
cat data.txt | sed -n '/^[0-9]/p'      # Print lines starting with number
sed -n '/[A-Z]\{5,\}/p' file.txt       # Print lines with 5+ uppercase letters

3. Address Ranges and Patterns

Line Addressing

# Single line addresses
sed '5 s/old/new/' file.txt     # Replace on line 5 only
sed '1,10 s/old/new/g' file.txt # Replace on lines 1-10
sed '$ s/old/new/' file.txt     # Replace on last line only
# Pattern addresses
sed '/^Start/,/^End/ s/old/new/' file.txt  # Replace within range
sed '/section/,/endsection/ d' file.txt    # Delete section
sed '/BEGIN/,/END/!d' file.txt              # Keep only between BEGIN and END
# Address negation
sed '/pattern/!s/old/new/' file.txt  # Replace where pattern doesn't match
sed '5,10!d' file.txt                 # Delete all except lines 5-10
# Step addressing (GNU sed)
sed '0~2 s/old/new/' file.txt  # Replace on even lines
sed '1~2 s/old/new/' file.txt  # Replace on odd lines
sed '0~5d' file.txt             # Delete every 5th line
# Examples
sed '/debug/,/enddebug/ s/^/# /' config.c    # Comment out debug section
sed '/^[[:space:]]*$/d' file.txt              # Delete empty lines

Regular Expressions in sed

# Basic regex (BRE)
sed 's/^[[:space:]]*//' file.txt     # Remove leading whitespace
sed 's/[[:space:]]*$//' file.txt      # Remove trailing whitespace
sed 's/[[:alpha:]]\+/WORD/g' file.txt # Replace words with WORD
# Extended regex with -r or -E
sed -r 's/[0-9]{3}-[0-9]{4}/(xxx)/' phones.txt
sed -E 's/[[:alnum:]_]+@[[:alnum:]_]+\.[a-z]{2,3}/[EMAIL]/' emails.txt
# Common regex patterns
sed -n '/^#/!p' config.txt           # Print non-comment lines
sed '/^$/d' file.txt                  # Remove empty lines
sed 's/[[:punct:]]//g' file.txt       # Remove punctuation
sed 's/[[:upper:]]/\L&/g' file.txt    # Convert to lowercase
sed 's/[[:lower:]]/\U&/g' file.txt    # Convert to uppercase
# Complex regex examples
sed -E 's/\b([0-9]{1,3}\.){3}[0-9]{1,3}\b/IP/' file  # Replace IP addresses
sed -E 's/^([^:]+):.*/\1/' passwd                     # Extract usernames
sed -E 's/<[^>]+>//g' html.html                       # Remove HTML tags

4. Advanced Commands

Transform (y Command)

# Character translation
sed 'y/abc/xyz/' file.txt        # a->x, b->y, c->z
sed 'y/[A-Z]/[a-z]/' file.txt    # Convert uppercase to lowercase
sed 'y/ \t/__/' file.txt          # Convert spaces/tabs to underscores
# Examples
echo "hello world" | sed 'y/helo/HELO/'  # HELLO wOrLd
echo "123-456-7890" | sed 'y/-/ /'       # 123 456 7890
# Note: y does character-by-character translation, not patterns
sed 'y/123456789/987654321/' numbers.txt  # Reverse digits

Hold Space and Pattern Space

# Hold space commands
# h - copy pattern space to hold space
# H - append pattern space to hold space
# g - copy hold space to pattern space
# G - append hold space to pattern space
# x - exchange pattern and hold spaces
# Reverse lines of a file (tac equivalent)
sed -n '1!G;h;$p' file.txt
# Double-space a file
sed 'G' file.txt
# Join lines
sed 'N;s/\n/ /' file.txt  # Join first two lines
sed 'N;N;s/\n/ /g' file    # Join every three lines
# Advanced examples
# Print even lines
sed -n 'n;p' file.txt
# Print odd lines
sed -n 'p;n' file.txt
# Remove duplicate consecutive lines
sed '$!N; /^\(.*\)\n\1$/!P; D' file.txt
# Extract text between markers
sed -n '/START/{:a;n;/END/q;p;ba}' file.txt

Branching and Conditional Operations

# Branching commands
# :label - define a label
# b label - branch to label
# t label - branch if substitution succeeded
# Example: Multiple replacements
sed ':a;s/foo/bar/;ta' file.txt  # Repeat substitution until no more changes
# Conditional processing
sed '/error/{ 
s/error/ERROR/
s/warning/WARNING/
}' log.txt
# Complex example - convert to title case
sed 's/.*/\L&/; s/[a-z]*/\u&/g' file.txt
# Loop until no more changes
sed ':start; s/[0-9][0-9]/&/; t start' numbers.txt

5. In-Place Editing

Basic In-Place Operations

# Edit file in-place (GNU sed)
sed -i 's/old/new/g' file.txt
# Create backup before editing
sed -i.bak 's/old/new/g' file.txt
# Multiple files
sed -i 's/old/new/g' file1.txt file2.txt file3.txt
# With backup extension
sed -i'.original' 's/old/new/g' config.txt
# Examples
# Replace all instances of "foo" with "bar" in all .txt files
sed -i 's/foo/bar/g' *.txt
# Remove trailing whitespace from multiple files
sed -i 's/[[:space:]]*$//' *.c *.h
# Add header to files
sed -i '1i # This file was modified' *.txt

Safe In-Place Editing

#!/bin/bash
# Safe in-place edit with backup and verification
safe_sed() {
local file="$1"
local backup="${file}.bak.$(date +%Y%m%d-%H%M%S)"
shift
# Create backup
cp "$file" "$backup"
# Perform edit
sed -i "$@" "$file"
# Ask for verification
echo "Changes made. Check $file"
echo "Backup saved at $backup"
echo -n "Keep changes? (y/n): "
read answer
if [ "$answer" != "y" ]; then
cp "$backup" "$file"
echo "Restored from backup"
fi
}
# Edit with diff preview
preview_sed() {
local file="$1"
shift
cp "$file" /tmp/sed_preview
sed -i /tmp/sed_preview "$@"
echo "=== Changes ==="
diff -u "$file" /tmp/sed_preview
echo "==============="
echo -n "Apply changes? (y/n): "
read answer
if [ "$answer" = "y" ]; then
cp /tmp/sed_preview "$file"
fi
rm /tmp/sed_preview
}

6. Multi-line Operations

Working with Multiple Lines

# Join lines
sed ':a;N;$!ba;s/\n/ /g' file.txt      # Join all lines
sed 'N;N;s/\n/ /g' file.txt             # Join three lines at a time
# Process paragraphs
sed -n '/^$/{p;d}; H; $ {x;s/\n/ /g;p}' file.txt  # Join paragraph lines
# Multi-line pattern matching
sed '/start/,/end/ {
/start/d
/end/d
s/.*/CHANGED/
}' file.txt
# Examples
# Convert CSV to single line
sed ':a;N;$!ba;s/\n/,/g' data.csv
# Wrap text at column 80
sed 's/.\{80\}/&\n/g' longlines.txt
# Remove HTML tags across lines
sed ':a;N;$!ba;s/<[^>]*>//g' html.txt

Complex Multi-line Examples

#!/bin/bash
# Extract function definitions from C file
extract_functions() {
sed -n '
/^[a-zA-Z_][a-zA-Z0-9_]* *(.*)/ {
:func
N
/\n[{}]/!b func
p
}
' "$1"
}
# Join lines ending with backslash
join_continuations() {
sed ':a;/\\$/N;s/\\\n//;ta' "$1"
}
# Wrap long lines
wrap_lines() {
local width="${1:-80}"
shift
sed "s/.\{$width\}/&\n/g" "$@"
}
# Number paragraphs
number_paragraphs() {
sed '/^$/ {
x
s/^$/Paragraph 0/
s/Paragraph \([0-9]*\)/Paragraph \1/
s/.*/&/
x
}
/^[^ ]/ {
H
x
s/^\(.*\)\n\(.*\)/\2 \1/
s/Paragraph \([0-9]*\)/\1. /
s/\([0-9]*\. \)\(.*\)/\1\2/
s/^[0-9]*\. //
}' "$1"
}

7. Practical Examples

Text Processing

#!/bin/bash
# Remove comments and blank lines
clean_config() {
sed -e 's/#.*$//' -e '/^$/d' "$1"
}
# Extract email addresses
extract_emails() {
sed -n 's/.*\([a-zA-Z0-9._%+-]\+@[a-zA-Z0-9.-]\+\.[a-zA-Z]\{2,\}\).*/\1/p' "$1"
}
# Convert to title case
title_case() {
sed 's/.*/\L&/; s/[a-z]*/\u&/g' "$1"
}
# Add line numbers
add_line_numbers() {
sed = "$1" | sed 'N;s/\n/ /'
}
# Format JSON (simple)
format_json() {
sed 's/[{,]/&\n  /g; s/}/&\n/g' "$1"
}
# URL decode
urldecode() {
sed 's/%/\\x/g' "$1" | while read -r line; do
echo -e "$line"
done
}
# Examples usage
# clean_config /etc/config.ini
# extract_emails contacts.txt
# title_case names.txt

File and Directory Operations

#!/bin/bash
# Rename files in directory (dry run)
rename_files() {
local pattern="$1"
local replacement="$2"
for file in *; do
newname=$(echo "$file" | sed "s/$pattern/$replacement/")
if [ "$newname" != "$file" ]; then
echo "mv '$file' '$newname'"
fi
done
}
# Batch rename with sed
batch_rename() {
local pattern="$1"
local replacement="$2"
shift 2
for file in "$@"; do
newname=$(echo "$file" | sed "s/$pattern/$replacement/")
if [ "$newname" != "$file" ]; then
mv -v "$file" "$newname"
fi
done
}
# Update file extensions
change_extension() {
local old_ext="$1"
local new_ext="$2"
for file in *."$old_ext"; do
[ -f "$file" ] || continue
base=$(echo "$file" | sed "s/\.$old_ext$//")
mv -v "$file" "$base.$new_ext"
done
}

Log Processing

#!/bin/bash
# Filter log by date range
filter_log_by_date() {
local start_date="$1"
local end_date="$2"
local logfile="$3"
sed -n "/^$start_date/,/^$end_date/p" "$logfile"
}
# Extract error messages with context
extract_errors() {
local logfile="$1"
local context="${2:-2}"
sed -n "
/ERROR/ {
= 
N
N
p
}
" "$logfile" | sed 'N;s/\n/ /g'
}
# Summarize log levels
log_summary() {
sed -n 's/.*\(ERROR\|WARN\|INFO\).*/\1/p' "$1" | sort | uniq -c
}
# Remove timestamp prefix
remove_timestamp() {
sed 's/^\[[0-9-]* [0-9:]*\] //' "$1"
}
# Format log for better readability
format_log() {
sed 's/\[\([^]]*\)\]/\n[\1]\n  /g' "$1" | sed '/^$/d'
}

CSV Processing

#!/bin/bash
# Extract specific column
extract_column() {
local col="$1"
local file="$2"
sed "s/[^,]*\(,[^,]*\)\{$((col-1))\},//" "$file"
}
# Change delimiter
change_delimiter() {
local old="$1"
local new="$2"
local file="$3"
sed "s/$old/$new/g" "$file"
}
# Remove header
remove_header() {
sed '1d' "$1"
}
# Add header
add_header() {
local header="$1"
local file="$2"
sed "1i $header" "$file"
}
# Filter rows by value in column
filter_by_column() {
local col="$1"
local value="$2"
local file="$3"
sed -n "/\([^,]*,\?\)\{$((col-1))\}$value/p" "$file"
}
# Examples
# extract_column 3 data.csv
# change_delimiter ',' '|' data.csv

8. sed Scripts

Creating sed Scripts

#!/bin/bash
# Create a sed script file
cat > cleanup.sed << 'EOF'
# Remove comments
s/#.*$//
# Remove empty lines
/^$/d
# Trim whitespace
s/^[[:space:]]*//
s/[[:space:]]*$//
# Normalize spaces
s/[[:space:]]\+/ /g
# Convert to lowercase
s/[A-Z]/\L&/g
EOF
# Use the script
sed -f cleanup.sed input.txt > output.txt
# Another example: markdown to plain text
cat > md2txt.sed << 'EOF'
# Remove headers
s/^#\+ //g
# Remove emphasis
s/\*\*\([^*]*\)\*\*/\1/g
s/\*\([^*]*\)\*/\1/g
# Remove links
s/\[\([^]]*\)\]([^)]*)/\1/g
# Remove code blocks
/^```/,/^```/d
# Remove images
s/!\[[^]]*\]([^)]*)//g
# Convert lists
s/^[*-] /• /g
EOF

Complex sed Script Example

#!/bin/bash
# HTML to text converter
cat > html2text.sed << 'EOF'
#!/usr/bin/sed -f
# Remove script and style sections
/<script/,/<\/script>/d
/<style/,/<\/style>/d
# Remove HTML tags
s/<[^>]*>//g
# Replace common entities
s/&nbsp;/ /g
s/&lt;/</g
s/&gt;/>/g
s/&amp;/\&/g
s/&quot;/"/g
# Handle multiple blank lines
/^$/{
N
/^\n$/d
}
# Trim lines
s/^[[:space:]]*//
s/[[:space:]]*$//
# Join lines that don't end with punctuation
:join
N
/\n[^A-Z]/s/\n/ /
t join
# Add paragraph breaks
s/\.  /.  \n\n/g
EOF
# Make executable and use
chmod +x html2text.sed
./html2text.sed webpage.html > webpage.txt

9. Performance Optimization

Efficient sed Usage

#!/bin/bash
# Use single sed command instead of multiple
# Bad: multiple processes
cat file.txt | sed 's/foo/bar/' | sed 's/baz/qux/' > output.txt
# Good: single sed with multiple commands
sed 's/foo/bar/; s/baz/qux/' file.txt > output.txt
# Use sed -e for multiple expressions
sed -e 's/foo/bar/' -e 's/baz/qux/' file.txt
# Avoid unnecessary pipes
# Bad: useless use of cat
cat file.txt | sed 's/foo/bar/'
# Good: sed reads file directly
sed 's/foo/bar/' file.txt
# Use address ranges to limit processing
sed '1000,2000 s/foo/bar/g' largefile.txt  # Only process relevant section
# Exit early when done
sed '/stop/q' file.txt  # Stop processing after finding "stop"
# Performance comparison
time sed 's/foo/bar/g' largefile.txt > /dev/null
time sed -n 's/foo/bar/gp' largefile.txt > /dev/null  # Usually slower

Handling Large Files

#!/bin/bash
# Process large files in chunks
process_large_file() {
local file="$1"
local chunk_size=10000
split -l $chunk_size "$file" chunk_
for chunk in chunk_*; do
sed -i 's/foo/bar/g' "$chunk" &
# Process in parallel
[ $(jobs | wc -l) -ge 4 ] && wait
done
wait
cat chunk_* > "${file}.processed"
rm chunk_*
}
# Stream processing for very large files
stream_process() {
while IFS= read -r line; do
echo "$line" | sed 's/foo/bar/'
done < "$1"
}
# Use sed with buffering
sed -u 's/foo/bar/' largefile.txt  # Unbuffered output
# Monitor progress with pv
pv largefile.txt | sed 's/foo/bar/' > output.txt

10. Error Handling and Debugging

Debugging sed Scripts

#!/bin/bash
# Debug with line numbering
sed -n '/pattern/{
=
p
}' file.txt
# Trace sed execution (GNU sed)
sed --debug 's/foo/bar/' file.txt
# Validate sed script
validate_sed() {
local script="$1"
local testfile="/tmp/sed_test.$$"
echo "test line" > "$testfile"
if sed -f "$script" "$testfile" >/dev/null 2>&1; then
echo "Script syntax OK"
else
echo "Script has errors"
sed -f "$script" "$testfile" 2>&1
fi
rm "$testfile"
}
# Dry run (show what would be changed)
sed_dry_run() {
local file="$1"
shift
diff -u "$file" <(sed "$@" "$file")
}
# Verbose mode for complex scripts
sed_verbose() {
local script="$1"
shift
echo "=== Input ==="
cat "$1"
echo "=== Processing ==="
sed -n "$script" "$1"
echo "=== Output ==="
sed "$script" "$1"
}

Common Error Patterns

#!/bin/bash
# Error: Unescaped delimiter in pattern
# sed 's/foo/bar/baz' file.txt  # Error
# Correct: Use different delimiter or escape
sed 's|foo/bar|baz|' file.txt
sed 's/foo\/bar/baz/' file.txt
# Error: Missing closing quote
# sed 's/foo/bar file.txt  # Error
# Error: Invalid range
# sed '10,5d' file.txt  # Error (start > end)
# Error: Using non-GNU extensions on non-GNU sed
# sed -i 's/foo/bar/' file.txt  # Works on GNU, may fail on BSD
# Fix for cross-platform
if sed --version 2>/dev/null | grep -q GNU; then
sed -i 's/foo/bar/' file.txt
else
sed -i '' 's/foo/bar/' file.txt  # BSD sed
fi

11. Integration with Other Commands

Pipeline Integration

#!/bin/bash
# With grep
grep "ERROR" log.txt | sed 's/^/*** /'  # Highlight errors
# With awk
awk '{print $1}' data.txt | sed 's/[0-9]/X/g'  # Mask numbers
# With sort
sed 's/^[[:space:]]*//' file.txt | sort -u  # Sort unique lines
# With find
find . -name "*.txt" -exec sed -i 's/foo/bar/g' {} \;
# With xargs
grep -l "pattern" *.txt | xargs sed -i 's/old/new/g'
# Complex pipeline
ps aux | 
sed -n '/^USER/d; s/  */ /g; p' |
sort -k3 -rn |
head -10
# Monitor and process
tail -f log.txt | sed -u 's/ERROR/\x1b[31m&\x1b[0m/'

Shell Functions with sed

#!/bin/bash
# Create reusable sed functions
trim() {
sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//'
}
lowercase() {
sed 's/[A-Z]/\L&/g'
}
uppercase() {
sed 's/[a-z]/\U&/g'
}
slugify() {
sed -e 's/[^[:alnum:]]/-/g' -e 's/--*/-/g' -e 's/^-//' -e 's/-$//' | lowercase
}
strip_ansi() {
sed 's/\x1b\[[0-9;]*m//g'
}
# Usage
echo "  Hello World  " | trim          # "Hello World"
echo "Hello World" | lowercase          # "hello world"
echo "Hello World" | uppercase          # "HELLO WORLD"
echo "My File Name.txt" | slugify       # "my-file-name-txt"

12. Advanced Patterns and Techniques

Text Transformations

#!/bin/bash
# Rot13 encoding
rot13() {
sed 'y/ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz/NOPQRSTUVWXYZABCDEFGHIJKLMnopqrstuvwxyzabcdefghijklm/'
}
# Base64-like encoding (simple)
simple_encode() {
sed 's/./\&#\1;/g' | sed 's/[^ ]/ /g'
}
# Word wrap
word_wrap() {
local width="${1:-80}"
shift
sed "s/.\{$width\}/&\n/g" "$@"
}
# Justify text
justify() {
local width="${1:-80}"
shift
sed -e ":a" -e "s/^.\{1,$((width-1))\}$/& /" -e "ta" "$@"
}
# Reverse lines (tac emulation)
reverse_lines() {
sed -n '1!G;h;$p' "$1"
}
# Reverse characters on each line
reverse_chars() {
sed '/\n/!G;s/\(.\)\(.*\n\)/&\2\1/;//D;s/.//' "$1"
}

Code Generation and Transformation

#!/bin/bash
# Add line numbers to code
number_code() {
sed = "$1" | sed 'N;s/^/  /; s/\n/ /'
}
# Comment out code blocks
comment_block() {
local start="$1"
local end="$2"
local file="$3"
sed "/$start/,/$end/ s/^/# /" "$file"
}
# Uncomment code blocks
uncomment_block() {
local start="$1"
local end="$2"
local file="$3"
sed "/$start/,/$end/ s/^# //" "$file"
}
# Convert indentation
spaces_to_tabs() {
sed 's/^\( *\)/\1/' | sed 's/  /\t/g'  # Simple version
}
tabs_to_spaces() {
local count="${1:-4}"
sed "s/\t/$(printf '%*s' $count)/g"
}
# Extract functions from source
extract_functions() {
sed -n '/^[a-zA-Z_][a-zA-Z0-9_]* *(.*)/,/^}/p' "$1"
}

13. sed One-Liners

Common One-Liners

# File operations
sed -i '1i Header line' file.txt          # Insert line at top
sed -i '$a Footer line' file.txt          # Append line at bottom
sed '2r insert.txt' file.txt              # Insert file at line 2
sed -n '2,$p' file.txt                    # Print from line 2 to end
sed -n '1!G;h;$p' file.txt                # Reverse lines (tac)
# Text transformations
sed 's/[[:punct:]]//g' file.txt           # Remove punctuation
sed 's/[0-9]//g' file.txt                  # Remove digits
sed 's/[[:alpha:]]//g' file.txt            # Remove letters
sed 's/\(.*\)/\L\1/' file.txt              # Lowercase
sed 's/\(.*\)/\U\1/' file.txt              # Uppercase
sed 's/^./\u&/' file.txt                   # Capitalize first letter
# Whitespace handling
sed 's/^[[:space:]]*//' file.txt          # Remove leading spaces
sed 's/[[:space:]]*$//' file.txt           # Remove trailing spaces
sed 's/[[:space:]]\+/ /g' file.txt         # Normalize spaces
sed '/^$/d' file.txt                        # Remove empty lines
sed '/./,$!d' file.txt                      # Remove leading blank lines
sed -n '/[^[:space:]]/p' file.txt          # Remove blank lines
# Line manipulation
sed 'n;d' file.txt                          # Print odd lines
sed -n 'n;p' file.txt                        # Print even lines
sed '$!N;$!D' file.txt                       # Print last 2 lines
sed 'N;s/\n/ /' file.txt                     # Join lines
sed ':a;N;$!ba;s/\n/ /g' file.txt            # Join all lines
# Pattern matching
sed -n '/pattern/p' file.txt                 # Print matching lines
sed '/pattern/d' file.txt                     # Delete matching lines
sed '/pattern/!d' file.txt                    # Delete non-matching
sed -n '/start/,/end/p' file.txt              # Print range
sed '/start/,/end/d' file.txt                  # Delete range
# Substitution
sed 's/foo/bar/' file.txt                     # First occurrence
sed 's/foo/bar/2' file.txt                    # Second occurrence
sed 's/foo/bar/g' file.txt                     # All occurrences
sed 's/foo/bar/gi' file.txt                    # Case-insensitive
sed 's/\(.*\)/[\1]/' file.txt                  # Wrap line in brackets

Practical One-Liners

# System administration
df -h | sed '1d' | sort -k5 -rn                # Sort filesystems by usage
ps aux | sed '1d' | sort -k3 -rn               # Sort processes by CPU
ifconfig | sed -n '/inet /p'                    # Show IP addresses
last | sed -n '/still/p'                        # Show logged-in users
# Text processing
sed -n '1,5p' /etc/passwd                       # First 5 lines of passwd
sed -n '/^root/=' /etc/passwd                    # Line number of root
sed 's/:.*//' /etc/passwd | sort                 # List all users
sed 's/.*:\(.*\):.*/\1/' /etc/passwd | sort -u   # List all shells
# File conversion
sed 's/\r$//' dosfile.txt > unixfile.txt        # DOS to Unix
sed 's/$/\r/' unixfile.txt > dosfile.txt        # Unix to DOS
sed 's/\t/    /g' file.txt                       # Tabs to spaces
sed 's/    /\t/g' file.txt                       # Spaces to tabs
# HTML/XML
sed 's/<[^>]*>//g' file.html                    # Remove HTML tags
sed -n 's/.*<title>\(.*\)<\/title>.*/\1/p' page # Extract title
sed 's/&amp;/\&/g; s/&lt;/</g; s/&gt;/>/g'      # Decode HTML entities
# CSV
sed 's/,/|/g' file.csv                          # Change delimiter
sed 's/"\([^"]*\)"/\1/g' file.csv               # Remove quotes
sed '1d' file.csv                                # Remove header

14. sed vs Other Tools

Comparison with awk, grep, cut

#!/bin/bash
# sed vs grep
grep "pattern" file.txt                         # Find lines with pattern
sed -n '/pattern/p' file.txt                    # Same as above
grep -v "pattern" file.txt                       # Lines without pattern
sed -n '/pattern/!p' file.txt                     # Same as above
# sed vs awk
awk '{print $2}' file.txt                        # Print second field
sed 's/[^[:space:]]*[[:space:]]*\([^[:space:]]*\).*/\1/' file.txt  # Complex
# sed vs cut
cut -d: -f1 /etc/passwd                          # First field
sed 's/:.*//' /etc/passwd                        # Same as above
# sed vs tr
tr 'A-Z' 'a-z' < file.txt                        # Lowercase
sed 's/[A-Z]/\L&/g' file.txt                      # Same as above
# Performance comparison
time grep "pattern" largefile.txt > /dev/null
time sed -n '/pattern/p' largefile.txt > /dev/null
# grep is usually faster for simple pattern matching

When to Use sed

# sed is best for:
# 1. Simple substitutions
sed 's/old/new/g' file.txt
# 2. Line-based operations
sed '1,10d' file.txt
# 3. Stream editing in pipelines
command | sed 's/^/  /' | another_command
# 4. Simple scripts
sed -f script.sed file.txt
# 5. In-place editing
sed -i 's/foo/bar/g' *.txt
# Use awk for:
# - Field-based processing
# - Complex calculations
# - Data reporting
# Use perl for:
# - Very complex regex
# - Multi-line patterns
# - One-liners with advanced features

15. Best Practices and Tips

sed Best Practices

#!/bin/bash
# 1. Quote your sed commands
# Good
sed 's/foo/bar/' file.txt
# Bad (if file has special characters)
sed s/foo/bar/ file.txt
# 2. Use different delimiters for paths
sed 's|/usr/local|/opt|' paths.txt
# 3. Test with -n first
sed -n 's/pattern/replacement/p' file.txt  # Preview changes
# 4. Create backups before in-place edits
sed -i.bak 's/foo/bar/g' important.txt
# 5. Use addresses to limit scope
sed '100,200 s/foo/bar/g' largefile.txt
# 6. Comment complex scripts
sed '
# Remove comments
s/#.*$//
# Remove empty lines
/^$/d
' file.txt
# 7. Use extended regex for readability
sed -E 's/[0-9]{3}-[0-9]{4}/(xxx)/' phones.txt
# 8. Handle errors gracefully
if ! sed -i 's/foo/bar/' file.txt 2>/dev/null; then
echo "sed failed"
fi

Performance Tips

#!/bin/bash
# 1. Use single sed instead of multiple
# Slow
sed 's/foo/bar/' file.txt | sed 's/baz/qux/' > out.txt
# Fast
sed 's/foo/bar/; s/baz/qux/' file.txt > out.txt
# 2. Use addresses to limit processing
sed '1000,2000 s/foo/bar/' largefile.txt
# 3. Avoid unnecessary pipes
# Slow
cat file.txt | sed 's/foo/bar/' > out.txt
# Fast
sed 's/foo/bar/' file.txt > out.txt
# 4. Use -n when you don't need all output
sed -n '/pattern/p' file.txt
# 5. For huge files, consider splitting
split -l 100000 largefile.txt part
for f in part*; do sed -i 's/foo/bar/' "$f" & done
wait
cat part* > processed.txt
# 6. Use -u for unbuffered output in pipelines
tail -f log.txt | sed -u 's/ERROR/!!!/' | grep -v "DEBUG"

Common Pitfalls and Solutions

#!/bin/bash
# Pitfall 1: Special characters in pattern
# Wrong
sed 's/$10/amount/' file.txt  # $ is special
# Correct
sed 's/\$10/amount/' file.txt
# Pitfall 2: Using / in patterns
# Wrong
sed 's/usr/local/opt/' file.txt  # / in pattern
# Correct
sed 's|/usr/local|/opt|' file.txt
# Pitfall 3: In-place on symlinks (GNU sed)
sed -i 's/foo/bar/' symlink  # Replaces symlink with regular file
# Better
sed -i 's/foo/bar/' "$(readlink -f symlink)"
# Pitfall 4: Portability between GNU and BSD sed
# GNU sed
sed -i 's/foo/bar/' file.txt
# BSD sed (macOS)
sed -i '' 's/foo/bar/' file.txt
# Portable version
if sed --version 2>/dev/null | grep -q GNU; then
sed -i 's/foo/bar/' file.txt
else
sed -i '' 's/foo/bar/' file.txt
fi
# Pitfall 5: Newline handling
# Won't match across lines
sed '/start/,/end/ s/foo/bar/' file.txt
# Use N for multi-line
sed '/start/{N;s/foo\nbar/bar\nfoo/}' file.txt

Conclusion

sed is an incredibly powerful stream editor that every Unix/Linux user should master:

Key Takeaways

Stream Processing: sed processes text line by line efficiently
Pattern Matching: Powerful regex support for text manipulation
In-Place Editing: Can modify files directly with -i option
Scripting: Complex transformations can be scripted
Pipeline Friendly: Works seamlessly with other Unix commands
Portability: Available on virtually all Unix-like systems

Command Summary

Command	Description
`s/pattern/replacement/`	Substitute
`/pattern/d`	Delete matching lines
`/pattern/p`	Print matching lines
`5,10d`	Delete lines 5-10
`y/abc/xyz/`	Translate characters
`=`	Print line numbers
`r file`	Read file
`w file`	Write to file
`q`	Quit

Best Practices Checklist

[ ] Quote sed commands to prevent shell interpretation
[ ] Test with -n before actual execution
[ ] Use different delimiters for paths
[ ] Create backups with -i.bak
[ ] Use addresses to limit scope
[ ] Comment complex scripts
[ ] Handle errors gracefully
[ ] Consider portability between GNU and BSD sed
[ ] Use extended regex (-E or -r) for complex patterns
[ ] Avoid useless use of cat

Common Use Cases

Text substitution in files and streams
Filtering lines by patterns
Data extraction from structured text
Code generation and transformation
Log processing and analysis
File format conversion
Batch renaming of files
Configuration file updates

The sed command's simplicity and power make it an indispensable tool for text processing tasks. While it has a learning curve, mastering sed will significantly enhance your productivity in the Unix/Linux environment.

Introduction to sed