Complete Guide to Bash sed Command (Stream Editor)

Introduction to `sed`

sed (Stream Editor) is a powerful text processing tool in Unix/Linux that performs basic text transformations on an input stream (a file or input from a pipeline). It's designed for efficient, non-interactive editing of text files using simple programming constructs.

Key Concepts

Stream Processing: Processes text line by line
Pattern Matching: Uses regular expressions for pattern matching
In-Place Editing: Can modify files directly
Scripting: Supports complex editing scripts
Pipeline Integration: Works seamlessly with other Unix commands

1. Basic Syntax and Structure

Command Structure

# Basic syntax sed [options] 'command' file.txt sed [options] -f script.sed file.txt # Common patterns sed 's/pattern/replacement/' file.txt # Substitute sed '/pattern/d' file.txt # Delete sed -n '/pattern/p' file.txt # Print matching lines sed -i 's/old/new/g' file.txt # In-place edit # Example: Simple substitution echo "hello world" | sed 's/world/universe/' # Output: hello universe # Multiple commands sed -e 's/foo/bar/' -e 's/baz/qux/' file.txt # Using semicolons sed 's/foo/bar/; s/baz/qux/' file.txt

Command Options

# Common options sed -n 'p' file.txt # Suppress automatic printing (-n) sed -i 's/old/new/' file.txt # Edit files in-place (-i) sed -i.bak 's/old/new/' file # Create backup before editing sed -e 'command' file.txt # Add multiple commands (-e) sed -f script.sed file.txt # Read commands from file (-f) sed -r 'pattern' file.txt # Use extended regex (-r or -E) # Example with backup sed -i.bak 's/error/warning/' config.txt # Creates config.txt.bak before modifying config.txt # Multiple expressions sed -e 's/foo/bar/' -e 's/[0-9]//g' data.txt

2. Basic Operations

Substitution (s Command)

# Basic substitution sed 's/old/new/' file.txt # Replace first occurrence per line sed 's/old/new/g' file.txt # Replace all occurrences (global) sed 's/old/new/2' file.txt # Replace second occurrence only sed 's/old/new/gi' file.txt # Case-insensitive global replace # Using different delimiters sed 's|/usr/local|/opt|' paths.txt # Use | as delimiter sed 's:/home:/users:' passwd # Use : as delimiter sed 's_old_new_' file.txt # Use _ as delimiter # Examples echo "foo foo foo" | sed 's/foo/bar/' # bar foo foo echo "foo foo foo" | sed 's/foo/bar/2' # foo bar foo echo "foo foo foo" | sed 's/foo/bar/g' # bar bar bar # Capture groups and backreferences echo "abc123" | sed 's/\([a-z]*\)\([0-9]*\)/\2\1/' # 123abc echo "John Doe" | sed 's/\(.*\) \(.*\)/\2, \1/' # Doe, John # Ampersand as matched pattern echo "hello world" | sed 's/.*/(&)/' # (hello world) echo "123-456-7890" | sed 's/[0-9]\+/(&)/g' # (123)-(456)-(7890)

Delete (d Command)

# Delete lines sed '3d' file.txt # Delete line 3 sed '5,10d' file.txt # Delete lines 5-10 sed '/pattern/d' file.txt # Delete lines matching pattern sed '/^#/d' file.txt # Delete comments (lines starting with #) sed '/^$/d' file.txt # Delete empty lines sed '/start/,/end/d' file.txt # Delete range between patterns # Complex deletions sed '1,5d' data.txt # Delete first 5 lines sed '$d' file.txt # Delete last line sed '/^[[:space:]]*$/d' file # Delete lines with only whitespace sed '/ERROR/,+3d' log.txt # Delete line with ERROR and next 3 lines # Examples cat > test.txt << EOF Line 1 Line 2 Line 3 Line 4 Line 5 EOF sed '2,4d' test.txt # Output: Line 1, Line 5 sed '/2/,/4/d' test.txt # Delete lines containing 2 through 4

Print (p Command)

# Print specific lines sed -n '5p' file.txt # Print line 5 sed -n '10,20p' file.txt # Print lines 10-20 sed -n '/pattern/p' file.txt # Print lines matching pattern sed -n '/start/,/end/p' file # Print range between patterns # Print with line numbers sed -n '/error/{=;p}' log.txt # Print line number and matching line sed -n '/error/{=;p}' log.txt | sed 'N;s/\n/ /' # Format: "number line" # Multiple print conditions sed -n '/ERROR/p; /WARNING/p' log.txt # Print ERROR or WARNING lines sed -n '1,5p; /important/p' file.txt # Print first 5 lines and important lines # Examples cat data.txt | sed -n '/^[0-9]/p' # Print lines starting with number sed -n '/[A-Z]\{5,\}/p' file.txt # Print lines with 5+ uppercase letters

3. Address Ranges and Patterns

Line Addressing

# Single line addresses sed '5 s/old/new/' file.txt # Replace on line 5 only sed '1,10 s/old/new/g' file.txt # Replace on lines 1-10 sed '$ s/old/new/' file.txt # Replace on last line only # Pattern addresses sed '/^Start/,/^End/ s/old/new/' file.txt # Replace within range sed '/section/,/endsection/ d' file.txt # Delete section sed '/BEGIN/,/END/!d' file.txt # Keep only between BEGIN and END # Address negation sed '/pattern/!s/old/new/' file.txt # Replace where pattern doesn't match sed '5,10!d' file.txt # Delete all except lines 5-10 # Step addressing (GNU sed) sed '0~2 s/old/new/' file.txt # Replace on even lines sed '1~2 s/old/new/' file.txt # Replace on odd lines sed '0~5d' file.txt # Delete every 5th line # Examples sed '/debug/,/enddebug/ s/^/# /' config.c # Comment out debug section sed '/^[[:space:]]*$/d' file.txt # Delete empty lines

Regular Expressions in sed

# Basic regex (BRE) sed 's/^[[:space:]]*//' file.txt # Remove leading whitespace sed 's/[[:space:]]*$//' file.txt # Remove trailing whitespace sed 's/[[:alpha:]]\+/WORD/g' file.txt # Replace words with WORD # Extended regex with -r or -E sed -r 's/[0-9]{3}-[0-9]{4}/(xxx)/' phones.txt sed -E 's/[[:alnum:]_]+@[[:alnum:]_]+\.[a-z]{2,3}/[EMAIL]/' emails.txt # Common regex patterns sed -n '/^#/!p' config.txt # Print non-comment lines sed '/^$/d' file.txt # Remove empty lines sed 's/[[:punct:]]//g' file.txt # Remove punctuation sed 's/[[:upper:]]/\L&/g' file.txt # Convert to lowercase sed 's/[[:lower:]]/\U&/g' file.txt # Convert to uppercase # Complex regex examples sed -E 's/\b([0-9]{1,3}\.){3}[0-9]{1,3}\b/IP/' file # Replace IP addresses sed -E 's/^([^:]+):.*/\1/' passwd # Extract usernames sed -E 's/<[^>]+>//g' html.html # Remove HTML tags

4. Advanced Commands

Transform (y Command)

# Character translation sed 'y/abc/xyz/' file.txt # a->x, b->y, c->z sed 'y/[A-Z]/[a-z]/' file.txt # Convert uppercase to lowercase sed 'y/ \t/__/' file.txt # Convert spaces/tabs to underscores # Examples echo "hello world" | sed 'y/helo/HELO/' # HELLO wOrLd echo "123-456-7890" | sed 'y/-/ /' # 123 456 7890 # Note: y does character-by-character translation, not patterns sed 'y/123456789/987654321/' numbers.txt # Reverse digits

Hold Space and Pattern Space

# Hold space commands # h - copy pattern space to hold space # H - append pattern space to hold space # g - copy hold space to pattern space # G - append hold space to pattern space # x - exchange pattern and hold spaces # Reverse lines of a file (tac equivalent) sed -n '1!G;h;$p' file.txt # Double-space a file sed 'G' file.txt # Join lines sed 'N;s/\n/ /' file.txt # Join first two lines sed 'N;N;s/\n/ /g' file # Join every three lines # Advanced examples # Print even lines sed -n 'n;p' file.txt # Print odd lines sed -n 'p;n' file.txt # Remove duplicate consecutive lines sed '$!N; /^\(.*\)\n\1$/!P; D' file.txt # Extract text between markers sed -n '/START/{:a;n;/END/q;p;ba}' file.txt

Branching and Conditional Operations

# Branching commands # :label - define a label # b label - branch to label # t label - branch if substitution succeeded # Example: Multiple replacements sed ':a;s/foo/bar/;ta' file.txt # Repeat substitution until no more changes # Conditional processing sed '/error/{ s/error/ERROR/ s/warning/WARNING/ }' log.txt # Complex example - convert to title case sed 's/.*/\L&/; s/[a-z]*/\u&/g' file.txt # Loop until no more changes sed ':start; s/[0-9][0-9]/&/; t start' numbers.txt

5. In-Place Editing

Basic In-Place Operations

# Edit file in-place (GNU sed) sed -i 's/old/new/g' file.txt # Create backup before editing sed -i.bak 's/old/new/g' file.txt # Multiple files sed -i 's/old/new/g' file1.txt file2.txt file3.txt # With backup extension sed -i'.original' 's/old/new/g' config.txt # Examples # Replace all instances of "foo" with "bar" in all .txt files sed -i 's/foo/bar/g' *.txt # Remove trailing whitespace from multiple files sed -i 's/[[:space:]]*$//' *.c *.h # Add header to files sed -i '1i # This file was modified' *.txt

Safe In-Place Editing

#!/bin/bash # Safe in-place edit with backup and verification safe_sed() { local file="$1" local backup="${file}.bak.$(date +%Y%m%d-%H%M%S)" shift # Create backup cp "$file" "$backup" # Perform edit sed -i "$@" "$file" # Ask for verification echo "Changes made. Check $file" echo "Backup saved at $backup" echo -n "Keep changes? (y/n): " read answer if [ "$answer" != "y" ]; then cp "$backup" "$file" echo "Restored from backup" fi } # Edit with diff preview preview_sed() { local file="$1" shift cp "$file" /tmp/sed_preview sed -i /tmp/sed_preview "$@" echo "=== Changes ===" diff -u "$file" /tmp/sed_preview echo "===============" echo -n "Apply changes? (y/n): " read answer if [ "$answer" = "y" ]; then cp /tmp/sed_preview "$file" fi rm /tmp/sed_preview }

6. Multi-line Operations

Working with Multiple Lines

# Join lines sed ':a;N;$!ba;s/\n/ /g' file.txt # Join all lines sed 'N;N;s/\n/ /g' file.txt # Join three lines at a time # Process paragraphs sed -n '/^$/{p;d}; H; $ {x;s/\n/ /g;p}' file.txt # Join paragraph lines # Multi-line pattern matching sed '/start/,/end/ { /start/d /end/d s/.*/CHANGED/ }' file.txt # Examples # Convert CSV to single line sed ':a;N;$!ba;s/\n/,/g' data.csv # Wrap text at column 80 sed 's/.\{80\}/&\n/g' longlines.txt # Remove HTML tags across lines sed ':a;N;$!ba;s/<[^>]*>//g' html.txt

Complex Multi-line Examples

#!/bin/bash # Extract function definitions from C file extract_functions() { sed -n ' /^[a-zA-Z_][a-zA-Z0-9_]* *(.*)/ { :func N /\n[{}]/!b func p } ' "$1" } # Join lines ending with backslash join_continuations() { sed ':a;/\\$/N;s/\\\n//;ta' "$1" } # Wrap long lines wrap_lines() { local width="${1:-80}" shift sed "s/.\{$width\}/&\n/g" "$@" } # Number paragraphs number_paragraphs() { sed '/^$/ { x s/^$/Paragraph 0/ s/Paragraph \([0-9]*\)/Paragraph \1/ s/.*/&/ x } /^[^ ]/ { H x s/^\(.*\)\n\(.*\)/\2 \1/ s/Paragraph \([0-9]*\)/\1. / s/\([0-9]*\. \)\(.*\)/\1\2/ s/^[0-9]*\. // }' "$1" }

7. Practical Examples

Text Processing

#!/bin/bash # Remove comments and blank lines clean_config() { sed -e 's/#.*$//' -e '/^$/d' "$1" } # Extract email addresses extract_emails() { sed -n 's/.*\([a-zA-Z0-9._%+-]\+@[a-zA-Z0-9.-]\+\.[a-zA-Z]\{2,\}\).*/\1/p' "$1" } # Convert to title case title_case() { sed 's/.*/\L&/; s/[a-z]*/\u&/g' "$1" } # Add line numbers add_line_numbers() { sed = "$1" | sed 'N;s/\n/ /' } # Format JSON (simple) format_json() { sed 's/[{,]/&\n /g; s/}/&\n/g' "$1" } # URL decode urldecode() { sed 's/%/\\x/g' "$1" | while read -r line; do echo -e "$line" done } # Examples usage # clean_config /etc/config.ini # extract_emails contacts.txt # title_case names.txt

File and Directory Operations

#!/bin/bash # Rename files in directory (dry run) rename_files() { local pattern="$1" local replacement="$2" for file in *; do newname=$(echo "$file" | sed "s/$pattern/$replacement/") if [ "$newname" != "$file" ]; then echo "mv '$file' '$newname'" fi done } # Batch rename with sed batch_rename() { local pattern="$1" local replacement="$2" shift 2 for file in "$@"; do newname=$(echo "$file" | sed "s/$pattern/$replacement/") if [ "$newname" != "$file" ]; then mv -v "$file" "$newname" fi done } # Update file extensions change_extension() { local old_ext="$1" local new_ext="$2" for file in *."$old_ext"; do [ -f "$file" ] || continue base=$(echo "$file" | sed "s/\.$old_ext$//") mv -v "$file" "$base.$new_ext" done }

Log Processing

#!/bin/bash # Filter log by date range filter_log_by_date() { local start_date="$1" local end_date="$2" local logfile="$3" sed -n "/^$start_date/,/^$end_date/p" "$logfile" } # Extract error messages with context extract_errors() { local logfile="$1" local context="${2:-2}" sed -n " /ERROR/ { = N N p } " "$logfile" | sed 'N;s/\n/ /g' } # Summarize log levels log_summary() { sed -n 's/.*\(ERROR\|WARN\|INFO\).*/\1/p' "$1" | sort | uniq -c } # Remove timestamp prefix remove_timestamp() { sed 's/^\[[0-9-]* [0-9:]*\] //' "$1" } # Format log for better readability format_log() { sed 's/\[\([^]]*\)\]/\n[\1]\n /g' "$1" | sed '/^$/d' }

CSV Processing

#!/bin/bash # Extract specific column extract_column() { local col="$1" local file="$2" sed "s/[^,]*\(,[^,]*\)\{$((col-1))\},//" "$file" } # Change delimiter change_delimiter() { local old="$1" local new="$2" local file="$3" sed "s/$old/$new/g" "$file" } # Remove header remove_header() { sed '1d' "$1" } # Add header add_header() { local header="$1" local file="$2" sed "1i $header" "$file" } # Filter rows by value in column filter_by_column() { local col="$1" local value="$2" local file="$3" sed -n "/\([^,]*,\?\)\{$((col-1))\}$value/p" "$file" } # Examples # extract_column 3 data.csv # change_delimiter ',' '|' data.csv

8. sed Scripts

Creating sed Scripts

#!/bin/bash # Create a sed script file cat > cleanup.sed << 'EOF' # Remove comments s/#.*$// # Remove empty lines /^$/d # Trim whitespace s/^[[:space:]]*// s/[[:space:]]*$// # Normalize spaces s/[[:space:]]\+/ /g # Convert to lowercase s/[A-Z]/\L&/g EOF # Use the script sed -f cleanup.sed input.txt > output.txt # Another example: markdown to plain text cat > md2txt.sed << 'EOF' # Remove headers s/^#\+ //g # Remove emphasis s/\*\*\([^*]*\)\*\*/\1/g s/\*\([^*]*\)\*/\1/g # Remove links s/\[\([^]]*\)\]([^)]*)/\1/g # Remove code blocks /^```/,/^```/d # Remove images s/!\[[^]]*\]([^)]*)//g # Convert lists s/^[*-] /• /g EOF

Complex sed Script Example

#!/bin/bash # HTML to text converter cat > html2text.sed << 'EOF' #!/usr/bin/sed -f # Remove script and style sections /<script/,/<\/script>/d /<style/,/<\/style>/d # Remove HTML tags s/<[^>]*>//g # Replace common entities s/&nbsp;/ /g s/&lt;/</g s/&gt;/>/g s/&amp;/\&/g s/&quot;/"/g # Handle multiple blank lines /^$/{ N /^\n$/d } # Trim lines s/^[[:space:]]*// s/[[:space:]]*$// # Join lines that don't end with punctuation :join N /\n[^A-Z]/s/\n/ / t join # Add paragraph breaks s/\. /. \n\n/g EOF # Make executable and use chmod +x html2text.sed ./html2text.sed webpage.html > webpage.txt

9. Performance Optimization

Efficient sed Usage

#!/bin/bash # Use single sed command instead of multiple # Bad: multiple processes cat file.txt | sed 's/foo/bar/' | sed 's/baz/qux/' > output.txt # Good: single sed with multiple commands sed 's/foo/bar/; s/baz/qux/' file.txt > output.txt # Use sed -e for multiple expressions sed -e 's/foo/bar/' -e 's/baz/qux/' file.txt # Avoid unnecessary pipes # Bad: useless use of cat cat file.txt | sed 's/foo/bar/' # Good: sed reads file directly sed 's/foo/bar/' file.txt # Use address ranges to limit processing sed '1000,2000 s/foo/bar/g' largefile.txt # Only process relevant section # Exit early when done sed '/stop/q' file.txt # Stop processing after finding "stop" # Performance comparison time sed 's/foo/bar/g' largefile.txt > /dev/null time sed -n 's/foo/bar/gp' largefile.txt > /dev/null # Usually slower

Handling Large Files

#!/bin/bash # Process large files in chunks process_large_file() { local file="$1" local chunk_size=10000 split -l $chunk_size "$file" chunk_ for chunk in chunk_*; do sed -i 's/foo/bar/g' "$chunk" & # Process in parallel [ $(jobs | wc -l) -ge 4 ] && wait done wait cat chunk_* > "${file}.processed" rm chunk_* } # Stream processing for very large files stream_process() { while IFS= read -r line; do echo "$line" | sed 's/foo/bar/' done < "$1" } # Use sed with buffering sed -u 's/foo/bar/' largefile.txt # Unbuffered output # Monitor progress with pv pv largefile.txt | sed 's/foo/bar/' > output.txt

10. Error Handling and Debugging

Debugging sed Scripts

#!/bin/bash # Debug with line numbering sed -n '/pattern/{ = p }' file.txt # Trace sed execution (GNU sed) sed --debug 's/foo/bar/' file.txt # Validate sed script validate_sed() { local script="$1" local testfile="/tmp/sed_test.$$" echo "test line" > "$testfile" if sed -f "$script" "$testfile" >/dev/null 2>&1; then echo "Script syntax OK" else echo "Script has errors" sed -f "$script" "$testfile" 2>&1 fi rm "$testfile" } # Dry run (show what would be changed) sed_dry_run() { local file="$1" shift diff -u "$file" <(sed "$@" "$file") } # Verbose mode for complex scripts sed_verbose() { local script="$1" shift echo "=== Input ===" cat "$1" echo "=== Processing ===" sed -n "$script" "$1" echo "=== Output ===" sed "$script" "$1" }

Common Error Patterns

#!/bin/bash # Error: Unescaped delimiter in pattern # sed 's/foo/bar/baz' file.txt # Error # Correct: Use different delimiter or escape sed 's|foo/bar|baz|' file.txt sed 's/foo\/bar/baz/' file.txt # Error: Missing closing quote # sed 's/foo/bar file.txt # Error # Error: Invalid range # sed '10,5d' file.txt # Error (start > end) # Error: Using non-GNU extensions on non-GNU sed # sed -i 's/foo/bar/' file.txt # Works on GNU, may fail on BSD # Fix for cross-platform if sed --version 2>/dev/null | grep -q GNU; then sed -i 's/foo/bar/' file.txt else sed -i '' 's/foo/bar/' file.txt # BSD sed fi

11. Integration with Other Commands

Pipeline Integration

#!/bin/bash # With grep grep "ERROR" log.txt | sed 's/^/*** /' # Highlight errors # With awk awk '{print $1}' data.txt | sed 's/[0-9]/X/g' # Mask numbers # With sort sed 's/^[[:space:]]*//' file.txt | sort -u # Sort unique lines # With find find . -name "*.txt" -exec sed -i 's/foo/bar/g' {} \; # With xargs grep -l "pattern" *.txt | xargs sed -i 's/old/new/g' # Complex pipeline ps aux | sed -n '/^USER/d; s/ */ /g; p' | sort -k3 -rn | head -10 # Monitor and process tail -f log.txt | sed -u 's/ERROR/\x1b[31m&\x1b[0m/'

Shell Functions with sed

#!/bin/bash # Create reusable sed functions trim() { sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//' } lowercase() { sed 's/[A-Z]/\L&/g' } uppercase() { sed 's/[a-z]/\U&/g' } slugify() { sed -e 's/[^[:alnum:]]/-/g' -e 's/--*/-/g' -e 's/^-//' -e 's/-$//' | lowercase } strip_ansi() { sed 's/\x1b\[[0-9;]*m//g' } # Usage echo " Hello World " | trim # "Hello World" echo "Hello World" | lowercase # "hello world" echo "Hello World" | uppercase # "HELLO WORLD" echo "My File Name.txt" | slugify # "my-file-name-txt"

12. Advanced Patterns and Techniques

Text Transformations

#!/bin/bash # Rot13 encoding rot13() { sed 'y/ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz/NOPQRSTUVWXYZABCDEFGHIJKLMnopqrstuvwxyzabcdefghijklm/' } # Base64-like encoding (simple) simple_encode() { sed 's/./\&#\1;/g' | sed 's/[^ ]/ /g' } # Word wrap word_wrap() { local width="${1:-80}" shift sed "s/.\{$width\}/&\n/g" "$@" } # Justify text justify() { local width="${1:-80}" shift sed -e ":a" -e "s/^.\{1,$((width-1))\}$/& /" -e "ta" "$@" } # Reverse lines (tac emulation) reverse_lines() { sed -n '1!G;h;$p' "$1" } # Reverse characters on each line reverse_chars() { sed '/\n/!G;s/\(.\)\(.*\n\)/&\2\1/;//D;s/.//' "$1" }

Code Generation and Transformation

#!/bin/bash # Add line numbers to code number_code() { sed = "$1" | sed 'N;s/^/ /; s/\n/ /' } # Comment out code blocks comment_block() { local start="$1" local end="$2" local file="$3" sed "/$start/,/$end/ s/^/# /" "$file" } # Uncomment code blocks uncomment_block() { local start="$1" local end="$2" local file="$3" sed "/$start/,/$end/ s/^# //" "$file" } # Convert indentation spaces_to_tabs() { sed 's/^\( *\)/\1/' | sed 's/ /\t/g' # Simple version } tabs_to_spaces() { local count="${1:-4}" sed "s/\t/$(printf '%*s' $count)/g" } # Extract functions from source extract_functions() { sed -n '/^[a-zA-Z_][a-zA-Z0-9_]* *(.*)/,/^}/p' "$1" }

13. sed One-Liners

Common One-Liners

# File operations sed -i '1i Header line' file.txt # Insert line at top sed -i '$a Footer line' file.txt # Append line at bottom sed '2r insert.txt' file.txt # Insert file at line 2 sed -n '2,$p' file.txt # Print from line 2 to end sed -n '1!G;h;$p' file.txt # Reverse lines (tac) # Text transformations sed 's/[[:punct:]]//g' file.txt # Remove punctuation sed 's/[0-9]//g' file.txt # Remove digits sed 's/[[:alpha:]]//g' file.txt # Remove letters sed 's/\(.*\)/\L\1/' file.txt # Lowercase sed 's/\(.*\)/\U\1/' file.txt # Uppercase sed 's/^./\u&/' file.txt # Capitalize first letter # Whitespace handling sed 's/^[[:space:]]*//' file.txt # Remove leading spaces sed 's/[[:space:]]*$//' file.txt # Remove trailing spaces sed 's/[[:space:]]\+/ /g' file.txt # Normalize spaces sed '/^$/d' file.txt # Remove empty lines sed '/./,$!d' file.txt # Remove leading blank lines sed -n '/[^[:space:]]/p' file.txt # Remove blank lines # Line manipulation sed 'n;d' file.txt # Print odd lines sed -n 'n;p' file.txt # Print even lines sed '$!N;$!D' file.txt # Print last 2 lines sed 'N;s/\n/ /' file.txt # Join lines sed ':a;N;$!ba;s/\n/ /g' file.txt # Join all lines # Pattern matching sed -n '/pattern/p' file.txt # Print matching lines sed '/pattern/d' file.txt # Delete matching lines sed '/pattern/!d' file.txt # Delete non-matching sed -n '/start/,/end/p' file.txt # Print range sed '/start/,/end/d' file.txt # Delete range # Substitution sed 's/foo/bar/' file.txt # First occurrence sed 's/foo/bar/2' file.txt # Second occurrence sed 's/foo/bar/g' file.txt # All occurrences sed 's/foo/bar/gi' file.txt # Case-insensitive sed 's/\(.*\)/[\1]/' file.txt # Wrap line in brackets

Practical One-Liners

# System administration df -h | sed '1d' | sort -k5 -rn # Sort filesystems by usage ps aux | sed '1d' | sort -k3 -rn # Sort processes by CPU ifconfig | sed -n '/inet /p' # Show IP addresses last | sed -n '/still/p' # Show logged-in users # Text processing sed -n '1,5p' /etc/passwd # First 5 lines of passwd sed -n '/^root/=' /etc/passwd # Line number of root sed 's/:.*//' /etc/passwd | sort # List all users sed 's/.*:\(.*\):.*/\1/' /etc/passwd | sort -u # List all shells # File conversion sed 's/\r$//' dosfile.txt > unixfile.txt # DOS to Unix sed 's/$/\r/' unixfile.txt > dosfile.txt # Unix to DOS sed 's/\t/ /g' file.txt # Tabs to spaces sed 's/ /\t/g' file.txt # Spaces to tabs # HTML/XML sed 's/<[^>]*>//g' file.html # Remove HTML tags sed -n 's/.*<title>\(.*\)<\/title>.*/\1/p' page # Extract title sed 's/&amp;/\&/g; s/&lt;/</g; s/&gt;/>/g' # Decode HTML entities # CSV sed 's/,/|/g' file.csv # Change delimiter sed 's/"\([^"]*\)"/\1/g' file.csv # Remove quotes sed '1d' file.csv # Remove header

14. sed vs Other Tools

Comparison with awk, grep, cut

#!/bin/bash # sed vs grep grep "pattern" file.txt # Find lines with pattern sed -n '/pattern/p' file.txt # Same as above grep -v "pattern" file.txt # Lines without pattern sed -n '/pattern/!p' file.txt # Same as above # sed vs awk awk '{print $2}' file.txt # Print second field sed 's/[^[:space:]]*[[:space:]]*\([^[:space:]]*\).*/\1/' file.txt # Complex # sed vs cut cut -d: -f1 /etc/passwd # First field sed 's/:.*//' /etc/passwd # Same as above # sed vs tr tr 'A-Z' 'a-z' < file.txt # Lowercase sed 's/[A-Z]/\L&/g' file.txt # Same as above # Performance comparison time grep "pattern" largefile.txt > /dev/null time sed -n '/pattern/p' largefile.txt > /dev/null # grep is usually faster for simple pattern matching

When to Use sed

# sed is best for: # 1. Simple substitutions sed 's/old/new/g' file.txt # 2. Line-based operations sed '1,10d' file.txt # 3. Stream editing in pipelines command | sed 's/^/ /' | another_command # 4. Simple scripts sed -f script.sed file.txt # 5. In-place editing sed -i 's/foo/bar/g' *.txt # Use awk for: # - Field-based processing # - Complex calculations # - Data reporting # Use perl for: # - Very complex regex # - Multi-line patterns # - One-liners with advanced features

15. Best Practices and Tips

sed Best Practices

#!/bin/bash # 1. Quote your sed commands # Good sed 's/foo/bar/' file.txt # Bad (if file has special characters) sed s/foo/bar/ file.txt # 2. Use different delimiters for paths sed 's|/usr/local|/opt|' paths.txt # 3. Test with -n first sed -n 's/pattern/replacement/p' file.txt # Preview changes # 4. Create backups before in-place edits sed -i.bak 's/foo/bar/g' important.txt # 5. Use addresses to limit scope sed '100,200 s/foo/bar/g' largefile.txt # 6. Comment complex scripts sed ' # Remove comments s/#.*$// # Remove empty lines /^$/d ' file.txt # 7. Use extended regex for readability sed -E 's/[0-9]{3}-[0-9]{4}/(xxx)/' phones.txt # 8. Handle errors gracefully if ! sed -i 's/foo/bar/' file.txt 2>/dev/null; then echo "sed failed" fi

Performance Tips

#!/bin/bash # 1. Use single sed instead of multiple # Slow sed 's/foo/bar/' file.txt | sed 's/baz/qux/' > out.txt # Fast sed 's/foo/bar/; s/baz/qux/' file.txt > out.txt # 2. Use addresses to limit processing sed '1000,2000 s/foo/bar/' largefile.txt # 3. Avoid unnecessary pipes # Slow cat file.txt | sed 's/foo/bar/' > out.txt # Fast sed 's/foo/bar/' file.txt > out.txt # 4. Use -n when you don't need all output sed -n '/pattern/p' file.txt # 5. For huge files, consider splitting split -l 100000 largefile.txt part for f in part*; do sed -i 's/foo/bar/' "$f" & done wait cat part* > processed.txt # 6. Use -u for unbuffered output in pipelines tail -f log.txt | sed -u 's/ERROR/!!!/' | grep -v "DEBUG"

Common Pitfalls and Solutions

#!/bin/bash # Pitfall 1: Special characters in pattern # Wrong sed 's/$10/amount/' file.txt # $ is special # Correct sed 's/\$10/amount/' file.txt # Pitfall 2: Using / in patterns # Wrong sed 's/usr/local/opt/' file.txt # / in pattern # Correct sed 's|/usr/local|/opt|' file.txt # Pitfall 3: In-place on symlinks (GNU sed) sed -i 's/foo/bar/' symlink # Replaces symlink with regular file # Better sed -i 's/foo/bar/' "$(readlink -f symlink)" # Pitfall 4: Portability between GNU and BSD sed # GNU sed sed -i 's/foo/bar/' file.txt # BSD sed (macOS) sed -i '' 's/foo/bar/' file.txt # Portable version if sed --version 2>/dev/null | grep -q GNU; then sed -i 's/foo/bar/' file.txt else sed -i '' 's/foo/bar/' file.txt fi # Pitfall 5: Newline handling # Won't match across lines sed '/start/,/end/ s/foo/bar/' file.txt # Use N for multi-line sed '/start/{N;s/foo\nbar/bar\nfoo/}' file.txt

Conclusion

sed is an incredibly powerful stream editor that every Unix/Linux user should master:

Key Takeaways

Stream Processing: sed processes text line by line efficiently
Pattern Matching: Powerful regex support for text manipulation
In-Place Editing: Can modify files directly with -i option
Scripting: Complex transformations can be scripted
Pipeline Friendly: Works seamlessly with other Unix commands
Portability: Available on virtually all Unix-like systems

Command Summary

Command	Description
`s/pattern/replacement/`	Substitute
`/pattern/d`	Delete matching lines
`/pattern/p`	Print matching lines
`5,10d`	Delete lines 5-10
`y/abc/xyz/`	Translate characters
`=`	Print line numbers
`r file`	Read file
`w file`	Write to file
`q`	Quit

Best Practices Checklist

[ ] Quote sed commands to prevent shell interpretation
[ ] Test with -n before actual execution
[ ] Use different delimiters for paths
[ ] Create backups with -i.bak
[ ] Use addresses to limit scope
[ ] Comment complex scripts
[ ] Handle errors gracefully
[ ] Consider portability between GNU and BSD sed
[ ] Use extended regex (-E or -r) for complex patterns
[ ] Avoid useless use of cat

Common Use Cases

Text substitution in files and streams
Filtering lines by patterns
Data extraction from structured text
Code generation and transformation
Log processing and analysis
File format conversion
Batch renaming of files
Configuration file updates

The sed command's simplicity and power make it an indispensable tool for text processing tasks. While it has a learning curve, mastering sed will significantly enhance your productivity in the Unix/Linux environment.

Complete Bash Exercises with Solutions

Practice Bash scripting with solved exercises to improve your command-line and scripting skills.
Link: https://macronepal.com/complete-bash-exercises-with-solutions/

Complete Guide to Bash Crontab Command

Learn how to schedule and automate tasks efficiently using the Bash crontab command.
Link: https://macronepal.com/complete-guide-to-bash-crontab-command-schedule-tasks/

Complete Guide to Bash Arrays

Understand how to create, manage, and use arrays in Bash scripting.
Link: https://macronepal.com/complete-guide-to-bash-arrays/

Bash Quiz: Test Your Knowledge

Test your Bash scripting knowledge with quizzes and practical questions.
Link: https://macronepal.com/bash-quiz-test-your-knowledge/

Complete Guide to Bash Functions

Learn how to write reusable Bash functions for cleaner and efficient scripts.
Link: https://macronepal.com/complete-guide-to-bash-functions/

Complete Guide to Bash Loops

Master loops in Bash for repetitive tasks and automation.
Link: https://macronepal.com/complete-guide-to-bash-loops/

Complete Guide to Bash If-Else Statements

Learn decision-making in Bash using conditional statements.
Link: https://macronepal.com/complete-guide-to-bash-ifelse-statements/

Complete Guide to Bash Operators

Explore arithmetic, logical, and comparison operators in Bash.
Link: https://macronepal.com/complete-guide-to-bash-operators/

Complete Guide to Bash Data Types

Understand variables and different data types used in Bash scripting.
Link: https://macronepal.com/complete-guide-to-bash-data-types/

Complete Guide to Bash Variables

Learn how to declare, assign, and use variables in Bash scripts.
Link: https://macronepal.com/complete-guide-to-bash-variables/

Complete Guide to Bash Scripting

A complete introduction to writing Bash scripts for automation.
Link: https://macronepal.com/complete-guide-to-bash-scripting/

Complete Guide to Basic Bash Syntax

Learn the fundamental syntax needed to write Bash commands and scripts.
Link: https://macronepal.com/complete-guide-to-basic-bash-syntax/

Complete Guide to Bash Chgrp Command

Learn how to change group ownership of files and directories.
Link: https://macronepal.com/complete-guide-to-bash-chgrp-command-change-group-ownership-2/

Complete Guide to Bash Chgrp Command (Alternate)

Another detailed explanation of Bash group ownership management.
Link: https://macronepal.com/complete-guide-to-bash-chgrp-command-change-group-ownership/

Complete Guide to Bash File Permissions and Ownership

Understand Linux file permissions and ownership management.
Link: https://macronepal.com/complete-guide-to-bash-file-permissions-and-ownership/

Complete Guide to Bash Chmod Command

Learn how to modify file permissions using chmod.
Link: https://macronepal.com/complete-guide-to-bash-chmod-command-change-file-permissions/

Complete Guide to Bash Chown Command

Change file ownership using the Bash chown command.
Link: https://macronepal.com/complete-guide-to-bash-chown-command-change-file-ownership/

Complete Guide to Bash SCP Command

Securely transfer files between systems using SCP.
Link: https://macronepal.com/complete-guide-to-bash-scp-command-secure-copy/

Complete Guide to Bash SSH Command

Learn secure remote access and server management using SSH.
Link: https://macronepal.com/complete-guide-to-bash-ssh-command/

Bash Wget Command Complete Guide

Download files from the internet using the wget command.
Link: https://macronepal.com/bash-wget-command-complete-guide-to-non-interactive-network-downloader/

Introduction to sed