Introduction to sort

The sort command is a powerful utility in Unix/Linux systems that sorts lines of text files. It can organize data alphabetically, numerically, by fields, and in various orders. Understanding sort is essential for data processing, log analysis, and general text manipulation.

Basic Syntax

sort [options] [file...]

If no file is specified, sort reads from standard input.

1. Basic Usage

Simple Sorting

# Sort alphabetically (default) sort file.txt # Sort in reverse order sort -r file.txt # Sort multiple files together sort file1.txt file2.txt # Save sorted output to new file sort file.txt > sorted.txt # Sort and display with line numbers sort file.txt | nl

Examples

$ cat fruits.txt banana apple cherry date $ sort fruits.txt apple banana cherry date $ sort -r fruits.txt date cherry banana apple

2. Common Options

Output Control

# -o: Write output to file (safe for in-place sorting) sort -o sorted.txt file.txt # -u: Unique lines (remove duplicates) sort -u file.txt # -c: Check if file is already sorted sort -c file.txt && echo "File is sorted" # -C: Check silently (exit code only) if sort -C file.txt; then echo "File is sorted" fi

Sorting Order

# -f: Ignore case (fold lower case to upper case) sort -f file.txt # -d: Dictionary order (ignore non-alphanumeric) sort -d file.txt # -i: Ignore non-printable characters sort -i file.txt # -M: Month sort (Jan, Feb, etc.) sort -M dates.txt # -h: Human numeric sort (2K, 1G, etc.) sort -h sizes.txt # -V: Version sort (natural version numbers) sort -V versions.txt

Examples with Options

$ cat mixed.txt Apple banana CHERRY date $ sort -f mixed.txt # Case-insensitive Apple banana CHERRY date $ cat versions.txt v1.0 v1.10 v1.2 v2.0 $ sort -V versions.txt # Version sort v1.0 v1.2 v1.10 v2.0

3. Numeric Sorting

Numeric Options

# -n: Numeric sort sort -n numbers.txt # -g: General numeric sort (handles scientific notation) sort -g scientific.txt # Sort by numeric value in reverse sort -nr numbers.txt # Sort with specific numeric format sort -n --numeric-sort numbers.txt

Examples

$ cat numbers.txt 10 2 33 5 111 $ sort -n numbers.txt 2 5 10 33 111 $ sort -nr numbers.txt # Reverse numeric 111 33 10 5 2 $ cat mixed_numbers.txt 10 apples 2 bananas 33 cherries 5 dates $ sort -n mixed_numbers.txt # Sorts by first field numerically 2 bananas 5 dates 10 apples 33 cherries

4. Field-Based Sorting

Field Specification

# -k: Sort by key field (1-based) sort -k2 file.txt # Sort by field 2 # -t: Specify field separator (default: whitespace) sort -t',' -k2 data.csv # Multiple keys sort -k1,1 -k2n file.txt # Sort by field1, then numerically by field2 # Field ranges sort -k2,3 file.txt # Sort by fields 2 through 3

Field Options

# n: numeric sort for specific field sort -k2n file.txt # r: reverse for specific field sort -k1r -k2n file.txt # b: ignore leading blanks sort -k2b file.txt # f: fold case for field sort -k2f file.txt

Complex Field Examples

$ cat data.txt John,30,NY Alice,25,CA Bob,30,TX Carol,25,NY # Sort by age (field 2) numerically sort -t',' -k2n data.txt Alice,25,CA Carol,25,NY John,30,NY Bob,30,TX # Sort by age then name sort -t',' -k2n -k1 data.txt Alice,25,CA Carol,25,NY Bob,30,TX John,30,NY # Sort by state then age sort -t',' -k3,3 -k2n data.txt Alice,25,CA Carol,25,NY John,30,NY Bob,30,TX $ cat /etc/passwd | sort -t':' -k3n | head -3 # Sort users by UID root:x:0:0:root:/root:/bin/bash daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin bin:x:2:2:bin:/bin:/usr/sbin/nologin

5. Advanced Sorting Techniques

Stable Sorting

# -s: Stable sort (preserves original order of equal lines) sort -s file.txt # Compare with unstable sort sort -k2 file.txt # May reorder equal fields sort -s -k2 file.txt # Preserves order of equal fields

Merge Sorted Files

# -m: Merge already sorted files sort -m sorted1.txt sorted2.txt > merged.txt # Check if merge is valid sort -c sorted1.txt && sort -c sorted2.txt && sort -m sorted1.txt sorted2.txt

Random Sort

# -R: Random sort (shuffle) sort -R file.txt # With seed for reproducibility sort -R --random-source=/dev/urandom file.txt

Examples

$ cat data.txt apple,10 banana,5 apple,20 cherry,15 banana,10 $ sort -t',' -k1 -s data.txt # Stable sort by fruit apple,10 apple,20 banana,5 banana,10 cherry,15 $ sort -R data.txt | head -3 # Random order (different each time) banana,5 apple,20 cherry,15

6. Practical Examples

Log File Analysis

# Sort log entries by timestamp sort -k1,2 access.log # Find most frequent IP addresses cut -d' ' -f1 access.log | sort | uniq -c | sort -nr | head -10 # Sort errors by count grep "ERROR" app.log | cut -d']' -f2 | sort | uniq -c | sort -nr # Sort by date in custom format sort -t' ' -k1.9,1.12n -k1.5,1.7M -k1.2,1.4n logfile.txt # Sorts by: year, month, day in format "10 Mar 2024"

Data Processing

# Sort CSV by column sort -t',' -k3n data.csv > sorted_data.csv # Sort with header preservation (head -1 data.csv && tail -n +2 data.csv | sort -k2) > sorted.csv # Remove duplicates after sorting sort -u file.txt > unique.txt # Sort by file size ls -l | sort -k5n # Sort processes by memory usage ps aux | sort -k4nr | head -10

Text Processing

# Sort words in a sentence echo "the quick brown fox" | tr ' ' '\n' | sort | tr '\n' ' ' # Sort paragraphs (using blank lines as record separators) tr '\n' '\0' < file.txt | sort -z | tr '\0' '\n' # Sort by line length awk '{print length, $0}' file.txt | sort -n | cut -d' ' -f2- # Sort and number lines sort file.txt | nl -w3 -s'. '

7. Script Examples

Log Analyzer Script

#!/bin/bash # analyze_logs.sh - Comprehensive log analysis with sort LOG_FILE="$1" if [[ ! -f "$LOG_FILE" ]]; then echo "Usage: $0 <logfile>" exit 1 fi echo "=== Log Analysis Report ===" echo "File: $LOG_FILE" echo "Date: $(date)" echo # Top IP addresses echo "Top IP Addresses:" grep -oE "\b([0-9]{1,3}\.){3}[0-9]{1,3}\b" "$LOG_FILE" | sort | uniq -c | sort -nr | head -10 | awk '{printf " %-15s %d\n", $2, $1}' echo # Top requested URLs echo "Top Requested URLs:" grep -oE "\"(GET|POST|PUT|DELETE) [^\"]+\"" "$LOG_FILE" | cut -d' ' -f2 | sort | uniq -c | sort -nr | head -10 | awk '{printf " %-40s %d\n", $2, $1}' echo # Response codes echo "Response Code Distribution:" grep -oE "\" [0-9]{3} " "$LOG_FILE" | awk '{print $2}' | sort | uniq -c | sort -nr | awk '{printf " %s: %d\n", $2, $1}' echo # Busy hours echo "Busy Hours:" grep -oE ":[0-9]{2}:[0-9]{2}:[0-9]{2}" "$LOG_FILE" | cut -d':' -f1 | sort | uniq -c | sort -nr | awk '{printf " %02d:00 - %d requests\n", $2, $1}'

CSV Processor Script

#!/bin/bash # process_csv.sh - Sort and analyze CSV files CSV_FILE="$1" SORT_COL="${2:-1}" DELIMITER="${3:-,}" if [[ ! -f "$CSV_FILE" ]]; then echo "Usage: $0 <csv_file> [sort_column] [delimiter]" exit 1 fi # Get header HEADER=$(head -1 "$CSV_FILE") # Sort data (excluding header) echo "=== Sorted CSV Data ===" echo "$HEADER" tail -n +2 "$CSV_FILE" | sort -t"$DELIMITER" -k"$SORT_COL"n echo echo "=== Column Statistics ===" echo # Calculate statistics for numeric columns COLS=$(echo "$HEADER" | awk -F"$DELIMITER" '{print NF}') for ((i=1; i<=COLS; i++)); do echo "Column $i Statistics:" tail -n +2 "$CSV_FILE" | cut -d"$DELIMITER" -f"$i" | { # Check if column is numeric if tail -n +2 "$CSV_FILE" | cut -d"$DELIMITER" -f"$i" | grep -qE '^[0-9]+(\.[0-9]+)?$'; then echo " Type: Numeric" echo " Min: $(sort -n | head -1)" echo " Max: $(sort -n | tail -1)" echo " Unique: $(sort -u | wc -l)" else echo " Type: Text" echo " Unique values: $(sort -u | wc -l)" echo " Top values:" sort | uniq -c | sort -nr | head -3 | while read count value; do echo " $value: $count" done fi } echo done

Duplicate Finder Script

#!/bin/bash # find_duplicates.sh - Find and sort duplicate lines find_duplicates() { local file="$1" local min_count="${2:-2}" if [[ ! -f "$file" ]]; then echo "File not found: $file" return 1 fi echo "Analyzing duplicates in: $file" echo "Minimum duplicate count: $min_count" echo # Find duplicates with counts sort "$file" | uniq -c | sort -nr | while read count line; do if [[ $count -ge $min_count ]]; then echo "$count: $line" fi done } # Find duplicate lines across multiple files find_cross_file_duplicates() { local pattern="$1" cat $pattern | sort | uniq -c | sort -nr | awk '$1 > 1 {print $0}' } case "$1" in single) find_duplicates "$2" "${3:-2}" ;; multi) find_cross_file_duplicates "$2" ;; *) echo "Usage: $0 {single|multi} <file/pattern> [min_count]" ;; esac

8. Sorting with Pipes and Redirections

Common Pipelines

# Sort and count unique lines cat file.txt | sort | uniq -c | sort -nr # Sort and page through less sort largefile.txt | less # Sort and display with head/tail sort -nr data.txt | head -20 # Sort multiple files and remove duplicates sort -u file1.txt file2.txt file3.txt # Sort and split output sort hugefile.txt | split -l 10000 - sorted_chunk_

Advanced Pipelines

# Find most common words in a file tr -cs '[:alpha:]' '\n' < file.txt | sort | uniq -c | sort -nr | head -20 # Sort files by modification time ls -l | sort -k6,7M -k7n # Sort process tree ps auxf | sort -k3nr # Sort by CPU usage # Sort and join files sort file1.txt > sorted1.txt sort file2.txt > sorted2.txt join sorted1.txt sorted2.txt

9. Performance Considerations

Large File Handling

# Use temporary directory for large sorts sort -T /tmp/large -o sorted.txt hugefile.txt # Increase buffer size sort -S 50% hugefile.txt # Use 50% of memory # Parallel sorting (with GNU sort) sort --parallel=4 hugefile.txt # External sort for very large files sort -S 1G -T /fast/disk/tmp hugefile.txt

Optimization Tips

# Use LC_ALL for faster sorting LC_ALL=C sort file.txt # Avoid unnecessary pipes # Bad: cat file.txt | sort # Good: sort file.txt # Use -u instead of sort | uniq # Bad: sort file.txt | uniq # Good: sort -u file.txt # Sort only necessary fields cut -f1,2 file.txt | sort > sorted_fields.txt

10. Sorting Different Data Types

IP Addresses

# Sort IP addresses sort -t. -k1,1n -k2,2n -k3,3n -k4,4n ips.txt # Using version sort for IPs sort -V ips.txt # Sort IPs with counts cut -d' ' -f1 access.log | sort | uniq -c | sort -nr

Dates and Times

# Sort dates (YYYY-MM-DD) sort -t'-' -k1n -k2n -k3n dates.txt # Sort timestamps sort -k1,2 logfile.txt # Sort by month sort -M -k2 data.txt # Sort by day of week awk '{print $1}' dates.txt | while read date; do date -d "$date" +"%u %s $date" done | sort -n | cut -d' ' -f3-

Human-Readable Sizes

# Sort file sizes ls -lh | sort -h -k5 # Sort by size with units du -sh * | sort -h # Sort free memory free -h | sort -h -k3

11. Special Sorting Cases

Version Numbers

# Sort version numbers sort -V versions.txt # Complex version sorting echo -e "1.0.0\n1.0.1\n1.1.0\n2.0.0-beta\n2.0.0" | sort -V # Sort package versions dpkg -l | grep ^ii | awk '{print $2"="$3}' | sort -V

Mixed Content

# Sort mixed alphanumeric sort -V mixed.txt # Sort with numeric prefix sort -n numbers.txt 2>/dev/null || sort -f mixed.txt # Sort ignoring leading symbols sort -b file.txt # Ignore leading blanks

Custom Sort Keys

# Sort by last field awk '{print $NF, $0}' file.txt | sort | cut -d' ' -f2- # Sort by word count while read line; do echo "$(echo "$line" | wc -w):$line" done < file.txt | sort -n | cut -d':' -f2-

12. Error Handling

Common Errors

# No such file sort nonexistent.txt # Output: sort: cannot read: nonexistent.txt: No such file or directory # Permission denied sort /etc/shadow # Output: sort: cannot read: /etc/shadow: Permission denied # Invalid option sort -z file.txt # if -z not supported

Error Handling in Scripts

# Check if sort succeeded if sort -o output.txt input.txt 2>/dev/null; then echo "Sorting successful" else echo "Sorting failed" fi # Validate input before sorting if [[ -r "$file" ]]; then sort "$file" else echo "Cannot read $file" >&2 exit 1 fi # Check if file is empty if [[ -s "$file" ]]; then sort "$file" else echo "File is empty" >&2 fi

13. Comparison with Related Commands

Command	Purpose	Example
sort	Sort lines	sort file.txt
uniq	Remove duplicates	uniq file.txt
shuf	Random permutation	shuf file.txt
comm	Compare sorted files	comm file1.txt file2.txt
join	Join on common field	join file1.txt file2.txt
tsort	Topological sort	tsort deps.txt

Combined Usage

# Find common lines in sorted files sort file1.txt > sorted1.txt sort file2.txt > sorted2.txt comm -12 sorted1.txt sorted2.txt # Find lines unique to file1 comm -23 sorted1.txt sorted2.txt # Join on first field sort -k1 file1.txt > s1.txt sort -k1 file2.txt > s2.txt join -t',' -1 1 -2 1 s1.txt s2.txt

14. Environment Variables

LC_COLLATE

Controls sort order (alphabetical rules):

# Default locale sort export LC_COLLATE=en_US.UTF-8 sort file.txt # C locale (ASCII order) export LC_COLLATE=C sort file.txt # Spanish locale (ll treated as separate letter) export LC_COLLATE=es_ES.UTF-8 sort spanish.txt

TMPDIR

Temporary directory for large sorts:

export TMPDIR=/fast/disk/tmp sort -S 1G hugefile.txt

15. GNU Extensions

GNU sort Additional Features

# --debug: Show sorting decisions sort --debug file.txt # --parallel: Parallel sorting sort --parallel=4 hugefile.txt # --compress-program: Compress temporary files sort --compress-program=gzip hugefile.txt # --batch-size: Number of merges sort --batch-size=16 hugefile.txt # --random-source: Random source for -R sort -R --random-source=/dev/urandom file.txt

16. Practical Applications

Database-like Operations

# SELECT DISTINCT column FROM table ORDER BY column cut -d',' -f2 data.csv | sort -u # SELECT column, COUNT(*) GROUP BY column ORDER BY count DESC cut -d',' -f1 data.csv | sort | uniq -c | sort -nr # ORDER BY multiple columns sort -t',' -k2,2 -k3n data.csv # LIMIT with OFFSET sort -k3n data.csv | tail -n +10 | head -5

Data Cleaning

# Remove duplicates and sort sort -u messy.txt > clean.txt # Sort and remove blank lines grep -v '^$' file.txt | sort > cleaned.txt # Normalize whitespace and sort tr -s ' ' < file.txt | sort # Sort and number lines sort file.txt | nl -w3 -s'. '

System Administration

# Sort users by UID cut -d: -f1,3 /etc/passwd | sort -t: -k2n # Sort running processes by memory ps aux | sort -k4nr | head -10 # Sort mounted filesystems by size df -h | sort -k2hr # Sort network connections by state netstat -tuna | sort -k6

17. Benchmarking

Performance Testing

# Time sort operations time sort largefile.txt > /dev/null # Compare different sort strategies time sort -S 1G largefile.txt > /dev/null time sort -T /tmp -S 1G largefile.txt > /dev/null # Test with different locales time LC_ALL=C sort largefile.txt > /dev/null time LC_ALL=en_US.UTF-8 sort largefile.txt > /dev/null

Memory Usage

# Monitor memory during sort /usr/bin/time -v sort largefile.txt > /dev/null # Check temporary file usage lsof -c sort

18. Quick Reference Card

Most Common Options

Option	Description
-n	Numeric sort
-r	Reverse order
-u	Unique lines
-k	Sort by key field
-t	Field separator
-o	Output file
-c	Check if sorted
-f	Ignore case
-M	Month sort
-h	Human numeric
-V	Version sort
-s	Stable sort
-R	Random sort
-m	Merge sorted files

Field Specifications

Syntax	Meaning
-k2	Sort by field 2 to end of line
-k2,2	Sort by field 2 only
-k2,3	Sort by fields 2 through 3
-k2n	Sort field 2 numerically
-k2nr	Sort field 2 numerically in reverse
-k2,2 -k3,3n	Sort by field 2, then numerically by field 3

Common Combinations

Command	Purpose
sort -n file	Numeric sort
sort -nr file	Reverse numeric sort
sort -u file	Unique sort
sort -k2 file	Sort by second field
sort -t',' -k3n file	Sort CSV by third column numerically
sort -V versions.txt	Version sort
sort -h sizes.txt	Human-readable sizes
sort -M dates.txt	Month sort
sort -R file	Randomize
sort -c file	Check if sorted

Conclusion

The sort command is an essential tool for data processing and organization:

Key Points Summary

1. Basic Sorting:

• Alphabetical sort (default)
• Numeric sort with -n
• Reverse order with -r
• Unique lines with -u

1. Field-Based Sorting:

• Specify fields with -k
• Set field separator with -t
• Multiple sort keys
• Field-specific options (n, r, b, f)

1. Special Sort Types:

• Version numbers (-V)
• Human-readable sizes (-h)
• Months (-M)
• Random (-R)

1. Performance:

• Use -T for temporary directory
• Set memory limit with -S
• Use LC_ALL=C for speed
• Parallel sorting with --parallel

Best Practices

1. Choose the right sort type for your data
2. Use field specifications for structured data
3. Consider performance for large files
4. Combine with other commands for complex processing
5. Use -u instead of sort | uniq for efficiency
6. Check sort results with -c before merging
7. Document complex sort commands in scripts

The sort command's versatility makes it indispensable for data processing, log analysis, and system administration. Mastering sort will significantly enhance your command-line data manipulation capabilities.

Complete Bash Exercises with Solutions

Practice Bash scripting with solved exercises to improve your command-line and scripting skills.
Link: https://macronepal.com/complete-bash-exercises-with-solutions/

Complete Guide to Bash Crontab Command

Learn how to schedule and automate tasks efficiently using the Bash crontab command.
Link: https://macronepal.com/complete-guide-to-bash-crontab-command-schedule-tasks/

Complete Guide to Bash Arrays

Understand how to create, manage, and use arrays in Bash scripting.
Link: https://macronepal.com/complete-guide-to-bash-arrays/

Bash Quiz: Test Your Knowledge

Test your Bash scripting knowledge with quizzes and practical questions.
Link: https://macronepal.com/bash-quiz-test-your-knowledge/

Complete Guide to Bash Functions

Learn how to write reusable Bash functions for cleaner and efficient scripts.
Link: https://macronepal.com/complete-guide-to-bash-functions/

Complete Guide to Bash Loops

Master loops in Bash for repetitive tasks and automation.
Link: https://macronepal.com/complete-guide-to-bash-loops/

Complete Guide to Bash If-Else Statements

Learn decision-making in Bash using conditional statements.
Link: https://macronepal.com/complete-guide-to-bash-ifelse-statements/

Complete Guide to Bash Operators

Explore arithmetic, logical, and comparison operators in Bash.
Link: https://macronepal.com/complete-guide-to-bash-operators/

Complete Guide to Bash Data Types

Understand variables and different data types used in Bash scripting.
Link: https://macronepal.com/complete-guide-to-bash-data-types/

Complete Guide to Bash Variables

Learn how to declare, assign, and use variables in Bash scripts.
Link: https://macronepal.com/complete-guide-to-bash-variables/

Complete Guide to Bash Scripting

A complete introduction to writing Bash scripts for automation.
Link: https://macronepal.com/complete-guide-to-bash-scripting/

Complete Guide to Basic Bash Syntax

Learn the fundamental syntax needed to write Bash commands and scripts.
Link: https://macronepal.com/complete-guide-to-basic-bash-syntax/

Complete Guide to Bash Chgrp Command

Learn how to change group ownership of files and directories.
Link: https://macronepal.com/complete-guide-to-bash-chgrp-command-change-group-ownership-2/

Complete Guide to Bash Chgrp Command (Alternate)

Another detailed explanation of Bash group ownership management.
Link: https://macronepal.com/complete-guide-to-bash-chgrp-command-change-group-ownership/

Complete Guide to Bash File Permissions and Ownership

Understand Linux file permissions and ownership management.
Link: https://macronepal.com/complete-guide-to-bash-file-permissions-and-ownership/

Complete Guide to Bash Chmod Command

Learn how to modify file permissions using chmod.
Link: https://macronepal.com/complete-guide-to-bash-chmod-command-change-file-permissions/

Complete Guide to Bash Chown Command

Change file ownership using the Bash chown command.
Link: https://macronepal.com/complete-guide-to-bash-chown-command-change-file-ownership/

Complete Guide to Bash SCP Command

Securely transfer files between systems using SCP.
Link: https://macronepal.com/complete-guide-to-bash-scp-command-secure-copy/

Complete Guide to Bash SSH Command

Learn secure remote access and server management using SSH.
Link: https://macronepal.com/complete-guide-to-bash-ssh-command/

Bash Wget Command Complete Guide

Download files from the internet using the wget command.
Link: https://macronepal.com/bash-wget-command-complete-guide-to-non-interactive-network-downloader/