Sort Text File and Remove Duplicate Lines

Copying and pasting text into a single text file can lead to duplicates. If you have ever needed a way to sort lines within a text file and then remove the duplicate lines, here is an easy method in Bash.

Method 1

sort infile.txt | uniq > outfile.txt

This uses the sort command to sort the file infile.txt alphabetically and then pipes the output to the uniq command to remove any duplicate lines. The result is redirected to outfile.txt.

Method 2

sort -u infile.txt > outfile.txt

This uses only the sort command with the -u (unique) option to remove duplicate lines.

The end result for both methods is an alphabetized text file with unique lines. Check the man pages for sort and uniq (man sort and man uniq) for more options.



  1. Leave a comment

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: