Archive for May 16th, 2012

Sort Text File and Remove Duplicate Lines

Copying and pasting text into a single text file can lead to duplicates. If you have ever needed a way to sort lines within a text file and then remove the duplicate lines, here is an easy method in Bash.

Method 1

sort infile.txt | uniq > outfile.txt

This uses the sort command to sort the file infile.txt alphabetically and then pipes the output to the uniq command to remove any duplicate lines. The result is redirected to outfile.txt.

Method 2

sort -u infile.txt > outfile.txt

This uses only the sort command with the -u (unique) option to remove duplicate lines.

The end result for both methods is an alphabetized text file with unique lines. Check the man pages for sort and uniq (man sort and man uniq) for more options.



Leave a comment