Removing Duplicate Lines in Bash

• author:: Nathan Acks

Quick-and-Dirty

cat $FILE | sort -u

This works in a pinch, but as a side effect will sort the lines lexicographicly.

Remove All But the First Occurrence of a Line

cat $FILE | cat -n | sort -uk2 | sort -nk1 | cut -f2-

Remove All But the Last Occurrence of a Line

cat $FILE | cat -n | sort -rk2 | sort -uk2 | sort -nk1 | cut -f2-

References

• Remove duplicate lines without sorting