How to remove duplicate lines in Bash

Quick-and-dirty

How to remove duplicate lines in Bash when line order isn’t important

cat $FILE | sort -u

This works in a pinch, but as a side effect will sort the lines lexicographically. Avoiding a sort is possible but more complex…

Link to original

Remove all but the first occurrence of a line

How to remove all but the first occurrence of a line in Bash

cat $FILE | cat -n | sort -uk2 | sort -nk1 | cut -f2-

Link to original

Remove all but the last occurrence of a line

How to remove all but the last occurrence of a line in Bash

cat $FILE | cat -n | sort -rk2 | sort -uk2 | sort -nk1 | cut -f2-

Link to original