Removing Duplicate Lines in Bash
- author:: Nathan Acks
Quick-and-Dirty
cat $FILE | sort -u
This works in a pinch, but as a side effect will sort the lines lexicographicly.
Remove All But the First Occurrence of a Line
cat $FILE | cat -n | sort -uk2 | sort -nk1 | cut -f2-
Remove All But the Last Occurrence of a Line
cat $FILE | cat -n | sort -rk2 | sort -uk2 | sort -nk1 | cut -f2-