uniq

Report or filter out repeated lines in a file.
Reads standard input comparing adjacent lines, and writes a copy of each unique input line to the standard output. The second and succeeding copies of identical adjacent input lines are not written. Repeated lines in the input will not be detected if they are not adjacent, so it may be necessary to sort the files first.

SYNTAX
       uniq [-c | -d | -u] [-f fields] [-s chars] [input_file [output_file]]

Options
     -c      Precede each output line with the count of the number of times
             the line occurred in the input, followed by a single space.

     -d      Distinct - Don't output lines that are not repeated in the input.

     -f fields
             Ignore the first fields in each input line when doing compar-
             isons.  A field is a string of non-blank characters separated
             from adjacent fields by blanks.  Field numbers are one based,
             i.e. the first field is field one.

     -s chars
             Ignore the first chars characters in each input line when doing
             comparisons.  If specified in conjunction with the -f option, the
             first chars characters after the first fields fields will be ig-
             nored.  Character numbers are one based, i.e. the first character
             is character one.

     -u      Don't output lines that are repeated in the input.
    

Notes
If additional arguments are specified on the command line, the first such argument is used as the name of an input file, the second is used as the name of an output file.

The historic +number and -number options have been deprecated but are still supported in this implementation.

"Human beings, who are almost unique in having the ability to learn from the experience of others, are also remarkable for their apparent disinclination to do so." - Douglas Adams

Related commands:

sort - Sort text files

Equivalent BASH command:

uniq - Report or filter out repeated lines in a file