Duplicate Line Remover

Lists pulled from logs, exports, or a copy-pasted spreadsheet column are full of repeats, and deleting them by hand is slow and easy to get wrong. This tool cleans the list and — just as usefully — tells you what was repeated and how often, so the removal is something you can check rather than something you have to trust.

Which copy survives is your choice. Keeping the first of each is the familiar behaviour and the right one for anything chronological. Keeping the last is what you want when a corrected row was appended below the one it replaces, where deduplicating from the top would throw the correction away. Two more options turn the tool around entirely: show only the entries that appear exactly once, or show only the ones that repeat, each with its count. A fifth collapses just the runs of identical lines sitting next to each other, the way the uniq command does, leaving separated repeats alone.

The harder question is what counts as the same. Exact comparison is the only unambiguous rule, but it is rarely what messy data needs, so the comparison is adjustable in six ways: fold case, ignore surrounding whitespace, ignore doubled spaces inside the line, drop punctuation and symbols, drop accents, and normalise Unicode so characters that look identical on screen but are stored differently still match. Every one of those changes only the comparison — the line that survives is written out exactly as you pasted it.

You can also narrow what is compared rather than loosening how. Compare a single column of a CSV and the whole row is still kept, so a contact list deduplicates on the email address while the name and phone number ride along. Compare only the part matching a regular expression and a log file deduplicates on the message with the timestamp ignored. Lists separated by commas, semicolons, tabs, or anything else work as well as one item per line, and the result can come back separated however suits the next step.

A CSV row is rarely duplicated in full. The same customer appears twice with a different phone number, or the same URL twice with a different tracking parameter — whole-line comparison sees two distinct rows and keeps both. Comparing a single column solves it: choose the separator and the column number, and the tool groups rows by that field alone while writing out the entire row it belongs to.

The pattern mode generalises the same idea to anything a regular expression can pick out. Give it a pattern with a capture group and the group becomes the comparison key; give it one without and the whole match is used. For log lines, a pattern that skips the first field compares the messages and ignores the timestamps, so a hundred repetitions of the same error collapse to one.

Both modes have to answer what happens to a line that has no key at all — a row with fewer columns than you asked for, or a line the pattern does not match. Treating those as sharing an empty key would silently merge unrelated lines, so they are kept exactly as they are, excluded from the grouping, and counted in a note beneath the controls. Note also that the column split is a plain separator split, not a full CSV parse: a quoted field containing the separator will be split at it.

Keeping the first occurrence preserves the original order and the earliest entry, which is correct for a log, an append-only export, or any list where position carries meaning. It is the default for that reason.

It is the wrong default when later entries carry corrections. If someone appended an updated address below the stale one, taking the first copy silently keeps the version that is out of date — the failure is invisible, because the output looks perfectly clean. Switching to the last occurrence keeps each value where its final copy appeared, which is what an append-corrections workflow actually means.

The two filtering modes answer different questions again. "Only lines that appear once" strips out every copy of anything repeated, which is how you find the entries that exist in one source but not the other after concatenating two lists. "Only the repeated lines" does the opposite and gives you the overlap. Neither is a cleanup; both are queries, and the counts beside them are the answer.

Every definition of duplicate looser than "byte-for-byte identical" is a judgement call, and the useful thing a tool can do is make that judgement explicit rather than bury it. Two email addresses differing only in capitals are almost certainly one inbox — the domain is case-insensitive by specification, and in practice providers treat the local part that way too. Two names differing only by an apostrophe are usually one person. Two words differing by an accent may or may not be the same word, depending entirely on the language.

So each of those is a separate switch you turn on knowingly, and none of them rewrite your data. The comparison runs against a normalised copy of the line; the line that reaches the output is untouched. The one exception is the trim option in the result panel, which does edit the kept text, and is labelled where it sits.

Unicode normalisation deserves a note because it is on by default and invisible when it works. The same accented character can be stored either as one code point or as a plain letter followed by a combining mark. The two render identically and compare as different strings, which is why a list assembled from more than one source sometimes appears to contain duplicates the tool refuses to remove. Normalising to a canonical form before comparing fixes that without discarding the accent.