Remove Duplicate Lines

Remove Duplicate Lines

Paste any list and get a deduplicated version back, with options for case sensitivity, whitespace trimming, and preserving original order vs. sorting. Works on emails, names, URLs, log lines, code identifiers, or anything separated by newlines.

Why Deduplication Comes Up Constantly

Lists from real-world sources almost always contain duplicates. Mailing lists merged from two events. URL lists scraped across multiple pages. CSV exports run twice. The duplicates are usually invisible until you're staring at the file, and they cause downstream pain — duplicate emails get flagged as spam, duplicate URLs waste crawl budget, duplicate IDs corrupt joins. Removing them is the single most common cleanup step in a data pipeline.

The trick is that duplicate is rarely as simple as byte-identical. Alice@example.com and alice@example.com are the same email for delivery purposes but different strings. hello and hello differ by leading whitespace but represent the same line. http://example.com and http://example.com/ are equivalent URLs. The dedup options below cover the cases that come up most often.

How It Works

The input gets split on newlines into an array. Each line is then normalized according to the active options — lowercased if case-insensitive, trimmed if whitespace-insensitive — and the normalized value becomes a key in a Set. If a normalized key is already in the Set, the line is dropped; otherwise it's kept. Preserve order keeps the first occurrence of each unique line in original position; sort output returns the deduplicated set in alphabetical order (which is functionally equivalent to piping through sort | uniq on Unix).

Need to dedupe and then alphabetize? Toggle both options, or pipe through the sort lines tool after. Need to dedupe a CSV column rather than a flat list? Use the CSV viewer to extract the column first, then dedupe.

Common Use Cases

Cleaning a mailing list before import — case-insensitive dedup is the standard pattern, since email addresses are case-insensitive in the local part for almost every major provider. Removing redundant entries from a sitemap or a list of canonical URLs. Deduping a list of git commit messages to find the unique work that happened across branches. Cleaning a vocabulary list, glossary, or tag set. Removing repeated log lines so you can read the unique error patterns instead of the same stack trace repeating.

Worked example: what each option does

The options interact, so here is one input run through them. These nine lines (· marks a space) were pasted into the tool in Chromium on October 12, 2026, switching on one option at a time:

banana
Apple
··apple
apple
banana·
cherry

Banana
apple

What the example shows. The first occurrence always wins and keeps its position, so with case-insensitive matching you get Apple, not apple, because it came first. Trim whitespace changes what is written as well as what is compared: ··apple comes out as apple, and banana· merges into banana. Turn Trim off if leading spaces matter, for example in indented code or YAML. Blank lines are deduplicated like any other line, so one survives unless you remove empty lines. Sorting happens after deduplication and uses your browser's dictionary order, so capitals and lower case sit together (apple, Banana, cherry) instead of all capitals first as in a plain byte sort.

For comparison, the usual command-line answers treat whitespace and case literally: awk '!seen[$0]++' keeps order but leaves 8 lines here, and sort -u also leaves 8 and reorders them. The counters under the boxes count lines the way wc -l does: a newline at the very end of the text does not add an extra empty line.

Frequently Asked Questions

Built by Derek Giordano · Part of Ultimate Design Tools

Privacy Policy · Terms of Service