Remove duplicate lines from a list
Remove duplicate lines from any list, keep the first or last copy, or show only the repeats — with case, trim and blank-line options.
- Runs in your browser
- Files are never uploaded
- Free, no account
One item per line. Works on email lists, log files, keyword exports — anything line based.
Off treats “Apple” and “apple” as the same line.
Ignores leading and trailing spaces when comparing.
Drops empty lines before comparing.
Adds “3⇥line” in duplicates-only mode.
How to remove duplicate lines
- Paste your list — one item per line — or drop a text or CSV file onto the box.
- Choose what you want out: the deduplicated list, only the lines that repeat, or only the lines that appear exactly once.
- Decide whether case matters. For email addresses and URLs it usually does not; for identifiers and passwords it always does.
- Leave “Trim each line” on unless trailing spaces are meaningful in your data.
- Read the counters to confirm the result before you copy or download it.
Keep first or keep last?
Both options remove exactly the same number of lines; they differ in which copy survives and therefore where it sits in the output. Keep first preserves the earliest occurrence, which is what you want for a chronological log or a priority-ordered list. Keep last preserves the most recent occurrence, which is what you want when later entries are updates — an append-only export where the newest row for a customer is the correct one, for example.
Either way the original order of the surviving lines is preserved. That is the main advantage
over the classic command-line trick sort | uniq, which silently reorders
everything alphabetically.
The three modes
| Mode | Input | Output |
|---|---|---|
| Remove duplicates | a, b, a, c, b | a, b, c |
| Show only duplicates | a, b, a, c, b | a, b — one entry per repeated value |
| Show only unique lines | a, b, a, c, b | c — the lines that appear exactly once |
Where it helps
- Mailing lists — merge exports from two systems and strip the overlap before an import that charges per contact.
- Keyword research — combine exports from several tools and keep one row per phrase.
- Log analysis — “show only duplicates” with counts turns a raw log into a frequency table you can paste into a spreadsheet.
- Redirect maps and URL lists — find the paths that appear twice before they cause a redirect loop.
- Data cleanup — “show only unique lines” quickly isolates the rows that exist in one export but not the other.
What counts as the same line
Two lines are duplicates when their comparison keys match. The key is the line itself,
optionally trimmed of leading and trailing whitespace, and optionally lowercased. Nothing else
is normalised: john.doe@example.com and johndoe@example.com are
different lines, as are values with different internal spacing. If you need looser matching,
run the list through find and replace first to normalise it,
then deduplicate.
Large lists and privacy
Lines are read in chunks with the browser given a chance to repaint between them, so a several-hundred-thousand-line list stays interactive rather than freezing the tab. Because everything runs locally, a customer list or a set of internal hostnames never leaves your device — which is exactly what you want before pasting it into any online tool.
Questions
Does it keep the original order?
Yes. Unlike sort-based deduplication, the surviving lines stay exactly where they were. Choose "keep first" to preserve the earliest occurrence or "keep last" to preserve the most recent one.
How do I see which lines were duplicated?
Switch to "Show only duplicates". You get one entry per repeated value, and the count prefix option adds the number of occurrences before each line, separated by a tab so it pastes cleanly into a spreadsheet.
Are trailing spaces treated as a difference?
Only if you turn "Trim each line" off. With trimming on (the default), "apple" and "apple " are the same line — which is usually what you want for lists exported from spreadsheets.
How many lines can it handle?
Hundreds of thousands. Lines are read in chunks with a pause between them so the browser stays responsive, and the whole job runs on your own machine.