All tools · Text tools

Remove Duplicate Lines

Strip repeated lines from a list, optionally case-insensitive.

Free to use, no sign-up required. Remove Duplicate Lines is one of 198 tools built into Xenition, alongside a chat that can use them for you.

Removes repeated lines from a list, or repeated sentences from a paragraph, and keeps the first copy of each. Handy for cleaning email lists, keyword lists, log output or pasted data.

How to use Remove Duplicate Lines

1. Paste your list into the box, one item per line.
2. Choose Lines for lists or Sentences for prose.
3. Tick case-insensitive if “Apple” and “apple” should count as the same item.
4. Check the Duplicates removed count and copy the cleaned text.

How it works

Each line (or sentence) is compared after trimming spaces from both ends, and with case-insensitive on it is also lowercased for the comparison. The first occurrence is kept in its original form and position; later matches are dropped. Blank lines are never treated as duplicates. Sentences mode splits after . ! ? … or । followed by whitespace and joins the kept sentences with single spaces.

Example

The six lines apple, Banana, apple, banana, “ cherry” and cherry leave apple, Banana, banana and “ cherry” with 2 duplicates removed. With case-insensitive ticked, banana also goes, leaving three lines and 3 removed.

Questions about Remove Duplicate Lines

Why are lines with different leading spaces treated as duplicates?

The comparison trims both ends of each line, so “ cherry” and “cherry” match. The copy that appears first is the one kept, spaces included.

Why were no duplicates found in my paragraph?

In Lines mode a paragraph is a single line, so a repeated sentence inside it is not a duplicate line. Switch to Sentences mode, which splits on sentence-ending punctuation. Note that Sentences mode rejoins everything with spaces, so paragraph breaks are not kept.

Use it from chat

You do not have to open Remove Duplicate Lines to use it. Ask Xenition in plain language and it runs the right tool, then carries the answer into whatever you are working on — a document, a spreadsheet, a deck.

Related tools

• Line Tools — Sort, dedupe, reverse, number, and clean up lines.
• Sort Lines — Sort lines alphabetically, numerically, or by length.
• Better English — Quick in-widget rewrite of a short snippet so it reads clearer and more natural. For full proofreading or tone rewrites of a document, ask in chat.
• Text ↔ Binary — Convert text to 8-bit binary and back.
• Count Occurrences — Count how many times a word or phrase appears.
• Find & Replace — Replace text or regex matches across a block of text.
• HTML Entities — Encode or decode HTML special characters.
• Initials Extractor — Get the initials from a full name.