Duplicate Line Remover
Lists copied from emails, spreadsheets, logs or websites often contain the same line more than once. This tool removes the repeats and keeps the first copy of each line in its original place. You can choose whether upper and lower case count as different, and whether spaces at the ends of lines are ignored. It runs in your browser, so your list is never uploaded.
How to use it
- Paste your list into Your lines, one item per line. A sample list is already there.
- Tick Ignore upper and lower case if Blue and blue should count as the same line.
- Leave Trim spaces at the start and end of each line ticked to ignore stray spaces.
- Click Remove duplicates and copy the clean list from the Result box. The line under it shows the counts before and after.
Worked example: the sample list
| Setting | Result | Lines before and after |
|---|---|---|
| Case sensitive | red, blue, green, Blue | 6 then 4, 2 removed |
| Ignore case | red, blue, green | 6 then 3, 3 removed |
The sample list is red, blue, red, green, Blue and blue. With case sensitivity on, the second red and the last blue are repeats, so two lines go and Blue stays because it is written differently. With case ignored, Blue is also a repeat, so three lines go. In both cases the first copy of each line stays where it was, so the order of your list is not changed.
How it decides what is a duplicate
Two lines are duplicates when they match exactly after the options are applied. If trimming is on, spaces at the start and end are removed first, so a line written as a with a space after it matches a line written without one. If case is ignored, both lines are compared in lower case. The copy that stays is the first one, written as it first appeared.
Blank lines
Blank lines count as lines too. If your list has several empty lines, the first one stays and the rest are removed as repeats, which is why the test list a, a, blank, blank, b became a, one blank line and b. If you want no blank lines at all, delete the remaining one by hand after the tool finishes.
Common uses
- Cleaning an email list before you import it.
- Removing repeated keywords from a research list.
- Tidying a list of links or file names.
- Reducing a long log to its unique messages.
- Checking how many different items a list really contains.
Example: a mailing list
You have merged two lists of email addresses and know that some people appear in both. Paste the combined list, tick Ignore upper and lower case, because email addresses are not case sensitive in practice, and click Remove duplicates. The line below the result might read Lines before 540, After 497, Removed 43. Now you know how many repeats there were, and you have a clean list that you can import without sending anyone two copies of a message.
Combine it with sorting
Duplicates are easier to see in a sorted list. If you need both, remove duplicates first, then sort the result with the Text Line Sorter. The order of the two steps does not change the final set of lines, but doing the removal first means you sort a shorter list.
Privacy and limits
Everything happens in your browser, so your list stays on your device. Very long lists, such as hundreds of thousands of lines, may take a moment because your computer does the work. For huge files a spreadsheet or a command line tool may be a better choice.
Checking the result
After you run the tool, compare the counts under the result with what you expected. If very few lines were removed from a list that you know is full of repeats, check that the case and trimming options match your data. Lines that look the same but differ by a hidden space inside the text, or by a different character such as a curly apostrophe, are treated as different lines. Scan the result once before you use it.
Common mistakes to avoid
- Forgetting to tick the case option and keeping near-duplicates such as Blue and blue.
- Expecting the tool to match lines that differ in small ways, such as a typo or an extra word.
- Pasting a table with several columns. The tool compares whole lines.
- Keeping only the cleaned list and losing the original. Save a copy first.
Frequently asked questions
Does it keep the first or the last copy?
It keeps the first copy of each line and removes later repeats.
Does the order of my list change?
No. The remaining lines keep their original order.
Is it case sensitive?
By default yes. Tick Ignore upper and lower case to change that.
Does it remove blank lines?
It keeps one blank line and removes the extra ones as duplicates.
Is my list uploaded?
No. It is processed in your browser.
Related tools and guides
This tool is provided for convenience. Keep a copy of your original list.