Clean messy text in the right order: spaces, duplicates, accents

Sequence five text cleaners so each one sees the problem the previous step removed, without creating new damage.

Cleaning text is not one action but a short sequence, and order changes the result. Collapsing spaces before removing duplicates merges lines that were distinct, while stripping accents first can hide the duplicates you meant to find. Here is an order that stays predictable.

Start with line-level noise: whitespace and blank lines

The whitespace cleaner trims each line, collapses runs of spaces and drops empty lines. Do this first so duplicates are compared on content rather than on invisible padding, and so later tools do not treat a blank line as a row.

Remove duplicates before changing characters

Duplicate removal keeps the first occurrence in order. Run it while lines still differ only by padding, not after accent stripping or case changes that could merge genuinely different entries into one.

Normalise characters last and deliberately

Accent removal, smart-quote conversion and character stripping change content, not layout. Apply only the transformation the destination needs: an English slug may want accents removed, while a names list usually should keep them.

Tools covered in this guide

Whitespace Cleaner
Trim lines, collapse stray spaces, and drop blank lines safely.
Remove Duplicate Lines
Delete repeated lines while keeping the first occurrence in order.
Remove Letter Accents
Strip accents and diacritics so accented letters become plain Latin characters.
Remove Unwanted Characters
Delete specific characters, or strip everything that is not a letter or digit.
Replace Smart/Straight Quotes
Convert curly quotes to straight ones, or straight quotes to typographic quotes.
Head Text (First N Lines)
Extract the first N lines of a text, exactly as written.
Tail Text (Last N Lines)
Extract the last N lines of a text, exactly as written.

Frequently asked questions about text cleanup order

Should I remove duplicates before or after trimming spaces?

After trimming. Otherwise two identical lines with different padding are treated as different and both survive the deduplication step.

Will removing accents change the words?

It changes letters such as é to e, which is fine for slugs and search keys but wrong for names, citations or anything a human will read as written.

Does converting smart quotes affect code?

Yes. Straight quotes are syntactically meaningful in code and JSON, while curly quotes look similar but break parsing, so convert before pasting into a code editor.