Unicode Edge Cases Every Form Should Survive

Text handling breaks in ways that are invisible until they are catastrophic. A user cannot log in because their password contains an é that their phone encoded differently from their laptop. A Turkish user's uppercase I becomes ı and a config key lookup fails. A family emoji in a display name truncates mid-character and the JSON response becomes invalid.

Every one of these is a real, repeatedly-shipped bug, and all of them are cheap to catch — if the input ever appears in a fixture. It usually does not, because the person writing the fixture types in ASCII.

Here is the catalogue, roughly in order of how often it causes production incidents.

1. The same character, two encodings#

é can be one code point (U+00E9, LATIN SMALL LETTER E WITH ACUTE) or two (U+0065 e + U+0301 COMBINING ACUTE ACCENT). Both render identically. Neither is wrong.

"café" === "café"  // false
"café".length      // 4
"café".length      // 5

Those two string literals look the same in every editor. The first is NFC — composed. The second is NFD — decomposed. And which one you get depends on where the text came from: macOS's HFS+ historically normalised filenames to a variant of NFD, most Linux tooling and the web platform prefer NFC, and iOS keyboards emit NFC while some IMEs emit NFD.

Where it breaks: username uniqueness checks pass when they should fail. A search for "café" misses the row. A password comparison fails intermittently between devices. A file written on a Mac cannot be found by a Linux service.

The fix is to normalise at the boundary — once, on input, before storage or comparison:

const canonical = input.normalize("NFC");

Pick NFC; it is what the W3C recommends for the web and what most systems already produce. The important part is not which form you choose but that you choose one and apply it consistently at every entry point. Normalising on write but not on query is the classic half-fix.

Note that NFC is not idempotent across all text in the naive sense you might hope — but for practical purposes, normalising twice is harmless. Normalising inconsistently is not.

2. Case folding is locale-dependent#

The most famous instance is the Turkish dotted and dotless i. Turkish has four letters where English has two: i/İ (dotted) and ı/I (dotless).

"TITLE".toLowerCase()            // "title"
"TITLE".toLocaleLowerCase("tr")  // "tıtle"  — dotless ı
"i".toLocaleUpperCase("tr")      // "İ"      — dotted capital

If your application lowercases a string for comparison using the user's locale, a Turkish user's "TITLE" no longer matches your "title". This has broken config parsing, HTTP header matching, and file extension checks in shipped software many times over.

German gives you the mirror problem. ß uppercases to SS, which means case conversion is not length-preserving and not round-trippable:

"straße".toUpperCase()  // "STRASSE"
"STRASSE".toLowerCase() // "strasse" — the ß is gone

There is a capital ẞ (U+1E9E) as of Unicode 5.1, but conventional uppercasing does not produce it.

The rule: for anything machine-facing — identifiers, protocol tokens, keys, file extensions — use locale-invariant case operations. toLowerCase() in JavaScript is already locale-invariant; toLocaleLowerCase() is the dangerous one. In .NET the distinction is ToLowerInvariant() versus ToLower(); in Java, toLowerCase(Locale.ROOT) versus the no-arg overload, which uses the default locale of whatever machine happens to be running. Reserve locale-aware folding for text you are displaying to that user.

For case-insensitive comparison, the technically correct operation is neither: it is String.prototype.localeCompare with sensitivity: "accent", or full Unicode case folding, which handles the ligatures and title-case characters that simple lowercasing does not.

3. Length means at least four different things#

Ask how long "👨‍👩‍👧‍👦" is and there are four defensible answers:

The family emoji is four person emoji joined by three ZERO WIDTH JOINER characters (U+200D). Each person emoji is itself outside the Basic Multilingual Plane, so each occupies two UTF-16 units.

"👨‍👩‍👧‍👦".length  // 11
[..."👨‍👩‍👧‍👦"].length  // 7 — spreading iterates code points

Where it breaks: a maxlength="20" on a display name field allows a string that a VARCHAR(20) rejects. Truncating to 20 characters slices a surrogate pair in half and produces a lone surrogate, which is not valid UTF-8 and will throw when serialised. A character counter shows "3 of 50" for a single emoji.

To count what a human counts, use Intl.Segmenter:

const seg = new Intl.Segmenter("en", { granularity: "grapheme" });
[...seg.segment("👨‍👩‍👧‍👦")].length;  // 1

To truncate safely, truncate on grapheme boundaries, never on code units. And decide explicitly whether your database column limit is in bytes or characters — VARCHAR(255) means 255 characters in PostgreSQL and 255 characters in MySQL, but Postgres text with a byte-based index and MySQL's index prefix limits both bring bytes back into it.

4. MySQL's utf8 is not UTF-8#

This deserves its own section because it has caused more emoji-related incidents than everything else combined.

MySQL's utf8 character set stores a maximum of three bytes per character. Every character outside the Basic Multilingual Plane — all emoji, many CJK extension characters, historic scripts, mathematical alphanumerics — needs four. Inserting one produces:

ERROR 1366: Incorrect string value: '\xF0\x9F\x98\x80' for column 'name'

or, in non-strict mode, silently truncates the value at the offending character.

The real UTF-8 is utf8mb4. MySQL 8.0 finally made utf8mb4 the default and formally deprecated the utf8 alias, but every schema created before that carries the three-byte version, and migrations frequently miss the connection charset even after the columns are converted. All four have to agree: the column, the table default, the server default, and the client connection.

5. Invisible and directional characters#

Some code points render as nothing at all, and they arrive in your input constantly — from copy-pasted rich text, from PDFs, from word processors.

• U+200B ZERO WIDTH SPACE — breaks equality comparisons; users cannot see why their input is "wrong"
• U+00A0 NO-BREAK SPACE — survives a trim() in many languages because it is not \s in some regex flavours
• U+FEFF BYTE ORDER MARK — appears at the start of files and, if not stripped, becomes part of your first CSV column name
• U+200D ZERO WIDTH JOINER — structural in emoji, meaningless elsewhere
• U+00AD SOFT HYPHEN — invisible until the text wraps

Then there are the bidirectional controls, which are a security issue rather than a nuisance. U+202E RIGHT-TO-LEFT OVERRIDE reverses the display order of everything after it. A file named invoice‮fdp.exe displays as invoiceexe.pdf. This has been an active malware technique for over a decade.

The same mechanism attacks source code. Trojan Source (CVE-2021-42574, Boucher and Anderson, 2021) uses bidi overrides to make code display in one order and compile in another — a comment that visually contains the whole if body while the compiler sees executable code. Most compilers and code hosts now warn on unbalanced bidi controls; your own diff viewer may not.

The defence is an explicit policy per field. For a display name, strip formatting characters (Unicode category Cf) entirely. For text that legitimately needs bidi — Arabic and Hebrew content — allow the controls but require them to be balanced within the string.

6. Homoglyphs#

а (U+0430, Cyrillic small a) and a (U+0061, Latin small a) are visually identical in most fonts and are different characters. So аpple.com is not apple.com.

This is the internationalised-domain-name spoofing problem, and browsers mitigate it by displaying Punycode (xn--pple-43d.com) when a domain mixes scripts. Your application probably does not.

Anywhere a name must be distinguishable by a human — usernames, display names, organisation names, subdomains — mixed-script strings deserve either rejection or a warning. The Unicode Consortium's UTS #39 defines confusability detection and a "single-script" restriction level. At minimum, flag a string that mixes Latin with Cyrillic or Greek, which is the overwhelming majority of spoofing attempts in practice.

Related: NFKC normalisation folds compatibility characters, so fi (U+FB01, the fi ligature) becomes fi, and ａ (fullwidth) becomes a. This is useful for search and for identifier canonicalisation, and destructive for text you intend to preserve exactly. NFKC is lossy in a way NFC is not; do not use it as a general input filter.

7. Names, specifically#

Personal names break more validators than any other field, because almost every rule a developer writes about them is false. Names contain apostrophes (O'Brien), hyphens (Sainte-Beuve), spaces, non-Latin scripts, combining marks, and characters like ʻ (U+02BB, the Hawaiian ʻokina) that look like punctuation but are letters. Names can be a single character. Some people have no surname, and forms that require one are simply broken for them.

The practical minimum for a name field: accept any Unicode letter, mark, apostrophe, hyphen and space; normalise to NFC; strip formatting characters; impose a generous grapheme-based length limit; and do not require more than one field to be non-empty.

A fixture set worth having#

The point of all of this is that these inputs should be in your test data, not discovered by users. A reasonable minimum set:

• The same accented word in NFC and NFD, asserted equal after normalisation
• A Turkish İstanbul and a German Straße, round-tripped through case conversion
• The ZWJ family emoji, for length, truncation and storage
• A right-to-left string — مرحبا بالعالم — for layout and for bidi handling
• A string with a leading BOM and a trailing no-break space, to test trimming
• A mixed-script homoglyph like pаypal (Cyrillic а), to test your confusability check
• A 4-byte character, to prove the database column is utf8mb4
• A string exactly one grapheme over the length limit, to test truncation

Eight strings. They take a morning to wire into a test suite and they cover the great majority of text-handling bugs that reach production.

The alternative is the way most teams find these: one user at a time, each convinced the software is broken specifically for them. Which it is.