Unicode Normalization: Four Text Identities a Page Reads

BotBrowser supports replaying an authorized same-origin fixture under a declared profile and keeping bounded observations about what a page reports, but it cannot make a page return a canonical string it never called normalize() on, change which Unicode version a given ICU build ships, or make a grapheme cluster count equal a code-unit length. This review runs the ICU in the Node runtime over a fixed set of strings and reports four different numbers a page can read for the same visible text: the code-unit length, the code-point count, the grapheme-cluster count and the normalized length.

TL;DR

• é written as one code point and as e plus a combining acute are not equal as strings, are equal after NFC, and produce two different String.length values.
• A Set holding both spellings has size two; the same set after NFC has size one, so a key match can fail on text a reader sees as identical.
• Emoji make the gap larger: one family emoji is eight code units, five code points and one grapheme cluster.
• An unpaired surrogate fails isWellFormed() and TextEncoder writes one replacement character, so a lossy encode is observable without any error.
• Collation, casing and word segmentation all depend on the locale passed to them, and the default sort() does not use a collator at all.
• Every number here comes from one ICU build and one Unicode version; another build can disagree.

Contents

• Which runtime and Unicode version this run used
• Four lengths for the same visible text
• Where a key match fails on identical text
• Where an unpaired surrogate becomes a replacement character
• Where casing changes the length of a string
• Where collation and segmentation depend on the locale
• Record a text reading without overclaiming

Which runtime and Unicode version this run used

The probe prints the runtime, ICU and Unicode versions before any measurement, because every number below is a property of that triple. This run reported node=v26.9.0, icu=78.3, unicode=17.0 and tz=2026a. A different ICU build can carry different case mappings, a different collation table and a different segmentation rule set, so a reading without those versions is not reproducible. That is the same scoping rule the ICU timezone alias review applies to zone names, where the resolved string is a lookup in a shipped table rather than a clock reading.

The Unicode version matters more than it first appears. New emoji sequences, new combining marks and corrections to existing properties all arrive with a version bump, and each of them changes at least one of the four lengths below for some string. A record that says "the page reported length 8" without the Unicode version cannot be compared against a second machine, because the same visible text may be eight units on one build and nine on another once a new sequence gains a property that changes its segmentation.

Four lengths for the same visible text

The first probe measures each string four ways: length counts UTF-16 code units, [...s].length counts code points, an Intl.Segmenter with grapheme granularity counts what a reader perceives as one character, and normalize('NFC').length counts units after canonical composition.

e-acute-precomposed: units=1 points=1 nfc=1 nfd=2 cps=U+00E9
e-acute-decomposed: units=2 points=2 nfc=1 nfd=2 cps=U+0065 U+0301
emoji-zwj-family: units=8 points=5 nfc=8 nfd=8 cps=U+1F468 U+200D U+1F469 U+200D U+1F467
emoji-skin-tone: units=4 points=2 nfc=4 nfd=4 cps=U+1F44D U+1F3FD
hangul-syllable: units=1 points=1 nfc=1 nfd=2 cps=U+AC00
hangul-jamo: units=2 points=2 nfc=1 nfd=2 cps=U+1100 U+1161

Six rows, and four distinct tuples among them: the precomposed é row and the precomposed Hangul row share one, and the two decomposed rows share another. The two é rows differ in units and code points but agree after NFC, which is the classic case. The family emoji row shows the widest spread: eight units, five code points and one grapheme. The Hangul rows show that NFC composes two jamo into one syllable, so the normalized length matches the precomposed row exactly.

The practical consequence is that "length" is not one measurement. A page that records str.length for a display name records a UTF-16 unit count, which for an emoji-heavy name can be double what a reader would call the number of characters. A page that records [...str].length records code points, which still overcounts the family emoji five times. Only the segmenter agrees with the reader.

The grapheme count is the one that matches perception, and the probe makes the agreement explicit:

segmenter graphemes=1 units=8 points=5
segmenter graphemes=1 units=4 points=2
segmenter graphemes=1 units=2 points=2
segmenter graphemes=1 units=4 points=2

Four different strings, three distinct unit counts, one grapheme each. A validator that caps a name at twenty code units would reject five family emoji that a human reads as five characters; a validator that caps at twenty graphemes would accept them and store forty units and twenty-five code points. The cap and the reader are measuring different things, and both are defensible as long as the record says which one was used.

Where a key match fails on identical text

The two é rows above already imply a failure mode, and the second probe measures it directly:

NFC equal: true
raw equal: false
name keys equal raw: false | nfc equal: true
Set size with two spellings: 2
Set size after NFC: 1

Two spellings of the same name are unequal as strings and equal after NFC. Placed in a Set as map keys, they produce two entries; mapped through normalize('NFC') first, they produce one. That is the whole mechanism behind a duplicate account that a support tool cannot find, a cache that misses on every request because the client sends the other spelling, and a blocklist that fails because the entry was typed on a keyboard that composes and the traffic arrives from one that does not.

Nothing in that failure is exotic. A macOS keyboard typically produces decomposed text for accented Latin letters, a Windows keyboard typically produces precomposed text, and a copy-paste from a PDF can produce either depending on the font's glyph mapping. The two spellings render identically on screen and compare unequal in every language that compares strings by code unit.

The fix is not to normalize everything everywhere, because NFKC changes characters that some systems treat as distinct, and normalization can also break a byte-exact signature. The fix is to normalize at the boundary where identity is decided: the key of a lookup, the value of a cache entry, the comparison in a dupe check. A record should say which form it stored, because a stored string that was normalized cannot be compared to one that was not without normalizing again.

Where an unpaired surrogate becomes a replacement character

Normalization does not repair invalid text. A string containing an unpaired surrogate is not well-formed, and the probe shows both the check and the loss:

lone-surrogate isWellFormed=false units=3
lone-surrogate bytes=61 ef bf bd 62
replacement count=1

isWellFormed() returns false, and TextEncoder writes ef bf bd in place of the bad unit: one U+FFFD replacement character. The encode does not throw. A page that sends such a string to a server sends a different string than the one it held, and nothing in the call reported the substitution.

That silent substitution is why a round trip can change a value. Encode, transmit, decode and the returned text is no longer equal to the input, yet every step succeeded. A fixture that only checks "the request completed" will not notice; a fixture that compares the string before encoding with the string after decoding will, and the difference is exactly the replacement characters.

Two habits keep this visible. Check isWellFormed() before encoding when the input may come from a decoder or a slice operation, because slicing a string mid-surrogate produces exactly this shape. And when a round trip must preserve the value, compare the original and the decoded string rather than comparing lengths, since a replacement character and the original unit both occupy one unit and the lengths agree.

The substitution also has a count consequence that is easy to miss. A string with one unpaired surrogate and a string with one replacement character are both three code units in the probe's input, so a length check passes while the value differs. Two replacement characters from two bad units produce a string that no longer round-trips at all, and a repeated encode-decode cycle does not add more replacements, which means a fixture that encodes twice and compares gets a false pass on data that was already lossy on the first pass.

Slicing is the most common way to create this shape by accident. Cutting a three-unit string at index two, or taking the first four units of a five-unit one, can split a surrogate pair and leave one half stranded. The resulting string is still a valid JavaScript string of the expected length, so nothing throws; only isWellFormed() or a byte comparison after encoding reveals it. A fixture that truncates user text to a display budget without checking the boundary will produce exactly this defect on any input containing a non-BMP character near the cut.

Where casing changes the length of a string

Case mapping is not a per-character substitution that preserves length, and three common characters show three different behaviours:

case "ß" upper="SS" upperUnits=2 lower="ß" lowerUnits=1
case "İ" upper="İ" upperUnits=1 lower="i̇" lowerUnits=2
case "Dž" upper="DŽ" upperUnits=1 lower="dž" lowerUnits=1

The German sharp s expands to two letters when uppercased. The Turkish capital I with a dot keeps one unit uppercased but expands to two units when lowercased in an English locale, because the lowercase form is i plus a combining dot. The titlecase digraph changes one character to another without changing length. Three characters, three outcomes, and only one of them preserves the unit count.

A system that stores a case-insensitive key by uppercasing will therefore change the length of some identifiers, and a system that enforces a maximum identifier length before casing can reject an identifier that becomes longer after casing. Both are easy to fix once the expansion is known, and both are invisible if the only test input is ASCII.

The locale dimension makes this worse in a way that is easy to miss, because the same letter maps differently depending on the locale passed:

case "I" lower(tr)="ı" upper(tr)="I" lower(en)="i"
case "i" lower(tr)="i" upper(tr)="İ" lower(en)="i"
case "İ" lower(tr)="i" upper(tr)="İ" lower(en)="i̇"
case "ı" lower(tr)="ı" upper(tr)="I" lower(en)="ı"

In the Turkish locale I lowercases to a dotless ı and i uppercases to a dotted İ, while in English both map to the familiar pair. The third row is the sharpest: İ lowercases to a single unit in Turkish and to two units in English. A case-insensitive comparison that pins a locale gets one behaviour; one that relies on the default locale gets whatever the runtime chose, which may differ from the locale the profile reports.

Text also reaches a page through a font stack before any of these counts matter, and the same string can measure differently once glyphs come from a different face. CJK font rendering covers how the stack changes measurement, and Intl API locale data and portable formatting covers the formatting side of the same locale dependence this section is about.

Where collation and segmentation depend on the locale

Sorting is a third place where the locale changes the answer, and the probe shows three different orders for the same four words:

sort en: apple < äpple < zebra < Zebra
sort sv: apple < zebra < Zebra < äpple
sort default: Zebra < apple < zebra < äpple
en compare zebra/Zebra: -1

English places äpple right after apple, Swedish places it last because ä is a separate letter after z, and the default sort places Zebra first because it compares code units and the capital Z sorts before lowercase a. A list that a page renders sorted will therefore differ between two sessions that only differ in locale, and a screenshot comparison of that list will report a difference that is entirely collation.

The same comparator also decides equality when a map is keyed by sorted text, because Intl.Collator returns zero for strings the default comparison treats as different. The probe measures that too, across four sensitivity levels:

sensitivity base: e-acute vs e=0 a vs A=0
sensitivity accent: e-acute vs e=1 a vs A=0
sensitivity case: e-acute vs e=0 a vs A=-1
sensitivity variant: e-acute vs e=1 a vs A=-1

At base sensitivity both pairs compare equal, so a key built from such a comparator would merge é with e and a with A. At accent sensitivity the two pairs split in opposite directions. A "duplicate detection" feature built on a collator with the wrong sensitivity therefore merges or splits exactly the cases it was meant to handle, and the setting is invisible in the output.

Word segmentation adds one more number that disagrees with the obvious tokenizer:

word segments=5 ["Version","155.0.0","ships","café","file"]
whitespace tokens=6 vs segmenter words=5

Splitting on whitespace gives six tokens while the segmenter gives five word-like segments, and the whole gap is the circled digit ①, which the segmenter reports with isWordLike false. Removing it makes both counts five, and replacing the ligature fi with f also leaves both at five, so the ligature is not the cause. A "word count" recorded in a fixture depends entirely on which method produced it and on whether the input contains a symbol the segmenter classifies as non-word.

Numeric collation is the last locale-independent switch that changes ordering, and it changes it substantially:

numeric on: 1.2.0 < 1.9.0 < 1.10.0
numeric off: 1.10.0 < 1.2.0 < 1.9.0

With numeric comparison enabled, versions sort the way a release page expects. Without it they sort as text, and 1.10.0 comes before 1.2.0 because 1 is followed by . in both and then 1 precedes 2. A release list sorted by a comparator without the numeric option looks wrong to every reader while remaining internally consistent, which makes it hard to spot in a diff.

Record a text reading without overclaiming

A text reading supports a narrow statement: this runtime, this ICU, this Unicode version and this locale, given this input, produced these four numbers. It does not say that a reader sees the same thing, that another build would agree, or that a server normalized the string the same way.

Keep six fields and the record stays checkable. The runtime, ICU and Unicode versions, because case tables, collation tables and segmentation rules all ship with them. The input as supplied, ideally as code points rather than as rendered text, because two inputs can render identically. The four lengths, with the method named for each, because length, spread length, segmenter count and NFC length are four different claims. The locale and the collator sensitivity, because both change the answer. The normalization form applied at the boundary, because a stored NFC string cannot be compared to a stored NFD one. And whether the encoder path produced a replacement character, because that substitution changes the value without an error.

The matching docs page covers how BotBrowser keeps font and navigator surfaces aligned with the loaded profile, and it is the right place for configuration. This review adds the reading side: what the runtime does with the text those surfaces carry, which of the four lengths a given API returns, and where a locale decides a comparison that a reader assumes is fixed. Consistent configuration and a correct reading of the resulting string are separate questions.

For any decision about a real session, use an authorized integration result from the system that makes the decision instead of extrapolating from this probe. A length recorded in a fixture and a production verdict are different kinds of evidence, and the gap between them is where overclaiming happens.

Practical conclusion

Record the runtime, ICU and Unicode versions, the input as code points, the four lengths with their methods, the locale and collator sensitivity, the normalization form at the boundary, and whether the encoder wrote a replacement character. In this build é has two different unit counts, a two-entry set collapses to one after NFC, a family emoji is eight units and one grapheme, an unpaired surrogate encodes as one replacement character without an error, casing can double a length, and collation order changes with the locale.

Sources

• MDN: String.prototype.normalize
• MDN: Intl.Segmenter
• MDN: Intl.Collator
• Unicode Standard Annex 15: Normalization Forms
• Unicode Standard Annex 29: Text Segmentation
• MDN: TextEncoder
• BotBrowser font documentation

Next reading

Use CJK font rendering for how the same text measures differently across font stacks, and Intl API locale data and portable formatting for the formatting side of the same locale dependence.