TextDecoder and TextEncoder: Bytes vs Strings

BotBrowser supports replaying an authorized same-origin fixture under a declared profile and keeping bounded observations about the bytes a page encoded and the string a decoder returned, but it cannot make TextEncoder emit a label other than UTF-8, cannot change which decoder a label resolves to, and cannot turn a decode result into an identity signal. This review decodes a fixed set of byte sequences through several labels and records which labels alias, how a byte-order mark is treated, when fatal mode throws instead of substituting, why a chunk boundary loses a multi-byte character, and why a byte length and a string length disagree for the same text.

TL;DR

• TextEncoder is always UTF-8 and ignores any label passed to its constructor; new TextEncoder({encoding:"latin1"}).encoding still prints utf-8.
• The labels latin1, iso-8859-1 and windows-1252 resolve to the same decoder, so café in UTF-8 bytes comes back as café and there is no way to tell which label was written.
• The UTF-8 byte-order mark is stripped by default and kept only with ignoreBOM: true, so the same bytes decode to one string or two depending on a flag most callers never set.
• A non-fatal decode replaces a bad byte with U+FFFD, while fatal: true throws TypeError; a truncated multi-byte sequence becomes one replacement character without any error.
• Splitting a multi-byte character across two decode calls loses it unless stream: true is set, and the streamed result joins back to the original string.
• A lone surrogate encodes to a single U+FFFD, so a round trip can change the value while every call succeeds; café is four UTF-16 units and five UTF-8 bytes.

Contents

• Which label resolves to which decoder
• How a byte-order mark and an invalid sequence are handled
• Why a chunk boundary drops a character
• Where byte length and string length disagree
• Record a codec reading

Which label resolves to which decoder

The constructor takes a label, not an exact encoding name, and the standard maps many labels onto one decoder. The probe decodes the five UTF-8 bytes of café (63 61 66 c3 a9) under four labels and prints the string each one returns:

TextDecoder=function TextEncoder=function node=v26.9.0
TextEncoder.encoding = "utf-8"
encode("café") -> 63 61 66 c3 a9
encode("😀") -> f0 9f 98 80
new TextEncoder({encoding:"latin1"}) -> "utf-8"
new TextDecoder("utf-8").decode(0x63 61 66 c3 a9) -> "café"
new TextDecoder("latin1").decode(0x63 61 66 c3 a9) -> "café"
new TextDecoder("iso-8859-1").decode(0x63 61 66 c3 a9) -> "café"
new TextDecoder("windows-1252").decode(0x63 61 66 c3 a9) -> "café"
new TextDecoder("utf-16le").decode(0x63 61 66 c3 a9) -> "慣썦�"

Two facts sit in those lines. First, TextEncoder reports utf-8 no matter what label is passed; the object simply has no other encoding, so a caller who writes latin1 and expects single-byte output gets multi-byte output and no warning. Second, the three single-byte labels collapse to one decoder. latin1, iso-8859-1 and windows-1252 are aliases in the standard's label table, and windows-1252 is the decoder all three resolve to. Decoding UTF-8 bytes through any of them reads each byte as one character, so the two-byte sequence c3 a9 becomes the two characters Ã and ©.

That collapse is the source of the classic mojibake defect: a producer wrote UTF-8 and a consumer declared Latin-1, or the reverse. The defect is not a bug in either side alone; it is a mismatch between the label a producer used and the label a consumer assumed. The two-character result is deterministic and reversible, which is why a re-decode of the same bytes under UTF-8 restores café exactly. The check that catches it is to compare the decoded string against the expected text, not to compare byte lengths, because both spellings of the mojibake occupy the same number of bytes.

The fourth label shows what a wrong family costs. utf-16le reads the same bytes two at a time as little-endian 16-bit units, so 63 61 becomes one character, 66 c3 another, and the leftover a9 becomes a replacement character. Nothing throws. A caller who receives bytes from a source that actually produced UTF-16 and declares LE gets a string, just not the intended one, and a caller who declares the wrong byte order for real UTF-16 gets a different string again. The label is part of the data contract, so a fixture that decodes without recording the label has recorded an unverifiable result.

There is a second reading of that line that matters just as much: the decoder consumed an odd number of bytes. A UTF-16 decoder reads one unit per two bytes, so a five-byte input leaves one byte with no partner, and the decoder turns the stranded byte into U+FFFD rather than discarding it. The character count of the result therefore tells a reader something about the input parity even though the label looked valid. A pipeline that decodes an even-length UTF-8 body under the wrong two-byte family will hide this signal, because an even byte count leaves nothing stranded and the substitution that would have flagged the mismatch never appears. The label mismatch is only cheap to spot when it also breaks the byte parity, which is a fragile form of detection to depend on.

An unknown label is the one case that fails loudly rather than silently:

new TextDecoder("utf-99") -> THREW RangeError

The constructor rejects a label that is not in the standard's table with RangeError, before any byte is read. That is the shape a fixture should rely on for a typo, because it names the problem at construction. Every other mismatch above returns a string and moves on, so the only reliable guard against a wrong-but-valid label is to assert the decoded value.

How a byte-order mark and an invalid sequence are handled

A byte-order mark is three bytes at the start of UTF-8 data (EF BB BF) or two at the start of UTF-16 data (FF FE for little-endian). The standard treats the UTF-8 mark specially: by default the decoder removes it from the output, and with ignoreBOM: true it leaves it in. The probe shows both results for the same input:

decode(EF BB BF 41) default -> "A"
decode(EF BB BF 41) ignoreBOM:true -> "A"
decode(FF FE 41 00) as utf-16le -> "A"

The first line decodes a mark followed by the byte 41 (A) and returns the single character A. The second returns two characters: an invisible U+FEFF followed by A, so result.length is 2 rather than 1. A comparison of the two strings fails, a comparison of their lengths differs by one, and neither is wrong; the flag decided the outcome. A consumer that uses ignoreBOM from a config and a producer that does not is exactly the split that turns one correct string into two unequal ones.

The UTF-16 line shows the mark handled by the same rule inside a different decoder: FF FE is the little-endian mark, the decoder consumes it, and the remaining 41 00 decodes to A. A UTF-16 decoder that expects the other byte order reads FF FE as a single unit and produces a different first character, so the mark and the byte order have to agree before the text does.

An invalid sequence is a separate decision from a mark, and fatal sets it:

decode(A FF B) non-fatal -> "A�B"
decode(A FF B) fatal -> THREW TypeError
decode(E4 B8) truncated -> "�"

The FF byte is not valid anywhere in UTF-8. Without fatal, the decoder emits one U+FFFD in its place and continues, so the result is A, a replacement character, then B: three characters for three input bytes, with one of them substituted. With fatal: true the same bytes throw TypeError. A pipeline that must not receive corrupted text should set fatal, because the non-fatal result is a valid string that silently differs from the input.

The truncated case is the one that surprises most callers. E4 B8 is the first two bytes of a three-byte character, and the decoder cannot complete it at end of input, so it emits one replacement character. No error is raised even though the data was cut short. That is why a byte count check passes on truncated input: the decoder always returns a string, and only the value reveals that a character was lost.

Why a chunk boundary drops a character

A network response, a file read and a worker message all arrive in chunks, and a chunk boundary can fall in the middle of a multi-byte character. TextDecoder has two modes for that, and the probe splits the five UTF-8 bytes of café after four bytes, so the first chunk ends on the first byte of the two-byte é:

stream split mid-sequence: first="caf" second="é"
non-stream split mid-sequence -> "caf�"

The non-stream path decodes each chunk as a complete unit. The first chunk is caf plus the byte c3, which starts a two-byte character whose continuation the chunk does not contain, so the decoder turns the stranded c3 into one replacement character and returns caf followed by U+FFFD. The é is gone. The streamed path with stream: true holds the trailing c3 instead of substituting it, so the first call returns caf, the second call receives a9 and returns é, and the two results concatenate to café.

This is why any decoder that reads a stream has to pass stream: true on every chunk except the last, and call the decoder once more with no argument to flush the tail. A loop that decodes each chunk without the flag loses one character at every boundary that lands inside a multi-byte sequence, and the loss is silent: the total length shrinks by one or more units and every call returns a string. A fixture that compares the concatenated result with the original string catches it; a fixture that only checks that decoding succeeded does not.

The same mechanism appears when a page reads a ReadableStream and decodes incrementally, and when a worker reassembles a message. In both cases the fix is the same flag and a final flush, and the failure without it is a fixed number of replacement characters equal to the number of split sequences. The count is a useful diagnostic: one replacement character at a chunk boundary means one split, and a run of them means the chunks were misaligned with the character width.

A final flush is easy to forget because every intermediate call already returned a string. After the last chunk, a decoder that still holds trailing bytes has to be called once with no input, and only that call emits the character the held bytes complete. A loop that stops after the last chunk leaves the tail inside the decoder, so the joined output is short by the characters those bytes would have produced. A fixture that compares the joined output against the original catches the omission; a fixture that checks only that no call threw does not, because none of them did.

Where byte length and string length disagree

TextEncoder writes UTF-8, so the byte length of a string is its UTF-8 length, not its UTF-16 length. The two agree only for text made of characters below U+0080. The probe measures four strings both ways:

"hello" stringLength=5 byteLength=5
"café" stringLength=4 byteLength=5
"中文" stringLength=2 byteLength=6
"😀" stringLength=2 byteLength=4

For hello, five ASCII characters cost five bytes because each ASCII code point is one unit and one byte. For café, the four letters cost five bytes because é is one UTF-16 unit but two UTF-8 bytes. For the two CJK characters, two units cost six bytes because each is three bytes in UTF-8. For the emoji, two UTF-16 units (a surrogate pair) cost four bytes, because one code point above the basic plane is four UTF-8 bytes. A storage limit expressed in bytes and a display limit expressed in characters therefore disagree by a different factor for each script, and a budget written for ASCII rejects or accepts the wrong amount of non-ASCII text.

The producer of those bytes is often a view over an ArrayBuffer, and ArrayBuffer and TypedArray shows why the same eight bytes report byte length 8 under a Uint8Array but element length 2 under a Uint32Array, and why a stored subarray alias dies when the source buffer is reused.

The surrogate case carries a second disagreement, because a lone surrogate is not valid text and TextEncoder rewrites it:

encode(lone high surrogate) -> ef bf bd
round trip lone surrogate length=1 equal=false

The encoded bytes are ef bf bd, one U+FFFD, because a lone surrogate has no UTF-8 form. Decoding those bytes returns a single replacement character, so the string length matches the input length of one, but the value differs and the equality check is false. A round trip can therefore preserve length and change value at the same time, which is why a length comparison is not a round-trip test. Slicing a string at an arbitrary index creates exactly this shape, because a cut inside a surrogate pair leaves one half stranded.

Record a codec reading

A codec reading supports a narrow claim: this runtime, this label, this flag, this input produced this string. Six fields keep it checkable, and none of them stores text that a decoder could reinterpret:

1. The label passed to the decoder and the byte order it implies, because several labels alias and one label can read bytes two ways.
2. The input as a byte sequence, not as rendered text, because two different byte sequences can render the same and one sequence can be read under several labels.
3. Whether ignoreBOM was set, because it changes the string length by one for the same bytes.
4. Whether fatal was set, because it decides between a thrown error and a substituted replacement character.
5. Whether stream was set per chunk and whether a final flush ran, because both decide whether a split character survives.
6. The UTF-8 byte length and the UTF-16 unit length recorded separately, because the two numbers answer different questions about the same text.

What such a record cannot show is anything about the bytes before they arrived. A decoder is given a byte sequence; it does not say whether the producer wrote UTF-8, Latin-1 or UTF-16, and a mismatch between producer and consumer is the most common cause of the mojibake above. For the wider question of how a page reports text that came from another layer, see Unicode normalization and text identity signals, which covers why two renderings of the same visible text compare unequal, and btoa and atob boundaries, which covers the older single-byte encode path that throws on the same non-ASCII text that UTF-8 accepts.

Two habits keep the reading honest over time. First, record the label and the flags alongside the decoded string rather than the string alone, because a bare string cannot be re-derived without knowing which options produced it. Second, compare the decoded value to the expected value and, for a round trip, compare the original string to the decoded result rather than comparing lengths, because a replacement character and the original unit can occupy the same length while differing in value.

Public sources

• MDN: TextEncoder
• MDN: TextDecoder
• MDN: TextDecoderStream
• WHATWG Encoding Standard
• Unicode Standard Annex 15: Normalization Forms
• BotBrowser: automation consistency reference

Practical conclusion

Treat the label as part of the data contract: latin1, iso-8859-1 and windows-1252 all resolve to one decoder, TextEncoder is always UTF-8 and ignores any label you pass, and an unknown label is the only one that throws at construction. Choose ignoreBOM on purpose and choose fatal when corrupted text must not pass silently. Decode streams with stream: true and flush the tail, or a split multi-byte character becomes a replacement character. Record the label, the bytes, the flags and both lengths separately, and compare the decoded value rather than the length so a lone surrogate that encodes to one U+FFFD cannot pass a green check.

Next reading

To go deeper into the text layer, Unicode normalization and text identity signals gives the several lengths and sort orders of one visible string, and btoa and atob boundaries gives the older way of packing bytes into a string and where it fails on non-ASCII text.