URL Parsing and Search Params: What Round Trips Change

BotBrowser supports replaying an authorized, same-origin fixture under a declared profile and keeping bounded observations about which URL a page parsed and what the parser rewrote, but it does not change how a runtime implements the WHATWG URL algorithm, cannot make a relative string resolve without a base, and does not turn a normalised URL into an identity signal. This article parses a set of awkward URLs and query strings, then records exactly where new URL throws, which characters normalisation rewrites, how duplicate keys and plus signs survive, and why setting a search parameter differs from assigning a raw search string.

TL;DR

• new URL with a relative string and no base throws TypeError. The same string resolves against a base, and spaces in the path become %20.
• Parsing is normalisation, not identity: HTTPS://Example.TEST:443/a/../b/./c becomes https://example.test/b/c, dropping the default port and resolving dot segments.
• URLSearchParams keeps duplicate keys, treats + as a space, and encodes & and = inside a value when it serialises.
• sort() reorders by key; insertion order is preserved until it is called, so a round trip can change the string without changing the data.
• searchParams.set encodes a value differently from assigning url.search raw: the first percent-encodes spaces, the second leaves them for the URL parser to fix.

Contents

• Where new URL throws and where a base saves it
• What normalisation rewrites
• Duplicate keys, plus signs and value encoding
• Sort order and the cost of a round trip
• Setting a parameter versus assigning search
• Record a URL reading

Where new URL throws and where a base saves it

The constructor has two forms, and only one of them can resolve a relative reference. With no base, the input must already be an absolute URL; a path-only string has nothing to attach to and the parser throws: A parsed URL is also a value that has to be copied before it is stored, and structuredClone versus JSON shows which types and references a deep copy keeps and which a JSON round trip silently drops.

new URL("/a/b?x=1", undefined) -> "THREW TypeError"
new URL("/a/b?x=1", "https://example.test") -> "https://example.test/a/b?x=1"
new URL("a b", "https://example.test/dir/") -> "https://example.test/dir/a%20b"
new URL("../../x", "https://example.test/a/b/c") -> "https://example.test/x"

The first line is the common defect: code that receives a value from configuration, a form field or a link and calls new URL(value) without a base will throw for every relative path, and a try block that swallows the error turns a wrong URL into a silent undefined. The second line shows the fix, resolving the same string against a base.

The third and fourth lines show that resolution is not string concatenation. A space in a relative path is percent-encoded to %20, and ../../x walks up three directory levels from /a/b/c to produce /x, not a literal append. A caller who expects the raw input to survive a round trip through the constructor is surprised in both cases, because the output is a normalised URL, not the string that was passed in.

That distinction matters for equality checks. Two strings that address the same resource can parse to the same href, and one string can parse to a different href after normalisation. A fixture that compares the input string to the output string to prove nothing changed will fail on any input that needed normalisation, while a fixture that compares two parsed URLs to prove they are equivalent can pass even when the inputs looked different.

What normalisation rewrites

The parser lowercases the scheme and the host, drops a default port, resolves dot segments and percent-encodes characters that are not allowed in each component. Four inputs show the rewrites side by side:

raw="HTTPS://Example.TEST:443/a/../b/./c"
  href="https://example.test/b/c"
  origin="https://example.test" protocol="https:" port=""
raw="https://example.test/a%2Fb?q=a b#frag ment"
  href="https://example.test/a%2Fb?q=a%20b#frag%20ment"
  pathname="/a%2Fb" search="?q=a%20b" fragment="#frag%20ment"
raw="https://user:[email protected]/x"
  username="user" password="pass"

The first input loses three things at once: the uppercase scheme and host become lowercase, the explicit :443 disappears because it is the default for https, and /a/../b/./c collapses to /b/c. A caller who stored the original string and compares it to href will see a mismatch that is entirely normalisation.

The second input shows a subtler point. The encoded slash %2F in the path survives unchanged, because it is a deliberate escape of a character that would otherwise change the path structure; the raw space in the query and fragment becomes %20; and the space in the fragment is encoded too. So the path preserves an encoded slash while the query and fragment encode a literal space. A naive "encode everything" or "decode everything" pass on the whole URL would corrupt the %2F, which is exactly why the parser treats each component with its own rules.

The third input shows that credentials are parsed into username and password and re-serialised into the href, while origin excludes them. That matters for logging: origin is safe to record and compare, while href can carry a user name and password that should not be written to a log or a fixture record. A reading that stores href for a URL with credentials stores the secret; one that stores origin and pathname does not.

Every component has its own rule set, and that is the reason a single encode or decode pass over a whole URL is unsafe. The scheme and host are compared and stored in lowercase; the port is dropped when it equals the scheme's default; the path resolves dot segments and keeps a percent-encoded slash as data; the query applies the + to space rule and percent-encodes reserved characters; and the fragment encodes a raw space but preserves the structure of the fragment itself. A caller who takes the whole href, runs decodeURIComponent on it and passes the result back into new URL will usually get a different URL, because the decode step turns encoded slashes and encoded ampersands into characters the parser then reads as structure. The safe unit is the component, not the whole string.

That component model also explains why two URLs that look different can be identical after parsing. HTTPS://Example.TEST:443/b/c and https://example.test/b/c produce the same href, because the parser lowercases and drops the default port before anything else. A deduplication step that compares raw strings will treat them as distinct and store both, while a step that compares parsed href collapses them. Which is correct depends on the goal: a cache key wants the collapsed form so both spellings hit the same entry, while an audit log may want the raw form so the original input is preserved. The reading should say which form it stores.

Duplicate keys, plus signs and value encoding

URLSearchParams is a list of pairs, not a map, and several behaviours follow from that. Duplicate keys survive, an empty value is present but empty, and a value that contains & or = is percent-encoded when the list serialises:

params.getAll(a) -> ["1","2"]
params.get(b) -> ""
params.get(c) -> "&="
params.toString() -> "a=1&a=2&b=&c=<encoded ampersand><encoded equals>"

getAll returns both a values, so a caller who uses get('a') sees only the first and silently drops the rest. get('b') returns an empty string, which is different from null for a missing key: a bare b= is present with an empty value, and code that checks if (value) treats both as falsy. A value that decoded to &= re-serialises with its ampersand and equals sign percent-encoded again, so the raw string round-trips but the decoded value shows the characters the caller actually wanted.

The plus sign is the other trap. In a query string, + means a space, so a value that legitimately contains a plus sign must be percent-encoded as %2B:

searchParams from a+b -> "a b"
Map-like set round trip -> "q=a%2Bb"

The first line shows new URLSearchParams('q=a+b').get('q') returning "a b", because the parser read the plus as a space. The second shows that constructing the same list from an object with the value a+b serialises it to q=a%2Bb, so the plus is preserved as data. The asymmetry is the defect: a caller who builds a query by string concatenation writes q=a+b and later reads back a b, while a caller who goes through URLSearchParams and a value object keeps the plus. The two paths disagree on the same intended value, and only one of them round-trips.

Three different escapes decode to two different characters, and only a reading that lists each one separately catches the difference. A raw + decodes to a space, a %20 decodes to a space, and a %2B decodes to a plus. So q=a+b, q=a%20b and q=a%2Bb are three distinct encodings: the first two carry the value a b, and the third carries a+b. A caller who normalises a query by decoding it and re-encoding with a generic encoder collapses the first two into one form and may either lose or invent the plus in the third, depending on which encoder is used. The recorded form ?a=%2B+%20 from the parse dump is exactly this triple in one parameter: a plus sign, then a space, then a space, decoded from %2B, + and %20 respectively, and a re-encode that treats all three as data would change the string even though the decoded value is unchanged.

Duplicate keys interact with the same decoding. A query of a=1&a=2&b= parses to three pairs, two with the key a and one with the key b and an empty value, and it serialises back to the identical string a=1&a=2&b=. That identity is worth recording because it is the exception rather than the rule: this particular string is already in the canonical serialisation, so a parse and re-serialise leaves it unchanged, while a string with a raw space or a non-canonical escape would change. A fixture that uses only already-canonical inputs will never see the normalisation, which is why at least one non-canonical input belongs in the set.

Sort order and the cost of a round trip

The list has an insertion order, and sort() reorders it by key. Until sort() is called, the order is whatever was appended:

insertion order -> "k9=9&k8=8&k7=7&k6=6&k5=5&k4=4&k3=3&k2=2&k1=1&k0=0"
sorted order -> "k0=0&k1=1&k2=2&k3=3&k4=4&k5=5&k6=6&k7=7&k8=8&k9=9"

Ten keys appended in descending order serialise in that order, and sort() turns them into ascending order. The data is identical before and after; only the serialised string changes. That is the cost of a round trip: if a fixture builds a query, serialises it, parses it again and compares strings, the comparison can fail on ordering alone. A fixture that compares the set of pairs, not the string, is stable.

There is a second consequence for duplicate keys. sort() is stable for keys that compare equal, so two entries with the same key keep their relative insertion order after sorting. A caller who expects sort() to also order the values within a duplicate key will not get that; the values stay in the order they were appended, and only the key grouping changes.

That stability is testable and worth stating plainly: after sort(), a=1&a=2 stays a=1&a=2, not a=2&a=1, because the two keys compare equal and the sort is stable. A caller who wants the values ordered inside a duplicate key has to sort them separately before appending, and a caller who assumes sort() did that will ship an unstable output that changes between runs as insertion order varies. The rule to record is that sort() sorts by key only, and that equal keys keep their input order.

Length is another reason a round trip is not a conversion. Parsing a query to a list and serialising it back can change the string length even when the pairs are identical, because one %20 becomes a + and a raw space becomes %20 depending on the path taken. A fixture that asserts a fixed byte length after a round trip will pass on some inputs and fail on others for reasons that have nothing to do with a defect. Measuring the pair count and the key set instead keeps the assertion stable across the encoding choices that are all correct.

Setting a parameter versus assigning search

There are two ways to change the query on a URL object, and they encode differently:

set("q","a b+c&d=e") href -> "https://example.test/?q=a+b%2Bc%26d%3De"
search = "?q=a b+c&d=e" href -> "https://example.test/?q=a%20b+c&d=e"

The first line goes through searchParams.set, which percent-encodes the value with the query-string rules: the space becomes +, the plus becomes %2B, and & and = become %26 and %3D, so the whole value stays inside one parameter. The second line assigns url.search directly with a raw string, so the parser sees & and = as separators: the value becomes a b (with the space percent-encoded to %20), and c and d become separate parameters.

The two results are not the same URL, and the difference is not cosmetic. set keeps the intended value intact; raw assignment splits it into three parameters because the characters that separate parameters are the characters the caller wanted as data. A caller who builds a search string by hand and assigns it must percent-encode every reserved character before assignment, while a caller who uses set hands over the raw value and lets the API encode it.

There is also a spacing difference worth noting. set produced + for the space, while the raw assignment produced %20. Both decode to a space, but the two serialisations differ, so a byte-for-byte comparison of the two href values fails even though they carry the same logical parameter. That is another reason to compare parsed values rather than serialised strings.

Record a URL reading

A URL reading that survives a review states what it parsed and what changed. Six fields cover the observable part, and none of them stores a credential:

1. The input string and the base used, or the fact that no base was supplied and the parse threw.
2. The resulting origin, pathname, search and fragment, recorded separately rather than as one href, so credentials and normalisation are visible.
3. The list of query pairs as [key, value] entries, including duplicates and empty values, rather than a single reconstructed string.
4. Whether a space was a + or %20 in the serialised query, because the two decode the same but serialise differently.
5. The order of the pairs and whether sort() was called, since ordering changes the string without changing the data.
6. How the query was changed, through searchParams.set or a raw search assignment, because the two encode reserved characters differently.

What such a record cannot show is anything about how a server or a proxy will interpret the same URL. A client parses one way, a server framework may decode and re-parse the query with its own rules, and a redirect can rewrite the string in transit. For the wider question of how a page's parsed URL interacts with the strings it reports, see URL protocol detection and privacy and btoa and atob boundaries, which covers the encoding layer that query values pass through.

Two habits keep the reading honest over time. First, record origin and the pair list rather than one href, so normalisation and credentials are visible rather than merged into an opaque string. Second, compare parsed pairs rather than serialised strings, so ordering and the + versus %20 choice cannot turn a correct round trip into a false failure.

Public sources

• MDN: URL
• MDN: URLSearchParams
• WHATWG URL Standard
• RFC 3986: Uniform Resource Identifier syntax
• MDN: encodeURIComponent

Practical conclusion

Always supply a base when a URL might be relative, and expect the parser to normalise rather than preserve the input string. Treat URLSearchParams as a list of pairs, so duplicate keys and empty values survive, and remember that + decodes to a space. Use searchParams.set to change a value and let it encode reserved characters, or encode every reserved character yourself before assigning search. Record origin and the pair list rather than one href, and compare parsed pairs rather than serialised strings.