URL Specification

Preliminaries

This section introduces basic concepts and notational conventions which are used throughout the rest of this document.

Sequences

This specification uses an algebraic notation for sequences that makes no distinction between one-element sequences and their first element. It uses the following notation:

The notation e : T may be used to inductively specify operations on sequences. For example, the length of a sequence is defined by length ε = 0 and length (e : T) = 1 + length T.

Parentheses are used for disambiguation and as visual aides. Paired parentheses are not part of a sequence-constructing operation.

Both component-types and their corresponding component-constructing operator are typeset in boldface. For example, scheme denotes a component-type, whilst (scheme value) denotes a scheme-component. Again, parentheses are used to disambiguate and as visual aides. They are not part of the component-constructing operator.

When a collection of components contains exactly one component for a given component-type, then the type may be used as a reference to the component's value. For example, given a sequence of components S := (dir x・file y・query z), the phrase “the query of S” identifies the value z whereas the phrase “the query component of S” identifies (query z).

Strings and Grammars

Strings and Code-Points

For the purpose of this specification,

• A string is a sequence of characters.
• A character is a single Unicode code point.
• A code point, is a natural number n < 17 × 2¹⁶.
• The empty string is a sequence of zero code points. It is denoted by ε.

Code points are denoted by a number in hexadecimal notation preceded by u+, in boldface. In addition, code points that correspond to printable ASCII characters are often denoted by their corresponding glyph, typeset in monospace and on a screened background. For example, u+41 and A denote the same code point. Strings that contain only printable ASCII characters are often denoted as a connected sequence of glyphs, typeset likewise.

The printable ASCII characters are codepoints in the range u+20 to u+7E, inclusive. Note that this includes the space character u+20.

Grammars

The notation name ::= expression is used to define a production rule of a grammar, where the expression uses square brackets ( [ … ] ) for optional rules, a postfix star ( * ) for zero-or-more, a postfix plus ( + ) for one-or-more, an infix vertical line ( | ) for alternatives, monospaced type for literal strings and an epsilon ( ε ) for the empty string. Parentheses are used for grouping and disambiguation.

Pattern Testing

The shorthand notation string :: rule is used to express that a string string can be generated by the production rule rule of a given grammar.
Likewise, the notation string :: expression is used to express that string can be generated by expression.

Percent Coding

This subsection is analogous to the section Percent-Encoding of RFC 3986 and the section Percent-encoded bytes of the WHATWG standard.

Bytes

For the purpose of this specification,

• A byte is a natural number n < 2⁸.
• A byte-sequence is a sequence of bytes.

Percent Encoded String

A percent-encoded–string is a string that may have zero or more percent-encoded–byte-sequences embedded within, as follows:

The grammar above may be relaxed in situations where error recovery is desired. This is achieved by adding a pct-invalid rule as follows:

There is an ambiguity in the loose grammar due to the overlap between pct-encoded-byte and pct-invalid combined with uncoded. The ambiguity must be resolved by using the pct-encoded-byte rule instead of the pct-invalid rule wherever possible.

Percent Encoding

To percent-encode a string string using a given character-set encode-set …

Wrap this up. Furthermore, this needs to discuss the encoding override, which is only allowed for the query; all other components must use UTF8.

Percent Decoding

Percent decoding of strings is stratified into the following phases:

• Analyse the string as into pct-encoded-bytes, pct-invalid and uncoded parts according to the pct-encoded-string grammar.
• Convert the percent-coded-bytes to a byte-sequence and decode the byte sequence to a string whilst leaving the pct-invalid and uncoded parts unmodified.
• Recompose the string.

URL Model

An URL is a sequence of components that is subject to a number of additional constraints. The ordering of the components in the sequence is analogous to the hierarchical syntax of an URI as described in Hierarchical Identifiers in RFC 3986.

It is important to stress the distinction between an URL and an URL-string.
An URL is a structure, indeed a special sequence of components, whereas an URL-string is a special kind of string that represents an URL. Conversions between URLs and URL-strings are described in the sections Parsing and Printing.

• Authorities have at most one component per type.
• If an Authority has a password component then it also has a username component.
• If an Authority has a username or a port component then it also has a host component.

URL components

Whenever present, the components of an URL are subject to the following constraints:

• scheme-string The scheme of an URL is a string scheme :: alpha (alpha | digit | + | - | .)*.
• The authority of an URL is an Authority.
• The path-root of an URL is the string /.
• The drive of an URL is a string drive :: alpha (: | |).
• The file of an URL is a nonempty string.
• The host of an Authority is a Host.
• The port of an Authority is either ε or a natural number n < 2¹⁶.
• For all other components present, the components' values are strings.

• A dir component is a component (dir name) where name is a string.
• The query of an URL, is a string.
• The fragment of an URL, is a string.

Note that arbitrary code points are allowed in component values unless explicitly specified otherwise. The restrictions on the code points in an URL-string is discussed in the Parsing section.

Types and Shapes

There is an additional number of soft constraints that must be met for an URL in order to be called valid. However, implementations must tolerate URLs that are not valid as an error recovery strategy.

There is a further categorisation of URLs that is used in several of the operations that will be defined later.

Reference Resolution

This section defines reference resolution operations that are analogous to the algorithms that are described in the chapter Reference Resolution of RFC 3986. The operations that are defined in this section are used in a subsequent section to define a parse-resolve-and-normalise operation that accurately describes the behaviour of the ‘basic URL parser’ as defined in the WHATWG URL standard.

Reference resolution as defined in this section does not involve URL-strings. It operates on URLs as defined in the URL Model section above. In contrast with RFC 3986 and with the WHATWG standard, it does not do additional normalisation, which is relegated to the section Equivalences and Normalisation instead.

Order and Prefix operations

A property that is particularly useful is the order of an URL. Colloquially, the order is the type of the first component of an URL. The order may be used as an argument to specify various prefixes of an URL.

The Goto Operation

Based on the order and the order-limited prefix one can define a “goto” operation that is analogous to the “merge” operation that is defined in the subsection Transform References of RFC 3986. I have chosen the name “goto” to avert incorrect assumptions about commutativity. The operation is not commutative, but it is associative.

Goto Properties

The goto operation has a number of pleasing mathematical properties, as follows:

• ord (url₁ goto url₂) is the least type of {ord url₁, ord url₂}.
• (url₁ goto url₂) goto url₃ = url₁ goto (url₂ goto url₃).
• ε goto url₂ = url₂.
• url₁ goto ε = url₁ — if url₁ does not have a fragment.
• url₂ is a postfix of (url₁ goto url₂).

Be aware that the goto operation does a bit more than sequence concatenation. In some cases it creates a path-root component to satisfy the path-root-constraint of the URL model. For example, if url₁ is the URL represented by //host and url₂ is the URL represented by foo/bar then (url₁ goto url₂) is represented by //host/foo/bar, thus containing a path-root even though neither url₁ nor url₂ has one.

Forcing

There is an additional operation on URLs that I have named force. The force operation is used as a final step in the process of resolving a web-URL or a file-URL. The operation ensures that the resulting URL has an authority token and a path-root token. Note that it is possible for the force operations to fail.

Force Properties

The force operation lacks a number of nice properties with respect to the goto operation. The following equalities do not hold in general:

Resolution

The subsection Transform References of RFC 3986 specifies two variants of reference resolution. A generic, strict variant and an alternative non-strict variant. This specification adds a third variant that is used to specify the behaviour of web-browsers. I have chosen to name the strict variant generic resolution and the non-strict variant legacy resolution, so that the words the words strict and non-strict can be used as modifiers later on.

Based on the generic resolution, and the legacy resolution and the force operations as defined above, it is now possible to define a reference resolution operation that can be used to characterise the behaviour that is specified in the WHATWG standard.

Parsing

Parsing is the process of converting an URL-string to an URL.
Parsing is stratified into the following phases:

1. Preprocessing.
2. Selecting a parser mode.
3. Parsing.
4. Decoding and parsing the host.

Preprocessing

The appendix Delimiting a URI in Context of RFC 3986 states that surrounding white-space should be removed from an URI when it is extracted from its surrounding context. The WHATWG standard makes this more explicit and specifies a preprocessing step that removes removes specific control– and white-space–characters from the input string before parsing.

Parser modes

Unfortunately, URL-parsing depends on the scheme of the URL being parsed. As a consequence, scheme-less URL parsing is ambiguous. This can be resolved by explicitly specifying a parser-mode. The parser-mode only influences how scheme-less URLs are parsed.

In practice, it is possible and advisable to begin parsing with a supplied parser-mode, and to update the parser-mode whilst parsing, as soon as a scheme has been detected.

URL Grammar

This subsection specifies the grammar for URL-strings. Implementations must use this grammar for parsing URLs from strings. The grammar is parameterised by a parser-mode. Specifically, the auth-path rule, the authority rule and the s rule have different versions for different parser-modes.

Strict URL Grammar

For an URL-string to be considered valid, it must conform to a more restricted grammar. The strict grammar is obtained by modifying the auth-drive and pcts rules and the url-char and s character-sets as follows:

IPv6 Addresses

Double check the following, and include it here

IPv6 Address — Strict

An IPv6 address-string is a representation of a natural number n < 2¹²⁸ that is used as an identifier. It is accurately described by the production rule IPv6address in the Host section of RFC 3986.

The IPv6 address parser in the WHATWG standard however implies a more tolerant definition of IPv6 address-strings.

IPv6 Address — loose

A note about multiple slashes

Web browsers interpret any amount of slashes after a web-scheme as the start of the authority component. Consider the following URL-strings:

Web browsers treat all these examples as equivalent to http://foo/bar. It is tempting to try to express this behaviour on the level of the grammar. For example one might consider using the following rule:

auth-path  ::=  s* authority [ path-root [ dir s ]* [ file ] ]

However, the examples above do behave differently with respect to reference resolution. For example, if they are resolved against the base-URL that is represented by http://host/, then the results are as follows:

As such, collapsing the multiples of slashes, cannot be expressed within the grammar. Instead, the force operation as defined in the section on Reference Resolution implements this behaviour.

Host Parsing

IPv4 Address

An IPv4 address-string consists of one up to four dot-separated numbers with an optional trailing dot. The numbers may use decimal, octal or hexadecimal notation, as follows:

Note that 0x is parsed as a hexadecimal number. (It will be interpreted as 0).

IPv4 Address – Strict

A valid IPv4 address-string consists of four dot-separated numbers, without a trailing dot. The numbers must use decimal notation, as follows:

Equivalences and Normalisation

This section is analogous to the section Normalization and Comparison of RFC 3986. The RFC however, does not prescribe a particular normal form. The WHATWG standard does, however implicitly.

Path Segment Normalisation

Path segment normalisation involves the interpretation of dotted-segments. Colloquially, a single-dot segment has the meaning of “select the current directory” whereas a double-dot segment has the meaning of “select the parent directory”. Dotted segments are defined by the following rules, where the addition of %2e and %2E has been motivated by security concerns. Again this is in accordance with the WHATWG standard.

Path equivalence is defined by the following equations. The equations must be exhaustively applied from left-to-right to normalise an URL.

Authority Normalisation

Authority equivalence is defined by the following equations. Like path normalisation, the equations must be exhaustively applied from left-to-right to normalise an URL. This has the same effect as removing any empty port or password component, and if the URL does not have a password after that, to also remove any empty username component.

Scheme-Based Authority Normalisation

If an URL has a scheme, then a number of additional equivalences apply to the authority. Normalisation according to these rules involves the removal of default ports, and similarly, removing the host from a file-URL if its value is localhost.

These rules apply in combination with the following rule that states that scheme equivalence is case-insensitive.

Percent Coding Normalisation

There is a natural notion of equivalence for percent-encoded-strings: one could consider two percent-encoded strings to be equivalent if their percent-decodings are equal. However, for URL-components, this notion of equivalence is too strong.

… RFC3986 specifies reserved characters to allow for additional, application-specific interpretation. For reserved characters, one cannot assume that they are equivalent to their percent-encoding.

The WHATWG standard does not specify the semantics of percent-encoded-bytes in the components of an URL other than the host at all, and any percent-encoded bytes that are present in other components are not decoded. It does however specify sets of characters that must be encoded.

Percent Encode Profile

A percent-encode-profile is a mapping that maps each component-type in the set { username, password, host, dir, file, query, fragment } to a percent-encode-set.

Unfortunately, like parsing, the percent-encoding of URLs as prescribed by the WHATWG standard, is scheme dependent.

Encode Profile Selection

The percent-encode-profile-for a given URL is

• the special profile — if the URL is a file-URL or a web-URL,
• the special profile — if the URL does not have a scheme,
• the generic profile — if otherwise the URL has an authority or a path-root, or
• the minimal profile — otherwise.

Printing

Printing is the process of converting an URL to an URL-string. To print an URL, it must first be subjected to an additional normalisation operation, and it must be percent encoded as follows.

Normalise for Printing

If an URL does not have a drive nor an authority but it does have an empty first dir component (dir ε) then it must be normalised for printing by inserting a component (dir .) immediately before its first dir component.

This additional normalisation step is necessary because it is not possible to represent the affected URLs as an URL-string.

Consider for example the URL (path-root /)・(dir ε)・(file foo). Printing this URL without an additional normalisation step would result in //foo which represents the URL (authority (host foo)) instead.

This is not sufficient. There are similar issues around URLs with drive letters, which is an open issue.

• To print to an ASCII URL-string, percent encode any \ printable-ASCII
• Otherwise percent encode control-c0, del-c1, surrogates and non-chars.
• In addition percent encode a subset of printable-ASCII, depending on the component, as indicated by the percent coding table.

• Let output be the empty string, then, for each of the components (tag, value) of url2 in tree order, convert the component to a string, depending on its type according to the following table, and append it to the output.

Concluding

This final section specifies the behaviour of a parse-resolve-and-normalise operation that characterises the behaviour of the ‘basic URL parser’ as described in the WHATWG URL standard.