URL Specification

Prelude

This section introduces basic concepts and notational conventions that are used throughout the rest of this document.

Components

typevalue The word component is used throughout this document to mean a tagged value, denoted by (type value), where type is taken from a predefined set of component-types.

The component-type is typeset in boldface and it may be used stand-alone as a component-type or in prefix position to denote a component.

For example, scheme denotes a component-type, whilst e.g. (scheme http) denotes a scheme-component that has the string http as its value.

When we are dealing with a collection of components, and the collection contains exactly one component with a given component-type, then we may use the type of the component to refer to its value directly.

For example, if S is a sequence of components (dir x・file y・query z) then the phrase “the query of S” is used to refer to the value z whereas “the query component of S” is used to refer to the component (query z) as a whole.

The value of a component may be a sequence of components itself, in which case the component may be referred to as a compound-component for clarity.

Strings and Code-Points

For the purpose of this specification,

• A string is a sequence of characters.
• A character is a single Unicode code point.
• A code point, is a natural number n < 17 × 2¹⁶.
• The empty string is a sequence of zero code points. It is denoted by ε.

Code points are denoted by a number in hexadecimal notation preceded by u+, in boldface. Code points that correspond to printable ASCII characters are often denoted by their corresponding glyph, typeset in monospace and on a screened background. For example, u+41 and A denote the same code point. Strings that contain only printable ASCII characters are often denoted as a connected sequence of glyphs, typeset likewise.

The printable ASCII characters are code points in the range u+20 to u+7E, inclusive. Note that this includes the space character u+20.

Fix the notaton of 2¹⁶ above

Grammars

The notation name ::= expression is used to define a production rule of a grammar, where the expression uses square brackets ( [ … ] ) for optional rules, a postfix star ( * ) for zero-or-more, a postfix plus ( + ) for one-or-more, an infix vertical line ( | ) for alternatives, monospaced type for literal strings and an epsilon ( ε ) for the empty string. Concatenation takes the highest precedence, followed by ( * ) and ( + ), and ( | ) is used with lowest operator precedence. Parentheses are used for grouping and disambiguation.

Pattern Testing

The shorthand notation string :: rule is used to express that a string string can be generated by the production rule rule of a given grammar.
Likewise, the notation string :: expression is used to express that string can be generated by expression.

Percent Coding

This subsection is analogous to the section Percent-Encoding of RFC 3986 and the section Percent-encoded bytes of the WHATWG standard.

Bytes

For the purpose of this specification,

• A byte is a natural number n < 2⁸.
• A byte-sequence is a sequence of bytes.

Percent Encoded Bytes

A non-empty byte-sequence may be percent-encoded as a string by rendering each individual byte in two-digit hexadecimal notation, prefixed by %.

Percent Encoded String

A percent-encoded–string is a string that may have zero or more percent-encoded–byte-sequences embedded within, as follows:

The grammar above may be relaxed in situations where error recovery is desired. This is achieved by adding a pct-invalid rule as follows:

There is an ambiguity in the loose grammar due to the overlap between pct-encoded-bytes and pct-invalid combined with uncoded. The ambiguity must be resolved by using the pct-encoded-bytes rule instead of the pct-invalid rule wherever possible.

Percent Encoding

To percent-encode a string string using a given character-set encode-set …

The only tricky bit in this is is the encoding of multi-byte UTF8 sequences, and the optional encoding override to specify a non-UTF8 encoding for the query component.

Percent Decoding

Percent decoding of strings is stratified into the following phases:

• Analyse the string into pct-encoded-bytes, pct-invalid and uncoded parts according to the pct-encoded-string grammar.
• Convert each of the the pct-encoded-bytes parts to a byte-sequence and decode the byte sequence to a string assuming that byte-sequence uses the Unicode UTF-8 encoding. Leave the pct-invalid and uncoded parts unmodified.
  
• Recompose the string by concatenating each of the parts

URL Model

An URL, or specifically an URL-structure, is conceptually distinct from an URL-string. An URL-structure is a collection of components that adheres to a number of constraints, whereas an URL-string is a string that represents an URL. An URL-string has an internal structure that can be parsed as an URL-structure. Conversely, an URL-structure may be converted to an URL-string.

In an URL-string the component values are delineated from each other by means of sigils. Most component-types have an associated sigil that is used either in a prefix– or postfix position to its value. However, in the case of the path-root the sigil is the single / character itself, and the file, host and username components do not have an associated sigil at all. The components of an URL are paired with sigils as follows:

Note that the second character : or | of a drive component value is not considered to be its sigil, but is instead a part of its value.

Types and Shapes

There is a further categorisation of URLs that is used in several of the operations that will be defined later.

Authority Model

The authority component of an URL is a compound-component and its value is an Authority structure:

In an URL-string, the components of an Authority are paired with sigils as follows:

Finally, the components of the Userinfo are paired with sigils as follows:

The WHATWG standard specifies scheme-dependent expectations on the Host of an URL. For any generic URL it merely enforces that its opaque-host does not contain certain characters, as is specified by the URL-grammar. However, for file and web-URLs it attempts to parse their opaque-host as a domain or ipv4-address. This additional Host parsing is stratified into the following phases:

• Percent decode.
• Apply domain-to-ASCII
• Err on 'forbidden domain codepoints'.
• Detect and interpret IPv4 addresses.

IPv6 Address

IPv6 Address — Strict

An IPv6 address-string is a representation of a natural number n < 2¹²⁸ that is used as an identifier. It is accurately described by the production rule IPv6address in the Host section of RFC 3986.

The IPv6 address parser in the WHATWG standard however implies a more tolerant definition of IPv6 address-strings.

IPv6 Address — Loose

Domain and IPv4 Address

Component Characters

When it comes to handling characters that may occur in URL-component values we have ‘for historical reasons’ ended up in a situation where we distinguish per component, characters that are:

• v:  valid;
• E:  valid but percent-encoded;
• T:  invalid but tolerated;
• F:  invalid and fixed by percent-encoding;
• R:  invalid and rejected.

In order to specify the how characters must be handled in components, it us useful to first divide the entire space of characters any into non-overlapping character sets as follows:

This collection of character sets covers the entire character space. To exhaustively specify how characters in components should be handled we divide each of these sets into further subdivisions and specify per component the status of the characters in this subset:

Note that applying the rules for encoding component characters does not necessarily produce valid URLs, specifically, if a component contains a character that is marked as T: invalid but tolerated.

Note that the apostrophe ' must instead be left untouched in the query of non-special urls; and the path of generic urls that have no auth nor root are handled as per the opaque path column.

Reference Resolution

This section defines reference resolution operations that are analogous to the algorithms that are described in the chapter Reference Resolution of RFC 3986. The operations that are defined in this section are used in a subsequent section to define a parse-resolve-and-normalise operation that accurately describes the behaviour of the ‘basic URL parser’ as defined in the WHATWG URL standard.

Reference resolution as defined in this section does not involve URL-strings. It operates on URLs as defined in the URL Model section above. In contrast with RFC 3986 and with the WHATWG standard, it does not do additional normalisation, which is relegated to the section Equivalences and Normalisation instead.

Order and Prefix

A property that is particularly useful is the order of an URL. Colloquially, the order is the type of the first component of an URL. The order may be used as an argument to specify various prefixes of an URL.

Reference Transformation

Based on the order and the order-prefix we define a “rebase” operation that slightly generalises reference transformation as specified in the section Transform References of RFC 3986.

Be aware that rebase does a bit more than simple sequence concatenation. It may add a path-root component to satisfy the path-root-constraint of the URL model. For example, if base is the URL represented by //host and input is the URL represented by foo/bar then (input onto base) is represented by //host/foo/bar, thus containing a path-root even though neither base nor input has one.

Rebase Properties

The rebase operation has a number of pleasing mathematical properties, as follows:

• ord (url₂ onto url₁)  is the least type of  { ord url₁, ord url₂ }
• (url₃ onto url₂) onto url₁  =  url₃ onto (url₂ onto url₁)
• url₁ onto ε  =  url₁
• ε onto url₁  =  url₁  — if url₁ does not have a fragment.

Forcing

Forcing is used as a final step in the process of resolving a web-URL or a file-URL. It ensures that the forced URL has an authority component and a path-root component. Note that it is possible for the force operations to fail.

Force Properties

The force operation does not have pleasing mathematical properties with respect to the rebase operation. The following equalities do not hold in general:

Resolution

The subsection Transform References of RFC 3986 specifies two variants of reference resolution. A generic, strict variant and an alternative non-strict variant. I have chosen to rename the non-strict variant to legacy resolution. This specification adds a third variant that characterises the behaviour that is specified in the WHATWG standard.

Parsing

Parsing is the process of converting an URL-string to an URL.
Parsing is stratified into the following phases:

1. Preprocessing.
2. Selecting a parser mode.
3. Parsing.
4. Decoding and parsing the host.

Preprocessing

The appendix Delimiting a URI in Context of RFC 3986 states that surrounding white-space should be removed from an URI when it is extracted from its surrounding context. The WHATWG standard makes this more explicit and specifies a preprocessing step that removes removes specific control– and white-space–characters from the input string before parsing.

Equivalences and Normalisation

This section is analogous to the section Normalization and Comparison of RFC 3986. The RFC however, does not prescribe a particular normal form. The WHATWG standard does, however implicitly.

Path Segment Normalisation

Path segment normalisation involves the interpretation of dotted-segments. Colloquially, a single-dot segment has the meaning of “select the current directory” whereas a double-dot segment has the meaning of “select the parent directory”. Dotted segments are defined by the following rules, where the addition of %2e and %2E has been motivated by security concerns. Again this is in accordance with the WHATWG standard.

Path equivalence is defined by the following equations. The equations can be exhaustively applied from left-to-right to normalise an URL.

Authority Normalisation

Authority equivalence is defined by the following equations. Like path normalisation, the equations must be exhaustively applied from left-to-right to normalise an URL. This has the same effect as removing any empty port or password component, and if the URL does not have a password after that, to also remove any empty username component.

Scheme-Based Authority Validation

If an URL has a scheme, then a number of additional requirements may be enforced on the authority; specifically these should be enforced or fixed whilst resolving (as opposed to rebasing) and URL.

Type it out nicely

• file and web URLs must have an authority
• file URL authority must not have a userinfo nor a port component
• web URL must have a non-empty authority
• file- and web URL host must not be an opaque host

Scheme-Based Authority Normalisation

If an URL has a scheme, then a number of additional equivalences apply to the authority. Normalisation according to these rules involves the removal of default ports, and similarly, removing the host from a file-URL if its value is localhost.

These rules apply in combination with the following rule that states that scheme equivalence is case-insensitive.

Printing

Printing is the process of converting an URL to an URL-string. Printing can be stratified into the following three phases:

1. Normalising the URL for printing.
2. Converting each of the components of the URL to a string.
3. Composing the final URL-string from the printed components.

Normalise for Printing

To print an URL, it must first be subjected to an additional normalisation operation, and it must be percent encoded as follows.

Normalise for Printing

If an URL does not have a drive nor an authority but it does have an empty first dir component (dir ε) then it must be normalised for printing by inserting a component (dir .) immediately before its first dir component.

Otherwise, if the first component of an URL is a dir or a file component and its value value starts with a scheme-like string, i.e.

value :: alpha (alpha | digit | + | - | .)* : any*,

then the first occurrence of : in value must be replaced with its percent-encoding %3A.

This additional normalisation step is necessary because it is not possible to represent the affected URL as an URL-string otherwise.

Consider for example the URL (path-root /)・(dir ε)・(file foo). Printing this URL without an additional normalisation step would result in //foo which represents the URL (authority (host foo)) instead.

There is a similar issue around URLs with drive letters, but these issues are not properly addressed in the current WHATWG standard, making it difficult for us to prescribe appropriate counter measures.

Printing URLs

To convert an Authority to an Authority-string, replace each of the sub-components with their component-value concatenated with their prefix– or postfix-sigil, if any. This results in a sequence of strings, which is converted to the final result by concatenating them in order.

To convert an URL to an URL-string, first convert the value of its authority component (if any) to an Authority-string. Then continue to replace each component with their component-value concatenated with their identifying sigil, if any. This results in a sequence of strings, which is converted to a single string by simply concatenating them in order.

Concluding

This final section specifies the behaviour of a parse-resolve-and-normalise operation that characterises the behaviour of the ‘basic URL parser’ as described in the WHATWG URL standard.