Character Model for the World Wide Web: String Matching

Character Model for the World Wide Web: String Matching
Character Model for the World Wide Web: String Matching
W3C
Working Draft
20 April 2018
This version:
Latest published version:
Latest editor's draft:
Bug tracker:
File a bug
open bugs
Previous version:
Editor:
Addison Phillips
(Invited Expert)
Github:
repository
2004-2018
W3C
MIT
ERCIM
Keio
Beihang
).
W3C
liability
trademark
and
permissive document license
rules apply.
Abstract
This document builds upon on
Character Model for the World Wide
Web 1.0: Fundamentals
CHARMOD
] to provide authors of
specifications, software developers, and content developers a common
reference on string identity matching on the World Wide Web and thereby
increase interoperability.
Status of This Document
This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current
W3C
publications and the latest revision of this technical report can be found in the
W3C
technical reports index
at https://www.w3.org/TR/.
Note
This version of the document represents a significant change from the
earlier
editions
. Much of the content is changed and the recommendations
are significantly altered. This fact is reflected in a change to the
name of the document from "Character Model: Normalization".
Note
Sending
comments on this document
If you wish to make comments regarding this document,
please raise them as
github issues
. When reviewing the document,
please refer to the latest
editor's
copy
. Only send comments by email if you are unable to raise issues on github (see
links below). All comments are welcome.
To make it easier to track comments, please raise
separate issues or emails for each comment, and point to the section
you are commenting on using a URL.
This document was published by the
Internationalization Working Group
as a Working Draft.

Comments regarding this document are welcome. Please send them to
www-international@w3.org
archives
).
Publication as a Working Draft does not imply endorsement by the
W3C
Membership. This is a draft document and may be updated, replaced or obsoleted by other
documents at any time. It is inappropriate to cite this document as other than work in
progress.
This document was produced by
a group
operating under the
W3C
Patent Policy

The group does not expect this document to become a
W3C
Recommendation.
W3C
maintains a
public list of any patent
disclosures
made in connection with the deliverables of
the group; that page also includes
instructions for disclosing a patent. An individual who has actual knowledge of a patent
which the individual believes contains
Essential
Claim(s)
must disclose the information in accordance with
section
6 of the
W3C
Patent Policy
This document is governed by the
1 February 2018
W3C
Process Document
1.
Introduction
1.1
Goals and Scope
The goal of the Character Model for the World Wide Web is to facilitate use of the Web by all people, regardless of their language, script, writing system, or cultural conventions, in accordance with the
W3C
goal of universal access
. One basic prerequisite to achieve this goal is to be able to transmit and process the characters used around the world in a well-defined and well-understood way.
Note
This document builds on
Character Model for the World Wide Web: Fundamentals
CHARMOD
]. Understanding the concepts in that document are important to being able to understand nd apply this document successfully.
This part of the Character Model for the World Wide Web covers string
matching—the process by which a specification or implementation defines
whether two string values are the same or different from one another. It
describes the ways in which texts that are semantically equivalent can
be encoded differently and the impact this has on matching operations
important to formal languages (such as those used in the formats and
protocols that make up the Web).
The main target audience of this specification is
W3C
specification
developers. This specification and parts of it can be referenced from
other
W3C
specifications and it defines conformance criteria for
W3C
specifications, as well as other specifications.
Other audiences of this specification include software developers,
content developers, and authors of specifications outside the
W3C
Software developers and content developers implement and use
W3C
specifications. This specification defines some conformance criteria
for implementations (software) and content that implement and use
W3C
specifications. It also helps software developers and content
developers to understand the character-related provisions in
W3C
specifications.
The character model described in this specification provides authors
of specifications, software developers, and content developers with a
common reference for consistent, interoperable text manipulation on
the World Wide Web. Working together, these three groups can build a
globally accessible Web.
1.2
Structure of this Document
This document defines one of the basic building blocks for the Web related
to this problem by defining rules and processes for String
Identity Matching in document formats. These rules are designed for
the identifiers and structural markup (
syntactic content
used in document formats to ensure consistent processing of each and are
targeted to Specification writers. This
section is targeted to implementers.
This document is divided into two main sections.
The
first section
lays out the
problems involved in string matching; the effects of Unicode and case
folding on these problems; and outlines the various issues and
normalization mechanisms that might be used to address these issues.
The
second section
provides
requirements and recommendations for string identity matching for use
in
formal languages
, such as many of the
document formats defined in
W3C
Specifications. This primarily is
concerned with making the Web functional and providing document
authors with consistent results.
1.3
Background
This section provides some historical background on the topics
addressed in this specification.
At the core of the character model is the Universal Character Set
(UCS), defined jointly by the
Unicode Standard
Unicode
] and ISO/IEC 10646 [
ISO10646
]. In this document,
Unicode
is used as a synonym for the Universal Character Set. A successful
character model allows Web documents authored in the world's writing
systems, scripts, and languages (and on different platforms) to be
exchanged, read, and searched by the Web's users around the world.
The first few chapters of the
Unicode Standard
Unicode
] provide useful background reading.
For information about the requirements that informed the development
of important parts of this specification, see
Requirements for
String Identity Matching and String Indexing
CHARREQ
].
1.4
Terminology and Notation
This section contains terminology and notation specific to this document.
The Web is built on text-based formats and protocols. In order to
describe string matching or searching effectively, it is necessary to
establish terminology that allows us to talk about the different kinds
of text within a given format or protocol, as the requirements and
details vary significantly.
Unicode code point
(or "code point") refers to the numeric value assigned to each Unicode character. Unicode code points range from 0 to 0x10FFFF
16
. (See Section 4.1 in [
CHARMOD
] for a deeper discussion of character encoding terminology.)
Unicode code points are denoted as
U+
hhhh
, where
hhhh
is a sequence of at least four, and at most six hexadecimal digits. For example, the character
U+20AC EURO SIGN
has the code point
U+20AC
Some characters used in this document's examples might not appear as intended on your specific device or display. Usually this is due to lack of a script-specific font installed locally or due to other limitations of your specific rendering system. This document uses a Webfont to provide fallback glyphs for many of the non-Latin characters, but your device might not support displaying the font. To the degree possible, the editors have tried to ensure that the examples nevertheless remain understandable.
legacy character encoding
is a character encoding not based on the Unicode character set.
transcoder
is a process that converts
text between two character encodings. Most commonly in this document it
refers to a process that converts from a
legacy character encoding
to a
Unicode encoding form
such as UTF-8.
Syntactic content
is any text in a document format or protocol that belongs to the structure of the format or protocol. This definition includes values that are typically thought of as "markup" but can also include other values, such as the name of a field in an HTTP header. Syntactic content consists of all of the characters that form the
structure
of a format or protocol. For example,
and
(as well as the element name and various attributes they surround) are part of the syntactic content in an HTML document.
Syntactic content usually is defined by a specification or specifications and
includes both the defined, reserved keywords for the given protocol or
format as well as string tokens and identifiers that are defined by
document authors to form the structure of the document (rather than
the "content" of the document).
Example 1
XML
XML10
] defines specific elements, attributes,
and values that are reserved across all XML documents. Thus, the
word
encoding
has a defined
meaning inside the XML document declaration: it is a reserved name.
XML also allows a user to define elements and attributes for a given
document, for example, by using a DTD. In a document that uses a DTD that defines an
element called

muffin
is a part of the syntactic content.
Natural language content
refers to the language-bearing
content in a document and
not
to any of the surrounding or embedded
syntactic content that form part of the document structure. You can think
of it as the actual "content" of the document or the "message" in a
given protocol. Note that syntactic content can contain natural
language content, such as when an [
HTML
img
element
has an
alt
attribute containing a description of
the image.
resource
, in the context of this document, is a given
document, file, or protocol "message" which includes both the
natural
language content
as well as the
syntactic content
such as identifiers surrounding or containing it. For example, in an
HTML document that also has some CSS and a few
script
tags with embedded JavaScript, the entire HTML document, considered as
a file, is a resource. This term is intentionally similar to the term
'resource' as used in [
RFC3986
], although here the term is applied
loosely.
user-supplied value
is unreserved syntactic
content in a
vocabulary
that is assigned by users, as distinct
from reserved keywords in a given format
or protocol. For example, CSS class names are part of the syntax of a CSS
style sheet. They are not reserved keywords, predefined by any CSS specification.
They are subject to the syntactic rules of CSS. And they may (or may not)
consist of natural language tokens.
vocabulary
provides the list of
reserved names as well as the set of rules and specifications
controlling how
user-supplied values
(such as identifiers) can be assigned in a
format or protocol. This can include restrictions on range, order, or
type of characters that can appear in different places. For example,
HTML defines the names of its elements and attributes, as well as
enumerated attribute values, which defines the "vocabulary" of HTML
syntactic content
Another example would be ECMAScript, which restricts the range of
characters that can appear at the start or in the body of an identifier
or variable name. It applies different rules for other cases, such as to
the values of string literals.
grapheme
is a sequence of
one or more characters in a visual representation of some text
that a typical user would perceive as being a single unit (
character
).
Graphemes are important for a number of text operations such as
sorting or text selection, so it is necessary to be able to compute
the boundaries between each user-perceived character. Unicode defines
the default mechanism for computing graphemes in
Unicode
Standard Annex #29: Text Segmentation
UAX29
] and calls
this approximation a
grapheme cluster
. There are two types
of default grapheme cluster defined. Unless otherwise noted, grapheme
cluster in this document refers to an extended default grapheme
cluster. (A discussion of grapheme clusters is also given in Section 2
of the
Unicode Standard
, [
Unicode
]. Cf. near the end of
Section 2.11
in version 8.0 of The Unicode Standard)
Because different natural languages have different needs, grapheme clusters
can also sometimes require tailoring. For example, a Slovak user might
wish to treat the default pair of grapheme clusters "ch" as a single
grapheme cluster. Note that the interaction between the language of
string content and the end-user's preferences might be complex.
Example 2
The Hindi word for Unicode
यूनिकोड
is composed of seven Unicode characters from the Devanagari script.
Most users would identify this word as containing four units of text. Each of the first three graphemes consists of two characters: a syllable and a modifying vowel character. So the word contains seven Unicode characters, but only four graphemes:
Word
यूनिकोड
Graphemes
यू
नि
को
Code Points
U+092F
U+0942
U+0928
U+093F
U+0915
U+094B
U+0921
1.4.1
Terminology Examples
This section illustrates some of the terminology defined above. For illustration purposes we'll use the following small HTML file as an example (line numbers added for reference):
html
lang
="
en
dir
="
ltr
">
head
meta
charset
="
UTF-8
">
title
Shakespeare
title
head
body
img
src
="
shakespeare.jpg
alt
="
William Shakespeare
id
="
shakespeare_image
">
What
’
s in a name? That which we call a rose by any other name would smell as sweet.
body
10
html
Everything inside the black rectangle (that is, in this HTML file) is part of the
resource
Syntactic content
in this case includes all of the HTML markup. There are only two strings that are
not
part of the syntactic content: the word
"Shakespeare"
on line 4 and the sentence
"What’s in a name? That which we call a rose by any other name would smell as sweet."
on line 8. (The HTML entity
’
embedded in the sentence on line 8
is
part of the syntactic content.)
Natural language content
is shown in a
bold blue font with a gray background
. In addition to the non-syntactic content, the
alt
value on line 7 (
William Shakespeare
) contains natural language text.
User-supplied values are shown in
italics
. In this case there are three user-supplied values on line 7: the values of the
src
alt
, and
id
attributes of the
img
tag. In addition, the value of the
lang
attribute on line 1 and the
charset
attribute on line 3 are user-supplied values.
Vocabulary
is shown with
red underlining
. The vocabulary of an HTML document are the elements and attributes (as well as some of the attribute values, such as the value
ltr
for the attribute
dir
in the example above) defined in [
HTML5
].
Note
All of the text above (all text in a text file) makes up the resource. It's possible that a given resource will contain no natural language content at all (consider an HTML document consisting of four empty
div
elements styled to be orange rectangles). It's also possible that a resource will contain
no
syntactic content and consist solely of natural language content: for example, a plain text file with a soliloquy from
Hamlet
in it. Notice too that the HTML entity
’
appears in the natural language content and belongs to both the natural language content and the syntactic content in this resource.
1.5
Conformance
As well as sections marked as non-normative, all authoring guidelines, diagrams, examples,
and notes in this specification are non-normative. Everything else in this specification is
normative.
The key words
MAY
MUST
MUST NOT
RECOMMENDED
SHOULD
, and
SHOULD NOT
are
to be interpreted as described in [
RFC2119
].
This document describes best practices for the authors of other specifications, as well as recommendations for implementations and content authors. These best practices can also be found in the Internationalization Working Group's document
INTERNATIONAL-SPECS
, which is intended to serve as a general reference for all Internationalization best practices in
W3C
specifications.
When a best practice or recommendation appears in this document, it has been styled to like this paragraph. Recommendations for specifications and spec authors are preceded by
[S]
. Recommendations for implementations and software developers are preceeded by
[I]
. Recommendations for content and content authors are preceeded by
[C]
Best practices in this document use [
RFC2119
] keywords in order to clarify the Internationalization Working Group's intentions regarding a specific recommendation. Following the recommendations in this document can help avoid issues during the
W3C
's "wide review" process, during implementation, or in the content that authors produce. This document is not, itself, normative and can be revised from time to time.
Specifications can claim conformance to this document if they:
do not violate any conformance criteria preceded by
[S]
where the imperative is
MUST
or
MUST NOT
document the reason for any deviation from criteria where the imperative is
SHOULD
SHOULD NOT
, or
RECOMMENDED
make it a conformance requirement for implementations to conform to this document
make it a conformance requirement for content to conform to this document
Note
Requirements placed on specifications might indirectly cause requirements to be placed on implementations or content that claim to conform to those specifications.
Where this specification contains a procedural description, it is to be understood as a way to specify the desired external behavior. Implementations
MAY
use other means of achieving the same results, as long as observable behavior is not affected.
2.
The String Matching Problem
The Web is primarily made up of document formats and protocols based on
character data. These formats or protocols can be viewed as a set of
resources
consisting mainly of text files that include some form of structural markup or
syntactic content
. Processing such syntactic content or document data requires
string-based operations such as matching (including regular expressions), indexing, searching, sorting,
and so forth.
Users, particularly implementers, sometimes have naïve expectations regarding the matching or non-matching
of similar strings or of the efficacy of different transformations they might apply to text, particularly to
syntactic content, but including many types of text processing on the Web.
Because fundamentally the Web is sensitive to the different ways in which text might be represented in a
document, failing to consider the different ways in which the same text can be represented can confuse
users or cause unexpected or frustrating results. In the sections below, this document examines the different
types of text variation that affect both user perception of text on the Web and the string processing on which
the Web relies.
2.1
Case Mapping and Case Folding
Some scripts and writing systems make a distinction between UPPER,
lower, and Title case characters. Most scripts, such as the Brahmic
scripts of India, the Arabic script, and the scripts used to
write Chinese, Japanese, or Korean do not have a case distinction, but
some important ones do. Examples of such scripts include the Latin
script used in the majority of this document, as well as scripts such
as Greek, Armenian, and Cyrillic.
Case mapping
is the process of transforming characters to a specific case, such as upper, lower, or titlecase. For those scripts that have a case distinction, Unicode defines a
default
UPPER, lower, and Title case character mapping for each Unicode code point. Case mapping, at first, appears simple. However there are variations that need to be considered when treating the full range of Unicode in diverse languages.
Note
For more information,
Unicode
Chapter 5 (in v8.0,
Section 5.18
) discusses case mappings and case folding in detail.
Example 3
For example here is a character with mappings to all three case variations. These mappings are defined in the Unicode Character Database (UCD).
Uppercase
Lowercase
Titlecase
U+01C4
U+01C6
U+01C5
Case folding
is the process of making two texts which differ only in case identical for comparison purposes, that is, it is meant for the purpose of string matching. This is distinct from
case mapping
, which is primarily meant for display purposes. As with the default case mappings, Unicode defines default case fold mappings for each Unicode code point. Unicode defines two forms of case fold mapping, which we'll examine below.
Since most scripts do not have a case distinction, as with case mappings, most Unicode code points do not require a case fold mapping. For those characters that
have a case fold mapping, the majority have a simple, straight-forward mapping to a single matching (generally lowercase) code point. Unicode
calls these the
common
case fold mappings, as they are shared by Unicode's case fold mappings.
Example 4
Here are some examples of
common
case fold mappings:
LATIN CAPITAL LETTER A
to
LATIN SMALL LETTER A
GREEK CAPITAL LETTER THETA
to
GREEK SMALL LETTER THETA
CYRILLIC CAPITAL LETTER DE
to
CYRILLIC SMALL LETTER DE
A few characters have a case fold mapping that map one Unicode code point to two or more code points during case folding. These are called the
full
case fold mappings. The
full
and
common
case fold mappings are used together to provide the default case fold mapping for all of Unicode. We refer to this form of case folding as
full casefolding
or
Unicode full
in this document.
Example 5
One well-known example of a
full
case fold mapping is the character
U+00DF LATIN SMALL LETTER SHARP S
, a letter that is commonly
used in the German language. The
full
case fold mapping and the lower case mapping of this character is to two ASCII letters 's'. The upper case mapping is to "SS".
ss
LATIN SMALL LETTER SHARP S
to
LATIN SMALL LETTER S
LATIN SMALL LETTER S
Because some applications cannot allocate additional storage when performing a case fold operation, Unicode provides a
simple
case fold mapping that maps characters that would normally map to more or
fewer code points to use a single code point for comparison purposes instead. Unlike the full mapping, this mapping invariably alters the content (and potentially the meaning) of the text. As with
full casefolding
, the
simple casefolding
or
Unicode simple
casefold, is a combination of
simple
and
common
mappings so as to cover the full range of Unicode.
Unicode simple
is not appropriate for use on the Web.
Example 6
Examples of
full
versus
simple
case fold variations can be found in the Greek script, where several precomposed characters have multi-character case fold mappings. The table below shows one such example, the character
U+1F9B GREEK CAPITAL LETTER ETA WITH DASIA AND VARIA AND PROSGEGRAMMENI
and its
full
and
simple
case fold mappings:
ἣι
full
case fold:
U+1F23 U+03B9
GREEK SMALL LETTER ETA WITH DASIA AND VARIA
GREEK SMALL LETTER IOTA
simple
case fold:
U+1F93
GREEK SMALL LETTER ETA WITH DASIA AND VARIA AND PROSGEGRAMMENI
Note that case folding removes information from a string which cannot
be recovered later. For example, two
letters in German do not necessarily represent
in unfolded text.
2.1.1
Language Sensitivity
Another aspect of case mapping and case folding is that it can be language sensitive.
Unicode defines
default
case mappings and case fold mappings for each encoded character, but
these are only defaults and are not appropriate in all cases. Some
languages need case-folding to be tailored to meet specific linguistic
needs. One common example of this are Turkic languages written in the
Latin script:
Example 7
Default Folding
Default folding of letter I
Turkic Language Folding
Turkic language folding of dotless (ASCII) letter I
Turkic language folding of dotted letter I
While the example above (and this document in general) focuses on case folding for the purpose of matching, note that case mapping is also language-specific. The name of the second largest city in Turkey is "
Diyarbakır
", which
contains both the dotted and dotless letters
Example 8
Diyarbakır
text-transform: uppercase
DİYARBAKIR
Notice that the ASCII letter
U+0069 LATIN SMALL LETTER I
maps to
U+0130 LATIN CAPITAL LETTER I WITH DOT ABOVE
, while the letter
U+0131 LATIN SMALL LETTER DOTLESS I
maps to the ASCII uppercase
U+0049 LATIN CAPITAL LETTER I
. Failure to apply this localized case mapping would change the meaning of the text in Turkish, even though this is the expected mapping in other languages, such as English or German.
This language-specific tailoring can also be applied to case folding. For example, if the uppercase text needed to be matched against some set of strings in a case-insensitive way:
DİYARBAKIR
case fold
diyarbakır
2.1.2
Uses for Case Folding
Some document formats or protocols seek to aid interoperability or provide an aid to content authors by ignoring case variations in the
vocabulary
they define or in
user-supplied values
permitted by the format or protocol.
Example 9
One example where this occurs is when matching element names between an HTML document and its associated style sheet. Consider this HTML fragment:
style
type
"text/css"
SPAN
.hello
text-decoration
: underline;
style
span
class
"hello"
Hello World!
span
The
SPAN
in the stylesheet
matches the
span
element in the
document, even though the stylesheet uses uppercase and the HTML markup
does not.
Sometimes case can vary in a way that is not semantically meaningful
or is not fully under the user's control. This is particularly true
when searching a document, but may sometimes also apply
when defining rules for matching user- or content-generated values,
such as identifiers. In these situations, case-
in
sensitive
matching might be desirable instead.
When defining a
vocabulary
, one important consideration is whether the values are restricted to the ASCII [
ASCII
] subset of Unicode or if the vocabulary permits the use of characters (such as accents on Latin letters or a broad range of Unicode including non-Latin scripts) that potentially have more complex case folding requirements. To address these different requirements, there are four types of casefold matching defined by this document for the purposes of string identity matching in document formats or protocols:
Case sensitive matching
: code points are compared directly with no case folding.
ASCII case-insensitive matching
compares a sequence of code points as if all ASCII code points in the
range 0x41 to 0x5A (A to Z) were mapped to the corresponding code
points in the range 0x61 to 0x7A (a to z). When a vocabulary is itself
constrained to ASCII, ASCII case-insensitive matching can be required.
Unicode case-insensitive matching
compares a sequence of code points as if the
Unicode full
casefolding (see above) had been applied to both input sequences.
Language-sensitive case-sensitive matching
is useful in the rare case where a document format or protocol contains information about the language of the syntactic content and where language-sensitive case folding might sensibly be applied. These case-fold mappings are defined in the
Common Locale Data Repository
UAX35
] project of the Unicode Consortium.
For advice on how to handle case folding see
3.1.4
Choice of Case Folding
2.2
Unicode Normalization
A different kind of variation can occur in Unicode text: sometimes several different
Unicode code point
sequences can be used to represent the same abstract character. When searching or matching text by comparing code points, these variations in encoding cause text values not to match that users expect to be the same.
Example 10
: Encoding Variations
Consider the character
U+01FA LATIN CAPITAL LETTER A WITH RING ABOVE AND ACUTE
. One way to encode this character is as
U+01FA LATIN LETTER CAPITAL A WITH RING ABOVE AND ACUTE
. Here are some of the different character sequences that a document could use to represent this character:
U+01FA
—A "precomposed" character.
Ǻ
A + U+030A + U+0301
— A
base
letter
followed by two combining marks (
U+030A COMBINING RING ABOVE
and
U+0301 COMBINING ACUTE ACCENT
Ǻ
U+00C5 + U+0301
—An accented letter (
U+00C5 LATIN CAPITAL LETTER A WITH RING ABOVE
) followed by a combining accent (
U+0301 COMBINING ACUTE ACCENT
Ǻ
U+212B + U+0301
—A compatibility character (
U+212B ANGSTROM SIGN
) followed by a combining accent (
U+0301 COMBINING ACUTE ACCENT
Ａ̊́
U+FF21 + U+030A + U+0301
— A compatibility character
U+FF21 FULLWIDTH LATIN LETTER CAPITAL A
) followed by two combining marks (
U+030A COMBINING RING ABOVE
and
U+0301 COMBINING ACUTE ACCENT
Each of the above strings contains the same apparent
meaning
as
U+01FA LATIN CAPITAL LETTER A WITH RING ABOVE AND ACUTE
, but each one is encoded slightly differently. More variations are possible, but are omitted for brevity.
Because applications need to find the semantic equivalence in texts
that use different code point sequences, Unicode defines a means of
making two semantically equivalent texts identical: the Unicode
Normalization Forms [
UAX15
].
Note Well
Unicode Normalization does not guarantee that two
identical-appearing strings that are in a given Unicode Normalization Form use the same sequence of code points.
See
2.3
Identical-Appearing Characters and the Limitations of Normalization
for more information.
Resources
are often susceptible to the
effects of these variations because their specifications and
implementations on the Web do not require Unicode Normalization of the
text, nor do they take into consideration the string matching
algorithms used when processing the syntactic content and natural language content later. For this
reason, content developers need to ensure that they have provided a
consistent representation in order to avoid problems later.
However, it can be difficult for users to assure that a given
resource
or set of resources uses a consistent textual representation because
the differences are usually not visible when viewed as text. Tools and
implementations thus need to consider the difficulties experienced by
users when visually or logically equivalent strings that "ought to"
match (in the user's mind) are considered to be distinct values.
Providing a means for users to see these differences and/or normalize
them as appropriate makes it possible for end users to avoid failures
that spring from invisible differences in their source documents. For
example, the
W3C
Validator warns when an HTML document is not fully in
Unicode Normalization Form C.
2.2.1
Canonical vs. Compatibility Equivalence
Unicode defines two types of equivalence between characters:
canonical
equivalence
and
compatibility equivalence
Canonical equivalence
is a fundamental equivalency
between Unicode characters or sequences of Unicode characters that
represent the same abstract character. When correctly displayed,
these should always have the same visual appearance and behavior.
Generally speaking, two canonically equivalent Unicode texts should
be considered to be identical as text. Unicode defines a process called
canonical decomposition
that removes these primary distinctions between two texts.
Examples of canonical equivalence defined by Unicode include:
vs.
Ç
Precomposed versus combining
sequences.
Some characters can be composed from a base
character followed by one or more combining characters. The same
characters are sometimes also encoded as a distinct "precomposed"
character. In this example, the character
U+00C7 LATIN CAPITAL LETTER C WITH CEDILLA
is canonically equivalent to the character sequence starting with the base character
U+0043 LATIN CAPITAL LETTER C
followed by
◌̧
U+0327 COMBINING CEDILLA​
. Such equivalence can extend to characters with multiple combining marks.
q̣̇
vs.
q̣̇
Order of combining marks.
When
a base character is modified by multiple combining marks, the
order of the combining marks might not represent a distinct
character. Here the sequence
U+0071 LATIN SMALL LETTER Q
U+0307 COMBINING DOT ABOVE​
U+0323 COMBINING DOT BELOW​
and
U+0071 LATIN SMALL LETTER Q
U+0323 COMBINING DOT BELOW​
U+0307 COMBINING DOT ABOVE​
are equivalent, even though the combining marks are in a different order. Note that this example is chosen
carefully: the dot-above character and dot-below character are on
opposite "sides" of the base character. The order of combining
diacritics on the same side have a positional meaning.
vs.
Singleton mappings.
These result from the need to separately encode otherwise equivalent characters to support legacy character encodings. In this example, the Ohm symbol
U+2126 OHM SYMBOL
is canonically equivalent (and identical in appearance) to the Greek letter Omega
U+03A9 GREEK CAPITAL LETTER OMEGA
. (Another example of a singleton is
U+212B ANGSTROM SIGN
in the
encoding variations example
above)
vs.
가
Hangul.
The Hangul script is
used to write the Korean language. This script is constructed
logically, with each syllable being a roughly-square
grapheme
formed from specific sub-parts that represent consonants and
vowels. These specific sub-parts, called
jamo
, are
encoded in Unicode. So too are the precomposed syllables. Thus the
syllable
U+AC00 [Hangul Syllable, First]
is canonically equivalent to its constituent
jamo
characters
U+1100 HANGUL CHOSEONG KIYEOK
U+1161 HANGUL JUNGSEONG A
Compatibility equivalence
is a weaker equivalence
between Unicode characters or sequences of Unicode characters that represent the
same abstract character, but may have a different visual appearance
or behavior. Generally the process called
compatibility decomposition
removes
formatting variations, such as superscript, subscript, rotated,
circled, and so forth, but other variations also occur. In many
cases, characters with compatibility decompositions represent a
distinction of a semantic nature; replacing the use of distinct
characters with their compatibility decomposition can therefore
change the meaning of the text. Texts that are equivalent after
compatibility decomposition often were not perceived as being
identical beforehand and
SHOULD NOT
be treated as equivalent by a formal
language.
Example 11
The following table illustrates various kinds of compatibility equivalence in Unicode:
Compatibility Equivalance
Original Character
Compatibility Mapping
Font variants
—characters that have a specific visual appearance (generally associated with a specialized use, such as in mathematics).
U+201C
U+0048
U+210D
Breaking versus non-breaking
—variations in breaking or joining rules, such as the difference between a
normal
and a non-breaking space.
NON-BREAKING SPACE
U+00A0
SPACE
U+0020
Presentation forms of Arabic
— characters that encode the specific shapes (initial, medial, final, isolated) needed by visual legacy encodings of the Arabic script.
U+FEE5
U+0646
U+FEE6
U+FEE7
U+FEE8
Circled
—numbers, letters, and other characters in a circled, bullet, or other presentational form; often used for lists, footnotes, and specialized presentation
U+2460
U+0031
U+2469
10
U+0031 U+0030
U+24B6
U+0041
U+329E
U+5370
Width variation, size, rotated presentation forms
—narrow vs. wide presentational forms of characters (such as those associated with legacy multibyte encodings), as well as "rotated" presentation forms necessary for vertical text.
U+FF76
U+30AB
U+FE37
U+007B
U+FF5B
U+FF21
U+0041
Superscripts/subscripts
—superscript or subscript letters, numbers, and symbols.
U+2079
U+0039
U+2089
U+00AA
U+0061
U+208A
U+002B
Squared
characters
—East Asian (particularly kana) sequences encoded as a presentation form to fit in a single ideographic "cell" in text.
U+3300
アパート
U+30A2 U+30D1 U+30FC U+30C8
U+3350
ユアン
U+30E6 U+30A2 U+30F3
U+3250
PTE
U+0050 U+0054 U+0045
U+3389
kcal
U+006B U+0063 U+0061 U+006C
Fractions
—precomposed vulgar fractions, often encoded for compatibility with font glyph sets.
U+00BC
1⁄4
U+0031 U+2044 U+0034
U+00BD
1⁄2
U+0031 U+2044 U+0032
U+215F
1⁄
U+0031 U+2044
U+2189
0⁄3
U+0030 U+2044 U+0033
Others
—compatibility characters encoded for other reasons, generally for compatibility with legacy character encodings. Many of these characters are simply a sequence of characters encoded as a single presentational unit.
U+01C6
dž
U+017E
U+2474
(1)
U+0028 U+0031 U+0029
U+2488
1.
U+0031 U+002E
U+2EF3
U+9F9F
In the above table, it is important to note that the characters
illustrated are
actual Unicode codepoints
, not just presentational
variations due to context or style. Each character was
encoded into Unicode for compatibility with various legacy character
encodings. They should not be confused with the normal kinds of
presentational processing used on their non-compatibility
counterparts.
For example, most Arabic-script text uses the characters in the Arabic script block of Unicode (starting at
U+0600
). The actual glyphs used to display the text are selected using fonts and text processing logic based on the position inside a word (initial, medial, final, or isolated), in a process called "shaping". In the table above, the four presentation forms of the Arabic letter
U+0646 ARABIC LETTER NOON
are shown. The characters shown are compatibility characters in the
U+FE00
block, each of which represents a specific "positional" shape and each of the four code points shown have a compatibility decomposition to the
regular
Arabic letter
U+0646 ARABIC LETTER NOON
Similarly, the variations in half-width and full-width forms and rotated
characters (for use in vertical text) are encoded as separate code
points, mainly for compatibility with legacy character encodings. In
many cases these variations are associated with the Unicode properties
described in
East Asian Width
UAX11
]. See also
Unicode
Vertical Text Layout
UTR50
] for a discussion of vertical text
presentation forms.
In the case of characters with compatibility decompositions, such
as those shown above, the
Unicode
Normalization forms convert the text to the "normal" or "expected"
Unicode code point. But the existence of these compatibility
characters cannot be taken to imply that similar appearance
variations produced in the normal course of text layout and
presentation are affected by Unicode Normalization. They are not.
2.2.2
Composition vs. Decomposition
These two types of Unicode-defined equivalence are then grouped by
another pair of variations: "decomposition" and "composition". In
"decomposition", separable logical parts of a visual character are
broken out into a sequence of base characters and combining marks
and the resulting code points are put into a fixed, canonical order.
In "composition", the decomposition is performed and then any
combining marks are recombined, if possible, with their base
characters. Note that this does
not
mean that all
of the combining marks have been removed from the resulting
normalized text.
Note
Roughly speaking,
NFC
is defined such that each combining character sequence (a base
character followed by one or more combining characters) is
replaced, as far as possible, by a canonically equivalent
precomposed character. Text in a Unicode character encoding form
(such as UTF-8 or UTF-16) is said to be in
NFC
if it doesn't
contain any combining sequence that could be replaced with a
precomposed character and if any remaining combining sequence is
in canonical order.
Users are cautioned that the resulting character sequence can still contain combining marks: not all character sequences have a precomposed equivalent and some scripts depend on combining marks for encoding. There are even cases where a given base character and combining mark is not replaced with a precomposed character because the combination is "blocked" by another combining mark in the sequence.
2.2.3
Unicode Normalization Forms
There are four Unicode Normalization Forms. Each form is named using a letter code:
(or NFD) stands for
canonical Decomposition
(or
NFC
) stands for
Composition
, which is canonical decomposition followed by composition.
KD
(or NFKD) stands for
Kompatibility decomposition
(K because the letter C is already used).
KC
(or NFKC) stands for compatibility decomposition followed by composition.
Example 12
Having converted a resource to a sequence of Unicode characters and unescaped any escape sequences, we can finally "normalize" the Unicode texts given in the example above. Here are the resulting sequences in each Unicode
Normalization form for the
U+01FA
example given earlier. Note that there are only three distinct code points sequences:
Original Codepoints
NFC
NFD
NFKC
NFKD
Ǻ
Ǻ
U+01FA
Ǻ
U+00C5 U+0301
Ǻ
U+212B U+0301
Ǻ
U+0041 U+030A U+0301
U+01FA
U+0041 U+030A U+0301
Ａ̊́
Ａ̊́
Ａ̊́
U+FF21 U+030A U+0301
U+FF21 U+030A U+0301
U+FF21 U+030A U+0301
U+01FA
U+0041 U+030A U+0301
Unicode Normalization reduces these (and other potential sequences
of escapes representing the same character) to just three possible
variations. However, Unicode Normalization doesn't remove all
textual distinctions and sometimes the application of Unicode
Normalization can remove meaning that is distinctive or meaningful
in a given context. For example:
Not all compatibility characters have a compatibility
decomposition.
Some characters that look alike or have similar semantics are actually distinct in Unicode and don't have canonical or compatibility decompositions to link them together. For example,
U+3002 IDEOGRAPHIC FULL STOP
is used as a
period
at the end of sentences in languages such as Chinese or Japanese. However, it is not considered equivalent to the ASCII
period
character
U+002E FULL STOP
Some character variations are not handled by the Unicode
Normalization Forms. For example, UPPER, Title, and lowercase
variations are a separate and distinct textual variation that must
be separately handled when comparing text.
Compatibility normalization removes meaning. For example, the character sequence
8½
(including the character
U+00BD VULGAR FRACTION ONE HALF
), when normalized using one of the
compatibility
normalization forms (that is,
NFKD
or
NFKC
), becomes an ASCII character sequence:
81/2
2.3
Identical-Appearing Characters and the Limitations of Normalization
Many users are surprised to find that two identical-looking strings—including
those that have had a specific Unicode normalization form applied—might not
in fact use the same underlying
Unicode code points
. This includes strings that have
had the more-destructive
NFKC
and
NFKD
compatibility normalization
forms applied to them. Even when strings, tokens, or identifiers appear visually to be the same,
they can be encoded differently.
The Unicode canonical normalization forms are concerned with folding the multiple different code point sequences that can be used for a given abstract character or grapheme cluster to use the same code point sequence. However, logically distinct characters or grapheme clusters can still look the same or very similar. When a pair of
graphemes
look identical (or very similar), they are called
homographs
. When a pair of graphemes look similar or are
homographs
but actually represent logically different characters or character sequences, they are said to be
confusable
Example 13
U+03A1 GREEK CAPITAL LETTER RHO
U+0420 CYRILLIC CAPITAL LETTER ER
U+0050 LATIN CAPITAL LETTER P
There are many cross-script examples, such as the characters shown above. These letters from the Greek, Cyrillic, and Latin scripts look identical in most fonts (that is, they are
homographs
),
but they are encoded separately, as they are logically distinct parts of their respective Greek, Cyrillic, or Latin alphabet. Unicode Normalization will not fold these characters together.
Examples of identical or identical-seeming appearance can appear even within a single script. This can take the form of similarly shaped characters, such as "0" and "O" or "l" and "1". But other scripts or the use of different compatibility characters can present much less readily distinguished variations. In some cases, Unicode Normalization brings these together, but in many other cases it does not.
Example 14
: Examples of homographs within a single script
Some examples include:
بٔ
U+08A1 ARABIC LETTER BEH WITH HAMZA ABOVE
vs.
ARABIC LETTER BEH
followed by
ARABIC HAMZA ABOVE
ij
U+0133 LATIN SMALL LIGATURE IJ
vs.
LATIN SMALL LETTER I
LATIN SMALL LETTER J
ក្ត
ក្ដ
Khmer sequences involving
U+17D2 KHMER SIGN COENG
such as
U+17D2 U+178F
and
U+17D2 U+178A
(each shown here, for legibility, with the
base character
U+1780 KHMER LETTER KA
ក)
Characters that are identical or
confusable
in appearance can present spoofing and
other security risks. This can be true within a single script or for similar characters in
separate scripts. For further discussion and examples of homoglyphs and confusability,
one useful reference is [
UTS39
].
In addition to identical or similar-appearing characters, the opposite problem also exists:
Unicode Normalization, even the
NFKC
and
NFKD
Compatibility forms,
does not bring together characters that have the same intrinsic meaning or function
but which vary in appearance or usage. For example,
U+002E
(.)
and
U+3002
(。) both function as sentence ending punctuation,
but the distinction is not removed by normalization because the characters have a distinct identity.
2.4
Character Escapes and Includes
Most document formats or protocols provide an escaping mechanism to
permit the inclusion of characters that are otherwise difficult to
input, process, or encode. These escaping mechanisms provide an
additional equivalent means of representing characters inside a given
resource. They also allow for the encoding of Unicode characters not
represented in the character encoding scheme used by the document.
Note
For further discussion of character escapes, including guidelines for the definition of escaping mechanisms in specifications, see: Section 4.6 of [
CHARMOD
].
Note
The expansion of character escapes and includes is dependent on context, that is, on which
syntactic content
or programming language is considered to apply when the string matching operation is performed. Consider a search for the string
suçon
in an XML document containing
suçon
but not
suçon
. If the search is performed in a plain text editor, the context is
plain text
(no
syntactic content
or programming language applies), the
ç
character escape is not recognized, hence not expanded and the search fails. If the search is performed in an XML browser, the context is
XML
, the character escape (defined by XML) is expanded and the search succeeds.
An intermediate case would be an XML editor that
purposefully
provides a view of an XML document with entity references left
unexpanded. In that case, a search over that pseudo-XML view
will deliberately
not
expand entities: in that
particular context, entity references are not considered
includes and need not be expanded
For example,
U+20AC
EURO SIGN
can also be encoded in HTML as the hexadecimal
entity
€
or as the decimal entity
€
In a JavaScript or JSON file, it can appear as
\u20ac
or as
\u{20AC}
while in a CSS stylesheet it can appear as
\20ac
. All of
these representations encode the same literal character value:
Character escapes are normally interpreted before a document is
processed and strings within the format or protocol are matched.
Returning to an example we used above:
Example 15
style
type
"text/css"
span
.h
e9llo
text-decoration
:underline;
style
span
class
"héllo"
Hello World!
span
You would expect that text to display like the following:
Hello world!
In order for this to work, the user-agent (browser) had to match two strings representing the class name
héllo
, even though the CSS and HTML each used a different escaping mechanism. The above fragment demonstrates one way that text can vary and still be considered "the same" according to a specification: the class name
h\e9llo
matched the class name in the HTML mark-up
héllo
(and would also match the literal value
héllo
using the code point
U+00E9 LATIN SMALL LETTER E WITH ACUTE
).
Formal languages and document formats often offer facilities for including a piece of text from one resource inside another. An
include
is a mechanism for inserting content into the body of a
resource
. Include mechanisms import content into a resource at processing time. This affects the structure of the document and potentially matching against the vocabulary of the document. Examples of includes are entity references in XML, the XInclude [
XInclude
] specification, and @import rules in CSS.
An include is said to be
include normalized
if it does not begin with a combining mark (either in the form of a character escape or as a character literal in the included resource).
2.5
Invisible Unicode Characters
Unicode provides a number of special-purpose characters
that help document authors control the appearance or performance of
text. Because many of these characters are invisible or do not have keyboard equivalents, users are not always aware
of their presence or absence. As a result, these characters can interfere with string matching when they are part of the encoded
character sequence but the expected matching text does not also include them. Some examples of these characters include:
The Unicode control characters
U+200D Zero Width Joiner
(also known
as
ZWJ
) and
U+200C Zero Width Non-Joiner
(also known as
ZWNJ
).
While these characters can be used to control ligature formation—either preventing the formation of undesirable
ligatures or encouraging the formation of desirable ones—their primary use is to control
the joining and shape selection in complex scripts such as the Arabic or various of the Indic scripts.
Some Indic scripts use the ZWJ and ZWNJ characters to allow authors to control the shape that certain conjuncts take. See the
discussion in Chapter 12 of [
Unicode
].
The
Zero Width Non-Joiner
is used in Persian to
prevent certain "normal" Arabic script joining. In these cases, the presence or absence of the
character
does
affect the meaning. For example, the word تنها ("alone") and the word تن‌ها  ("bodies"
or "corpuses") are encoded as "
U+062A
U+0646 U+0647 U+0627
" and "
U+062A U+0646
U+200C
U+0647 U+0627
respectively, the only difference being the ZWNJ in the latter word.
The ZWJ character is also used in forming certain emoji sequences, which is discussed in more
detail
below
Variation selectors (
U+FE00
through
U+FE0F
) are
characters used to select an alternate appearance or glyph
(see Character Model: Fundamentals [
CHARMOD
]). For example, they are used to select between black-and-white and color emoji.
These are also used in predefined ideographic variation sequences (
IVS
). Many
examples are given in the "Standardized Variants" portion of the Unicode Character Database (UCD).
A few scripts also provide a way to encode visual variation selection: a prominent example of this
are the Mongolian script's free variation selectors (
U+180B
through
U+180D
).
The character
U+034F Combining Grapheme Joiner
whose name is misleading (as it does not join graphemes or affect line
breaking), is used to separate characters that might otherwise be
considered a grapheme for the purposes of sorting or to provide a
means of maintaing certain textual distinctions when applying Unicode
normalization to text.
Whitespace variations can also affect the interpretation and
matching of text. For example, the various non-breaking space
characters, such as NBSP, NNBSP, etc.
U+200B Zero Width Space
is a character used to
indicate word boundaries in text where spaces do not otherwise appear.
For example, it might be used in a Thai language document to assist
with word-breaking.
The
U+00AD Soft Hyphen
can be used in text
to indicate a potential or preferred hyphenation position. It only
becomes visible when the text is reflowed to wrap at that position.
The
U+2060 WORD JOINER
, sometimes called
WJ
is a zero-width
non-breaking space character. Its purpose is to replace the functionality of the character
U+FEFF ZERO WIDTH NO-BREAK SPACE
because that character also serves as the "Byte Order Mark" character (used as a Unicode
signature in plain text files). The Word Joiner is used to separate words in languages that do not use explicit spacing. An example
would be the Thai language.
Finally, some scripts, such as Arabic and Hebrew, are written prodominently from right-to-left. Text written in these scripts can also
include character sequences, such as numbers or quotes in another script, that are left-to-right. This intermixing of text direction
is called
bidirectional
text or
bidi
for short. The Unicode Bidirectional Algorithm
UAX9
] describes how such mixed-direction text is processed for display. For most text, the directional handling can be derived
from the text itself. However, there are many cases in which the algorithm needs additional information in order to present text
correctly. For more examples, see [
html-bidi
].
One of the ways that Unicode defines to address the ambiguity of text direction are a set of invisible control characters to
mark the start and end of directional runs. While bidirectional controls can have an affect on the appearance of the text
(since they help the Unicode Bidirectional Algorithm with the presentation of text), they might have no effect on the
text if the text would naturally have fallen into bidirectional runs without the controls. Because these controls are, like the characters
mentioned above, invisible, they can have an unintentional effect on matching.
In almost all of these cases, users may not be aware of or cannot
be sure if a given document or text string has included or omitted one
of these characters. Because text matching depends on matching the
underlying codepoints, variation in the encoding of the text due to
these markers can cause matches that ought to succeed to mysteriously
fail (from the point of view of the user).
2.6
Emoji Sequences
A newer feature of Unicode are the emoji characters. In [
UTR51
], Unicode describes these as:
Emoji are pictographs (pictorial symbols) that are typically presented in a colorful cartoon
form and used inline in text. They represent things such as faces, weather, vehicles and buildings,
food and drink, animals and plants, or icons that represent emotions, feelings, or activities.
Emoji can be used with a variety of emoji modifiers, including
U+200D ZERO WIDTH JOINER
or
ZWJ
, to form more complex emoji.
For example, the emoji (
U+1F46A FAMILY
) can also be formed by using ZWJ between emoji characters in the sequence
U+1F468
U+200D
U+1F469
U+200D
U+1F466
Altering or adding other emoji characters can alter the composition of the family. For example the sequence
👨‍👩‍👧‍👧
U+1F468 U+200D U+1F469 U+200D U+1F467 U+200D U+1F467
results in a composed
emoji character for a "family: man, woman, girl, girl" on systems that support this kind of
composition. Many common emoji can
only
be formed using ZWJ sequences. For more
information, see [
UTR51
].
Emoji characters can be followed by emoji modifier characters. These modifiers allow for the selection of skin tones for emoji that represent people. These characters are normally invisible modifiers that follow the base emoji that they modify. For example: 👨 👨🏻 👨🏼 👨🏽 👨🏾 👨🏿
An emoji character can also be followed by a
variation
selector
to indicate text (black and white, indicated by
U+FF0E Variation Selector 15
) or color
(indicated by
U+FF0F Variation Selector 16
) presentation
of the base emoji.
Still another wrinkle in the use of emoji are flags. National flags can be composed using country codes derived from the [
BCP47
] registry, such as the sequence
U+1F1FF REGIONAL INDICATOR SYMBOL LETTER Z
U+1F1F2 REGIONAL INDICATOR SYMBOL LETTER M
, which is the country code (
ZM
) for the country Zambia: 🇿🇲. Other regional or special purpose flags can be composed using a flag emoji with various symbols or with regional indicator codes terminating in a cancel tag. For example, the flag of Scotland (🏴󠁧󠁢󠁳󠁣󠁴󠁿) can be composed like this:
U+1F3F4 WAVING BLACK FLAG
U+E0067 TAG LATIN SMALL LETTER G
U+E0062 TAG LATIN SMALL LETTER B
U+E0073 TAG LATIN SMALL LETTER S
U+E0063 TAG LATIN SMALL LETTER C
U+E0074 TAG LATIN SMALL LETTER T
U+E007F CANCEL TAG
Each of these mechanisms can be used together, so quite complex sequences of characters
can be used to form a single emoji grapheme or image. Even very similar emoji sequences might
not use the same exact encoded sequence. Many of the modifiers and combinations mentioned above
are generated by the end-user's keyboard (where they are presented as a single emoji "character"),
so users may not be aware of the underlying encoding complexity. Emoji sequences are evolving rapidly,
so there could be additional developments to either help or hinder matching of emoji in the near
future. Currently Unicode normalization does not reorder these
sequences or insert or remove any of the modifiers. Users and implementers are therefore cautioned that
users who employ emoji characters in namespaces and other matching contexts might encounter
unexpected character mismatches.
2.7
Legacy Character Encodings
Resources
can use different character encoding
schemes, including
legacy character encodings
, to serialize
document formats on the Web. Each character encoding scheme uses
different byte values and sequences to represent a given subset of the
Universal Character Set.
Note
Choosing a Unicode character encoding, such as UTF-8, for all documents, formats, and protocols is a strongly encouraged
recommendation
, since there is no additional utility to be gained from using a legacy character encoding and the considerations in the rest of this section would be completely avoided.
For example,
U+20AC EURO SIGN
) is encoded as the byte sequence
0xE2.82.AC
in the
UTF-8
character encoding. This same
character is encoded as the byte sequence
0x80
in the
legacy character encoding
windows-1252
(Other legacy character encodings may not provide any byte sequence to
encode the character.)
Specifications mainly address these resulting variations by
considering each document to be a sequence of Unicode characters after
converting from the document's character encoding (be it a legacy
character encoding or a Unicode encoding such as UTF-8) and then
unescaping any character escapes before proceeding to process the
document.
Note
Even within a single legacy character encoding there can
be variations in implementation. One famous example is the legacy
Japanese encoding
Shift_JIS
. Different
transcoder implementations faced choices about how to map specific
byte sequences to Unicode. So the byte sequence
0x80.60
0x2141
in the JIS X 0208 character set) was mapped by
some implementations to
U+301C
WAVE DASH
while others chose
U+FF5E
FULL WIDTH TILDE
. This means that two reasonable,
self-consistent, transcoders could produce different Unicode character
sequences from the same input. The
Encoding
Encoding
specification exists, in part, to ensure that Web implementations use
interoperable and identical mappings. However, there is no guarantee
that transcoders consistent with the Encoding specification will be
applied to documents found on the Web or used to process data
appearing in a particular document format or protocol.
One additional consideration in converting to Unicode is the existence of
legacy character encodings of bidirectional scripts (such as Hebrew and Arabic) that use a
visual storage order. That is, unlike Unicode and other modern encodings,
the characters are stored in memory in the order that they are printed
on the screen from left-to-right (as with a line printer). When converting
these encodings to Unicode or when comparing text in these encodings, care
must be taken to place both the source and target text into logical order.
For more information, see Section 3.3.1 of [
CHARMOD
2.8
Other Types of Equivalence
Note
There are additional kinds of equivalence or processing that are appropriate when performing natural language searching or "find" features. These are described in another part of the Character Model series of documents ([
STRING-SEARCH
]). Specifications for a
vocabulary
or which define a matching algorithm for use in a formal syntax
SHOULD
avoid trying to apply additional custom folding, mapping, or processing such as described in that document, since these interfere with producing consistent, predictable results.
3.
String Matching of Syntactic Content in Document Formats and Protocols
In the Web environment, where strings can be encoded in different
encodings, using different character sequences, and with variations such
as case, it's important to
establish a consistent process for evaluating string identity.
This chapter defines the implementation and requirements for string
matching in
syntactic content
3.1
The Matching Algorithm
This section defines the algorithm for matching strings in a formal language or syntax. Specifications need to specify certain options called out below. Recommendations are provided for best practices in the sub-sections below.
Convert the strings to be compared to a sequence of Unicode code points. This might entail
transcoding
from a legacy character encoding.
Expand all
character escapes and includes
If Unicode normalization is
specified
, apply the appropriate normalization form to the text.
If case sensitivity is
specified
, proceed to the next step. Otherwise apply the appropriate case folding operation:
Unicode full
case folding
: map all code points to their
Unicode full
case fold equivalents. Note that this can change the length of the string.
ASCII case folding
: map all code points in the range U+0041 to U+005A (A to Z) to the corresponding code points in the range U+0061 to U+007A (a to z).
Perform any
additional matching tailoring
specific to the specification.
Compare the resulting sequences of code points for identity.
3.1.1
Converting to a Sequence of Unicode Code Points
[C]
Content authors
SHOULD
enter and store resources in a Unicode character encoding (generally UTF-8 on the Web).
[C]
Content authors
SHOULD
choose a
normalizing transcoder
when converting legacy encoded text or resources to Unicode unless the mapping of specific characters interferes with the meaning.
[S]
Specifications
MUST
allow a Unicode character encoding.
[S]
Specifications
MUST
specify a default character encoding and
SHOULD
specify UTF-8 as the default encoding.
[S]
Specifications
SHOULD
disallow encodings other than UTF-8.
The first step in comparing text is to ensure that both use the same digital representation. This means that implementations need to convert any text in a
legacy character encoding
to a sequence of Unicode code points. Normally this is done by applying a
transcoder
to convert the data to a consistent Unicode encoding form (such as UTF-8 or UTF-16). This allows bitwise comparison of the strings in order to determine string equality.
normalizing transcoder
is a
transcoder
that performs a conversion from a
legacy character encoding
to Unicode
and
ensures that the result is in Unicode Normalization Form C (
NFC
). For most legacy character encodings, it is possible to construct a normalizing transcoder (by using any transcoder followed by a normalizer); it is not possible to do so if the
legacy character encoding
's
repertoire
contains characters not represented in Unicode. While normalizing transcoders only produce character sequences that are in
NFC
, the converted character sequence might not be
include normalized
(for example, if it begins with a combining mark).
Because document formats on the Web often interact with or are processed using additional, external resources (for example, a CSS style sheet being applied to an HTML document), the consistent representation of text becomes important when matching values between documents that use different character encodings. Use of a normalizing transcoder helps ensure interoperability by making legacy encoded documents match the normally expected Unicode character sequence for most languages.
Most transcoders used on the Web produce
NFC
as their output, but several do not. This is usually to allow the transcoder to be round-trip compatible with the source legacy character encoding, to preserve other character distinctions, or to be consistent with other transcoders in use in user-agents. This means that the Encoding specification [
Encoding
] and various other important transcoding implementations include a number of non-normalizing transcoders. Indeed, most compatibility characters in Unicode exist solely for round-trip conversion from legacy encodings and a number of these have singleton canonical mappings in
NFC
. You saw an example of this
earlier in the document
with
U+212B ANGSTROM SIGN
Bear in mind that most transcoders produce
NFC
output and that even those transcoders that do not produce
NFC
for all characters produce
NFC
for the preponderence of characters. In particular, there are no commonly-used transcoders that produce decomposed forms where precomposed forms exist or which produce a different combining character sequence from the normalized sequence (and this is true for
all
of the transcoders in [
Encoding
]).
3.1.2
Expanding Character Escapes and Includes
Most document formats and protocols provide a means for encoding characters as an escape sequence or including external data, including text, into a
resource
. This is discussed in detail in Section 4.6 of [
CHARMOD
] as well as
above
When performing matching, it is important to know when to interpret character escapes so that
a match succeeds (or fails) appropriately. Normally, escapes, references, and includes are processed
or expanded before performing matching (or match-sensitive processing), since these syntaxes exist to allow difficult-to-encode
sequences to be put into a document conveniently, yet allowing the characters to behave as-if they were
directly encoded as a codepoint sequence in the document in question.
One area where this can be complicated is deciding how syntactic content and natural language content interact.
For example, consider the following snippet of HTML:
Example 16

"̀"
>Combining mark used
as
the value
of
'id'
attribute

Although technically the combining mark
U+0300 COMBINING GRAVE ACCENT​
combines with the preceding quote mark, HTML does not consider the character (whether or not it is encoded as an entity) to form part of the HTML syntax.
When performing a matching operation on a resource, the general rule is to expand escapes on the same "level" as the user is interacting with. For example, when considering the above example, a tool to view the source of the HTML would show the escape sequence
̀
as a string of characters starting with an ampersand. A JavaScript program, by contrast, operates on the browser's interpretation of the document and would match the character
U+0300
as the value of the attribute
id
When processing the syntax of a document format, escapes are usually converted to the character sequence they represent before the processing of the syntax, except where explicitly forbidden by the format's processing rules. This allows resources to include characters of all types into the resource's syntactic structures.
In some cases, pre-processing escapes creates problems. For example, expanding the sequence
<
before parsing an HTML document would produce document errors.
3.1.3
Choice of Normalization Form
A specific Unicode normalization form is not always appropriate or available to content authors and the text encoding choices of users might not be obvious to downstream consumers of the data. As shown in this document, there are many different ways that content authors or applications could choose to represent the same semantic values when inputting or exchanging text. Normalization can remove distinctions that the users applied intentionally. Therefore:
[S]
Specifications
SHOULD NOT
specify the Unicode normalization in string matching for vocabularies.
[I]
Implementations
MUST NOT
alter the normalization form of syntactic or natural language content being exchanged, read, parsed, or processed except when required to do so as a side-effect of text transformation such as transcoding the content to a Unicode character encoding, case mapping or folding, or other user-initiated change, as consumers or the content itself might depend on the de-normalized representation.
[I]
Authoring tools
SHOULD
provide a means of normalizing resources and warn the user when a given resource is not in Unicode Normalization Form C.
[S]
Specifications of text-based formats and protocols that as part of their syntax definition require the text be in a normalized form
MUST
define string matching in terms of normalized string comparison and
MUST
define the normalized form to be
NFC
Note
A specification that defines comparison of text in a normalized format needs to address the requirements in
3.1.3.1
Considerations When Requiring Normalization
Specifications are generally discouraged from requiring formats or protocols to store or exchange data in a normalized form unless there are specific, clear reasons why the additional requirement is necessary. As many document formats on the Web do not require normalization, content authors might occasionally rely on denormalized character sequences. A normalization step could negatively affect such content.
The canonical normalization forms (form
NFC
or form NFD) are intended to preserve the meaning and presentation of the text to which they are applied. This is not always the case, which is one reason why normalization is not recommended.
NFC
has the advantage that almost all legacy data (if transcoded trivially, one-to-one, to a Unicode encoding), as well as data created by current software or entered by users on most (but not all) keyboards, is already in this form.
NFC
also has a slight compactness advantage and is a better match to user expectations in most languages with respect to the relationship between characters and graphemes.
[S]
Specifications
SHOULD NOT
specify compatibility normalization forms (NFKC, NFKD).
[I]
Implementations
MUST NOT
apply compatibility normalization forms (NFKC, NFKD) unless specifically requested by the end user.
The compatibility normalization forms (form NFKC and form NFKD) change the structure and lose the meaning of the text in important ways. Users sometimes use characters with a compatibility mapping in Unicode on purpose or they use characters in a legacy character encoding that have a compatibility mapping when converted to Unicode. This has to be considered intentional on the part of the content author. Although NFKC/NFKD can sometimes be useful in "find" operations or string searching natural language content, erasing compatibility differences is harmful.
Note
Requiring
NFC
requires additional care on the part of the specification developer, as content on the Web generally is not in a known normalization state. Boundary and error conditions for denormalized content need to be carefully considered and well-specified in these cases.
[S]
Specifications
MUST
document or provide a health-warning if canonically equivalent but disjoint Unicode character sequences represent a security issue.
[C]
Content authors
SHOULD
use Unicode Normalization Form C (
NFC
) wherever possible for content.
Note that
NFC
is not always appropriate to the content or even available to content authors in some languages.
[C]
Content authors
SHOULD
always encode text using consistent Unicode character sequences to facilitate matching, even if a Unicode normalization form is included in the matching performed by the format or implementation.
In order for their content to be processed consistently, content authors should try to use a consistent sequence of code points to represent the same text. While content can be in any normalization form or might use a de-normalized (but valid) Unicode character sequence, inconsistency of representation will cause implementations to treat the different sequences as different. The best way to ensure consistent selection, access, extraction, processing, or display is to always use
NFC
[C]
Content authors
SHOULD NOT
include combining marks without a preceding base character in a resource.
There can be exceptions to this. For example, when making a list of characters (such as a list of [
Unicode
] characters), an author might want to use combining marks without a corresponding base character. However, use of a combining mark without a base character can cause unintentional display or, with naive implementations that combine the combining mark with adjacent syntactic content or other natural language content, processing problems. For example, if you were to use a combining mark, such as the character
U+0301 COMBINING ACUTE ACCENT​
, as the start of a
class
attribute value in HTML, the class name might not display properly in your editor and be difficult to edit.
Some recommended base characters include
U+25CC DOTTED CIRCLE
(when the base character needs to be visible) or
U+00A0 NO-BREAK SPACE
(when the base character should be invisible).
Since content authors do not always following these guidelines:
[S]
Specifications of vocabularies
MUST
define the boundaries between syntactic content and character data as well as entity boundaries (if the language has any include mechanism).
These need to include any boundary that may create conflicts when processing or matching content when instances of the language are processed, while allowing for character escapes designed to express arbitrary characters.
3.1.3.1
Considerations When Requiring Normalization
When a specification requires Unicode normalization for storage, transmission, or string matching, some additional considerations need to be addressed by the specification authors as well as by implementers of that specification:
[S]
Where operations can produce denormalized output from normalized text input, specifications
MUST
define whether the resulting output is required to be normalized or not. Specifications
MAY
state that performing normalization is optional for some operations; in this case the default
SHOULD
be that normalization is performed, and an explicit option
SHOULD
be used to switch normalization off.
[S]
Specifications that require normalization
MUST NOT
make the implementation of normalization optional.
Interoperability of matching cannot be achieved if some implementations normalize while others do not.
An implementation that is required to perform normalization needs to consider these requirements:
[I]
Normalization-sensitive operations
MUST NOT
be performed unless the implementation has first either confirmed through inspection that the text is in normalized form or it has re-normalized the text itself. Private agreements
MAY
be created within private systems which are not subject to these rules, but any externally observable results
MUST
be the same as if the rules had been obeyed.
[I]
A normalizing text-processing component which modifies text and performs normalization-sensitive operations
MUST
behave as if normalization took place after each modification, so that any subsequent normalization-sensitive operations always behave as if they were dealing with normalized text.
[I]
Authoring tool implementations
SHOULD
warn users or prevent the input or creation of syntactic content starting with a combining mark that could interfere with processing, display, or interchange.
3.1.4
Choice of Case Folding
One important consideration in string identity matching is whether the comparison is case sensitive or case insensitive.
[C]
Content authors
SHOULD
always spell identifiers using consistent upper, lower, and mixed case formatting to facilitate matching, even if case-insensitive matching is supported by the format or implementation.
3.1.4.1
Case-sensitive matching
[S]
Case-sensitive
matching is
RECOMMENDED
for matching syntactic content, including user-defined values.
Vocabularies usually put a premium on predictability for content authors and users. Case-sensitive matching is the easiest to implement and introduces the least potential for confusion, since it generally consists of a comparison of the underlying Unicode code point sequence. Because it is not affected by considerations such as language-specific case mappings, it produces the least surprise for document authors that have included words, such as the
Turkish examples
above, in their syntactic content.
Case insensitivity is usually reserved for processing
natural language content
, such as running a feature for searching text. However, cases exist in which case-insensitivity is desirable. When case-insensitive matching is necessary, there are several implementation choices that a formal language needs to consider.
3.1.4.2
Unicode case-insensitive matching
[S]
Specifications that define case-insensitive matching in vocabularies that include more than the Basic Latin (ASCII) range of Unicode
MUST
specify
Unicode full
casefold matching.
[S]
Specifications
SHOULD
allow the full range of Unicode for user-defined values.
Vocabularies generally should allow for a wide range of Unicode characters, particularly for
user-supplied values
, so as to enable use by the broadest range of languages and cultures without disadvantage. As a result, text operations such as case folding need to address the full range of Unicode and not just selected portions. When case-insensitive matching is desired, this means using
Unicode case folding
The
Unicode simple
casefolding form is not appropriate for string identity matching on the Web.
3.1.4.3
ASCII case-insensitive matching
[S]
Specifications that define case-insensitive matching in vocabularies limited to the Basic Latin (ASCII) subset of Unicode
MAY
specify
ASCII case-insensitive
matching.
A formal language whose
vocabulary
is limited to ASCII and which does not allow user-defined names or identifiers can specify
ASCII case-insensitive
matching. An example of this is HTML, which defines the use of ASCII case-insensitive comparison for element and attribute names defined by the HTML specification.
A vocabulary is considered to be "ASCII-only" if and only if all tokens and identifiers are defined by the specification directly and these identifiers or tokens use only the Basic Latin subset of Unicode. If user-defined identifiers are permitted, the full range of Unicode characters (limited, as appropriate, for security or interchange concerns, see [
UTR36
]) should be allowed and Unicode case insensitivity used for identity matching.
Note
An ASCII-only vocabulary can exist inside a document format or protocol that allows a larger range of Unicode in identifiers or values. For example [
CSS-SYNTAX-3
] defines the format of CSS style sheets in a way that allows the full range of Unicode to be used for identifiers and values. However, CSS specifications always define CSS keywords using a subset of the ASCII range. The vocabulary of CSS is thus ASCII-only, even though many style sheets contain identifiers or data values that are not ASCII.
3.1.4.4
Language-specific tailoring
Locale- or language-specific tailoring is most appropriate when it is part of natural language processing operations (which is beyond the scope of this document). Because language-specific tailoring of case mapping or case folding produces different results from the generic case folding rules, these should be avoided in formal languages, where predictability is at a premium.
[S]
Specifications that define case-insensitive matching in vocabularies
SHOULD NOT
specify language-sensitive case-insensitive matching.
[S]
If language-sensitive case-sensitive matching is specified, Unicode case-fold mappings
SHOULD
be tailored according to language and the source of the language used for each tailoring
MUST
be specified.
Two strings being matched can be in different languages and might appear in yet a third language context. Which language to use for case folding therefore depends on the application and user expectations.
Language specific tailoring is not recommended for formal languages because the language information can be hard to obtain, verify, or manage and because the resulting operations can produce results that frustrate users or which fail for some users and succeed for others depending on the language configuration that they are using or the configuration of the system where the match is performed.
[S]
Operations that are language-specific
SHOULD
include language-specific case folding where appropriate.
For example, the CSS operation
text-transform
is language-sensitive when used to case map strings.
Note
Although Unicode case folding is the preferred case-insensitive matching for document formats and protocols, content authors and users of languages that have mappings different from the default can still be surprised by the results, since their expectations are generally consistent with the languages that they speak.
Note
Language-sensitive string comparison is often referred to as being
locale-sensitive
, since most programming languages and operating environments access language-specific tailoring using their respective locale-based APIs. For example, see the
java.text.Collator
class in the Java programming language or
Intl.Collator
in JavaScript.
3.1.5
Additional Match Tailoring
[S]
Specificiations
MUST
clearly define any additional tailoring done as part of the matching process.
Some specifications might wish to include additional tailoring to assist with matching in a given vocabulary. Examples of this might include removing additional textual differences described in
Section 2
, mapping together or removing characters that are part of the syntax, or performing a whitespace trim.
Any additional tailoring needs to avoid interfering with the way that different languages are represented in Unicode. For example, a process that attempts to remove accents from letters by decomposing the text and then removing all of the combining characters will break languages that rely on combining marks. An example of this would be as the Devanagari text in
Example 2
. (Such a process would also fail to remove all of the potential accents and probably do harm to the meaning and representation of the text.)
4.
Other Matching and Processing Considerations
While matching strings and tokens in a formal language is the primary concern of this document, sometimes a specification needs to consider additional types of matching beyond pure string equality.
4.1
Regular Expressions
[S]
Specifications that define a regular expression syntax
MUST
provide at least Basic Unicode Level 1 support per [
UTS18
] and
SHOULD
provide Extended or Tailored (Levels 2 and 3) support.
Regular expression syntaxes are sometimes useful in defining a format or protocol, since they allow users to specify values that are only partially known or which can vary in predictable ways. As seen in the various sections of this document, there is variation in the different ways that characters can be encoded in Unicode and this potentially interferes with how strings are specified or matched in expressions. For example, counting characters might need to depend on grapheme boundaries rather than the number of Unicode code points used; caseless matching might need to consider variations in case folding; or the Unicode normalization of the expression or text being processed might need to be considered.
Unicode Regular Expressions Level 1 support includes the ability to specify Unicode code points in regular expressions, including via the use of escapes, and to access Unicode character properties as well as certain kinds of boundaries common to most regular expression syntaxes.
Level 2 extends this with a number of important capabilities, notably the ability to select text on certain kinds of
grapheme cluster
boundary and support for case conversion (two topics mentioned extensively above). Level 3 provides for locale [
LTLI
] based tailoring of regular expressions, which are less useful in formal languages but can be useful in processing
natural language content
5.
Changes Since the Last Published Version
This document has been extensively revised and rewritten since the
Working Draft
of 2014-07-15. Please see the
github commit log
for more details.
6.
Acknowledgements
The
W3C
Internationalization Working Group and Interest Group, as well as others, provided many comments and suggestions. The Working Group would like to thank:
Mati Allouche,
Ebrahim Byagowi,
John Cowan,
Martin Dürst,
Behdad Esfahbod,
Asmus Freitag,
John Klensin,
Peter Saint-Andre,
Amir Sarabadani,
and all of the CharMod contributors over the twenty (!!) years of this document's development.
The previous version of this document was edited by:
François Yergeau, Invited Expert (and before at Alis Technologies)
Martin J. Dürst, (until Dec 2004 while at
W3C
Richard Ishida,
W3C
(and before at Xerox)
Misha Wolf, (until Dec 2002 while at Reuters Ltd.)
Tex Texin, (until Dec 2004 while an Invited Expert, and before at
Progress Software)
A.
References
A.1
Normative references
[CHARMOD]
Character Model for the World Wide Web 1.0: Fundamentals
. Martin Dürst; François Yergeau; Richard Ishida; Misha Wolf; Tex Texin et al. W3C. 15 February 2005. W3C Recommendation. URL:
[Encoding]
Encoding
. Anne van Kesteren; Joshua Bell; Addison Phillips.URL:
[INTERNATIONAL-SPECS]
Internationalization Best Practices for Spec Developers
. Richard Ishida. W3C. 7 October 2016. W3C Working Draft. URL:
[ISO10646]
Information Technology - Universal Multiple- Octet Coded CharacterSet (UCS) - Part 1: Architecture and Basic Multilingual Plane
ISO/IEC10646-1:1993.
[RFC2119]
Key words for use in RFCs to Indicate Requirement Levels
. S. Bradner. IETF. March 1997. Best Current Practice. URL:
[UAX15]
Unicode Normalization Forms
. Mark Davis; Ken Whistler. Unicode Consortium. 26 May 2017. Unicode Standard Annex #15. URL:
[UAX29]
Unicode Standard Annex #29: Unicode Text Segmentation
. Mark Davis.URL:
[Unicode]
The Unicode Standard
. Unicode Consortium. URL:
[UTS18]
Unicode Technical Standard #18: Unicode Regular Expressions
. Mark Davis; Andy Heninger.URL:
A.2
Informative references
[ASCII]
ISO/IEC 646:1991, Information technology -- ISO 7-bit coded character set for information interchange
. URL:
[BCP47]
Tags for Identifying Languages
. A. Phillips; M. Davis. IETF. September 2009. IETF Best Current Practice. URL:
[CHARREQ]
Requirements for String Identity Matching and String Indexing
. Martin Dürst. W3C. 15 September 2009. W3C Note. URL:
[CSS-SYNTAX-3]
CSS Syntax Module Level 3
. Tab Atkins Jr.; Simon Sapin. W3C. 20 February 2014. W3C Candidate Recommendation. URL:
[HTML]
HTML Standard
. Anne van Kesteren; Domenic Denicola; Ian Hickson; Philip Jägenstedt; Simon Pieters. WHATWG. Living Standard. URL:
[html-bidi]
Additional Requirements for Bidi in HTML & CSS
. Aharon Lanin; Richard Ishida. W3C. 21 July 2015. W3C Note. URL:
[HTML5]
HTML5
. Ian Hickson; Robin Berjon; Steve Faulkner; Travis Leithead; Erika Doyle Navara; Theresa O'Connor; Silvia Pfeiffer. W3C. 27 March 2018. W3C Recommendation. URL:
[LTLI]
Language Tags and Locale Identifiers for the World Wide Web
. Felix Sasaki; Addison Phillips. W3C. 23 April 2015. W3C Working Draft. URL:
[RFC3986]
Uniform Resource Identifier (URI): Generic Syntax
. T. Berners-Lee; R. Fielding; L. Masinter. IETF. January 2005. Internet Standard. URL:
[STRING-SEARCH]
Character Model for the World Wide Web: String Searching
. Addison Phillips.URL:
[UAX11]
Unicode Standard Annex #11: East Asian Width
. Ken Lunde 小林劍.URL:
[UAX35]
Unicode Locale Data Markup Language (LDML)
. Mark Davis; CLDR committee members. Unicode Consortium. 15 March 2017. Unicode Standard Annex #35. URL:
[UAX9]
Unicode Standard Annex #9: Unicode Bidirectional Algorithm
. Mark Davis; Aharon Lahnin; Andrew Glass.URL:
[UTR36]
Unicode Technical Report #36: Unicode Security Considerations
. Mark Davis; Michel Suignard.URL:
[UTR50]
Unicode Technical Report #50: Unicode Vertical Text Layout
. Koji Ishii 石井宏治.URL:
[UTR51]
Unicode Technical Report #51: Unicode Emoji
. Mark Davis; Peter Edberg.URL:
[UTS39]
Unicode Technical Standard #39: Unicode Security Mechanisms
. Mark Davis; Michel Suignard.URL:
[XInclude]
XML Inclusions (XInclude) Version 1.0 (Second Edition)
. Jonathan Marsh; David Orchard; Daniel Veillard. W3C. 15 November 2006. W3C Recommendation. URL:
[XML10]
Extensible Markup Language (XML) 1.0 (Fifth Edition)
. Tim Bray; Jean Paoli; Michael Sperberg-McQueen; Eve Maler; François Yergeau et al. W3C. 26 November 2008. W3C Recommendation. URL: