Code points, U+, planes and surrogate pairs

The first job of Unicode is turning a character into a number — a code point. How that number is stored (bytes) is a separate layer. Get this straight and "why is the length 2" and "why is one character seven code points" both make sense.

A code point is a number#

Every character has a number between 0 and 0x10FFFF, written U+ plus at least four hex digits:

The digit count is not fixed: four is conventional inside the BMP, five or six beyond it.

Planes#

Divide 0–0x10FFFF into blocks of 65,536 and you get planes:

Private-use planes are for anyone's own definitions: a font may put any drawing there, and Unicode says nothing about what it is called.

Surrogates are not characters#

U+D800–U+DFFF never correspond to characters. They exist for UTF-16: a character outside the BMP is spelled with two 16-bit code units, and those units land in this range.

So "is there a character in the surrogate range" is a meaningless question — they are only an encoding halfway house.

UTF-8 and UTF-16#

• UTF-8 uses 1–4 bytes per code point and is ASCII-compatible; the default for files and the web;
• UTF-16 uses 2 bytes (one code unit) inside the BMP and two units (a surrogate pair) outside it.

JavaScript strings are UTF-16 code unit sequences, hence:

'😀'.length                  // 2  ← two code units
[...'😀'].length             // 1  ← one code point
'😀'.codePointAt(0)          // 128512 (0x1F600)
String.fromCodePoint(0x1F600) // '😀'

Likewise slice(0, 1) cuts an emoji into half a surrogate pair and yields garbage. To iterate "characters", use for...of or spread into an array, not indices.

One "character" can be several code points#

Code points are encoding units, not what the eye sees as one character:

• é can be U+00E9, or U+0065 + U+0301 (letter plus accent);
• the family emoji 👨‍👩‍👧 is man + zero-width joiner + woman + zero-width joiner + girl — five code points;
• flag emoji are two "regional indicator" code points.

The process that groups these into one visual character is the grapheme cluster, defined by Unicode segmentation rules — not by byte counts.

Variation selectors#

Some characters have several shapes (text-style versus emoji-style), chosen with a variation selector:

• U+FE0E: text presentation;
• U+FE0F: emoji presentation.

They render nothing themselves; they are modifiers attached to the preceding character. A font must declare support in cmap's variation-sequence subtable, otherwise the selector is ignored.

How this site shows it#

The character drawer shows the standard U+4E2D form plus a decimal value, so you can look the character up elsewhere. Blocks use the names and ranges from Unicode's Blocks.txt.

Further reading#

• The Unicode Standard, chapter 2 (General Structure)
• UAX #29 (grapheme cluster boundaries)
• MDN: String#codePointAt, String.fromCodePoint