UTF-8

UTF-8 is a character encoding capable of encoding all 1,112,064^[1] valid code points in Unicode using one to four 8-bit bytes.^[2] The encoding is defined by the Unicode standard, and was originally designed by Ken Thompson and Rob Pike.^[3][4] The name is derived from Unicode (or Universal Coded Character Set) Transformation Format – 8-bit.^[5]
It was designed for backward compatibility with ASCII. Code points with lower numerical values, which tend to occur more frequently, are encoded using fewer bytes. The first 128 characters of Unicode, which correspond one-to-one with ASCII, are encoded using a single octet with the same binary value as ASCII, so that valid ASCII text is valid UTF-8-encoded Unicode as well. Since ASCII bytes do not occur when encoding non-ASCII code points into UTF-8, UTF-8 is safe to use within most programming and document languages that interpret certain ASCII characters in a special way, such as "/" in filenames, "\" in escape sequences, and "%" in printf.

Shows the usage of the main encodings on the web from 2001 to 2012 as recorded by Google,^[6] with UTF-8 overtaking all others in 2008 and nearing 50% of the web in 2012.
Note that the ASCII only figure includes web pages with any declared header if they are restricted to ASCII characters.

UTF-8 has been the dominant character encoding for the World Wide Web since 2009, and as of October 2017 accounts for 90.1% of all Web pages. (The next-most popular multibyte encodings, Shift JIS and GB 2312, have 0.8% and 0.6% respectively).^[7][8][6] The Internet Mail Consortium (IMC) recommended that all e-mail programs be able to display and create mail using UTF-8,^[9] and the W3C recommends UTF-8 as the default encoding in XML and HTML.^[10]

Contents

• 1 Description
  • 1.1 Examples
  • 1.2 Codepage layout
  • 1.3 Overlong encodings
  • 1.4 Invalid byte sequences
  • 1.5 Invalid code points
• 2 Official name and variants
• 3 Derivatives
  • 3.1 CESU-8
  • 3.2 Modified UTF-8
  • 3.3 WTF-8
• 4 Byte order mark
• 5 History
• 6 Comparison with single-byte encodings
• 7 Comparison with other multi-byte encodings
• 8 Comparison with UTF-16
• 9 See also
• 10 Notes
• 11 References
• 12 External links

Description

Since the restriction of the Unicode code-space to 21-bit values in 2003, UTF-8 is defined to encode code points in one to four bytes, depending on the number of significant bits in the numerical value of the code point. The following table shows the structure of the encoding. The x characters are replaced by the bits of the code point. If the number of significant bits is no more than seven, the first line applies; if no more than 11 bits, the second line applies, and so on.

Number of bytes	Bits for code point	First code point	Last code point	Byte 1	Byte 2	Byte 3	Byte 4
1	7	U+0000	U+007F	0xxxxxxx
2	11	U+0080	U+07FF	110xxxxx	10xxxxxx
3	16	U+0800	U+FFFF	1110xxxx	10xxxxxx	10xxxxxx
4	21	U+10000	U+10FFFF	11110xxx	10xxxxxx	10xxxxxx	10xxxxxx

The first 128 characters (US-ASCII) need one byte. The next 1,920 characters need two bytes to encode, which covers the remainder of almost all Latin-script alphabets, and also Greek, Cyrillic, Coptic, Armenian, Hebrew, Arabic, Syriac, Thaana and N'Ko alphabets, as well as Combining Diacritical Marks. Three bytes are needed for characters in the rest of the Basic Multilingual Plane, which contains virtually all characters in common use^[11] including most Chinese, Japanese and Korean characters. Four bytes are needed for characters in the other planes of Unicode, which include less common CJK characters, various historic scripts, mathematical symbols, and emoji (pictographic symbols).
Some of the important features of this encoding are as follows:

• Backward compatibility: Backwards compatibility with ASCII and the enormous amount of software designed to process ASCII-encoded text was the main driving force behind the design of UTF-8. In UTF-8, single bytes with values in the range of 0 to 127 map directly to Unicode code points in the ASCII range. Single bytes in this range represent characters, as they do in ASCII. Moreover, 7-bit bytes (bytes where the most significant bit is 0) never appear in a multi-byte sequence, and no valid multi-byte sequence decodes to an ASCII code-point. A sequence of 7-bit bytes is both valid ASCII and valid UTF-8, and under either interpretation represents the same sequence of characters. Therefore, the 7-bit bytes in a UTF-8 stream represent all and only the ASCII characters in the stream. Thus, many text processors, parsers, protocols, file formats, text display programs etc., which use ASCII characters for formatting and control purposes will continue to work as intended by treating the UTF-8 byte stream as a sequence of single-byte characters, without decoding the multi-byte sequences. ASCII characters on which the processing turns, such as punctuation, whitespace, and control characters will never be encoded as multi-byte sequences. It is therefore safe for such processors to simply ignore or pass-through the multi-byte sequences, without decoding them. For example, ASCII whitespace may be used to tokenize a UTF-8 stream into words; ASCII line-feeds may be used to split a UTF-8 stream into lines; and ASCII NUL characters can be used to split UTF-8-encoded data into null-terminated strings. Similarly, many format strings used by library functions like "printf" will correctly handle UTF-8-encoded input arguments.
• Fallback and auto-detection: UTF-8 provided backwards compatibility for 7-bit ASCII, but much software and data uses 8-bit extended ASCII encodings designed prior to the adoption of Unicode to represent the character sets of European languages. Part of the popularity of UTF-8 is due to the fact that it provides a form of backward compatibility for these as well. A UTF-8 processor which erroneously receives an extended ASCII file as input can "fall back" or replace 8-bit bytes using the appropriate code-point in the Unicode Latin-1 Supplement block, when the 8-bit byte appears outside a valid multi-byte sequence. Though it does happen, the 8-bit characters in extended ASCII encodings do not usually have the correct form for UTF-8 multi-byte sequences. This is because the 8-bit bytes which introduce multi-byte sequences in UTF-8 are primarily accented letters (mostly vowels) in the common extended ASCII encodings, and the UTF-8 continuation bytes are punctuation and symbol characters. To appear as a valid UTF-8 multi-byte sequence, a series of 2 to 4 extended ASCII 8-bit characters would have to be an unusual combination of symbols and accented letters. In short, extended ASCII character sequences which look like valid UTF-8 multi-byte sequences are unlikely. Fallback errors will be false negatives, and these will be rare. Moreover, in many applications, such as text display, the consequence of incorrect fallback is usually slight. Only legibility is affected, and not significantly. These two things make fallback feasible, if somewhat imperfect. Indeed, as discussed further below, the HTML5 standard requires that erroneous bytes in supposed UTF-8 data be replaced upon display on the assumption that they are Windows-1252 characters. The presence of invalid 8-bit characters outside valid multi-byte sequences can also be used to "auto-detect" that an encoding is actually an extended ASCII encoding rather than UTF-8, and decode it accordingly. A UTF-8 stream may simply contain errors, resulting in the auto-detection scheme producing false positives; but auto-detection is successful in the majority of cases, especially with longer texts, and is widely used.
• Prefix code: The first byte indicates the number of bytes in the sequence. Reading from a stream can instantaneously decode each individual fully received sequence, without first having to wait for either the first byte of a next sequence or an end-of-stream indication. The length of multi-byte sequences is easily determined by humans as it is simply the number of high-order 1s in the leading byte. An incorrect character will not be decoded if a stream ends mid-sequence.
• Self-synchronization: The leading bytes and the continuation bytes do not share values (continuation bytes start with 10 while single bytes start with 0 and longer lead bytes start with 11). This means a search will not accidentally find the sequence for one character starting in the middle of another character. It also means the start of a character can be found from a random position by backing up at most 3 bytes to find the leading byte. An incorrect character will not be decoded if a stream starts mid-sequence, and a shorter sequence will never appear inside a longer one.
• Sorting order: The chosen values of the leading bytes and the fact that the continuation bytes have the high-order bits first means that a list of UTF-8 strings can be sorted in code point order by sorting the corresponding byte sequences.

Examples

Consider the encoding of the Euro sign, €.

1. The Unicode code point for "€" is U+20AC.
2. According to the scheme table above, this will take three bytes to encode, since it is between U+0800 and U+FFFF.
3. Hexadecimal 20AC is binary 0010 0000 1010 1100. The two leading zeros are added because, as the scheme table shows, a three-byte encoding needs exactly sixteen bits from the code point.
4. Because the encoding will be three bytes long, its leading byte starts with three 1s, then a 0 (1110...)
5. The first four bits of the code point are stored in the remaining low order four bits of this byte (1110 0010), leaving 12 bits of the code point yet to be encoded (...0000 1010 1100).
6. All continuation bytes contain exactly six bits from the code point. So the next six bits of the code point are stored in the low order six bits of the next byte, and 10 is stored in the high order two bits to mark it as a continuation byte (so 1000 0010).
7. Finally the last six bits of the code point are stored in the low order six bits of the final byte, and again 10 is stored in the high order two bits (1010 1100).

The three bytes 1110 0010 1000 0010 1010 1100 can be more concisely written in hexadecimal, as E2 82 AC.
Since UTF-8 uses groups of six bits, it is sometimes useful to use octal notation which uses 3-bit groups. With a calculator which can convert between hexadecimal and octal it can be easier to manually create or interpret UTF-8 compared with using binary.

• Octal 0200–3777 (hex 80-7FF) shall be coded with two bytes. xxyy will be 3xx 2yy.
• Octal 4000–177777 (hex 800-FFFF) shall be coded with three bytes. xxyyzz will be (340+xx) 2yy 2zz.
• Octal 200000-4177777 (hex 10000-10FFFF) shall be coded with four bytes. wxxyyzz will be 36w 2xx 2yy 2zz.

The following table summarises this conversion, as well as others with different lengths in UTF-8. The colors indicate how bits from the code point are distributed among the UTF-8 bytes. Additional bits added by the UTF-8 encoding process are shown in black.

Character		Octal code point	Binary code point	Binary UTF-8	Octal UTF-8	Hexadecimal UTF-8
$	U+0024	044	010 0100	00100100	044	24
¢	U+00A2	0242	000 1010 0010	11000010 10100010	302 242	C2 A2
€	U+20AC	020254	0010 0000 1010 1100	11100010 10000010 10101100	342 202 254	E2 82 AC
𐍈	U+10348	0201510	0 0001 0000 0011 0100 1000	11110000 10010000 10001101 10001000	360 220 215 210	F0 90 8D 88

Codepage layout

The following table summarizes usage of UTF-8 code units (individual bytes or octets) in a code page format. The upper half (0_ to 7_) is for bytes used only in single-byte codes, so it looks like a normal code page; the lower half is for continuation bytes (8_ to B_) and (possible) leading bytes (C_ to F_), and is explained further in the legend below.

UTF-8

	_0	_1	_2	_3	_4	_5	_6	_7	_8	_9	_A	_B	_C	_D	_E	_F
0_	NUL 0000 0	SOH 0001 1	STX 0002 2	ETX 0003 3	EOT 0004 4	ENQ 0005 5	ACK 0006 6	BEL 0007 7	BS 0008 8	HT 0009 9	LF 000A 10	VT 000B 11	FF 000C 12	CR 000D 13	SO 000E 14	SI 000F 15
1_	DLE 0010 16	DC1 0011 17	DC2 0012 18	DC3 0013 19	DC4 0014 20	NAK 0015 21	SYN 0016 22	ETB 0017 23	CAN 0018 24	EM 0019 25	SUB 001A 26	ESC 001B 27	FS 001C 28	GS 001D 29	RS 001E 30	US 001F 31
2_	SP 0020 32	! 0021 33	" 0022 34	# 0023 35	$ 0024 36	% 0025 37	& 0026 38	' 0027 39	( 0028 40	) 0029 41	* 002A 42	+ 002B 43	, 002C 44	- 002D 45	. 002E 46	/ 002F 47
3_	0 0030 48	1 0031 49	2 0032 50	3 0033 51	4 0034 52	5 0035 53	6 0036 54	7 0037 55	8 0038 56	9 0039 57	: 003A 58	; 003B 59	< 003C 60	= 003D 61	> 003E 62	? 003F 63
4_	@ 0040 64	A 0041 65	B 0042 66	C 0043 67	D 0044 68	E 0045 69	F 0046 70	G 0047 71	H 0048 72	I 0049 73	J 004A 74	K 004B 75	L 004C 76	M 004D 77	N 004E 78	O 004F 79
5_	P 0050 80	Q 0051 81	R 0052 82	S 0053 83	T 0054 84	U 0055 85	V 0056 86	W 0057 87	X 0058 88	Y 0059 89	Z 005A 90	[ 005B 91	\ 005C 92	] 005D 93	^ 005E 94	_ 005F 95
6_	` 0060 96	a 0061 97	b 0062 98	c 0063 99	d 0064 100	e 0065 101	f 0066 102	g 0067 103	h 0068 104	i 0069 105	j 006A 106	k 006B 107	l 006C 108	m 006D 109	n 006E 110	o 006F 111
7_	p 0070 112	q 0071 113	r 0072 114	s 0073 115	t 0074 116	u 0075 117	v 0076 118	w 0077 119	x 0078 120	y 0079 121	z 007A 122	{ 007B 123	| 007C 124	} 007D 125	~ 007E 126	DEL 007F 127
8_	• +00 128	• +01 129	• +02 130	• +03 131	• +04 132	• +05 133	• +06 134	• +07 135	• +08 136	• +09 137	• +0A 138	• +0B 139	• +0C 140	• +0D 141	• +0E 142	• +0F 143
9_	• +10 144	• +11 145	• +12 146	• +13 147	• +14 148	• +15 149	• +16 150	• +17 151	• +18 152	• +19 153	• +1A 154	• +1B 155	• +1C 156	• +1D 157	• +1E 158	• +1F 159
A_	• +20 160	• +21 161	• +22 162	• +23 163	• +24 164	• +25 165	• +26 166	• +27 167	• +28 168	• +29 169	• +2A 170	• +2B 171	• +2C 172	• +2D 173	• +2E 174	• +2F 175
B_	• +30 176	• +31 177	• +32 178	• +33 179	• +34 180	• +35 181	• +36 182	• +37 183	• +38 184	• +39 185	• +3A 186	• +3B 187	• +3C 188	• +3D 189	• +3E 190	• +3F 191
2-byte C_	192	193	Latin 0080 194	Latin 00C0 195	Latin 0100 196	Latin 0140 197	Latin 0180 198	Latin 01C0 199	Latin 0200 200	IPA 0240 201	IPA 0280 202	IPA 02C0 203	accents 0300 204	accents 0340 205	Greek 0380 206	Greek 03C0 207
2-byte D_	Cyril 0400 208	Cyril 0440 209	Cyril 0480 210	Cyril 04C0 211	Cyril 0500 212	Armeni 0540 213	Hebrew 0580 214	Hebrew 05C0 215	Arabic 0600 216	Arabic 0640 217	Arabic 0680 218	Arabic 06C0 219	Syriac 0700 220	Arabic 0740 221	Thaana 0780 222	N'Ko 07C0 223
3-byte E_	Indic 0800 224	Misc. 1000 225	Symbol 2000 226	Kana, CJK 3000 227	CJK 4000 228	CJK 5000 229	CJK 6000 230	CJK 7000 231	CJK 8000 232	CJK 9000 233	Asian A000 234	Hangul B000 235	Hangul C000 236	Hangul D000 237	PUA E000 238	Forms F000 239
4‑byte F_	SMP, SIP 10000 240	40000 241	80000 242	SSP, SPUA C0000 243	SPUA-B 100000 244	245	246	247	248	249	250	251	252	253	254	255

Orange cells with a large dot are continuation bytes. The hexadecimal number shown after a "+" plus sign is the value of the six bits they add.
White cells are the leading bytes for a sequence of multiple bytes, the length shown at the left edge of the row. The text shows the Unicode blocks encoded by sequences starting with this byte, and the hexadecimal code point shown in the cell is the lowest character value encoded using that leading byte.
Red cells must never appear in a valid UTF-8 sequence. The first two red cells (C0 and C1) could be used only for a two-byte encoding of a 7-bit ASCII character which should be encoded in one byte; as described below such "overlong" sequences are disallowed. The red cells in the F row (F5 to FD) indicate leading bytes of 4-byte or longer sequences that cannot be valid because they would encode code points larger than the U+10FFFF limit of Unicode (a limit derived from the maximum code point encodable in UTF-16), and FE and FF were never defined for any purpose in UTF-8.
Pink cells are the leading bytes for a sequence of multiple bytes, of which some, but not all, possible continuation sequences are valid. E0 and F0 could start overlong encodings, in this case the lowest non-overlong-encoded code point is shown. F4 can start code points greater than U+10FFFF which are invalid. ED can start the encoding of a code point in the range U+D800–U+DFFF; these are invalid since they are reserved for UTF-16 surrogate halves.

Overlong encodings

In principle, it would be possible to inflate the number of bytes in an encoding by padding the code point with leading 0s. To encode the Euro sign € from the above example in four bytes instead of three, it could be padded with leading 0s until it was 21 bits long – 000 000010 000010 101100, and encoded as 11110000 10000010 10000010 10101100 (or F0 82 82 AC in hexadecimal). This is called an overlong encoding.
The standard specifies that the correct encoding of a code point use only the minimum number of bytes required to hold the significant bits of the code point. Longer encodings are called overlong and are not valid UTF-8 representations of the code point. This rule maintains a one-to-one correspondence between code points and their valid encodings, so that there is a unique valid encoding for each code point. This ensures that string comparisons and searches are well-defined.
Modified UTF-8 uses the two-byte overlong encoding of U+0000 (the NUL character), 11000000 10000000 (hexadecimal C0 80), instead of 00000000 (hexadecimal 00). This allows the byte 00 to be used as a string terminator.

Invalid byte sequences

Not all sequences of bytes are valid UTF-8. A UTF-8 decoder should be prepared for:

• the red invalid bytes in the above table
• an unexpected continuation byte
• a leading byte not followed by enough continuation bytes (can happen in simple string truncation, when a string is too long to fit when copying it)
• an overlong encoding as described above
• a sequence that decodes to an invalid code point as described below

Many earlier decoders would happily try to decode these. Carefully crafted invalid UTF-8 could make them either skip or create ASCII characters such as NUL, slash, or quotes. Invalid UTF-8 has been used to bypass security validations in high-profile products including Microsoft's IIS web server^[12] and Apache's Tomcat servlet container.^[13]
RFC 3629 states "Implementations of the decoding algorithm MUST protect against decoding invalid sequences."^[14] The Unicode Standard requires decoders to "...treat any ill-formed code unit sequence as an error condition. This guarantees that it will neither interpret nor emit an ill-formed code unit sequence."
Many UTF-8 decoders throw exceptions on encountering errors.^[15] This can turn what would otherwise be harmless errors (producing a message such as "no such file") into a denial of service bug. Early versions of Python 3.0 would exit immediately if the command line or environment variables contained invalid UTF-8,^[16] making it impossible to handle such errors.
More recent converters translate the first byte of an invalid sequence to a replacement character and continue parsing with the next byte. These error bytes will always have the high bit set. This avoids denial-of-service bugs, and it is very common in text rendering such as browser display, since mangled text is probably more useful than nothing for helping the user figure out what the string was supposed to contain. Popular replacements include:

• The replacement character "�" (U+FFFD) (or EF BF BD in UTF-8)
• The invalid Unicode code points U+DC80–U+DCFF where the low eight bits are the byte's value.^[17] Sometimes it is called UTF-8B^[18]
• The Unicode code points U+0080–U+00FF with the same value as the byte, thus interpreting the bytes according to ISO-8859-1^{[citation needed]}
• The Unicode code point for the character represented by the byte in CP1252,^{[citation needed]} which is similar to using ISO-8859-1, except that some characters in the range 0x80–0x9F are mapped into different Unicode code points. For example, 0x80 becomes the Euro sign, U+20AC.

These replacement algorithms are "lossy", as more than one sequence is translated to the same code point. This means that it would not be possible to reliably convert back to the original encoding, therefore losing information. Reserving 128 code points (such as U+DC80–U+DCFF) to indicate errors, and defining the UTF-8 encoding of these points as invalid so they convert to 3 errors, would seem to make conversion lossless. But this runs into practical difficulties: the converted text cannot be modified such that errors are arranged so they convert back into valid UTF-8, which means if the conversion is UTF-16, it cannot also be used to store arbitrary invalid UTF-16, which is usually needed on the same systems that need invalid UTF-8. U+DC80–U+DCFF are reserved for UTF-16 surrogates, so that when they are used for UTF-8 in this way, and the string is converted to UTF-16 this can lead to bugs or the string being rejected.
The large number of invalid byte sequences provides the advantage of making it easy to have a program accept both UTF-8 and legacy encodings such as ISO-8859-1. Software can check for UTF-8 correctness, and if that fails assume the input to be in the legacy encoding. It is technically true that this may detect an ISO-8859-1 string as UTF-8, but this is very unlikely if it contains any 8-bit bytes as they all have to be in unusual patterns of two or more in a row, such as "£".

Invalid code points

Since RFC 3629 (November 2003), the high and low surrogate halves used by UTF-16 (U+D800 through U+DFFF) and code points not encodable by UTF-16 (those after U+10FFFF) are not legal Unicode values, and their UTF-8 encoding must be treated as an invalid byte sequence.
Not decoding surrogate halves makes it impossible to store invalid UTF-16, such as Windows filenames, as UTF-8. Therefore, detecting these as errors is often not implemented and there are attempts to define this behavior formally (see WTF-8 and CESU below).