ASCII, Unicode, and Text Representation

ASCII (American Standard Code for Information Interchange), standardized in 1963, encodes 128 characters in 7 bits (values 0-127). It covers the English alphabet (upper and lowercase), digits, punctuation, and control characters.

ASCII was designed for English and American telecommunication equipment. It worked well for its intended purpose but was fundamentally inadequate for any language beyond English.

The first 32 ASCII characters (0-31) plus DEL (127) are control characters. They were originally designed for controlling Teletype machines: LF (line feed, 10), CR (carriage return, 13), BEL (bell, 7), TAB (9), ESC (27).

In cryptographic contexts, control characters can cause problems. A null byte (0x00) terminates C strings early, and a DEL or ESC character might be interpreted by terminal emulators.

Different countries created their own extensions of ASCII using the 8th bit (values 128-255), creating hundreds of incompatible encodings: ISO-8859-1 (Latin-1) for Western European, KOI-8R for Russian, Big5 for Chinese.

A document saved in one encoding appears as gibberish when opened with a different encoding. The lack of a universal standard made international software development a significant challenge throughout the 1980s and 1990s.

Unicode was created to provide a single universal character set covering all human writing systems. It currently defines over 149,000 characters covering 161 scripts, including historical scripts and symbols.

Unicode separates character identity (a code point, like U+0041 for "A") from encoding (how that code point is stored in bytes). This separation allows multiple encoding formats to represent the same characters.

UTF-8 encodes Unicode code points using 1 to 4 bytes. ASCII characters (U+0000 to U+007F) use exactly 1 byte, identical to their ASCII values, making UTF-8 backward compatible with ASCII.

Characters from U+0080 to U+07FF use 2 bytes. U+0800 to U+FFFF use 3 bytes (covering most common scripts including Chinese, Japanese, Korean). U+10000 and beyond use 4 bytes.

UTF-16 uses 2 bytes for the most common characters (the Basic Multilingual Plane, U+0000 to U+FFFF) and 4 bytes (surrogate pairs) for characters beyond U+FFFF. It is used internally by Windows and Java.

UTF-32 uses exactly 4 bytes per code point, making it fixed-width and easy to index by character position, but wasteful for text that is mostly ASCII. It is used in some internal representations for fast random access.

The Byte Order Mark (BOM) is the Unicode character U+FEFF placed at the start of a file to indicate byte order and encoding. In UTF-16, it distinguishes big-endian (FE FF) from little-endian (FF FE).

In UTF-8, the BOM (EF BB BF) is unnecessary since UTF-8 has no byte order issues, but some Windows software adds it anyway. This causes issues in cryptographic applications where the BOM is treated as data rather than a marker.

Cryptographic hash functions and MACs operate on byte sequences, not abstract characters. The same string "café" encodes differently in UTF-8 (4 bytes: 63 61 66 C3 A9) vs Latin-1 (4 bytes: 63 61 66 E9).

If two systems hash the same string but use different encodings, they will produce different hashes and authentication will fail. Crypto protocols must specify encoding explicitly to ensure interoperability.

Emoji are Unicode characters in the Supplementary Multilingual Plane. The "Grinning Face" emoji (U+1F600) encodes to 4 bytes in UTF-8: F0 9F 98 80.

In security contexts, emoji and full-width Unicode characters have been used in homograph attacks, where a URL like "xn--pple-43d.com" (which looks like "apple.com") tricks users into visiting a malicious site.

Some characters can be represented in multiple Unicode forms. "e with acute accent" can be U+00E9 (precomposed) or U+0065 U+0301 (e followed by combining accent, decomposed). These look identical but have different byte representations.

Cryptographic systems that do not normalize Unicode before hashing may produce different hashes for visually identical strings. NFKC normalization is commonly recommended before applying cryptographic operations to text.

When implementing cryptographic systems, the encoding choice is a critical decision that must be documented. Passwords should be encoded as UTF-8 before hashing. Protocol fields should specify encoding in their specification documents.

Interoperability failures caused by encoding mismatches are a common source of bugs in cryptographic systems. Two implementations of the same protocol can produce different authentication results if they encode strings differently.

Test your understanding of UTF-8 encoding.

ASCII covers 128 characters in 7 bits. Unicode provides a universal code point space for all human scripts. UTF-8 encodes Unicode in 1-4 bytes, with ASCII as a 1-byte subset.

In cryptography, encoding matters because hash functions operate on bytes. The same text in different encodings produces different hashes. Unicode normalization is essential before cryptographic operations on text.