Normalise text before you compare or store it

How to do it

"é" can be one code point or an "e" followed by a combining accent. Both look the same, but without normalisation they never match, so search, duplicate checks and unique keys fail. Store NFC. For identifiers, compare the NFKC_Casefold form, which also folds full-width letters such as "Ａ" and ligatures such as "fi". Change case without a locale, as Change the case of identifiers without a locale says. Refuse names that only look like another one, as Reserve names that look official says, with the UTS #39 confusable check.

Examples

Python example

import unicodedata

def stored_text(text: str) -> str:
    return unicodedata.normalize("NFC", text)

def identifier_key(name: str) -> str:
    # Close to NFKC_Casefold for usernames and handles
    return unicodedata.normalize("NFKC", unicodedata.normalize("NFKC", name).casefold())

Java example

import com.ibm.icu.text.Normalizer2;

String stored = Normalizer2.getNFCInstance().normalize(text);
String key = Normalizer2.getNFKCCasefoldInstance().normalize(name);

How to check

A unit test saves "e" plus U+0301 and finds the record by searching "é". Signing up as "ＡＬＩＣＥ" when "alice" exists is refused.

Sources

• UAX #15 (Unicode Normalization Forms)
• UTS #39 (Unicode Security Mechanisms)

Related rules

• LOC-15 Sort and compare text by locale in Localization
• LOC-29 Change the case of identifiers without a locale in Localization
• ACC-5 Reserve names that look official in Accounts and identity