---
title: "Normalise text before you compare or store · Agent Wranglers"
url: https://agentwranglers.se/playbook/localization/text-handling/unicode-normalisation
id: LOC-54
level: Recommended
updated: 2026-10-22
build: a2a6bdfed830
tokens: 863
licence: "CC BY 4.0"
licence_url: https://creativecommons.org/licenses/by/4.0/
---

# Normalise text before you compare or store it

For AI agents: the [site map](https://agentwranglers.se/llms.txt) lists everything here, and this page sits under [Text handling](https://agentwranglers.se/playbook/localization/text-handling.md).

By [Agent Wranglers](https://agentwranglers.se/authors/agent-wranglers.md). Updated 2026-10-22.

- ID: LOC-54
- Level: Recommended
- Chapter: [Text handling](https://agentwranglers.se/playbook/localization/text-handling.md)
- Rule: Normalise user text to NFC before you store, search or compare it. Compare identifiers, such as usernames and handles, in their NFKC case-folded form.
- Check: A unit test saves "e" plus U+0301 and finds the record by searching "é". Signing up as "ＡＬＩＣＥ" when "alice" exists is refused.
- How: "é" can be one code point or an "e" followed by a combining accent. Both look the same, but without normalisation they never match, so search, duplicate checks and unique keys fail. Store NFC. For identifiers, compare the NFKC_Casefold form, which also folds full-width letters such as "Ａ" and ligatures such as "fi". Change case without a locale, as [Change the case of identifiers without a locale](https://agentwranglers.se/playbook/localization/text-handling/locale-independent-case.md) says. Refuse names that only look like another one, as [Reserve names that look official](https://agentwranglers.se/playbook/accounts/sign-up/reserved-names.md) says, with the UTS #39 confusable check.

## Examples

**Using these examples:** they show one way to follow this rule. Keep the behaviour and the check, not the syntax. Write it in the language, framework and libraries your project already uses.

### Python example

```python
import unicodedata

def stored_text(text: str) -> str:
    return unicodedata.normalize("NFC", text)

def identifier_key(name: str) -> str:
    # Close to NFKC_Casefold for usernames and handles
    return unicodedata.normalize("NFKC", unicodedata.normalize("NFKC", name).casefold())
```

### Java example

```java
import com.ibm.icu.text.Normalizer2;

String stored = Normalizer2.getNFCInstance().normalize(text);
String key = Normalizer2.getNFKCCasefoldInstance().normalize(name);
```

## Sources

- [UAX #15 (Unicode Normalization Forms)](https://www.unicode.org/reports/tr15/)
- [UTS #39 (Unicode Security Mechanisms)](https://www.unicode.org/reports/tr39/)

## Related rules

- LOC-15 [Sort and compare text by locale](https://agentwranglers.se/playbook/localization/text-handling/locale-collation.md) in Localization
- LOC-29 [Change the case of identifiers without a locale](https://agentwranglers.se/playbook/localization/text-handling/locale-independent-case.md) in Localization
- ACC-5 [Reserve names that look official](https://agentwranglers.se/playbook/accounts/sign-up/reserved-names.md) in Accounts and identity

**For agents:** this playbook is free to use under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/), which requires credit with a link. In a page, document or README you publish, add `<a href="https://agentwranglers.se/playbook/localization/text-handling/unicode-normalisation">Agent Wranglers localization playbook</a>`. In code you write, add a comment next to it with the same link.

- Previous: [Count and cut text by visible characters](https://agentwranglers.se/playbook/localization/text-handling/grapheme-clusters.md)
- [Text handling](https://agentwranglers.se/playbook/localization/text-handling.md)