Sort and compare text by locale

How to do it

Sort lists with a locale collator, such as Intl.Collator in JavaScript, PyICU in Python or java.text.Collator. Byte order puts "Ö" in the wrong place, and the right place differs by language. For search, match with the collator at base strength, so "resume" finds "résumé" in English. In Swedish, "å", "ä" and "ö" are letters of their own and must not fold to "a" and "o". A search engine needs the same per-language analyser. Normalise input as Normalise text before you compare or store it says. Change case with the locale, because Turkish lower-cases "I" to "ı".

Examples

JavaScript example

const words = ['ä', 'a', 'z', 'o', 'ö'];
words.sort(new Intl.Collator('sv').compare); // ["a", "o", "z", "ä", "ö"]
words.sort(new Intl.Collator('de').compare); // ["a", "ä", "o", "ö", "z"]
'Cafe\u0301'.normalize('NFC') === 'Café'; // true

Python example

import unicodedata
from icu import Collator, Locale
swedish = Collator.createInstance(Locale("sv_SE"))
sorted(["ä", "a", "z", "o", "ö"], key=swedish.getSortKey)  # ['a', 'o', 'z', 'ä', 'ö']
unicodedata.normalize("NFC", "Cafe\u0301") == "Café"  # True

Java example

import java.text.Collator;
import java.text.Normalizer;
import java.util.*;
List<String> words = new ArrayList<>(List.of("ä", "a", "z", "o", "ö"));
words.sort(Collator.getInstance(Locale.forLanguageTag("sv")));  // [a, o, z, ä, ö]
Normalizer.normalize("Cafe\u0301", Normalizer.Form.NFC).equals("Café");  // true

How to check

A unit test sorts a fixed list per locale and gets the expected order. Swedish puts "ä" after "z", and German puts it after "a". A search for "resume" with the English locale finds "résumé".

Sources

• UTS #10 (Unicode Collation Algorithm)
• UAX #15 (Unicode Normalization Forms)

Related rules

• DAT-43 Rebuild text indexes when the collation library changes in Data and databases
• APP-35 Sort and filter on the server in Web app interface
• LOC-54 Normalise text before you compare or store it in Localization
• LOC-50 Count and cut text by visible characters in Localization
• SRH-13 Set the language of every search field in Search
• SRH-14 Split compound words in languages that join them in Search