Cover every script with fonts and break words with a segmenter

How to do it

Thai, Lao, Khmer, Chinese and Japanese put no spaces between words. Code that splits on spaces sees one long word, so word counts, excerpts and highlights break in the middle of words. Use Intl.Segmenter with granularity: 'word' in JavaScript, or ICU's BreakIterator in Java and Python. Browsers break lines in these scripts only when lang names the language. The same Han character also takes a different shape in Japanese and Chinese, and lang picks the right one. End each font stack with a family that covers the script, such as a Noto font. Cut text by visible character as Count and cut text by visible characters says.

Examples

JavaScript example

const words = [...new Intl.Segmenter('th', { granularity: 'word' }).segment('สวัสดีครับ')]
  .filter(part => part.isWordLike).map(part => part.segment);

Java example

import java.text.BreakIterator;
import java.util.Locale;

BreakIterator words = BreakIterator.getWordInstance(Locale.forLanguageTag("th"));
words.setText("สวัสดีครับ");

How to check

A unit test segments "สวัสดีครับ" with the Thai locale and gets more than one word. A browser test renders a sample sentence in every supported language at phone width, and a screenshot shows no missing-glyph boxes.

Sources

• UAX #29 (Unicode Text Segmentation)
• UAX #14 (Unicode Line Breaking Algorithm)
• ECMA-402 (Segmenter objects)

Related rules

• LOC-50 Count and cut text by visible characters in Localization