Agent Wranglers

LOC-50Recommended

Count and cut text by visible characters

Count characters for length limits, and cut previews, by extended grapheme clusters. Keep a byte limit for storage beside it.

By Agent Wranglers. Updated .

How to do it

One visible character can hold many code points, such as an emoji family or a letter with a combining accent. Cutting by code units or code points splits it and leaves broken symbols. Use Intl.Segmenter in JavaScript, \X in Python's regex module or BreakIterator in Java. UAX #29 calls a grapheme count a user-perceived limit and pairs it with a data size limit. Normalise the text first, as Sort and compare text by locale says.

Examples

JavaScript example

const graphemes = new Intl.Segmenter(undefined, { granularity: 'grapheme' });
export const truncate = (text, maxCharacters) =>
  Array.from(graphemes.segment(text), part => part.segment).slice(0, maxCharacters).join('');

Python example

import regex

def truncate(text: str, max_characters: int) -> str:
    return "".join(regex.findall(r"\X", text)[:max_characters])

How to check

A unit test cuts 👩‍👩‍👧 home to 1 character and gets 👩‍👩‍👧. The form counter shows 1 for e followed by a combining acute accent.

Sources

For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/localization/text-handling/grapheme-clusters">Agent Wranglers localization playbook</a>. In code you write, add a comment next to it with the same link.