Split compound words in languages that join them

How to do it

These languages write compounds as one word. A Swedish shop sells "vinterjacka", and a search for "jacka" finds nothing with a plain stemmer. A dictionary decompounder splits a word into known words from a list, and keeps the whole word too. Elasticsearch and OpenSearch offer dictionary_decompounder and hyphenation_decompounder token filters. PostgreSQL splits compounds with an Ispell or Hunspell dictionary whose affix file allows compounds. Build the word list from your own catalogue and a public dictionary. Set a minimum part length, such as 3 letters, so short parts do not match everything. Sorting and comparing text follow Sort and compare text by locale.

Examples

JSON example

{
  "settings": {
    "analysis": {
      "filter": {
        "sv_parts": { "type": "dictionary_decompounder", "word_list_path": "analysis/sv-words.txt", "min_subword_size": 3 },
        "sv_stem": { "type": "stemmer", "language": "swedish" }
      },
      "analyzer": {
        "swedish_compound": { "tokenizer": "standard", "filter": ["lowercase", "sv_parts", "sv_stem"] }
      }
    }
  }
}

Python example

def split_compound(word: str, known: set[str], min_part: int = 3) -> list[str]:
    word = word.lower()
    for cut in range(min_part, len(word) - min_part + 1):
        head, tail = word[:cut], word[cut:]
        if head in known and tail in known:
            return [word, head, tail]
    return [word]

JavaScript example

export function splitCompound(word, known, minPart = 3) {
  const lower = word.toLowerCase();
  for (let cut = minPart; cut <= lower.length - minPart; cut++) {
    const [head, tail] = [lower.slice(0, cut), lower.slice(cut)];
    if (known.has(head) && known.has(tail)) return [lower, head, tail];
  }
  return [lower];
}

How to check

A test holds at least 20 compound pairs per supported language, such as "vinterjacka" and "jacka", or "Kinderwagen" and "Wagen". A search for each part returns the record holding the compound in the top 10.

Sources

• OpenSearch dictionary decompounder token filter
• PostgreSQL text search dictionaries, Ispell

Related rules

• LOC-15 Sort and compare text by locale in Localization