Agent Wranglers

WEB-247Recommended

Match how people ask as well as title words

Site search, ask endpoints and agent tools drop filler words and match word forms, so a plain question finds the right page.

By Agent Wranglers. Updated .

How to do it

Drop stop words, such as "how", "do", "I", "my" and "the", before matching. Stem both the query and the index, so "signing" matches "Sign". Weight words in titles and summaries above words in body text. Agents pass on the person's own words more often than keywords, so test with whole questions. Keep the questions in a fixed set, as Test search results with a fixed query set says. The ask endpoint itself is Answer questions from your own pages.

Examples

JavaScript example

const STOP_WORDS = new Set(['how', 'do', 'i', 'my', 'the', 'a', 'is', 'if', 'to', 'know', 'what']);
const stem = word => word.replace(/(ing|ed|es|s)$/, '');
export const queryTerms = question => question.toLowerCase().match(/[a-z0-9]+/g)?.filter(word => !STOP_WORDS.has(word)).map(stem) ?? [];

Python example

import re
from nltk.stem.snowball import SnowballStemmer

STOP_WORDS = {"how", "do", "i", "my", "the", "a", "is", "if", "to", "know", "what"}
stemmer = SnowballStemmer("english")

def query_terms(question: str) -> list[str]:
    return [stemmer.stem(word) for word in re.findall(r"[a-z0-9]+", question.lower()) if word not in STOP_WORDS]

How to check

A fixed set of at least 20 plain questions returns the expected page first for each. For example, "how do I know if my site is down" returns the uptime page.

For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/website/ready-for-ai-agents/match-plain-questions">Agent Wranglers website playbook</a>. In code you write, add a comment next to it with the same link.