Check output before anyone sees it

How to do it

Validate the schema first. Then check for content the feature must never produce, such as prices, promises, other customers' data, links to unknown domains or personal data. Compare any number or price in the answer with your own data. Use the provider's moderation endpoint or a classifier for harmful content where the feature is public. On a failure, show a fixed safe reply and log the case. Rendering rules are in treat model output as untrusted.

Example

const ALLOWED_LINK_HOSTS = new Set(['example.com', 'help.example.com']);
const EMAIL = /[\w.+-]+@[\w-]+\.[\w.]+/;

export function outputProblems(text: string): string[] {
  const problems: string[] = [];
  for (const match of text.matchAll(/https?:\/\/[^\s)"]+/g)) if (!ALLOWED_LINK_HOSTS.has(new URL(match[0]).hostname)) problems.push('unknown_link');
  if (EMAIL.test(text)) problems.push('email_address');
  if (/\b(we will|we'll) refund\b/i.test(text)) problems.push('refund_promise');
  return problems;
}

ALLOWED_LINK_HOSTS = {"example.com", "help.example.com"}
EMAIL = re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+")

def output_problems(text: str) -> list[str]:
    problems = ["unknown_link" for url in re.findall(r"https?://[^\s)\"]+", text)
                if urlparse(url).hostname not in ALLOWED_LINK_HOSTS]
    if EMAIL.search(text):
        problems.append("email_address")
    if re.search(r"\b(we will|we'll) refund\b", text, re.I):
        problems.append("refund_promise")
    return problems

How to check

Unit tests feed the output checks an answer with an unknown link, one with an email address and one promising a refund. Each is blocked and logged, and the user sees the safe reply.