Check every generated image and video before anyone sees it

How to do it

Generators can produce illegal images from harmless-looking prompts, and producing them is a crime in most countries. Check the prompt first, then the output, because each catches what the other misses. Use the provider's safety settings and a classifier of your own on the output. Set the strictest level for anything that looks like a minor. Never store a blocked output. Log the block with the account, and suspend accounts that keep trying. Report child sexual abuse material to the body your law names, such as the national hotline. Label what you do show, as say when people talk to an AI says. Text output gets the same treatment in check output before anyone sees it.

Examples

TypeScript example

export async function generateImage(prompt: string, user: User): Promise<StoredImage> {
  if ((await safety.checkPrompt(prompt)).blocked) return blocked(user, 'prompt');
  const image = await ai.generateImage({ prompt, userId: user.id });
  if ((await safety.checkImage(image.bytes)).blocked) return blocked(user, 'output');
  return storage.save(image, { aiGenerated: true, userId: user.id });
}

Python example

def generate_image(prompt: str, user: User) -> StoredImage:
    if safety.check_prompt(prompt).blocked:
        return blocked(user, "prompt")
    image = ai.generate_image(prompt=prompt, user_id=user.id)
    if safety.check_image(image.data).blocked:
        return blocked(user, "output")
    return storage.save(image, ai_generated=True, user_id=user.id)

How to check

A test sends a set of blocked prompts and fake outputs that the classifier flags. Each is blocked before display, nothing is written to storage and a block event is logged. A search of the code finds no path from the generator to storage that skips the check.

Sources

• NIST AI 600-1, Generative AI Profile, section 2.11

Related rules

• AIF-60 Say when people talk to an AI in AI features
• AIF-34 Check output before anyone sees it in AI features
• AIF-79 Refuse likenesses and protected works you hold no rights to in AI features