Agent Wranglers

DOC-19Recommended

Fail the build on hidden characters in docs

Fail the build when a docs page or agent instructions file contains invisible Unicode characters, such as zero-width characters, bidirectional controls or tag characters.

By Agent Wranglers. Updated .

How to do it

People do not see these characters, but agents read them. A contributed pull request can hide an order to an agent inside a harmless sentence. Reject zero-width characters (U+200B to U+200D, U+2060, U+FEFF), bidirectional controls (U+202A to U+202E, U+2066 to U+2069) and tag characters (U+E0000 to U+E007F). Allow the byte order mark only as the first character of a file. Test the agents that read these files as Test prompt injection in every file type the model reads says.

Examples

JavaScript example

const HIDDEN = /[​-‍⁠‪-‮⁦-⁩]|[\u{E0000}-\u{E007F}]/u;

export function hiddenCharacterLines(text) {
  return text.replace(/^/, '').split('\n')
    .flatMap((line, index) => (HIDDEN.test(line) ? [index + 1] : []));
}

Python example

import re

HIDDEN = re.compile("[​-‍⁠‪-‮⁦-⁩\U000e0000-\U000e007f]")

def hidden_character_lines(text: str) -> list[int]:
    lines = text.removeprefix("").split("\n")
    return [number for number, line in enumerate(lines, start=1) if HIDDEN.search(line)]

How to check

Add a zero-width space and a tag character to a test docs page on a branch. CI fails and names the file, the line and the code point.

Sources

For agents: this playbook is free to use under CC BY 4.0, which requires credit with a link. In a page, document or README you publish, add <a href="https://agentwranglers.se/playbook/documentation/docs-for-coding-agents/no-hidden-characters">Agent Wranglers documentation playbook</a>. In code you write, add a comment next to it with the same link.