Try plain code before a model

How to do it

Describe the task in one sentence. Build or sketch the simplest version without a model first. Examples are a regular expression, a lookup table, a keyword search or a fixed form. Measure it on the same examples you will use for the model. Choose the model only when it beats that baseline by a margin you set in advance, such as 10 points of accuracy. Models suit fuzzy language tasks, such as summaries, free-text questions and messy extraction. They suit exact sums, dates, prices and permission checks badly.

Example

// A fixed rule handles most order-status questions. The model gets only the rest.
const ORDER_NUMBER = /\b[A-Z]-\d{4,}\b/;

export async function answerSupportQuestion(question: string, user: User) {
  const orderId = ORDER_NUMBER.exec(question)?.[0];
  if (orderId && /\b(where|status|shipped)\b/i.test(question)) return orderStatusReply(orderId, user);
  return assistant.answer(question, user);
}

ORDER_NUMBER = re.compile(r"\b[A-Z]-\d{4,}\b")

def answer_support_question(question: str, user: User) -> Reply:
    match = ORDER_NUMBER.search(question)
    if match and re.search(r"\b(where|status|shipped)\b", question, re.I):
        return order_status_reply(match.group(0), user)
    return assistant.answer(question, user)

How to check

Each AI feature has a short decision note in the repository. It names the baseline without a model, both scores on the same eval set and the margin the model had to beat.