The assistant answers questions it should refuse

Guide, October 2026. Code is Python with the Anthropic SDK; the pattern works with any provider. The categories are examples; yours come from your own risk list.

Why the prompt alone is not enough

A line in the system prompt saying "only answer questions about X" is advice to the model, and the model weighs it against everything else in the conversation. A long message, a role-play framing, or a request that looks adjacent to X will get through often enough to matter. Worse, the prompt cannot do the thing that matters most in sensitive domains: recognise that a message is not a question at all but a person in trouble, and route it to a human.

A classifier is a separate, small, fast model call whose only job is to label the input before the main model sees it. It cannot be argued with, because it does not converse. It returns a category and the system acts on the category.

The design

1. Define the categories before writing code

Four is usually enough to start: in_scope (answer normally), out_of_scope (decline politely and say what the assistant is for), needs_human (stop, show the escalation message, notify a person), and unsafe (refuse and log). The needs_human category is the one that earns the whole system its keep; write its definition with the people who will receive the notification.

2. Call a small model with a strict output

import anthropic, json, time

client = anthropic.Anthropic()
CATEGORIES = ["in_scope", "out_of_scope", "needs_human", "unsafe"]

def classify(text: str, timeout_s: float = 8.0) -> dict:
    t0 = time.time()
    try:
        msg = client.with_options(timeout=timeout_s, max_retries=0).messages.create(
            model="claude-haiku-5-5",
            max_tokens=20,
            system=("Label the user message with exactly one of: " + ", ".join(CATEGORIES) +
                    ". The assistant answers questions about office facilities and bookings for employees. "
                    "needs_human means a safety hazard, an injury, or a person in distress. Reply with the label only."),
            messages=[{"role": "user", "content": text}],
        )
        label = msg.content[0].text.strip().lower()
        if label not in CATEGORIES:
            label = "needs_human"           # an unexpected answer fails closed
        return {"label": label, "ms": int((time.time() - t0) * 1000), "source": "classifier"}
    except Exception as e:
        return {"label": None, "ms": int((time.time() - t0) * 1000), "source": "error", "error": type(e).__name__}

Three details matter. The timeout is short and explicit, because the classifier sits on the critical path of every request. Retries are off at the SDK level so the system controls them (one retry with a shorter budget is reasonable). An unexpected label fails closed to needs_human, never open.

3. Add a keyword fallback for when the classifier is unavailable

Models have slow minutes. When the classifier times out, the request must still be handled, and "let it through" is the wrong default in any domain where needs_human exists. A keyword list, maintained in a table you can edit without a deploy, catches the phrases that must never reach the main model unflagged. It is crude, which is fine; it only runs when the careful check is down, and a false positive costs a person a minute while a false negative costs much more.

def route(text: str, keywords: list[str]) -> dict:
    hit = [k for k in keywords if k in text.lower()]
    if hit:
        return {"label": "needs_human", "source": "keyword", "hit": hit}
    c = classify(text)
    if c["label"] is None:
        c = classify(text, timeout_s=6.0)        # one retry, shorter budget
    if c["label"] is None:
        return {"label": "unavailable", "source": "fallback"}   # answer withheld, draft kept
    return c

Run the keyword check first, on every request, not only when the classifier fails: it is cheap, and a known phrase should not depend on a network call. When both the classifier and the keywords give nothing, the honest response is "the safety check is unavailable, try again in a moment", with the user's draft preserved, and an alert to the people who run the system.

4. Log the decision with the request

Every request should carry a trace: the classifier's label, its latency, whether a keyword fired, which attempt succeeded, and what the system did as a result. Without this, you cannot answer the question that will be asked after the first incident: did the check run, and what did it say? With it, the answer is a query.

How to test it

• Write at least twenty messages per category, including awkward ones: an in-scope question that mentions a sensitive word in passing, an out-of-scope request phrased politely, a needs_human message written in the third person or in another language your users speak.
• Run them through route and check every label. This is the start of your eval set.
• Simulate a slow classifier (set the timeout to a millisecond) and confirm the keyword fallback and the unavailable path both behave, and that the alert fires.
• Measure latency on real traffic for a week. A classifier that takes two seconds on sensitive inputs and one on benign ones is normal; the timeout should sit comfortably above the slow tail.

Mistakes we have seen

• Trusting the classifier's default retries. SDK retries with backoff can turn an 8 second budget into 30, and the user sees a hang. Own the retry.
• Failing open on timeout. It feels user-friendly. It means the one message that mattered was the one that got through unchecked.
• Keywords only in one language. If users write in two, the list covers two.

When to ask for help

If the assistant is already in front of users without this layer, adding it is days of work, and we would start with the needs_human definition and the keyword list, since those are the pieces that need your domain knowledge. The AI agents FAQ lists the other parts of a production assistant.

Related: safety classifier, trace and guardrail in the glossary.