A system prompt rule is a request, not a guarantee

You built an agent that calls tools, and it works. Then one day it does the exact thing you told it not to do, and the rule was right there, in capitals, at the top of the system prompt. If you ship LLM agents, this post is for you: it explains why a system prompt rule cannot enforce itself, and what can.

Everything measured here comes from one production analytics agent we run. We walk one ordinary question from a silently wrong answer to the graph edge that made the wrong answer impossible. By the end you will know the three places a rule can live — the prompt, the tool code, the graph — and which one to use for the rules you cannot afford to lose.

TL;DR

• A system prompt rule is a request. The model can decline it, and the decline is silent.
• A rule can live in three places: the prompt (a default), tool code (a check that only fires when the tool is called), or a graph edge (a guarantee).
• Fail closed means a blocked run ends as a question to the user, never as an answer nobody can trace.
• Keep the prompt rule anyway. The prompt sets the default; the edge sets the guarantee.

The wrong answer that passes every check

Here is the most ordinary question in a company: what is our average monthly revenue growth? An agent with the revenue table in front of it can answer in a second, and it will.

The trouble is that the question means two things:

• Average the month-over-month changes across the year.
• Measure growth inside each month, first day to last, and average that.

Both are standard. Both come off the same table. They give different numbers, and nothing in the question says which one was meant.

In the diagram below the two readings land on +4.7% and +1.9%. The figures are illustrative, picked to make the gap visible. What is not illustrative is that one table answers the same question two ways.

The agent picks one reading and computes. There is no error and no warning, and nothing in the reply records which definition was used. The number that comes back is clean and wrong in a way nobody can see: not wrong arithmetic, but the answer to a question that was never asked. Every check you have, it passes. A loud failure gets fixed the same day; this one gets pasted into a monthly report.

You write the rule down, and you write it well

The honest engineering response is to state the rule. This one is real, quoted from a production analytics agent's system prompt:

It is not buried. It sits in an Operating Protocol block at the very top of the prompt, under the heading "read before every answer", and it is restated in the rules section as the highest-priority rule for derived metrics. Capitals where they carry weight, no hedging. If prompt wording were the lever, this is what pulling it looks like.

It computed anyway

We did not guess this outcome; we measured it. We gave that exact prompt — rule at the top, in capitals — to two different frontier models and asked an ambiguous growth question. Both models went straight to the tools and returned a number. Neither stopped to ask which meaning we wanted. (The measurement comes from our production agent, on a real question of this shape; the revenue figures in the diagram stay illustrative.)

Look at where the rule sits at runtime and the result stops being surprising. The turn travels from the model, through a router, to the tools. The rule is text that arrived in the same context window as everything else. Nothing joins it to the path the turn actually takes.

Why do LLMs ignore system prompt instructions?

Because a system prompt is not a constraint on the system. It is one input to a function, and the model weighs it against the tool schema offering the call, the history showing what worked last turn, and its own strong prior that a helpful assistant answers the question it was asked. No interpreter reads the word MUST.

You did not write the rule badly, and you could not have written it much better. Against those other inputs, a sentence asking the model to stop is one voice among four, and nothing in the runtime can refuse the call on the model's behalf. That is why rewording keeps failing: the problem is structural, not stylistic.

Three places a rule can live

Only one of the three is a guarantee.

The middle tier is the one people mistake for safety, and the mistake is subtle because the check does work: raise inside the function and the run stops. But it only fires if the model calls that tool. Any route that avoids the tool avoids the check, and the unchecked answer comes out the other side looking like every other answer.

Move the rule onto the edge

So move it. In a graph, a router decides where each turn goes next. When the first turn of a conversation tries to compute an ambiguous derived metric, the router sends it to a clarify gate instead of to the tools.

That is one edge changed, on the same four boxes as before. The tools are not discouraged or deprioritised. From this state there is no path to them at all.

What does fail closed mean for an AI agent?

Fail closed means that when the safeguard fires, the run ends as a question to the user, never as an answer of unknown origin. A gate that only asks nicely is not a gate, so the sequence matters.

The gate tells the model to ask one plain question. The model tries to compute instead: blocked. It tries again: blocked. After two attempts the gate stops asking the model to ask, poses the clarifying question itself, and ends the turn. There is no state left from which that run reaches a guess.

On exactly this class of question, that gate flipped both frontier models to the correct answer with no domain-specific configuration: no list of metric names, no per-question tuning. To be exact about what was measured: the two blocks and the gate's own question are the router's behaviour; the flip was measured on a different question of the same class, not on the revenue example the diagrams carry.

A guarantee that annoys people has a short life

The gate stays narrow on purpose. A question like "top ten customers by contract value" has one reading, so it never trips the gate: the router sends it to the tools like any other turn. Ambiguity is the trigger, not the topic.

That scope is drawn in code, not in prose. The gate covers every tool that computes a metric, and it skips the support tool, because the support tool computes no numbers.

The width of that scope decides whether the guarantee survives. A gate that questions people about questions that were never ambiguous becomes a daily annoyance, and within weeks someone tired of it will remove the gate. Keep it narrow and it only fires when the ambiguity is real.

The rule that was followed and still failed

Ignoring a rule is not the interesting failure. Here is the interesting one, found in that agent's staging smoke test. Another rule said: when you cannot answer from the data, call the support tool. The model did not refuse. It replied:

Read the words and the rule was followed. Read the trace and no tool was called. The words are compliant; the calls are empty. A prompt rule does not only get ignored. It gets routed around, in a form that survives a human reading the transcript.

Keep the prompt

None of this makes prompts useless. In that same system the rule is still in the prompt, and it was strengthened rather than dropped, so the words and the structure say the same thing. A model told plainly to clarify does clarify more often, and on the turns nobody audits, that is the whole of your quality control.

What changes is the division of labour. The prompt sets the default. The edge sets the guarantee. Here is the same question with one thing added: the definition now travels attached to it. Month over month, averaged, and then the number. Nothing about the model changed. What changed is that the turn could not reach a number until the ambiguity was resolved.

Where should a system prompt rule live?

In the prompt, when losing the rule costs you little. On a graph edge, when it costs you trust. The prompt is how your agent behaves well on the turns nobody audits — and most turns are turns nobody audits. The structure is what holds when the phrasing is strange, the context is long, or somebody swapped in a cheaper model on a Friday afternoon.

So the next time your agent does the thing you told it not to do, the first question is not how to word the rule. It is where the rule lives. Take the rule you care about most and ask: what, in the runtime, would stop the call if the model ignored it? If the answer is "the prompt says not to", you have a default. If correctness depends on the model choosing well, it is not a guarantee.

FAQ

What is the difference between a system prompt and a guardrail?

A system prompt shapes behaviour; a guardrail enforces it. The prompt is text the model weighs together with everything else in its context, so it can be outweighed. A guardrail sits outside the conversation, in tool code or on the graph, where the model cannot argue with it. Use the prompt for tone and defaults, and a guardrail for any rule you would not accept seeing broken.

How do you force an LLM agent to ask a clarifying question before it answers?

Put the requirement on the path, not in the prompt. In a graph framework, route ambiguous first turns to a gate that has no edge to the tools, and let the gate ask the question itself when the model will not. Frameworks call this pattern an interrupt, or human-in-the-loop. Asking in the prompt raises how often the model clarifies; only the structure makes it certain.

Are LLM guardrails enough to stop agent actions?

Tool-level guardrails protect only the honest path. A check inside a tool fires when the tool is called, so any run that avoids the tool avoids the check — and the unchecked answer comes out looking like every other answer. A guarantee needs to sit on the path itself: a graph edge that closes the route before the call is made.

Do firmer words like MUST make a model comply?

Firmer words help, and they cannot bind. Plain, firm instructions raise the rate at which a model complies, which is why the rule stays in the prompt. But a word has no mechanism to stop a tool call, so MUST in capitals is still a request. The prompt sets the default; only structure can set a guarantee.

References

• A Closer Look at System Prompt Robustness (Mu et al., 2025) — arxiv.org
• Human-in-the-loop, LangChain documentation — docs.langchain.com
• Guardrails, OpenAI Agents SDK documentation — openai.github.io
• Why Large Language Models Skip Instructions — unite.ai