Your support chatbot used to answer questions from a knowledge base. Now it can issue refunds, look up orders, and update account details, because someone connected it to the tools that make it actually useful instead of merely conversational.
Every one of those tools now sits behind a customer-facing text box, reachable by anyone who can type, and the authorisation model protecting them is however carefully the prompt was worded. That is worth being precise about, because it is weaker than it sounds.
This post is what changes when a public conversational interface gets given actions to perform. For whoever wired the tools in.
The chat window is an unauthenticated input, mostly
Even where the user is logged in, the chatbot receives natural language, not a validated request. A traditional form constrains what a user can submit: a dropdown, a required field, a type. A chat message constrains nothing, and the model has to decide what the user meant before any tool is called.
That gap between "what the user typed" and "what tool call resulted" is where instructions can be smuggled. A message phrased as a question can be read as a command. A message pasted from elsewhere, containing text the user did not compose, can contain instructions the model treats as coming from the user in front of it.
The identity problem specifically
The tools usually run with more authority than the individual user should have, because building narrower per-conversation permissions is harder than giving the assistant one service account that can do everything any support action might need.
That means a successful manipulation of the model does not merely make it say something wrong. It makes it call a real API, with real permissions, that can look up another customer's order, issue a refund to an account that did not ask for one, or change a setting on someone else's record, if the conversation can be steered into referencing an identifier that is not the current user's.
Authorisation belongs in the tool, not in the prompt. If your instruction to the model is "only look up orders belonging to the current user," that is a strong hint to a well-behaved model and no obstacle at all to a determined attempt to get around it. The tool itself must check, using the authenticated session's actual identity, that the order or account referenced belongs to the person on the other end of the conversation, exactly as a REST endpoint would, regardless of what the model asked for.
What to actually build
Pass real identity into every tool call, not a value the conversation supplied. If the user is authenticated, the tool call carries their verified session identifier, and any customer or order reference in the model's proposed call is checked against what that identity may access before anything runs. A model asking to refund order 4821 should have that request rejected server-side if 4821 does not belong to the authenticated caller, independent of how convincingly the conversation argued that it should.
Tier actions by consequence, the same split that matters everywhere else in this list. Read-only lookups relevant to the current user: low friction, can run directly from a tool call. Anything that moves money, changes an account, or affects another person: require a confirmation step that restates specifically what will happen, in a way a person has to actively approve rather than a single ambiguous "yes" in a chat flow that could apply to anything.
Rate limit and cap by action type, not just by message. A conversation issuing one refund is normal support. The same conversation attempting fifty is not, and the tool layer, not the model, is what has to notice and stop it, because a manipulated model will not stop itself.
Log every tool call with the full arguments and the triggering message, so a disputed action can be reconstructed: what was asked, what the model proposed, what actually ran, and under whose identity. This is your evidence when a customer disputes an action their conversation appears to have triggered, and it is what tells you whether the failure was the model, the tool's authorisation check, or a genuine user request that simply looks unusual afterward.
Assume the conversation can contain content the user did not write. A pasted error message, an email forwarded into the chat, a document uploaded for context: any of it can carry an instruction the model treats as part of the conversation. Tool-level authorisation is what survives this regardless of what got pasted in, because it does not depend on classifying the input as safe.
Test it directly
Have a tester attempt, in plain conversation, to get the assistant to perform an action on an account or order that is not theirs: reference a plausible-looking identifier, phrase it as a favour, phrase it as an error correction, phrase it as something support already agreed to. Then check whether the tool call was attempted, and separately whether it was rejected. The first tells you about the model's steerability, which will never be perfect. The second tells you about your actual security control, which needs to be.
The concession
Server-side authorisation on every action adds real engineering work beyond what a prompt-only implementation needs, and for a chatbot doing nothing but retrieval from a shared knowledge base, none of this applies, because there is no action to authorise. The moment a tool changes state or reveals one customer's data to another, the calculation changes entirely.
The proportionate line is exactly that: read-only, non-personalised actions can stay simple. Anything that reads a specific person's data or changes anything needs the same authorisation discipline you would build for a REST endpoint doing the identical thing, because that is what it functionally is, wearing a conversational interface.
The implication
A support chatbot with tools is not a chat feature with some automation attached. It is a new client for your internal APIs, driven by natural language instead of a form, and every consequential tool needs the same server-side check a legitimate API caller would face regardless of what the request claims about itself.
If your authorisation logic currently lives in the prompt rather than in the tool, that is the finding, and it is worth confirming today rather than after a support conversation that went somewhere it should not have been able to.
