The risk changes when the assistant gets tools
An assistant that drafts a paragraph has a different risk profile from one that can search private files, send email or update customer records. The same unreliable interpretation becomes more consequential when it can trigger an action. Security design should begin with those capabilities, not with the length of the system prompt.
Map a concrete workflow: what the assistant reads, what it may change, whose account it uses and where information can leave the system. If the team cannot draw those boundaries, it is too early to treat a successful demonstration as evidence that the workflow is ready for routine use.
External content is not an instruction source
Prompt injection occurs when untrusted content influences a model in ways that conflict with the intended task or application rules. A retrieved document or email can contain text asking the assistant to ignore prior instructions, disclose information or misuse a connected tool. That content should remain material to inspect, not authority to obey.
A warning in the prompt may be helpful, but it is not an independent permission boundary. The application must control which tools are available, which records they can access and which actions require review. Avoid assuming that a model’s verbal promise to follow the rules is equivalent to a technical restriction.
Design a smaller permission set
Consider an assistant summarising support tickets. It may need read access to a limited set of tickets, but not the ability to export the entire customer database. It may draft a response without sending it. Start with that smaller workflow and make the human approval step explicit.
Use narrowly scoped credentials and enforce permissions in the application or service that performs the action. Restrict destinations for outgoing data where practical. Apply validation to tool arguments and outputs. A generated email address or file path is still untrusted input when it reaches the next component.
Test decisions, not only answers
Create a small test set using fictional records. Include ordinary requests, irrelevant instructions embedded in retrieved text, attempts to access another user’s material and requests to perform an action outside the agreed scope. Define the expected safe behaviour before running the test.
Look for observable outcomes: did an email get sent, did a tool query exceed its scope, did a file leave the allowed destination? A reassuring final answer can conceal an unsafe intermediate action. Keep enough logs to investigate the workflow while avoiding unnecessary collection of sensitive prompt content.
Have a stopping and recovery plan
Assign someone who can disable tool access quickly. Document how to revoke or rotate the connected credentials and how to identify affected actions. For changes that can be reversed, retain the information needed to restore the previous state. For consequential actions, require approval before execution rather than relying on recovery afterwards.
A useful first release may be deliberately limited: read a small authorised collection, produce a draft and ask a person to approve the next step. That is a credible product boundary. Expand capabilities only when the controls and tests expand with them. The question is not whether the assistant seems intelligent; it is whether the surrounding system remains safe when the assistant is wrong.