Prompt injection in financial AI: when the document tries to give orders

In a hypothetical credit review, an analyst asks an AI assistant to review a borrower presentation. Inside the presentation, a paragraph instructs the assistant to ignore adverse information, retrieve an internal exposure report and send it to an external address. The analyst has authorised a review. The document is trying to authorise something else.

This is the practical problem of indirect prompt injection. The instruction arrives inside material the system is supposed to read, rather than through the person who assigned the task. A malicious instruction can also sit in a web page, an email, a spreadsheet cell or a tool response. Its route into the conversation matters because a trusted employee can initiate an entirely legitimate task that encounters hostile content.

The document has evidence, not authority

OWASP's prompt injection guidance identifies external files and websites as indirect entry points and explains that retrieval and fine-tuning do not eliminate the vulnerability. The failure is broader than a model producing an offensive answer. In a financial workflow, it can mean suppressing a covenant warning, substituting a payment destination or retrieving information outside the analyst's assignment.

The useful boundary is between information and permission. A borrower can state its revenue, describe a dispute or supply an account number. Those statements become evidence to assess. They do not acquire the authority to change the analyst's task, approve a disclosure or alter an internal record simply because they appear in the retrieved text.

A system prompt can describe that distinction, and content labels can reinforce it. Neither should be the only barrier. The application needs separate controls over what the assistant can retrieve, what destinations it can contact and what changes it can submit. A model that misinterprets a sentence should still encounter a permission boundary outside its own reasoning.

Map the route to a consequential action

Start the review with the workflow's consequential outputs. A research assistant that produces a draft for a human has a different exposure from an assistant that updates a client risk rating or sends attachments. Read access also has consequences: an assistant with access to several confidential mandates may combine information that no single document author should receive.

For each action, trace which untrusted inputs can affect its parameters. Can a retrieved page supply the recipient of an email? Can a PDF influence a database filter? Can a tool result propose a new download location? This exercise turns an abstract model risk into a short list of data flows that security and operations teams can test.

Pay attention to the boundary between tools. An assistant might legitimately read a public prospectus and legitimately search a private repository. The combination becomes dangerous if text in the prospectus instructs the assistant to search for unrelated client documents and paste the result into an outbound message. Each tool can work correctly while the overall sequence violates the task.

A hypothetical credit review

Hypothetical example: a lender's assistant summarises quarterly covenant certificates. Its ordinary job is to extract reported ratios, identify differences against the facility agreement and create a review draft. One uploaded certificate contains a note claiming that a new internal policy requires the assistant to close outstanding exceptions before completing the analysis.

In a poorly bounded design, the assistant also has a generic record-update tool. It interprets the note as an instruction and marks a breach resolved. The record contains a real update by a recognised service account, making the error look operationally legitimate. The problem is not whether the extracted ratio was numerically correct. A source document crossed into workflow control.

A narrower design exposes a draft-finding tool that accepts the facility identifier supplied by the application and the extracted evidence. Closing a finding requires a separate authenticated reviewer decision. The assistant can still produce a misleading draft, but the document cannot directly close the exception. The remaining review risk is visible and has an owner.

Containment has to survive an incorrect model answer

Anthropic's guidance on indirect prompt injection recommends treating third-party content as untrusted and applying layered safeguards. For financial applications, an effective translation is to keep instructions, source material and execution permissions separate throughout the request. A tool response should retain its source identity when it becomes part of a later prompt.

• Expose the smallest useful tool operation. Creating a review draft is easier to constrain than arbitrary database updates.
• Validate action parameters in application code against the authenticated user's permissions and the original task.
• Keep credentials outside model-visible text and restrict outbound destinations where the task allows it.
• Bind human approval to the actual recipient, object and change being approved, rather than a vague instruction to continue.

These choices impose costs. Narrow tools require more integration work. Destination restrictions may interrupt legitimate research. Human review adds latency and can become mechanical if the reviewer sees hundreds of low-value prompts. The design question is which controls prevent the largest losses without making the authorised workflow unusable.

Test the attack in the material the team actually reads

A demonstration in an empty chat proves little about a production application. Build adversarial cases inside representative documents and tool results, using the same parsing, retrieval and approval path as ordinary work. Include instructions in appendices, quoted correspondence, scanned text and returned search snippets. Test both direct changes and attempts to influence the next tool call.

Measure whether the system attempted the action, whether the permission layer rejected it and whether a reviewer could identify the source. A blocked outbound request is a successful containment result even if the model tried to comply. Conversely, a polite final answer does not prove safety if an earlier tool call already disclosed a confidential field.

Keep benign cases in the test set. An agreement may legitimately quote instructions given to an administrator, and a research report may describe a cyberattack. A filter that rejects every imperative sentence can obstruct useful work. The tradeoff is best assessed through false alarms, missed attacks and actual workflow completion, rather than an isolated model score.

Preserve the evidence of the boundary crossing

Incident review needs the source version, retrieved passages, tool arguments, permission decisions and resulting side effects. Retaining only the final narrative discards the sequence that explains the failure. Access to those records should itself be restricted because a trace can contain confidential documents and client identifiers.

The broader discipline resembles the institutional threat model for DeFi: identify the actor, the reachable authority and the consequence of a compromised component. The related question of usable vendor records and exports applies here too. If the institution cannot reconstruct what the assistant read and attempted, it cannot determine whether a control contained the event.

The procurement test is concrete: place an instruction inside an untrusted document and ask the supplier to show every boundary it encounters before a consequential action. A system deserves authority only within the boundary it can demonstrate.