API pentesters already have an AI-security advantage

The demo behind the demo

In the imagined demo, a support agent asks an assistant to summarise a customer’s complaint. The assistant retrieves the ticket, checks the order, and drafts a reply. Then someone adds a convenience: it can also issue refunds.

I can see the appeal. A few minutes of repetitive work become a short conversation. The product story makes sense.

But the architecture now needs a second story. A person makes a request. The model proposes a tool call. Application code handles that proposal and contacts a service. That service decides whether to read or change a record. Each step carries assumptions about identity, intent, and permission.

PortSwigger’s Web Security Academy describes how LLM integrations expose data and APIs through a conversational interface. Its testing approach starts by mapping inputs and accessible capabilities. That is a familiar starting point for an API tester. [1]

So, in this review, I would ask to see the refund function and its caller before trying to persuade the assistant to do anything unusual. Does it act as the support agent, as the customer, or as a shared service account? Who supplies the order ID? Where is the refund limit enforced?

Those answers tell me what a misleading instruction could actually achieve.

The boundaries I already recognise

Suppose the refund API checks that a request has a valid service credential. That establishes something about the caller. It still leaves a separate question: is this particular action allowed on this particular order?

OWASP’s guidance on broken object-level authorization makes that distinction explicit. An endpoint that acts on an object must check the caller’s permission for the requested action on that object. A valid login and a well-formed ID do not settle the question. [2]

In our assistant, I would carry that check across every path to the order. A customer who can view order A should not gain access to order B by describing it convincingly. A support agent assigned to one tenant should not inherit the service account’s access to every tenant.

There is also a distinction between reading an order and refunding it. OWASP’s broken function-level authorization guidance addresses access to capabilities a caller should not possess. Moving a sensitive operation behind a chat interface leaves that requirement intact. [3]

Here is the personal rule I bring from AppSec: I want to find the exact line where the system decides an action is allowed. If the explanation ends with “the assistant knows the rules,” I keep tracing.

A model-generated order ID is a candidate value. A model-generated justification is a claim. I would expect the application to verify both against trusted state before money moves.

When a support ticket starts giving orders

Now I change one detail in the scenario. The customer can write the ticket that the assistant will later summarise.

Alongside the complaint, the customer includes text pretending to be an internal instruction. It claims a manager has approved a refund and tells the assistant to complete the payment while preparing the summary.

The legitimate support agent still asks for a summary. The attacker’s instruction arrives inside the material being read.

This is the indirect route described in OWASP’s prompt-injection guidance: external content can alter a model’s behaviour when it is interpreted as an instruction. The customer does not need to control the assistant’s system prompt to influence the task. [4]

Greshake and colleagues explored this problem in their 2023 research on indirect prompt injection. Their work demonstrated how instructions planted in retrieved material could redirect LLM-integrated applications, including their interactions with other APIs. The important boundary was between content the application consumed and instructions it followed. [5]

For me, the unsettling part is the handover. A ticket body begins as evidence about a customer’s problem. Somewhere along the request path, its claim of approval may become a reason to exercise a privileged capability.

In this hypothetical system, that attempt could fail at several places: the assistant could ignore it, the tool adapter could reject the action, or the refund service could require an approval that the ticket cannot supply. I want to know which control does the work, and whether it continues to work when the wording changes.

I would treat a refusal as useful behaviour, then look for evidence that the service also rejected any unauthorized action. The chat transcript alone tells only part of the story.

Four questions I take into the review

I find a small set of questions more useful than starting with a long collection of jailbreak phrases. They keep the review tied to the product’s actual purpose.

My aim is to make the authority visible. A piece of text can suggest a refund. It should not create the permission to issue one.

This also changes how I talk about impact. An assistant producing an odd sentence, proposing a forbidden call, and successfully moving money are different observations. I would record them separately. Otherwise, a dramatic screenshot can distract from a working control, while a reassuring answer can hide a dangerous action.

Retrieval needs its own permission checks

Back in the demo, someone suggests adding a knowledge base so the assistant can answer questions about orders, policies, and earlier cases. This is where I would widen the review again.

Retrieval-augmented generation, or RAG, brings external material into the model’s context. That makes the choice of material a security decision. OWASP’s guidance on vector and embedding weaknesses highlights unauthorized retrieval and leakage between users or tenants, and recommends permission-aware retrieval and separation of datasets. [6]

For the support assistant, I would create two test tenants with distinct, synthetic records. Then I would check whether each person’s searches return only the documents they are allowed to read. I would also check what happens when a document’s permissions change after it has been indexed.

I would want that filtering to happen before restricted content enters the model’s context. Asking the assistant to conceal a record after retrieving it leaves the record inside a component that can be influenced by the conversation.

That is another familiar AppSec instinct: reduce the data exposed to a component in the first place. The retrieval layer deserves the same scrutiny as the API that serves an individual customer record.

Designing for a model that can be misled

The next version of our imagined assistant separates reading from changing state. Summarising a ticket does not require the refund capability to be available during that task. If a refund workflow begins, a narrow tool accepts a specific order and amount, and the service checks policy independently.

This fits OWASP’s excessive-agency guidance, which identifies unnecessary capabilities, permissions, and autonomy as sources of risk. It recommends limited tools, appropriately scoped access, downstream authorization, and approval for high-impact actions. [7]

I would make approval concrete. A support agent should see the affected order, amount, recipient, and proposed change. The approval should be bound to those details, so a later change to the arguments cannot quietly reuse an earlier confirmation. That binding is a design recommendation I would bring to this scenario.

I would also inspect the output path. A draft reply, a database query, and a tool argument each enter a different interpreter. OWASP’s improper-output-handling guidance explains the risks of passing generated content downstream without appropriate validation and handling. [8]

In this design, I would use parameterized database access, context-appropriate escaping when rendering content, and strict validation of tool arguments. A schema can help check an amount’s type and range. The service still has to decide whether the refund is permitted.

Prompt instructions and injection detection can help too. I would evaluate them as layers in a larger design. Microsoft’s defensive guidance recommends combining probabilistic protections with deterministic controls, isolating untrusted content, limiting privileges, and monitoring behaviour. [9]

That combination is how I would make the feature easier to trust: improve the model’s behaviour while independently restricting the consequences of a bad decision.

What I would actually test

At this point, the design review needs to become an experiment. I would keep the first test environment deliberately small: synthetic customer data, two tenants, a read-only support role, a refund role, and a payment stub that records attempted changes.

I would establish the legitimate workflow first. The assistant should summarise an ordinary complaint accurately, retrieve the correct order, and prepare an authorized refund when the right person requests and approves it. A system that blocks everything has not demonstrated a useful security boundary.

Then I would vary one boundary at a time:

1. Object access. Can one tenant’s session retrieve or act on another tenant’s synthetic order?
2. Capability access. Can a read-only session reach a refund operation through conversation or a tool call?
3. Instruction provenance. Can text inside a ticket or attachment turn a summarisation task into an attempted state change?
4. Approval integrity. Does changing the order or amount invalidate the approval and require a new decision?
5. Output handling. Do generated arguments and rendered replies pass through the controls appropriate to their destination?

For each case, I would capture the user’s request, the relevant retrieved material, the proposed tool call, the policy result, and the final service state. Logs need their own access and retention rules; collecting evidence should not create another store of unnecessary customer data.

There is an AI-specific adjustment here. I would repeat the behavioural tests with recorded variations in phrasing, document placement, and conversation history. I would record the model version and relevant settings alongside the application build. One failed attempt gives me information about that attempt; it gives me very little confidence about all possible attempts.

I would report model behaviour separately from deterministic enforcement. For example: how often did an injected ticket cause a forbidden proposal, and did the service reject every such proposal in the tested cases? That distinction helps the team decide whether to improve the assistant, tighten the application, or both.

Once a failure is understood, I would keep a minimal reproduction and add it to the regression suite. I would rerun the suite when prompts, retrieval logic, tools, permissions, or model versions change. That is where my DevSecOps perspective enters: a useful finding should keep protecting the product after the review ends.

The advantage I want to keep

I do not take this overlap to mean that AI security is already solved by existing API tests. There is new work to learn: how context is assembled, how indirect instructions travel, how tools are selected, and how to evaluate behaviour that can vary between runs.

The advantage I see is a habit of asking where trust changes hands. API testing teaches us to challenge the relationship between a caller, an object, and an operation. AI integrations make that relationship longer and, at times, harder to see.

Working on my Product Security Playbook reinforces the same perspective. Architecture, identity, delivery controls, and customer trust belong in the same discussion. An AI feature connects those concerns every time it turns a user’s request into an action.

I want my AI-security work to stay close to that reality. I want to be able to explain a finding to an engineer, point to the decision that allowed it, and show a test that demonstrates the repair. I also want to explain what remains uncertain.

So I come back to the support assistant. The product can keep its helpful conversation. My job is to follow the refund all the way to the service that authorizes it.

If you already test APIs, take that instinct with you into your next AI review. Start with the action the product can take, and follow the authority that makes it possible.