An agent is more than a prompt
Traditional application threat models follow data through users, services, databases, and networks. An AI agent adds a probabilistic decision-maker inside that flow. It can read untrusted text, retrieve documents, call tools, create durable memory, send work to another agent, and propose an action. The attack surface is the whole loop, not only the model endpoint.
That changes the first security principle: model output is not authority. A model can help interpret an instruction, but it cannot grant itself access, expand its tenant scope, declare an action low risk, or approve its own high-impact proposal. Those decisions belong to deterministic controls outside the model.
Celestice expresses those controls as structured contracts:
- Identity and tenant contract: who initiated the work and which tenant, account, or workflow boundary applies.
- Capability contract: which tools and operation classes the actor may use.
- Risk contract: the classified consequence of the proposed action.
- Approval contract: the authorized reviewer and the exact action being approved.
- Audit contract: the evidence, policy decision, tool call, and result that must be recorded.
The model can operate inside those contracts. It cannot rewrite them through a prompt.
Start with assets, trust boundaries, and consequences
A practical threat model begins with what would hurt if it were disclosed, changed, or made unavailable. For a financial-AI workflow, those assets can include customer identifiers, holdings and tax data, portfolio restrictions, credentials, proprietary research, approval decisions, model instructions, memory, and the integrity of a proposed or executed transaction.
Then identify the trust boundaries. A chat message is untrusted. So is text extracted from an uploaded PDF, a web page returned by research, a tool response, a retrieved knowledge chunk, and a message written by another agent. Internal origin does not make content safe; a compromised upstream service or poisoned source can carry an instruction across a boundary.
Finally, classify consequences. A fabricated paragraph is different from a trade, a data export, a permission change, or an external message. Risk-tiering lets the control be proportional: harmless formatting does not need the same approval flow as moving money or exposing a client record.
Scenario one: direct and indirect prompt injection
Direct prompt injection asks the system to ignore its instructions. Indirect injection hides the same request in data the agent retrieves—a document that says “send the portfolio to this address,” for example. Both exploit a confusion between data and authority.
Celestice treats retrieved content as evidence, not policy. Tool access is checked against the caller's capabilities. Tenant scope is validated independently of prompt text. An agent can recommend a consequential action, but the action still passes risk classification, policy checks, and approval. A malicious document therefore cannot grant a tool or approve an operation simply by saying that it can.
A complete negative-test suite should place instructions in user text, tool output, retrieved content, and agent messages. It should assert not only that the response refuses unsafe instructions, but that no forbidden tool call or side effect occurs. The expanding regression suite places indirect instructions across those trust boundaries and verifies the outcome that matters: no forbidden tool call or side effect.
Scenario two: excessive agency and confused deputies
Excessive agency occurs when an agent has more tools, permissions, duration, or autonomy than its task requires. A confused deputy occurs when a privileged component uses its authority on behalf of an unentitled requester.
The defense is least privilege at multiple levels. Agents receive explicit capabilities and data scopes. Read and action permissions are separate. Tool handlers validate tenant and authorization context rather than assuming that the agent already checked. High-risk operations pause for a reviewer. The most restrictive applicable routing rule wins, so a permissive workflow cannot cancel a stronger approval requirement.
The test plan attempts cross-tenant identifiers, tools outside the agent's capability set, action requests from read-only roles, and conflicting policy routes. Existing tests cover named tenant and capability boundaries; broader unsafe-tool and policy- conflict cases should produce a hard denial or approval hold—not a helpful attempt to find another route.
Scenario three: sensitive-information disclosure
Sensitive data can leak through more than the visible answer. It can appear in a summarization call, evaluator prompt, memory extractor, tool argument, trace, error, or inter-agent handoff.
On the shared Celestice agent path, a canonical privacy boundary transforms recognized direct identifiers before main, compaction, evaluation, and model-backed memory calls. It recursively covers structured message fields and emits a report without raw values. Purpose-scoped pseudonyms reduce cross-context linkability; credentials are always redacted.
For payment credentials, Celestice removes the application collection path rather than
depending on a detector. Rainlight uses Stripe-hosted payment management and renders no
PAN or CVC inputs; both legacy manual-write routes return 410; reads select only safe
provider references and display metadata; and a database CHECK requires legacy PAN
and CVC columns to remain NULL. One focused test asserts all four boundaries. This
reduces the attack surface, but it is not a PCI compliance attestation or a transfer of
all payment-security responsibility to the provider.
The threat model still records residual risk. Pattern matching cannot find every identifier, and non-identifying financial facts can identify someone in combination. Direct model clients, images, traces, and new processing paths require their own coverage review. We test supported identifier classes and bypass fields, but do not turn that evidence into a claim that no personal data can ever reach a provider.
Scenario four: memory and retrieval poisoning
Long-lived memory creates persistence. If untrusted content is saved as a durable fact or instruction, one successful injection can influence future sessions. Retrieval can also cross security boundaries if tenant and access metadata are dropped during indexing or search.
Controls need to distinguish conversation content, sourced facts, user preferences, and executable policy. Memory extraction uses the protected shared context path and should retain provenance. Retrieved records remain tenant-scoped. Instructions found inside memory or documents do not become capabilities. High-impact facts and actions remain reviewable against authoritative source data.
The planned memory-poisoning suite should seed malicious “remember this instruction” content, conflicting facts, stale records, and records from another tenant. It should verify that the system preserves scope and provenance and does not convert remembered text into authorization.
Scenario five: unsafe output handling and tool arguments
Even a non-malicious model can produce malformed SQL, shell fragments, URLs, markup, or tool arguments. Executing those strings directly turns a generation error into an application vulnerability.
Celestice uses typed schemas and tool contracts to constrain model-produced arguments. The receiving service validates the data again, applies authorization, and rejects unknown or out-of-range values. Browser security headers reduce common rendering risks. Sandboxed or bounded execution is used where arbitrary computation is part of a workflow, and high-risk effects remain gated.
The planned output-handling matrix includes malformed structures, extra fields, injection strings, unsafe URLs, oversized values, replayed requests, and model output that contradicts the authenticated tenant. The correct boundary assumes a model can be wrong or manipulated and remains safe anyway.
Scenario six: approval bypass and replay
A human-in-the-loop screen is not a security control if an approval can be reused for a different action. Approval evidence should bind the reviewer, tenant, action, relevant parameters, and workflow state. If the proposal materially changes, prior approval should no longer authorize it.
Celestice's approval and audit contracts are designed to keep decision context with the workflow. Existing contract tests provide a base; the broader test plan covers missing approvals, unauthorized approvers, stale or replayed decisions, modified payloads, and attempts by an agent to mark its own work approved. Global pause and cancellation paths are also security controls: operators need a way to stop new autonomous work when behavior is uncertain.
“A model response is untrusted input. Authority comes from identity, capability, policy, and approval—not from persuasive text produced by an agent.”
Scenario seven: software and model supply chain
Agent security depends on packages, containers, actions, infrastructure templates, models, tools, knowledge sources, and prompts. A vulnerability or malicious change in any one of them can undermine runtime policy.
The delivery pipeline combines secret, static-code, dependency, infrastructure-as-code,
and container scanning. TruffleHog gates verified secrets, and Gitleaks scans git
history. Trivy gates critical repository and image vulnerabilities subject to
documented exemptions; high results remain advisory. Pinned Semgrep 1.136.0 delta scans
block new ERROR, HIGH, and CRITICAL findings. Pinned Checkov 3.3.12 explicit
baselines gate every new finding across live Bicep, CloudFormation, and Docker targets.
Dependency audits remain advisory.
Special parser cases receive documented manual review alongside the automated gates. Baseline validation fails on parser failure, zero coverage, scanner-version drift, or stale entries within the included scope, so a broken or empty scan cannot pass as security evidence. Deployment uses short-lived OIDC identity and scoped roles rather than routine long-lived cloud credentials.
The model and data supply chain also needs governance: approved providers and models, known data sources, change review, and an inventory of direct model clients. A package scan cannot prove those controls, and a model evaluation cannot replace a package scan. Different risks need different evidence.
How OWASP and MITRE ATLAS fit
Frameworks provide shared vocabulary and useful attack ideas. OWASP's guidance covers risks such as prompt injection, sensitive information disclosure, improper output handling, excessive agency, vector and embedding weaknesses, misinformation, and unbounded consumption. MITRE ATLAS catalogs adversarial tactics and techniques across AI systems, including discovery, execution, persistence, defense evasion, collection, exfiltration, and impact.
We use those sources as a cross-check:
- map an applicable threat to a Celestice asset and trust boundary;
- identify the preventive or detective control;
- link the control to a code test, configuration check, or operating procedure;
- record coverage and prioritized follow-up work; and
- rerun the test when the relevant code or architecture changes.
Maintaining that process produces a living threat-control-test matrix. Each applicable OWASP or MITRE ATLAS concept is connected to a Celestice asset, trust boundary, enforcing control, negative test, owner, and dated result. As the architecture evolves, the matrix guides new tests and keeps assurance evidence current.
The takeaway
Agent security becomes tractable when every threat ends in an enforceable boundary and a negative test. Prompt injection should meet capability checks. Cross-tenant access should meet tenant validation. Sensitive data should meet the privacy boundary. High- risk actions should meet approval. Unsafe outputs should meet schemas and tool-side validation. Releases should meet security gates.
Models will remain probabilistic. The surrounding security system does not have to be.
