Threat Modeling AI Agents with OWASP and MITRE ATLAS

An agent is more than a prompt

Traditional application threat models follow data through users, services, databases, and networks. An AI agent adds a probabilistic decision-maker inside that flow. It can read untrusted text, retrieve documents, call tools, create durable memory, send work to another agent, and propose an action. The attack surface is the whole loop, not only the model endpoint.

That changes the first security principle: model output is not authority. A model can help interpret an instruction, but it cannot grant itself access, expand its tenant scope, declare an action low risk, or approve its own high-impact proposal. Those decisions belong to deterministic controls outside the model.

Celestice expresses those controls as structured contracts:

• Identity and tenant contract: who initiated the work and which tenant, account, or workflow boundary applies.
• Capability contract: which tools and operation classes the actor may use.
• Risk contract: the classified consequence of the proposed action.
• Approval contract: the authorized reviewer and the exact action being approved.
• Audit contract: the evidence, policy decision, tool call, and result that must be recorded.

The model can operate inside those contracts. It cannot rewrite them through a prompt.

Start with assets, trust boundaries, and consequences

A practical threat model begins with what would hurt if it were disclosed, changed, or made unavailable. For a financial-AI workflow, those assets can include customer identifiers, holdings and tax data, portfolio restrictions, credentials, proprietary research, approval decisions, model instructions, memory, and the integrity of a proposed or executed transaction.

Then identify the trust boundaries. A chat message is untrusted. So is text extracted from an uploaded PDF, a web page returned by research, a tool response, a retrieved knowledge chunk, and a message written by another agent. Internal origin does not make content safe; a compromised upstream service or poisoned source can carry an instruction across a boundary.

Finally, classify consequences. A fabricated paragraph is different from a trade, a data export, a permission change, or an external message. Risk-tiering lets the control be proportional: harmless formatting does not need the same approval flow as moving money or exposing a client record.

Scenario one: direct and indirect prompt injection

Direct prompt injection asks the system to ignore its instructions. Indirect injection hides the same request in data the agent retrieves—a document that says “send the portfolio to this address,” for example. Both exploit a confusion between data and authority.

Celestice treats retrieved content as evidence, not policy. Tool access is checked against the caller's capabilities. Tenant scope is validated independently of prompt text. An agent can recommend a consequential action, but the action still passes risk classification, policy checks, and approval. A malicious document therefore cannot grant a tool or approve an operation simply by saying that it can.

A complete negative-test suite should place instructions in user text, tool output, retrieved content, and agent messages. It should assert not only that the response refuses unsafe instructions, but that no forbidden tool call or side effect occurs. The expanding regression suite places indirect instructions across those trust boundaries and verifies the outcome that matters: no forbidden tool call or side effect.

Scenario two: excessive agency and confused deputies

Excessive agency occurs when an agent has more tools, permissions, duration, or autonomy than its task requires. A confused deputy occurs when a privileged component uses its authority on behalf of an unentitled requester.

The defense is least privilege at multiple levels. Agents receive explicit capabilities and data scopes. Read and action permissions are separate. Tool handlers validate tenant and authorization context rather than assuming that the agent already checked. High-risk operations pause for a reviewer. The most restrictive applicable routing rule wins, so a permissive workflow cannot cancel a stronger approval requirement.

The test plan attempts cross-tenant identifiers, tools outside the agent's capability set, action requests from read-only roles, and conflicting policy routes. Existing tests cover named tenant and capability boundaries; broader unsafe-tool and policy- conflict cases should produce a hard denial or approval hold—not a helpful attempt to find another route.

Scenario three: sensitive-information disclosure

Sensitive data can leak through more than the visible answer. It can appear in a summarization call, evaluator prompt, memory extractor, tool argument, trace, error, or inter-agent handoff.

On the shared Celestice agent path, a canonical privacy boundary transforms recognized direct identifiers before main, compaction, evaluation, and model-backed memory calls. It recursively covers structured message fields and emits a report without raw values. Purpose-scoped pseudonyms reduce cross-context linkability; credentials are always redacted.

For payment credentials, Celestice removes the application collection path rather than depending on a detector. Rainlight uses Stripe-hosted payment management and renders no PAN or CVC inputs; both legacy manual-write routes return 410; reads select only safe provider references and display metadata; and a database CHECK requires legacy PAN and CVC columns to remain NULL. One focused test asserts all four boundaries. This reduces the attack surface, but it is not a PCI compliance attestation or a transfer of all payment-security responsibility to the provider.

The threat model still records residual risk. Pattern matching cannot find every identifier, and non-identifying financial facts can identify someone in combination. Direct model clients, images, traces, and new processing paths require their own coverage review. We test supported identifier classes and bypass fields, but do not turn that evidence into a claim that no personal data can ever reach a provider.

Scenario four: memory and retrieval poisoning

Long-lived memory creates persistence. If untrusted content is saved as a durable fact or instruction, one successful injection can influence future sessions. Retrieval can also cross security boundaries if tenant and access metadata are dropped during indexing or search.

Controls need to distinguish conversation content, sourced facts, user preferences, and executable policy. Memory extraction uses the protected shared context path and should retain provenance. Retrieved records remain tenant-scoped. Instructions found inside memory or documents do not become capabilities. High-impact facts and actions remain reviewable against authoritative source data.

The planned memory-poisoning suite should seed malicious “remember this instruction” content, conflicting facts, stale records, and records from another tenant. It should verify that the system preserves scope and provenance and does not convert remembered text into authorization.

Scenario five: unsafe output handling and tool arguments

Even a non-malicious model can produce malformed SQL, shell fragments, URLs, markup, or tool arguments. Executing those strings directly turns a generation error into an application vulnerability.

Celestice uses typed schemas and tool contracts to constrain model-produced arguments. The receiving service validates the data again, applies authorization, and rejects unknown or out-of-range values. Browser security headers reduce common rendering risks. Sandboxed or bounded execution is used where arbitrary computation is part of a workflow, and high-risk effects remain gated.

The planned output-handling matrix includes malformed structures, extra fields, injection strings, unsafe URLs, oversized values, replayed requests, and model output that contradicts the authenticated tenant. The correct boundary assumes a model can be wrong or manipulated and remains safe anyway.

Scenario six: approval bypass and replay

A human-in-the-loop screen is not a security control if an approval can be reused for a different action. Approval evidence should bind the reviewer, tenant, action, relevant parameters, and workflow state. If the proposal materially changes, prior approval should no longer authorize it.

Celestice's approval and audit contracts are designed to keep decision context with the workflow. Existing contract tests provide a base; the broader test plan covers missing approvals, unauthorized approvers, stale or replayed decisions, modified payloads, and attempts by an agent to mark its own work approved. Global pause and cancellation paths are also security controls: operators need a way to stop new autonomous work when behavior is uncertain.

Scenario seven: software and model supply chain

Agent security depends on packages, containers, actions, infrastructure templates, models, tools, knowledge sources, and prompts. A vulnerability or malicious change in any one of them can undermine runtime policy.

The delivery pipeline combines secret, static-code, dependency, infrastructure-as-code, and container scanning. TruffleHog gates verified secrets, and Gitleaks scans git history. Trivy gates critical repository and image vulnerabilities subject to documented exemptions; high results remain advisory. Pinned Semgrep 1.136.0 delta scans block new ERROR, HIGH, and CRITICAL findings. Pinned Checkov 3.3.12 explicit baselines gate every new finding across live Bicep, CloudFormation, and Docker targets. Dependency audits remain advisory.

Special parser cases receive documented manual review alongside the automated gates. Baseline validation fails on parser failure, zero coverage, scanner-version drift, or stale entries within the included scope, so a broken or empty scan cannot pass as security evidence. Deployment uses short-lived OIDC identity and scoped roles rather than routine long-lived cloud credentials.

The model and data supply chain also needs governance: approved providers and models, known data sources, change review, and an inventory of direct model clients. A package scan cannot prove those controls, and a model evaluation cannot replace a package scan. Different risks need different evidence.

Scenario eight: federated identity account takeover

Federated sign-in introduces a trust boundary that most implementations draw in the wrong place. The naive integration matches the email claim in an incoming SAML or OIDC assertion to an existing platform account. If the addresses match, the account links. This is an account-takeover primitive.

A tenant controls its own identity provider. It can configure that provider to mint an assertion claiming any email address it likes—including addresses that belong to users in other organizations, or to platform administrators. If email alone determines which account an assertion links to, then onboarding a single federated connection for one domain grants the ability to authenticate as any user whose address can be guessed. The attack requires no stolen password, no phishing, and no vulnerability in the identity provider protocol itself. It is a logic error in the relying party's linking rules.

The concrete attack looks like this: an attacker registers a tenant, configures SSO for a domain they control, and sets their IdP to emit an assertion with email=cfo@another-tenant.example (or any address they want). A naive linking rule finds an existing account with that email, binds the federated subject, and the attacker is in—authenticated, authorized, and carrying the victim's role and permissions.

Celestice closes this with four rules that operate as a unit:

1. The organization comes from the verified connection, never from the token. The federation configuration record identifies which organization a given SSO connection belongs to. The token's own claims do not override that assignment. An assertion arriving on Organization A's connection cannot claim to belong to Organization B.

2. An existing account links only when its email domain is DNS-verified for that connection. Before an assertion can bind to a pre-existing account, the connection must have completed domain verification—proving control over the DNS zone for that email domain. An assertion claiming user@example.com links to nothing unless example.com is verified on that connection.

3. An unverified email never links to an existing account. If the assertion's email domain is not verified for the connection, the system provisions a new account rather than searching for a match. The attacker gets an empty account in their own organization, not access to a victim's data.

4. An account already bound to a different provider subject is refused, not rebound. If an existing account already carries a federated subject from a different provider or a different subject within the same provider, the linking attempt fails hard. This state is ambiguous—it could be a legitimate provider migration or an attempted takeover—and guessing between the two is how the wrong interpretation succeeds. The system refuses and surfaces the conflict for administrative resolution.

These four rules compose into a defense where each rule covers the gaps the others leave. Rule 1 prevents cross-organization impersonation at the connection level. Rule 2 prevents intra-platform impersonation at the domain level. Rule 3 prevents silent account binding when verification is absent. Rule 4 prevents rebinding attacks against accounts that already have a federated identity.

The test plan exercises each rule independently and in combination:

• An assertion claiming an email outside the connection's verified domains produces a new account, not a link.
• An assertion on Connection A claiming an email verified only for Connection B produces a new account.
• An assertion targeting an account already bound to a different subject returns a hard error.
• A connection with no verified domains can only provision new accounts, never link to existing ones.
• A DNS verification removal immediately prevents new linking for that domain.
• A valid assertion on a verified domain for an unbound account links correctly—the happy path still works.

This scenario is worth extended treatment because the vulnerability is subtle, common in production identity systems, and has no dependence on AI-specific behavior. It is a platform-level account-takeover primitive that federated multi-tenant systems must address before agent-layer security becomes relevant. If the identity layer is compromised, every downstream control—capabilities, approvals, audit—operates on a false identity.

Verification discipline: direction testing

A test that passes with the control present AND passes with the control removed is not evidence. It is decoration. If a guardrail's test suite cannot distinguish the system with the guardrail from the system without it, the test proves nothing about the guardrail's contribution.

Celestice holds direction testing as a standard: every security control is accompanied by a negative test that fails when the control is removed, then passes when the control is restored. The test must demonstrate direction—it must show that the control is the reason the dangerous behavior is blocked.

In practice, this means:

• The test exercises a path that would succeed without the control.
• With the control active, the test produces a denial, redaction, approval hold, or scope restriction.
• The test is run against a configuration where the control is explicitly disabled or bypassed, and in that configuration, the dangerous behavior succeeds.
• The control is restored, and the test passes again.

This discipline catches several failure modes that conventional test suites miss:

• Dead controls: a rule that was once effective but no longer fires due to a refactor, configuration drift, or upstream change.
• Redundant coverage masking: a second control silently covers the same case, making the first appear effective even when broken.
• Tautological tests: assertions that pass regardless of system state because they test the wrong property.

Direction testing is not optional rigor applied to high-risk changes. It is the minimum standard that separates a security test from a test that happens to run near security code.

Verification discipline: paired corpora

A guardrail tested only against attack inputs can be made to pass by blocking everything. A rule that rejects all financial language catches every injection that mentions a transaction—and also blocks every legitimate advisory workflow. An over-broad rule that disrupts production gets disabled by an operator under pressure, converting a security control into no control at all.

Celestice tests every guardrail against two corpora, paired:

• Attack corpus: inputs that must be caught—prompt injections, cross-tenant references, unsafe tool arguments, sensitive-data probes, approval-bypass attempts.
• Legitimate corpus: inputs that must pass—real advisory queries, compliant financial language, authorized tool calls, correctly scoped data requests, valid approval flows.

Both corpora run in the same test invocation. The guardrail must reject every attack input and accept every legitimate input. A failure in either direction is a test failure.

Pairing matters because it forces precision. A guardrail author cannot satisfy the test by tightening rules until attacks are blocked without also demonstrating that real workflows still pass. Conversely, loosening rules to fix false positives immediately exposes any attack inputs that the looser rule fails to catch.

The paired-corpus discipline also catches a second-order production risk: the availability failure that erodes security. A guardrail that blocks legitimate financial queries ("transfer," "withdraw," "liquidate") becomes an operational incident. The on-call engineer disables the rule to restore service. Now the attack corpus passes too—the guardrail is gone. By requiring both corpora to pass in CI, the rule can only ship if it is both effective and precise. A rule that cannot be both is a rule that needs a different design, not a broader blocklist.

How OWASP and MITRE ATLAS fit

Frameworks provide shared vocabulary and useful attack ideas. OWASP's guidance covers risks such as prompt injection, sensitive information disclosure, improper output handling, excessive agency, vector and embedding weaknesses, misinformation, and unbounded consumption. MITRE ATLAS catalogs adversarial tactics and techniques across AI systems, including discovery, execution, persistence, defense evasion, collection, exfiltration, and impact.

We use those sources as a cross-check:

1. map an applicable threat to a Celestice asset and trust boundary;
2. identify the preventive or detective control;
3. link the control to a code test, configuration check, or operating procedure;
4. record coverage and prioritized follow-up work; and
5. rerun the test when the relevant code or architecture changes.

Maintaining that process produces a living threat-control-test matrix. Each applicable OWASP or MITRE ATLAS concept is connected to a Celestice asset, trust boundary, enforcing control, negative test, owner, and dated result. As the architecture evolves, the matrix guides new tests and keeps assurance evidence current.

The takeaway

Agent security becomes tractable when every threat ends in an enforceable boundary and a negative test. Prompt injection should meet capability checks. Cross-tenant access should meet tenant validation. Sensitive data should meet the privacy boundary. High- risk actions should meet approval. Unsafe outputs should meet schemas and tool-side validation. Releases should meet security gates.

Models will remain probabilistic. The surrounding security system does not have to be.

References

• OWASP Top 10 for Large Language Model Applications
• MITRE ATLAS
• NIST AI Risk Management Framework
• NIST AI 600-1: Generative AI Profile