What Is AI Security? Threats, Defenses, and Why It Matters

Why AI Security Is a Separate Discipline

Traditional software security assumes deterministic behavior: given the same input, a program produces the same output, and vulnerabilities are bugs in code that can be found and patched. AI applications break this assumption fundamentally. A language model is a probabilistic function that generates different outputs for the same input depending on sampling parameters, context window contents, and the specific weights produced by its training process. This means that security testing cannot rely on reproducing specific input-output pairs. An attack that works on one attempt may fail on the next, and a defense that blocks an attack today may miss a slightly rephrased version tomorrow.

The attack surface of an AI application is also qualitatively different from traditional software. In a conventional web application, the attack surface consists of input fields, API endpoints, authentication mechanisms, and data stores, all of which handle structured data according to explicit code logic. In an AI application, the attack surface includes everything the model reads as context: system prompts, user messages, retrieved documents, conversation history, tool definitions, memory stores, and any other text that enters the model's context window. All of this text is interpreted by the model as natural language, meaning that malicious instructions can be embedded anywhere the model reads data, not just in fields explicitly designed for user input.

The consequence is that AI security requires specialized knowledge beyond what traditional security professionals typically possess. A penetration tester skilled at finding SQL injection and cross-site scripting may not know how to craft multi-turn prompt injection chains, test for indirect prompt injection through RAG knowledge bases, or evaluate whether a model's tool access permissions create privilege escalation paths. Conversely, an AI researcher studying adversarial robustness may not understand the practical implications of their findings for production deployments. Effective AI security bridges both disciplines.

The Seven Categories of AI Security Threats

AI security threats fall into seven broad categories, each targeting a different component of the AI application stack. Understanding all seven is essential because most production AI applications are vulnerable to threats across multiple categories, and attackers will target whatever is weakest.

Prompt injection manipulates the model's behavior by embedding instructions in user input or retrieved content. Direct injection targets the user input field; indirect injection embeds instructions in documents, web pages, or data that the model processes as context. This is the most frequently exploited AI vulnerability because it requires no technical sophistication, just the ability to write a convincing instruction in natural language. OWASP has ranked it as the number one LLM security risk since 2023.

Jailbreaking bypasses the model's safety training to elicit content the model was aligned to refuse. Unlike prompt injection (which targets application-level instructions), jailbreaking targets model-level alignment, attempting to undo the reinforcement learning from human feedback (RLHF) and other safety training that prevents the model from generating harmful content. Jailbreak techniques include role-playing, hypothetical framing, adversarial token sequences, and multi-turn social engineering.

Data poisoning corrupts the information that the AI relies on for grounding. In RAG systems, this means inserting false or malicious documents into the knowledge base. In fine-tuning scenarios, this means contaminating the training data with adversarial examples that create backdoor behaviors in the resulting model. Data poisoning is difficult to detect because the AI presents poisoned information with the same confidence as legitimate information.

Model extraction and theft attempts to reconstruct a proprietary model's capabilities by systematically querying it and using the responses to train a clone. This threat applies primarily to API-based model providers, where an attacker with API access can submit thousands of carefully crafted queries designed to maximize the information extracted from each response. Model extraction violates intellectual property, undermines competitive advantages, and creates uncontrolled copies of the model that lack the original's safety training and monitoring.

Supply chain attacks compromise components in the AI stack: model files containing executable code, compromised training frameworks that modify model weights during fine-tuning, backdoored embedding models that degrade retrieval quality, or malicious dependencies in orchestration libraries. The open-source nature of the AI ecosystem creates a broad supply chain attack surface, because most AI applications depend on dozens of packages from public repositories with varying levels of security review.

Excessive agency occurs when an AI agent is granted more permissions, tools, or autonomy than its task requires. This is not an attack in itself, but it amplifies the impact of every other attack. An AI agent with read-only access to one database can leak information if compromised; an AI agent with read-write access to multiple databases, email sending capability, and file system access can cause catastrophic damage under the same compromise. Excessive agency transforms a contained security incident into an unrestricted one.

Memory and state attacks target the persistent state that AI systems maintain across interactions. Conversation histories, user preference stores, persistent memories, and cached responses all influence the model's future behavior. Corrupting these stores creates persistent backdoors that affect every subsequent interaction, unlike prompt injection which only affects the current conversation. Memory attacks are especially dangerous because they can be introduced through normal-looking interactions that the system stores without recognizing them as adversarial.

Defense Strategies That Work

Effective AI security follows the defense-in-depth principle: multiple overlapping layers of protection so that no single point of failure compromises the system. The layers specific to AI applications include:

Input classification runs every user input through a classifier trained to detect prompt injection, jailbreak attempts, and other adversarial patterns before the input reaches the model. The classifier should be a separate, smaller model specifically fine-tuned for this task, not the main application model. Input classifiers catch roughly 85-95% of known injection patterns when properly trained, but they cannot catch novel attacks, which is why additional layers are necessary.

System prompt hardening reduces the model's susceptibility to injection by structuring the system prompt to resist override attempts. Techniques include explicit instruction hierarchy (stating that system instructions always take precedence over user input), boundary markers that separate system instructions from user content, and response format constraints that limit the model's output to specific structures. These are soft defenses, the model may still violate them under sufficient adversarial pressure, but they significantly raise the bar for successful attacks.

Output validation inspects the model's response before it reaches the user. Output validators check for information leakage (system prompt content, internal URLs, API keys), harmful content, hallucinated claims, PII, and responses that deviate from the expected format or scope. Output validation is the safety net for attacks that bypass input classification: even if a prompt injection succeeds in manipulating the model's behavior, the output validator can block the harmful response before the user sees it.

Tool permission scoping implements the principle of least privilege for agent tool access. Each tool available to the agent should have explicit permission boundaries: which data it can access, which operations it can perform, which parameters are required versus optional, and what rate limits apply. Permission scoping should be enforced in code at the tool execution layer, not just requested in the system prompt, because the system prompt is the exact thing that prompt injection attacks target.

Continuous monitoring detects attacks that bypass active defenses by identifying anomalous patterns in the system's behavior. Monitoring should track injection detection rates, tool call patterns, response characteristics, memory writes, and retrieval accuracy over time. Sudden changes in any of these metrics can indicate an active attack, and gradual changes can indicate slow data poisoning or model degradation.

AI Security and Compliance Frameworks

Regulatory frameworks increasingly require specific AI security controls. The EU AI Act, which entered full enforcement in 2025 with additional provisions taking effect through 2026, classifies AI systems by risk level and mandates security measures proportional to that classification. High-risk AI systems (those used in employment, credit scoring, law enforcement, and other domains with significant individual impact) must implement risk management systems, data governance controls, technical documentation, transparency measures, human oversight mechanisms, and accuracy, robustness, and cybersecurity requirements. The cybersecurity requirement specifically mandates that high-risk AI systems are resilient against attempts to alter their intended purpose through adversarial manipulation, including the input validation and output filtering controls described above.

SOC 2 compliance for AI systems requires demonstrable security controls over data handling, access management, and system integrity. AI-specific controls that map to SOC 2 criteria include: tool access controls with audit logging (maps to the Security principle), memory isolation and data partitioning (maps to the Confidentiality principle), input and output validation (maps to the Processing Integrity principle), and comprehensive interaction logging (maps to the Availability and Monitoring criteria). Organizations pursuing SOC 2 certification for AI products need to document these controls and demonstrate their effectiveness through testing.

HIPAA covered entities using AI to process protected health information (PHI) must ensure that AI systems cannot inadvertently disclose PHI through model outputs, tool calls, or memory leakage. This requires PII/PHI detection guardrails on both input (preventing PHI from being sent to third-party model APIs) and output (preventing PHI from appearing in model responses), strict user isolation in memory systems, and audit trails that track every interaction involving health data. GDPR's right to erasure applies to AI memory systems: if a user requests deletion of their data, all stored memories, conversation histories, and cached responses associated with that user must be deletable.

Industry-specific regulations add additional requirements. Financial services firms subject to FINRA and SEC oversight need auditability of AI-generated financial advice. Government agencies under FedRAMP need AI systems deployed in authorized cloud environments with specific access controls. Each regulatory context adds constraints that shape the specific AI security controls an organization must implement, but the foundational layers (input validation, output filtering, access control, monitoring, and audit logging) are universal across compliance frameworks.

Where AI Security Intersects with Related Disciplines

AI security overlaps with several related fields but is distinct from each. AI guardrails focus on output quality and safety, ensuring the model does not generate harmful, inaccurate, or off-topic content. Security focuses on defending against deliberate attacks rather than accidental misbehavior. Both use similar technical mechanisms (input classification, output filtering), but their threat models differ: guardrails assume benign users who might trigger unintended behavior, while security assumes adversarial actors who are actively trying to exploit the system.

LLM evaluation measures model quality, accuracy, and safety through systematic testing. Security testing (red teaming, adversarial testing) is a subset of evaluation focused specifically on the model's resistance to attack. Evaluation tells you whether the model works correctly; adversarial testing tells you whether it can be made to work incorrectly on purpose.

Enterprise AI memory governance addresses access control, compliance, and audit trails for AI memory systems. Security addresses the threat of memory corruption and poisoning. Both are necessary for a complete memory protection strategy: governance ensures authorized users can access the right data, while security ensures unauthorized users and adversarial inputs cannot corrupt the data that authorized users depend on.

AI security also connects directly to function calling and tool use, because agent capabilities are the primary mechanism through which successful attacks translate into real-world damage. A prompt injection that only affects the model's text output is an information disclosure risk; a prompt injection that triggers tool calls is an arbitrary action risk. Securing tool access is where AI security and traditional application security converge most closely.