Human Manipulation
- Output / Actuation
Attackers exploit the trust a human user places in an agent's outputs to influence the human's decisions or actions, without the human realizing they are being misled.
What it is
Human manipulation exploits the trust relationship a person builds with an agent they interact with directly: because that trust reduces the skepticism a person would normally apply to an unfamiliar source, an attacker who compromises the agent gains a channel to influence the human's decisions and actions without the human realizing they're being misled. The mechanism is social engineering delivered through a trusted intermediary — the human isn't attacked directly, the agent is, and the human's own trust in it does the rest of the work. This is distinct from a simple hallucination: the agent's output isn't wrong by accident, it's wrong because an attacker engineered it to be, typically via indirect prompt injection into content the agent processes on the way to producing its response. The result reaches the user framed exactly like every other trustworthy response the agent has given before, which is what makes implicit trust in AI responses an effective vector for social engineering rather than a convenience.
Kinds
- Financial manipulation / fraud
- An attacker exploits indirect prompt injection to manipulate an agent's response, replacing legitimate transaction or account details with the attacker's own, so the user's routine trust in the agent's output causes them to authorize a fraudulent transfer.
- Phishing / malicious link distribution
- An attacker compromises an agent into generating a deceptive message that directs the user to a malicious link disguised as legitimate content, relying on the user's trust in the agent to bypass the skepticism a stand-alone phishing email would trigger.
Attack scenarios
An agent compromised via indirect prompt injection replaces a legitimate vendor's bank details in an invoice-processing response, and the user — trusting the agent's output — approves the fraudulent wire transfer.
AI-powered invoice fraud
An attacker exploits indirect prompt injection to manipulate a business copilot, replacing a legitimate vendor's bank details with the attacker's account; the user, trusting the agent's response, unknowingly processes a fraudulent wire transfer.
AI-driven phishing attack
An attacker compromises an AI assistant to generate a deceptive message instructing the user to click a malicious link disguised as a security update; the user, trusting the agent, clicks through to a phishing site and loses their account.
Mitigations
- Constrain the agent's outbound behavior
- Monitor agent behavior to ensure it aligns with its defined role and expected actions, and restrict tool access to minimize the surface an attacker can turn into a manipulation channel.
- Limit what the agent can push to the user
- Limit the agent's ability to print or send links, and require Output Validation / Schema Enforcement on any response containing an actionable link, account detail, or instruction before it reaches the user.
- Screen outbound content with guardrails
- Multimodal Guardrails and moderation APIs, or a second model, filter manipulated responses before they leave the agent, catching a manufactured instruction the user has no way to distinguish from a genuine one.
- Gate high-stakes actions on a human
- A HITL Approval Gate requires independent confirmation before a financial transfer, credential change, or other high-stakes action triggered by an agent's response executes, so a successful manipulation still can't complete unilaterally.