AI risk discourse has largely focused on hallucinations, bias, privacy and explainability, but a new threat is approaching: the manipulation of autonomous agents by fraudsters.The latest generation of AI agents interacts with customers, coordinates workflows and performs other tasks with limited human interaction. Businesses and other organizations enjoy lower costs, productivity increases and 24-hour operations, while consumers appreciate AI replacing the time and effort they would spend on tedious tasks.
But alongside these benefits, new attack surfaces and liability gaps are emerging. What happens when an AI agent becomes the victim of a scam?
The Growing Risk of AI Manipulation
Fraud prevention and scam models usually focus on protecting humans from deception. Organizations train employees to recognize phishing emails, verify payment instructions and challenge suspicious requests: Is it really the CEO calling? AI agents, however, are being entrusted with increasing levels of authority without the instinctive skepticism that experienced humans develop over time.
As businesses and consumers deploy thousands of autonomous and semi-autonomous agents, we need to ask, “Can AI agents be socially engineered?”
The answer is yes.
Traditional fraud relies on manipulating human psychology. Criminals exploit trust, urgency, deference to authority, fear and social pressure to scam victims. AI agents don’t experience emotions, but that doesn’t make them immune to manipulation. Instead, they can be exploited through their decision logic, training, objectives and permissions. Attackers can embed malicious instructions inside emails, documents, websites or other content that an AI agent processes to influence its reasoning, without any direct interaction from the attacker.
As we give agents more independence to browse websites, read documents and perform tasks on behalf of human users, the attack surface widens. AI agents can be scammed through malicious content hidden on web pages and induced to ignore their original instructions, performing attacker-directed actions instead. In agentic environments, these actions could include data disclosure, unauthorized tool usage or covert communication with external systems. For organizations deploying AI-powered procurement, finance, customer service or operations agents, this represents the machine equivalent of phishing.
Malicious instructions can also be planted in an AI agent's long-term memory through seemingly normal interactions. Once stored, this “poisoned memory” can influence future decisions and recommendations long after the original attack. In this scenario, a fraudster may only need to deceive an agent once if the malicious information becomes part of the agent's trusted memory, compromising future decisions.
Updating Fraud Controls to Include AI Agents
As fraud evolves to target AI agents more often, it’s vital that fraud controls adapt as well, covering agents just like they do human employees. Organizations must assume that AI agents will be targeted. Criminals use analogous techniques: prompt injection (phishing), executive impersonation against agents (business email compromise), synthetic supplier data (fake invoices), instruction manipulation (social engineering), memory poisoning (insider manipulation) and agent credential compromise (account takeover). Just as organizations train employees not to fall for phishing emails or fraudulent payment requests, they will need to do the same for their agentic workforce.
When an AI agent is deceived, the scale and speed of the resulting loss may be dramatically greater. A human employee may process tens of invoices per day; an autonomous procurement agent could process thousands.
Although an AI application that can only generate text poses relatively limited risk, an agent directly connected to enterprise systems that can approve transactions, access sensitive data or perform other important tasks represents a completely different category of exposure.
This rare combination of autonomy and authority creates an attractive opportunity for fraudsters.
Legal and Regulatory Implications of Deceiving AI
What happens when an AI agent is deceived? Traditional Strong Customer Authentication (SCA) and card-scheme dispute rules assume a human is involved at the point of decision; however, when an agent initiates a purchase or payment, "Who authorized this?" becomes ambiguous. No consensus liability model exists yet among consumer, merchant, payment service provider and the AI vendor.
AI systems themselves aren’t legal entities. They can’t be sued, fined, or held accountable in any practical sense. Responsibility generally falls on the humans and organizations that authorized, deployed and supervised them.
Customers, regulators, auditors and shareholders are unlikely to accept "our AI made the decision" as a sufficient explanation. In many situations, the organization operating the AI agent may remain responsible for financial losses and regulatory breaches, including data protection failures.
As AI autonomy increases, liability discussions will increasingly resemble those already used in model risk management, fraud prevention, cybersecurity and operational risk disciplines.
The most important question is who pays when AI agents are deceived. You may delegate decisions to AI agents, but you can’t delegate accountability.
Building An Operational Plan to Protect AI Agents
Fraud professionals are well placed to meet the challenge. Fraud teams have long worked with the fundamental principle that every control will eventually be targeted by attackers. This same mindset should now be applied to AI agents.
Rather than assuming agents will always behave correctly, organizations should expect agents to be attacked, manipulated, misled, impersonated, poisoned and exploited.
How can organizations create a resilient Agentic AI operating model and protect themselves?
1.
Limit Agent Authority - AI agents should operate according to the principle of least privilege. The greater the authority granted, the stronger the controls required. Granting agents excessive privileges is a dangerous mistake to make.
2.
Introduce Risk-Based Human Oversight - Recognize that not every AI action requires review; however, high-risk transactions should always trigger human involvement without exception.
3.
Authenticate Requests, Not Just Identities - Traditional authentication focuses on verifying the originator of a request. Agentic systems must now also assess whether the request itself makes sense. The question to answer is "Does this action fit expected behavior?" not just "Is this an authorized user?"
4.
Maintain Complete Audit Trails - When an incident occurs, organizations must be able to explain what the agent did and why. To do this, they will need the data that influenced the decision, the permissions the agent used, and importantly, the controls that were in place. Comprehensive logging, explainability and auditability are becoming essential governance requirements. If organizations can’t reconstruct an agent's decisions, then defending regulatory, legal or insurance claims becomes significantly harder.
5.
Apply Fraud Controls to AI Agents - Many existing fraud prevention techniques are highly relevant. AI agents should be monitored using controls similar to those applied to customers and employees, such as behavioral analysis, transaction monitoring, anomaly detection, velocity monitoring, peer-group analysis, and real-time fraud risk scoring.
In effect, organizations may need to become experts at detecting suspicious behavior from their own digital workforce.
The irony is that AI agents will increasingly become both the target of fraudsters and a key defense
against them. The same technologies that enable autonomous agents can also help identify suspicious activity, detect manipulation attempts and block fraudulent transactions in real time.
Those that apply the same level of scrutiny, controls, and monitoring to AI agents as they do to human employees will be far better positioned to manage the risks ahead.
