Here at NICE Actimize, we are “all-in on AI”: that is, we believe that not only is Artificial Intelligence here to stay, but if employed intelligently, that it creates incredible opportunities and will (like major innovations before it) increase productivity and efficiency. For these reasons, we believe that AI is to be embraced, rather than feared. We see AI as an enabler, not necessarily a replacer or a supplanter: in the areas in which we operate, human input, oversight and judgment will, ultimately, remain necessary.
That said, there are definitely areas of human activity where technology should be trusted without question; one such area is traffic collision avoidance in aviation. In 2002, two commercial aircraft collided in the skies over Switzerland. All 71 people on both aircraft died, including 52 children on a school trip from Russia. Both aircraft were fitted with the then-new technology called the Traffic Collision Avoidance System (TCAS). TCAS is fitted to all commercial aircraft around the world and each aircraft’s TCAS communicates with others nearby; if a potential conflict is identified, the TCAS on the two aircraft concerned works out a resolution (faster than could be achieved if the two aircraft had to coordinate with each other and air traffic control) and displays a resolution advisory (RA) in the cockpits. In this case, one flight followed the TCAS RA, while the other followed ground instructions. As a result of this crash, international rules were standardised to
require pilots to unquestioningly follow a TCAS RA, so as to resolve potential mid-air conflict situations. The technology decides what to do, and there is no human in the decision-making process; the humans in the loop – the pilots – don’t check the TCAS’s work, but just execute the TCAS RA. The trust placed in TCAS is absolute.
Hitherto, situations where technology should be followed without question have been the exception rather than the rule; the human has usually been seen as the ultimate decision-maker. However, reports suggest that as the adoption of AI has become widespread, society – including compliance staff – is increasingly both offloading its thinking to AI, and assuming the AI must be right, and this has financial regulators worried.
In this context, their concerns are that blind faith in AI output is creating new regulatory risks and potential points of failure in the financial regulatory system; as the FCA’s recent Mills Review into AI and the future of retail financial services put it:
“As [AI model] autonomy grows, the nature of regulatory risk changes. As AI moves from recommending to acting, and firms and consumers delegate more, risks shift from harm within a single firm towards system-wide harms.” The risks can differ depending on the degree of autonomy of the model and the respective roles of the human and AI in various situations. The Mills Review identified five roles the human plays, as we move across the AI autonomy spectrum:
- Role 1, Human as an operator: the human uses AI as a tool.
- Role 2, Human as a collaborator: the human and AI plan and act together.
- Role 3, Human as a consultant: AI compares options and recommends, while the human decides.
- Role 4, Human as an approver: AI prepares actions that the human authorises.
- Role 5, Human as an observer: AI acts continuously within agreed limits while the human monitors outcomes.
In the first three roles, the human is the ultimate decision-maker; in the fourth, the AI has a high degree of autonomy, but again, the ultimate decision is down to the human. In the fifth, however, the AI has autonomy to act, and the human is there merely to observe, and “monitor outcomes” – after the event.
On the face of it, the fifth role is also potentially the riskiest: that is, the AI can plan and execute on its own. We might argue that this has been going on for some time in the area of high-frequency or algorithm-driven trading: orders to trade are placed by a system, with human oversight only coming after the fact to check position risk or detect where the algo might’ve gone off the rails due to a programming bug. Most surveillance officers with more than a year or two of experience will likely have encountered such situations; I would suggest that those of us who have, can clearly attest to the potential risks this introduces to traded markets, and there are instances of large market moves having been caused by a trading algo going rogue.
What appears to be currently scaring regulators, however, is not just the risk of an algo passively ‘going rogue’ or having a bug, but effectively creating a sixth role, where the AI acts continuously on its own, without human oversight (or worse, perhaps even actively hiding from human oversight): think of an autonomous AI-driven trading algo – whose only goal is to maximise profitability and has not been given any training in regulatory guardrails – actually planning and executing market abuse. Humans will certainly see the outcome of such actions, but only well after the event. This is not an unfounded fear: we are already seeing reports of AI models by themselves creating new AI agents and collaborating with other models to create an ‘agentic swarm’ to overwhelm a target, be that a utility, a system at a target company, or an organised market. These models and agents act entirely autonomously – there is no human in, on or near the loop.
But simply inserting a human somewhere in the loop does not necessarily create a circuit-breaker, nor does it address other, growing risks: one, where the putative human decision-maker in Roles 1 – 4 above suffers from ‘automation bias’; that is, they come to trust the automation too much. The human may employ AI many times each day, and because they find over time that the AI is usually correct (or nearly correct), the human decreasingly questions the AI and, when errors do occur is decreasingly willing to question or challenge it – if the human even spots the error at all. The human in the loop becomes a mere rubber-stamper for the AI; as the Mills Review puts it, there is a risk of over-reliance or weak challenge.
The other risk which is surfacing is ‘cognitive offloading’ to AI, whereby humans get the tool to perform tasks, rather than do it themselves, and gradually lose the skill to perform the task. This phenomenon is known as ‘cognitive atrophy’: the human loses the necessary cognitive skills they’ve spent a career building up. Most parts of the human body work on a use-it-or-lose-it basis and the brain is no different, so if we don’t work to maintain cognitive skills, they are lost. Fortunately, just like atrophied muscles, with time and effort neural pathways can be rebuilt (just like going to the gym).
In both cases, not only is the human outsourcing their thinking to AI, but they are knowingly putting too much trust in systems which themselves carry ‘health warnings’ that they can make mistakes, and they are losing the ability to critically assess the tool’s output. As a result (given that AI tends to learn from itself and our reactions to it) that erroneous training becomes baked into the model, and the deviation from ‘correct’ grows over time. The human in Roles 1 - 4 above becomes pointless.
To its great credit, the FCA has been on this particular bandwagon for some time, requiring transparency and explainability: it is OK to have technical data science explanations, but outcomes must also be explainable in plain English. Similarly, the EU AI Act requires that operators of high-risk AI systems must conform to requirements around explainability and human supervision. Importantly, explainability implies that the human not only understands what is going on within the model, but is able to challenge the model, and does when necessary.
So as the industry goes ever more all-in on AI, I will conclude by repeating what I wrote on this topic last year: “trust, but verify”; in light of recent autonomous hacks by AI agents of government databases, this has never been more important. But you’ll need to maintain your cognitive skills in order to do that – which itself will mean that you will continue to be better-equipped to provide the necessary challenge. Wordle, anyone?
