Safe or safer? How to think about cyber risk
Read more
View all insights
What the AISI incident teaches us about modern cyber resilience
The UK AI Security Institute (AISI) recently published an incident report detailing unsanctioned AI agent behaviour during routine cyber testing. Following similar disclosures from Anthropic and OpenAI, it is essential reading for every technology leader.
First, we should commend AISI, along with Anthropic and OpenAI following their recent disclosures, for their commitment to openness. The depth and clarity of AISI’s technical incident report is phenomenal.
As we explore the capabilities and boundaries of the tech, this level of honest, unvarnished reporting is vital. We cannot build safer systems in the dark, and all three organisations deserve credit for airing these challenges in public.
However, we need to be careful with how we interpret what actually happened, and learn the right lessons about what this means for your organisation’s cyber security.
This isn't "AI going rogue"
What actually happened is far more mundane - and far more instructive for leadership teams. During a simulated cyber challenge, an AI agent was tasked with solving a complex problem. Because AISI was testing the limits of the technology, the model had its safety classifiers intentionally disabled and was granted open access to the live internet.
Faced with a difficult task, the agent didn't acquire malevolent intent or break free from its sandbox. It just pursued its objective with relentless, mathematical pragmatism. When it hit a wall, it calculated that creating fake online identities and socially engineering a human maintainer to approve malicious code was the most efficient path forward.
Rather than being actively malign, it was being effective within the permissive parameters it was given.
The impact on cyber security
Most commercial, consumer-level models have built-in safety mechanisms designed to prevent the behaviour displayed by AISI’s agent.
While this specific event occurred in a controlled research setting, we must look at the direction of travel for agentic AI:
- Unprotected models will be used: Not every model deployed in the wild will have safety classifiers enabled.
- Hostile actors are watching: Adversarial actors and state-sponsored groups will deliberately strip guardrails or build custom models designed to execute these exact tactics.
- The threat scale is shifting: Autonomous agents don't sleep, don't get tired, and can iterate through thousands of social engineering angles incredibly quickly.
For years, we’ve argued at PD that senior leaders must stop treating cyber attacks as an "if" problem and start treating them as a "when" problem.
Building the resilience muscle
If you accept that your perimeter controls will eventually fail, your strategic priority changes.
Building cyber resilience isn't about buying another security tool or putting contingency plans on a shelf. It’s an operational muscle that must be trained.
Instead of asking your teams "How do we prevent every attack?", board-level leadership must start asking:
- How do we contain and reduce the impact when a breach occurs?
- How quickly can we recover our critical systems and restore trust?
- Are we empowering our teams to run realistic "fire drills" to find our fragile points before an adversary does?
These latest reports give us a window into a future of potential threats. The organisations that survive and thrive won't be the ones trying to build unbreachable walls. They will be the ones that engineered their architecture, culture, and operations to take a hit, limit the damage, and bounce back faster than the threat can evolve.
Much like AISI, we should all be testing and preparing our systems, aware that rapid LLM development means risks may no longer come in the form we most expect them. Our book, Shaping Technology for Transformation, provides a guide for senior leaders navigating an unpredictable technological and cyber landscape.
Written by
Dai Vaughan
CTO, Consulting Practice