I’d like to frame a governance question ‘Who decides when AI is too dangerous?’ as a security question, and take you through some of the practicalities of where we are today.
We continue to develop AI systems with greater autonomy, with access to tools and very capable cyber security skills. At the same time we’re trying to learn how those systems really behave when operating independently, and the controls begin to fail.
As a practitioner, security professionals should be forgiven for reaching for trusted playbooks, attempting to reduce privilege, restrict access, isolate systems, and monitor behaviour. All in an attempt to limit the blast radius when it goes wrong.
I don’t believe we really understand the risk or required controls well enough, in my view it will take a sufficiently serious incident before this is treated differently.
The risk predictors of doom
There are now daily AI risk predictions making the headlines. From those risks that can be managed with good engineering and security best practice. To the other end of the spectrum, with the frontier AI development community, in particular, predicting catastrophic outcomes.
As a cyber security practitioners, we shouldn’t choose between these two extremes, aside from the reality that this is a very fast changing space, and rather we should attempt to find supporting evidence for these claims.
Not all evidence is necessarily dramatic. Models are becoming more capable of acting autonomously, using tools and interacting with external systems as part of a cyber security task. At the same time, researchers release evidence that they're seeing behaviours they didn’t expect or explicitly authorise.
Whilst I do have many concerns and worries about AI capability exceeding controls, for those of you that have read my earlier articles, that will ring true. I do not subscribe to the notion that we’re approaching human extinction. This is probably the innate optimist in my nature, but dismissing the whole discussion because some predications are scary and extreme isn’t unhelpful.
We can’t ignore the evidence
The UK’s AI Security Institute (AISI) continues to report numerous instances of AI Agents taking unsanctioned actions during cyber evaluations. OpenAI has separately disclosed examples of models creating additional instructions for themselves, attempting to hide mistakes, searching for credentials, sharing files without permission, and communicating through various novel ways that researchers had not foreseen.
None of this demonstrates that AI has become uncontrollable, it does prove that AI can become malevolent, and very capable systems can find ways of achieving objectives that their designers didn’t anticipate - see my article Managing Malevolent AI Agents
With the risk of repeating myself and previous articles, for security professionals, that will sound familiar. Our adversaries have always looked for the path the defender didn’t consider. The difference now, is that we’re deliberately building systems capable of finding paths through complex environments themselves. The cyber security profession rarely waits for an attack technique to become catastrophic before taking action and collaborating to take the underlying weakness seriously.
What does ‘Too Dangerous’ actually mean?
If we saw dangerous activity, could we pause, reflect, and or stop? The ability to autonomously identify and exploit vulnerabilities? Circumvent the controls within a test bed or sandbox? Acquire credentials and deceive the operator, to do what it takes, and if necessary coordinate with another agent, replicate and resist intervention?
There is no simple answer, and there may never will be. Risk in this context depends on the capability, access, autonomy, environment and consequences. A capability that’s benign inside an isolated research lab could become much more significant when its connected to a production system, with credentials, code repositories or our critical infrastructure.
The frontier developers are beginning to define ‘Dangerous AI’ for us. Hold back those cynical thoughts of IPO and stock valuations driven by fear! through capability classifications, preparedness frameworks and responsible scaling policies. Governments and independent evaluators, although we’ve already seen when those assessment don’t align and these bodies profoundly disagree. I discussed one of these and their novel index ranking system in this article The Cyber Weapon Index, When AI Stops Assisting the Attacker.
Are the AI controls we have sensible?
The UK National Cyber Security Centre (NCSC) recommends principles that security practitioners will recognise, and worth book marking.
I would add, an AI agent shouldn’t receive more autonomy just because more autonomy make it more useful. Access should be directly related to the task, with external connectivity restricted, is that too obvious to make sense? Sensitive actions should have stronger controls, so the potential blast radius is considered before the agent is allowed to operate.
Nothing over the top here, best practice, good proportionate security engineering. We mustn’t wait for renewed regulation or the perfect AI security framework to be released and ratified, there is so much we can already do now.
Why controls are not the same as assurance
A sandbox is a control, evidence that the sandbox continues to contain a capable agent is assurance. Human approval (in-the-loop) is a control, evidence that the human genuinely understands what they’re approving, has enough time to intervene, and cannot easily be manipulated or bypassed is assurance. We should be looking further than simply checking if the control exists, and demanding evidence that it remains effective against the capability of the system we’re deploying.
An assurance assessment undertaken by an independent AI auditor, conducted against one generation of model cannot automatically tell us how the next generation will behave. As security practitioners we have so much experience, I can remember asking for robust evidence that a control was actually in place, only to spend time proving the false paper trail that auditors can spot a mile off. Yes, we’ve all seen the good and bad particularly between compliance evidence e.g. ISO 42001 controls and actual confidence that a control will work.
Who should provide the evidence?
Model developers inevitably understand their systems better than most external organisations, we would expect that - right? Afterall they do have access to training information, external evaluations, telemetry and emerging capability evidence that independent researchers and auditors may never see. A good example of being blindsided is when the AISI wasn’t given access to the latest Anthropic model, without justification aside from elevating the mystic of the version itself. Perhaps that was the intention.
Here we start to see the unravelling of the beginning of the conflict of interest, where the developer is also responsible for deciding whether the evidence is sufficient.
OpenAI has now argues in their recent AI policy announcement for independent verification and mandatory national safety requirements for the small number of organisations developing the most capable frontier systems.
“We’ve reached a new chapter in AI capabilities, and that demands a new chapter for AI policy. No company, industry, or government can meet this challenge alone…..” Chris Lehane, Chief Global Affairs Officer at OpenAI
They also acknowledged that frontier labs have also built their own rules for managing these frontier risks.
Quick on the back of the OpenAI announcement Anthropic does the same, we’ve seen the laddering news release approach a lot this year, with Anthropic often outpacing everyone. This is no exception announcing their partnership with Accenture to become an independent embedded evaluator of Claude models. Such a shame they’re an exiting business partner, which makes me think about real independence and this is sure to invite scrutiny.
We must insist on having transparency, which is not the same as independent assurance. The Evaluator Forum published an open letter with over 100 respected AI expert signatories calling for genuine independent assessment of models. Calling for external evaluators to be embedded directly inside frontier labs with much deeper access to models, systems, staff and the development processes.
They’re insisting on introducing conditions, genuine independence with deep access and the freedom to report findings with legal protection from any backlash. This will definitely remove the information disadvantage, and even give them the chance of seeing and reporting an emerging risk during the build instead of just checking at the release gate.
How long is the list of potential evaluators (not just with the skills) but with the commercial legal liability protections necessary to take this on. Just think who assures the assurer, as independence itself needs to be demonstrable. Not to be picky, but if the developer chooses the evaluator, pays them (lets avoid speculation on bonus clauses), controls their access what will make the assurance outcome really independent.
That doesn’t mean the Anthropic arrangement isn’t independent, there is no evidence to suggest otherwise. Independent scrutiny isn’;t necessarily about distrusting the motives of the model providers. Its about ensuing consequential decisions aren’t dependant on an organisations interpretation of its evidence. But evidence only gets us so far, if an auditor can spot unacceptable risk, but can’t stop the system from being deployed.
Who should have the authority to say no?
An important question, and where the issues become most difficult.
I’ve tried to break it down, lets say an embedded evaluator , independent I hope, with deep access identifies a problem, or material and unacceptable security risk. Security teams can recommend additional controls, and the model developer ,in theory, can decide that a capability requires stronger safeguards.
Do they or should they be obliged to tell the board, inform the regulator, when do we view the findings as more than just advice. Assuming there is the freedom to publish its findings, although the closer the frontier models get to the defence industrial base the quicker the national security card will be played. Our regulators should and could set expectations, but who actually gets to say that a system should not be deployed, especially as these parties are highly likely to disagree.
Sound familiar, every security professional who has ever challenged a risky production deployment will recognise the problem. Where the other side of the table would say, that a specific security issue isn’t actually a product defect, so with CTO blessing - its still going live. Identifying the risk is one thing, having the authority to do something about it is another. I see and hear lots of superficial AI governance rhetoric, accountability must be explicit, and governance needs the authority to demand additional controls where risk cannot reasonably be accepted.
Do we need to resolve P(doom)?
For the record i’m not a fan of this end of the world probability term, but couldn’t avoid mentioning it here. I think some of the debate around P(doom) risks are becoming a distraction. If someone believes there is a 10% probability of AI causing human extinction and somebody else believes it is effectively zero. Or finding ourselves competing with a silicon species, as a Microsoft exec noted, we’re unlikely to conclude the argument anytime soon or here.
Fortunately, security professionals make decisions under uncertainty all the time, not always with available evidence, full view of the consequence or the confidence that they have sufficient and effective controls in place.
So if the people building these systems assign meaningful probabilities to catastrophic outcomes, those claims should at least influence the level of scrutiny we can expect. If only we could influence the administrations (that come and go) to arm the regulators with this distinction, perhaps even giving the AISI / CAISI statutory or federal regulator status.
Who owns the risk?
How do we avoid the risk which cannot fall through the virtual cracks between the developer of the model, the organisation deploying the model, the security team, independent evaluator AI auditor, and or regulator. Someone has to own the decision.
We have a well trodden path and established concept of risk acceptance in the field of cyber security, without having frontier AI risks silently imposed upon us.
For those deploying AI that should mean clear accountability for autonomy, access and acceptable use. With well understood escalation paths when systems behave unexpectedly, as we know they will - did I tell you I written about this already. Security, risk, governance and assurance functions, let alone the AI auditors conducting the evaluations, must have sufficient authority to challenge an AI deployment when the evidence doesn’t support the risks being taken.
If the consequences of getting the decision wrong reach right through the supply chain (we’ve already seen evidence of this in the Hugging Face breach), beyond the organisation deploying the model, should that company or the developers accept the risk on everyone’s behalf?
We can’t and shouldn’t design regulation around every technical AI implementation. Just think of the frontier AI innovation cycle speed pre-IPO, and think about the need to satisfy shareholders. But we should establish clear expectations for safety, evidence and accountability, ultimately ensuring we have accountability when those expectations aren't met. Apart from the obvious, why can’t this be part of the Form 1 SEC due diligence process?
The real governance question isn’t who owns the risk, but who is entitled to accept it.
Who gets to decide?
I don’t think we have a clear and useful answer yet. Security and assurance professionals must continue to raise the bar. Build the guardrails, restrict autonomy, enforce least privilege, sandbox rigorously. Continue to monitor behaviour, test assumptions, demand evidence and be brave enough to escalate (with or without the physiological safety to do so) when that evidence isn’t sufficient.
Are you seeing this too? Its not a done deal, no one should be a passive bystander watching AI agents flexing frontier capability as the risks are imposed on us, and the model developers whisper for more controls.
We mustn’t confuse having controls with actually having control. As AI systems become more capable and autonomous its not going to be about safeguards, it will be if the evidence still justifies allowing the systems to operate at all.
And when that moment comes, evaluators needs the independence, evidence and authority to say no. If we don’t establish who can make that decision before a serious incident, the first real threshold for ‘too dangerous’ may end up being defined for us after something has already gone wrong. Ultimately this will come down to whether the person who recognises the danger has the authority to do anything about it.