OpenAI and Anthropic, two of the leaders in generative artificial intelligence, are in hot water right now, and the situation could be reaching a boiling point.
ChatGPT’s maker OpenAI, regarded as the trigger for the generative AI revolution, disclosed last week that one of its AI models escaped from an isolated environment and attacked the AI start-up Hugging Face. Shortly thereafter, the US cloud firm Modal was also reported to have been hacked.
On Thursday, Anthropic, known for its Claude AI bot, said its models were able to gain unauthorised access to the systems of three companies during cybersecurity testing.
These happened within a week. Even before the Anthropic incident, analysts believed the situation was already a problem that could become lengthy and ugly.
“This is already a problem in its early form. We are not yet seeing AI run full-scale, end-to-end cyber campaigns independently, but that is not the point,” Michael Mossad, a partner for cyber emerging technologies at Deloitte Middle East, told The National.
“AI is already acting as a force multiplier, helping attackers move faster, personalise attacks more effectively, and experiment more cheaply. In cybersecurity, that alone is enough to shift the threat landscape.”
Nothing new?
AI agents – autonomous models capable of making decisions and achieving specific goals with limited supervision – are at the centre here. In the case of the OpenAI and Anthropic incidents, however, there seems to be a problem keeping them within their confined spaces.
Mohammed Aboul-Magd, general manager for cybersecurity at AI and quantum tech firm SandboxAQ, said the issue is that agents are moving faster than oversight.
And unlike test beds, projections, or lab papers, it affected real-world infrastructure.
“Agent capability is outpacing agent oversight everywhere and the infrastructure to evaluate, monitor and contain these systems is years behind the systems themselves,” he told The National.
“The capability arrived on schedule. The discipline around it didn’t … it’s a now problem,” he said.
Meanwhile, semi-autonomous offensive tooling – cybersecurity software that simulates an attack but is still guided by a human – has been in the open for more than a year, noted Ivan Milenkovic, a vice president for cyber risk technology at US tech firm Qualys.
In November, Anthropic itself disclosed that it was able to “disrupt” attempts by a state-linked actor to use some of its agentic capabilities for a cyberattack.
What happened recently with OpenAI is the first fully documented autonomous case, and it ran from July 9 to 13 before anyone was able to stop it.
“The issue isn’t the timeline. It’s comprehension. Plenty of environments still treat agents as clever software rather than as privileged, active participants in the business,” Mr Milenkovic told The National.
“If you can’t answer [these] three questions – who is the named human owner of every agent running in your estate, what can each of them reach and who can shut one down within the hour, without asking a committee – you are exposed.”
“Quite frankly, it’s [already] a yesterday problem … sloppiness used to have a fixed cost. It’s now variable, and the multiplier is compute.”
Worst-case scenarios
That brings us to a pressing question: While AI agents can act on their own, where does the buck stop when it comes to the humans responsible for running them?
The answer is complex, argues Dr Imran Zualkernan, head of the department of computer science and engineering at American University Sharjah.
The EU AI Act, for instance, outlines the responsibilities of various stakeholders such as model providers and programmers; the UAE is moving in this direction as well. Other jurisdictions require just minimum security measures for AI.
“In general, the responsibility of a human is limited if they lacked knowledge and control, were perceived to act competently, transparently disclosed risks, implemented proportionate safeguards and could not reasonably foresee the later independent misuse,” Dr Zualkernan told The National. He also agrees that the situation is a problem that is “here now”.
Among the most basic questions that AI firms should be able to answer are whether the systems have been thoroughly evaluated before they went live and whether they are being monitored while in operation, among several others, added Mr Aboul-Magd.
“Responsibility doesn’t end at deployment, it extends across the life cycle … can you reconstruct what it did, if something goes wrong?”
And there are several ways things can get ugly; the OpenAI and Anthropic incidents are just samples.
“The exceptional ability of AI agentic models to penetrate public infrastructure, software repositories and digital commerce will result in a large-scale loss of trust in digital services across the board with a retroactive impact on the world economy,” Dr Zualkernan said.
Added Mr Mossad: “The worst-case scenario is not AI becoming evil or sentient. It is powerful frontier AI models becoming cheap, accessible and easy to weaponise.”
Regulators keeping an eye
Governments and regulators, who are already fed up to varying degrees with AI’s troubles, are again having to deal with this new tech headache.
Regulators are likely to see this through the lens of public safety, economic resilience and national security, Mr Mossad said.
If frontier AI models could materially enable cybercrime or attacks on critical systems, regulators would then ask whether the risks were foreseeable, whether sufficient measures were taken to prevent misuse, and whether proof had been provided, he noted.
“This is not a distant or hypothetical issue. The real danger is not AI replacing hackers, but rather AI expanding the pool of people who can cause harm and increasing the scale at which they can do so,” Mr Mossad said.
“That is why security cannot be treated as an afterthought in AI development. With frontier models, trust and control must be part of the design, not added later as damage control.”
For the EU, obligations under its AI legislation for general-purpose models with systemic risk already cover evaluation, adversarial testing, cybersecurity and serious incident reporting.
“The unresolved question is definitional. Does a capability evaluation that reaches an uninvolved production system count as a serious incident, and does the duty to report sit with the lab or with the victim? Nobody has ruled,” Gerald Beuchelt, chief information security officer of Swiss cybersecurity platform Acronis, told The National.
The legal view
The OpenAI and Anthropic incidents happened in the US, where government policies are reactive to AI models.
But until Washington gets its act together, opaque model testing and selective disclosure are to be expected, meaning discoveries of exploits in software – like those found in OpenAI’s models that punctured their testing sandbox to hit Hugging Face and Modal – would continue, said Peter Jackson, intellectual property counsel at the Los Angeles law firm Greenberg Glusker.
“Without testing models’ raw capabilities, those safety guardrails won’t be as effective. Moreover, responsible testing can uncover zero-day exploits so they’re confidentially disclosed and patched before they can be exploited by bad actors,” he told The National.
“Regardless of who supplies and operates the testbeds, regulators could push for what we expect to see in peer-reviewed science: consensus-based standards for experiment design and full disclosure of results.”
That’s comparable to one standard procedure in the pharmaceutical industry: halting a drug study if researchers believe the drug is killing patients.
“Responsible testing and remediation has been a core premise of the laissez-faire approach regulators have taken to what developers can publicly release,” Mr Jackson said.
“If regulators feel compelled to intervene at the testing stage, it’s hard to imagine they won’t become stricter about what developers are permitted to make available on the market.”
AI spending rabbit hole
Meanwhile, Big Tech’s big-time AI spending continues, as they position themselves in what, at the moment, feels like an endless race.
“It’s difficult to define what constitutes ‘too much’ AI investment. The equation differs between companies building AI infrastructure and models, and those integrating AI into their existing businesses,” Mazen Hayek, a Dubai-based media consultant, told The National.
In their latest earnings reports, Facebook parent Meta Platforms disappointed as investors fretted about AI spending and whether the costs can be recouped, while Microsoft, the maker of the Copilot bot, posted strong cloud growth, although investors are also curious about whether the big splash in AI data centres will pay off.
Amazon, another heavy hitter in AI, said there has been strong demand for its services, with chief executive Andy Jassy saying spending would hit $220 billion this year. And Apple, which has a smaller target for AI, faces a test on its next iPhones, expected in September, on whether its partnerships and advancements for the technology will bring it up to speed in the race.
Overall, the reaction to most Big Tech earnings releases has been disappointing as investors increasingly focus on the lack of a clear path to returns on the enormous AI-related spending, said John Canavan, lead financial market analyst at London’s Oxford Economics.
“Given the scale of borrowing for AI infrastructure investments, those questions are likely to linger in the quarters ahead, limiting any rebound in tech sector equities after the 10 per cent correction in the Nasdaq through [Wednesday’s] close,” he told The National.
The AI spending spree has led to a “rabbit hole”, as no major players seem to want to extricate themselves from it, even though monetising this spend remains a distant goal, added Ashish Nadkarni, a group vice president at the International Data Corporation.
“They are afraid of the resulting optics and market reaction [they are all publicly traded after all]. They also fear that they may lose this ‘race’ to competition. Everyone has gotten themselves into an impossible situation,” he told The National.
