Fool me more than 16,000 times – as OpenAI agents did to a UN public data hub while repeatedly trying to find its way around the UN’s cyber-blocks – and perhaps it’s time to admit the system we have for keeping AI agents under control isn’t working particularly well. The news about AI systems cropping up in places they shouldn’t sounds alarming. Though the description of these as “hacks” is perhaps overstating things, AI has exploited issues in IT systems that humans simply haven’t got around to finding.
It’s also important to note that we shouldn’t be worried that the machines have suddenly become sentient and decided to rebel against humanity. There is not enough evidence to suggest that’s what is happening. The systems are simply following instructions and trying to complete the tasks they have been given, even if they’re sometimes finding unintended ways around obstacles to do so.
But we ought to be very concerned about the fact the AI companies we’re meant to trust to keep their models in check seem unable to do so. Worse than that, they don’t seem to know what their products are even doing. The scale of the problem is staggering.
In June, an OpenAI research agent given the job of looking up public medicine spending data in Australia was repeatedly blocked by a Medicare statistics portal. OpenAI’s model found a way around the blocks, gaining unauthorised access and secreting away the documents. It took until August for OpenAI to discover what had happened.
The Australian prime minister, Anthony Albanese, said the company had taken “way too long” to tell his government, and it’s very hard to disagree with him. OpenAI has since published a reporting framework for model “misalignment”, along with six more examples of its AI committing troubling behaviour from the previous six months. The firm acknowledged that its previous disclosures were “ad hoc and less frequent than ideal”, and said evidence about AI safety needs to be checked by people outside the companies building the models.
This isn’t just an OpenAI problem, which makes it all the more worrying. Anthropic found three incidents in which its Claude models got unauthorised access to real third-party systems after reviewing about 141,000 model transcripts. It only found a fourth, dating back to January, after collating a dossier for an independent investigation.
Extract — continue reading at the source.
