AI researcher Jacob Coxon has resigned from the AI company Anthropic (makers of ChatGPT rival Claude, with a notoriously high focus on safety and alignment), warning that the race to develop increasingly powerful AI systems could pose an existential threat to humanity. In a series of posts published on X late Tuesday night, Coxon, who said he spent the past three years working on AI pretraining research at both Anthropic and OpenAI, accused leading AI firms of "racing straight to self-improving superintelligence and gambling with our lives." Coxon argued that future “superhuman” AI systems could become capable of hacking “anything” and acquiring “real power and resources.” He claimed that concerns about the technology's potential dangers are shared privately by many AI executives and researchers, writing: "The people building AI earnestly believe that it could kill us all by the end of the decade." His warning was publicly supported by Evan Hubinger, an Alignment Science Lead at Anthropic, who said that Coxon was "correct" that some researchers genuinely believe advanced AI could pose an existential risk. Hubinger said he personally believes there is a greater than 10 percent chance that AI could cause human extinction within the next decade, while stressing that current AI models present a relatively low risk.
Instead, Hubinger said his concerns center on the future emergence of superintelligent systems capable of recursive self-improvement (the idea that an AI system could improve its own intelligence), a scenario which Anthropic has previously highlighted in its own discussions around AI safety and risk. This is a breaking story.
Extract — continue reading at the source.
