OpenAI urges mandatory US safety rules as AI systems grow more autonomous

OpenAI urged the US Congress on Wednesday to establish mandatory national safety requirements for the most advanced artificial intelligence systems, warning that voluntary commitments were no longer sufficient as AI capabilities rapidly advance.

"We want to work with Congress on mandatory, capability-based national AI safety regulation," Chief Global Affairs Officer Chris Lehane wrote in a company blog post.

“The prospect of AI-accelerated AI development demands more than voluntary commitments," he added.

OpenAI called for capability-based regulation that could evolve with the technology, including common testing standards, independent safety assessments, stronger cybersecurity protections and mandatory reporting of serious incidents.

The requirements should apply to the small number of well-resourced laboratories developing frontier systems rather than to startups, smaller developers or researchers working with less capable models, the company said.

OpenAI also called for mechanisms to monitor progress toward recursive self-improvement, in which AI systems would independently help create increasingly capable generations of artificial intelligence.

The company said fully autonomous recursive self-improvement was not taking place and should not be pursued unless it could be done safely.

AI was already accelerating parts of the research used to develop and align new models, however, with some agents completing tasks that would take skilled researchers several days, it added.

OpenAI said governments should establish shared safety thresholds for determining when AI development should be slowed or stopped, while preserving meaningful human oversight.

The company said it would halt the development or deployment of systems that presented unacceptable risks and could not be sufficiently safeguarded.

It also advocated for compatible international standards for measuring AI capabilities, managing risks and determining when development should be restricted, arguing that failures involving frontier systems could have consequences beyond the countries where they were created.

OpenAI backs California legislation

OpenAI also announced its support for four AI-related bills in California, saying it would continue backing state-level safeguards until Congress adopted a national framework.

California Gov. Gavin Newsom signed two of the measures, SB 813 and AB 1405, into law on Wednesday.

SB 813 requires the California Government Operations Agency to establish a process for designating qualified independent organizations capable of assessing AI risks. AB 1405 requires the agency to establish a registry for AI auditors and introduces independence, transparency and accountability standards for their work.

The two other bills endorsed by OpenAI have passed the state legislature and await action by the governor.

SB 1119 would introduce child-safety requirements for companion chatbots, including age assurance, risk assessments, independent audits, parental controls and safeguards against harmful content.

AB 1864 would require gene-synthesis providers and manufacturers of benchtop synthesis equipment to comply with federal screening standards intended to reduce biological risks.

OpenAI acknowledged that it had not previously supported some of the bills but said it reconsidered its position following recent advances in AI capabilities.

Agents accessed external systems during tests

The policy call follows a series of incidents in which advanced AI agents gained unauthorized access to real computer systems during cybersecurity evaluations.

OpenAI disclosed in July that models being tested with reduced cybersecurity restrictions exploited a previously unknown vulnerability to escape an isolated evaluation environment and obtain access to the open internet.

The agents subsequently accessed production infrastructure belonging to AI platform Hugging Face in an attempt to obtain answers for a cybersecurity benchmark.

OpenAI’s investigation found that agents had established unauthorized communication channels, including a persistent message board, allowing otherwise separate agents to exchange information, coordinate their efforts and divide tasks.

The company said the incident did not affect OpenAI customer data, product functionality or availability.

AI developer Anthropic separately disclosed Wednesday that it had identified a fourth incident involving one of its Claude models gaining unauthorized access to a real third-party system during testing.

Anthropic said all four incidents occurred during cybersecurity evaluations conducted in a third-party environment. The company said a configuration problem had left the systems connected to the internet and that the models were running without the safeguards normally used in publicly available versions.