跳到正文
CNBC·· 43 分钟前AI 评分64

OpenAI与四家独立评估机构签约开展安全评估

OpenAI inks agreements with four independent evaluators to boost company's safety work

AI 导读

OpenAI宣布与Apollo Research、METR、Transluce和Goodfire签署协议,由四家机构开展模型行为与安全防护试点评估。评估方将获得部分内部系统、代码、模型检查点和训练数据等资源,并可发布评估结果,但OpenAI保留删去机密信息的权利。

正文

OpenAI CEO Sam Altman speaks at Semafor's "Silicon Valley and The World" conference in Menlo Park, California, on Nov. 16, 2026.

Godofredo A. Vasquez | AFP | Getty Images

OpenAI on Monday announced it has signed agreements with four independent evaluators to help support the artificial intelligence company's safety work.

The evaluators, Apollo Research, METR, Transluce and Goodfire, will initially conduct distinct pilot assessments around model behavior and safeguards, OpenAI said. The company will give the organizations "deep access" to its internal systems as part of the agreements, adding that it plans to use what it learns to expand its work with independent experts going forward.

"We expect this first phase of assessments to lead to longer-term, consistent embedded engagements over time that can continuously provide third party safety signals and recommended improvements on our most load bearing safety claims and safety cases," OpenAI said in a blog post.

Independent evaluators have occupied a quiet corner of the multi-trillion-dollar AI industry thus far, but they've taken on outsized importance following a series of high-profile safety and security incidents this year. The nascent ecosystem is still ironing out best practices, including how its work should be funded and what the reporting structures should ultimately look like.

OpenAI CEO Sam Altman pledged to grant independent evaluators "employee-like access" to the company in September, shortly after Anthropic CEO Dario Amodei said his company would do so as part of his call to slow down the pace of advanced model development. Anthropic has since announced agreements with evaluators including Accenture and METR, and it said in October that it's engaging in discussions with others.

Read more CNBC tech news

OpenAI's safety practices have been under intense scrutiny since July, when two of its models escaped containment, accessed the open internet and hacked a company called Hugging Face. Following the incident, OpenAI said it would invest more in its safeguards and alignment work, which is the process by which the company works to ensure that its models act in accordance with human interests and values.

The company said Monday that its pilot assessment with Apollo Research will explore whether its models covertly work against their developers' intentions, and whether its safeguards detect that behavior. Meanwhile, Goodfire will embed researchers within OpenAI to focus on training run monitoring, debugging and alignment, and METR intends to carry out two assessments focused on model alignment and monitoring.

OpenAI said Transluce will conduct an initial assessment of its post-training alignment evaluations, and that the organization plans to conduct an eight-week-long follow-on assessment with broader access to additional models and information next year.

The evaluators will get access to some of the "same internal resources" that OpenAI's own researchers use, including its source code, model checkpoints, training data and other documentation, according to the release. The organizations will also have access to relevant researchers and engineers within the company.

OpenAI said Apollo Research and Goodfire "accept financial compensation for their work," but that it is not contingent on the results of their evaluations. METR and Transluce do not accept compensation for their assessments, OpenAI said, but they might accept compute or API credits to help support their work.

"We do not believe that accepting financial compensation prevents assessors from maintaining independence — but we believe it is important to disclose this," OpenAI said.

The evaluators will be able to publish their findings, but OpenAI said it reserves the right to redact confidential information.

Disputes that arise between the company and the third parties will be overseen by OpenAI's Safety Advisory Group, which is a group of senior officials from across the company that assesses model capabilities and safeguards, and other OpenAI executives. If disagreements are not resolved, they'll be escalated to the Safety and Security Committee, an independent oversight committee of the OpenAI Foundation's board.

"We are committed to expanding this work and helping build the capacity and shared standards needed to sustain it," OpenAI said.

WATCH: OpenAI unveils ChatGPT Wallet: Here’s what you need to know

OpenAI unveils ChatGPT Wallet: Here's what you need to know

watch now

来源:CNBC · cnbc.com