Anthropic’s AI progress comes with a disclosure gap: some of the evidence explaining its speed is confidential.
In a November 5 assessment, Model Evaluation and Threat Research (METR) estimated that Anthropic’s benchmark scores have improved at 1.9 times their earlier rate since October 2025. The broader finding concerns capabilities on well-specified tasks, with substantial uncertainty about how far those gains extend.
For IT buyers, the distinction matters: faster benchmark progress does not establish how much a new model will improve their own workflows.
The numbers behind the speedup
| Benchmark progress | AECI scores growing at an estimated 1.9 times the earlier rate since October 2025; 95% confidence interval: 1.5–2.8 |
| Estimated AI contribution | Perhaps 80% of current acceleration, with substantial uncertainty |
| Training compute | Faster scaling of training compute did not explain the acceleration |
| Public evidence | Anthropic redacted much of the evidence METR considered most informative |
METR’s 11-week evaluation examined internal research data, including benchmark results for unreleased models, and roughly 15 staff interviews. AECI is Anthropic’s version of the Epoch Capabilities Index. METR judged AI use likely responsible for most current acceleration, while acknowledging other contributing factors.
“We are confident that the acceleration was not due to an acceleration in the scaling of training compute,” METR said in its report.
Informal Slack polls put average self-reported speedup from the most capable internal model at approximately 12x in October. METR considered this weak evidence from a likely unrepresentative sample, rather than a measured company-wide productivity gain.
The transparency problem
Anthropic exercised its redaction rights to remove what METR called “a majority of the evidence we found most informative for these conclusions.” The company described the redactions as “aimed at removing confidential information that could accelerate other AI developers.”
The findings also provide context for Anthropic CEO Dario Amodei’s call to slow frontier AI development. His essay argued that additional time could help developers strengthen safeguards. The assessment does not establish whether Anthropic met or missed a specific pacing commitment.
More must-read AI coverage
- SS&C Intralinks DealCentre AI vs. Datasite: Which platform is built for the future of dealmaking?
- SS&C Intralinks FundCentre AI vs. Juniper Square: Which platform better supports modern private markets fund managers?
- Why Data, Not Models, Determines AI Success
- The Rise of the AI-Native Factory: How Physical AI Is Transforming Manufacturing
What IT buyers should check before upgrading
For organizations evaluating AI tools, the useful question is whether a newer model completes their work more reliably and economically. A vendor’s development speed should inform what buyers investigate, rather than determine purchasing decisions.
Build a repeatable evaluation around representative tasks. Include straightforward requests and difficult cases involving incomplete information, conflicting instructions, or actions that require human approval.
| Task completion | Whether the requested work was completed correctly |
| Human review | Time spent checking and correcting results |
| Reliability | Errors, unsupported answers, and inconsistent behavior |
| Cost and speed | Total cost per successful task and response time |
| Access controls | Whether the model respects permissions and approval requirements |
For development teams, research on AI coding agents offers useful context for designing evaluations. Track review effort, defects, and accepted changes alongside output volume; producing more code is only one part of the result.
Apply the same scrutiny to independent assessments, including proposals for rival AI companies to test one another’s models. Ask vendors which model versions were evaluated, what access evaluators received, which conclusions remain uncertain, and what supporting evidence buyers can inspect.
Keep evaluation results tied to specific model versions and configurations. Before expanding deployment, set acceptance criteria, assign responsibility for reviewing failures, and establish a rollback option if quality deteriorates.
The purchasing decision should rest on demonstrated improvements in your environment. Faster development makes a repeatable testing process more useful, especially when public evidence cannot answer every question.
Read more: Put model evaluations in context with this guide to Claude’s models, tools, pricing, and privacy controls.