OpenAI Chief Scientist Jakub Pachocki published internal pretraining perplexity measurements on November 21, 2026, proposing the metric as a shared standard for comparing model developers and informing industry safety norms.
Why it matters
OpenAI published internal perplexity measurements to help establish shared safety measures across the AI industry. Chief Scientist Jakub Pachocki argued that tracking pretraining progress through compression metrics is key to understanding the arc of advances in artificial intelligence.
Evolution of Model Perplexity
OpenAI outlined how perplexity scores evolved across its models, noting GPT-3.5 reached a log-perplexity of ~0.99 and GPT-4 hit ~0.91. Pachocki stated that recent losses below 0.8 coincided with the emergence of transformative capabilities in fields like cybersecurity and mathematics.
Proposed Industry Standards
Pachocki proposed that frontier model developers jointly establish shared norms and common limits based on perplexity until monitoring and alignment capabilities achieve concrete milestones. He acknowledged challenges such as the difficulty of comparison across labs and detachment from actionable evaluations.
Key facts and where they come from
- OpenAI's GPT-3.5 reached a log-perplexity of ~0.99 on the company's current main evaluation dataset.
GPT‑3.5, at a log-perplexity of ~0.99 on our current main evaluation dataset, was capable of fluent conversation
- OpenAI's GPT-4 reached a log-perplexity of ~0.91.
GPT‑4, at ~0.91, was proficient at utilizing chain-of-thought reasoning productively
- OpenAI models have achieved losses below 0.8.
Now, as we have reached losses below 0.8, we have seen the emergence of transformative capabilities in fields like cybersecurity and mathematics.
The catch
- The announcement does not say what specific datasets other labs should use to achieve a standardized comparison.
- Pachocki notes that perplexity does not directly correspond to a task users care about.
- OpenAI acknowledges that perplexity does not account for advances outside of pretraining, such as chain-of-thought reasoning.
The open question: Can competing AI laboratories agree on a standardized perplexity metric despite using different evaluation datasets and training data?
Tell us what you think · Discuss it in the AI room
Reader poll: Should AI developers adopt perplexity as a shared metric for safety standards?
A show of hands from readers, not a scientific poll. Say why in the comments.
