> ## Content Index
> Fetch the complete content index at: https://www.notatechguy.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# OpenAI publishes perplexity scores, proposes shared AI safety metric
- URL: https://www.notatechguy.com/openai-publishes-perplexity-scores-proposes-shared-ai-safety-metric/
- Published: 2026-11-22T18:28:07.000Z
- Updated: 2026-11-22T18:28:08.000Z
- Description: OpenAI disclosed perplexity measurements showing losses below 0.8 on its evaluation dataset, proposing the metric as a shared safety standard for AI
- Author: Marcello Babbili
- Tags: Technology & AI, OpenAI, Google

OpenAI published its internal perplexity scores on November 22, 2026, disclosing losses below 0.8 on its evaluation dataset, and proposed the metric as a shared AI safety standard [S¹](https://openai.com/index/perplexity-and-intelligence?ref=notatechguy.com). The company concedes the disclosure hands competitors a clearer picture of where OpenAI stands, but says the urgency of shared safety measures outweighs that cost [S¹](https://openai.com/index/perplexity-and-intelligence?ref=notatechguy.com).

**My read:** The number I would watch is the gap between GPT-4's 0.91 and the sub-0.8 losses OpenAI says it has now reached. That band is where the company claims "transformative capabilities" in cybersecurity and mathematics emerged. But nobody outside OpenAI has verified these measurements, and the post is an argument for a standard, not the standard itself. No other developer has signed on.

### What perplexity actually measures

Perplexity, in this context, is not the search company of the same name. It is an information-theoretic score: how well a model predicts the next token (the next chunk of text) on average, given everything before it [S¹](https://openai.com/index/perplexity-and-intelligence?ref=notatechguy.com). Lower is better. A model with a perplexity of 1 would be perfectly certain about every next word; real models sit well above that. OpenAI says it has tracked perplexity on a held-out dataset for years as the primary measure of progress on what it calls the "intelligence substrate" [S¹](https://openai.com/index/perplexity-and-intelligence?ref=notatechguy.com).

The metric is equivalent to cross-entropy loss, a standard training objective, expressed on a logarithmic scale [S¹](https://openai.com/index/perplexity-and-intelligence?ref=notatechguy.com).

### The numbers behind four generations

OpenAI's blog post maps each generation to a perplexity milestone on its current main evaluation dataset [S¹](https://openai.com/index/perplexity-and-intelligence?ref=notatechguy.com). GPT-2 set a new state-of-the-art on older datasets and was, in OpenAI's words, the first model to consistently produce coherent paragraphs [S¹](https://openai.com/index/perplexity-and-intelligence?ref=notatechguy.com). GPT-3 added in-context learning, the ability to pick up a task from a few examples in the prompt [S¹](https://openai.com/index/perplexity-and-intelligence?ref=notatechguy.com). GPT-3.5, at a log-perplexity of approximately 0.99, was fluent enough in conversation to become the original ChatGPT [S¹](https://openai.com/index/perplexity-and-intelligence?ref=notatechguy.com)[P⁷](https://openai.com/index/chatgpt/?ref=notatechguy.com). GPT-4, at approximately 0.91, used chain-of-thought reasoning well enough to motivate the reasoning models that followed [S¹](https://openai.com/index/perplexity-and-intelligence?ref=notatechguy.com).

![OpenAI log-perplexity on main evaluation dataset lower is better](https://storage.ghost.io/c/6e/89/6e896869-22ef-4281-a213-b4c462c17cff/content/images/2026/11/chart_b3bbd207778fdc50a834.png)

Now OpenAI says it has reached losses below 0.8 on the same dataset, and that this is where it observed "transformative capabilities" in cybersecurity and mathematics [S¹](https://openai.com/index/perplexity-and-intelligence?ref=notatechguy.com). The post does not name a specific model at that level, nor does it publish error bars or methodology details that would let an outside party reproduce the measurement.

### Why OpenAI is giving away the scoreboard

The blog post frames perplexity as a candidate for a shared industry metric, one that would let developers compare models on a common scale and coordinate safety thresholds [S¹](https://openai.com/index/perplexity-and-intelligence?ref=notatechguy.com). OpenAI calls this a step toward "a shared scientific basis for future pacing efforts" [S¹](https://openai.com/index/perplexity-and-intelligence?ref=notatechguy.com).

The admission that publishing helps competitors is unusual. Most labs treat training-loss curves as proprietary. But the post stops short of proposing a binding standard or naming partners. No other developer has adopted the metric for safety comparisons, and OpenAI has not disclosed the dataset, compute budget, or training methodology behind the numbers [S¹](https://openai.com/index/perplexity-and-intelligence?ref=notatechguy.com).

The post drew discussion on Hacker News, gathering between 80 and 96 points and 36 to 43 comments across two snapshots taken roughly an hour apart on November 22 [S³](https://openai.com/index/perplexity-and-intelligence/?ref=notatechguy.com)[S⁴](https://openai.com/index/perplexity-and-intelligence/?ref=notatechguy.com).

A team evaluating whether to trust OpenAI's safety claims, say a policy group advising regulators, would need the evaluation dataset itself to verify the scores. Without it, the shared standard OpenAI proposes has no shared way to check it.

---

*Sources: [S1 — Perplexity and Intelligence](https://openai.com/index/perplexity-and-intelligence?ref=notatechguy.com) · [S2 — Perplexity and Intelligence - OpenAI](https://news.google.com/rss/articles/CBMiY0FVX3lxTE96Sm1EaVlsMHctNGtBMDBYdVRreWpySVdVS1RhSVZCUzk3enZHQmlIS3lWY2Rmajh2Q3RNZWU2RDEwc3ZEX2tONEpsbEN6RXowZk1ySVdaWXVVR09jX21aelV6UQ?oc=5&ref=notatechguy.com) · [S3 — Perplexity and Intelligence](https://openai.com/index/perplexity-and-intelligence/?ref=notatechguy.com) · [S4 — Perplexity and Intelligence](https://openai.com/index/perplexity-and-intelligence/?ref=notatechguy.com) · [P5 — Perplexity trusts GPT-6 Astra with end-to-end systems | OpenAI](https://openai.com/index/perplexity-improving-accuracy-with-astra/?ref=notatechguy.com) · [P6 — openai (OpenAI)](https://huggingface.co/openai?ref=notatechguy.com) · [P7 — Introducing ChatGPT | OpenAI](https://openai.com/index/chatgpt/?ref=notatechguy.com) · [P8 — gaojing8500/AIGCOpenSource](https://github.com/gaojing8500/AIGCOpenSource?ref=notatechguy.com)*

---

*Written from 8 sourced items, 5 of them primary.*

## More from Not A Tech Guy

- [Google DeepMind benchmark: AI agents score below 57% on visual work](https://www.notatechguy.com/google-deepmind-benchmark-ai-agents-score-below-57-on-visual-work/)
- [GeForce NOW Black Friday cuts 30% off six-month plans](https://www.notatechguy.com/geforce-now-black-friday-cuts-30-off-six-month-plans/)
- [SGLang cache system doubles AI agent turns on NVIDIA chips](https://www.notatechguy.com/sglang-cache-system-doubles-ai-agent-turns-on-nvidia-chips/)