A July 2026 arXiv preprint reports that visual pretraining — feeding models raw documents with their charts, equations, and layouts intact — consistently outperforms text-only pretraining across multiple benchmarks [S1]. If that holds up, it undermines a choice the AI field made years ago: strip every PDF, webpage, and slide into plain text before training. What happens to the models, and the companies building on them, when the pictures were always carrying more signal than the words?

The information we've been throwing away

Foundation models — the large AI systems behind tools like ChatGPT — have been trained almost entirely on text [S1]. The recipe is simple in concept: gather enormous collections of written material, and have the model learn to predict the next word. This approach has driven the rapid progress of the past several years [S1].

But text is a lossy format. A scientific paper's figure, a typeset equation, the two-column layout of a news page: all of these carry information that plain text cannot faithfully represent [S1]. When a PDF gets converted to text for training, the chart becomes a string of numbers. The equation loses its structure. The layout that told you which paragraph mattered most is gone.

The standard fix has been to accept the loss. Current pretraining approaches convert visually rich sources into plain text and train on that [S1]. The assumption, rarely questioned, is that language models should learn from language. Text in, text out.

Training on the raw page

The new paper, "Scalable Visual Pretraining for Language Intelligence," challenges that assumption directly [S1]. The authors ran a systematic study of unsupervised visual pretraining: training models on visual documents without first extracting their text [S1]. No OCR pass. No conversion step. The model sees the page the way a human reader does.

The result: across multiple model architectures and multiple test suites, visual pretraining on the same underlying data consistently outperformed text-only pretraining [S1]. Same corpora, different training signal. The visual version won.

This connects to a broader shift in the field. Separate work on scaling vision pretraining to 4K resolution [P2] has shown the field is hitting the limits of low-resolution image processing. The common thread: vision is no longer just an add-on to language models. It may be a better training substrate.

What it means

The finding, if it holds under peer review, reframes a basic trade-off in AI development. The field has spent years building better text extraction pipelines — OCR systems, PDF parsers, HTML cleaners — to convert the world's visual documents into training data. This paper suggests that work may have been solving the wrong problem. The visual document was never an obstacle to work around. It was the richer signal.

For a reader with no background: think of it like studying a painting from a written description versus looking at the actual canvas. The description tells you what's there. The painting shows you how it fits together. The paper's claim is that models trained on the "painting" — the raw visual document — learn something the text-only models miss.

The paper frames this as an efficient pathway to scalable language intelligence [S1], meaning the approach could let models learn more from the same amount of source material, without the cost and error introduced by text extraction.

What it means for business

For small operators, the implications are concrete but early.

A two-person legal tech firm that currently runs OCR over case documents before feeding them to a language model might, in time, skip that step entirely. If visual pretraining becomes standard, the model could learn directly from scanned court filings, preserving the structure of footnotes and tables that OCR routinely mangles.

A suburban accounting practice that relies on AI to read financial statements faces a similar shift. Today, a chart in an annual report gets flattened into text, losing the visual relationships between data points. A model trained on the raw visual document could, in principle, learn those relationships directly.

But this is a preprint, not a product. No one can download a visual-pretraining pipeline today and swap it into their workflow. The research direction signals where infrastructure investment is heading: away from text extraction and toward native visual training. Companies building document-AI pipelines should watch whether model vendors begin offering visually pretrained checkpoints.

What we don't know yet

The paper provides no specific quantitative figures in its abstract — no percentage improvements or benchmark scores [S1]. The claim that visual pretraining "consistently outperforms" text-only is stated without numbers we can verify from the abstract alone. The full paper would need to be examined for those details.

The study has not been peer-reviewed [S1]. Findings may change under review, and the comparative claims are limited to the same underlying corpora. Whether the advantage holds across different data regimes is unverified.

The term "visual documents" refers specifically to figures, typeset equations, and page layouts [S1]. The paper does not claim the approach generalises to photographs, video, or other non-document visual data.

No independent research group has replicated the findings, and no production system has adopted the approach. The deep research sources surfaced alongside this paper — including work on 4K-resolution vision pretraining [P2], visual tokenizers for generation [P3], and generative language-image pretraining [P4] — suggest the broader field is moving toward visual training, but each takes a different approach. Whether they converge remains open.

The next concrete signal to watch: whether the authors release code or model weights, and whether a peer-reviewed version appears with full benchmark tables.

If this kind of plain-English decode is what you want from AI news, subscribe — more stories like this are on the way.

Sources

Sources

More from Not A Tech Guy

Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.