A 21 July arXiv preprint introduces TPIPS, a text-prompted image similarity metric that lets you specify which dimension of similarity you care about, then scores it against human judgment on that specific axis [S1]. Frontier vision-language models, the systems behind image search and generation across the industry, show a "considerable performance gap" against human consensus on the authors' benchmark [S1]. If that gap holds, the AI deciding what images look alike is missing something basic about how people see.

The one-number problem

Ask someone whether two photos are similar and the answer depends on what you mean. Shape, colour, texture, mood? Today's perceptual similarity metrics, the ones built into image generation pipelines, don't ask. They collapse every dimension of similarity into a single scalar value, with no way to condition on a specific aspect [S1]. A red apple and a blue apple get one score. A red apple and a red car get another. The metric can't tell you they match in colour but differ in shape, because it was never asked.

This matters because image generation has exploded. When the metric is a blunt instrument, the feedback loop is blunt too.

The dataset and the fix

The authors built a large-scale dataset of human similarity judgments over image triplets, with each triplet annotated across multiple free-form semantic aspects of similarity [S1]. Rather than asking annotators for a single "more similar" judgment, they collected judgments conditioned on specific aspects: similar in shape, similar in colour, similar in style.

They then fine-tuned a vision-language model to produce TPIPS, short for Text-Prompted Image Perceptual Similarity [S1]. You give it two images and a text prompt specifying the dimension of similarity you care about. It returns a score calibrated to human judgment on that specific axis. The authors report that TPIPS aligns more closely with human perception than prior approaches and generalises reliably beyond its training distribution [S1].

The approach sits in a growing cluster of research pushing image evaluation beyond single numbers. DreamSim, a related project on GitHub, explores new dimensions of human visual similarity [P2]. RelSim, accepted to CVPR 2026, tackles relational visual similarity for image retrieval and analogy tasks [P3]. UniPercept, an ICML 2026 spotlight, unifies perceptual-level understanding across aesthetics, quality, structure, and texture [P5]. A separate arXiv paper on multi-dimensional human preference for text-to-image generation makes the same case from the generation side: one number is not enough [P4]. TPIPS adds a twist the others lack. You don't just get multiple dimensions. You choose which one matters, in plain language, at query time.

What it means

The core finding has two layers. First, the tool. TPIPS lets a developer say "compare these two images by texture, not colour" and get a score that tracks what a human would say. That is new. Existing metrics either give you one number or require a fixed set of pre-defined dimensions. TPIPS takes free-form text, which means the dimension of comparison can be anything a human can describe.

Second, the benchmark. The authors tested frontier vision-language models on their dataset and found a "considerable performance gap" compared to human annotators' consensus [S1]. These are the same VLMs being used to evaluate generated images, rank search results, and filter content. If they cannot match human similarity judgments when the dimension is specified, they are making decisions about what looks alike that diverge from what people actually see. The gap is described qualitatively in the abstract, with no specific numbers, so the size of the problem remains an open question.

What it means for business

A two-person design studio using AI image generation to produce client mockups currently relies on automated metrics to pick the best output. Those metrics give one number. If the client says "I want something similar to this reference but in a warmer palette," the studio has no automated way to measure similarity on the colour axis while ignoring shape. TPIPS, if it works as described, would let them prompt "similar in colour and tone" and get a score.

For a stock photo platform, the implications are in search. Current image retrieval uses embedding similarity, which blends all dimensions into one vector. A text-prompted similarity metric could let a user search for "same composition, different subject" or "similar lighting, different angle" and get results that match on the specified axis. The authors demonstrate TPIPS in text-guided retrieval and compositional search [S1], exactly these use cases.

For a team evaluating generative models, TPIPS offers fine-grained evaluation: whether this output is similar to the reference on the specific dimension we asked for, rather than a blanket "is this output good" score [S1]. That is the difference between knowing a model failed and knowing why.

The code, data, and trained models are publicly available [S1], which means a small team can test this today without licensing fees.

What we don't know yet

This is an unreviewed preprint [S1]. Every performance claim, from the "considerable performance gap" to the claim that TPIPS "aligns more closely with human perception" and "generalises reliably," is self-reported by the authors without independent verification or specific quantitative results in the abstract [S1].

The dataset, while described as large-scale, covers a subset of possible semantic aspects. We don't know how TPIPS handles edge cases: abstract art, medical imaging, satellite imagery, or culturally specific visual cues. The generalisation claim is about distribution shift within the authors' test set, not about entirely new domains.

The "considerable performance gap" for frontier VLMs is stated without numbers. Whether that gap is 5 per cent or 50 per cent changes the story considerably. Peer review, or an independent reproduction, will need to quantify it.

The next concrete event to watch: whether TPIPS or its dataset is submitted to a major computer vision conference, where reviewers will test the generalisation and alignment claims against held-out data. The related projects in this space, RelSim at CVPR 2026 [P3] and UniPercept at ICML 2026 [P5], suggest the venue and the competition are already set.

If you want to see how AI image evaluation is changing, this is the paper to track. Subscribe for the follow-up when the numbers land.

Sources

More from Not A Tech Guy

Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.