The best AI models in the world transcribe real pop music with just 38% accuracy on a new benchmark called MulTTiPop, released as a preprint on arXiv this month [S1]. That means the systems designed to convert audio into playable notation are getting more notes wrong than right. Nobody knew the gap was this wide until researchers built a test that actually reflects what pop sounds like across seven decades.

The number that should embarrass AI

38% Onset F1. That is the score the top-performing automatic music transcription model achieved when a Carnegie Mellon team ran it against MulTTiPop [S1]. Onset F1 measures whether a model correctly identifies the start of a note — the right pitch, the right instrument, at the right moment. A perfect score is 100. Random guessing sits near zero. Thirty-eight is barely passing.

The dataset itself is modest: 572 segments of popular music, totalling 3.5 hours of audio [S1]. But it spans an unusually wide range, with songs from the 1930s through the 2000s across diverse genres [S1]. That breadth matters because most existing transcription benchmarks draw from narrow stylistic pools, which lets models overfit to specific production styles and instrumentation.

To build MulTTiPop, the team performed metadata-based matching on song segments from two existing collections: Lakh MIDI and TheoryTab [S1]. The alignment process was not fully automated. Researchers manually identified an anchor beat between each audio clip and its corresponding MIDI, then used beat tracking on the audio and warped the MIDI to match tempo and timing [S1]. Even constructing the test required human hands at every step, a detail that shows how hard it is to map audio to notation without human judgment.

The dataset is now publicly available on Hugging Face under the organisation gclef-cmu [P3], with sound examples and further details at the project website [S1]. The authors are Nathan Pruyne, Benjamin Stoler, William Chen, Chien-yu Huang, Shinji Watanabe, and Chris Donahue [P3].

What it means

Automatic music transcription — the task of turning an audio recording into a machine-readable score, with each note assigned to the right instrument — has been a standing challenge in AI research. Systems like YourMT3+, which uses transformer architectures and cross-dataset training, have pushed the field forward on existing benchmarks [P4]. But those benchmarks have a blind spot: they tend to test on the same kinds of music the models were trained on.

MulTTiPop exposes that blind spot. When state-of-the-art models face real pop recordings spanning seven decades, their performance drops to a level the authors describe as showing "substantial room for improvement" [S1]. That is researcher-speak for: the models are failing.

A model that gets 38% of note onsets right is not ready for practical use. Feed it a guitar riff and it will misidentify more than half the notes. Ask it to separate a bass line from a drum kit in a dense mix and it will guess wrong more often than not. The gap between benchmark performance and real-world usefulness turns out to be enormous.

This matters beyond music. Transcription is a microcosm of a broader AI problem: models that perform well on curated training data often collapse when tested on messier, more varied real-world inputs. MulTTiPop is a reminder that the distance between lab and living room can be measured in decades of pop music.

What it means for business

For music software companies, the 38% figure is a reality check. Any product that claims to automatically convert audio to MIDI or sheet music — the kind of feature found in digital audio workstations, music education apps, and sample-creation tools — is likely performing far worse than its marketing suggests when tested on diverse, real-world pop.

A two-person music production startup building an auto-transcription feature would need to treat MulTTiPop as a stress test before shipping. If their model was trained on classical piano or isolated stems, pop mixes from the 1970s could break it.

For music education platforms that promise students they can play along with any song, the benchmark suggests the underlying transcription layer is unreliable enough that manual correction would still be needed for most tracks. That changes the cost model. Instead of a fully automated pipeline, the workflow requires human review, which is slower and more expensive.

The dataset's availability on Hugging Face [P3] means any developer can now run the same evaluation. That transparency could pressure vendors who have been quoting accuracy numbers from friendlier benchmarks.

What we don't know yet

The MulTTiPop paper is an arXiv preprint, categorised under cs.AI and cs.LG, and has not been peer-reviewed [S1]. All dataset statistics, methodology claims, and the 38% benchmark result are author-reported. Independent researchers have not yet verified the evaluation protocol or reproduced the scores.

The dataset covers only 3.5 hours of audio across 572 segments [S1] — segments, not full songs. Whether a larger or more contemporary set (the collection stops at the 2000s) would produce different results remains an open question.

The 38% Onset F1 reflects the authors' specific evaluation setup. Different benchmarking protocols — alternative metrics, different segment lengths, or varied model configurations — could yield different numbers. Until independent teams run their own tests, the headline figure should be treated as a signal, not a verdict.

The next concrete event to watch: whether the dataset gets adopted by other research groups for benchmarking, and whether any team publicly reports a model that meaningfully beats 38%. The dataset and sound examples are live now at gclef-cmu.org/multtipop [S1], so the clock is already running.

If this kind of ground-truth reporting is worth your time, subscribe — there is more where this came from.

Sources

More from Not A Tech Guy

Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.