Joyce Chai's lab at the University of Michigan built a simulated household where AI agents observe 15 hours of daily life spread across 38 in-game days, and found that frontier models still can't work out the rules governing human behaviour S¹P².

The models can guess what you'll do next by counting what you've done most often. What they can't do is reason about why you did it, or adjust when your habits change S¹.

My read: This is the first benchmark I've seen that tests whether an AI agent can actually understand a person over time, rather than retrieve facts about them. The frequency-shortcut finding rings true from what we've seen in agent reliability more broadly, as when SearchAuditor identified 32% of AI agent failures traced to surface-level reasoning. I'm skeptical of how much a synthetic household tells us about real homes, and the paper hasn't been peer-reviewed. But the diagnostic is sharp: if your assistant can't tell the difference between "you always cook on Tuesdays" and "you cook when you have ingredients," it will fail the moment you run out of groceries.

What SimLife actually is

SimLife is a simulation platform that generates long stretches of household activity: visual observations, ground-truth action logs, and synthetic dialogues with audio S¹. Think of it as a soap opera for AI agents to watch, except every action is logged and tagged so researchers can test whether the viewer understood the plot.

The benchmark built on top of it, SimLife-BP, contains 106 episodes averaging 15.49 hours of activity and 38.57 in-game days, with 1,439 question-answer pairs S¹. Those 15.49 hours are simulated time, not wall-clock time. Each question probes one of four reasoning types: direct (what happened), counterfactual (what would change if), noisy (what if the observation is incomplete), and inverse (what rule explains this behaviour), each tested under different levels of hint about the underlying rules S¹.

The authors include Run Peng, Zinnia Nie, and Jing Ding from the University of Michigan, with collaborators at ByteDance, Osaka University, Amazon, and the University of Pennsylvania P². The code is public on GitHub at dfy37/lifesim, which had 17 stars at the time of writing P³.

The finding that matters

When the researchers ran current frontier models on SimLife-BP, the models produced surface-level predictions that looked right but broke under scrutiny S¹. They fell back on frequency-based shortcuts, counting how often things happened and guessing they'd happen again, instead of building if-then rules from what they observed S¹.

That distinction matters because human behaviour is conditional. You don't always make coffee at 7am. You make coffee at 7am when you have beans, when you didn't sleep through your alarm, when you're not running late. A model that has learned "coffee at 7" will fail the morning you run out. A model that has learned "coffee when beans available and not rushing" will adapt.

The models also struggled when behavioural patterns shifted mid-episode S¹. If a simulated resident started cooking dinner at 6pm for two weeks and then switched to 7pm, the models kept predicting 6pm. They couldn't update their understanding, which echoes the agent-reliability problems we've tracked before, like when a new technique cut AI agent wait time by 45% by addressing the kind of stale-context problem that trips up long-running agents.

What would have to hold

The paper is an arXiv preprint, not peer-reviewed, and the findings have not been independently replicated S¹. The household data is entirely synthetic, generated by the simulation rather than recorded from real homes, which limits how far the results generalise to actual human behaviour S¹. The authors flag this themselves, noting that SimLife opens a space for studying memory, personalisation, and adaptation rather than closing it S¹.

No specific commercial models are named as top or bottom performers in the available evidence, so any claim about whether GPT-4o outperforms Claude on household reasoning would be speculation.

Who needs this first

The immediate audience is researchers building embodied agents, systems that act in physical or simulated environments, and personalised assistants meant to persist across months of use. A team designing a home robot that needs to learn a resident's routines over weeks could use SimLife-BP to test whether their system is actually reasoning about habits or just counting them. The same long-horizon memory problem connects to broader agent safety work, as we saw when an AI coding agent defence cut malware severity 83% by forcing agents to verify rather than assume.

The GitHub repository, dfy37/lifesim, is public and written in Python P³. A research engineer could clone it today and run the benchmark against their own model, though with 17 stars and one open issue, the project is early-stage.

The authors describe long-context pattern understanding as a major bottleneck for embodied agents S¹. The next checkpoint is whether the simulation holds up against real household data, something the current paper does not attempt.

Sources: S1 — SIMLIFE: Pattern Understanding for Long-Horizon Human-Agent Partnershi · P2 — SIMLIFE: Pattern Understanding for Long-Horizon Human-Agent Partnershi · P3 — dfy37/lifesim · P4 — YinpeiDai/m3-agent · P5 — DDigimon/GraphPattern

Written from 5 sourced items, 4 of them primary.

SimLife-BP benchmark scale