A recent evaluation of 100 genuine office assignments, which typically take 2.32 hours of human effort, reveals that advanced AI systems finish the work quicker and at a lower cost than individuals, yet they fail to reach the standard of human output S¹. Known as OmegaUse-OfficeVal and published as an arXiv preprint on July 29, this evaluation is unique in attaching an economic indicator to each assignment: the financial cost and time required for a person to complete it S¹. That framing lets you ask the question every business actually cares about, and the one most AI benchmarks dodge. Is the cost saving worth the quality loss?

My read: This is the first benchmark I've seen that treats office-task evaluation as an economics problem rather than a pure capability test. Most agent benchmarks tell you pass rates. This one tells you whether the trade-off pencils out. I'm skeptical of the human baseline methodology, which the preprint does not detail in the available material, and the fact that the same group built and evaluated the benchmark is a real bias risk. But the core finding, cheap and fast but not good enough, aligns with what we are seeing across agent research.

Why economics changes the benchmark game

Most AI benchmarks evaluate models on a simple pass or fail basis. OmegaUse-OfficeVal does something different. Every one of the 100 assignments includes two metrics: the duration a human needs to finish the job, and a cost estimate representing the expense of hiring a person for the work S¹. That means you can line up the cost of running an AI agent (inference, the compute cost of actually running the model) against the cost of a human doing the same job, and weigh that against the quality gap.

The assignments are sourced from actual office software requests made by professionals, which are then modified using a method that protects privacy S¹. The average assignment requires 2.32 hours of human effort S¹, which puts them in the range of what an office worker might do in a morning: build a spreadsheet model, format a multi-section report, merge data across applications. These are long-horizon tasks, multi-step, multi-tool jobs that test whether an agent can sustain a chain of actions without breaking.

To verify if an agent successfully finished the assignment, the team created automated grading scripts based on detailed rubrics S¹. These are automated grading scripts that check specific deliverable properties rather than relying on subjective human judgement.

The gap that matters

The research assessed multiple leading large language models against a human control group S¹. The results cut two ways. All examined AI systems were significantly more affordable and quicker than their human counterparts S¹. However, none achieved the standard of quality found in human-produced results S¹.

That is the finding buried under the headline. Speed and cost are solved problems, or close to it. Quality is not. An agent that finishes a 2.32-hour task in seconds but produces a spreadsheet with broken formulas or a report with missing sections has not done the job. It has done a draft of the job. The economic framing makes that distinction measurable: you can now ask, for a specific task worth a specific amount of human labour, whether the AI's output is good enough to ship, or whether a human still needs to fix it.

This connects to a broader pattern in agent research. Earlier benchmarks like OfficeBench P⁴ and AgentBench P⁵ tested agents on office automation and general agent tasks, but without the economic layer. The initial OmegaUse initiative P³, released in January 2026, developed a versatile graphical user interface agent for independent task completion. OmegaUse-OfficeVal extends that line by adding the question that decides whether any of this ships in practice: does it save money once you account for the quality gap?

What to do about it

For a mid-sized accounting firm, the benchmark's finding is directly actionable. Consider a junior accountant who spends two hours building a client financial summary in Excel: pulling data from a ledger, formatting it into a standard template, and checking the totals. An AI agent might do the pull and format in under a minute for a few cents of inference cost. But if the agent drops a row or misapplies a formula, the senior reviewer still needs to check every figure by hand, and the time saving evaporates.

The practical move is to split the task. Use the agent for the mechanical parts (data extraction, initial formatting, first-pass assembly) and keep the verification step human. The benchmark's economic framing lets you quantify that split. If the agent saves 1.5 hours of assembly but adds 45 minutes of review, the net saving is 45 minutes, not two hours. That is the number that belongs in the business case.

This week, if you are testing any AI agent on office work, time the human review of its output. Compare that review time against the human time the agent replaced. If the review takes more than half the original task time, the agent is not saving labour. It is relocating it.

What we don't know yet

The preprint does not name which frontier LLMs were tested S¹, which makes it impossible to compare specific models or judge whether the quality gap is uniform across providers. The human baseline methodology, including who the humans were, how their work was graded, and what "human-level deliverable quality" actually means, is not detailed in the available material. The benchmark was built and evaluated by the same research group, which creates a design risk: the rubrics and verifiers may reflect that group's assumptions about what a correct deliverable looks like.

The paper is a non-peer-reviewed preprint S¹, so all findings are preliminary and author-reported. No independent replication exists yet. The code and dataset are open-sourced S¹, which means other groups can test the claims, but nobody has done so.

The next signal: peer review and independent evaluation of OmegaUse-OfficeVal by a separate research group. If the benchmark gains traction, expect results tables naming specific models within the next few months. We'll check those claims against the quality gap when they land.

If this kind of plain-English decode is what you need from AI coverage, subscribe. There is more of this coming every week.

Sources: S1 — OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Sui · P2 — OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Sui · P3 — OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Exe · P4 — OfficeBench: Benchmarking Language Agents across Multiple Applications · P5 — THUDM/AgentBench

More from Not A Tech Guy

Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.

Human labour per task vs AI agent performance