A new arXiv preprint reports that GUI agents can improve their click accuracy by an average of 7.4% after deployment, without any human-annotated data S¹. The paper, posted on 11 August by Shiyu Xuan and Zechao Li, describes a framework where the model learns from its own attempts at runtime: exploring, evaluating, reflecting, and internalising what worked S¹P². If the claim holds, it points to a path where the agents that operate your apps get better the longer they run. The question is whether a 7.4% gain on six benchmarks translates to real screens, and whether the "first work" claim survives a look at the prior literature.

My read: This is the first test-time adaptation method I've seen that specifically targets GUI visual grounding with on-policy self-distillation, but I'm not fully sold on the novelty claim. A May 2026 preprint from a different group already explored on-policy self-distillation for GUI grounding P⁴, and a self-evolutionary reinforcement learning approach appeared in May 2025 P³. The distinction may live in the "test-time" and "reflection-guided" framing, but the authors' claim of being first needs independent verification. The 7.4% figure is self-reported across six benchmarks with no peer review, and the code has not been released yet S¹. I'd watch whether the GitHub repo (horizon-llm/RESD, already public with 16 stars P⁵) gets the implementation that lets others reproduce the number.

The problem: frozen agents on changing screens

GUI visual grounding is the task of looking at a screen and figuring out where to click. It's the foundation for any agent that operates software on your behalf: booking a flight or filing an expense report. According to the paper, current models generally lock their parameters once training is complete P². That means when the app changes (a button moves, a menu restructures, a new version ships), the agent's accuracy degrades with no way to recover unless someone retrains it with fresh labelled data.

This is the gap the paper targets. The authors propose a Test-Time Self-Evolving framework that lets the model improve after deployment without human-annotated ground truth S¹. No labels, no retraining pipeline, no human in the loop. The agent teaches itself from its own attempts.

How the four-step loop works

The framework runs a closed loop with four stages: Exploration, Evaluation, Reflection, and Internalisation S¹.

First, the agent tries to ground a target on the screen. It generates a click prediction. Then an MLLM-based Reflector (a multimodal large language model) assesses whether the result looks right and produces a reasoning reflection: a natural-language critique of what went well or wrong S¹. That reflection feeds into the third step, Reflection-Guided On-Policy Self-Distillation, turning that broad reasoning into dense token-level guidance through a conditioned self-teacher S¹. In plain terms, the model uses its own reflection to generate fine-grained training signal at the level of individual output tokens, then distils that signal back into itself.

The fourth ingredient is Contrastive Calibration. When the agent's exploration fails and it clicks the wrong element, the early tokens it generated were wrong, and those wrong prefixes can poison the self-teaching signal. Contrastive Calibration stops flawed auto-regressive prefixes from polluting the supervision signal when those attempts fail S¹.

The whole loop runs at test time, meaning during inference: when the model is actually being used, not during a separate training phase.

What 7.4% means on a real screen

The authors report an average accuracy improvement of 7.4% over the base model across six benchmarks S¹. That's a single averaged number. The paper doesn't break out per-benchmark results in the abstract, and the full paper is a preprint with no peer review S¹.

GUI grounding accuracy is measured by whether the predicted click coordinate falls within the target element's bounding box. A 7.4% average lift means roughly seven more correct clicks per hundred attempts. On a screen with 40 clickable elements, that's the difference between the agent completing a task flow and stalling on a wrong button.

What to do about it

If you build or deploy GUI agents, say you run an automation team at an insurance company where agents use a claims portal that updates quarterly, this paper sketches a direction worth tracking. Today, when the portal changes, your agents break and someone has to collect new screenshots, label new click targets, and retrain. A test-time self-evolving approach could, in principle, let the agent adapt to the new layout on its own, using its own attempts as feedback.

The practical step this week: check the horizon-llm/RESD GitHub repository P⁵. It's already public with an Apache 2.0 licence, 16 stars, and five open issues. The authors say code will be released S¹, but the repo's current state suggests partial implementation. If you work with GUI agents, watch whether the full training and inference code lands there, and whether the six-benchmark evaluation is reproducible from what's published.

What we don't know yet

Several things remain open:

  • The 7.4% figure is self-reported and has not been independently verified or replicated S¹.
  • The paper has not been peer-reviewed S¹.
  • The source code has not been fully released, so implementation details and reproducibility can't be checked S¹.
  • The "first work" novelty claim is unverified and may be challenged by prior literature, including a May 2026 preprint on on-policy self-distillation for GUI grounding P⁴ and a May 2025 paper on self-evolutionary reinforcement learning for GUI agents P³.
  • It's unclear how the method performs on real-world screens versus benchmark datasets, or what the computational cost of running the four-step loop at inference time is.

The next signal: whether the full code lands in the horizon-llm/RESD repository and whether an independent group reproduces the 7.4% claim. We'll check the repo against that benchmark. Subscribe for the follow-up when the code drops and the numbers get checked.

Sources: S1 — Test-Time Self-Evolving GUI Visual Grounding via Reflection-Guided On- · P2 — Test-Time Self-Evolving GUI Visual Grounding via Reflection-Guided On- · P3 — Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinfo · P4 — Learn where to Click from Yourself: On-Policy Self-Distillation for GU · P5 — horizon-llm/RESD

More from Not A Tech Guy

Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.

Average accuracy improvement over base model across six benchmarks