> ## Content Index > Fetch the complete content index at: https://www.notatechguy.com/llms.txt > Use this file to discover other available public pages before exploring further. # Agent memory boosts gpt-oss 16 points but does nothing for GLM-5 - URL: https://www.notatechguy.com/agent-memory-boosts-gpt-oss-16-points-but-does-nothing-for-glm-5/ - Published: 2026-08-18T21:27:28.000Z - Updated: 2026-08-18T21:27:28.000Z - Description: IBM Research tested agent memory on eight models and found the biggest model gained nothing while a 117B model jumped 16 points at 5% extra token cost. - Author: Marcello Babbili - Tags: Technology & AI gpt-oss-120b gained 16.1 percentage points in task completion when given a selective set of self-mined guidelines, while GLM-5, a larger 745-billion-parameter model, gained nothing from the same system [S¹](https://huggingface.co/blog/ibm-research/altk-evolve-hmm?ref=notatechguy.com). Same memory, same 585-task benchmark, opposite outcomes. What decides whether an agent actually learns from its own past, or just burns tokens pretending to? **My read:** This is the first agent-memory study I've seen that honestly reports a zero-result for a frontier-scale model instead of burying it. The temptation with any "agent learning" paper is to claim universal gains, but IBM Research's own data shows memory is conditional, not additive. I don't buy the prompt-caching affordability claim yet, because it comes with no cost figures attached. And I'd want to see the saturated pattern tested on more than one model before treating it as a real category rather than a single data point. ## What the agent actually remembers "Memory" in this system doesn't mean replaying a past transcript. It means a guideline set: distilled lessons from the agent's own earlier runs, covering what worked, what failed, and unusual cases worth remembering [S¹](https://huggingface.co/blog/ibm-research/altk-evolve-hmm?ref=notatechguy.com). ALTK-Evolve, described in IBM Research's August 18 Hugging Face post, extracts those guidelines from previous trajectories and supplies them to the agent at inference time, right when the agent is doing its work [S¹](https://huggingface.co/blog/ibm-research/altk-evolve-hmm?ref=notatechguy.com). The model's parameters stay frozen. No human annotation is involved. What changes is the guidance the agent can draw on, not the underlying model itself [S¹](https://huggingface.co/blog/ibm-research/altk-evolve-hmm?ref=notatechguy.com). The evaluation ran on AppWorld, a benchmark from Stony Brook NLP [P⁵](https://github.com/StonyBrookNLP/appworld/?ref=notatechguy.com) that packages 585 multi-step tasks spanning nine simulated apps, divided into 168 standard tasks and 417 harder challenge tasks [S¹](https://huggingface.co/blog/ibm-research/altk-evolve-hmm?ref=notatechguy.com). Eight models were tested, from a 30-billion-parameter dense model up to frontier proprietary systems [S¹](https://huggingface.co/blog/ibm-research/altk-evolve-hmm?ref=notatechguy.com). ## Three patterns, not one rule The results fall into three patterns, and the pattern a model lands in depends on more than size. Strong models with headroom, models that haven't yet saturated the benchmark, do best with the full guideline set: every guideline, including rare edge-case lessons [S¹](https://huggingface.co/blog/ibm-research/altk-evolve-hmm?ref=notatechguy.com). DeepSeek-V3.2, a 671B mixture-of-experts model (MoE, a design that activates only a fraction of total parameters per query), climbed 9.5 percentage points in task completion when given its full self-mined guideline set [S¹](https://huggingface.co/blog/ibm-research/altk-evolve-hmm?ref=notatechguy.com). Smaller or weaker models do better with a tight, high-confidence core of guidelines plus a handful retrieved per task [S¹](https://huggingface.co/blog/ibm-research/altk-evolve-hmm?ref=notatechguy.com). gpt-oss-120b, a 117B MoE model, gained 16.1 percentage points with this selective approach [S¹](https://huggingface.co/blog/ibm-research/altk-evolve-hmm?ref=notatechguy.com). Already-saturated models show no measurable gain [S¹](https://huggingface.co/blog/ibm-research/altk-evolve-hmm?ref=notatechguy.com). GLM-5, a 745B MoE model, sat in this category [S¹](https://huggingface.co/blog/ibm-research/altk-evolve-hmm?ref=notatechguy.com). The researchers call this the "saturated pattern" but are explicit that the label describes what they observed, not a proven cause [S¹](https://huggingface.co/blog/ibm-research/altk-evolve-hmm?ref=notatechguy.com). ![Task completion gain from agentic memory, by model](https://storage.ghost.io/c/6e/89/6e896869-22ef-4281-a213-b4c462c17cff/content/images/2026/08/chart_d983054413c23e46c1bc.png) What puts a model in one pattern rather than another involves more than parameter count. Several factors, including how much room the model has left on the benchmark, its context-window size, its architecture, the quality of the guidelines it receives, and the distribution of tasks, all appear to influence the outcome [S¹](https://huggingface.co/blog/ibm-research/altk-evolve-hmm?ref=notatechguy.com). ## Why selective retrieval beats the full dump For gpt-oss-120b, the full guideline set gained less than the selective approach and cost roughly 50% more tokens [S¹](https://huggingface.co/blog/ibm-research/altk-evolve-hmm?ref=notatechguy.com). Curated retrieval, a compact core plus per-task retrieval, achieved the same 16.1-point gain at only 5% extra token cost [S¹](https://huggingface.co/blog/ibm-research/altk-evolve-hmm?ref=notatechguy.com). More memory was both less effective and more expensive for this model. Selective retrieval works because it gives the agent only what's relevant to the current task, rather than flooding its context with every lesson it has ever learned. IBM Research says prompt caching, which lets providers store and reuse the initial portion of a prompt across multiple requests, keeps even the full guideline set affordable in production [S¹](https://huggingface.co/blog/ibm-research/altk-evolve-hmm?ref=notatechguy.com). The post provides no specific cost figures to back this up. The broader current in agent design is moving toward smarter retrieval rather than bigger context windows. ALTK-Evolve pushes the same direction. Filtering matters more than volume. ## What to do about it If you're running agents in production, the takeaway is practical: don't assume more memory helps. Test it. Consider a customer-support automation team deploying agents to handle ticket triage across a product suite. They've been dumping every past resolution into the agent's context, hoping more examples mean better performance. The ALTK-Evolve findings suggest the opposite might be true. If their model has headroom on the task, a full guideline set could help. If the model is already performing near its ceiling, the extra context is dead weight that inflates token costs for no gain. And if the model is smaller or mid-tier, selective retrieval, a compact core of high-confidence lessons plus a few retrieved per ticket, could match the full-dump approach at a fraction of the cost. The first step is measuring where your model sits. Run your agent on a held-out task set without memory, then with the full guideline set, then with selective retrieval. The gap between no-memory and full-memory tells you whether your model has headroom. The gap between full-memory and selective tells you whether you're paying for tokens that don't earn their keep. This week, check your agent's token usage per task. If memory injection is consuming more than 10% of your token budget, test whether stripping it back to a selective core changes your completion rate. The result tells you whether your memory budget is working or wasted. ## What we don't know yet All statistics in this study are self-reported by IBM Research through a Hugging Face blog post, with no independent corroboration yet [S¹](https://huggingface.co/blog/ibm-research/altk-evolve-hmm?ref=notatechguy.com). The findings haven't been peer-reviewed. The "saturated pattern" rests on a single model, GLM-5, and the researchers themselves say the label describes an observation, not a proven cause [S¹](https://huggingface.co/blog/ibm-research/altk-evolve-hmm?ref=notatechguy.com). The prompt-caching affordability claim comes with no cost figures [S¹](https://huggingface.co/blog/ibm-research/altk-evolve-hmm?ref=notatechguy.com). The evaluation covers eight models, a reasonable spread but not enough to draw firm lines between the three patterns. Whether a model falls into the "strong with headroom" or "saturated" category could depend on the specific benchmark, the guideline quality, or the task mix in ways this single evaluation can't isolate. The code is public on GitHub under the AgentToolkit organisation, with 93 stars and 52 open issues [P³](https://github.com/AgentToolkit/altk-evolve?ref=notatechguy.com), so replication is possible. The next signal: the first independent replication using that public code on a new set of models. The repo already has 52 open issues to work through, so the work is underway. We'll check whether the three-pattern split holds or collapses when that replication lands. Subscribe and you'll get that answer the day we do. --- *Sources: [S1 — How Much Memory Does Your Agent Actually Need?](https://huggingface.co/blog/ibm-research/altk-evolve-hmm?ref=notatechguy.com) · [P2 — ALTK‑Evolve: On‑the‑Job Learning for AI Agents](https://huggingface.co/blog/ibm-research/altk-evolve?ref=notatechguy.com) · [P3 — AgentToolkit/altk-evolve](https://github.com/AgentToolkit/altk-evolve?ref=notatechguy.com) · [P4 — benchmark/appworld/quickstart.md at main · agentscope-ai/ReMe](https://github.com/agentscope-ai/ReMe/blob/main/benchmark/appworld/quickstart.md?ref=notatechguy.com) · [P5 — StonyBrookNLP/appworld](https://github.com/StonyBrookNLP/appworld/?ref=notatechguy.com)* ## More from Not A Tech Guy - [OpenAI adds safeguards to pace frontier AI model development](https://www.notatechguy.com/openai-adds-safeguards-to-pace-frontier-ai-model-development/) - [Hugging Face adds multi-vector retrieval to sentence-transformers](https://www.notatechguy.com/hugging-face-adds-multi-vector-retrieval-to-sentence-transformers/) - [Euclid-Omni: AI for Olympiad geometry with far less compute](https://www.notatechguy.com/euclid-omni-ai-for-olympiad-geometry-with-far-less-compute/) --- *Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.*