A new arXiv paper describes an AI agent that rewrites its own prompts, rules, and code after every round of testing, evolving nine versions of itself to answer questions against the DBpedia knowledge graph S¹. It settles at 0.22 accuracy on the 2025 DBpedia validation set. That number looks like a failure, but the agent's errors cluster in one specific place, and several questions the benchmark marks wrong appear to be correct. What the self-improvement loop reveals about the benchmark itself may matter more than the score.

My read: This is the first agent architecture I've seen where the system treats its own prompt and code as the thing to debug, in addition to the query output. The 0.22 accuracy will grab attention as a failure, but the paper's finding that the bottleneck is predicate selection, not SPARQL syntax, tells me the hard part of knowledge-graph QA was never query construction. It was always knowing which relationship in a messy ontology to trust. I don't buy the "converges quickly" claim yet without seeing the actual curve, and the silence on results from the two stronger backbone models is conspicuous. But the benchmark critique, that correct queries get penalised because DBpedia's properties are ambiguous, is the kind of observation that only surfaces when you let an agent fail in public.

The agent debugs its own instructions as well as its answers

Most AI agents today follow fixed instructions: a prompt tells them which tools to call, in what order, and how to format the result. This system, described in arXiv:2608.07700v1, adds a second layer S¹. After each round of inference on a validation set, a "researcher agent" reviews what went wrong and suggests modifications to its own prompts and tool-orchestration code. It then tests those changes on the same validation set to see if they help.

The researchers ran this loop on DBpedia, a knowledge graph built from Wikipedia data, and evolved nine successive versions of the agent using what they describe as a low-cost reasoning model S¹. The best-performing configuration was then deployed on two stronger backbone models, though the paper does not report those results S¹.

The "researcher agent" here treats its own configuration as an experiment to run, rather than relying on a human to tune prompts by hand.

Where nine versions keep hitting the same wall

The paper identifies one consistent bottleneck across all nine versions: basic-graph-pattern predicate selection S¹. In plain terms, the agent struggles to pick the right relationship label from DBpedia's ontology when building the core of its query.

SPARQL, the query language for knowledge graphs, has two layers of difficulty. The syntax, the brackets and keywords that structure a query, is something modern language models handle well. The modifiers, such as filters and sorting clauses, are also manageable. But predicates, the specific property names that connect entities in the graph, are where things break down.

DBpedia's ontology is dense and inconsistent. The same real-world relationship might be represented by multiple property names, or a property name might be ambiguous about what it actually connects. The agent can write a syntactically perfect SPARQL query and still get the wrong answer because it chose the wrong predicate.

The benchmark is marking correct answers wrong

Here is the finding that raises this paper above a simple system report. Several benchmark items appear to penalise correct queries due to property ambiguity in DBpedia S¹. The agent writes a query that would return the right answer, but because it uses a different but equivalent property name than the benchmark's reference query, the system marks it as incorrect.

The researchers suggest future Text-to-SPARQL benchmarks should be scored using a combination of machine translation metrics and information retrieval metrics S¹, rather than exact string matching against a reference query. Machine translation metrics measure semantic similarity between queries, so two queries that ask the same question in different ways would both score well. Information retrieval metrics check whether the query returns the correct results, regardless of how it phrases the request.

If the benchmark cannot distinguish between a wrong query and a differently-worded correct query, every system evaluated on it is being measured against a flawed standard.

What to do about it

For teams building knowledge-graph question-answering systems, the paper offers two practical signals.

First, if your agent is failing, check where. The paper's finding that predicate selection, not syntax, is the bottleneck means engineering effort spent on better SPARQL templates or syntax validation is likely wasted. The leverage is in ontology mapping: helping the agent understand which DBpedia property corresponds to which natural-language relationship. A pharmaceutical company building an internal knowledge graph over drug interactions, for example, would get more from investing in a clean, unambiguous property schema than from a more sophisticated query generator.

Second, if you are evaluating any text-to-SPARQL system, do not rely on exact-match scoring. Run a secondary check that executes the generated query and compares results against the expected answer set. A query that returns the right data through a different property path is not wrong.

The one thing to check this week: if your team uses a knowledge graph in production, pull ten sample queries and compare the predicates your system selects against the ontology documentation. The gap between those two lists is your accuracy ceiling.

What we don't know yet

The paper leaves several questions open. The two stronger backbone models were deployed but their results are not reported in the abstract, so we cannot tell whether more capable models close the predicate-selection gap or merely produce more confident wrong predicates S¹. The claim that self-improvement "converges quickly" is stated without a learning curve, so the rate and stability of convergence remain unclear S¹. The paper has not been peer reviewed; all findings are the authors' own claims S¹. And while a GitHub repository under the LiberAI organisation appears to exist for the project P², we cannot confirm it is a complete or official code release.

Related work is still arriving. AgenticT²S, a separate text-to-SPARQL system using agentic collaborative reasoning, was submitted to the ACL ARR 2026 January review cycle P⁴. RPO-RAG, which applies relation-aware preference optimisation to small language models for knowledge-graph retrieval, appeared at WWW 2026 P⁵. Both tackle the same predicate-selection problem from different angles.

The next signal: the full paper text on arXiv, which should contain the two stronger backbone model results and the convergence curve the abstract omits. If those numbers show a jump, the self-improvement loop earns its keep. If they flatline, the bottleneck is in the ontology, not the model. We will check this claim against the full paper when results are published.

If you want these dispatches in your inbox before they hit the web, the subscribe button is right there.

Sources: S1 — Towards Researcher Agents for Knowledge-Graph Question Answering · P2 — LiberAI/researcher-agents-for-kgqa · P3 — OPPO-PersonalAI/O-Researcher · P4 — $AgenticT^2S$: Robust Text-to-SPARQL via Agentic Collaborative Reasoni · P5 — KaeHyun/RPO-RAG

More from Not A Tech Guy

Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.