Change2Task, a system described in a July 30 arXiv preprint, converts merged pull requests into verified, executable coding agent tasks with a 79.6% construction success rate across 1,130 eligible source changes S¹. Training coding agents stalls on tasks, not models. You need verified, runnable problems drawn from real codebases, the kind that teach an agent what "done" actually means. Change2Task claims to build those tasks automatically from the code changes developers already merge every day, and whether the 79.6% figure holds up under scrutiny is the question that determines if this is a tool or a footnote.

My read: This is the most direct attack on the coding-agent training-data problem I've seen. The 79.6% success rate is self-reported and unreviewed, and the "up to 98.0%" agreement figure is a maximum, not an average. But the approach, mining pull requests that already exist by the millions on GitHub and reversing them into tasks, is structurally sound. The real test is whether reconstructed tasks actually transfer to agent improvement, and the paper does not show that yet.

How the reversal works

The system takes a merged pull request and works backwards. A developer fixed a bug, added a feature, wrote tests, migrated an API, or patched a security hole. Change2Task takes the end state, the healthy repository with the fix applied, and reconstructs the "before" state: the broken version the agent would need to fix S¹.

It does this through three methods. Patch Reversal strips the merged changes back to the pre-fix state. Code Mapping links the change to the specific code region it touched. Agent Reconstruction uses an agent itself to rebuild the task environment when simple reversal is not enough S¹.

Next comes validation: start from a healthy base, apply the task state (the broken version), confirm the agent can work on it, and confirm the fix restores the healthy state S¹. This validation loop is what separates a verified task from a broken one. A task where the tests do not run, or where the "fix" does not actually fix anything, gets filtered out.

Why task construction is the real bottleneck

SWE-bench needs 90% of tasks to be solvable for benchmark results to be reliable. Quantity alone does not solve it. Tasks need to actually run and represent real work.

The five task families Change2Task targets cover the bulk of what coding agents actually do: Bug Fix, Feature Addition, Test Generation, API Migration, and Security Repair S¹. These are drawn from real repositories where real developers merged real changes.

This connects to a broader current. HuggingFace's Repo2RLEnv project, which turns any GitHub repository into a verifiable reinforcement learning environment, has gathered 282 stars on GitHub P⁵. Repo2Run, a February 2025 preprint, tackled the related problem of automatically building executable environments for code repositories at scale P². The ICML 2024 code-act paper showed that executable code actions produce better LLM agents P⁴. The shared thread across all of these: the environment matters as much as the model.

The numbers that matter

Starting from 1,130 source changes eligible for construction, Change2Task achieved 79.6% verified task construction success across the five task families S¹. On a matched candidate set, it recovered 29.2% more verified tasks than a construction baseline based on pull requests S¹.

Historical and reconstructed cases achieved up to 98.0% matched outcome agreement under agent evaluation S¹. That "up to" matters. It is a maximum, not an average, and the paper does not report the distribution.

Reusing modern repository bases, rather than rebuilding environments from scratch, reduced measured expenditure across the complete pipeline by 10.8% S¹. In a field where inference costs compound across thousands of training episodes, a 10.8% saving on environment setup is real money.

What to do about it

If you are building or evaluating coding agents, the implication is practical. Consider a mid-sized fintech that maintains a Python trading library with 200 merged pull requests per quarter. Today, turning those merges into training tasks for an internal coding agent means a human reading each PR, writing a task description, setting up the environment, and verifying the agent can solve it. That is days of engineering work per batch. Change2Task's approach, if it holds up, could automate most of that pipeline.

The immediate action: audit your own merged pull requests from the last quarter. Sort them by the five task families Change2Task uses (bug fix, feature, test, API migration, security). If your distribution skews heavily toward one or two families, any automated task generator will produce a skewed training set. Knowing your own mix is the first step to judging whether a system like this would work for your codebase.

What we don't know yet

The paper is an unreviewed preprint. All performance figures are self-reported S¹, the baseline is internal to the paper, and no external state-of-the-art benchmark comparison is cited. The system and its code have not been confirmed as publicly released, nor has the derived dataset. The results have not been independently replicated.

That 79.6% success rate is measured across the five evaluated task families and may not generalise to all repository types. The 98.0% agreement figure is a maximum, not a guaranteed rate.

The paper does not show that agents trained on Change2Task-generated tasks actually perform better on downstream work. Task construction is necessary but not sufficient. The gap between "we can build tasks" and "these tasks make agents better" is where the evidence stops.

The next signal: watch for a v2 revision on the paper's arXiv page, which would typically carry reviewer responses or a public data release. We will check the listing at arxiv.org/abs/2607.28591 for updates and test the 79.6% claim against any independent replication. If you want that follow-up in your inbox, subscribe and we will send it the day it lands.

Sources: S1 — Change2Task: From Repository Changes to Executable Coding Agent Tasks · P2 — Repo2Run: Automated Building Executable Environment for Code Repositor · P3 — yogurts-sy/Seg2Change · P4 — xingyaoww/code-act · P5 — README.md at main · huggingface/Repo2RLEnv

More from Not A Tech Guy

Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.