A new open-source tool called Kernel Forge uses AI agents to automatically write and optimize CUDA GPU kernels, achieving up to 2.83x speedup over PyTorch's default execution mode on real models including Stable Diffusion 3.5 and Gemma 4 S¹. The tool, listed on arXiv this week as paper 2607.24762, accepts any unmodified PyTorch model and handles the entire pipeline from kernel generation to testing to integration, no CUDA programming knowledge required S¹. The question is whether agent-generated kernels can hold up against hand-tuned code from experienced GPU engineers, or whether this opens a door that was previously bolted shut for most developers.

My read: This is the first agent harness I have seen that takes an unmodified PyTorch model and produces working, faster CUDA kernels end to end. The 2.83x softmax result on Gemma 4 E2B is genuinely impressive for 50 iterations. I am skeptical of the broader claim until we see numbers against hand-written kernels rather than just PyTorch eager mode, which is a relatively low bar. The MCTS approach is smart, but the evaluation covers only 14 kernels across four models on one GPU, so the ceiling is unknown.

Why CUDA kernel writing is the bottleneck nobody talks about

Writing CUDA kernels is where performance goes to either live or die. A GPU runs thousands of threads in parallel, but squeezing real speed out of it requires manually crafting code in NVIDIA's C-like language, tuning memory access patterns, managing shared memory, and debugging race conditions that only appear under specific workloads. Most machine learning teams never touch this layer. They rely on PyTorch's built-in operators, which work but leave substantial performance on the table.

The gap between a well-written CUDA kernel and PyTorch's default eager mode, the interpreter-like execution path where each operation runs separately, can be enormous. Kernel Forge targets exactly this gap S¹.

How the agent explores optimization paths

Most existing tools that use LLMs to generate CUDA code follow a single chain: generate a kernel, test it, ask the model to fix what failed, repeat. Kernel Forge takes a different approach. It uses Monte Carlo Tree Search, or MCTS, a decision-tree algorithm that branches across multiple possible optimization paths simultaneously rather than walking one linear refinement chain S¹.

Think of it like a chess engine evaluating several moves ahead instead of committing to the first line that looks decent. The agent can explore different kernel strategies in parallel, compare results, and backtrack from dead ends without losing the work from promising branches.

The tool also ships with a graphical interface for monitoring progress, inspecting candidate kernels, and debugging failures S¹. That matters because agent-generated code fails often, and being able to see why is the difference between a useful tool and a black box.

The numbers that matter

The authors evaluated Kernel Forge on four PyTorch models spanning vision, diffusion, and LLM workloads, all running on an NVIDIA DGX Spark with a GB10 GPU S¹. With 50 optimization iterations per kernel, the system optimized 14 kernels to beat PyTorch eager mode S¹.

The standout results:

  • 1.52x speedup on adaptive_avgpool2d in ResNet-50 S¹
  • 1.70x speedup on group_norm in Stable Diffusion 3.5 Medium S¹
  • 2.83x speedup on softmax in Gemma 4 E2B S¹
  • 1.54x speedup on softmax in Qwen 3.5 35B-A3B S¹

Kernel Forge speedup over PyTorch eager mode by operation

The softmax result on Gemma 4 is the headline number. Softmax is a core operation in every transformer model, used to convert raw scores into probabilities. A 2.83x improvement on a single operator will not make the whole model run 2.83x faster, but across many operators, the compounding effect adds up.

A separate research group at the University of Minnesota published a similar framework called CudaForge, which also uses agents with hardware feedback to optimize CUDA kernels P⁴. The field is converging on the idea that LLM agents can handle GPU kernel optimization, but the approaches differ. Kernel Forge's distinguishing feature is its drop-in design: it accepts an unmodified PyTorch model directly, while many existing tools require you to extract standalone CUDA code and manually reintegrate the results S¹.

The GitHub repository, created in September 2025 under TheJoshBrod/KernelForge, has 13 stars and an MIT license P³. It is early days for community adoption, but the code is publicly available.

Kernel Forge fits squarely into a broader trend: the harness, the scaffolding around the model that determines what it can actually do, is becoming the unit of engineering rather than the model itself.

What to do about it

Consider a small computer vision startup running ResNet-50 inference on edge devices. Their team of four has nobody with CUDA experience, so they accept PyTorch's default performance. With Kernel Forge, they could point the tool at their model, let the agent run 50 iterations on the adaptive_avgpool2d operator, and potentially pick up a 1.52x speedup on that layer without writing a line of CUDA S¹. For a company paying by the GPU hour, that is real money.

The tool supports vision, diffusion, and LLM workloads S¹, so the same workflow applies whether you are running image classification, generating images with Stable Diffusion, or serving a language model. The practical step this week: clone the repository from GitHub, point it at a PyTorch model you already run, and see what the agent produces on a single operator. The GUI will show you which kernels failed and why, which tells you quickly whether this approach fits your workload.

What we don't know yet

All performance figures are author-reported from a single hardware configuration, the NVIDIA DGX Spark with GB10 GPU, and have not been independently verified S¹. The evaluation covers only 14 kernels across four models, so claims about broader performance are unsupported. The paper is an arXiv preprint and has not undergone peer review S¹.

The comparisons are against PyTorch eager mode, which is the easiest baseline to beat. We do not know how Kernel Forge's output compares against hand-tuned kernels from experienced CUDA engineers, or against PyTorch's compiled mode (torch.compile), which already provides automatic optimisation. The speedups may not generalise beyond the specific GB10 GPU used in testing.

The next signal: watch the GitHub repository for community contributions and independent benchmarks on different hardware. If third-party developers reproduce the speedups on consumer GPUs like the RTX 4090, that will tell us whether this approach scales beyond one lab's setup. We will check the repo's issue tracker and any follow-up papers against this claim.

If this kind of reporting is useful, subscribe to keep reading.

Sources: S1 — Kernel Forge: An Agent Harness for LLM-based Generation and Optimizati · P2 — Kernel Forge: An Agent Harness for LLM-based Generation and Optimizati · P3 — TheJoshBrod/KernelForge · P4 — CudaForge: An Agent Framework with Hardware Feedback for CUDA Kernel O · P5 — china-qijizhifeng/agentic-harness-engineering

More from Not A Tech Guy

Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.