A July 2026 arXiv preprint presents CRAFT, a technique designed to pinpoint the reasons behind language model failures on particular tasks and create specific training data to address those shortcomings [S1]. When evaluated on four open-source models within finance and legal contexts, CRAFT yielded the top average finance scores for all four systems and outperformed alternatives in legal tasks for three of them [S1]. This addresses a common challenge in fine-tuning, and the underlying approach is surprisingly straightforward.

The diagnosis problem

Current evaluation methods typically highlight underperforming examples or topics without explaining the specific capability deficit [S1]. A system might reveal that a model struggles with legal reasoning, yet fail to specify if the issue lies in citing precedents, organizing arguments, or applying the right laws. Consequently, developers must guess when constructing fine-tuning datasets.

CRAFT approaches this differently [S1]. It utilizes rubric-based evaluation datasets, where answers are assessed against detailed criteria lists. Every individual grading criterion acts as a probe for a specific capability [S1]. For each prompt and rubric combination, CRAFT derives a straightforward description of the skill under examination [S1].

These descriptions are then organized into a hierarchical tree of capabilities [S1]. The model receives scores at each node, and CRAFT adaptively picks the weakest nodes across various levels, focusing on the scale where the failure is most apparent [S1]. These identified weak points then guide the creation of focused supervised fine-tuning data [S1].

This research follows previous efforts in the field. EvalTree, presented at COLM 2025, created hierarchical capability trees to profile model weaknesses but functioned at the prompt level [P5]. Another 2026 preprint examining capability gaps in fine-tuning data highlighted a similar issue: developers must understand which skills a dataset lacks before undertaking a costly training process [P4]. The OpenRubrics project, launched on GitHub in June 2026, applies rubric-based methods to reward modeling for LLMs [P3]. CRAFT's innovation lies in linking rubric-level diagnostics directly to the generation of training data.

How they tested it

The researchers maintained consistent data generation, fine-tuning, and evaluation conditions while evaluating CRAFT against two baselines: prompt-level EvalTree clustering and random untargeted generation [S1]. The experiment involved four open-source models across two professional fields, finance and law, utilizing 13 separate benchmarks that did not overlap with the diagnostic data [S1].

For finance tasks, CRAFT secured the highest domain average across all four models using repeated temperature decoding [S1]. In the legal domain, it topped three of the four models; for the fourth, its performance stayed within the decoding variance bands of the leading baseline [S1]. Essentially, the fourth model's results were sufficiently close that the disparity might be statistical noise.

What it means

The fundamental realization is that diagnosing and addressing a problem should occur at an identical level of detail. Informing a model that it failed a legal question provides minimal learning value. Conversely, specifying that it failed to identify the correct jurisdictional standard for a summary judgment motion gives it a precise area to improve.

CRAFT automates this specificity. By using each rubric criterion as an individual capability probe, it transforms a general weakness into identifiable, fixable components. The hierarchical tree enables it to function at the most appropriate scale for each failure, avoiding the need to force all diagnoses into a uniform category.

This is significant for developers because generating fine-tuning data is costly, and misdirected efforts are even more so. Creating a training set aimed at the wrong deficiency wastes computational resources without yielding improvements. CRAFT's strategy of diagnosing first and then targeting data generation at those specific gaps offers a more methodical approach than simply generating broad datasets and hoping they address the right issues.

What it means for business

A small legal tech startup fine-tuning an open-source model for contract review often encounters a vague issue: the model struggles with indemnity clauses, but the team cannot determine if the problem is clause extraction, risk classification, or basic reading comprehension. CRAFT's rubric-level diagnostics could distinguish between these failures and direct training data toward the exact deficiency.

A regional financial advisory firm employing a fine-tuned model for compliance document analysis faces a similar challenge. General fine-tuning might improve overall accuracy without resolving the most critical capability: accurately identifying missing disclosures. Rubric-level diagnostics would pinpoint that specific need.

The operational shift involves running CRAFT's diagnostics before blindly generating fine-tuning data, identifying the weakest capability nodes, and focusing data generation on those specific areas. This should result in less wasted training data and more precise improvements, although the paper does not explicitly report cost or time savings.

For Australian businesses, a key limitation is that the finance and legal evaluations relied on generic professional domains rather than Australian regulations, market structures, or institutions [S1]. An organization would need to create or obtain rubric-based evaluation data tailored to Australian law or ASIC rules before CRAFT could identify local deficiencies.

What we don't know yet

As an arXiv preprint, the paper has not undergone peer review [S1]. All performance assertions are self-reported by the authors.

The abstract omits the names of the four open-source models and the 13 benchmarks, restricting the extent to which external readers can verify the findings [S1]. Without knowing the model names, it is difficult to assess whether they represent the models commonly used by practitioners.

The legal domain outcomes are mixed. One of the four models did not demonstrate clear superiority over the best baseline, staying only within decoding variance bands [S1]. Based solely on the abstract, it is uncertain whether CRAFT reliably benefits legal contexts or if the fourth model's result indicates a boundary condition.

The evaluation includes only two professional fields. The paper does not test whether rubric-level diagnostics apply to creative writing, code generation, customer service, or other typical fine-tuning objectives.

A key development to monitor is whether the authors publish their code and model names during the peer review process. The OpenRubrics GitHub repository [P3] and EvalTree code [P5] are both publicly accessible, establishing a reproducibility precedent in this research area. If CRAFT does the same, developers could evaluate the method on their own models and domains.

Subscribe to catch the next piece, where we will track whether CRAFT's code drops and whether independent teams can reproduce the finance and legal gains on their own models.

Sources

More from Not A Tech Guy

Generated from an audited evidence pack with primary-source research. Social-media items are discussion signals, not verified facts. Nothing here is financial, legal or medical advice.