Fine-tuning a 35B model on agent traces beats bigger judges at lower cost

LangChain and Fireworks showed that fine-tuning a 35B model on actual agent traces can match or beat the big frontier models at judging agent output. The cost drop is massive.

Matching big models on narrow tasks changes where the real edge sits

They took Qwen-3.5-35B and trained it on multi-turn traces from two agent setups. Labeling mixed model assistance with human review to catch error signals. The fine-tuned model held its own against larger judges while slashing inference costs.

Evaluation tasks seem especially open to this kind of specialization. They rely on spotting patterns in how agents interact rather than broad reasoning, so the generalist models carry extra baggage that isn't needed.

Teams sitting on their own trace data now have a real advantage

| What people are saying | What the data showed | How it shifts thinking | Practical takeaway | |--------------------------------|-----------------------------------------|----------------------------------------------------|-------------------------| | Open models can close gaps on narrow work | The 35B model matched frontier judges using only traces from chat-langchain and Fleet | Labs are questioning whether every benchmark needs full frontier scale | Specialized models speed up observability and hit API-heavy providers hardest | | Data quality beats size for eval work | Filtering plus hybrid labeling gave solid ground truth at 18-24% positive error rate | Companies focus on collecting their own traces instead of shopping models | Platforms with live user loops build lasting edges research labs don't have | | Lower costs change what gets monitored | Up to 100x cheaper inference on production traces | Smaller teams can now afford full-trace monitoring | Inference layers like Fireworks pull value away from raw model builders | | General claims need testing across domains | Holdout sets from two different agent products checked transfer | Single-environment judge results get more scrutiny | Good error detectors become baseline, not a moat |

• More production traces improve the judge, which then finds better signals for iteration.
• Teams stuck on frontier API judges pay a running cost penalty that adds up fast.
• Skipping tool calls in training showed that user-assistant patterns carry the key signals anyway.

Raw benchmark scores miss the bigger move toward distillation tuned to specific domains.

The practical takeaway is simple. Teams that already have production traces and know how to fine-tune are set up to improve agent reliability faster. Chasing undifferentiated scale looks less compelling once the economics are this clear.