Cline Agents Harness Unlocks Benchmark-Leading Performance from NVIDIA Nemotron 3.5 Super VL

Open weights models have become the new frontier of achieving intelligence more efficiently, and we are super excited to see how the latest NVIDIA Nemotron 3.5 Super VL model continues to push the frontier of open weights models.

We’ve been testing NVIDIANemotron 3.5 Super VL in Cline, both on benchmarks and in our own day-to-day work. In our Terminal-Bench 2.1 evaluation, the model scored 61.8% with Cline and thinking mode turned off. That was the highest score among the four harnesses we tested with this model.

We’ve also built an internal agent around Nemotron that works with our team in Slack. It triages tasks, investigates user-reported bugs, and uses Cline to work through issues. The benchmark gives us a way to compare performance; the internal agent lets us see how the model handles the work our team needs done.

Unlocking powerful models with tailored harness

A strong model matters. But the system wrapped around it can matter just as much.

An agent harness is the software around the model: the instructions it receives, the tools it can use, and the checks that guide its work. We’ve been refining those parts of Cline for open-weight models, including Nemotron 3.5 Super VL.

In this evaluation, Nemotron-3.5 Super VL 120B A12B scored 61.8% with Cline, compared with 53.9%, 51.7%, and 44.9% in three other open-source harnesses. We’ve labeled those harnesses A, B, and C in the chart below.

Terminal-Bench 2.1 scores for Nemotron-3.5 Super 120B. Cline: 61.8%; open-source harnesses A: 53.9%, B: 51.7%, C: 44.9%.

Cline reached its score with thinking off; harnesses A and C used high reasoning or effort settings. This comparison shows how much results can vary with the software around the same model. It doesn’t isolate which Cline features account for the difference or measure how much compute they save.

What we changed in Cline

Plan before changing code

Cline prompts the model to plan before implementation: understand the task, choose an approach, then start making changes. That gives the model an explicit sequence to follow as it works.

We think this structure can help the model spend less effort deciding what to do next. The result with thinking off is encouraging, but we haven’t measured token savings or isolated the effect of the planning instruction. That would take token-usage data and a comparison with the instruction enabled and disabled.

Detect when an approach has stalled

An agent can keep repeating an action long after it stops being useful. Cline’s stall detector can end that attempt and record a structured event, including the error class, whether it is recoverable, and the iteration number.

Those details make failed runs easier to investigate. In the other runs we reviewed, we often had to read through a sequence of tool calls and identify the repetition ourselves. A recorded stall gives us a specific place to start debugging.

Check the behavior the task depends on

Two very specific tasks from the evaluation show what this looks like in practice.

On chess-best-move, Cline parsed the board into FEN notation and ran an exhaustive checkmate search. It found two mates in one, e2e4 (Qe4#) and g2g4 (g4#), in a run spanning 58 log lines. Harness A made 29 tool calls and reopened the board image at least six times before timing out.

On headless-terminal, harnesses A and B both ran real tests. Cline completed its checks in five commands, using an itemized checklist that covered both whether Ctrl-C delivered a signal and whether the shell survived afterward. The narrower checks in the other runs missed the behavior that eventually broke.

These runs help us understand the score. They also give us concrete things to improve: how the agent chooses an approach, recognizes a dead end, and verifies that its fix works.

Using Nemotron inside Cline

Our internal Nemotron-based agent lives in Slack and runs Cline under the hood. It can spin up sandboxes and containers, and call more advanced models when a task needs them. We use it to triage internal tasks and investigate and fix issues reported by users.

One example is a Windows diagnostics export failure. A teammate asked the agent to investigate a report in which Cline tried to create an existing Downloads folder, hit an EEXIST error, and failed to write the diagnostic report.

A teammate asks our internal agent to investigate a Windows diagnostics export failure.

That is the kind of work we are super excited to see Nemotron super being able to help with: taking a user report, tracing the failure, and giving the team a useful next step. We’re encouraged by what Nemotron 3.5 Super VL can already do, and we’re continuing to refine Cline through benchmark runs and the issues our team handles every day!