Mistral Large 4 Benchmark on FrontierCode 2.0: How Far Open Weight Coding Models Trail on Mergeable Code

By Mistral's own report, Mistral Large 4 is the strongest open weight model on Cognition's FrontierCode 2.0. It trails the best closed models by a wide margin. Mistral reports a score of 19.7%. Cognition's table puts Claude Opus 5.5 at 45.5% and GPT-6 Astra at 42.8%. Your coding agents may need to produce changes that a maintainer merges with little correction. For that work, Mistral Large 4 does not yet replace a closed frontier model.

The model is a better fit for tasks with a clear spec and an automatic check, such as terminal work. It is also a good fit when the code and data must stay on your own infrastructure.

The key numbers

Two numbers explain most of this article. The gap from Mistral Large 4 to Opus 5.5 on FrontierCode 2.0 is 45.5 − 19.7 = 25.8 points. Opus 5.5 scores 2.3 times higher. The lead over the next open model is 0.4 points.

The problem: spec-style benchmarks overstate coding agent readiness

Most coding benchmarks test whether an agent can meet a written spec. Examples are "make the failing test pass" and "finish this terminal task." Real repository work also has rules that nobody wrote down. Examples are the code style, the scope of a change, and what a maintainer expects the change to touch.

Cognition, which builds the Devin coding agent, made FrontierCode 2.0 to measure this second kind of work. In its words, "agentic software engineering remains far from solved. Models struggle to infer user intent, respect unstated constraints, and anticipate long-term consequences" (Cognition).

Open-source maintainers wrote the tasks. Cognition says more than 20 developers "each spent over 40 hours per task building realistic coding evals from the repositories they maintain".

Each task has blocking criteria. A blocking criterion is a rule that, if broken, makes the change unmergeable. Cognition states that "Solutions that don't pass blocking criteria receive 0." A solution that passes every blocker gets a weighted rubric score.

Terminal-Bench works differently. A task passes when the terminal state matches the expected result. That makes it a good test of a written spec. It cannot tell you whether a maintainer would accept the change.

What changed from FrontierCode 1.1 to 2.0

Cognition made version 2.0 harder in measurable ways (Cognition):

• Reference solutions change 67% more files and 89% more lines.
• Codebases are 25% larger, and tasks come from 55% more repositories.
• Tasks have 43% more blocking criteria.
• The reference solution differs from the upstream pull request in 80% of tasks, up from 60%.

Thoroughness is now the main source of failure. Thoroughness blockers went from 168 to 501, about 3 times as many, and the thoroughness failure rate rose from 11.7% to 19.2%. Cognition loosened the scope criteria, and the scope failure rate fell from 9.0% to 3.8%.

Every model in Cognition's table lost points. The size of the loss is where the models separate.

The last two columns are our arithmetic from Cognition's numbers. "Share lost" is points lost divided by the 1.1 score. For example, Opus 5.5 lost 9.1 points of 54.6, which is 17%.

The table has three groups. Opus 5.5 and GPT-6 Astra lost 9 to 11 points. Grok 4.7, Gemini 4 Argon, and Muse Wonder 1.0 lost 19 to 23 points. The two open weight models in the table, DeepSeek V4.1 Flash and Kimi K3, lost more than half their scores. Cognition's own model, SWE-2, also lost more than half, so this is not a problem that only open models have.

Cognition summarizes the open-model result directly: "significant room for improvement for open source models, even more so for western OSS models". Mistral is a European lab, so Mistral Large 4 is a western open model.

Where Mistral Large 4 sits

Mistral Large 4 is Mistral's largest model, with open weights. Mistral says the model is "fully open-weight" and gives a FrontierCode 2.0 score of 19.7% (Mistral).

Read the 19.7% carefully, for three reasons:

1. It is self-reported. Mistral Large 4 is not in Cognition's published table. FrontierCode 2.0 is private, so no one else can run it again.
2. The lead is too small to matter. A 0.4-point lead over DeepSeek V4.1 Flash is likely inside normal run-to-run variance. Our guide to leaderboard gaps smaller than error bars explains how to test this.
3. The two sources do not fully agree. Mistral's chart shows Kimi K3 at 17.3%, and Cognition's table shows 17.7%. We use Cognition's value.

The model does better on spec-style benchmarks. On Terminal-Bench 4.0, Mistral reports that Mistral Large 4 "improved by 13 percentage points to 41%" compared with its public preview. Mistral says this result is "taking the lead among open-weight models". The model also used 30% fewer output tokens. On the AA Coding Agent Index, Mistral reports 48%, "effectively tied" with GLM-5.3, "while using 47% fewer tokens per task on average."

Mistral's published charts on these benchmarks compare the model mostly with other open models. Its Surge Code human preference results point the same way as FrontierCode. Mistral says the model ranks "behind only Claude Opus 5.5 among the models evaluated."

How different organizations are responding

The sources show different responses to the same problem. Coding agents look stronger on spec tests than on real merge decisions.

Two limits apply to this table. First, every adopter quote comes from Mistral's own launch post, so it is marketing. Second, almost all of the quotes are about security or data sovereignty. No source reports a company that replaced a closed coding agent model with Mistral Large 4.

The harness research matters for open models. The paper reports that "Planning shifts from an accuracy scaffold for weaker models to a cost saver for stronger models". It also reports that "Predefined tools improve performance for models with weaker bash proficiency". The study used four unnamed models, not Mistral Large 4. It still suggests a direction. If you choose a weaker model, put more structure around it. Our post on harness A/B testing shows how to prove that a harness change helped.

Why fewer tokens do not close the quality gap

Token efficiency lowers the price of one agent run. A merged change also needs human review, and every rejected run still takes review time. The worked example below shows the effect. All numbers are illustrative.

Assume this:

• Model A costs $1 per run, and 20% of its runs are accepted.
• Model B costs $4 per run, and 45% of its runs are accepted.
• A reviewer spends 15 minutes on each run, at $100 per hour. That is $25 per run.

We use the two FrontierCode scores as acceptance rates only to make the example concrete. The real score is a weighted rubric, not a merge rate.

1. Find the runs per accepted change. Model A needs 1 / 0.20 = 5 runs. Model B needs 1 / 0.45 = 2.22 runs.
2. Find the token cost. Model A: 5 × $1 = $5.00. Model B: 2.22 × $4 = $8.89.
3. Find the review cost. Model A: 5 × $25 = $125.00. Model B: 2.22 × $25 = $55.56.
4. Add them. Model A: $130.00. Model B: $64.44.

Model A is 4 times cheaper per run, but it costs about twice as much per accepted change. If Model A's tokens were free, it would still cost $125.00. The review time decides the total. The result reverses only when review is very cheap, for example when an automatic test does the review. That is the case for spec-style tasks, and it is why the decision depends on the workload.

Which workloads can use an open weight model

The table below is our recommendation from the sources. It is not a finding from Cognition or Mistral.

A routing layer lets you use both. Send spec-heavy tasks to the open model and maintainer-grade tasks to the closed model. Our guide to LLM routing in production explains the setup. Before you self-host, read the license. Mistral says the weights are open but its post does not list the license terms. See open-weight model license terms.

Which Mistral reasoning mode to start with

Mistral describes three reasoning modes. The third column is our inference for coding work. Mistral does not give it.

Do not assume that High is always better. Cognition found that "models can score lower at higher reasoning effort". From low to max effort, scope failures rose from 3.8% to 4.9%. Thoroughness failures fell from 26.0% to 15.3%. That data comes from four closed models only. No source gives Mistral Large 4 results by mode, so test it yourself. Our post on thinking budgets explains how to set effort per task.

How to test Mistral Large 4 on your own repository

Public benchmarks give a starting point. Your own merged pull requests give the answer for your team.

1. Collect 30 to 50 recent merged pull requests from the repository where the agent will work.
2. Write each one as a task, with the issue text as the only input.
3. Ask the maintainer of each area to list the blocking criteria for each task.
4. Run Mistral Large 4 at Medium and High, and run your current closed model, on every task.
5. Run each configuration at least 3 times, because a single run hides variance.
6. Record tokens, wall time, and reviewer minutes for every run.
7. Calculate the cost per accepted change, as in the worked example.
8. If the open model's cost per accepted change is lower for a task type, route that task type to it.
9. If it is higher, keep the closed model for that task type and test again on the next model release.

For a deeper method, see workflow-specific evals and reviewing AI-generated code at scale.

Put the right model on each coding task

The model choice for coding agents decides how much engineering time goes to review and how much goes to new work. OpenNash puts forward-deployed engineers on your team, and we work with Fortune 2000 companies. A typical engagement for this decision has these steps:

1. Build an eval set from your own merged pull requests, with blocking criteria from your maintainers.
2. Measure the cost per accepted change for open weight and closed models on each task type.
3. Set up routing so that each task type goes to the model that wins on your data.
4. Deploy self-hosted models where data residency requires it, with review gates and audit trails.
5. Train your engineers to run the evals again when a new model is released.

At the end, you own the eval set, the routing rules, the harness, and the deployment.

Book a call with an OpenNash forward-deployed engineer or email us at hello@opennash.com. In the first session, we review your coding agent workloads. We show which tasks an open weight model can take at lower cost per merged change.