Frontier vs. Open-Weight Models: What Our Legal Evals Show
We ran twelve open-weight models through the evals that gate every model change at August. On every benchmark, a frontier model matched or beat the best of them on quality, and frontier models were faster than every open-weight model that came close.

Share on:
Open-weight models, whose weights are public so anyone can host them, have improved quickly this year. The case for them is usually made on price: per token, many cost a fraction of what the closed frontier models from the major labs do. For a legal AI product, that only matters if the cheaper model can do the work. A model that is cheap per token but needs more tokens, takes longer and gets more answers wrong isn't cheaper in any way a law firm would recognize.
So we tested twelve of them against five frontier models, GPT-6.1 Terra, GPT-6.1 Sol, GPT-6 Luna, Claude Opus 5.5 and Claude Sonnet 5.5, on the evals behind every model decision at August.
GPT-6.1 Terra matched or beat Kimi K3, the strongest open-weight model that ran every benchmark, on six of seven. On those six, Kimi K3 cost 1.8–3.5× as much per task and took 2.2–4.5× as long. The newest releases close much of the quality gap: MiMo-V2.6-Pro roughly matched Terra on the benchmarks we could run it on, but took 18–20× as long.
Why open-weight models deserve a serious look
The appeal of open weights in legal work goes beyond price:
Control over data. A firm, or a vendor serving it, can run open weights inside its own cloud, so documents never leave infrastructure it controls.
Specialization. Open weights can be fine-tuned on a firm's own precedents, playbooks and drafting style, in ways closed models allow only partly.
Stability. A model you host doesn't change underneath you. Closed models are updated and retired on their maker's schedule.
Price and choice. Dozens of hosts compete to serve the same weights, which pushes prices down and avoids depending on a single vendor.
How we evaluate
Every eval runs through the same agent runtime that serves our customers: the same prompts, tools, context management and model wrappers. Search and reading tools make real calls instead of returning canned results. We measure a model the way a lawyer would experience it inside August, not as a bare API call.
The suite has seven benchmarks, each built around work lawyers actually bring to us (Table 1).

The four document benchmarks are built from anonymized, realistic matters from participating partner firms. Documents drafted from the same template sit side by side, so the distractors are natural ones: two versions of a clause that differ only in a date or a cap. Some questions have no answer in the documents at all, and the right response is to say so. The three behavioral benchmarks (Memory Management, Skill Recommendation and Template Adherence) are hand-written cases modeled on how lawyers use the product, in five languages.
A question that every model gets right, or that a model can answer without reading, tells us nothing, so each question has to earn its place:
Frontier models from three labs draft questions with quoted evidence.
Code checks that every quote exists in its source, and that answers meant to be absent really are.
At least two of three models must answer correctly from the evidence alone.
Any question that can be answered without the documents is dropped.
Of roughly 550 drafts for Long-Context QA, 333 survived. Our comparison runs use the 96 that actually separate models.
Grading is done by code wherever possible. Answers are normalized by type: dates in any language, amounts in either decimal convention, business days kept distinct from calendar days. Where a model judge is needed, any evidence it quotes must appear word for word in the answer. The Template Adherence grader reads the Word file's XML directly. Every trial records cost, latency and token use alongside the score.
The models and their prices
We put six of the twelve open-weight models through the full suite: Kimi K3, the strongest of them, plus GLM 5, DeepSeek V3.2, MiniMax M2.5, Kimi K2.5 and gpt-oss-120b. Six newer releases aren't yet available on the infrastructure our document benchmarks run on, so they ran only the three behavioral benchmarks: Kimi K3.1 Pro, DeepSeek V4 Pro, GLM 5.3, Qwen3.8-2.4T, Mistral Large 4 and MiMo-V2.6-Pro. All twelve ran as released, without fine-tuning.
On list price, most of them look cheap (Figure 1, Table 2). MiniMax M2.5, DeepSeek V3.2 and gpt-oss-120b charge 70–90% less than Terra per input token. The exceptions are Kimi K3, Kimi K3.1 Pro and Qwen3.8-2.4T, at about twice Terra's input price; Kimi K3 also charges three times its output price. GPT-6 Luna, the small frontier model, costs the same per token as the cheapest of them, and Claude Opus 5.5 is the most expensive model here, at four times Terra's price.
Open-weight models are sold by many hosts at different prices. To give them every advantage, we price each at the lower of its maker's own price (what the company that built the model charges) and the median across hosts, and credit it with the better of its own prompt-cache hit rate and Terra's. Where a model's maker charged us less than its list price, we use what it charged. The Claude models get the same cache credit. Every cost below is on that basis. Every model ran each benchmark once, at its low reasoning-effort setting unless noted.


Results
Agentic Document Search is the clearest single view, because it exercises the whole agent: searching, reading and answering. Figure 2 plots each model's accuracy against its cost per successful result: here, the cost of one correct answer.

Every open-weight model is less accurate than GPT-6.1 Terra, and all but one cost more per correct answer. Kimi K3 comes closest at medium effort, with 90% correct. It still costs 2.1× as much per question as Terra at the same effort, takes 3.6× as long, and trails Terra's 94%.
gpt-oss-120b costs a tenth as much as Terra per question, but it answers only 17% of questions correctly. It's cheap because it rarely does the work. At the same price per token, GPT-6 Luna answers 74% correctly, at about a quarter of gpt-oss-120b's cost per correct answer. Every model on the line in Figure 2 is a frontier model.
Claude Opus 5.5 and Claude Sonnet 5.5 each answer 83% correctly: ahead of Kimi K3 at the same effort, behind Terra and Sol. Both answer faster than Terra, and Sonnet 5.5 costs slightly less per question.
Figure 3 shows the frontier models and the six open-weight models we ran on the full suite. Below Kimi K3, the open-weight models fall away quickly:
GLM 5, DeepSeek V3.2 and MiniMax M2.5 couldn't attempt Long-Context QA at all, because the documents don't fit in their context windows.
On Memory Management and Skill Recommendation, which test whether an agent makes the right judgment calls, they scored between 0% and 22%.
Kimi K2.5 rarely produced an answer at all.

The frontier models aren't interchangeable. The GPT-6.1 models lead both question-answering benchmarks, and the Claude models lead Multi-Document Retrieval.
Kimi K3 is the only one of the six open-weight models that competes on quality, and where it ties, it loses on cost and speed. Context Compaction is the one benchmark where Terra trails it; there, Sol and both Claude models are more accurate, and Kimi K3 wrote no summary at all in 10 of 32 conversations. Figure 4 compares Kimi K3 with Terra benchmark by benchmark: it takes longer on every one, and costs more on all but Context Compaction.

The gap isn't just a slow host. Even at its low effort setting, Kimi K3 writes far more to reach a worse answer: on Agentic Document Search it produced 798 output tokens per question against Terra's 273, and answered 11.5 points fewer correctly. Turning reasoning off makes it fast, but then it stops making judgment calls: correct saves on Memory Management fell from 77% to 13%.
The newest releases
The six newer open-weight releases are a clear step up from the generation before them (Table 3). The strongest, MiMo-V2.6-Pro, was level with Terra on Memory Management, ahead of it on Skill Recommendation and a little behind on Template Adherence. DeepSeek V4 Pro matched Terra on Template Adherence, and Kimi K3.1 Pro edged past it on Skill Recommendation.
What none of them matched was speed: every newcomer was slower than Terra on every benchmark. MiMo-V2.6-Pro took 18–20× as long, a median of 1.5 minutes per memory decision and 7.4 minutes per document against 5 and 25 seconds. The other newcomers each fell well behind on at least one benchmark, and GPT-6.1 Sol still led Skill Recommendation by a wide margin.

Price per token vs. cost per successful result
Price per token is what a model charges. Cost per successful result is what the work costs: total spend divided by the number of results that were actually usable. It's the number that matters, because a failed result is still paid for. Template Adherence is the one benchmark every model completed, including the newest releases, so it gives the cleanest comparison of the two. Figure 5 puts each model's list price next to what it cost to produce one successful document, meaning one that passed every check. That is a stricter bar than the Template Adherence score above, which counts individual checks: Terra passes 96.4% of checks, but only 24 of its 28 documents pass every one.

Two things turn a cheap token into an expensive result:
Failure. A document that breaks the template has to be redone. Only 36% of MiniMax M2.5's documents passed every check, one in 28 of gpt-oss-120b's, and none of Kimi K2.5's.
Volume. The weaker models flailed. GLM 5, DeepSeek V3.2 and MiniMax M2.5 each read 0.96–1.26 million input tokens per document, against Terra's 0.43 million.
The upshot: DeepSeek V3.2 has about a quarter of Terra's list price but cost 3.6× as much per successful document. MiniMax M2.5, at 30% of the price, cost twice as much, and gpt-oss-120b, at a tenth, cost 5.5×.
The newest releases narrow that gap. MiMo-V2.6-Pro and DeepSeek V4 Pro, at their makers' prices, cost $0.072 and $0.076 per successful document against Terra's $0.087, but took 18× and 4.5× as long per document. For a lawyer waiting on a draft, that's the difference between a pause and a coffee break. GPT-6 Luna was cheaper still, at $0.031, though only 64% of its documents passed every check. The Claude models cost the most of the frontier models here: $0.26 for Sonnet 5.5 and $0.50 for Opus 5.5.
Figure 6 shows the trade-off. On cost, the Pareto frontier runs from GPT-6 Luna through MiMo-V2.6-Pro and DeepSeek V4 Pro to GPT-6.1 Terra. On time, only Claude Sonnet 5.5 is faster than Terra, and it scores lower; every open-weight model is both slower and less accurate.

Counting the cost of failure
Cost per successful result is a better measure than price per token, but it still flatters a cheap model. It assumes every failure is caught and simply run again. In legal work, a failure costs far more than its tokens: someone has to notice the broken clause or the invented answer, and then fix it. A fuller measure is the expected cost of a task:
expected cost = model cost + P(failure) × cost of a failure
The first term is measured in cents. The second is measured in a lawyer's time, and it dominates. Table 4 shows what a failure would have to cost for each model to beat GPT-6.1 Terra on expected cost.

For every model that is cheaper than Terra but fails more often, the break-even is under 40 cents. If a lawyer spends even a minute catching and fixing a failure, the cheaper model costs more in practice.
Template Adherence has the exceptions. Terra failed four of 28 documents; DeepSeek V4 Pro, Kimi K3 and Qwen3.8-2.4T each failed one or two. With so few documents that edge is within noise, and each took 2.2–4.5× as long per document, but this is the benchmark where open-weight models come closest.
Answering when the answer isn't there
For a lawyer, the most damaging error isn't a missing answer. It's a fluent, cited answer to a question the documents don't actually answer. Our question set includes near-misses whose answers sit only in a document the model never sees. Figure 7 shows how often each model answered them anyway.

Terra answered none of these questions, and Sol and both Claude models answered 4%. Kimi K3 answered 15–21%, depending on effort: several times as often, in the error that's hardest for a lawyer to catch. The small frontier model, GPT-6 Luna, is no better than Kimi K3 here.
Where we stand
We don't have a side in the open-versus-closed debate; we have evals. August uses whichever models win them, and the same evals will tell us when that changes. On today's evidence, this is what we believe:
Price per token is the wrong number to compare. What matters is the cost of a successful result, failures included. By that measure, frontier models come out ahead on most of what we tested.
The remaining gap is speed and knowing when not to answer. The best open-weight models can now fill a template about as well as a frontier model, but they take several times as long, and they more often answer a question the documents don't answer.
Open weights could win on cost-efficiency in narrow jobs first. The likeliest openings are high-volume steps where nobody is waiting, a failure is cheap to catch and a model can be fine-tuned for a single task, rather than the agent a lawyer works with.
Privacy doesn't require open weights, but only if you trust the labs. Frontier labs offer enterprise terms, including zero data retention, that promise customer data is neither stored nor used for training. A firm that trusts those promises gets data control from its contract rather than its choice of model. A firm that doesn't can only be sure by hosting open weights itself.
Caveats and next steps
We tested the open-weight models as released. Fine-tuning one on legal work could change its results, and we didn't test that.
August's prompts and tools were developed on frontier models, and a harness tuned for another model might narrow some gaps.
Frontier models helped build some of the benchmarks, which could favor them. They drafted and verified the Long-Context QA questions, and Claude models wrote the Multi-Document Retrieval answer key and grade it, on the benchmark where they score highest.
Every run is a single trial. Kimi K3's gaps on Long-Context QA and Agentic Document Search are larger than the variation between repeated frontier runs, but on Multi-Document Retrieval they are within it.
Open-weight latency depends on where a model is served, and we measured each model in one place. The output-token counts suggest most of the gap is the model itself.
The newest releases ran on their makers' own APIs, or on a host serving full-precision weights. Where a host mangled tool calls, we re-ran the model elsewhere and report only the clean run.
When the newest releases become available on the infrastructure our document benchmarks run on, they'll go through those benchmarks too, and we'll publish what we find.





