Mistral Large 4 vs Kimi K3.1 Pro: refusal or removal for cyber risk

Moonshot AI and Mistral AI each released a frontier-scale open-weight model in the past three weeks, and each said something specific about cyber misuse. They chose opposite controls. Moonshot removed cyber-related data from Kimi K3.1 Pro’s training and reported how much exploit capability that cost. Mistral trained Mistral Large 4 on vulnerability work, measured how often it refuses malicious cyber requests, and kept a more capable cyber variant behind vetting and contracts. The short answer to which approach works better on downloadable weights: the published evidence cannot say, because each company measured a different property, and neither has published a test of what a determined user can undo after download.

That gap matters more for open weights than for an API. Once the files are public, the only safeguards that travel with them are the ones inside the parameters. Moonshot’s choice changes what the model has learned. Mistral’s choice changes what the model is willing to do, plus who gets the stronger variant. Research on fine-tuning suggests these behave differently under tampering, but that research was done on other models and other domains.

The Frontier Wire has not run either model. This comparison reads the two launch posts, both Hugging Face model cards and licence files, Mistral’s technical report, the UK AI Security Institute’s July measurement of the open-weight cyber gap, press reporting that carried comments from Mistral, and three research papers on how safeguards survive fine-tuning. Every number below is attributed to whoever measured it.

The short version

• Moonshot removed cyber data from mid-training and reinforcement learning. On its own exploit test the release scored 81.0% at pass@3, against 90.5% for the same pipeline without removal and 33.3% for Kimi K3.
• Mistral reports a 96.6% average refusal rate on the cyber subsets of three public benchmarks, measured internally. It published no over-refusal rate and no exploit-development score of the same kind.
• Mistral told CSO that its guardrail updates reach its API and future releases, not weights already downloaded. Its specialised cyber variant is not public.
• Neither company published a fine-tuning or tampering test, and no third party has yet published an evaluation of either released checkpoint.

The criteria before the verdict

A fair comparison has to fix what is being compared before reading any scores. The criteria below follow from one question: what protection does a person who downloads the weights actually receive, and how do we know?

The table does not crown either approach. It shows that the two companies answered different questions. Moonshot answered “how much offensive capability did we choose not to ship?” Mistral answered “how often does the shipped model say no?” A reader who wants to know which download is harder to misuse needs both answers for both models, and the record contains one of each.

Reconstructed timeline

The dates need care, because the API launches, the weight uploads and the reports did not happen together.

Two consequences follow. First, Mistral’s 29 October comparisons include Kimi K3 but not K3.1 Pro, whose weights were not yet public. Second, for about nine days Kimi K3.1 Pro existed only behind Moonshot’s API. Moonshot ties its data removal to “the open-weight release of K3.1 Pro”; its post does not say whether the API model served from launch day is the same checkpoint.

Kimi’s approach: remove the data

Moonshot’s account is unusually specific about cause. During development it saw vulnerability-research performance rise steadily as post-training scaled. Tracing the trend, it found cyber-related content inside the general mid-training corpus and cyber-related tasks inside the reinforcement learning environment mix. In preliminary experiments on internal models, the company says, keeping that data at either stage would have raised downstream cyber capability substantially.

Its response, in its own words: “we identified and removed cyber-related data from both training stages, while retaining the general-purpose coding tasks needed for model usefulness.” The post does not describe the classifier or rules used to identify cyber data, how much data was removed, or how the line between cyber content and general coding was drawn. Those are the details an outside reviewer would need to judge whether the removal was thorough or cosmetic.

The measurement is xdev-bench-hard, which Moonshot built in house. Its description: expert-curated, source-code-based vulnerability discovery and exploitation tasks, most reproducing recently disclosed vulnerabilities in software such as the Linux kernel, Node.js, JavaScriptCore, Exim and VirGL, with mitigations enabled. Each task supplies the vulnerable source, a short bug hint and the binary. The agent must develop an exploit, achieve arbitrary code execution in a separate sandbox and retrieve a secret string. Moonshot reports pass@3 over a random subset of runs, with no hard timeouts or output-token limits. K3.1 Pro ran in Moonshot’s Kimi Code harness; GLM-5.3 and DeepSeek-V4.1-Flash-1017 ran in Claude Code.

On that test, per Moonshot’s chart: GPT-5.6 Sol (labelled a third-party evaluation) 100, the K3.1 checkpoint without cyber risk reduction 90.5, DeepSeek-V4.1-Flash 85.7, Kimi K3.1 Pro 81.0, GLM-5.3 57.1 and Kimi K3 33.3.

Moonshot’s interpretation is measured: “The comparison shows that data removal meaningfully reduces cyber capability relative to what the same training pipeline would otherwise produce.” It also concedes the other half. K3.1 Pro improves substantially over its predecessor, which Moonshot reads as general coding and long-horizon debugging skill transferring to cyber tasks even after removal. Data removal slowed the climb. It did not stop it.

What 90.5% to 81.0% measures

Moonshot does not state how many tasks xdev-bench-hard contains. The published figures allow an inference. Every score Moonshot reported on the test, 100, 90.5, 85.7, 81.0, 57.1 and 33.3, is exactly a multiple of 1/21 rounded to one decimal: 21, 19, 18, 17, 12 and 7 out of 21. A score of 100 fits any grid, but the chance that five other arbitrary one-decimal percentages all land on this one is very small. This is an inference from rounding, not a disclosed fact; the set could be a multiple of 21, or the scores could be computed in a way that happens to produce these values.

Take 21 tasks as an explicit hypothetical input and the 9.5-point gap becomes two tasks. The unmitigated checkpoint solved 19; the release solved 17. That changes how much weight the comparison can bear.

The intervals are this publication’s arithmetic on the hypothetical denominator. Because both Kimi checkpoints ran on the same tasks, a paired test is more appropriate than comparing intervals. If the release failed exactly two tasks the unmitigated model passed, and passed none it failed, an exact McNemar test gives a two-sided p-value of 0.5. On this reading, the published comparison is consistent with a real reduction but cannot by itself distinguish it from run-to-run variation. The jump from K3 to K3.1 Pro, ten tasks, is the robust result in the chart.

Our analysis of benchmark sizes makes the general point: on small task sets, gaps of a few points are often noise. Moonshot says the benchmark is “calibrated to provide meaningful differentiation at our current capability level”, and pass@3 with no time limits is a generous setting for an attacker. A disclosed task count, per-task results and repeated runs would settle whether two tasks is what the removal bought.

Kimi’s other cyber numbers

The model card adds two external benchmarks, with comparison scores Moonshot took from the other developers’ reports. On ExploitBench (official harness, 300-turn budget, pass@5 maximum-capability score out of 16 flags), K3.1 Pro scores 74.7, against 100 for GPT-6 Astra, 91.0 for Claude Opus 5.5, 83.0 for Claude Fable 5.1 and 54.4 for GLM-5.3. On ExploitGym (six-hour wall-clock budget per task, Kimi Code harness), it scores 20.7, against 42.4, 34.5, 30.4 and 15.0 respectively.

These support Moonshot’s claim that K3.1 Pro “remains behind that of frontier proprietary models”. The stronger version, that it trails models “publicly available for at least three months”, is Moonshot’s characterisation and the post does not name which models it means. AISI’s July finding gives an outside reference: across a 70-task suite and simulated network ranges, the most cyber-capable open model it tested, GLM-5.2, matched closed models released 4 to 7 months earlier. AISI has not published results for either model in this comparison.

Mistral’s approach: train defence, measure refusal

Mistral did the opposite with its training data. Its technical report describes an automated pipeline that converts publicly disclosed vulnerabilities into executable repository-level environments, spanning “tens of thousands of CVEs across roughly 600 weakness classes”. Test suites probe each vulnerability with adversarial exploits and standard inputs. “A candidate patch receives positive reward only if it completely neutralizes the exploit while preserving existing test-suite passes.” The reward targets patching, but the environments contain working exploits, and the policy is asked both to remediate known flaws and to identify and fix undisclosed defects given a weakness class.

The safeguard is behavioural. The launch post says the model’s built-in guardrails distinguish defensive from malicious offensive tasks, and that Large 4 “handles these legitimate defensive security tasks without unnecessary refusals.” The technical report lists safety as one domain in its reinforcement learning mixture but does not describe how refusal behaviour was trained.

The headline figure, 96.6%, is defined precisely in the report. It is the unweighted mean of three numbers, each measured internally on a benchmark’s cybersecurity subset: JailbreakBench safe-response rate 97.3%, StrongREJECT refusal rate 92.6%, and 100 minus the AgentHarm harm score, 100.0%. The mean of those is 96.63. The preview scored 95.3, according to the value in the post’s chart.

Three limits apply. First, these are refusal rates on known harmful prompts, not measurements of what the model can do when it complies. A model can refuse often and still be highly capable on the requests it accepts or after its refusals are removed. Second, the subset sizes are not stated. The JailbreakBench figures in Mistral’s table for all seven models are consistent with a 37-prompt denominator, which would make Large 4’s 97.3% one failure; again, an inference from rounding. Third, no over-refusal rate is reported, so “without unnecessary refusals” is a claim without a number. Gartner’s Niyati Daftary, quoted by InfoWorld, put the general caution plainly: “Mistral’s 96.6% figure tells you what their model does in their test scenario, not how it will behave with your data.”

Checking Mistral’s tables against its claims

Mistral’s post calls 96.6% “the highest we measure among open models.” The report’s Table 5 includes Muse Spark Open, with components of 100.0%, 90.7% and 99.4%. Their mean is 96.7. Given rounding in the one-decimal components, the fair reading is a tie with Large 4, not a lead for either. Muse Spark Open does not appear in the post’s refusal chart.

The post’s chart and the report’s table also disagree on competitors. The chart’s embedded value for Kimi K3 is 93.6. Averaging Kimi K3’s components in Table 5 (86.5, 77.8 and 100.0) gives 88.1. The chart gives GLM-5.3 87.1, where its Table 5 components (81.1, 87.0 and 92.2) average 86.8, a gap too large for rounding. The chart’s DeepSeek-V4-Pro bar reads 50.7, while the table’s DeepSeek V4-Pro 0813 components (64.9, 77.4 and 95.0) average 79.1; the chart label carries no version date, so these may be different checkpoints. The report is the more detailed document, so this article uses its table. Mistral has not explained the differences.

On capability, the report’s Table 4 shows Large 4 at 50.9% on the Artificial Analysis Cyber Index, which Artificial Analysis measures. DeepSeek-V4.1-Flash-1017 scores 56.8% on the same index in the same table, and also leads on CWE-Bench-AA (69.2% against 58.0%) and DeepsecBench-AA (24.9% against 16.0%). Large 4 leads on CyberGym-E2E-AA (79.0% against 76.3%) and on Cybench, run internally (90.0% against 87.5% for GLM-5.3 and Kimi K3). The post’s own Cyber Index chart also places MiMo-V2.6-Pro, which is not in Table 4, at 56% against Large 4’s 51%, and InfoWorld noted that both DeepSeek V4.1 Flash and MiMo-V2.6-Pro score higher. The model card’s description of Large 4 as “state of the art among open-weight models for security work” is therefore stronger than Mistral’s own table supports on three of its four Artificial Analysis measures.

The gated variant and guardrails that stay behind

Mistral’s second control is distribution. The post says enterprise design partners such as Orange Cyberdefense have preview access to “a specialized variant of Mistral Large 4 optimized for advanced cybersecurity workflows, including penetration testing, vulnerability assessment, and red-teaming.” The conditions: “Access is subject to rigorous vetting and contractual safeguards, with testing restricted to owned or explicitly authorized systems in controlled environments.” Mistral has published no benchmark for this variant, and its cybersecurity industry page does not mention it. Closed labs have taken the same vetting route for their own cyber models; our comparison of gated cyber models covers how OpenAI, Google and Anthropic run those programmes.

The third control is monitoring, and Mistral was candid about what it found. “Though we observed the model refusing to comply with black-box attacks against live targets, we see a potential for uplift to conduct real attacks. We are modifying our guardrails accordingly, and will continue updating them in the months to come.”

The question is which copies those updates reach. Pierre Stock, Mistral’s vice president of science, told CSO that later changes “are applied to our managed API and future model releases,” as InfoWorld reported on 30 October. That is the structural fact of open weights: the monitoring that surfaced the concern happens on Mistral’s API traffic, and the fixes land there too. The Hugging Face commit history after the initial release lists README and technical-report updates; no commit is titled as a weights change.

So Mistral’s design has three layers, and only one ships in the download. Refusal behaviour travels with the weights. The gated variant and the evolving guardrails stay with Mistral.

Why the layer matters once weights are public

AISI’s July report states the general problem. Open release removes options such as monitoring, classifiers, user bans and model withdrawal, and “open weight release therefore creates a persistent and irreversible risk of misuse.” Both companies’ arguments for release rest on defenders benefiting more than attackers; Moonshot says broad access helps teams audit code “including legacy systems that rarely receive dedicated security reviews.” AISI frames the 4-to-7-month gap as a window for defenders to prepare.

Within the weights, research gives reasons to expect refusal and data removal to behave differently under tampering, with caveats on transfer.

• Refusal is fragile under fine-tuning. Qi and colleagues reported in 2023 that fine-tuning GPT-3.5 Turbo on 10 adversarial examples, at a cost under $0.20 through OpenAI’s API, removed its safety behaviour for most harmful instructions, and that even benign fine-tuning degraded it. With open weights, no API stands between the user and the fine-tune.
• Refusal can be edited directly. Arditi and colleagues found that across 13 open chat models up to 72B parameters, refusal was mediated by a single direction in the model’s activations; removing that direction stopped refusals. Whether the result holds for trillion-parameter mixture-of-experts models trained with reinforcement learning is untested in public.
• Removed knowledge is harder to restore, but not unreachable. The Deep Ignorance paper (O’Brien et al., 2025) pretrained 6.9B models with biothreat text filtered out and reported resistance to adversarial fine-tuning for up to 10,000 steps and 300M tokens, more than an order of magnitude better than post-training baselines. The authors add that filtered models can still use dangerous information supplied in context, for example through search tools.

Read together, the papers favour Moonshot’s layer on durability and warn about both. Data removal is the more tamper-resistant kind of safeguard in the published experiments, but Kimi’s removal was partial by design: 81.0% on an exploit-development test is a lot of retained capability. And an agentic model with tool access can retrieve exploit write-ups at run time, which is precisely the gap Deep Ignorance flags. There is a further transfer gap. Deep Ignorance filtered pretraining data, while Moonshot describes removal at mid-training and in reinforcement learning, and says K3.1 Pro is a post-trained version of Kimi K3. Its post does not say whether cyber material was filtered from K3’s pretraining corpus. None of these studies tested cyber capability at this scale, so they predict direction, not magnitude.

Models both vendors measured

Neither company evaluated the other’s release. Three models do appear in both companies’ cyber tables, which allows a limited cross-reference. The worksheet below collects them. Different benchmarks, harnesses and measuring parties mean values in different columns cannot be compared to each other.

Two patterns hold within columns. Both vendors rank DeepSeek-V4.1-Flash-1017 above GLM-5.3 and Kimi K3 on cyber capability. And each vendor’s own headline cyber table shows its new release below DeepSeek-V4.1-Flash-1017 on at least one measure: Large 4 on the Artificial Analysis Cyber Index, K3.1 Pro on xdev-bench-hard, though the harness differed. If those measures are representative, neither release extends the open-weight cyber frontier that an already-public model set. That fits Moonshot’s stated principle of not extending “the absolute frontier of cyber capability”, and it is a fact about the field as much as about either safeguard.

The worksheet also exposes the missing cells. Nobody has published a refusal rate for K3.1 Pro or an exploit-development score for Large 4 on a shared test. Reading a model card like an auditor means treating those blanks as findings.

Licences as a control layer

A licence is another place a developer could restrict misuse. Neither does so for cyber, and both Hugging Face repositories carry the metadata tag license: other, so the terms have to be read in the files themselves.

The Mistral Ecosystem Open-Weight License is a modified MIT licence. It grants rights to use, modify, further train and distribute. Its one substantive condition concerns hosting: any licensee whose group “operates a Model as a Service business” must take a separate commercial licence before commercial use. Internal use and use through Mistral’s official products are exempt. The model card adds a line that the model must not be used to infringe third-party rights. There is no acceptable-use policy and no cyber clause.

The Kimi K3.1 Pro License, which this publication read in full, is also MIT-derived. It adds that “Licensee’s use of the Software must comply with applicable laws and regulations.” The hosting condition applies only above a threshold: a Model-as-a-Service business whose group revenue exceeds $20 million over any consecutive 12 months needs a separate agreement. Products above 100 million monthly active users or $20 million in monthly revenue must display “Kimi K3.1-Pro” prominently. Internal use, and use through Moonshot’s products or “certified inference partners”, are exempt from both. There is no cyber-specific prohibition.

For cyber risk, the licences matter in one way. Both conditions point at commercial hosts, the parties most likely to run monitoring and abuse detection on their own traffic. Neither licence obliges a host to do so. Our survey of open-weight licensing maps how these clauses compare with the wider field.

What neither release has shown

The comparison can be definitive only about the evidence that exists. Several missing pieces would change the conclusion.

1. A tampering test. Neither company reports what happens to refusals or exploit success after a fine-tune on cyber material. For Mistral, this is the decisive test, because its main shipped safeguard is behavioural. For Moonshot, it would show whether removal resists recovery as the Deep Ignorance results suggest.
2. Matched metrics. Moonshot published no refusal rate; Mistral published no exploit-development score comparable to xdev-bench-hard. Each company’s headline number cannot be read against the other’s.
3. Benchmark disclosure. xdev-bench-hard is unpublished. Its task count, per-task outcomes and run variance are unknown. Mistral’s refusal subsets have unstated sizes.
4. Benign-task costs. Moonshot says general coding was retained; Mistral says unnecessary refusals were avoided. Neither number exists for defensive cyber work, which is the use both companies cite to justify release.
5. Third-party evaluation of the released weights. AISI said in July it planned to test Kimi K3. No independent evaluation of either release has been published, as far as this publication could find.
6. The gated variant. Mistral has published nothing about how much more capable the specialised variant is. That number would show how much capability Mistral judged too risky to release openly, which is the closest analogue to Moonshot’s 90.5%.
7. Which checkpoint runs where. Whether Kimi’s API model matches the uploaded weights, and how far Mistral’s API guardrails now diverge from its download, are not documented.

A proposed test

The following is a proposed evaluation, not one The Frontier Wire has run. It would put both safeguards on one scale.

Take each released checkpoint and build four arms: the weights as shipped; the weights after a fixed, small fine-tune on public exploit write-ups; the weights with a refusal direction ablated, following Arditi and colleagues; and the shipped weights with exploit material supplied in context through a search tool. In every arm, measure three things on public, disclosed task sets: exploit-development success (for example a Cybench-style suite), refusal rate on the same cyber prompt subsets Mistral used, and success on benign defensive tasks such as patching.

The prediction from existing research is that Large 4’s refusal rate falls sharply in the fine-tune and ablation arms while its capability stays the same, and that K3.1 Pro’s capability moves less under fine-tuning but rises in the in-context arm. If the results came out that way, the comparison would favour removal for downloadable weights and show the limits of both. If Large 4’s refusals proved durable, or K3.1 Pro’s removed capability came back after a small fine-tune, the conclusion would reverse.

What the comparison supports

Moonshot’s approach produced the more informative evidence. It reported a counterfactual: the same pipeline, with and without the control, on an exploit test. Its own numbers also show the control’s limit. The released model is far more capable at exploitation than its predecessor, and on a likely small task set the measured reduction may be two tasks.

Mistral’s approach produced the more conventional evidence and the more layered design. It reports a high refusal rate, discloses how the aggregate is built, keeps its most capable cyber model off the open internet, and admits that live monitoring revealed potential uplift. But two of its three layers stay with Mistral, and the one that ships is the kind research has repeatedly shown to be removable. Its own tables do not support a clean lead on either refusal or cyber capability among open models.

For a security team deciding which to download, the useful conclusion is narrow. Neither release has shown that its safeguards survive a user who wants them gone. Both companies say defenders benefit from open access, and both models sit, by their makers’ own tables, near rather than beyond the open-weight cyber capability already available. Our October ranking of open-weight models predates both releases but covers their nearest comparators, Kimi K3, GLM-5.3 and DeepSeek V4.1 Flash, on general capability. The cyber question will be settled by the tampering and matched-metric tests above, not by the launch posts.