Model merging versus distillation: what the new evidence compares

Model merging and distillation solve related problems by changing different things. Merging combines parameter information from existing models. Distillation trains a student using a teacher’s behavior. They can be alternatives, complements or stages in the same workflow, so a useful comparison begins with the required artifacts and the resource budget rather than a universal winner.

The November 2 P²-MOPD preprint explores a hybrid at Qwen3 scales of 1.7B and 4B across four domains. Its central idea keeps the student trainable while learning contributions associated with specialist parameter differences. The reported evidence belongs to that setup, not to arbitrary combinations of frontier models.

The Frontier Wire has not reproduced the experiments. This comparison reads the new work against earlier primary research on weight averaging, task vectors, interference and on-policy distillation. The aim is to explain what each method requires and which conclusions a matched experiment can support.

The short version

• Parameter merging needs a meaningful relationship among the artifacts being combined.
• Distillation needs a usable teacher signal and a student-training procedure.
• A hybrid can use parameter structure while continuing to learn from behavior.
• Final quality, preparation cost and deployment cost are separate comparison axes.

The criteria before the verdict

A method can be attractive because it is inexpensive to prepare, because it retains several skills or because it produces an efficient deployed model. Those advantages need not come together. The comparison should expose the tradeoffs before deciding which matters most for a particular task.

These are comparison criteria, not claims that every method in a category has the same implementation. A technique may relax a particular constraint, and its evidence should be examined on those terms. The category name alone does not determine compatibility or quality.

The new preprint is interesting because it crosses the table’s columns. Asking whether merging beats distillation is therefore too coarse to capture its contribution. The more precise question concerns what parameter structure adds when the student is already being trained on teacher feedback.

A merged model is not an inference ensemble

An ensemble runs several models and combines their predictions. Parameter averaging instead constructs a model whose parameters incorporate information from several sources. The resulting deployment can have a different resource profile because it need not execute every source model for each request.

Model soups studies weight averaging among fine-tuned models. Its findings belong to the tested architectures and settings. The broader conceptual distinction is that averaging weights and averaging outputs are different operations, even when their behavior can be related under particular conditions.

For a practitioner, that distinction affects where the cost is paid. An ensemble may retain several specialists and incur repeated inference work. A merge may pay preparation and selection costs, then run one artifact. Neither profile automatically dominates when quality and hardware constraints differ.

A comparison should therefore name the deployed object. Is it one merged checkpoint, one distilled student or a router choosing among several models? A result attributed to one model should not silently include an ensemble or another system that performs additional inference work.

Task vectors describe a particular relationship

The task-arithmetic paper represents a task-specific change as a difference between related model weights. That gives a concrete operation: take a base, identify the change associated with fine-tuning and combine such changes using stated coefficients.

This representation is meaningful because the artifacts have a relationship. Arbitrarily subtracting tensors with matching shapes does not establish that the difference encodes a comparable skill. The interpretation depends on alignment, provenance and the way the specialists were produced.

The useful mathematical intuition is a direction in parameter space, not a detachable module containing a complete subject. A model’s behavior is distributed across interacting parameters. Moving along a task vector changes the whole computation, and the effect need not remain isolated to the named task.

That is why a merge requires evaluation rather than only arithmetic. The operation can be inexpensive to specify while its behavioral consequences remain empirical. A coefficient that helps one task may hurt another, and a combination that works for one family may fail elsewhere.

Matching shapes is weaker than compatibility

Two checkpoints can share tensor dimensions without occupying compatible positions in a useful parameter coordinate system. Their training histories, internal representations and conventions may differ. A simple parameter operation assumes more structure than the fact that the files can be loaded together.

Related fine-tuned models often provide a clearer starting point because they descend from a common artifact. That does not guarantee a successful merge, but it makes the comparison more interpretable. The base-to-specialist changes have a defined reference rather than an arbitrary pairing.

A study should state which compatibility conditions it relies on and which it tests. Are tokenizers and architectures the same? Do the specialists share initialization? Were they trained with similar objectives? If a technique claims to bridge differences, examine the evidence for that bridge instead of assuming the category already includes it.

This also changes how a negative result should be read. Failure to combine incompatible artifacts does not show that all merging is ineffective. Success among closely related specialists does not show that unrelated large models can be combined by the same procedure.

Interference is a behavior problem with parameter causes

Combining useful changes can damage them. Some parameter updates may be redundant; others may point in conflicting directions. A final student can improve on an average while losing a capability that made one specialist valuable.

TIES-Merging studies interference in merging and proposes operations addressing small changes and sign conflicts. It provides a concrete example of a method responding to structure in parameter differences, rather than treating an unweighted average as the whole category.

The general comparison question is which conflict the method resolves. A technique may reduce one form of interference while leaving another. The resulting model still needs to be tested on the relevant tasks, including those that might be harmed by the intervention.

An explanatory paper should connect its mechanism to an observed pattern. If the method claims to preserve several skills, per-task results matter more than a single aggregate. A mechanism is more convincing when a controlled ablation changes the failure mode it was intended to address.

Sparsity changes the combination strategy

The DARE paper investigates dropping and rescaling parameter differences in related fine-tuned language models. Its relevance is that not every recorded weight change must contribute equally to a useful combination. The paper’s specific findings should remain tied to its tested models.

This opens a different axis from choosing one global mixing coefficient. A method can alter which parts of an update survive, how they are scaled and how conflicts are handled. The behavior of the final model depends on that entire construction, not just the label model merge.

A fair comparison should disclose any selection performed over these choices. Trying many sparsity levels or coefficient settings consumes validation opportunities and computation. A final merge may require no additional gradient training while still benefiting from a substantial search process.

The correct claim is therefore narrower than free capability. A method can be inexpensive relative to retraining and still have preparation, memory and selection costs. Those costs should be reported at the level relevant to the intended use.

Distillation transfers a behavioral signal

The original knowledge-distillation paper develops learning from model behavior as a way to obtain a more convenient deployed model. A teacher can provide information beyond a single hard answer, depending on what its interface exposes and what the training procedure uses.

The student does not simply receive a copy of a skill. It is optimized against a signal on selected inputs and outputs. What transfers depends on the training distribution, objective, student capacity and teacher quality. Behavior absent from the training evidence may not survive.

This can make distillation more flexible than direct parameter combination. The teacher and student need not always share the same architecture if the signal is defined through outputs. But some procedures require richer access, compatible token distributions or other implementation conditions.

A comparison should therefore say what teacher access is available. Text responses from a hosted API, token probabilities and local teacher weights are different resources. A method relying on one cannot be assumed available under another without changing the experiment.

The student must learn from the states it reaches

A student trained only on polished teacher trajectories may encounter different situations at inference time. An early mistake changes the subsequent context, and the student may lack useful supervision for recovering from that state. This is a distribution problem, not merely a shortage of examples.

Generalized Knowledge Distillation studies training on student-generated sequences with teacher feedback. The central contrast is between learning exclusively from a fixed set of demonstrations and receiving guidance on the student’s own behavior. Its experiments belong to the tasks described in that paper.

The analytical implication is that data generation and optimization are connected. Changing which trajectories the student visits changes the evidence it learns from. A comparison should identify whether it uses teacher-generated data, student-generated data or a mixture, and how the teacher evaluates those trajectories.

On-policy training adds practical costs. The system must generate sequences and obtain feedback during learning. Those costs can be worthwhile if they improve the deployed student, but they should not disappear from an efficiency claim that counts only optimizer steps.

Teacher quality and teacher disagreement

A teacher can be stronger than the student on one domain and weaker elsewhere. Several teachers can also disagree. Distillation needs a policy for deciding which signal applies to which example, or how competing signals are combined.

That policy is part of the method. Routing prompts to domain specialists assumes some way to identify the domain. Mixing teacher distributions makes another assumption about how their preferences should be balanced. A high-quality teacher set does not automatically provide an unambiguous objective.

The student may also inherit shared errors. Agreement among teachers is not independent confirmation when they share training data, architecture or a common misconception. A useful evaluation needs external ground truth or task-specific checks rather than treating teacher consensus as correctness by definition.

This is where deployment requirements matter. A user may value a student that preserves a reliable specialist behavior even at the expense of a small average gain elsewhere. The paper should expose that tradeoff so the reader can choose a weighting appropriate to the task.

What the new hybrid changes

The full P²-MOPD paper treats specialist parameter differences as learnable contributions while continuing to update student weights. That distinguishes the hybrid from a fixed merge and from a student whose learning path uses no adjustable parameter-prior contribution.

The important mechanism is the extra structure available to optimization. The method can move along directions associated with the specialists while retaining a broader trainable student. The parameter contribution and the behavioral objective are cooperating parts of one training procedure.

This does not imply that a specialist’s entire capability is contained in one interpretable coefficient. A coefficient controls a contribution within a defined parameterization. The model’s actual behavior still depends on the resulting weights, input and inference procedure.

The comparison should therefore avoid a misleading binary verdict. The work does not establish that a training-free merge universally defeats distillation. It explores whether an additional parameter-based mechanism helps a particular distillation setup under the tested conditions.

Initialization is part of the evidence

A student can begin from the base model, a merged checkpoint or a warm-start procedure. Those starting points may already differ in capability before the main training begins. Comparing only final scores can attribute an initialization advantage to the later method.

The clean causal question is what happens when the relevant starting point is held constant and one mechanism changes. A separate practical comparison can evaluate complete recipes, including initialization. Both are useful, but they should not be presented as the same experiment.

Training curves help reveal the distinction. A method may start stronger but learn at the same rate, or start similarly and improve faster. A final checkpoint alone can conceal which pattern occurred. Reporting the initial state and intermediate measurements makes the claim more precise.

The same applies to preparation budgets. If one initialization required additional teacher data, validation search or training, count that work in the complete-recipe comparison. A narrowly matched mechanism study can exclude shared preparation, provided it states the boundary clearly.

Steps, tokens and time measure different resources

Fewer training steps can be an important result, but a step is not a universal unit of work. It may include different rollout lengths, teacher evaluations, batch sizes or parameter operations. Comparing step counts without those details can confuse algorithmic progress with a changed amount of computation per step.

A useful resource account separates student generation, teacher scoring, backward computation, memory and elapsed time. It also identifies hardware and parallelism. The relative importance of each component depends on the method and the environment in which it is run.

This is particularly relevant to a hybrid. Additional structure may introduce overhead while improving the number of useful updates needed. The complete result could be favorable or unfavorable depending on the balance. A paper should measure that balance rather than infer it solely from an optimization curve.

A method can also trade training cost for inference cost. A more expensive distillation run may produce a smaller student that serves many requests efficiently. The appropriate comparison depends on the expected deployment volume, which should be stated rather than assumed.

Memory can determine which method is available

An algorithm’s arithmetic may be simple while its artifact requirements remain large. Merging several checkpoints requires storing or streaming their parameters and constructing the output. Distillation may require keeping teachers, student, optimizer state and generated data available during training.

The exact implementation can reduce those requirements through offloading, sequential processing or other techniques. A comparison should describe the actual strategy instead of estimating memory from model size alone. Different strategies can change elapsed time and hardware needs.

This matters for access to research. A method that uses fewer gradient steps may still be impractical for a team unable to hold the necessary artifacts. Another method may be slower but fit available infrastructure. Practical suitability is not a universal ranking of scientific merit.

A useful report would state peak requirements, persistent storage and any external service dependencies. If a commercial API supplies teacher feedback, the cost and availability of that service belong in the experiment’s resource boundary.

Deployment is a separate comparison stage

After training or merging, inspect the actual deployed artifact. Does it require extra routing, multiple models or a particular inference stack? Were learnable contributions folded into ordinary weights, or does execution retain additional machinery? The answer affects both reproducibility and serving cost.

A merged artifact can have the same nominal architecture as its sources while behaving differently. A distilled student can have a different size or architecture. A comparison should therefore measure task quality and resource use for the final system rather than infer both from the preparation method.

The expected number of requests also changes the economic decision. A one-time research tool and a high-volume service amortize preparation differently. A method that is attractive for one may be poor for the other without either result being inconsistent.

This is why preparation cost and deployment cost need separate rows in a decision table. Combining them into a single unlabeled cheapness claim can conceal the tradeoff the reader needs to make.

Recovery is relative to a reference

The new paper uses teacher-relative recovery to describe how much of a base-to-teacher advantage the student regains. That is a useful normalization for the specified comparison. It should not be read as raw accuracy or as a percentage of general intelligence.

The denominator matters. A small teacher advantage can make a modest score change large on a recovery scale. A large advantage can make a practically useful gain look smaller. Raw per-domain scores are needed to interpret the normalized summary.

The reader should also inspect the weakest retained domain. A high mean can conceal one capability falling below the starting model. Whether that is acceptable depends on the intended use, but the tradeoff should be visible before a universal success claim is made.

A comparison of consolidation methods should therefore preserve a capability profile, not only one number. The relevant question is which skills survived, which improved and which were lost under the stated budget and measurement procedure.

Capability outside the teacher domains

A student may recover selected specialist behavior while degrading unrelated capabilities. Testing only the teacher domains can miss that cost. A deployment-oriented evaluation should include relevant controls outside the consolidation objective, chosen before results are known.

Earlier catastrophic-forgetting research frames the wider difficulty of acquiring competence without losing what was already useful. The consolidation setting is not identical to every continual-learning problem, but it shares that need to inspect retained behavior rather than only new gains.

Mixed-domain tasks are another useful check. A model that performs well on each specialist benchmark separately may still fail when one request requires several skills together. Separate competence does not automatically establish reliable composition.

These follow-ups would extend the evidence. They should not be described as failures of the new method merely because a particular deployment test was not part of its research question. The fair interpretation names the tested scope and the additional claims still requiring experiments.

Validation search belongs to the method

A merge can be selected from many coefficient settings, and a distilled student can be selected from many checkpoints. If one method receives more opportunities to inspect validation performance, its final result may benefit from that search even when the underlying training procedure is unchanged.

A matched study should declare the selection budget and preserve the distinction between validation and final testing. Repeatedly choosing settings against the final test weakens its role as independent evidence. A useful artifact includes the candidate-selection rule, not only the configuration that won.

This is especially relevant when a simple method competes with a more elaborate one. The fair question is whether the added mechanism improves outcomes under a comparable opportunity to tune and select. Otherwise, a claimed algorithmic advantage may partly describe a larger search process. That process can be worthwhile, but it should be named and counted.

A decision that follows the available artifacts

A team with closely related specialist checkpoints and little training budget may reasonably begin with a merging approach. A team with output access to a strong teacher and a smaller target model may instead investigate distillation. A team with both kinds of access can evaluate a hybrid, provided it accounts for the combined resources.

These are starting points for experimentation, not guaranteed recommendations. The actual decision depends on the compatibility of the artifacts, the quality threshold, the deployment constraints and the cost of testing alternatives. A simple baseline can be valuable precisely because it makes the incremental benefit of complexity easier to measure.

Our overview of open-weight model releases provides context for artifact availability. Our account of the public multimodal research stack examines why training recipes and intermediate artifacts affect which comparisons researchers can reproduce.

The new evidence favors a precise conclusion: parameter structure and behavioral learning can be combined and studied under matched controls. It does not retire either merging or distillation as a category. The useful winner is the method that meets a stated capability requirement with a reproducible account of preparation, retained skills and deployment cost.