Some of the most capable open-weight models on Hugging Face’s leaderboards are not the product of a single training run at all. They are Frankenstein assemblies of other models, built by averaging or arithmetically combining the weights of several separately fine-tuned networks into one, with no additional training and no access to any of the original training data. The technique is called model merging. It costs almost nothing compared to training a model from scratch, and it has become part of serious post-training pipelines at a level of sophistication well beyond the leaderboard-climbing hobbyist projects that first made it popular.
The premise sounds like it should not work. Neural network weights are the product of a specific, path-dependent optimization process, millions or billions of numbers shaped by the exact sequence of gradient updates a particular training run happened to take. Averaging the weights of two models trained separately, even from the same starting point, sounds like it should produce structural nonsense, the numerical equivalent of averaging two people’s fingerprints and expecting a third valid fingerprint. It usually does not produce nonsense, and the reason why is a genuinely interesting piece of empirical deep learning that most coverage of merged models skips entirely in favor of just reporting the benchmark score.
Why Averaging Weights Doesn’t Just Break Everything
The property that makes merging possible at all is called linear mode connectivity. When two models are fine-tuned starting from the same pretrained checkpoint, rather than trained from independent random initializations, they tend to land in the same broad region of the loss surface, close enough in weight space that a straight line drawn between their two sets of weights stays in a region of similarly low loss for most of that line. That is not true in general for two arbitrary neural networks. It is specifically true for models that share a common ancestor, because the shared starting point means both fine-tuning runs are exploring the same neighborhood of the loss surface rather than two unrelated ones. Model merging as a field exists almost entirely because this property holds for fine-tuned variants of the same base model, and it mostly does not hold, or holds far less reliably, for models trained from scratch with different initializations.
Model soups, the paper that got the modern wave of interest started, demonstrated the simplest version of this directly: averaging the weights of multiple models fine-tuned from the same pretrained checkpoint with different hyperparameters produced a single model more accurate than any of the individual ingredients, at zero additional inference cost, because the merged model is still exactly one network of the same size. That result on its own would be a mildly interesting stability trick. What turned it into an entire subfield is what happened when researchers stopped averaging models fine-tuned on the same task and started merging models fine-tuned on completely different tasks, effectively trying to build one model that inherited multiple separate specializations for free.
The foundational evidence for merging came largely from computer vision and comparatively small models, not the billion-parameter LLMs the technique gets applied to today. A 2026 systematic study comparing merging techniques head to head on contemporary LLMs found that simple task arithmetic held up more consistently than the more elaborate interference-aware methods described below, which did not always transfer as cleanly from the smaller-scale settings where they were first validated. That does not invalidate the mechanism. It means the newer, more sophisticated techniques deserve more skepticism at LLM scale than their vision-model origins might suggest.
Task Vectors: Treating Fine-Tuning as Arithmetic
The idea that made merging genuinely useful rather than just a training-stability trick is called task arithmetic. A task vector is defined as simply the difference between a fine-tuned model’s weights and the pretrained base model’s weights, the specific direction and magnitude that fine-tuning moved the model in weight space to make it good at a particular task. Once you have that vector, the paper’s central claim is that these deltas behave enough like real vectors to support meaningful arithmetic: adding a task vector to the base model reintroduces that task’s capability, subtracting one suppresses it, and adding multiple task vectors from different fine-tuned models onto the same base model can combine their separate capabilities into one, without ever running a training step on the combination.
In practice this means merging is not really averaging entire models. It computes the task vector for model A minus the base, the task vector for model B minus the base, then forms a new model as the base plus a weighted sum of both task vectors. The base model’s own pretrained knowledge stays intact as the anchor point, and each task vector layers a specific fine-tuned skill on top of it. This reframing is why task arithmetic generalizes so much further than naive weight averaging: two task vectors pointing in genuinely different, non-conflicting directions in weight space can be summed with much less interference than averaging the raw fine-tuned weights directly, because the shared pretrained component is factored out before the combination happens.
Where Simple Arithmetic Breaks: Interference
Merging more than two or three task vectors this way runs into a specific, well-characterized failure mode called interference. Different fine-tuning runs push some of the same parameters in conflicting directions, and different runs also produce large numbers of small, mostly noisy parameter changes that do not carry much real task-specific signal but still add up when summed across several task vectors. Left unaddressed, both problems degrade the merged model’s performance as more models get added to the combination, which is exactly the opposite of what you want from a technique whose entire value proposition is combining more capabilities for free.
TIES-Merging addresses this with a three-step procedure its authors call Trim, Elect Sign, and Disjoint Merge. Trimming keeps only the largest-magnitude changes within each task vector and zeroes out the rest, on the reasoning that small parameter changes are disproportionately noise rather than meaningful task-specific signal. Electing the sign resolves direct conflicts: for every parameter position, TIES looks at whether the surviving task vectors agree on whether that parameter should move up or down, and keeps only the changes that agree with the majority direction, discarding contributions that push the opposite way. The final disjoint merge step then averages together only the parameter values that survived both the trimming and the sign election, producing a combined task vector with dramatically less internal conflict than a naive sum would have.
DARE, published under the title “Language Models Are Super Mario,” takes a different, almost counterintuitive approach to the same interference problem: it randomly drops out somewhere between 90 and 99 percent of the values in each task vector, setting them to zero, and then rescales the small fraction of surviving values upward to preserve the task vector’s overall expected magnitude. The premise is that most individual parameter changes from fine-tuning are redundant enough that a model’s task-specific capability survives having the overwhelming majority of them deleted, provided the remaining changes are rescaled to compensate. DARE is typically layered on top of task arithmetic or TIES rather than used alone, and the paper’s own experiments found it measurably improves results when combined with both, in some cases producing a merged model that outperforms one of the individual source models on that source model’s own specialty task, a result the authors describe as previously undocumented.
The Geometric Alternative: Interpolating Instead of Averaging
A separate family of merging methods sidesteps linear averaging entirely in favor of spherical linear interpolation, or SLERP, which treats the two models’ weight vectors as points on the surface of a high-dimensional sphere rather than points in ordinary flat space, and interpolates along the curved arc connecting them rather than along a straight line. The practical motivation is that high-dimensional weight vectors from trained neural networks tend to concentrate their magnitude in a way that makes the straight-line path between two of them pass through a region with a smaller effective norm than either endpoint, which can quietly degrade the merged model’s behavior even when linear mode connectivity technically holds. SLERP’s curved path avoids that dip, though it is an interpolation geometry, not a general performance guarantee: it preserves vector norm along the way, but that does not by itself guarantee lower loss or better downstream results than a well-tuned linear merge would produce. The original formulation only handles two models cleanly, which is why mergekit, the open-source toolkit that has become the de facto standard implementation for all of these techniques, extended it into a MultiSlerp variant capable of interpolating across more than two source models at once.
This Is Not Just a Hobbyist Leaderboard Trick
Model merging’s reputation took a real hit from Hugging Face’s Open LLM Leaderboard era, when a wave of low-effort merges, sometimes combining models with no clear complementary specialization at all, climbed the rankings on the strength of benchmark overlap rather than genuine capability gains, prompting real skepticism about whether merged models were doing anything beyond gaming evaluation sets. That history is worth stating honestly rather than glossing over, because it is the most common objection anyone technical raises the first time they hear about the technique, and it is part of why serious open-weight model rankings now weigh benchmark contamination and provenance as carefully as raw scores.
What separates that era from current practice is that merging has since shown up as a documented component inside frontier model training pipelines rather than only inside leaderboard-chasing community projects. Meta’s own technical report for Llama 3 describes model merging as part of its post-training process, used to combine checkpoints trained with different data mixtures, different hyperparameters, or different fine-tuning objectives into a single final model, the same underlying technique as a community mergekit recipe, applied with the calibration and evaluation discipline of a frontier lab rather than a weekend leaderboard push. Reports on Gemma 2’s development describe a similar checkpoint-averaging step in its own post-training pipeline. The distinction between a gamed leaderboard entry and a genuine capability improvement was never really about the algorithm. It was always about whether the models being merged had real, complementary, non-overlapping specializations to contribute in the first place, and whether the result was evaluated honestly rather than cherry-picked against the benchmark it was optimized to win.
What the Technique Still Can’t Do
The formal literature on model merging has a limitation worth stating plainly: most of it has been validated on relatively small models and a narrow set of benchmark tasks, and the field’s own researchers have flagged that results at genuinely large scale, the regime frontier labs actually operate in, are far less thoroughly studied than the volume of published merging papers might suggest. A technique that reliably combines two 7-billion-parameter models fine-tuned on well-separated tasks does not automatically imply the same reliability merging a dozen specialized experts at 70 billion parameters and up, and the honest state of the evidence is that this gap has not been closed, only narrowed by scattered individual results.
Merging also has a hard structural requirement that limits where it applies at all: the source models need architectural compatibility, meaning the same base model and the same parameter shapes, which is exactly why every mainstream merging technique from task arithmetic through TIES and DARE assumes a shared pretrained ancestor rather than working across genuinely different architectures. Merging a Llama-family fine-tune with a Mistral-family fine-tune is not a harder version of the same problem. It is a different problem these methods were not designed to solve, since there is no shared weight space for a task vector to even be defined against. That architectural constraint is also why model merging and the token-level routing used inside a mixture-of-experts layer solve genuinely different problems despite both being ways of combining multiple specialized components into one system. MoE keeps every expert as a distinct, separately addressable set of weights and routes individual tokens to a small subset of them at inference time. Merging collapses several separately trained models into a single, smaller set of weights before inference ever happens, trading the ability to selectively activate specific expertise for a system that costs nothing extra to run.
What Happens Next
The active research edge right now is less about inventing new merging arithmetic and more about deciding, automatically, which parameters from which source models actually deserve to survive the trim-and-merge process. Recent work on sensitivity-aware and curvature-aware merging methods tries to weight each parameter’s contribution by how much it actually matters to that source model’s task performance, rather than treating every surviving parameter after trimming as equally important. Whether that finer-grained approach scales to merging many models at once without reintroducing the same interference problem TIES and DARE were built to solve is still an open question in the literature rather than a settled result.
The practical trajectory is clearer than the research frontier. mergekit and its command-line recipes have made merging accessible enough that it functions as a real, cheap experimentation tool for teams without the budget for a full fine-tuning run, and its documented use inside frontier post-training pipelines means the technique has graduated from leaderboard curiosity to a legitimate, if still imperfectly understood, tool for combining separately trained capability without paying for a from-scratch training run to get it.
Leave a Reply