Skip to content

LLM : Model Merging/Model Fusion⚓︎

What is model merging ?

A techniques that combines the parameters(weights) of two or more pre-trained/fine-tuned language models with same/diff architectures into a single model, to inherit(blend) the capabilities or knowledge of each parent models without retraining from scratch. Example : Model-1 : Text Summarization Model-2 : Text Translation Then Merged M1 + M2 = Single model by combining serveral tasks specific models into a single models without any training or finetuning with just performating the mathematical operations onto the model parameters.

Why Model merging ?

  1. Pre-training a large model from scratch or Post-training a foundation model is

  2. Expensive

  3. Time-consuming

  4. Compute-heavyS

  5. Model merging can provide us a cheap, fast alternative that can

  6. Merge several task specific finetuned models into single one.

  7. Single Unified model capable of handling multiple tasks.
  8. We don't need for additional gradient-based training or finetuning.
  9. Everything runs on CPU, no GPU required.

How model merging works ? If we have the same architecture then we can merge the model parameters based onto the interpolations of the each model parameters such that they preserve the embedding space geoemetry and properties.

  • It algebraically mixes checkpoints—often as full weights or fine-tuning deltas—so the merged model inherits behaviors from multiple specialized models without re-training on raw data.
  • It done offline via weight-space operations like linear averaging, spherical interpolation, or structured selection, and can run entirely on CPU with libraries such as mergekit.

Why does model merging works ?

  • The fine-tuned models from a shared initialization often lie in a connected low-loss basin, so averaging their weights can improve accuracy and robustness—an effect popularized as “model soups”.
  • The main failure mode is parameter interference (e.g., redundant updates and sign conflicts across models), which specialized schemes like TIES address by trimming small deltas and resolving sign disagreements before merging.
  • Thus, well designed merging can generalize better than naive averages and provide stronger starting points for subsequent fine-tuning.

What are the most common methods for it ?

  1. Model-soups
  2. Spherical Linear Interpolation(SLERP)
  3. Task Arithmetic
  4. Trim, Elect Sign and Merge(TIES)
  5. Drop and Rescale
  6. Franken-Merging

Linear/Soup Averaging :

Spherical Linear Interpolation(SLERP)

Motivation : In computer graphics, problem of finding smooth transitions between two camera rotations.

What is SLERP ?

It only works with two models and we can favour one of them also.

It helps to preserve magnitude of weights and curvature of the embedding space.

TIES : TRIM, ELECT SIGN & MERGE

Parameter interference can be degrade merged performance

References⚓︎

  1. LLM Marathon Series Lecture : Model Merging
  2. Deep Dive : Model Merging - Part-1
  3. Deep Dive : Model Merging - Part-2
  4. https://huggingface.co/blog/mlabonne/merge-models
  5. https://github.com/arcee-ai/mergekit
  6. https://www.slideshare.net/slideshow/julien-simon-deep-dive-model-merging/270921708 7.