gensyn (@gensynai): Training-Free Composition of Expert Models

Training-Free Composition of Expert Models The dominant approach to building capable language models has been to scale monolithically: add parameters, add data, add compute, and train everything together in one large centralized run. This works, up to a point, but it creates a fundamental problem that becomes harder to ignore as model development spreads across organizations, research groups, and individual contributors. When expertise is already distributed, when someone has trained a strong mathematics model, someone else has a strong coding model, and another has a strong model for biology - how do you combine them without destroying what each one knows? The naive answer is weight averaging. Take the parameters from each model, form a weighted mean, and use the result. The problem is that these models, which have been trained to solve different tasks in different domains, have been optimized in different loss landscapes. If you just average their weights, you fail to produce a model that is optimized for the global distribution. In other words, your final aggregated model is good at none of the original tasks. This is destructive interference, and it is a well-documented failure mode in model merging. Most of the literature has tried to work around it, and most of the literature has not fully succeeded. MoErging and the Partial Merge A different angle on the problem comes from treating model composition through the lens of Mixture-of-Experts construction. Rather than attempting to merge all parameters into a single set of weights, MoErging (Mixture-of-Experts Merging) tries to sidestep destructive interference by being selective about what actually gets merged. Self-attention blocks and embedding layers encode relatively general structure; the MLP blocks within each transformer layer are where domain specialisation actually concentrates. Rather than averaging everything, MoErging averages the general layers across source models while keeping the MLP blocks separate, treating each source model's MLPs as distinct experts in the resulting architecture. The MoErged model gains both breadth and depth, inheriting shared structural knowledge while retaining the specialised processing that makes each contributor useful. This approach existed before DUME. The unresolved problem was routing. Once you have assembled a set of MLP experts under a unified attention scaffold, you need a mechanism to decide which expert should process each incoming token. Prior work addressed this by doing additional training on the merged model, using collected data to learn the routing function from scratch. This is exactly the requirement that collapses the practical appeal of the approach. If you need to run another training pass, you need centralised data, you reintroduce catastrophic forgetting, and you lose the key economic advantage of reusing already-trained models. DUME: Closing the Last Gap DUME (Dynamic Upcycling of Mixture-of-Experts) solves the router initialisation problem without any training. The approach frames router construction as a classification problem: given an input, which domain does it belong to? This is treated mathematically using ridge regression, a well-understood regularised least-squares technique. Rather than learning a router by gradient descent, DUME computes the router analytically, requiring only forward passes through the source models on a small number of labelled prompts per domain. In some experimental settings, as few as 80 prompts per domain are sufficient. The computational cost of this step is low by construction. There is no backpropagation, no optimiser state, no learning rate schedule. The process reduces to matrix multiplications and a matrix inversion per each MoE block in the model, which can be computed cheaply even at the scale of modern large models. The result is a router that reflects the actual geometry of the latent representations produced by the source experts, without requiring any direct access to the original training data beyond these small domain-labelled prompt sets. The "dynamic" aspect of the name refers to something equally important: the ability to add or remove experts after the fact without retraining or corrupting the router. If a new expert needs to be incorporated, the router is upgraded using the same ridge regression procedure applied to the new expert's statistics. If an expert needs to be removed, whether for operational reasons or because a data owner wants to exercise something resembling a right to be forgotten, the router can be downgraded, cleanly subtracting that expert's contribution without affecting the knowledge retained from all other contributors. This is not a minor operational convenience. It means the composition of the model can be governed over time, which matters enormously in any setting where data ownership and model provenance are real concerns. Privacy and Decentralised Deployment There is an additional structural property worth examining carefully. The statistics that DUME requires from each source expert are aggregated feature representations, not the underlying data. This has meaningful implications for deployment in federated or decentralised settings. In federated learning, the persistent concern with any gradient-sharing or feature-sharing protocol is that shared quantities can leak information about private training data through adversarial inference attacks. Because DUME's aggregation step produces aggregated quantities, a Secure Aggregation layer can be applied directly on top of this aggregation to make it secure, providing formal privacy guarantees without any additional modification to the core algorithm. This makes DUME unusual among model composition methods: it is simultaneously training-free, privacy-compatible, and incrementally updatable. These three properties do not typically coexist. The reason they coexist here is that the design choices that make DUME computationally cheap - forward-only statistics extraction and closed-form router computation - are the same choices that make it compatible with privacy-preserving aggregation protocols. Open Ecosystems and What Comes Next The broader implication of this work is that model composition can be made genuinely modular. Today, a practical obstacle to assembling generalist models from specialist contributors is the cost and complexity of the post-composition training step. DUME removes that obstacle. Models already available on public repositories could be combined into multi-domain systems without any centralised training infrastructure, using only the small labelled prompt sets needed to initialise the router. The current limitation is that DUME requires source experts to share the same base architecture. The MLPs being kept separate must be structurally compatible for the method to work as described. This is a real constraint, but it is also a tractable one. A natural extension is to ask what happens when experts share the same architectural family but differ in scale: a 3B and a 9B model with the same layer structure but different hidden dimensions. Handling heterogeneous MLP sizes would open the method to the broader population of open-source models where size variation is the norm rather than the exception. A complementary line of ongoing work addresses the inference-time routing problem. Rather than merging experts into a single model, the question becomes how to select the right expert for each incoming query without using a large model to make that decision. Using a large language model to decide which large language model to call doubles the cost of every inference. A lightweight classifier trained on the same ridge regression principles would preserve the efficiency gains that motivate the whole approach. This is where the research is heading, and the connection to the core DUME ideas is direct. The broader research direction here, covering training-free composition, incrementally updatable models, privacy-preserving statistics sharing, and open ecosystems of specialist contributors, is coherent with how decentralised machine learning infrastructure needs to develop. The value of a network of independently trained expert models depends entirely on whether they can be composed into useful systems without requiring all contributors to coordinate on a single training run. DUME is a meaningful step toward making that composition tractable. (This article was created from a conversation with Eros Fani. You can listen to the recording here - https://youtu.be/2xoKSqxp8Lc)