SourceJS-LoRA: Source-Referenced Jensen-Shannon Divergence for Learning LoRA Merge Coefficients

Abstract

Merging multiple task-specific LoRA adapters into a single multi-task adapter is challenging due to parameter interference. Existing adaptive methods learn merge coefficients by minimizing entropy on unlabeled data, but this reference-free objective can lead to ``confidently wrong'' predictions where the model becomes overconfident on incorrect outputs. We propose SourceJS-LoRA, which learns merge coefficients by minimizing Jensen-Shannon divergence between the merged model's predictions and each task expert's predictions. This source-referenced objective anchors the optimization to task-specific expert models, preventing confidence collapse. On 8 GLUE tasks with T5-base, SourceJS-LoRA achieves 83.10% average accuracy, outperforming the state-of-the-art DO-Merging by +2.29 points and entropy-based coefficient learning by +5.55 points. Analysis reveals that our method produces stable, balanced coefficients while entropy minimization leads to extreme values with negative weights.