2025-04-18 ジョージア工科大学（Georgia Tech）
＜関連情報＞
- https://research.gatech.edu/new-approach-easily-merging-data-models-brings-multi-tasking-ais-closer-reality
- https://www.cc.gatech.edu/news/new-approach-easily-merging-data-models-brings-adaptable-multi-tasking-ais-closer-reality
- https://arxiv.org/abs/2410.19735
結び目を結ぶSVDによるモデル統合 Model merging with SVD to tie the Knots
George Stoica, Pratik Ramesh, Boglarka Ecsedi, Leshem Choshen, Judy Hoffman
arXiv Submitted on 25 Oct 2024
DOI:https://doi.org/10.48550/arXiv.2410.19735
Abstract
Recent model merging methods demonstrate that the parameters of fully-finetuned models specializing in distinct tasks can be combined into one model capable of solving all tasks without retraining. Yet, this success does not transfer well when merging LoRA finetuned models. We study this phenomenon and observe that the weights of LoRA finetuned models showcase a lower degree of alignment compared to their fully-finetuned counterparts. We hypothesize that improving this alignment is key to obtaining better LoRA model merges, and propose KnOTS to address this problem. KnOTS uses the SVD to jointly transform the weights of different LoRA models into an aligned space, where existing merging methods can be applied. In addition, we introduce a new benchmark that explicitly evaluates whether merged models are general models. Notably, KnOTS consistently improves LoRA merging by up to 4.3% across several vision and language benchmarks, including our new setting.
