2024-02-13 マサチューセッツ工科大学(MIT)
◆彼らの方法は、多くの大規模言語モデルの核となるキー・バリューキャッシュ（会話のメモリのようなもの）を微調整することに関連しています。この方法により、最初のいくつかのデータポイントがメモリに残るようにすることで、研究者たちはチャットボットが会話を続けられるようにしました。この方法は、「StreamingLLM」と呼ばれ、会話が400万単語を超えてもモデルを効率的に保つことができます。
＜関連情報＞
- https://news.mit.edu/2024/new-way-let-ai-chatbots-converse-all-day-without-crashing-0213
- https://arxiv.org/abs/2309.17453
アテンションシンクによる効率的なストリーミング言語モデル Efficient Streaming Language Models with Attention Sinks
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, Mike Lewis
arXiv last revised:12 Dec 2023
DOI:https://doi.org/10.48550/arXiv.2309.17453
Abstract
Deploying Large Language Models (LLMs) in streaming applications such as multi-round dialogue, where long interactions are expected, is urgently needed but poses two major challenges. Firstly, during the decoding stage, caching previous tokens’ Key and Value states (KV) consumes extensive memory. Secondly, popular LLMs cannot generalize to longer texts than the training sequence length. Window attention, where only the most recent KVs are cached, is a natural approach — but we show that it fails when the text length surpasses the cache size. We observe an interesting phenomenon, namely attention sink, that keeping the KV of initial tokens will largely recover the performance of window attention. In this paper, we first demonstrate that the emergence of attention sink is due to the strong attention scores towards initial tokens as a “sink” even if they are not semantically important. Based on the above analysis, we introduce StreamingLLM, an efficient framework that enables LLMs trained with a finite length attention window to generalize to infinite sequence lengths without any fine-tuning. We show that StreamingLLM can enable Llama-2, MPT, Falcon, and Pythia to perform stable and efficient language modeling with up to 4 million tokens and more. In addition, we discover that adding a placeholder token as a dedicated attention sink during pre-training can further improve streaming deployment. In streaming settings, StreamingLLM outperforms the sliding window recomputation baseline by up to 22.2x speedup. Code and datasets are provided at this https URL.
