2026-11-06 東京大学
自己回帰生成モデル（特に大規模言語モデル）の概念図
＜関連情報＞
- https://www.t.u-tokyo.ac.jp/press/pr2026-11-06-001
- https://journals.aps.org/prxintelligence/abstract/10.1103/tv38-b23y
自己回帰生成モデルのための確率熱力学：非マルコフ的視点 Stochastic Thermodynamics for Autoregressive Generative Models: A Non-Markovian Perspective
Takahiro Sagawa
PRX Intelligence Published: 5 November, 2026
DOI:https://doi.org/10.1103/tv38-b23y
Abstract
Autoregressive generative models—including Transformers, recurrent neural networks, classical Kalman filters, state space models, and Mamba—all generate sequences by sampling each output from a deterministic summary of the past, producing genuinely non-Markovian observed processes. We develop a general theoretical framework based on stochastic thermodynamics for this class of architectures and introduce the entropy production, which can be efficiently estimated from sampled trajectories without exponential overhead, despite the non-Markovian nature of the observed dynamics. As a proof-of-concept experiment with a large language model (LLM), we evaluate the entropy production for a pretrained Transformer-based model, GPT-2. We find that the token-level entropy production is dominated by a syntactic artifact, while the sentence-level entropy production tends to be larger for causally ordered than for noncausal text sets. This observation is supported by a reevaluation with a substantially larger model, Qwen3-4B-Base. We also demonstrate the framework in the linear Gaussian case, where the model reduces to the Kalman innovation representation and the entropy production admits an analytical expression. We also show that the entropy production decomposes exactly into non-negative per-step contributions in terms of retrospective inference, and each of those terms further splits into information-theoretically meaningful terms: a compression loss and a model mismatch. Our results establish a bridge between stochastic thermodynamics and modern generative models, and provide a starting point for using irreversibility as a quantitative probe of the highly non-Markovian processes generated by models such as LLMs.
