2026-07-30 中国科学院（CAS）
Illustration of outcome reward-induced shortcuts and the corrective mechanism of process supervision. (Image by DING Zenghui)
＜関連情報＞
- https://english.cas.cn/newsroom/research-news/202607/t20260731_1186800.shtml
- https://aclanthology.org/2026.acl-long.925/
- https://aclanthology.org/2026.findings-acl.769/
結果最適化のパラドックス：LLMにおける推論ショートカットに対する因果情報理論的限界 The Paradox of Outcome Optimization: A Causal Information-Theoretic Bound on Reasoning Shortcuts in LLMs
Zihan Chen, Yiming Zhang, Wenxiang Geng, Zenghui Ding, Yining Sun
Association for Computational Linguistics Published:July 2026
DOI:https://doi.org/10.18653/v1/2026.acl-long.925
Abstract
Large Language Models (LLMs) aligned via outcome-based Reinforcement Learning (RL) frequently exhibit a critical failure mode: they achieve high performance on in-distribution benchmarks while demonstrating brittle reasoning capabilities on out-of-distribution (OOD) tasks. We term this phenomenon Reward-Induced Manifold Collapse. We establish a theoretical framework bridging Structural Causal Models (SCM) and the Information Bottleneck (IB) principle to explain this paradox. We define reasoning as a high-complexity causal process and shortcut learning as the exploitation of low-complexity spurious correlations. Under the implicit inductive bias of Stochastic Gradient Descent (SGD), models optimized for outcome rewards are biased toward shortcut solutions whenever the training distribution allows for a “Markovian Screening” of the true causal mechanism. We derive a new generalization bound based on Semantic Coverage Measure () rather than sample size, showing why data scaling on homogeneous distributions may fail to correct reasoning flaws. We also show that Process Reward Models (PRMs) function as Topological Filters, enforcing step-wise mutual information constraints that render the low-complexity shortcut manifold inadmissible. These results provide a mathematical grounding for the role of process supervision beyond simple credit assignment.
強化学習の評価を再考する：ベンチマークは強化学習手法の失敗を本当に明らかにできるのか？ Rethinking RL Evaluation: Can Benchmarks Truly Reveal Failures of RL Methods?
Zihan Chen, Yiming Zhang, Hengguang Zhou, Zenghui Ding, Yining Sun, Cho-Jui Hsieh
Association for Computational Linguistics Published:July 2026
DOI:https://doi.org/10.18653/v1/2026.findings-acl.769
Abstract
Current benchmarks are inadequate for evaluating progress in reinforcement learning (RL) for large language models (LLMs). Despite recent benchmark gains reported for RL, we find that training on these benchmarks’ training sets achieves nearly the same performance as training directly on the test sets, suggesting that the benchmarks cannot reliably separate further progress. To study this phenomenon, we introduce a diagnostic suite and the Oracle Performance Gap (OPG) metric that quantifies the performance difference between training on the train split versus the test split of a benchmark. We further analyze this phenomenon with stress tests and find that, despite strong benchmark scores, existing RL methods struggle to generalize across distribution shifts, varying levels of difficulty, and counterfactual scenarios: shortcomings that current benchmarks fail to reveal. We conclude that current benchmarks are insufficient for evaluating generalization and propose three core principles for designing more faithful benchmarks: sufficient difficulty, balanced evaluation, and distributional robustness.
