学習過程を少数合成データに圧縮する仕組みの理論的解明 －汎用性のある情報の抽出と効率的な再利用－

2026-07-08 理化学研究所,東京大学

データ蒸留と低次元構造の抽出メカニズム

＜関連情報＞

• https://www.riken.jp/press/2026/20260708_1/index.html#wrapper
• https://arxiv.org/abs/2603.14830

データセット蒸留は、非線形タスクの勾配ベース学習から低次元表現を効率的に符号化する Dataset Distillation Efficiently Encodes Low-Dimensional Representations from Gradient-Based Learning of Non-Linear Tasks

Yuri Kinoshita, Naoki Nishikawa, Taro Toyoizumi
arXiv  last revised 3 Jul 2026 (this version, v3)
DOI:https://doi.org/10.48550/arXiv.2603.14830

Abstract

Dataset distillation, a training-aware data compression technique, has recently attracted increasing attention as an effective tool for mitigating costs of optimization and data storage. However, progress remains largely empirical. Mechanisms underlying the extraction of task-relevant information from the training process and the efficient encoding of such information into synthetic data points remain elusive. In this paper, we theoretically analyze practical algorithms of dataset distillation applied to the gradient-based training of two-layer neural networks with width L. By focusing on a non-linear task structure called multiindex model, we prove that the low-dimensional structure of the problem is efficiently encoded into the resulting distilled data. This dataset reproduces a model with high generalization ability for a required memory complexity of ˜ Θ(r²d+L), where d and r are the input and intrinsic dimensions of the task. To the best of our knowledge, this is one of the first theoretical works that include a specific task structure, leverage its intrinsic dimensionality to quantify the compression rate and study dataset distillation implemented solely via gradient-based algorithms.