2024-03-13 カリフォルニア大学サンディエゴ校(UCSD)
＜関連情報＞
- https://today.ucsd.edu/story/how-do-neural-networks-learn-a-mathematical-formula-explains-how-they-detect-relevant-patterns
- https://www.science.org/doi/10.1126/science.adi5639
ニューラルネットワークとバックプロパゲーションを使わない機械学習モデルにおける特徴学習のメカニズム Mechanism for feature learning in neural networks and backpropagation-free machine learning models
ADITYANARAYANAN RADHAKRISHNAN, DANIEL BEAGLEHOLE , PARTHE PANDIT, AND MIKHAIL BELKIN
Science Published:7 Mar 2024
DOI:https://doi.org/10.1126/science.adi5639
Abstract
Understanding how neural networks learn features, or relevant patterns in data, for prediction is necessary for their reliable use in technological and scientific applications. In this work, we presented a unifying mathematical mechanism, known as Average Gradient Outer Product (AGOP), that characterized feature learning in neural networks. We provided empirical evidence that AGOP captured features learned by various neural network architectures, including transformer-based language models, convolutional networks, multi-layer perceptrons, and recurrent neural networks. Moreover, we demonstrated that AGOP, which is backpropagation-free, enabled feature learning in machine learning models, such as kernel machines, that apriori could not identify task-specific features. Overall, we established a fundamental mechanism that captured feature learning in neural networks and enabled feature learning in general machine learning models.
