2026-01-14 コロンビア大学
＜関連情報＞
- https://www.engineering.columbia.edu/about/news/robot-learns-lip-sync
- https://www.science.org/doi/10.1126/scirobotics.adx3017
ヒューマノイド顔ロボットのためのリアルな唇の動きの学習 Learning realistic lip motions for humanoid face robots
Yuhang Hu, Jiong Lin, Judah Allen Goldfeder, Philippe M. Wyder, […] , and Hod Lipson
Science Robotics Published:14 Jan 2026
DOI:https://doi.org/10.1126/scirobotics.adx3017
Abstract
Lip motion represents outsized importance in human communication, capturing nearly half of our visual attention during conversation. Yet anthropomorphic robots often fail to achieve lip-audio synchronization, resulting in clumsy and lifeless lip behaviors. Two fundamental barriers underlay this challenge. First, robotic lips typically lack the mechanical complexity required to reproduce nuanced human mouth movements; second, existing synchronization methods depend on manually predefined movements and rules, restricting adaptability and realism. Here, we present a humanoid robot face designed to overcome these limitations, featuring soft silicone lips actuated by a 10–degree-of-freedom mechanism. To achieve lip synchronization without predefined movements, we used a self-supervised learning pipeline based on a variational autoencoder (VAE) combined with a facial action transformer, enabling the robot to autonomously infer more realistic lip trajectories directly from speech audio. Our experimental results suggest that this method outperforms simple heuristics like amplitude-based baselines in achieving more visually coherent lip-audio synchronization. Furthermore, the learned synchronization successfully generalizes across multiple linguistic contexts, enabling robot speech articulation in 10 languages unseen during training.
