指先の色の変化でAR/MR操作を実現する新入力技術 ―特別な装置なしで壁や机をタッチパネルとして利用可能にー

Current Mixed Reality (MR) headsets are typically controlled by mid-air hand inputs, which require significant arm effort and consequently reduce input speed and accuracy during extended sessions. We propose BlanchTouch, a research prototype that explores an alternative touch input approach for MR by appropriating uninstrumented flat surfaces as interactive areas, with reduced arm effort. It incorporates a CNN model that detects the fingertip’s pressure-induced blanching phenomenon when a firm touch is applied to hard surfaces, serving as a confirmation of selection. Additionally, hand tracking via the headset is employed to determine the fingertip’s position, thereby providing targeting information for virtual buttons or objects. BlanchTouch operates using only the headset’s built-in cameras and sensors, without requiring additional hardware or prior surface calibration, enabling walk-up use in unfamiliar environments. We conducted two user studies to evaluate the system and compare its performance against a baseline (a real touchscreen) and the state-of-the-art mid-air interaction. Results demonstrated that BlanchTouch achieved an average input positional error of 5.33mm and a recall rate of 97.88%. Furthermore, the findings indicate that BlanchTouch enables competitive input speeds, alleviates arm strain, and delivers satisfactory feedback for each touch.