From Pose to Muscle: Multimodal Learning for Piano Hand Muscle Electromyography

RUOFAN LIU (Institute of Science Tokyo) · YICHEN PENG (Japan Advanced Institute of Science and Technology, Tokyo Institute of Technology) · Takanori Oku (Shibaura Institute of Technology) · Chen-Chieh Liao (Institute of Science Tokyo, Tokyo Institute of Technology) · Erwin Wu (Institute of Science Tokyo) · Shinichi Furuya (Sony Computer Science Lab) · Hideki Koike (Tokyo Institute of Technology, Tokyo Institute of Technology)

adaptationanatomical variabilitydexterous skillselectromyographyembodied intelligence generalizationhand muscle engagementhand muscle estimation frameworkhigh-fidelity emg inferencelatent pose-emg correspondencemultimodal datasetmuscle coordinationpianokpm datasetpianokpm nettask-specific contextstraining set scale

Muscle coordination is fundamental when humans interact with the world. Reliable estimation of hand muscle engagement can serve as a source of internal feedback, supporting the development of embodied intelligence and the acquisition of dexterous skills. However, contemporary electromyography (EMG) sensing techniques either require prohibitively expensive devices or are constrained to gross motor movements, which inherently involve large muscles. On the other hand, EMGs exhibit dependency on individual anatomical variability and task-specific contexts, resulting in limited generalization. In this work, we preliminarily investigate the latent pose-EMG correspondence using a general EMG gesture dataset. We further introduce a multimodal dataset, PianoKPM Dataset, and a hand muscle estimation framework, PianoKPM Net, to facilitate high-fidelity EMG inference. Subsequently, our approach is compared against reproducible competitive baselines. The generalization and adaptation across unseen users and tasks are evaluated by quantifying the training set scale and the included data amount.