| Title |
Enhanced human-robot teaming through attention multi convolutional neural network-based multi-modal sensor fusion for hand gesture recognition and orientation control |
| Authors |
Sundo Gonsholt, Alf Stian ; Greca, Eivind Enea ; Haddad, Mustapha ; Zafar, Muhammad Hamza ; Sanfilippo, Filippo |
| DOI |
10.24251/HICSS.2025.070 |
| ISBN |
9780998133188 |
| Full Text |
|
| Is Part of |
Proceedings of the 58th Hawaii International Conference on System Sciences, HICSS 2025: Hilton Waikoloa Village, 7-10 January 2025 / edited by Tung X. Bui.. Honolulu : University of Hawaii at Manoa, 2025. p. 590-599.. eISSN 2572-6862. ISBN 9780998133188 |
| Keywords [eng] |
Human-Robot Interaction and Collaboration ; deep neural network ; electromyography (emg) sensors ; gesture recognition ; hand gesture recognition ; inertial measurement units (imus) ; multi-modal sensor fusion |
| Abstract [eng] |
Our study aims at enhancing Human-Robot Interaction, Collaboration, and Teaming (HRI/C/T) in industrial automation by developing a novel framework for real-time gesture control of a robotic hand. We use an Inertial Measurement Unit (IMU) sensor for precise orientation control of the end effector, and surface Electromyography (sEMG) sensors to detect muscle movements. The sEMG signals are processed by an Attention-based Multi Convolutional Neural Network (A-MCNN) for accurate gesture detection, enabling the robotic hand to mimic these gestures in real-time. Our method achieves notable results for gesture recognition, with the A-MCNN model attaining an accuracy of 97.89%, a precision of 97.49%, a recall of 97.71%, and an F1 score of 97.65%. This integration of IMU and sEMG technologies with advanced neural networks creates a responsive and intuitive control mechanism, improving safety, usability, and interaction of collaborative robots in shared workspaces. Our approach aims to transition towards Human-Robot Teaming (HRT), significantly advancing the seamless and safe integration of robots in industrial environments, enhancing productivity and collaboration. |
| Published |
Honolulu : University of Hawaii at Manoa, 2025 |
| Type |
Conference paper |
| Language |
English |
| Publication date |
2025 |
| CC license |
|
