| Title |
Multi-scale spatiotemporal EEG and self-supervised audio fusion: a mixture-of-experts approach to continuous affect |
| Authors |
Amini, Ali ; Maqsood, Sarmad ; Abbas, Irfan ; Sarwar, Muhammad Abdullah ; Maskeliūnas, Rytis ; Damaševičius, Robertas |
| DOI |
10.1145/3776574.3831113 |
| ISBN |
9798400723186 |
| Full Text |
|
| Is Part of |
ICMI '26: proceedings of the 28th international conference on multimodal interaction, 5-9 October, Napoli, Italy.. New York : Association for Computing Machinery (ACM), 2026. p. 27-35.. ISBN 9798400723186 |
| Keywords [eng] |
Electroencephalography (EEG) ; Continuous emotion recognition ; Multimodal fusion ; Mixture of experts ; Transfer learning |
| Abstract [eng] |
Emotion-aware human–computer interaction increasingly relies on continuous emotion recognition (CER) to track affective states over time. This paper investigates continuous valence prediction on the MAHNOB-HCI database using a multimodal EEG+audio framework. The proposed model combines (i) a hierarchical spatiotemporal EEG encoder that captures per-channel temporal dynamics and inter-channel dependencies and (ii) self-supervised audio embeddings extracted from SSLAM in a feature-based transfer setting. To integrate modalities robustly, we introduce a modality-sensitive mixture-of-experts (MoE) fusion module with modality-specific experts and a learned gating network that dynamically weights EEG and audio contributions. Under session-preserving partitioning with 5-fold cross-validation, the proposed approach achieves strong agreement with ground-truth valence trajectories (PCC=0.778, CCC=0.742, RMSE=0.034), comparing favorably with representative prior results reported under broadly related MAHNOB-HCI protocols. Finally, we provide interpretability evidence by linking the EEG encoder’s alpha-band saliency asymmetry to classical alpha-band power asymmetry patterns, thereby supporting the neurophysiological plausibility of the learned representations. |
| Published |
New York : Association for Computing Machinery (ACM), 2026 |
| Type |
Conference paper |
| Language |
English |
| Publication date |
2026 |
| CC license |
|
