A Riemannian multimodal representation to classify parkinsonism-related patterns from noninvasive observations of gait and eye movements

Parkinson’s disease is a neurodegenerative disorder principally manifested as motor disabilities. In clinical practice, diagnostic rating scales are available for broadly measuring, classifying, and characterizing the disease progression. Nonetheless, these scales depend on the specialist’s expertise, introducing a high degree of subjectivity. Thus, diagnosis and motor stage identification may be affected by misinterpretation, leading to incorrect or misguided treatments. This work addresses how to learn multimodal representations based on compact gait and eye motion descriptors whose fusion improves disease diagnosis prediction. This work introduces a noninvasive multimodal strategy that combines gait and ocular pursuit motion modalities into a geometrical Riemannian Neural Network for PD quantification and diagnostic support. Markerless gait and ocular pursuit videos were first recorded as Parkinson’s observations, which are represented at each frame by a set of frame convolutional deep features. Then, Riemannian means are computed per modality using frame-level covariances coded from convolutional deep features. Thus, a geometrical learning representation is adjusted by Riemannian means, following early, intermediate, and late fusion alternatives. The adjusted Riemannian manifold combines input modalities to obtain PD prediction. The geometrical multimodal approach was validated in a study involving 13 control subjects and 19 PD patients, achieving a mean accuracy of 96% for early and intermediate fusion and 92% for late fusion, increasing the unimodal accuracy results obtained in the gait and eye movement modalities by 6 and 8%, respectively. The proposed method was able to discriminate Parkinson’s patients from healthy subjects using multimodal geometrical configurations based on covariances descriptors. The covariance representation of video descriptors is highly compact (with an input size of 625 and an output size of 256 (1 BiRe)), facilitating efficient learning with a small number of samples, a crucial aspect in medical applications.