Real-time multimodal interaction in virtual reality - a case study with a large virtual interface

The values of VR and multimodal interaction technologies offer creative, virtual alternatives to manipulate a large data set in a virtual environment. This work presents the design, implementation, and evaluation of a real-time multimodal interaction framework that enables users to navigate, select, and move data elements. The novel multimodal fusion method is able to recognize freehand gestures, voice commands, and head gaze pointer in real-time and fuse them to meaningful actions for interacting with the virtual environment. We worked with imagery analysts who were defense and security experts on designing and testing the interface and interaction modalities. The evaluation of the framework was conducted with a case study of photo management tasks based on a real-world scenario. Users are able to select photos in a large virtual interface and move them to the bins on the left and right sides of the main view. The evaluation focuses on performance, task completion time, and users’ experience amongst several different combinations of input modalities. The evaluation result shows it is important to make multiple interaction modalities available to users, and the interaction design implications are concluded based on the evaluation.