Improving Text Classification Performance Through Multimodal Representation

Traditionally, text classification research has predominantly focused on extracting single text features, with limited exploration of integrating other modal information (such as speech and images) to enhance classification performance. To address this research gap, we propose the Multimodal Representation for Text Classification (MRTC) framework. This framework aims to boost text classification performance by incorporating speech, image, and text features. Specifically, we employ advanced text-to-speech models to convert text content into audio features. Simultaneously, we retrieve images closely associated with the text content and extract their visual features to further enrich the information dimension of text representation. Subsequently, we utilize an efficient triplet structure network to fuse the speech, image, and text features, thereby constructing a multimodal feature representation for application in text classification tasks. The proposed MRTC framework achieves high-precision text classification across multiple datasets without requiring additional multimodal annotated data. This characteristic not only reduces the cost of data annotation but also enhances the model’s practical flexibility and scalability. To validate the effectiveness of the MRTC framework, we conduct experiments on six distinct text classification tasks. The experimental results demonstrate the significant effectiveness of our MRTC framework across various text classification tasks.