Retrieval In Decoder benefits generative models for explainable complex question answering

Large-scale Language Models (LLMs) utilizing the Chain-of-Thought prompting demonstrate exceptional performance in a variety of tasks. However, the persistence of factual hallucinations remains a significant challenge in practical applications. Prevailing retrieval-augmented methods treat the retriever and generator as separate components, which inadvertently restricts the generator’s capabilities to those of the retriever through intensive supervised training. In this work, we propose an unsupervised Retrieval In Decoder framework for multi-granularity decoding called RID, which integrates retrieval directly into the decoding process of generative models. It dynamically adjusts decoding granularity based on retrieval outcomes, and duly corrects the decoding direction through its direct impact on the next token. Moreover, we introduce a reinforcement learning-driven knowledge distillation method for adaptive explanation generation to better apply to Small-scale Language Models (SLMs). The experimental results across six public benchmarks surpass popular LLMs and existing retrieval-augmented methods, which demonstrates the effectiveness of RID in models of different scales and verifies its applicability and scalability.