Effect of Loss Functions on Language Models in Question Answering-Based Generative Chat-Bots

Question answering mechanism plays a pivotal role in helping generative chat-bots provide accurate answers to the questions asked by the questioner. Language models are an integral part of question/answer mechanisms where they answer two things, namely classifying the question correctly and also help provide the most suitable answer for the question asked. This paper takes into account the effect of various loss functions on the answer-predicting capability of a language model. Various loss functions were implemented like cross-entropy loss, negative log-likelihood loss, cosine embedding loss, and Kullback–Leibler divergence loss and found that there is a significant impact due to selection of loss functions on accuracy. Based on the research and implementation in this paper, it is found that choosing the multi-label cross-entropy loss for all the general question answering problems has a 0.5–1% raise.