Data Annotation for Support Ticket Data: A Literature Review

Supervised Machine Learning is still the most prevalent Machine Learning approach used across the field of Natural Language Processing. As it needs labels to work properly, labeling text data sets is a discerning step in supervised Machine Learning projects. Many industry projects involving supervised Machine Learning never reach a productive phase due to the absence of sufficient labeled data. Against this background, we conducted a Literature Review investigating state of the art approaches to label text data sets for later Natural Language Processing projects. We concentrated on solutions that could be applicable to annotate a support ticket data set. We found that there are three major approaches: Crowdsourcing, Learning Algorithms and Weak Supervision. We also found, that in annotation projects there seems to be an assessment between label quality and cost/effort. We discuss our findings and share our thoughts on the special challenges of annotating a support ticket data set.