Assessing deep and shallow learning methods for quantitative prediction of acute chemical toxicity

Animal-basedmethods for assessing chemical toxicity are struggling tomeet testing demands. In silico approaches, including machine-learningmethods, are promising alternatives. Recently, deep neural networks (DNNs) were evaluated and reported to outperform othermachine-learningmethods for quantitative structure-activity relationshipmodeling ofmolecular properties. However,most of the reported performance evaluations relied on global performancemetrics, such as the root mean squared error (RMSE) between the predicted and experimental values of all samples, without considering the impact of sample distribution across the activity spectrum. Here, we carried out an in-depth analysis of DNN performance for quantitative prediction of acute chemical toxicity using several datasets.We found that the overall performance of DNN models on datasets of up to 30 000 compounds was similar to that of random forest (RF)models, asmeasured by the RMSE and correlation coefficients between the predicted and experimental results. However, our detailed analyses demonstrated that global performancemetrics are inappropriate for datasets with a highly uneven sample distribution, because they show a strong bias for themost populous compounds along the toxicity spectrum. For highly toxic compounds, DNN and RFmodels trained on all samples performed much worse than the global performancemetrics indicated. Surprisingly, our variable nearest neighbormethod, which utilizes only structurally similar compounds tomake predictions, performed reasonably well, suggesting that information of close near neighbors in the training sets is a key determinant of acute toxicity predictions.