Measuring orthodontic treatment impact: Description or judgment, challenge or result

Determination of improvement in orthodontic treatment may depend on the measurement method used and the purpose. Improvement after orthodontic treatment (from T1 to T2 [beginning to end of treatment]) was assessed 3 ways from a set of 98 patient records: (1) calculated by subtracting judges’ assessments at T2 from T1 for records presented in random order, (2) judged as a holistic impression viewing T1 and T2 records side by side, and (3) determined from proxies (American Board of Orthodontics Discrepancy Index, the American Board of Orthodontics Objective Grading System, and the Peer Assessment Rating index). High levels of intramethod consistency were observed, with intraclass correlation coefficient clustering around an intraclass correlation coefficient of 0.900, and distributions were normal. Calculated and judged improvements correlated at r  = 0.606. Calculated or judged improvements were correlated at a lower level with proxies. Calculated improvement was significantly associated with “challenge” (T1) scores and judged improvement associated with “results” (T2) scores. Common method bias was observed, with higher correlations among similar indexes than among indexes at the same time that used various methods. Relative to differences in Peer Assessment Rating scores, calculated improvement overestimated low scores and underestimated high ones. The same effect, but statistically greater, was observed using direct judgment of improvement. These findings are consistent with decision science and measurement theory. In some circumstances, such as third-party reimbursement and research, operationally defined measures of occlusion are appropriate. In practice, the determination of occlusion and improvement are best performed by judgment that naturally corrects for biases in proxies and incorporates background information.