Receiver operating characteristic (ROC) curves and their use in psychometric simulation studies

Simulation studies are commonly used in psychometric research to compare existing methods or to highlight the outperformance of a newly developed approach with respect to standard techniques. In several specific situations, the output of performance evaluations can be summarized by pairs of statistics such as false alarm and hit rates (or Type I error and power). Adequate analysis of these rates, however, is often subject to discussion. The purpose of this ongoing work (jointly with Francis Tuerlinckx) is to advocate the usefulness of receiver operating characteristic (ROC) curves to analyze the output of simulation studies in terms of pairs of summary statistics. Two particular psychometric applications will be considered and illustrated: differential item functioning (DIF) and person fit identification. By means of simple examples, ROC curves will be shown to be efficient in capturing more output than standard analyses, thus allowing for a more refined and precise discussion of the study results. Limitations and other extensions will also be outlined.