How do music experts and non-experts evaluate the vocal accuracy of operatic singing voices?

Professional singers are expected to sing in tune. However, when an operatic singing technique is employed, the objective measurement of the vocal accuracy (i.e. pitch interval deviation) shows particularly low scores, whatever the melody performed. This study focuses on the perceptual judgment of operatic voices in order to observe the evaluation process of singing voice accuracy by music experts and non-experts. In addition, this study aims to better understand the relationship between the subjective and objective measurements of operatic singing voices.
22 music experts and 22 non-experts paired in age and gender participated in a test and a retest (8 to 15 days in between). Fourteen sung performances performed by professional operatic singers were presented with a pairwise comparison paradigm. The participants were asked to indicate the most “in tune” melody for each pair (N = 91). The performances obtained thus a ranking by each judge. In addition, the 14 sung performances were objectively analyzed in order to confront the objective measurement of singing voice accuracy with the perceptual rating of the judges.
Computing the variances of rank differences between the test and the retest, we observed that 20 music experts and 16 non-experts were consistent in their judgments. Among each group, the correlations between consistent raters were positive. However, 67.38% of these correlations were significant (p < .05) for the music experts whereas only 42.10% were significant for the non-experts. In addition, no relationship occurred between the objective measurements (from 9.5 to 115.5 cents, M = 40.57, SD = 34.42) and the perceptual ratings, except for two music experts.
This study highlights the consistency of a judge when rating operatic singing voices and the difference between music experts and non-experts concerning the inter-judges reliability. Despite the fact that the majority of the music experts used similar strategies to evaluate the vocal accuracy of operatic voices, their judgment was obviously not linked with the objective measurement of vocal accuracy. This finding supports the tolerance of music expert listeners regarding the singing voice accuracy of operatic singers. Furthermore, this study provides some directions about the perception of operatic singing voices, which are particularly complex.