A comparison between subjective and objective methods for evaluating the vocal accuracy of a popular song

Vocal accuracy of a sung performance can be evaluated by two methods: acoustical analyses and subjective judgements. For one decade, acoustic analyses have been presented as a more reliable solution to evaluate vocal accuracy, avoiding the limitation of experts’ perceptive system and their variability. This paper presents for the first time a direct comparison of these methods.
166 occasional singers were asked to sing the popular song « Happy Birthday ». Acoustic analyses were performed to quantify the pitch interval deviation, the number of contour errors and the number of tonality modulations for each recording. Additionally, eighteen experts in singing voice or music rated the global pitch accuracy of these performances.
The results showed a high inter-rater concordance within the judges. In addition, a high correlation occurred between acoustic measurements and subjective rating. The total model of acoustic analyses explained 81% of the variance of the judges’ scores. Their rating was influenced by both tonality modulations and interval deviations.
This study highlights the congruence between objective and subjective measurements of vocal accuracy while the assessment is done by music or singing voice experts. Our results confirm the relevance of the “pitch interval deviation” criterion in vocal accuracy assessment. Furthermore, the “number of tonality modulations” is a salient criterion in perceptive rating and should be taken into account in studies using acoustical analyses.