Disease Severity Estimates – Effects of Rater Accuracy and Assessment Methods for Comparing Treatments

Assessment of disease severity is required for several purposes in plant pathology; most often
the estimates are made visually. It is established that visual estimates can be inaccurate and
unreliable. The ramifications of biased or imprecise estimates by raters have not been fully
explored using empirical data; partly because of the logistical difficulties involved in different
raters assessing the same leaves for which actual disease has been measured in a replicated
experiment with multiple treatments. In this study nearest percent estimates (NPEs) of
Septoria leaf blotch (SLB) on leaves of winter wheat from non-treated and fungicide treated
plots were assessed in both 2006 and 2007 by four raters and compared to assumed true
values measured using image analysis. Lin’s concordance correlation (LCC, ρc) was used to
assess agreement between the two approaches. NPEs were converted to Horsfall-Barratt (HB)
mid-points and again compared for agreement with true values. The estimates of SLB severity
from fungicide-treated and non-treated plots were analyzed using generalized linear mixed
modeling to ascertain effects of rater using both the NPE and HB values. Rater 1 showed
good agreement with image analysis (ρc = 0.986 to 0.999), while raters 3 and 4 had less good
agreement (ρc = 0.205 to 0.936). Conversion to the HB scale had little effect on bias or
accuracy, but reduced both precision and agreement for most raters on most assessment dates
(precision, r = -0.001 to -0.132; and agreement, ρc = -0.003 to -0.468). Inter-rater reliability
was also reduced slightly by conversion of estimates to HB midpoint values. Estimates of
mean SLB severity were significantly different between image analysis and raters 2, 3 and 4,
and there were frequently significant differences among raters (F=151 to 1260, P=0.001 to
<0.0001). Conversion to the HB scale changed the means separation ranking of rater
estimates on 26 June 2007. Nonetheless, image analysis and all raters were able to
differentiate control and treated plots treatments (F=116 to 1952, P=0.002 to <0.0001,
depending on date and rater). Conversion of NPEs to the HB scale tended to reduce F-values
slightly (2006: NPEs, F=116 to 276, P=0.002 to 0.0005, and for the HB converted values
F=101 to 270, P=0.002 to 0.0005, and in 2007, NPEs, F=164 to 1952 P=0.001 to <0.0001,
and for HB converted values F=126 to 1633 P=0.002 to <0.0001). The results demonstrated
the need for accurate and reliable disease assessment to minimize over or underestimates
compared to actual disease, and where multiple raters are deployed, they should be assigned
in a manner to reduce any potential effect of rater differences on the analysis.