Examiner performance in calibration exercises compared with field conditions when scoring caries experience

Abstract

The objective of this study was to verify how valid misclassification measurements obtained from a ‘pre-survey’ calibration exercise are by comparing them to validation scores obtained in ‘field’ conditions. Validation data were collected from the ‘Smile for Life’ project, an oral health intervention study in Flemish children. A calibration exercise was organized under ‘pre-survey’ conditions (32 age-matched children examined by eight examiners and the benchmark scorer). In addition, using a pre-determined sampling scheme blinded to the examiners, the benchmark scorer re-examined between six and 11 children screened by each of the dentists during the survey. Factors influencing sensitivity and specificity for scoring caries experience (CE) were investigated, including examiner, tooth type, surface type, tooth position (upper/lower jaw, right/left side) and validation setting (pre-survey versus field). In order to account for the clustering effect in the data, a generalized estimating equations approach was applied. Sensitivity scores were influenced not only by the calibration setting (lower sensitivity in field conditions, p < 0.01), but also by examiner, tooth type (lower sensitivity in molar teeth, p < 0.01) and tooth position (lower sensitivity in the lower jaw, p < 0.01). Factors influencing specificity were examiner, tooth type (lower specificity in molar teeth, p < 0.01) and surface type (the occlusal surface with a lower specificity than other surfaces) but not the validation setting. Misclassification measurements for scoring CE are influenced by several factors. In this study, the validation setting influenced sensitivity, with lower scores obtained when measuring data validity in ‘field’ conditions. Results obtained in a pre-survey calibration setting need to be interpreted with caution and do not (always) reflect the actual performance of examiners during the field work.

Appendix

Logistic regression model for sensitivity and specificity

Suppose that the binary score for CE is denoted as Y for the benchmark and Y* is the score attributed by the examiner. Thus, Y = 1 corresponds to CE truly present, and Y = 0 means no CE present. Using this notation, sensitivity is equal to ${\pi_{\rm{se}}} = { \Pr }\left( {Y* = {1}|Y = {1}} \right)$ and specificity is equal to ${\pi_{\rm{sp}}} = { \Pr }\left( {Y* = 0|Y = 0} \right)$.

A logistic regression model relating π _se and π _sp to p factors x ₁, x ₂,…x _p is given by:

$$ {\hbox{logit}}\left( {{\pi_{\rm{se}}}} \right) = {\beta_0} + {\beta_{{1}}}{x_{{1}}} + {\beta_{{2}}}{x_{{2}}} + ... + {\beta_{\rm{p}}}{x_{\rm{p}}}, $$

(1)

$$ {\hbox{logit}}\left( {{\pi_{\rm{sp}}}} \right) = {\gamma_0} + {\gamma_{{1}}}{x_{{1}}} + {\gamma_{{2}}}{x_{{2}}} + ... + {\gamma_{\rm{p}}}{x_{{{\rm{p}},}}} $$

(2)

respectively, where logit $\left( \pi \right) = { \log }\left( {\pi /\left[ {{1}-\pi } \right]} \right)$. The coefficients β ₀, β ₁,…, β _p and γ ₀, γ ₁,…, γ _p are called regression coefficients and are estimated according to the method of maximum likelihood. The coefficients β ₀ and γ ₀ are called intercepts. The other coefficients measure the strength of the relationship between the regressors and sensitivity and specificity, respectively.

Models 1 and 2 assume that the data are independent. However, here, the data are clustered, since surfaces are nested within teeth, and teeth are nested within mouths. Therefore, the models were extended further to account for this clustering using the GEE approach. This approach is based on supposing at the start a correlation structure for the outcomes, called working correlation matrix. The term ‘working’ refers to the fact that it is good enough that the correlation matrix roughly represents the true correlation structure. Here, an exchangeable working correlation was assumed, which means that the correlation of CE among surfaces on the same tooth and on teeth in the same mouth are all equal.