Assessment of diagnostic reasoning with standardized patients: Testing the reliability and validity of the diagnostic reasoning assessment

Abstract

The purpose of this study was to test the psychometric properties of the Diagnostic Reasoning Assessment (DRA), an assessment instrument developed by the author. The DRA assesses the level of expertise of the diagnostic reasoning process in graduate level nurse practitioner (NP) students during performance exams using standardized patients. The study employed a one group cross-sectional design for the purpose of evaluating the psychometric properties of the DRA. Content validity was assessed by content experts and the instrument was revised based on their feedback. NP students' performance was assessed by two NP faculty raters and a SP rater using the DRA. Generalizability studies were performed for all the DRA, SOAP (SOAP Note Evaluation Tool) and SCT (Script Concordance Test) to determine the variance components of the three measures. Evidence for the construct validity of the DRA was examined by comparing correlations between the DRA and two other instruments, the SOAP note evaluation tool and the Script Concordance Test. Generalizability analysis of the DRA followed a two facet, person x rater x item design. The generalizability coefficient (rho 2) was 0.81 for all raters. The largest source of variance for this study was the person component accounting for 44% of the variance, followed by the residual with 24% and the person by rater interaction at 23%. The relationship was non-significant with r (49) = .26, p = .071 for the analysis of all raters on the DRA and SCT. For the DRA (all raters) and SOAP, the relationship was significant with r (47) = .39, p = .006. The scores on the DRA (faculty raters) were significantly related to the scores on the SCT, r (49) = .32, p = .024 and the SOAP r (47) = .44, p = .002. The relationship of the SP ratings on the DRA and the other instruments was nonsignificant with the SCT, r (49) = .04, p = .79 and the SOAP, r (47) = .17, p = .26. Though some of these correlations were significant, the strength of the relationships are low. This study offers support for the psychometric properties of the Diagnostic Reasoning Assessment.