Translate this page into:
Experience with 3-anchor script concordance test in undergraduate clinical assessment: A cross-sectional study
*Corresponding author: Kaberi Biswas, Department of Ophthalmology, Sree Narayana Institute of Medical Sciences, Chalakka, Kerala, India. kaberiferoze@gmail.com
-
Received: ,
Accepted: ,
How to cite this article: Kaliyadan F, Biswas K, Al Dossari S. Experience with 3-anchor script concordance test in undergraduate clinical assessment: A cross-sectional study. J Skin Sex Transm Dis. 2025;7:25-8. doi: 10.25259/JSSTD_73_2024
Abstract
Objectives:
The aim of this study was to assess the item analysis parameters of a 3-anchor script concordance test (SCT) in terms of discrimination and difficulty indices.
Materials and Methods:
After getting the required permissions, four SCT questions with a 3-point Likert scale were used among a total of 30 questions after faculty and student orientation.
Results:
The average discrimination index of the SCTs was 0.12 (average discrimination for all questions was 0.22), and the average difficulty index for the SCTs was 0.79 (average difficulty index for all questions was 0.69).
Limitations:
There are very few studies available to assess the validity of SCT. A larger study to compare three-item and five-item SCT is planned.
Conclusion:
The study showed that both discrimination and the difficulty of the SCT were acceptable but less than the average for the whole test. We attribute this to the element of guessing, which can be part of the SCT. While SCTs may be an effective and more objective method of assessing higher-level PS as compared to open-ended questions, the element of guessing needs to be factored in while designing the test.
Keywords
Difficulty index
Discrimination index
Multiple choice questions
Problem-solving questions
Script concordance test
INTRODUCTION
Clinical assessment in undergraduate Dermatology is challenging because of multiple reasons. Undergraduate exposure to dermatology cases often tends to be less compared to some other broad specialties, and therefore, ensuring good quality, valid, and reliable assessment becomes important. While skill assessment can be done using standard tools like objective structural clinical examinations (OSCEs) (including e-OSCEs), written assessment based on tools like multiple choice questions (MCQs), self-assessment questionnaires (SAQs), or structured essays are fraught with issues related to validity in the context of clinical dermatological assessment (especially when clinical reasoning needs be assessed and considering the fact that adding good quality clinical images for written assessment is often not very practical). One of the tools which could help in this context is the Script-Concordance Test (SCT). Undergraduate curriculum in Dermatology can be covered with a relatively lesser number of scripts as compared to broader specialties. We aimed to study the use of SCT in Dermatology in conjunction with other specialties that have a similar credit hour system in our curriculum, like ophthalmology.
Since the adoption of the problem-based learning (PBL) curriculum in 1969 at the Mac Master University, there has been an increasing acceptance of this curriculum, with studies showing that students from a PBL curriculum were found to have superior problem-solving (PS) skills, interpersonal communication, and adjustment to the challenges of clinical clerkship.[1,2] The PBL curriculum in the present medical college was utilizing mainly MCQs as an assessment tool in the first 3 years (preclinical years) and open-ended PS questions to assess clinical reasoning skills in years 4 and 5 (clinical years).[3-9] Item analysis is an advantage of MCQ questions and testing higher order skills that of PS questions. To combine the advantages of each format, SCT questions were brought into play, which help to assess competency in situations similar to real-life clinical problems.[8,9] Conventionally, a 5-item SCT is used. However, the difficulty in constructing a 5-item SCT led the authors to think of developing a 3-item SCT. This study was conducted with the aim of assessing the item analysis parameters of a 3-anchor SCT in terms of discrimination and difficulty indices.
MATERIALS AND METHODS
This was a single institutional cross-sectional study for assessing the effectiveness of this novel assessment tool conducted in a medical college in Saudi Arabia. Prior training was given to the faculty and students were oriented about SCTs before the exams. All of the year 4 students (200 students) in the medical college attempted the SCT questions in examinations of two specialties. Data was acquired with the appropriate permission of the authorities. Institutional Ethical Committee clearance was not necessary for this study, as per institutional guidelines.
Construction of SCTs
The “script theory,” which forms the basis for the SCT question, suggests that clinicians use networks or organized knowledge (scripts) to approach and solve clinical problems.[10] For example, on seeing a patient presenting with scaly, erythematous plaques, a dermatologist activates his/her knowledge networks related to the possible differential diagnosis to reach a final diagnosis and plan further management. SCT questions assess the clinical judgment of the candidate and compare it to the scoring given by expert judges based on the degree of concordance.[11] Short scenarios with some degree of ambiguity are used. A possible hypothesis is suggested, and each scenario is followed by sub-questions, each of which gives some additional information based on which the student has to decide on whether the hypothesis is strengthened or weakened.[6]
A simple example is given [Table 1]. For scoring, the questions are scored by expert judges (the one disadvantage of the classic SCT is that it requires a large number of judges, like in standard setting methods like the Angoff ’s method). The score for each question is equal to the number of judges that chose that answer, divided by the mode for the same question.
| A 25-year-old female patient presents to the dermatology clinic with a hypopigmented patch on her face, which has been there for the past 3 months. A clinical diagnosis of vitiligo is considered. Chose: -2, -1, 0, 1, 2 (–2 hypothesis strongly weakened to 2 –hypothesis strongly strengthened) based on the extra information provided. | |
| The lesions show significant scaling | –2, –1, 0, 1, 2 |
| There is itching over the lesions | –2, –1, 0, 1, 2 |
| Wood’s light examination show accentuation of the hypopigmentation | –2, –1, 0, 1, 2 |
| White hairs are seen over the affected areas | –2, –1, 0, 1, 2 |
SCT: Script concordance test
For example, if, for the above question A, 8 panel members chose “−2,” and 2 chose “−1,” mode will be 8. Credit for a student who chose −2 will be 8/8 = 1, and for one who chose −1, the credit will be 2/8 = 0.25.[8]
Modification of the SCT
Keeping in mind the disadvantage of a large number of judges in the classic SCT, a modified form of the prototype SCT was constructed to reduce the need for a large panel of judges by reducing the Likert scale to a three-option one. The 3-point scale would also obviate the tendency of students to select extreme scores. The advantages and disadvantages of the 3-item SCT are listed in Table 2.
| Advantage | Disadvantage |
|---|---|
| Easy to design | Item analysis parameters are lower compared to 5 point SCT |
| Requires less number of judges | Guesswork is possible |
| Acceptableitem analysis parameters | |
| Effective problem-solving assessment tool |
SCT: Script concordance test
A modified 3-point scale-based SCT was used in exams of two specialties – dermatology and ophthalmology. Four SCT questions with a 3-point Likert scale were used among a total of 30 questions. Four brief case vignettes with four subquestions and a three-option response were administered in the dermatology and ophthalmology examinations, respectively.
Some of the sample questions are given in Tables 3 and 4. Following this, the effectiveness of the 3-item SCT was evaluated by item analysis in terms of difficulty and discrimination indices. The difficulty and discrimination indices of the SCT questions were then compared to the average difficulty and discrimination indices of all the questions in the examination.
| A 16-year-old male patient presents with a vesicular rash on his trunk of 5 day duration. Choose -1, 0, 1 (1-hypothesis strengthened, 0-hypothesis unchanged, –1-hypothesis weakened). | |
| You suspect herpes zoster and you find that the lesions are unilateral | –1, 0, 1 |
| You suspect varicella and during history taking the patient says that he has severe pain over the lesions | –1, 0, 1 |
| You suspect a localized irritant contact dermatitis and a Tzanck smear shows many multinucleate giant cells | –1, 0, 1 |
| You suspect tinea corporis – vesicular type and start the patient on a topical antifungal. The lesions disappear in 10 days but the patient still has some pain | –1, 0, 1 |
SCT: Script concordance test
| A 45-year-old man comes to the ophthalmologist with complaints of a right-sided red eye, which started 2 days back. Choose –1, 0,1 (–1-hypothesis weakened, 0-hypothesis unchanged, 1-hypothesis strengthened) | |
| You suspect a diagnosis of iridocyclitis. The pupil is widely dilated and fixed. | –1, 0, 1 |
| You suspect a Gonococcal conjunctivitis. The discharge is watery | –1, 0, 1 |
| The upper tarsal conjunctiva shows a follicular reaction. The patient is suspected to have an allergic conjunctivitis | –1, 0, 1 |
| You suspect a herpetic keratitis. There is a dendritic lesion on the cornea. | –1, 0, 1 |
SCT: Script concordance test
The discrimination index is the point biserial correlation coefficient between the item in question and the mean score.[12,13] For the discrimination index of open-ended questions, there are no standard methods. We used the formula ([SUM (scores of top quartile)-SUM (lowest quartile)]/4), where “4” is the maximum possible score on an item. On the other hand, the difficulty index refers to the total number of correct responses to the item in question.[14] For difficulty analysis, the standard method followed everywhere is “total score obtained for the item (all students)/maximum total score” The same was done here. The optimum range is 0.2–0.8.[15]
RESULTS
In the fourth year theory examination, four 3-item SCT questions were administered out of a total of 30 PS scenario-based questions. The average discrimination index of the SCT questions was compared to the universal discrimination index. The average discrimination index of the SCT questions was 0.12, and the universal discrimination index of the examination was 0.22.
The average difficulty index of the SCT questions was compared to the universal difficulty index of all questions. The average difficulty index of the SCT questions was 0.7925, and the examination, in general, was found to be 0.695.
DISCUSSION
In this study, a modified 3-point scale-based SCT was used in exams of two specialties and the effectiveness of the same was evaluated in terms of discrimination and difficulty indices. Four SCT questions with a 3-point Likert scale were used among a total of 30 questions. The questions were prepared after the faculty and the students were given adequate orientation regarding the concept of SCTs.
As in SCT-based examinations elsewhere, the item analysis involved the difficulty index and discrimination index.[16] The average difficulty index of the SCT questions was 0.7925, and the average discrimination index was 0.12. In a 5-item SCT test, it was noted that the difficulty index ranged from 0.39 to 0.59 and the discrimination index from 0.32 to 0.66.[16] Item analysis of this novel 3-item SCT examination shows that both difficulty and discrimination were acceptable but lower than the average for the whole test. This could be attributed to an element of guessing, which can be part of the SCT. Making the SCT scenarios more elaborate with subquestions, which make the students think and analyze the options could help to make guesswork difficult and, thus, have more acceptable item analysis results. Studies have shown that SCT scores positively correlate with increasing clinical training and levels of knowledge.[15] In medical postgraduate teaching, where assessment has traditionally been by the long essay mode, SCTs could provide an assessment tool to assess both background knowledge and clinical reasoning skills.
Limitations of this study
Although the study was undertaken to assess the efficacy of an easy-to-design 3-point SCT, there are very few studies available to assess SCT validity, especially this novel form of SCT. It is also planned to design research on a larger scale to compare the 3-item SCT with the conventional 5-item SCT.
CONCLUSION
While SCTs may be an effective and more objective method of assessing higher-level PS skills as compared to open-ended questions, the fact that SCTs can have an element of guessing needs to be factored in while designing the test. Making more complex scenarios that require an enhanced level of analysis and PS skills could help. It is planned to include SCT scenarios with therapeutic problems to make the examination more objective, especially in the fifth year medical curriculum. Follow-up feedback questionnaires are also planned to further improve the quality of the SCTs. Although SCTs are acknowledged as an effective assessment tool, this novel 3-point SCT was found to have acceptable item analysis parameters and could be used as an effective PS tool, with some minor tweaks and modifications. However, further evaluation in more examinations and deeper statistical analysis is needed before the 3-item analysis is implemented as an effective assessment tool in both the clinical years of medical school as well as in post-graduate examinations.
Ethical approval:
The Institutional Review Board approval is not required according to the institutional policy.
Declaration of patient consent:
Patient’s consent was not required as there are no patients in this study.
Conflicts of interest:
Dr. Feroze Kaliyadan is on the editorial board of the Journal.
Use of artificial intelligence (AI)-assisted technology for manuscript preparation:
The authors confirm that there was no use of artificial intelligence (AI)-assisted technology for assisting in the writing or editing of the manuscript and no images were manipulated using AI.
Financial support and sponsorship: Nil.
References
- The McMaster M.D. Program: A case study of renewal in medical education. Acad Med. 1989;64:423-32.
- [CrossRef] [PubMed] [Google Scholar]
- Undergraduate medical education: Comparison of problem-based learning and conventional teaching. Hong Kong Med J. 2000;6:301-6.
- [Google Scholar]
- Evaluation of modified essay questions (MEQ) and multiple choice questions (MCQ) as a tool for assessing the cognitive skills of undergraduate medical students. Int J Health Sci (Qassim). 2011;5:39-43.
- [Google Scholar]
- Should essays and other “open-ended”-type questions retain a place in written summative assessment in clinical medicine? BMC Med Educ. 2014;14:249.
- [CrossRef] [PubMed] [Google Scholar]
- Assessment of higher order cognitive skills in undergraduate education: Modified essay or multiple choice questions? Research paper. BMC Med Educ. 2007;7:49.
- [CrossRef] [PubMed] [Google Scholar]
- Script concordance tests: Guidelines for construction. BMC Med Inform Decis Mak. 2008;8:18.
- [CrossRef] [PubMed] [Google Scholar]
- Using the script concordance test to assess clinical reasoning skills in undergraduate and postgraduate medicine. Hong Kong Med J. 2015;21:455-61.
- [CrossRef] [PubMed] [Google Scholar]
- Script concordance test acceptability and utility for assessing medical students' clinical reasoning: A user's survey and an institutional prospective evaluation of students' scores. BMC Med Educ. 2022;22:277.
- [CrossRef] [PubMed] [Google Scholar]
- Minimum accepted competency examination: Test item analysis. BMC Med Educ. 2022;22:400.
- [CrossRef] [PubMed] [Google Scholar]
- The script concordance test: A tool to assess the reflective clinician. Teach Learn Med. 2000;12:189-95.
- [CrossRef] [PubMed] [Google Scholar]
- Medical students' assessment preferences at king Faisal university, Saudi Arabia. Adv Med Educ Pract. 2011;2:95-103.
- [CrossRef] [PubMed] [Google Scholar]
- Measuring gains in the clinical reasoning of medical students: Longitudinal results from a school-wide script concordance test. Acad Med. 2014;89:1046-50.
- [CrossRef] [PubMed] [Google Scholar]
- Relationship between item difficulty and discrimination indices in true/false-type multiple choice questions of a para-clinical multidisciplinary paper. Ann Acad Med Singap. 2006;35:67-71.
- [CrossRef] [Google Scholar]
- Difficulty index, discrimination index, sensitivity and specificity of long case and multiple choice questions to predict medical students' examination performance. J Taibah Univ Med Sci. 2014;9:110-4.
- [CrossRef] [Google Scholar]
- Script concordance tests for formative clinical reasoning and problem-solving assessment in general pediatrics. MedEdPORTAL. 2022;18:11274.
- [CrossRef] [PubMed] [Google Scholar]
- Development and psychometrics of script concordance test (SCT) in midwifery. Med J Islam Repub Iran. 2018;32:75.
- [CrossRef] [PubMed] [Google Scholar]
