Open Journal Systems

ORIGINAL CONTRIBUTIONS

Received: March 10^th 2014

Accepted: October 27^th 2014

Reliability and validity of an instrument to evaluate integral clinical competence in medical residents

Ana Carolina Sepúlveda-Vildósola,^a Nadia Gaspar-López^b Lucina Isabel Reyes-Lagunes,^c Héctor Jaime Gonzalez-Cabello^d

^aDirección de Educación e Investigación en Salud

^bMédico residente de la especialidad de Pediatría

^cFacultad de Psicología, Universidad Nacional Autónoma de México

^dJefatura de la División de Educación Médica

^a,b,dHospital de Pediatría “Dr. Silvestre Frenk Freund”, Centro Médico Nacional Siglo XXI, Instituto Mexicano del Seguro Social

Distrito Federal, México

Communication with: Ana Carolina Sepúlveda-Vildósola

Telephone: (55) 5627 6900, extension 22306

Email: ana.sepulvedav@imss.gob.mx

Background: The evaluation of clinical competence in medical residents is a complex procedure. Teachers need reliable and valid instruments to evaluate objectively the, clinical competence. The aim of this study was to determine the reliability and validity of an instrument designed to evaluate the clinical competence of medical residents.

Methods: We designed an instrument taking into consideration every part of the clinical method, and three different levels of competence were determined for each one. The instrument was examined with regards to the clarity, pertinence and sufficiency of each clinical indicator by five expert pediatricians. The instrument was finally constituted by 11 indicators. Each resident was evaluated independently by three pediatricians.

Results: A total of 651 measurements were done in 234 residents. The instrument distinguished between extreme groups, had a value of Cronbach´s alpha of 0.778 and the factorial analysis set apart two factors: clinical competence and complementary competences. No statistical differences were found between evaluators neither in global evaluation or in each indicator.

Conclusion: The instrument here proposed is valid and reliable. It may be used in formative evaluation of medical residents in clinical specialization programs.

Keywords: Evaluation; Clinical competence; Medical residents

---

Mexico has been slowly moving toward a competency approach to education in medicine. The objective is that the professional be able to solve health problems that our society demands, and adapt to uncertainty, change, and complex new scenarios. Health institutions that train human resources should provide processes and environments that let individuals develop the skills required to adapt to constantly changing organizations and to the needs of the society in which they live. Learning must be done in the context of real experiences that allow them to be creative in solving these problems.¹

In recent years, the evaluation of clinical competency has become more important and, despite being a complex activity, it is an essential and fundamental task for feedback in the educational process and is the mechanism by which institutions and organizations can guarantee the public adequate performance by trained doctors. However, there are serious problems in the process of evaluating the clinical competency of doctors, both from a conceptual perspective and from the instruments available to do it. Most of these instruments assess one competency or, at best, two.²

The available tools can be divided into three categories: formal, structured, circumscribed, and ex vivo assessments (written, oral examinations or OSCE);^3-7 formal, structured, circumscribed, in situ examinations (Mini Clinical Evaluation Exercise or mini-CEX); and informal, unstructured, summative, in situ assessments (In-training Evaluation Report). In general, the evaluation of clinical competency requires direct observation of the clinical skills of students and the use of valid and reliable instruments to objectivize the process. However, the systematic literature review conducted in 2009 by Kogan et al. found that, of 32 instruments that assess the competency of residents, only half reported the evaluation of the reliability of the instrument; the mini-CEX was the most used and most reliable.^8,9 However, despite attempts to develop objective, standardized, valid, and reliable tools to achieve objectivity,¹⁰ one of the greatest difficulties is the subjectivity of the assessors, which has been attributed to factors such as idiosyncratic judgments in the "first impression", their frames of reference, inference levels used in the process of observation, or their clinical skills. These authors found no differences by demographic factors such as age, gender, or clinical experience.^11-13 Margolis et al. recommend a greater number of assessors for each observation to improve the stability of reported tests.¹⁴ Other strategies include the use of portfolios, reviewing records, simulations and models, examination with standardized patients, or reviewing videos of performance with a patient.^15-21

According to this context, it is understood that there is no single way to assess clinical competencies and that forms of evaluation are complementary, since no one comprehensively evaluates the process.

In Mexico, evaluation of clinical competency has been done in the undergraduate of the Facultad de Medicina of the UNAM through OSCE.²² However, its limitations have been documented: the knowledge and skills of students are examined piecemeal, some of the "stations" are artificial, and also the cost, time spent, and personnel involved in the development and implementation is higher than in traditional tests. In graduate medical education, clinical expertise of the resident can only be determined by observation and the evaluation that they provide patients during their training. Other authors found adequate reliability in the comprehensive evaluation of clinical competency through professional examination,²³ or the use of written tests built with real cases in various degrees.^24-26 At Hospital Infantil de Mexico, Federico Gomez began to use the mini-CEX as a tool in the evaluation of clinical competency of pediatric residents.²⁷ In Spanish-language literature we found no objective instrument that is contextually relevant and that validly and reliably evaluates clinical performance of specialty students with a real patient.   

In the Hospital de Pediatría of the Unidad Médica de Alta Especialidad of the Centro Médico Nacional Siglo XXI, an instrument for the evaluation of clinical competency (IECC) was developed. To be considered effective, the instrument must be valid and reliable. The aim of this study was to estimate the reliability and validity of the IECC in the evaluation of clinical competency of medical residents of the Hospital de Pediatría of the Centro Médico Nacional Siglo XXI, and to determine inter-observer agreement.

Methods

The study was conducted at the Unidad Médica de Alta Especialidad (UMAE), Hospital de Pediatría, Centro Médico Nacional Siglo XXI.

Phase 1. Instrument design

1. In 2012 an instrument for assessing overall clinical competency was made by the following steps:

a) A checklist of the different steps carried out in the clinical method was made, and three proficiency levels were determined for each. The instrument was built based on what is proposed in the literature, namely from indicators mentioned in instruments used to measure clinical competency.

b) The instrument was subjected to review by a group of five pediatric doctors who are experts in the hospital with over 20 years of clinical experience. These experts determined the representativeness of the items and the use of sensitive methods for the construction of the test. They were asked their opinions on the clarity, relevance, and sufficiency of the indicators proposed. The instrument was supplemented in accordance with their comments. 

2. The instrument was composed of 11 indicators, described in Annex 1. Three competency levels were determined for each (satisfactory, unsatisfactory, and unacceptable).

The instrument was applied to a pilot group of second and third year residents of specialization in pediatrics to ensure understanding and clarity of the prompts.

Phase 2. Implementation of the instrument

Records obtained from the comprehensive clinical competency evaluation done on resident physicians who completed a specialty in UMAE Hospital de Pediatría of the CMN Siglo XXI from August 2012 to December 2013 were included. Incomplete or illegible evaluations were removed.

Each resident was evaluated by three evaluators (one from the Dirección de Educación e Investigación en Salud, one regular or deputy assessor of the specialty course, and another evaluator-teacher assistant in the specialty in which the resident would be evaluated). The professors chose the patient based on the illnesses in the university modules of each grade level. The parent/guardian of the child, and, when older than 9 years, the patient were asked for voluntary cooperation in the clinical evaluation. Residents were explained the examination procedure, which consisted of an initial phase, in which they would examine and interview the patient, and a second phase in the classroom, where residents would be questioned about their reasoning and to explain their medical knowledge. Residents were given as much time as they needed for interviewing and physical examination of the patient without intervention, except if the examination could injure the patient or if it could present complications.

Each evaluator was asked to independently qualify each category of the instrument. At the end, the points obtained for the execution of each indicator were added, and the final grade was assigned, as referred to in Annex 1. The final exam score of each student was obtained from the average of the three grades of the various assessors.

Scores were captured in an SPSS, version 20 database. Descriptive analysis of the variables was done using frequencies and percentages for qualitative variables and mean and standard deviation for quantitative variables. The discriminant analysis of the prompts was done with the Student’s t test for extreme groups. Determining the validity of the instrument was done using factorial analysis of principal components with orthogonal rotation. The reliability of the instrument was determined by Cronbach's alpha coefficient. Inter-observer agreement was determined with ANOVA.

The protocol was approved by the Local Research and Ethics Committee, with the registration number R-2012-3603-87.

Results

651 total measurements were obtained from 234 residents with an average of 2.78 measurements per resident. 67% of evaluations (438 records) were obtained in the 2012-2013 school year and 213 records were obtained in the 2013-2014 school year.

By grade level of residents, seven evaluations were of first year residents, 135 second year, 181 third year, 60 fourth year, 123 fifth year, and 145 sixth year.

51.9% of evaluations were of residents in the specialty of Pediatrics, while the remaining 48.1% was represented by specialties within Pediatrics.

As for the scores obtained, 13.2% obtained a rating of 5, 13.4% got a rating of 6, 18.9% of 7, 18.1% of 8, 21.4% of 9, and 15% of 10. The average score was 7.66 (± 1.617).

The discriminant analysis found a statistically significant difference between the groups with lower and higher test performance (25th percentile or under and 75th or higher, respectively) (Table I).

Reliability analysis of 11 elements obtained a Cronbach's alpha value of 0.778, with a Cronbach's alpha based on established elements of 0.818 and a Hotelling T2 of 0.000.

In Table II it can be seen that none of the test items adversely affect the test reliability.

According to the maximum possible score for each of the categories, the sections of presentation, communication, and priority programs showed the best performance, followed by current condition, prognosis, and diagnostic complement. Areas with lower performance were treatment plan, diagnostic integration, physical examination, and theoretical foundation (Table III).

Factorial analysis was conducted to determine the validity of the instrument, and it found that the items were clearly grouped into two main factors (Table IV).

a) Factor 1: Clinical competency, which includes the components of the medical act, such as obtaining information through the current condition, interview, and physical examination, as well as the components of clinical diagnostic reasoning, composed of diagnostic complement, diagnostic integration, treatment plan, theoretical foundation, and prognosis.

b) Factor 2: Complementary skills, composed of presentation, communication, and knowledge of priority programs.

Regarding agreement between observers, no statistically significant differences were found in the values ​​obtained by each evaluator in different test items, so there was a good inter-observer agreement (Table V).

Discussion

The evaluation of comprehensive clinical competency is a primary objective of the institutions involved in the training of health professionals. It is not enough to design excellent training programs that receive accreditation by the relevant bodies, but one must also prove that their application produces the positive impact desired. Therefore, it is essential to do continuous, rigorous, and specific assessment of the medical resident in specialization.¹⁶

However, the evaluation of clinical competency is difficult to carry out, because the set of knowledge, skills, and attitudes have to be assessed integrally.^11-14 Traditionally, student evaluation is done in a fragmented way, i.e., in different times and through different instruments.

At IMSS, cognitive evaluation is undertaken through multiple-choice tests, while psychomotor skills are evaluated in an open format called CEM 2, in which professors note the expertise gained by the resident, without any checklist to endorse the measure, so that its reliability and validity are questionable.

The attitudes of the residents are assessed through the CEM 3 format, which includes aspects such as professional behavior, judgment, interpersonal relationships, discipline, fulfillment of academic activities, criticism, responsibility, and commitment, which the professor rates on a scale from 0 to 100. To be clear, psychometric study of these evaluation forms has never been done.

The proposed instrument for evaluation of clinical competency attempts to integrate the main aspects of knowledge, skills, and attitudes, which, when performed in a real environment, offer results of resident performance roughly similar to how it will be in their professional life.

Given the results of discrimination, it is useful in evaluating the clinical competency of residents, given that, beyond just being easy to apply, it integrates the qualities required to conduct an assessment of clinical competency.

It is noteworthy that the areas of interview and physical examination obtained the lowest average scores, as they are widely covered in undergraduate medical education, and one would expect students to already have mastery of these skills. In regard to treatment plan, diagnostic integration, and theoretical foundation, which also had lower scores, these correspond to knowledge and skills that each resident must acquire for the specialty, which should be reinforced by their professors.

We believe that the main bias of the test is related to student performance, which may be affected by two main aspects, the stress of the evaluation itself, and the fact of being observed, making the resident more meticulous in the interviewing and physical examination of the patient, which is not necessarily how they do it daily. Other problems were related to the very process of patient care, such as phone calls to relatives, interruptions from the nursing area to apply medication or request tests from the lab, and so on.

The time of application of the test ranged from 45 to 150 minutes, with an average of 120. In this case, residents had free time for interview and physical examination. One would have to determine if the reliability and validity of the instrument could be affected if time is limited.

The main purpose of the evaluation was to give feedback on the educational process, in order to establish areas for improvement in the process. The integration of this evaluation instrument is not intended to replace other forms of evaluation that are done with medical residents; rather it is intended as a complementary form of evaluation, and the frequency of application should be weighed by each of the professors or institutions that adopt it.

New trends in medical education allow us to envision an educational scenario in which aspects of the quality standards of education will acquire increasing relevance, with new teaching methodologies, and with advances in the evaluation of knowledge and acquired skills, so that having valid and reliable instruments would allow us to comprehensively assess the training of residents and plan necessary changes in the academic programs, so that in the future the performance of specialists graduated from various training institutions gets better results.

Conclusion

The proposed instrument is valid and reliable. It is proposed as a training tool in the evaluation of medical residents in clinical specialties.

References

1. Fraser S., Greenhalgh T. Coping with complexity: educating for capability. BMJ 2001; 323: 799-803.
2. Regehr G. Kevin E., Ginsburg S. et al. Assessment in postgraduate medical education: trens and issues in assessment in the workplace. Members of the FMEC PG consortium; 2011.
3. Sloan DA, Donelly MB, Schwarts RW. Strodel WE. The Objetive structured clinical examination. The new gold standard for evaluating posgraduate clinical performance. Annals of Surgery 1995; 222(6):735-42
4. Hull AL, Hodder MD, Berger B, Ginsberg D, Lindheim N, Quan J.Validity of three clinical perfomance assessments of International Medicine Clerks. Acad Med 1995; 70(6): 517-522.
5. Harden RM, Gleeson FA. Assessment of Clinical Competence Using an Objetive Structured Clinical Examination (OSCE) Med Educ 1979;13: 41-54.
6. Dupras DM, Li JT. Use of an Objetive Structured Clinical Competence Acad Med 1995;70 (11): 1029-34.
7. Selby C, Osman L, Davis M, Lee M. How to do it. Set up and run.an objetive structured clinical exam. Br Med Journal. 1995; 310: 1187-90.
8. Kogan J., Holmboe E., Hauer K. Tools for direct observation and assessment of clinical skills of medical trainees. A systematic review. JAMA 2009; 302(12): 1316-1326.
9. Cook D., Bechman T., Mandrekar J, et al. Internal structure of mini-CEX scores for internal medicine residents: factor analysis and generalizability. Adv Health Sci Educ 2010; 15(5): 633-645
10. Ginsburg S., Mciloy J., Oulanova O., et al. Toward authentic clinical evaluation: pitfalls in the pursuit of competency. Academic Medicine 2010; 85(5): 780-786
11. Gingerich A., Regehr G., Eva K. Rater-based assessments as social judgements:rethinking the etiology of rater errors. Academic Medicine 2011; 86(10): 51-57
12. Kogan J., Conforti L., Bernabeo E, et al. Opening the black box of clinical skills assessment via observation: a conceptual model. Medical Education 2011; 45: 1048-1060.
13. Kogan J., Hess B., ConfortiL, et al. What drives faculty ratings of residents´clinical skills? The impact of faculty´s own clinical skills. Academic Medicine 2010; 85(10): 525-528.
14. Margolis M., Clauser B., Cuddy M, et al. Use of mini-clinical evaluation exercise to rate examinee performance on a multiple-station clinical skills examination: a validity study. Academic Medicine 2006; 81(10): 556-560
15. Epstein RM. Assessment in medical education. N Engl J Med 2007; 356 (4): 387-96.
16. Van der Vleuten C, Swanson DB. Assessment of clinical skills with standardized patients: state of the art. Teach Learn Med 1990; 2: 58–76.
17. Hays RB, Van der Vleuten C, Fabb WE, Spike NA. Longi- tudinal reliability of the Royal Australian College of General Practitioners certification examination. Med Educ 1995; 29: 317-321.
18. Page G, Bordage G, Allen T. Developing key-feature problems and examinations to assess clinical decision- making skills. Acad Med 1995; 70: 194-201.
19. Challis M. AMEE medical education guide no. 11: Portfolio- based learning and assessment in medical education. Med Teach 1999; 21: 370-386.
20. Van Der Vleuten C. Validity of final examinations in under- graduate medical training. BMJ 2000; 321: 1217-1219.
21. Accreditation Council for Graduate Medical Education (ACGME)-American Board of Medical Specialties (ABMS). ACGME/ABMS Joint Initiative Toolbox of Assessment Methods Version 1.1 September 2000. p. 1-3.
22. Trejo Mejia JA., Martinez Gonzalez A., Mendz Ramirez I, et al. Evaluación de la competencia clínica con el examen clínico objetivo estructurado en el internado médico de la Universidad Nacional Autónoma de México. Gac Med Mex 2014; 150: 8-17
23. Ponce de Leon-Castañeda ME., Ortiz-Montalvo A., Bonilla-Gonzalez I., et al. Evaluacion integral de la competencia clínica a través del examen profesional. Arch Med Fam 2006; 8(2):119-129
24. Soler-Huerta E., Sabido-Sighler C., Sainz-Vazquez L., et al. Confiabilidad de un instrumento para evaluar la aptitud clínica en residentes de medicina familiar. Arch Med Fam 2005; 7(1): 14-17
25. Pantoja Palmeros M., Barrera Monita J., Insfran Sanchez M. Instrumento para evaluar aptitud clínica en anestesiología. Rev Med Imss 2003; 41(1): 15-22.
26. Degollado Bardales L., Leyva Gonzalez FA., Viniegra Velazquez L., et al. Competencia clínica de los residentes de anestesiología en pacientes que requieren atención médica en un hospital de especialidades. Gac Med Mex 2009; 145(6): 481-489
27. Mendez Zarandona AP., Gutierrez Camacho C., Peña Alonso R. Estudio piloto para la validación de un instrumento de evaluación de habilidades y destrezas clínicas en pediatras en formación. Available from http://www.facmed.unam.mx/sem/presentaciones/ARF/EV1.pdf

Enlaces refback

• No hay ningún enlace refback.