Open Journal Systems

EDUCATION

Received: April 14^th 2015

Accepted: October 21^st 2015

Teacher’s perfomance assessment in Family Medicine specialization

Adrián Martínez-González,^a Francisco J. Gómez-Clavelina,^b Isaías Hernández-Torres,^b Fernando Flores-Hernández,^a Melchor Sánchez-Mendiola^a

^aSecretaría de Educación Médica

^bSubdivisión de Medicina Familiar, División de Estudios de Posgrado

Facultad de Medicina, Universidad Nacional Autónoma de México, Ciudad de México, México

Communication with: Adrián Martínez-González

Telephone: (55) 3939 5204

Email: adrianmartinez38@gmail.com

Background: In Mexico there is no systematic evaluation of teachers in medical specialties. It is difficult to identify appropriate teaching practices. The lack of evaluation has limited the recognition and improvement of teaching. The objective of this study was to analyze feedback from students about teaching activities of teachers-tutors responsible for the specialization course in family medicine, and evaluate the evidence of reliability and validity of the instrument applied online.

Methods: It was an observational and cross-sectional study. Seventy eight teachers of Family Medicine of medical residency were evaluated by 734 resident´s opinion. The anonymous questionnaire to assess teaching performance by resident’s opinion and it is composed of 5 dimensions using a Likert scale. Descriptive and inferential statistics (t test, one-way ANOVA and factor analysis) were used.

Results: Residents stated that teaching performance is acceptable, with an average of 4.25 ± 0.93. The best valued dimension was “Methodology” with an average of 4.34 ±.92 in contrast to the “assessment” dimension with 4.16 ± 1.04.

Conclusions: Teachers of specialization in family medicine have acceptable performance by resident’s opinion. The online assessment tool meets the criteria of validity and reliability.

Keywords: Educational measurement; Learning; Medical faculty; Graduate medical education

---

Family Medicine is a medical specialty that is taught in 84 academic venues of institutions of the Sistema Nacional de Salud, such as the Instituto Mexicano del Seguro Social (IMSS), the Instituto de Seguridad y Servicios Sociales de los Trabajadores del Estado (ISSSTE), the Secretaría de Salud (SSA) of Mexico City and the State of Queretaro, and the Instituto de Seguridad Social of the State of Mexico and Municipalities (ISSEMyM). The academic venues are located in 30 states of Mexico with 1168 students.

The academic program is run by the Single Plan of Medical Specializations (Plan Único de Especializaciones Médicas, PUEM), which includes four areas of training: Seminar on Health Care, Health Care Work, Research Seminar, and Medical Education Workshop, which are held for three years. Resident doctors do rotations in secondary and tertiary level hospital subsites for six months and annually, where they receive training in core medical areas such as internal medicine, obstetrics and gynecology, pediatrics, and emergency medicine.

The profile of the Family Medicine physician graduate includes three aspects of educational attainment expected from the formal education process; this is: professional-humanistic orientation (being), intellectual formation (knowing), and the operating performance of the medical specialist (know-how). In this profile, the medical professional is considered responsible for granting comprehensive, continuous medical attention to the individual and their family, so they must have great clinical, teaching, administrative, and research training, as well as links of shared accountability with the health team to address the health-disease process; they attend patients in need appropriately and in a timely manner, and act with humanism and adherence to professional ethics, with strong social and institutional awareness.

Through a medical practice supported by an educational methodology in problem-solving, it is proposed to make the graduate a highly competent doctor specialized in their specific field of action. The educational methodology proposed gives great importance to the development of knowledge, the acquisition of attitudes and values, the development of independent study habits, and the stimulation of critical and reflective thinking by residents to make smart decisions related to problem-solving for individual health as well as training specialists to high academic standards. It follows that the teacher has a key role to achieve this purpose, because their functions and activities include not only providing information during classes, but also participating as a consultant in solving clinical problems, facilitator of learning, creator of educational resources, model for future specialists, and evaluator.

To be appointed professor of the family medicine specialization course at the Universidad Nacional Autónoma de México (UNAM), one must be nominated by the Jefatura Nacional de Enseñanza of the health institution concerned, and meet the following requirements of the PUEM: a specialist in family medicine, certified or recertified by the Consejo Mexicano de Certificación en Medicina Familiar, with teaching experience and regular participation in the dissemination of medical knowledge. For tenure in giving the course, the professor is annually subjected to academic and administrative supervision by the UNAM and the health institution itself. 

The main teaching activities of the professor are academic teaching and coordination of all academic activities to be performed by the resident during their training period, which is three years. At the same time, they are part of a team of tutors in charge of the theoretical training, but also involved in the learning field, tutoring their residents in the clinic. Similarly, they are responsible for planning the different student rotations with other tutors in the same academic headquarters and the rotation by academic hospital subsite. The academic venues are medical units or health centers in primary care, while the hospital subsites are secondary care units. One must mention that depending on the health institution to which the resident physician is assigned, the tutor: student ratio is one tutor for every two to three residents. The annual turnover of pupils, obeying the needs of the health institution, is 6 to 8 months in the academic headquarters, and the rest in the hospital subsidiary, maintaining self-taught tutorial and tutored field learning in both rotations. Through continuous interaction with other tutors, the professor periodically collects the evaluations corresponding to each student and, at the end of the school year, is responsible for specifying the final evaluation and stating their grades in official documents to formally emit the resident’s definitive assessment from both the health institution and the university granting the academic endorsement (UNAM).

For the present study, we evaluated only official teachers of each academic venue, as it is the tutor-professor who constantly performs teaching activities with residents.

Evaluation is understood as any systematic method of obtaining information through reviews and other sources used to extract information about the characteristics of people, objects, and programs.¹ Teacher performance is complex and multidimensional, involving knowledge, skills, attitudes, and values ​​that, interrelated, allow successful performance according to established standards for the achievement of the graduate profile, in this case the Family Medicine specialization. The evaluation of teaching performance is defined as a process of analysis and teacher training, which values ​​the concept, practice, planning, and development of professional teaching.²

Within the context of medical education in our country, there is little research regarding teacher performance, which has limited the recognition and improvement of this work.^3,4

To conduct teacher performance assessments, there are more than twelve strategies that can be used,^5,6 such as evaluation by student opinion, peer assessment, self-assessment, evaluation by administrators, objective structured teaching exams, and others, each with strengths and weaknesses.

In connection with the evaluation of teacher performance by student opinion, the weaknesses include that some teachers feel that students do not have the maturity to evaluate the quality of teaching, and that their limited knowledge of the subject prevents them from issuing judgment; however, the assessment of student opinion has become the most used and studied teaching evaluation strategy in North America, Europe, and Asia⁷ as a source of consistent, valid, and useful information for feedback and improving teaching.^5,8,9

In the literature there is a great diversity of approaches, experiences and paradigms associated with teaching and teacher evaluation. However, their application may vary depending on the institutional context, the teaching model, the objective, and the study variables.^10-12 In the Mexican context, in general, professors of medicine specializations have heterogeneous and not always systematized teacher training.¹³

In Mexico there is no systematic evaluation of the teaching function of teachers of medical specializations, reducing the possibility that teachers fully fulfill the functions and responsibilities of a teacher, making it difficult to identify criteria to support teacher training activities, and hindering the identification of appropriate teaching practices.

The objectives of this study were to analyze students’ views of the teaching activities of teacher-tutors responsible for the family medicine specialization course, and to assess the evidence of reliability and validity of the instrument applied online.

Methods

It was an observational, cross-sectional census study. The study population consisted of full professors of family medicine at the primary care medical residency at the Faculty of Medicine of the Universidad Nacional Autónoma de México (UNAM) in the 2012-2013 school year. Inclusion criteria: teachers with recognition from UNAM; family medicine teachers who participated with informed consent. Exclusion criteria: teachers who refused to be evaluated by their residents. The variables considered were teaching performance, age, and gender.

The questionnaire used has evidence of validity and reliability, and considers five basic dimensions for the evaluation of teaching work performed by teachers of Family Medicine on the Faculty of Medicine, in terms of residents’ perception or opinion, to find out the degree to which their teachers in their job performance exhibit a set of characteristics or abilities defined as good teaching practice under the Single Plan of Medical Specializations (PUEM).¹⁴

The dimensions included in the questionnaire are: 1. Teacher-student relationship and motivation; 2. Methodology; 3. Evaluation; 4. Problem solving ability; and 5. Knowledge of the subject. The questionnaire has 37 items, each with five Likert response options: 1. never (0-19%); 2. almost never (20-39%); 3. sometimes (40-59%), 4. most of the time (60-79%), 5. always (80-100%). These dimensions are related to the tutor’s activities and functions based on the specialty’s curriculum and academic plan. For more detailed information on the process of validity and reliability of the instrument, we refer the reader to the original article.¹⁵

Upon approval by the Coordination of Teaching and Continuing Education of the institutions in which the Family Medicine Specialization Course is taught, the teachers who were evaluated were asked for informed consent. The protocol was approved by the research and ethics committee of the Faculty of Medicine with No. 050-2012.

Data collection was performed at the end of the school year using the Surveymonkey program via an electronic link in an email message to residents, thus ensuring the accuracy and confidentiality of their responses. The compiler was configured to associate the email address and Internet Protocol (IP) computer for each student to one questionnaire response, so multiple responses were avoided. Access to the electronic compiler and responses is private and can only be handled by one person. A one-week deadline was given to respond to the questionnaire. Therefore, although questionnaires applied written on paper and in person have evidence of validity and reliability, it was considered necessary to evaluate it again this time as applied online.

The literature review focused on a metasearch using EBSCO Discovery Service (EDS) through the Dirección General de Bibliotecas of the UNAM, using the keywords: “desempeño docente”, “evaluación”, “competencias docentes”, “medicina”, “teacher performance assessment”, “teaching skills” and “medicine”.

SPSS v. 21 was used. Descriptive statistics were applied, including measures of central tendency and dispersion, and Student’s t-test for related samples for independent samples, and ANOVA and factor analysis.

Results

78 teachers from 81 participating sites were evaluated; information from three academic venues was removed due to not having a tenured professor at the time of the study. A record of 734 questionnaires was obtained with responses from 1149 residents registered at four health institutions: IMSS, ISSSTE, ISSEMyM, and SSA. The residents’ response rate was 63.8%.  

As shown in Figure 1, 73.1% of teachers are evaluated at a level between acceptable and excellent in the residents’ opinion, and 10.3% are perceived as having insufficient performance.

Table I compares the dimensions of the instrument. The highest average is in dimension 2, and the lowest is in dimension 3, the minimum score given by a student is 0 in four dimensions and 1 in the first, the highest score given by a student is 5 in all dimensions.

In analyzing the means and standard deviations of each dimension of the instrument and the general average, the highest score is in methodology, which is dimension 2 of the instrument, with 4.34 ± 0.92; in contrast, the evaluation dimension had the lowest score, with an average of 4.16 ± 1.04. Both in the five dimensions and the overall average, the average is within the 95% confidence interval with a typical error of 0.03 and 0.04.

Figure 2 analyzes the students’ evaluation by age, observing that with increasing age, the residents tend to think better of their teachers.

When comparing the residents’ age by rank in the overall average, significant differences tend to appear F = 9.69; sig. (Alpha = 0.05); reviewing the various intervals on the results of the post hoc analysis developed using Tukey’s HSD test, significant differences are observed in the averages at a level of 0.05 among students 20-29, and the three subsequent ranges 30-39, 40-49, and 50-59, tending to be greater as age increases.

When comparing the overall average by teacher gender, significant differences are found between the means, the scores of male teachers tending to be higher by 0.14 with a difference of .1387 that becomes significant verified by Student’s t-test with a 95% interval (Table II).

When comparing the overall average of the assessments considering the gender of residents, significant differences are found among the means, scores given by male residents tending to be higher by 0.31 (Table III).

As seen in Table IV, the results of the exploratory factor analysis of the instrument applied online, with varimax rotation, yielded optimal values adjusted ​​to the model, with an estimated variation of 82.2% and Chronbach’s alpha of 0.97.

Discussion

The main objective of the study was to identify the perception of residents about their teachers’ performance. It is assumed that this perception is determined by the academic quality of teaching, although intervening variables such as age and gender were considered in this study.

Teacher evaluation is an essential element in any organization carrying out a program of institutional evaluation for continuous improvement with open communication and a climate of trust that will impact favorably on the quality of teaching practice. This assessment should be objective, ethical, valid, reliable, and useful.

Regarding the questionnaire, statistical analysis identified a high level of reliability, the variance accounted for 82.2% by the factor analysis model was considered adequate, so the measurement obtained by the instrument is consistent with the construct and dimensions. While, as observed in the present study, the instrument tends towards uni-dimensionality, identifying the different dimensions presented a priori in its prior application, and maintaining its conceptual structure, implies an opportunity for improving teaching by identifying which specific factors need training.^16,17

It should be noted that the online instrument obtained better psychometric indicators compared with the paper questionnaire.¹⁵ This can be explained statistically by there being a large number of cases subject to statistical analysis (NV = V734), and 78 teachers. The doctor-resident also had the opportunity to complete the questionnaire personally and according to their schedule to evaluate the performance without the presence of another person as an observer, which could have a negative or intimidating influence for residents, as happens with the paper application. Online application also has the advantage of being able to evaluate a large number of teachers who are geographically dispersed, as in our study, consequently saving time and financial resources.

The teaching performance of 73.1% of teachers was considered acceptable or excellent. Based on the aim of this study, the remaining 26.9% deserve attention from both the educational institution and the administration to establish programs to update and improve teaching. The average of the scores obtained on the Likert scale shows a general homogeneity that tends to manifest in the good level of teacher performance. One notices that deviations from the average are small, and that no one dimension is identified much more than the others.

In the analysis of the results obtained in the five dimensions of the instrument, we see the possibility of strengthening training in evaluation, considering that this dimension got the lowest score. This result could be explained by considering that in general teachers do not have sufficient tools to fully evaluate residents’ learning in multiple dimensions.

These results are similar to those obtained in other studies, which found a need for further development of conceptual and methodological mastery of the various assessment techniques that can be used by the teacher, in particular the importance of assessing competence in various clinical scenarios.^15,18 In contrast, for undergraduates, when evaluating teaching, the basic level teachers have average knowledge of assessment, while preclinical teachers have good knowledge of planning, implementation, and evaluation.¹⁹

Vasquez and Gabalán^20,21 evaluated teacher performance in graduate school, through various information sources such as students, teachers, and immediate supervisors, noting that, in general, they have high competence in the discipline but low pedagogical competence, an area of ​​opportunity to improve and optimize teaching and learning processes in graduate school. This result is similar to ours, although we have the limitation of only using one source of information for assessing teacher performance, which opens a line of research in our context that will have to use other sources of information, as recommended by Smith,²² who developed the "four quadrant" approach, which are self-reflection, peer review, learning, and student experience; this approach allows data collection on a complex phenomenon from a variety of angles or sources to provide a more accurate picture of teacher performance.

Villavicencio²³ found that tenured teachers get a better assessment of their teaching performance compared to teachers located in other academic categories, also noting that academic production and research increases between different evaluation periods; in this case, due to the characteristics of the program itself, only tenured teachers were evaluated. A line derived from this study could be the addition of some other teachers on the faculty to generate other types of comparisons.

It is worth noting that in this study, the evaluation of teachers by gender found male teachers with better scores. In this regard, it is thought that the gender of the students can also have some influence on the evaluation of teachers. In a study by Martinez et al., with undergraduate students, unlike this work, students gave higher scores to female teachers.²⁴ It is likely that differences between men and women teachers are real, and responsive to different forms of behavior in teaching situations; however, not being conclusive studies, we believe that this aspect is worthy of further exploration to gauge its true value.

It should be noted that younger students tend to assess their teachers’ performance with lower scores. It could be inferred that the expectations of younger residents are not met by teachers, but older residents seem gauge the performance of their teachers better. The latter could be driven by further development of academic independence and clinical problem-solving as they move through their course of specialization.

This work evaluated five educational dimensions (teacher-student relationship and motivation, methodology, evaluation, problem-solving, and knowledge of the subject), all part of adequate teaching practice under PUEM. However, it should be noted that the academic program has not identified professional competencies to be teachers of Family Medicine.¹⁴ A study in Canada reported the skills of medical school teachers, namely program evaluation, communication skills, curriculum development, educational theory, educational leadership, dissemination, and clinical teaching. In the same paper, the authors conclude that good clinical educators must have the following characteristics: being active in clinical practice, applying theory to educational practice, promoting educational research, and serving as a consultant to other health professionals on educational issues and problems.²⁵ It would be desirable for family medicine teachers to possess the above features, which would certainly improve their teaching quality.

The results of this work may allow the development of institutional activities to strengthen the areas where teachers scored better, and may implement training strategies aimed at areas that were scored lower, which will help teaching competence in teachers of family medicine.

In order to improve the education of future family medicine specialists, and based on the experience gained in this study, one should consider research projects aimed at identifying the specific competencies needed by teachers of family medicine to better perform their teaching.

Acknowledgments

The authors thank the teachers and residents in the study of family medicine specialization.

References

1. American Educational Research Association, American Psychological Association, National Council on Measurment in Education. The Standards for Educational and Psychological Testing. Washington, DC. 2014.
2. Valdés VH. “En un mundo de cambios rápidos, sólo el fomento de la innovación en las escuelas permitirá al sistema educacional mantenerse al día con los otros sectores.” En: OEI, editor. Encuentro Iberoamericano sobre Evaluación del Desempeño Docente [Internet]. México: OEI Encuentro Iberoamericano sobre Evaluación del Desempeño Docente.; 2000. [Cited 2015 Jan 7]. Available from: http: //www.oei.es/de/rifad01.htm
3. Rodríguez CR, Martínez GA, Ponce RER, Contreras E, Colina RC, Cerritos A. Una Nueva Estrategia para Evaluar la Calidad del Desempeño Docente en las Instituciones de Educación Superior. Rev la Educ Super. México; 2000; 29:115.
4. Valle R, Alaminos I, Contreras E, Salas LE, Tomasini P, Varela M. Student questionnaire to evaluate basic medical science teaching (METEQ-B). Rev Med IMSS. 2004; 42:405-12. [Cited 2015 Jan 7]
5. Berk RA. Survey of 12 Strategies to Measure Teaching Effectiveness. International Journal of Teaching and Learning in Higher Education. 2005; 17(1): 48-62.
6. Mc Andrew M, Eidtson WH, Pierre GC, Gillespie CC. Creating an Objective Structured Teaching Examination to Evaluate a Dental Faculty Development Program. Journal of Dental Education 2012; 76(4): 461-471.
7. Luna SE, Torquemada GAD. Los cuestionarios de evaluación de la docencia por los alumnos: balance y perspectivas de su agenda. Rev Electrónica Investig Educ [Internet]. 2008;(1):10. [Cited 2015 Jan 7]. Available from: http://redie.uabc.mx/index.php/redie/article/view/201
8. Marsh HW, Roche LA. Making students’ evaluations of teaching effectiveness effective: The critical issues of validity, bias, and utility. Am Psychol [Internet]. 1997 [cited 2014 Jun 21]; 52(11). [Cited 2015 Jan 9]. Available from: http://psycnet.apa.orgjournals/amp/52/11/1187
9. McKeachie, W. Valoraciones de los alumnos: la validez del uso American Psychologist, 1997; 52(11): 1218-1225.
10. Molero López-Barajas D, Ruiz CJ. La evaluación de la docencia universitaria. Dimensiones y variables más relevantes [Internet]. Revista de Investigación Educativa. 2005: 57-84. [Cited 2014 Jun 21] Available from: http://revistas.um.es/rie/article/view/98341
11. Fuentes MME, Herrero SJR. Dialnet [Internet]. Revista electrónica interuniversitaria de formación del profesorado. 1999. [Cited 2015 Jan 9). Available from: http://dialnet.unirioja.es/servlet/autor?codigo=2342713
12. CINDE. La evaluación del desempeño de los docentes: Teoría e investigación. 2007. Centro interuniversitario de desarrollo, Chile. [Cited 2015 Jan 10]. Available from: http://www.cinda.cl/download/libros/EVALUACI%C3%93N%20DEL% 20DESEMPE%C3%91O%20DOCEN.pdf
13. Pérez RBA, Viniegra VL. La formación de profesores de medicina. Rev Invest Clin [Internet]. 2003;55 (3):281–288. {Cited 2015 Jan 10]. Available from: http://www.imbiomed.com.mx/1/1/articulos.php? method=showDetail&id_articulo=15729&id_seccion =6&id_ejemplar=1623&id_revista=2
14. Facultad de Medicina. División de Estudiosvde Posgrado. Plan Único de Especialidades Médicas. México; 2009.
15. Martínez-González, A; Lifshitz-Guinzberg, A;Ponce-Rosas, R; Aguilar, V. Evaluación del desempeño docente en cursos de especialización médica. Validación de un cuestionario. Rev. Med. Inst Mex Seguro Social 2008; 46 (4):375-382.
16. Sáenz ML, Cárdenas ML, Rojas E. Efectos de la capacitación pedagógica en la práctica docente universitaria en salud. Rev. Salud pública 2010; 12 (3): 425-433, 2010.
17. Triviño X, Sirhan M, Moore P, Montero L. Experiencias en la implementación de un programa de formación en docencia para profesores de medicina en una universidad de Chile. Rev Peru Med Exp Salud Pública. 2014; 31(3):417-423.
18. Correa JE. La importancia de la evaluación por competencias en contextos clínicos dentro de la docencia universitaria en salud. Rev. Cienc. Salud 2012; 10 (1):73-82.
19. Alterio GH, Pérez HA. Evaluación de la función docente según el desempeño de los profesores y la opinión estudiantil. Educación Médica Superior. 2009; 23(3): 1-14.
20. Vásquez FE, Gabalán J. La evaluación docente en posgrado: variables y factores influyentes. Educ Educ. 2012;15 (3):445-460.
21. Vásquez FE. Gabalán J. Percepciones estudiantiles y su influencia en la evaluación del profesorado. Un caso en la Universidad Autónoma de Occidente, Cali-Colombia. RELIEVE, 2006;12 (2):219-245. [Cited 2015 Jan 11]. Available from: http://www.uv.es/RELIEVE/v12n2/RELIEVEv12n2_3.htm
22. Smith, C. La construcción de la eficacia en la enseñanza a través de la evaluación y la respuesta específica: Conexión de evaluación para la mejora docente en educación superior Valoración y Evaluación de la Educación Superior, 2008; 33 (5):517-533.
23. Villavicencio JE. Evaluación del desempeño docente: la experiencia del Programa Académico de Odontología de la Universidad del Valle, 2005-2007. Colombia Médica. 2008;39 Supl 1.57-63.
24. Martínez-González A, Martínez-Stack J, Buquet-Corneto A, Díaz-Bravo P, Sánchez-Mendiola M. Satisfacción de los estudiantes de medicina con el desempeño de sus docentes, género y situaciones de enseñanza. Rev Investig en Educ Médica [Internet]. Secretaría de Educación Médica. Facultad de Medicina, Universidad Nacional Autónoma de México. 2012; 10:64–74. [Cited 2014 May 4]. Available from: http://riem.facmed.unam.mx/node/195.
25. Sherbino J, Frank JR, Snell L. Defining the key roles and competencies of the clinician-educator of the 21st century: a national mixed-methods study. Acad Med [Internet]. 2014; 89(5):783-9. [Cited 2015 Jan 12]. Available from: http://www.ncbi.nlm.nih.gov/pubmed/24667507.

Enlaces refback

• No hay ningún enlace refback.