Using team-based learning to teach evidence-based medicine – University of Sydney
Prior to introducing its new research framework in 2020, the University of Sydney Medical School taught evidence-based medicine as a one-week intensive program in Year 1. Its new research framework uses clinical integration of research training, and a student–centred, team-based approach, to maximise student competency.
|Title of the Program
Rapid Evidence for Practice modules, Doctor of Medicine (MD)
Deonna Ackermann (Clinical Epidemiology, School of Public Health, University of Sydney)
Sharon Reid (Clinical Epidemiology, School of Public Health, University of Sydney)
Annette Burgess (Education Office, Sydney Medical School, University of Sydney)
Fiona Blyth (Clinical Epidemiology, School of Public Health, University of Sydney)
|Date project commenced
Why was the program undertaken?
Prior to introducing its new research framework in 2020, the University of Sydney Medical School taught evidence-based medicine (EBM) as a one-week intensive program scheduled late in Year 1. Content was delivered through mandatory face-to-face lectures and small group practical workshops. However, feedback suggested that students could not engage with the large amount of content delivered in traditional lectures and, while the critical appraisal workshops were well received, students had not retained content from the lectures and were not ready to apply it. Many students reported that they did not see the practical and clinical relevance of EBM.
EBM and research methods are core components in medical education curricula globally. Best methods for effective teaching of EBM have not been definitively determined, however two key strategies have emerged for optimising curriculum delivery. First, clinical integration of EBM teaching is likely to bring about changes in student skills, attitudes and behaviour . Second, blended interactive approaches, when compared with traditional didactic methods, have been shown to be more effective in improving student attitudes towards, and increasing self-reported use of, EBM in clinical practice . Additionally, the blended learning approach has associated efficiencies, such as freeing up in-class time for student participation in active learning, and cost-effectiveness of delivery, which leads to improved value for institutions with repeated iterations .
Team-based learning (TBL) has gained popularity in medical education , in line with the expansion of student-centred learning strategies, including flipped learning approaches and peer-assisted learning. TBL uses active, collaborative learning strategies to provide an authentic experience of approaching a clinical problem. TBL classes have been shown to provide an enhanced learning environment where students can build on basic knowledge and apply what has been learnt [5, 6]. The classic TBL framework is illustrated in Figure 1. 
Figure 1: TBL framework
|Phase 1: Preparation|
|Students read study material or view pre-recorded lectures independently.|
|Phase 2: Readiness Assurance|
|Learners complete an individual readiness assurance test (IRAT) to assess their understanding of facts and concepts learnt in Phase I.|
|The same test is administered to teams of 5 to 7 learners. The team forms a consensus about each answer in this group readiness assurance test (TRAT).|
|Phase 3: Application|
|Students work in teams to apply Phase 1 and 2 knowledge to real-world complex problems.|
In 2020, The University of Sydney Medical School launched a new curriculum and course structure for its four-year graduate MD program, with Research, Evidence and Informatics established as a vertical theme. The aims of the revised EBM curriculum were to ensure student competency in EBM learning outcomes, emphasise the clinical relevance of EBM, and improve student engagement and application of EBM teaching. The new program shifted EBM teaching from didactic face-to-face lectures to interactive, student-centred and clinically integrated modules.
We used a modified TBL framework to develop Rapid Evidence for Practice (REP) modules that would provide students with EBM knowledge and skills, and prepare them to apply evidence within short timeframes – simulating real world clinical contexts.
How was the program implemented?
In the revised curriculum, EBM teaching is integrated into the Back to Basics (B2B) block, a nine-week term in Year 2, which introduces students to a range of clinical specialties, from paediatrics to aged care, following the human lifespan. This formal coursework program also provides foundational knowledge of research methods prior to students undertaking projects in a dedicated 14-week MD Research Project Block in Year 3.
EBM content is informed by the Australian Medical Council (AMC) guidelines and a set of core competencies developed for health professionals . The course design allows for a rapid change to online only format, if necessary, due to the uncertainty created by the Covid-19 pandemic regarding face-to-face teaching options. A logic model is used as a guide in developing, implementing and evaluating REP modules (Table 1).
Table 1: REP modules Logic Model
Clinical epidemiology staff (3) time
|REP modules comprising:
|Readiness quiz submission
Canvas data analytics
Decision map submission
|Student acceptance of REP modules.
Feasibility of REP modules.
(Staff review, participation data)
Increased EBM knowledge and skills:
(Decision Map meet expected standards
Students’ performance in REI questions in KAT exam)
|Learning objectives met
Students better prepared for MD project
Students are prepared for practice
Four REP modules were developed, with each module focussing on a specific EBM study design in the following sequence: Randomised Controlled Trials, Systematic Reviews, Observational studies, Diagnostic Test Evaluations. An REP module development guide was created for each module (see Appendix, Table 2), detailing learning objectives, content suggestions, activities and assessment tasks, and ensuring consistency in structure.
The REP module structure (Figure 2) comprises online informational content, an individual readiness quiz, a forum featuring a guided critical appraisal, and a student team-based rapid appraisal and application of a published article to a clinical scenario.
Based on our previous experience of students not recognising the clinical importance of EBM and requiring considerable additional support to complete application tasks, we expanded the readiness assurance phase of REP modules to include an additional live forum. The forum features a panel session with real world clinician researcher experiences, a guided critical appraisal task led by our teaching team, and an opportunity for questions and discussion providing students with targeted appropriate knowledge and skills to apply in the subsequent REP activity.
Figure 2: REP module structure
|Preparatory phase – 2 hours, online|
1. Series of short videos provide information on basic concepts.
|2. Students read the journal article for the REP activity.|
|Readiness assurance phase – 3 hours, online and/or face to face.|
3. Individual Readiness quiz (RQ).
4. Forum – Panel session with clinician researchers who discuss their research involvement, career path and how they use evidence in patient management.
5. Forum – Critical appraisal walkthrough. Interactive, practical session where students are guided through a rapid appraisal of a journal article.
6. Forum – Q&A. Provide feedback and clarification.
|Application phase – 2 hours online, and/or face to face|
7. REP activity – Working in teams of 4-5, students use critical appraisal skills to solve a clinical problem.
8. REP activity – Facilitator feedback on student responses. Summary of session.
9. REP activity – Assessment: submission of a team Decision Map.
10. Feedback – Individual and year group feedback on Decision Map.
In REP activities, students are given a clinical scenario and a relevant journal article, and they work through a series of questions with facilitator clarification of challenging questions. A summary of the session is then provided to the cohort. After the REP activity, students finalise and submit as a team a Decision Map (see Appendix, Figure 3), which illustrates how evidence, clinical knowledge and patient values intersect. General feedback on the Decision Map is provided to the cohort and direct feedback is provided to teams if a rubric element does not meet the expected standard.
What is the program achieving?
In May 2021, Year 2 medical students completed a REP module on randomised controlled trials (RCTs). A mixed methods approach to evaluation included:
- Curriculum document analysis to assess content coverage of core EBM competencies.
- Student acceptance assessed using a questionnaire distributed immediately following the REP activity.
- Review of student participation and task completion using learning management system analytics (see Appendix).
Online videos, forum critical appraisal PowerPoint presentations and REP activity tasks were reviewed to determine the extent of their alignment with learning objectives specific to the RCT module and EBM competencies. The document review indicated that learning objectives specific to the RCT module were covered adequately. EBM competencies – including asking a focussed clinical question, applying the PICO framework to research questions, critical appraisal of a journal article and the application of research results to a clinical problem – were met. While acquiring appropriate evidence could be included within the REP module framework, we did not do so, due to time allocation constraints within the Back to Basics Block. Students have a separate lecture and access to library modules on literature searching.
Student experience of the REP module.
In total, 55/258 (21%) of participants completed the post-module questionnaire. Student responses to closed items regarding their experience of REP modules are presented in Figure 3 (see Appendix). The overall response was positive, with 81% of students agreeing or strongly agreeing that they were satisfied with this method of teaching. Responses to items 1–2 indicate that the preparation stage worked well, with 91% agreeing that short online videos were their preferred method of acquiring basic knowledge and 83% completing the pre-readings. Responses to items related to the readiness assurance phase were similarly positive, with 79% agreeing that the RQ improved understanding of concepts and 83% agreeing that the critical appraisal walk-through prepared them for the REP activity. Notably, 91% of students agreed that REP helped them develop a way to use evidence to approach clinical questions.
Students’ responses to open-ended questions are illustrated in Table 3 (what students valued) and Table 4 (what could be improved) in the Appendix. Students valued the overall structure of REP modules due to the staged transition from knowledge acquisition to practice to application: “The stepwise approach to learning. The process has been very methodical and logical”. In addition, each component was valued for its role in the process. Short online videos were perceived as an effective method of delivering basic content: “The online videos are really good, easy to listen to and understand and not too long. They give a really good foundation of concepts that get reiterated in the activities”. The readiness assurance phase was valued as a practice task and REP activities were valued as real-life simulations: “It is good to have the chance to appraise an article and put the theory into practice in a manner very similar to how it would be in real life.” In addition, REP modules were perceived by students as being useful for their future career: “Very useful for me as I engage in the latter years of my training.”
Suggested improvements included avoidance of repetition of basic content in the readiness assurance phase and changes to the clinician researcher presentations, which some students did not perceive as relevant: “The lectures with research examples do not benefit my learning. I feel like the information they give only apply to that one research project and nothing else”. Students also suggested expanded Q&A sessions, more practice activities, shorter duration of classes (two hours) and avoidance of replication of administrative tasks.
All student groups submitted their Decision Maps and only one student from the cohort of 258 did not complete the RQ.
We evaluated the REP module on RCTs embedded in Year 2 based on content, student participation and acceptance. Students found the TBL structure effective for their learning, enabling application of new knowledge to clinically relevant problems that may be encountered in their future careers. Student participation in the RCT REP module activities was high, with 100% completion of the outcome assessments.
Some challenges we faced related to the introduction of a new teaching intervention, including the need for clarity regarding new terminology and scheduling of tasks. A welcome video explaining REP modules was created, all activities were included in students’ calendars and the assessment page detailed a timetable for the release of learning materials and due dates. There was some duplication of administrative tasks for students, which will be improved. Staff found it challenging to identify journal articles of an appropriate difficulty level for medical students that allowed them to focus on appraisal and application of EBM concepts and skills. Students have different levels of prior experience in critical appraisal, suggesting the need for additional targeted training in future iterations.
The REP framework has successfully shifted our EBM and research methods teaching from didactic face-to-face lectures to interactive and clinically integrated modules. Students build on basic knowledge with practice activities and gain real-world experience in the rapid appraisal and application of EBM to clinical problems. The REP activity is an authentic task which promotes student collaboration, reflection and communication, with support from expert input. It provides students with the experience of small group learning in a large group setting, reducing the number of teaching sessions and facilitators required. In addition, REP modules can be run completely online, which provided a safeguard during the Covid-19 pandemic.
 Khan KS, Coomarasamy A. A hierarchy of effective teaching and learning to acquire competence in evidenced-based medicine. BMC Medical Education. 2006;6(1):59.
 Ilic D, Nordin RB, Glasziou P,et al. A randomised controlled trial of a blended learning education intervention for teaching evidence-based medicine. BMC Med Educ. 2015;15:39.
 Maloney S, Nicklen P, Rivers G, et al. A Cost-Effectiveness Analysis of Blended Versus Face-to-Face Delivery of EBMto Medical Students. J Med Internet Res. 2015;17(7):e182.
 Burgess AW, McGregor DM, Mellis CM. Applying established guidelines to team-based learning programs in medical schools: a systematic review. Academic medicine : journal of the Association of American Medical Colleges. 2014;89(4):678-88.
 Burgess A, Bleasel J, Haq I, et al. Team-based learning (TBL) in the medical curriculum: better than PBL? BMC Med Educ. 2017;17(1):243.
 Burgess A, Haq I, Bleasel J, et al. Team-based learning (TBL): a community of practice. BMC Med Educ. 2019;19(1):369.
 Albarqouni L, Hoffmann T, Straus S, et al.Core Competencies in Evidence-Based Practice for Health Professionals: Consensus Statement Based on a Systematic Review and Delphi Survey. JAMA netw. 2018;1(2):e180281.
Table 2: RCT Module development guide
|INTERVENTION STUDIES (B2B WEEKLY THEME – ADULT ACUTE)
At the end of this REP module on RCTs students should be able to:
|Online Content||Forum||REP activity|
|Allocated time||2 hours of student time||3 hours||2 hours|
|Delivery||Online (Canvas)||Face to face or online (Zoom)||Face to face sessions but need to be able to pivot online.|
||Critically appraise an RCT for risk of bias. Trial of Pregabalin for Acute and Chronic Sciatica
Apply the evidence to a clinical scenario using a structured approach based on the 5As
Assess applicability and individualise results
Communicate research methods and results
|Blooms Taxonomy||remember, understand||remember, understand, apply, analyse||remember, understand, apply, analyse, evaluate, create|
|Assessment||Readiness Quiz||Decision Map|
Figure 3: Decision Map outline
Figure 4: Student experience of the REP module
Table 3: Valued aspects of REP modules
|Content||The content was perceived as relevant|
|The level of difficulty was appropriate||Information was relevant and aimed at our level of understanding|
|The content was perceived as being useful for future career||I often found that I would be quite slow getting through research papers and this rapid-fire approach is very useful for me as I engage in the latter years of my training.|
|Can see myself using this strategy as a clinician in the future|
|A framework for critical appraisal was valued||Providing a structured framework to assess primary research articles|
|Effective overall REP module structure||Students valued the structure of REP modules as knowledge was built, consolidated, and then applied.||The stepwise approach to learning. The process has been very methodical and logical|
|Preparation||Short online videos were perceived as an effective method of delivering basic content||The online videos are really good, easy to listen to and understand and not too long. they give a really good foundation of concepts that get reiterated in the activities.|
|Concise||Videos were succinct and explained information well.|
|Quality of video||The online videos were very concise and well made.|
|Clarity of explanations|
|Readiness assurance||The readiness assurance phase was valued.|
|The RQ motivated students to prepare.||The quiz challenged me to learn|
|Critical appraisal walk throughs||Walk-through followed by practical session was good|
|REP activities||REP activities were valued as an opportunity to apply knowledge.||It is good to apply our knowledge to a case scenario soon after learning the concepts.|
|REP activities were valued as real life simulations||It is good to have the chance to appraise an article and put the theory into practice in a manner very similar to how it would be in real life.|
|Helpful practice using a real RCT from a high-impact journal|
|Clinical Integration||Students value clinical integration in EBM teaching.||Realistic application of how we might use research in clinical practice|
|Feedback||Students value feedback via the Zoom Chat||It was great that the questions from Zoom chat etc were addressed quickly.|
|Assessment||Students reported that assessment tasks were appropriate.||I also enjoyed the REP decision map, as I believe it’s extremely helpful in making us critically analyse research.|
Table 4: Improvements suggested by students for REP modules.
|Clinician researcher presentations||Clinician researcher presentations perceived as not relevant||The lectures with research examples do not benefit my learning. I feel like the information they give only apply to that one research project and nothing else.|
|Clinician researcher presentations could be more structured||The panel discussion of clinicians/researchers sharing their experiences could benefit from structure (e.g. Common questions, downfalls or weaknesses seen in the field). Having slides or other presentation material could help.|
|The first lecture today (9-10) with the panel of lecturers seemed a bit directionless. I wasn’t really sure what I was supposed to be getting out of it and found it difficult to stay tuned in.|
|Repetition of basic content||Avoid repetition of basic content in readiness assurance phase and post-rep reflection||Each concept reiterated at least 3 times (prework, forums, Kuracloud activity, recap forum), which seemed excessive.|
|Readiness Assurance phase||More practice tasks||I believe that there could have been more practice activities, like there were in the first two weeks, rather than just the readiness quiz.|
|I would also appreciate mini quizzes etc during/after the lecture to check my understanding|
|Feedback||Expand Q&A (rather than repeat basic content)||An opportunity for feedback and questions would be much more beneficial in terms of learning and efficiency (than repeating basic content).|
|Effective use of time||Avoid duplication of tasks (Kuracloud activity and Decision map)||This process was a bit repetitive from Kuracloud to Canvas submission, etc|
PDF of case studies available here: Research in the Medical Curriculum, Volume 1 – A window on innovation and good practice 2022