Driving student engagement in research through a revitalised online journal club

Driving student engagement in research through a revitalised online journal club

What was the main aim of this project?

Journal clubs are a mainstay of medical education: they are a practical and collaborative way to improve content knowledge and promote evidence-based medicine; [1]  they play an important role in developing students’ ability to critically appraise the literature and research methodologie[2]; and they help students recognise the importance of keeping up to date with the medical literature to help inform their future clinical practice [3]. Journal clubs also facilitate group discussion with peers, which is a valuable addition to the scholarship of teaching and learning [4].

A recent study of the prevalence and implementation of journal clubs within Australian medical schools found that over 80% of schools held journal clubs as part of the curriculum, though the majority were held during later-stage clinical years and tended to be clinician-led[1].

The COVID-19 pandemic provided a stimulus to restructure the journal club format. The new virtual journal club format delivered through this project was designed to be student-centred, with students encouraged to lead their session and drive interaction with their audience. Also, it capitalised on the idea that introducing and engaging students in journal clubs earlier in their medical education may help to increase and maintain an ongoing interest in and understanding of research and its role in medicine.

Integration with the curriculum

Research and critical analysis (RCA) content is integrated into the curriculum in each of the four years of the graduate MD program at University of Wollongong [5,6]. Development of this award-winning RCA program [7] was based on the frameworks of Miller’s pyramid [8] and Boyer’s domains of academic scholarship [9].

RCA principles taught in the first 18 months of the MD program (Phase 1) are delivered in large group sessions and practiced through small group journal club presentations (see Figure 1, below). The journal club presentations align with the lecture content to ensure that students can build on this knowledge through interpretation, application and demonstration of skills. Group journal club presentations provide the students with one of their first opportunities to apply their skills and reflect on their skills development by receiving structured feedback on the task.

Figure 1. Research and critical analysis (RCA) assessment tasks over the four Phases, as aligned with the framework of Miller’s triangle. POEM: patient-orientated evidence that matters. Figure adapted from [7] Mansfield et al., (2022).

The key learning outcomes of the journal club address five of the nine RCA Learning Outcomes, as defined in Table 1 below.

Table 1. Learning outcomes for the research and critical analysis (RCA) content across the four-years of the Graduate Medicine course at UOW and those that align with outcomes of the journal club task.

What changes were implemented?

Prior to the COVID-19 pandemic a traditional in-person journal club format was employed, which involved a non-assessable task and voluntary attendance. As part of these sessions, Phase 1 medical student groups were encouraged to critically appraise an article selected by an academic staff member, which focussed on the RCA principles and the body system block being taught at the time.

Student groups presented a critique of their assigned articles to their peers during a casual lunchtime journal club session, as is traditional in many medical and educational institutions. Academics involved in the journal club often commented that while the presenting students engaged with the task, there were very few other students attending to watch and engage with the presentations, so that little to no discussion was generated following the presentation. Additionally, feedback from students suggested it was considered a low-priority activity.

The new virtual journal club format developed during the COVID-19 pandemic was devised to be an interactive space where discussion was encouraged. A safe and supportive environment was established through appropriate questioning and positive encouragement from participating academics. Students were also guided through an information session about the task and how to give respectful and constructive feedback when engaging in discussion.

The journal club presentations became an assessable component, with clear learning objectives. It was anticipated that this would encourage greater peer interaction, collaboration and student-led learning.

A marking rubric was designed to assess each group presentation against a range of RCA learning objectives. Based on the rubric, grades of ‘excellent’, ‘satisfactory’ or ‘unsatisfactory’ were awarded by attending academics for the following activities of the groups: literature searching skills and selection of an appropriate article; appraisal of the study design and research methods; appraisal of the results and discussion; appraisal of the strengths and weaknesses of the study; articulation of the implications for clinical relevance. This change was based on the premise that the use of a structured critical appraisal is integral to running a successful journal club.[10] Presentation skills were also assessed as part of the rubric and a student peer assessment score incorporated to encourage all students to contribute equitably to the group work, knowing that their contribution would be rated by other group members.

How was it delivered?

Students were assigned a tutorial group for their regular activities in Phase 1 and these same groups were utilised for their journal club task. Each group was given a topic incorporating a methodology or analysis technique relevant to RCA principles and aligned with the RCA content being delivered at large group teaching sessions during the period. Students were advised to search for a recent and appropriate article, clinically relevant to the current medical case being examined and addressing the appropriate RCA principles. This facilitated the development of their literature searching skills within a confined topic, and increased student ownership and interest in the task.

The assessment criteria aimed to build an understanding of how to apply medical literature to evidence-based medicine and assess its clinical relevance. Two Phase 1 RCA academic leads attended each of the virtual journal club presentations to maintain consistency in marking and student feedback. A guest academic also joined each journal club presentation to provide additional perspectives and act as a third assessor.

How has the impact of this project been evaluated?

Specific goals for the reinvigorated and restructured journal club were to: increase students’ research and critical analysis skills; improve attendance and peer interaction; and increase engagement with learning about evidence-based medicine. To evaluate the effectiveness of the journal club, student assessment marks, student reflections and staff reflections were considered.

Overall, student presentations were assessed to be of a high standard, demonstrating a deep understanding and strong engagement in the assigned topics. Further, discussion was greatly improved, with students asking numerous questions and the full hour allocation (30-minute presentation and 30-minute discussion) was often used due to the vigour of the discussion  – in contrast to previous iterations which routinely ended following the 30-minute presentation.

Student reflections

Student reflections, via an anonymous online survey and spontaneous email feedback, indicated the achievement of the goals for the virtual journal club.

Students were asked whether attendance at journal club increased or improved their confidence over a range of research and critical analysis skills, such as: searching databases and finding relevant literature, understanding methodology, biostatistics, and interpretation of results. They were also asked to reflect on what they found useful about journal club and possible session improvement. Over one third of the cohort responded to the survey (37%).

Attendance was increased with the new online approach. While attendance at the nine journal club sessions was not compulsory, 44% of respondents self-reported attendance at 7-9 sessions and 78% attended more than half of the sessions (5-9). Online attendance counts across the sessions were on average 58% (±18%; range 28% – 85%), with the lower attendances being near exams. Most feedback on improvements to journal club related to attendance and, specifically, the timing of presentations to reduce impact on schedules near exam periods, or the desire for in-person or hybrid delivery of journal club.

Increasing peer interaction and providing a forum for discussion about research was another goal achieved by the reinvigorated journal club. Students overwhelmingly reported that they enjoyed the interaction with peers during journal club (70% agreed) and that it was an engaging way to learn about research and critical analysis (93% agreed), see Figure 2 below. Students who attended less than half of the journal club sessions reported a lower level of enjoyment in peer interaction.

Figure 2. Students reflected positively on their engagement with journal club, survey responses shown for the questions

A. “I enjoyed the interaction with peers during journal club”
B. “I found the Journal Club presentations an engaging way to learn about medical research and critical analysis”. Results averaged from n=28 responses (37% of cohort).

Table 2. Student and staff reflections on the updated journal club activity.

For all RCA skills examined, over 50% of respondents agreed that their skills had improved (Figure 3).

Figure 3. Student evaluation of their skills following participation in the journal club sessions (data represents average of n = 28 responses: 37% of the cohort).

The increased confidence in RCA skills was apparent, regardless of prior level of research experience or how many journal club sessions were attended, suggesting that the completion of the task ensured students developed and applied many of the required skills (Quote 2, Table 2). Student-led learning and peer collaboration was a large component of this skill development, and several students reflected on the benefits of peer learning when undertaking the task (Quotes 3, 4 and 5, Table 2). The structured rubric used to assess the journal club appraisals was also highlighted as a useful tool in the student reflections (Quotes 6 and 7, Table 2).

The effectiveness of journal clubs can also be evaluated by measuring changes in reading habits of participants [10,11]. Student reflections demonstrated that in addition to increased confidence in their critical appraisal skills, 60% of students agreed that their motivation to read research articles had increased following journal club participation (Figure 3). 

Staff reflections

Staff discussions occurred regularly during implementation of the new-format journal club program. Staff agreed that students were more engaged with the new group presentation assessment task and that the interactive format was of benefit to their skills and understanding. Because groups were meeting regularly as part of their ongoing tutorials, they often incorporated their journal club preparation into their group meetings and discussed concepts with their tutors and RCA facilitators. This increased engagement was demonstrated in the depth of knowledge displayed by the students, both in the construction of their appraisal and the discussion of questions raised following the presentation, which was of an exceptional standard. This was acknowledged by both staff and students as a positive outcome following journal club (see quotes 8 and 9, Table 2).

Academic staff engagement with journal club also increased, with all academic staff attending at least one session and some staff attending additional sessions to support students they tutored or out of personal interest in the topics. This further fostered a sense of comradery and collegiality with the students. While it was not specified in the marking criteria, each student group spontaneously incorporated entertaining and original components to their talk, with students dressing up in formal or themed attire, having special thematic slides, polls, quizzes and animations in their presentations, and one even producing an entertaining video demonstrating some important concepts from their research topic. It was evident that these efforts to engage the audience were successful and built up over time.

Conclusion

The development of a supportive and interactive environment for an assessable online journal club resulted in students overwhelmingly agreeing that it was an engaging way to learn about research. There was increased student attendance and engagement, both with the new format journal club sessions and the RCA course content generally. Learning outcomes for RCA content were detailed in the marking rubric and the survey of student confidences was mapped to these outcomes. Students rated their confidence in critical appraisal skills as having improved following their participation in journal club across all RCA appraisal and interpretation components including literature searching, methodology, statistical analysis, interpretation of results, discussion, and clinical relevance.

One of the key learning outcomes for the journal club was to facilitate discussion and debate of evidence-based medicine and to encourage an appreciation for research. Students reported that their motivation to incorporate research literature into their regular studies had also increased. Further, both students and academics acknowledged the depth of knowledge displayed within the presentations and the high standard of discussions that were generated.

References

[1] Ilic D, de Voogt A, & Oldroyd J. (2020). The use of journal clubs to teach evidence-based medicine to health professionals: A systematic review and meta-analysis. Journal of Evidence Based Medicine, 13(1), 42-56. https://doi.org/https://doi-org.ezproxy.uow.edu.au/10.1111/jebm.12370

[2] Ianno, D. J., Mirowska-Allen, K., Kunz, S. A., & O’Brien, R. (2020). Journal clubs in Australian medical schools: prevalence, application and educator opinion. Journal of Educational Evalulation for Health Professionals, 17, https://doi.org/10.3352/jeehp.2020.17.9

[3] Shrivastava, S. R., & Shrivastava, P. S. (2021). Promoting the Conduct of Medical Education Journal Clubs in Teaching Medical Institutions. Avicenna Journal of Medicine, 11(3), 156-159. https://doi.org/10.1055/s-0041-1735126

[4] Szucs, K. A., Benson, J. D., & Haneman, B. (2017). Using a Guided Journal Club as a Teaching Strategy to Enhance Learning Skills for Evidence-Based Practice. Occupational Therapy In Health Care, 31(2), 143-149. https://doi.org/10.1080/07380577.2016.1278296

[5] Mullan, J. R., Mansfield, K. J., Weston, K. M., Rich, W., Burns, P., Brown, C., & McLennan, P. L. (2017). ‘Involve Me and I Learn’: Development of an Assessment Program for Research and Critical Analysis. Journal of Medical Education and Curricular Development, 4, 2382120517692539. https://doi.org/10.1177/2382120517692539

[6] Mullan, J. R., Weston, K. M., Rich, W. C., & McLennan, P. L. (2014). Investigating the impact of a research-based integrated curriculum on self-perceived research experiences of medical students in community placements: a pre- and post-test analysis of three student cohorts. BMC Medical Education, 14(1), 161. https://doi.org/10.1186/1472-6920-14-161

[7] Mansfield, K., Metusela, C., & Mullan, J. R. (2022). Building research capacity for rural and regional medical students. University of Wollongong Research and Critical Analysis Program – case study. In: Research in the Medical Curriculum Volume 1: A Window on Innovation and Good Practice. Medical Deans Australia and New Zealand

[8] Miller G. E. (1990) The assessment of clinical skills/competence/performance. Academic Medicine, 65(9), S63-67.

[9] Boyer E. L. (1990) Scholarship Reconsidered. Princeton, NJ: Carnegie Foundation for the Advancement of Teaching

[10] Deenadayalan, Y., Grimmer-Sommers, K., Prior, M., & Kumar, S. (2008). How to run an effective journal club: a systematic review. Journal of Evaluation in Clinical Practice, 14(5), 898-911. https://doi.org/https://doi-org.ezproxy.uow.edu.au/10.1111/j.1365-2753.2008.01050.x

[11] Linzer, M., Brown, J. T., Frazier, L. M., DeLong, E. R., Siegel, W. C. (1988) Impact of a Medical Journal Club on House-Staff Reading Habits, Knowledge, and Critical Appraisal Skills: A Randomized Control Trial.  260(17):2537–2541. https://doi:10.1001/jama.1988.03410170085039