What was the main aim of this program?

The JCU six-year undergraduate MBBS degree is the only medical program based entirely in northern Australia and has demonstrated success in producing graduates who serve rural, regional, and remote communities [1,2]. Inspiring and supporting the next generation of clinician researchers to address the health and health care challenges of these communities is a key part of the mission of the JCU MBBS course.

(more…)

What was the main aim of this project? 

MD students of the Sydney Medical Program (SMP) undertake a 14-week research block to develop an understanding of the principles and requirements of medical research and thereby facilitate their practice of evidence-based medicine. Students on extended placements in Dubbo, Orange and Broken Hill in western and far western New South Wales (NSW) are offered rural research projects with local supervision.

This MD Project block reflects the vision of the SMP: “To develop compassionate, diverse, and innovative lifelong learners, who work in partnership with individuals and communities to improve health through clinical care, education and research”. It also enables students at graduation to meet the Australian Medical Council’s Graduate outcomes (Table 1).

Table 1: Australian Medical Council’s Graduate outcomes (Domain 1 – Science and Scholarship)Source: Standards for Assessment and Accreditation of Primary Medical Programs by the Australian Medical Council 2012. 

For rural students, this research experience fosters meaningful connections to local communities, clinicians and services, and a deeper understanding of health care delivery in a rural context. Advantages also accrue to rural communities and health services through the projects: by undertaking a rural MD Project, students provide a much-needed boost to rural research outputs and support for evidence-based rural health care.

Learning outcomes

By the end of the MD Project block students can conduct a health research project from conception to completion (Table 2).

Table 2: Learning Objectives of the University of Sydney SMP MD Research Block

The MD Project is a compulsory part of the MD 2022 curriculum and an important element of the vertical theme of Research, Evidence and Informatics, which runs throughout the four-year graduate MD program. Research methods are introduced in Year 2 of the program and refreshed in Year 3. The Programmatic assessment strategy used requires students to meet standards in the 14-week MD Project to be eligible to progress to Year 4.

The MD program is based on the “pedagogical framework of active, experiential, project-based learning”[1]  applied here in a rural context.  Each rural research project is supervised by rural academic staff and rural clinical topic experts. Learning is scaffolded to the individual interests of the student. Having topic experts who are practicing clinicians, health service workers or community members helps students to see their project in the context of the community and health service.

How was the project implemented?

A subset of University of Sydney students live and study in rural and remote NSW whilst undertaking their medical degree. Extended rural placement students undertake one or two year-long placements; students in the new Dubbo Medical Stream of the SMP are permanently located in Dubbo for the four years of their degree. The School of Rural Health (SRH) and University Department of Rural Health (UDRH) provide supervision for these students.

While overall leadership of the MD Project block is provided by the SMP, the rural team is supported by a local academic coordinator. In 2022, 25 Year-3 MD students worked across 19 different projects in Dubbo, Orange and Broken Hill, supervised by nine academics and 18 local clinical topic experts. This was possible as a result of the long-term relationships built between SRH/UDRH academics and Aboriginal communities and services, Local Health District clinicians and Health Intelligence Unit staff.

The number of students will increase in 2024 as students in the University of Sydney Dubbo Medical Stream Year 3 cohort join the extended placement students to do their MD Project.

Students in rural sites choose projects from a list presented to them six months prior to the MD Project block. In 2022, 18 (72%) received their first preference and the remainder received their second preference. A broad range of topics were offered (cancer, cardiovascular disease, diabetes, pregnancy, virtual healthcare, public health, gout, falls, emergency medicine, psychiatry, mental health, drug and alcohol, paediatrics), with a range of priority populations (Indigenous, rural, adolescent) (Table 3).

Table 3: Examples of rural and clinically relevant research projects completed in 2022

Several types of research report are acceptable (Table 4 shows report types submitted in 2022).

Table 4: Type of MD Project report submitted for assessment in 2022

Students worked with their rural academic supervisor and clinical topic experts (and often community members) to understand issues in the field of study in a rural context and scope the appropriate literature. Establishing a rationale for their project, they formulated a research question and considered relevant, feasible research methods to answer the question. The ethical issues regarding the proposed project in a rural area were carefully considered as rural populations are more likely to include Aboriginal people and those who can be easily identified. Students completed a literature review to find, appraise and integrate existing knowledge relevant to their research question. They collected and analysed data and presented their findings to an audience which included their team, peers, academics and clinicians. Finally, they prepared a 3,000-word scientific report.  

Achievement of learning objectives was assessed at key milestones as the students progressed through the block, to support completion of their projects. The final reports were marked by University of Sydney academics or affiliates, with no conflict of interests, using a standard rubric. Dissemination of research can follow through the direct incorporation of the findings into local practice, policy briefs, the submission of abstracts to local and national conferences, and the drafting of manuscripts for journal publication.

HREC processes were determined by the nature of the project: projects such as systematic or narrative literature reviews, or those using publicly available data, did not require HREC review; students were added to ongoing HREC-approved projects by a modification in some cases. Alternatively, low/negligible risk research was undertaken, such as interrogating existing databases. All HREC processes were finalised prior to the commencement of student involvement and local supervisory teams were supported by the rural clinical schools’ research governance officer to ensure these were in place.

How has the impact of this program been evaluated?

Student reflection

Students commenced the block with varying levels of prior research experience, ranging from no experience to holding a higher degree by research (Table 5).

Table 5: Prior research experience of MD Project students in 2022

Students were asked to undertake a self-assessment of their research experience using the Research Spider [2] on commencement and completion of the MD Project. This tool asks respondents to score their level of experience in ten aspects of the research process on a scale from 1, no experience, to 5, very experienced, and plots responses on a diagram that resembles a spider’s web. In a post-research experience survey, they were asked the three most important things learnt from doing the MD Project and what advice they would give to incoming students. Responses were received from 24 of the 25 students.

Prior to the MD Project, Research Spider group mean scores ranged from 1.4 (Applying for research funding) to 3.2 (Finding relevant literature). After completion of the MD Project the scores ranged from 1.3 (Applying for research funding) to 3.9 (Finding relevant literature) (n=15). The biggest changes in group mean scores were on the dimensions of Writing and presenting a research report (increase 2.7 to 3.7), Critical review of the literature (2.9 to 3.9), Analysing and interpreting results (2.8 to 3.6) (Figure 1).

Figure 1: University of Sydney School of Rural Health and University Department of Rural Health MD student self-assessment of research experience prior to commencement and on completion of the MD Project block 2022

Schmidt et al[3] published Research Spider scores of a group of experienced rural clinicians, but novice researchers, who undertook a workplace-based research training program known as the Rural Research Capacity Building Program (RRCBP)[4]. Post program Research Spider scores for the MD Project students and RRCBP participants were similar, with a maximum difference of +/- 0.5 for all dimensions except applying for research funding, where RRCBP candidates had more experience. The post-program Research Spider data are limited (n=10 or 40% missing), but nevertheless show that the MD Project block prepares rurally based medical students to a level of research experience like that reported by rurally-based clinician researchers.

Students described research-related skills they had learnt during the MD Project, such as generating hypotheses and presenting an argument in a research paper. Students also described ‘soft’ skills, such as the importance of being consistent and time management, as the most important things learned during their MD Project.

Students were asked to share their advice for future students. These comments also refer to research-related skills and transferrable ‘soft’ skills (Figures 2, 3 & 4).

Figure 2: Student S5 shares advice to future MD Project students

–

Figure 3: Student S6 shares advice to future MD Project students

–

Figure 4: Student S23 shares advice to future MD Project students

Staff reflection

The rural supervisors and rural clinical topic experts were invited to reflect on how the block prepares students to be clinician researchers. All nine supervisors and two of 18 topic experts responded, identifying curriculum, local and general strengths of the MD Project.

The structured nature of the placement with dedicated time for the MD Project, existing projects and focused tasks with clear criteria were strengths that reflected the curriculum’s influence.

Locally, there was a strength in having a team approach which included supervision from clinical topic experts and researchers for projects that were practical, and had local relevance and impact for improving health outcomes. Clinician content experts provided the students with role models of clinicians who collaborate to research and advance their practice.

Generally, the projects supported practical research skill development (collection, cleaning, analysis and presentation of data). The team approach required organisation and collaboration to provide local lectures and workshops that enhanced the centrally provided university lectures with local context. The two topic expert respondents noted the value of being included in the research projects and the support of the team approach with access to researcher skills. “Collaboration is the key in any research and this is one of the most important behaviours [the students] can learn.”

Although we did not ask, almost everyone shared challenges. This was a short intense block with milestone schedules that made completion or revision difficult. The short block also removed important engagement and skill development in project design and ethics processes, and favoured existing projects with pre-approvals. There were also practical challenges with data availability, student time management, student engagement with the projects and the research process.

The weekly meeting of the team of local supervisors was valued as a collegiate way to problem solve and benchmark. Supervisors recognised that they could improve their communication about student expectations to enhance preparedness and dedication of students undertaking their MD Project in a rural location. It was noted that the MD Project was a clear opportunity to help students understand research and its impact and embrace or at least support it in their future.

Conclusion

We present here a model of supervision for medical students undertaking a research block in rural locations. Rural clinical schools have fewer research academics than metropolitan schools and our unique supervisory teams, comprising rural researchers and rural clinical topic experts, have been successful. Students have learned both research skills and other transferable skills. Projects have resulted in practical and academic outputs, and have given students a deeper understanding of health care delivery in a rural context.

Acknowledgments

Artwork by David Schmidt of Battered Old Wreck Comics https://m.webtoons.com/en/challenge/battered-old-wreck-comics/list?title_no=83262&page=1

References

[1] Hart, J., Hakim, J., Kaur, R. et al. Research supervisors’ views of barriers and enablers for research projects undertaken by medical students; a mixed methods evaluation of a post-graduate medical degree research project program. BMC Med Educ 22, 370 (2022). https://doi.org/10.1186/s12909-022-03429-0

[2] Smith, H., et al., The ‘Research Spider’: a simple method of assessing research experience. Primary Health Care Research & Development, 2002. 3(3): p. 139-140.

[3] Schmidt, D., Webster, E., Duncanson, K. (2019). Building research experience: Impact of a novice researcher development program for rural health workers. Australian Journal of Rural Health, 27(5), 392-397.

[4] Webster, E., Thomas, M., Ong, N., Cutler, L. (2011). Rural research capacity building program: capacity building outcomes. Australian Journal of Primary Health, 17(1), 107-113.

What was the main aim of this program?

The program of Rural Health Projects (RHPs) run by University of Queensland (UQ) provides medical students the opportunity to undertake scholarly research projects that encourage critical thinking and logical reasoning – an essential part of students’ learning and development for a medical career. [1,2,3]

More than 270 RHPs are conducted every year through UQ’s Rural and Remote Medicine (RRM) unit, in over 50 rural and remote (MMM3-7) communities. Each RHP is undertaken with ~40-47 students during a 6-week block placement, as part of the Year 3 medical curriculum (MD).

Using a Quality Improvement (QI) framework, students develop an understanding of rural health service delivery, while learning to work with others in improving health outcomes for the community in which they are placed. Students collaboratively investigate a specific aspect of a chosen topic and use scientific methods in data collection, analyses and reporting to meet a community’s ongoing needs. Students are assessed based on their written academic reports, which aim to highlight potential solutions or resources for the local communities, relevant to the chosen topic.

The RHPs harness the opportunity of placement at a rural site to provide the following learning outcomes for students:

• gain an understanding of rural communities, including Aboriginal health issues and cultural safety
• identify relevant population health issues associated with a clinical topic as it affects a rural community
• engage with relevant health professionals, organisations, patients, and carers
• design and evaluate an intervention to address health-related rural issues, specific to a local population
• apply past learning about research skills to a practical task using a quality improvement framework
• reflect upon ethical issues
• gain experience collating a written academic report
• obtain insight into research possibilities in the maintenance of standards and achievement of optimum health outcomes for rural communities.

Learning Theory

The RHP approach is underpinned by a sociocultural theory. [4,5,6] Students work under interactive guidance, supervision and modelling regarding the cognitive and experiential aspects of their work, with intensive immersion in the tasks being carried out, and rely on self-motivation, initiative and problem-solving to achieve outcomes.

Integration with the MD/MBBS curriculum

The RHPs are integrated with the flow of Phase 1 pre-clinical programs and fit in with other RRM assessments and practical experiences. RHP students can apply previously taught research skills during their learning. Students motivated to continue research after completing their RHP are referred to the Medicine School’s Regional Clinical Research units to take advantage of other available research opportunities.

How was the program implemented?

The RHPs are designed to be carried out within a QI framework – a systematic formal approach to the analysis of practice performance and effort to improve performance and patient outcomes. The methodologies and research methods used are compliant with the guidelines of Queensland Health and other organisations related to QI activities.

Students undertake these projects for the services where they are placed, as ‘temporary members of their staff working under delegation and supervision’, and are not engaged in publishable original research. The wide variety of student placement sites, the short timeframe of the placements, and the finding of a suitable topic in collaboration with the community only on arrival, preclude engagement with research and ethics committees. The projects are expected to comply with the ethical and research standards of the Human Research Ethics Committee, learned by students during the pre-clinical years.

During the students’ time on site in a rural location, the RHPs are carried out alongside clinical teaching and training in a rural hospital, general practice or a combination of both. Students also have the option of conducting their project with another health-related service (e.g. community health or a relevant non-government organisation), with an approved supervisor who can support their work.

The data, information, and practical resources the students construct for the project belong to the health service where they are placed. The topic, and need to be addressed, is identified in conjunction with key health providers in the local health service and community, and must be approved through consultation and submission of a plan to the RHP Academic Coordinator, as well as endorsed for suitability by the student’s local clinical preceptor and/or other involved stakeholders. Students can negotiate an equivalent to a half-day per week to carry out practical tasks such as interviews, community visits or data audits associated with the QI component of their project, with other tasks such as literature and document review, data analysis and collation, and assembling their report occurring outside of clinical working hours.

Assessment

Completion of the QI investigation culminates in the presentation or production of resources for the health service’s staff and patients, about which students can obtain feedback. The completed report, with attachments illustrating the work done, is submitted via the Academic Coordinator to the RHP team, who organises examination and feedback to inform a student’s future learning.

The submitted report is assessed against ten key criteria, encompassing:

• ability to present a written academic report and abstract
• understanding of health-related rural issues and the selection and overview of a relevant topic to be investigated
• ability to plan, organise and evaluate data relating to a QI task
• involvement with key stakeholders in the health service and community, and contribution of practical resources or outcomes for them
• ability with critical analysis and problem-solving regards ongoing needs and solutions.

The RHP program is an integral part of the decision-making and operational processes of the RRM clinical unit. The RHP Academic Coordinator, employed part-time as a Senior Lecturer specifically to fulfil this role, is responsible for leading the RHP program.

Four Level B lecturers are employed on a casual basis for the assessment and marking of the RHPs. Recruitment for these positions uses a targeted approach to employ professionals within an interprofessional team, with extensive rural health experience, postgraduate qualifications in a variety of areas, experience with research or health-related projects and an interest in student teaching. This team also has access to university staff and academic title holders from various areas of medicine and nursing, Indigenous Health, and research, for advice and assistance, as well as collaboration with other members of the RRM unit and designated support officers.

How has the impact of this program been evaluated?

The RHPs provide students an opportunity to gain valuable understanding of the process of QI research activities and develop important outcomes or resources relevant to their rural placement communities.

Figure 1. Number of Rural Health Projects conducted within Australia during 2011-2021.

From 2011 to 2021, more than 2,974 RHPs were conducted in rural or remote communities (Figure 1). Most of these RHPs were conducted in Queensland (Figure 2), in small rural towns (MMM 3-5).

Figure 2. Number of Rural Health Projects conducted within Queensland during 2011-2021.

The RHPs not only provided a practical, beneficial approach for medical students to use in their research projects, they also addressed key issues relevant to the local communities through the development of practical resources or important baseline information for future research.

Staff reflections

The impact of the RHPs is further described in the reflections provided by staff and students below.

• ‘I have been involved for well over 11 years and in that time, these projects have developed from an essay-style report around a topic to something much more beneficial and useful to both the GP/ medical staff and the rural community as a whole. The QI component has become more strongly developed and the necessary research skills have been refined… Having to work the full process, from deciding a topic, reviewing what is happening now, then working through evidence and collected data to devise a hypothesis and recommendations is quite unique. Being placed in rural and remote medical surrounds, gives these students a view of how to provide good health care for patients when they do not have the easy access to equipment and specialists that city-based practitioners do.’

• ‘Rural health is an area where diversity can often be overlooked, and the variety of student projects we have seen over the years is encouraging. Another gratifying aspect of the RHPs has been to witness the responsiveness of student work to situations affecting rural communities. Student projects have responded to major events such as flooding or the pandemic, as well as to more localised events impacting on specific communities. One of the greatest strengths of the project format in my view is that it encourages students to develop a response to current community needs and to do work that addresses the challenges of a particular place and time. Both for the practical learning experience this provides students, and the creativity and energy students can bring to the work being done in rural communities, the Rural Health Project is a highly rewarding program. It has been a pleasure to see how it has developed over time and built a body of student work which reflects the variety of rural communities and the health challenges they experience.’
• ‘The program has had to evolve to keep up with the dynamic nature of the health industry. The ways in which students see and access their target communities has evolved and so have project outcomes to involve more innovative solutions, such as using Telehealth. For me the most academically rewarding aspect of the program is that it helps students learn that a one-size-fits-all approach doesn’t work in health care, particularly when you’re in rural and remote healthcare, so they are learning to take the research and best practice evidence and to adapt and use their local health teams to make it fit the community need, not make the community fit the best practice research.’
• ‘A positive aspect of the RHP is finding most students listen and discover the complex nature of healthcare from a patient perspective. The benefits of feedback and reflection as a result of critical analysis are also evident in the projects.’
• ‘Where ‘research skills’ are concerned, there will always be diverse expectations and motivations in student groups: for some, the program has produced a basic working knowledge of how to apply assessment and planning of services and specific projects to their daily clinical practice and an appreciation of different types of investigation; for many more it has motivated an interest in extending or sharing the work they have done or being involved with original research. Much of the Quality Improvement activity has produced pilot data or observations which indicate areas in which more extensive rural health research could be undertaken, and our coding over the entire life of the RHP program, of the topic choices and issues students have chosen to address, certainly has indicated the evolving trends which would benefit from meticulous researching.’

Student Reflections

• (The program was) ‘very hands on and very diligent…. Offered great feedback and ways for students to improve their work in the future.’
• ‘…there was so much of value in this feedback that just wasn’t communicated or able to be interpreted from marked criteria alone. … it has exposed me to a higher level of academic rigour than I’ve been exposed to in this or previous degrees.’
• ‘I’m currently writing up two papers for a research project ongoing through…hospital.’
• ‘…extremely helpful and reassuring… thank you for the time and thought put into it…’
• ‘It was really informative and has given me a lot of pointers and directions to go into…’.

References

[1] Alrefaie Z, Al-Hayani A, Hassanien M, et al. Implementing group research assignment in undergraduate medical curriculum; impact on students’ performance and satisfaction. BMC Med Educ 2020;20(1):229. doi: 10.1186/s12909-020-02137-x [published Online First: 2020/07/22]

[2] Conroy MB, Shaffiey S, Jones S, et al. Scholarly Research Projects Benefit Medical Students’ Research Productivity and Residency Choice: Outcomes From the University of Pittsburgh School of Medicine. Acad Med 2018;93(11):1727-31. doi: 10.1097/ACM.0000000000002328 [published Online First: 2018/06/21]

[3] Zee M, de Boer M, Jaarsma AD. Acquiring evidence-based medicine and research skills in the undergraduate medical curriculum: three different didactical formats compared. Perspect Med Educ 2014;3(5):357-70. doi: 10.1007/s40037-014-0143-y [published Online First: 2014/11/15]

[4] Bleakley A. Broadening conceptions of learning in medical education: the message from teamworking. Med Educ 2006;40(2):150-7. doi: 10.1111/j.1365-2929.2005.02371.x [published Online First: 2006/02/03]

[5] Badyal D, Singh T. Learning Theories: The Basics to Learn in Medical Education. Int J App Basic Med Res 2017;7

[6] International Clinical Educators Blog. Sociocultural Theory. Education Theory Made Practical 2021; 5(10).

What was the main aim of this project/program?

In 2020, Monash University graduated its first cohort of approximately 500 Doctor of Medicine (MD) students across Monash Australian and Malaysian campuses. A key element of transitioning from a Bachelor of Medicine, Bachelor of Surgery (MBBS(Hons)) degree to a vertical double Bachelor of Medical Science/MD degree was the introduction of the Year 3/B module called Research Methods in Medicine (RMM).  

RMM aims to enhance students’ research knowledge and skills, specifically in biostatistics and research methods. It teaches complex, high-level statistical knowledge, building on foundational knowledge gained from Years 1 and 2, whilst integrating with the current Year 3 and 4 evidenced-based medicine clinical practice content, and preparing students for their Year 5 Scholarly Intensive Placement (SIP) projects.  

A number of challenges were identified during the design and development stage of RMM: 

• Starting in 2018, there was an annual cohort of approximately 500 students  
• Passing this unit was a hurdle requirement for entry into 5th year medicine for all students 
• Authentic assessment and comprehensive assessment strategies were required for failing students 
• It had to be an accessible and consistent learning experience for metropolitan, rural and Malaysian student cohorts 
• It had to be flexible, with students taught whilst on clinical rotations at various sites 
• It had to demonstrate to the Australian Medical Council (AMC) how this work supported the transition for AQF 9 [1].

To address these challenges, RMM was designed using connectivism and a cognitive constructivist approach.  

How was the program implemented?

RMM was the first completely online component in the Monash medical degree, comprising an 11-week program with six modules developed as interactive e-books.  

For the most part students in RMM work on their projects autonomously, with the project tasks building on the learning modules. However, as students are randomly allocated to unique datasets, they can connect online and support each other through the scientific process, whilst not having the same answer. Further, the online discussion boards are structured with weekly prompts that lead directly into the weekly learning objectives and explore key topics. This encourages students to connect online and has stimulated opportunities for robust discussion (e.g. a discussion of the variable gender). 

Beginning in mid-February, all Year 3B medical students are enrolled into the unit and introduced to the program via a series of introductory videos aimed at putting the unit into context with the rest of their learning.  Students are allocated two self-directed hours per week to complete a module and an additional two hours per week to work on the corresponding section of the assessment. 

The six modules are: 1) The role of statistics in health, 2) Graphical and chart representation, 3) Quantifying uncertainty in data analysis, 4) Evaluation of statistical significance 5) Additional statistical methods: chi-square & non-parametric methods 6) Introduction to qualitative methods.  

These modules have been designed as bespoke activities and are structured so that each module starts with a learning objective which directly links to the Check Your Learning activity at the end. Each module is further supported with explanations and worked examples that are relevant to clinical practice and research. Although the examples differ, there are some key concepts that link the modules and provide an opportunity for cascading depth learning activities. In these instances, a worked example with a concurrent student activity will bridge multiple modules to scaffold the learning and build on previous concepts. For example, the same activity can use the same data variables where they are defined, graphed, tabled and then analysed.  

Other innovative elements of learning resources are the bespoke educational videos with H5P questions embedded, hands on application with the drawing of graphs, gamification of key concepts in form of consolidation crosswords, activities that require application and data manipulation, and interactive infographics. Student feedback of the resource has been overwhelming positive but also informative, as it has allowed for the program’s continued improvement. A snapshot of student feedback highlights how the design of RMM has enhanced student learning: 

‘The research methods in medicine content was really clearly presented and comprehensive. This definitely helped fill in those holes and aided me in learning and understanding a lot more of this content. Thanks for delivering this module helping us all get our head around the stats!’  

 ‘was very easy to use. The questions following each section allowed for excellent content revision and assisted with information retention’  

‘Helpful in consolidating concepts. Life-saver in understanding tricky concepts especially in an online learning environment where it’s more difficult to reach out for assistance. Enjoyed the extra questions to reinforce concepts!’ 

One of the prized outcomes of RMM is that this student learning experience was scalable, robust, applicable and flexible in nature. These characteristics allowed the program to be adapted to other programs, such as being incorporated into the coursework of the honours year of Bachelor of Biomedical Science (BMS). Further, the COVID-19 pandemic saw a dramatic increased usage of these syllabi into a number of other units at both a postgraduate and undergraduate level and across the Faculty including BMS, Bachelor of Health Sciences, Bachelor of Public Health, other years of the MD, Master of Public Health and Bachelor of Physiotherapy.  

This is evidenced by the data analytics and increased uptake of all modules since 2021, with the ebooks increasing in student views by 48-58% (see Table 1). As an example, The role of statistics in health was viewed by 221,649 students from the 2017 pilot phase to December 2020. The learning analytics show students returning to the materials throughout their studies and to support their learning throughout their MD and other studies.

Table 1. Research methods in medicine data analytics

This increase in usage is consistent with the positive feedback received from colleagues who have eagerly adopted these resources into their programs: 

‘I wanted to get in touch as I am using your modules from “Research Methods in Medicine” in the X this year to support the student’s understanding of biostatistics. The modules themselves look really clear and engaging and I am particularly impressed by the functionality to display the content in a dyslexia friendly font. Thank you for making this resource and I will let you know how the students go as they work through the modules this semester.’  

Assessment – Application, interpretation and working with data

The RMM assessment strategy was aligned and developed alongside the learning resources, allowing students to work on it progressively as they completed each module.  

The major assessment task is called The Sunnyville Report, which consists of 43 questions that require students to use the different skills they have developed, such as data description, an understanding of sampling, simple to complex calculations, graphing, use of statistical software and interpretation of the findings, all built into an online quiz platform to allow for automatic marking (see Appendix 1: Example of assessment questions).  

Sections of the report are complete with bespoke videos that link back to the learning resource as a lesson plan, and the questions directly link to the learning objectives of the modules. The learning outcomes of the assessment ensure students will be able to: 

• demonstrate an ability to present data clearly and interpret its results 
• conduct and show workings, where appropriate, for various statistical tests 
• describe the epidemiological foundations for obtaining the data 

• generate and interpret results based on various statistical tests and public health tasks. 

Students are randomly allocated to one of ten different datasets but have the same question set. Different datasets ensure that students need to be confident when choosing, creating/applying and interpreting their results. It also allows the students to support each other, as can been seen with high-level questions being asked on the discussion boards on the Moodle site.  

The Sunnyville Report is a form of experiential learning, as gamification is used with a strong narrative to ensure that the research task is exciting, engaging and is a problem that must solved. The assignment takes approximately 10-12 hours to complete and is approximately 2,500-word equivalent in length, with a focus on data analysis, calculations and reporting of findings. It is designed in a manner that allows students to receive their overall mark and feedback on each question, as per the marking criteria provided on the learning management system Moodle. 

How has the impact of this program been evaluated?

RMM was supported by the Faculty of Medicine, Nursing and Health Sciences and received funding through a competitive internal learning and teaching grant entitled Evaluating the development of research knowledge and skills.  

The aim of this project was to 1) develop the content material and then 2) evaluate the RMM program by assessing research knowledge and skills among the new MD student cohort. Students were asked to complete a before and after questionnaire with thirteen sections covering domains beyond the RMM program, to identify gaps in student learning. The research team is currently validating the questionnaire, however preliminary results have shown that skills in research methods require specific learning, with students benefiting from the structure of the program and activities of the RMM. 

RMM is well suited for the asynchronous learning environment, as it is fully online with a quiz-based assessment which is feasible for large student cohorts in different locations while completing their clinical placements. Importantly, student engagement with the module has been demonstrated by students returning to the resources throughout their studies. 

APPENDIX

Appendix 1. Example of assessment questions

Reference

[1] Australian Qualifications Framework Council, Australian Qualifications Framework, 2nd Edition, 2013. p. 59. 

What was the main aim of this project?

Journal clubs are a mainstay of medical education: they are a practical and collaborative way to improve content knowledge and promote evidence-based medicine; [1]  they play an important role in developing students’ ability to critically appraise the literature and research methodologie[2]; and they help students recognise the importance of keeping up to date with the medical literature to help inform their future clinical practice [3]. Journal clubs also facilitate group discussion with peers, which is a valuable addition to the scholarship of teaching and learning [4].

A recent study of the prevalence and implementation of journal clubs within Australian medical schools found that over 80% of schools held journal clubs as part of the curriculum, though the majority were held during later-stage clinical years and tended to be clinician-led[1].

The COVID-19 pandemic provided a stimulus to restructure the journal club format. The new virtual journal club format delivered through this project was designed to be student-centred, with students encouraged to lead their session and drive interaction with their audience. Also, it capitalised on the idea that introducing and engaging students in journal clubs earlier in their medical education may help to increase and maintain an ongoing interest in and understanding of research and its role in medicine.

Integration with the curriculum

Research and critical analysis (RCA) content is integrated into the curriculum in each of the four years of the graduate MD program at University of Wollongong [5,6]. Development of this award-winning RCA program [7] was based on the frameworks of Miller’s pyramid [8] and Boyer’s domains of academic scholarship [9].

RCA principles taught in the first 18 months of the MD program (Phase 1) are delivered in large group sessions and practiced through small group journal club presentations (see Figure 1, below). The journal club presentations align with the lecture content to ensure that students can build on this knowledge through interpretation, application and demonstration of skills. Group journal club presentations provide the students with one of their first opportunities to apply their skills and reflect on their skills development by receiving structured feedback on the task.

Figure 1. Research and critical analysis (RCA) assessment tasks over the four Phases, as aligned with the framework of Miller’s triangle. POEM: patient-orientated evidence that matters. Figure adapted from [7] Mansfield et al., (2022).

The key learning outcomes of the journal club address five of the nine RCA Learning Outcomes, as defined in Table 1 below.

Table 1. Learning outcomes for the research and critical analysis (RCA) content across the four-years of the Graduate Medicine course at UOW and those that align with outcomes of the journal club task.

What changes were implemented?

Prior to the COVID-19 pandemic a traditional in-person journal club format was employed, which involved a non-assessable task and voluntary attendance. As part of these sessions, Phase 1 medical student groups were encouraged to critically appraise an article selected by an academic staff member, which focussed on the RCA principles and the body system block being taught at the time.

Student groups presented a critique of their assigned articles to their peers during a casual lunchtime journal club session, as is traditional in many medical and educational institutions. Academics involved in the journal club often commented that while the presenting students engaged with the task, there were very few other students attending to watch and engage with the presentations, so that little to no discussion was generated following the presentation. Additionally, feedback from students suggested it was considered a low-priority activity.

The new virtual journal club format developed during the COVID-19 pandemic was devised to be an interactive space where discussion was encouraged. A safe and supportive environment was established through appropriate questioning and positive encouragement from participating academics. Students were also guided through an information session about the task and how to give respectful and constructive feedback when engaging in discussion.

The journal club presentations became an assessable component, with clear learning objectives. It was anticipated that this would encourage greater peer interaction, collaboration and student-led learning.

A marking rubric was designed to assess each group presentation against a range of RCA learning objectives. Based on the rubric, grades of ‘excellent’, ‘satisfactory’ or ‘unsatisfactory’ were awarded by attending academics for the following activities of the groups: literature searching skills and selection of an appropriate article; appraisal of the study design and research methods; appraisal of the results and discussion; appraisal of the strengths and weaknesses of the study; articulation of the implications for clinical relevance. This change was based on the premise that the use of a structured critical appraisal is integral to running a successful journal club.[10] Presentation skills were also assessed as part of the rubric and a student peer assessment score incorporated to encourage all students to contribute equitably to the group work, knowing that their contribution would be rated by other group members.

How was it delivered?

Students were assigned a tutorial group for their regular activities in Phase 1 and these same groups were utilised for their journal club task. Each group was given a topic incorporating a methodology or analysis technique relevant to RCA principles and aligned with the RCA content being delivered at large group teaching sessions during the period. Students were advised to search for a recent and appropriate article, clinically relevant to the current medical case being examined and addressing the appropriate RCA principles. This facilitated the development of their literature searching skills within a confined topic, and increased student ownership and interest in the task.

The assessment criteria aimed to build an understanding of how to apply medical literature to evidence-based medicine and assess its clinical relevance. Two Phase 1 RCA academic leads attended each of the virtual journal club presentations to maintain consistency in marking and student feedback. A guest academic also joined each journal club presentation to provide additional perspectives and act as a third assessor.

How has the impact of this project been evaluated?

Specific goals for the reinvigorated and restructured journal club were to: increase students’ research and critical analysis skills; improve attendance and peer interaction; and increase engagement with learning about evidence-based medicine. To evaluate the effectiveness of the journal club, student assessment marks, student reflections and staff reflections were considered.

Overall, student presentations were assessed to be of a high standard, demonstrating a deep understanding and strong engagement in the assigned topics. Further, discussion was greatly improved, with students asking numerous questions and the full hour allocation (30-minute presentation and 30-minute discussion) was often used due to the vigour of the discussion  – in contrast to previous iterations which routinely ended following the 30-minute presentation.

Student reflections

Student reflections, via an anonymous online survey and spontaneous email feedback, indicated the achievement of the goals for the virtual journal club.

Students were asked whether attendance at journal club increased or improved their confidence over a range of research and critical analysis skills, such as: searching databases and finding relevant literature, understanding methodology, biostatistics, and interpretation of results. They were also asked to reflect on what they found useful about journal club and possible session improvement. Over one third of the cohort responded to the survey (37%).

Attendance was increased with the new online approach. While attendance at the nine journal club sessions was not compulsory, 44% of respondents self-reported attendance at 7-9 sessions and 78% attended more than half of the sessions (5-9). Online attendance counts across the sessions were on average 58% (±18%; range 28% – 85%), with the lower attendances being near exams. Most feedback on improvements to journal club related to attendance and, specifically, the timing of presentations to reduce impact on schedules near exam periods, or the desire for in-person or hybrid delivery of journal club.

Increasing peer interaction and providing a forum for discussion about research was another goal achieved by the reinvigorated journal club. Students overwhelmingly reported that they enjoyed the interaction with peers during journal club (70% agreed) and that it was an engaging way to learn about research and critical analysis (93% agreed), see Figure 2 below. Students who attended less than half of the journal club sessions reported a lower level of enjoyment in peer interaction.

Figure 2. Students reflected positively on their engagement with journal club, survey responses shown for the questions

A. “I enjoyed the interaction with peers during journal club”
B. “I found the Journal Club presentations an engaging way to learn about medical research and critical analysis”. Results averaged from n=28 responses (37% of cohort).

Table 2. Student and staff reflections on the updated journal club activity.

For all RCA skills examined, over 50% of respondents agreed that their skills had improved (Figure 3).

Figure 3. Student evaluation of their skills following participation in the journal club sessions (data represents average of n = 28 responses: 37% of the cohort).

The increased confidence in RCA skills was apparent, regardless of prior level of research experience or how many journal club sessions were attended, suggesting that the completion of the task ensured students developed and applied many of the required skills (Quote 2, Table 2). Student-led learning and peer collaboration was a large component of this skill development, and several students reflected on the benefits of peer learning when undertaking the task (Quotes 3, 4 and 5, Table 2). The structured rubric used to assess the journal club appraisals was also highlighted as a useful tool in the student reflections (Quotes 6 and 7, Table 2).

The effectiveness of journal clubs can also be evaluated by measuring changes in reading habits of participants [10,11]. Student reflections demonstrated that in addition to increased confidence in their critical appraisal skills, 60% of students agreed that their motivation to read research articles had increased following journal club participation (Figure 3). 

Staff reflections

Staff discussions occurred regularly during implementation of the new-format journal club program. Staff agreed that students were more engaged with the new group presentation assessment task and that the interactive format was of benefit to their skills and understanding. Because groups were meeting regularly as part of their ongoing tutorials, they often incorporated their journal club preparation into their group meetings and discussed concepts with their tutors and RCA facilitators. This increased engagement was demonstrated in the depth of knowledge displayed by the students, both in the construction of their appraisal and the discussion of questions raised following the presentation, which was of an exceptional standard. This was acknowledged by both staff and students as a positive outcome following journal club (see quotes 8 and 9, Table 2).

Academic staff engagement with journal club also increased, with all academic staff attending at least one session and some staff attending additional sessions to support students they tutored or out of personal interest in the topics. This further fostered a sense of comradery and collegiality with the students. While it was not specified in the marking criteria, each student group spontaneously incorporated entertaining and original components to their talk, with students dressing up in formal or themed attire, having special thematic slides, polls, quizzes and animations in their presentations, and one even producing an entertaining video demonstrating some important concepts from their research topic. It was evident that these efforts to engage the audience were successful and built up over time.

Conclusion

The development of a supportive and interactive environment for an assessable online journal club resulted in students overwhelmingly agreeing that it was an engaging way to learn about research. There was increased student attendance and engagement, both with the new format journal club sessions and the RCA course content generally. Learning outcomes for RCA content were detailed in the marking rubric and the survey of student confidences was mapped to these outcomes. Students rated their confidence in critical appraisal skills as having improved following their participation in journal club across all RCA appraisal and interpretation components including literature searching, methodology, statistical analysis, interpretation of results, discussion, and clinical relevance.

One of the key learning outcomes for the journal club was to facilitate discussion and debate of evidence-based medicine and to encourage an appreciation for research. Students reported that their motivation to incorporate research literature into their regular studies had also increased. Further, both students and academics acknowledged the depth of knowledge displayed within the presentations and the high standard of discussions that were generated.

References

[1] Ilic D, de Voogt A, & Oldroyd J. (2020). The use of journal clubs to teach evidence-based medicine to health professionals: A systematic review and meta-analysis. Journal of Evidence Based Medicine, 13(1), 42-56. https://doi.org/https://doi-org.ezproxy.uow.edu.au/10.1111/jebm.12370

[2] Ianno, D. J., Mirowska-Allen, K., Kunz, S. A., & O’Brien, R. (2020). Journal clubs in Australian medical schools: prevalence, application and educator opinion. Journal of Educational Evalulation for Health Professionals, 17, https://doi.org/10.3352/jeehp.2020.17.9

[3] Shrivastava, S. R., & Shrivastava, P. S. (2021). Promoting the Conduct of Medical Education Journal Clubs in Teaching Medical Institutions. Avicenna Journal of Medicine, 11(3), 156-159. https://doi.org/10.1055/s-0041-1735126

[4] Szucs, K. A., Benson, J. D., & Haneman, B. (2017). Using a Guided Journal Club as a Teaching Strategy to Enhance Learning Skills for Evidence-Based Practice. Occupational Therapy In Health Care, 31(2), 143-149. https://doi.org/10.1080/07380577.2016.1278296

[5] Mullan, J. R., Mansfield, K. J., Weston, K. M., Rich, W., Burns, P., Brown, C., & McLennan, P. L. (2017). ‘Involve Me and I Learn’: Development of an Assessment Program for Research and Critical Analysis. Journal of Medical Education and Curricular Development, 4, 2382120517692539. https://doi.org/10.1177/2382120517692539

[6] Mullan, J. R., Weston, K. M., Rich, W. C., & McLennan, P. L. (2014). Investigating the impact of a research-based integrated curriculum on self-perceived research experiences of medical students in community placements: a pre- and post-test analysis of three student cohorts. BMC Medical Education, 14(1), 161. https://doi.org/10.1186/1472-6920-14-161

[7] Mansfield, K., Metusela, C., & Mullan, J. R. (2022). Building research capacity for rural and regional medical students. University of Wollongong Research and Critical Analysis Program – case study. In: Research in the Medical Curriculum Volume 1: A Window on Innovation and Good Practice. Medical Deans Australia and New Zealand

[8] Miller G. E. (1990) The assessment of clinical skills/competence/performance. Academic Medicine, 65(9), S63-67.

[9] Boyer E. L. (1990) Scholarship Reconsidered. Princeton, NJ: Carnegie Foundation for the Advancement of Teaching

[10] Deenadayalan, Y., Grimmer-Sommers, K., Prior, M., & Kumar, S. (2008). How to run an effective journal club: a systematic review. Journal of Evaluation in Clinical Practice, 14(5), 898-911. https://doi.org/https://doi-org.ezproxy.uow.edu.au/10.1111/j.1365-2753.2008.01050.x

[11] Linzer, M., Brown, J. T., Frazier, L. M., DeLong, E. R., Siegel, W. C. (1988) Impact of a Medical Journal Club on House-Staff Reading Habits, Knowledge, and Critical Appraisal Skills: A Randomized Control Trial.  260(17):2537–2541. https://doi:10.1001/jama.1988.03410170085039

What was the main aim of this project?

The overall aim of the Sydney Medical Program’s MD Project is to teach students the basic tenets of medical research through a hands-on research project experience. During the project, students develop key research and generic skills, including self-motivation, time management, organisation problem-solving, and building relationships in clinical or research laboratory environments.

The University of Sydney introduced a new postgraduate Doctor of Medicine (MD) Program in 2020, including a new format for the compulsory, individual student research project now run in a dedicated block in Year 3. With the renewal of the MD curriculum, we had the opportunity to revise our MD Project and address a key concern previously raised by both students and supervisors: the lack of a dedicated time for the MD Project. [1, 2].

The pre-2020 MD Project was very successful, but the students were required to complete their projects in their own time on top of the rest of the curriculum. Competing learning priorities meant project milestone assessment tasks acted as extrinsic motivators and barriers to overcome, instead of activities that meaningfully contributed to student learning [2]. Thus, with the introduction of the new MD curriculum, a 14-week block in Year 3 was quarantined for completing the MD Project.

Learning theory underpinning the project

The pedagogical framework for the MD Project employs active, experiential, project-based learning in a research context, with individual projects being supervised by academic or clinical staff mentors or mentoring teams. Throughout the MD Project block an additional series of workshops on various research methods, literature search strategies, statistical analysis, scientific presentations and scientific writing is provided to support the student projects.

The MD Project is scaffolded with milestone tasks that guide the students through the research process, from project planning to the submission of a scientific report.

Integration with other components of the MD curriculum

The MD Project is a key component of the vertical curriculum theme of Research, Evidence and Informatics, which runs throughout the four-year MD Program. Research methods and research ethics are taught in Year 2, along with evidence-based medicine, where students are taught to be effective users of medical research knowledge. The MD Project then requires the students to do the research and puts them on the pathway to future development of independent research capabilities.

Learning outcomes

The learning objectives of the MD Project are listed below:

How was the MD Project implemented?

The University of Sydney MD Program has about 300 students per year. In Year 3, students undertake an individual research project on any topic related to health or medicine. Project data may be quantitative, qualitative or mixed methods, and a wide scope of project types are accepted, including:

• narrative, scoping or systematic literature review
• study from inception to completion, in laboratory, clinical, public health or other fields
• project centred on development of a product (e.g. an educational video)
• project using publicly available health-related data
• developing a study protocol and ethics application.

The MD Project team is comprised of several academics and education support staff. Projects are delivered at nine locations across NSW, including urban and rural clinical schools and affiliated research institutes. This involves recruitment of 150+ research supervisors, ~10 research co-ordinators and a further ~10 education support staff. The MD Project team oversees the recruitment for all projects and the allocation of students to their projects.

General information about the MD Project Program is available to students from commencement of the MD Program. Students in Year 2 are then introduced to the MD Project by the MD Project team staff via information sessions and online Q&A sessions. Project topics are posted for students to select from, with projects allocated according to student preference. All information about the MD Project is presented on a dedicated Canvas website that students and supervisors can access.

Many of the projects require human or animal ethics approvals. All ethics approvals are the responsibility of the research supervisor, however the MD Project team ensures that these are in place before the MD Project block. The MD Project team can also advise research supervisors on the basics of ethics applications, study design and statistical analysis.

Assessment of learning objectives

The MD Program has programmatic assessment, thus the milestone tasks, the final scientific report, and professionalism and engagement with the project are all included as data for the programmatic assessment process. Submitted tasks graded borderline or not meeting standards require revision and resubmission.

Detailed rubrics are available for all the assessed tasks and they are all mapped back to the stated learning objectives. The final assessment task is a 3000-word written report in the form of a scientific paper. These reports are assessed by academic staff with no conflicts of interest using a standard rubric, and students receive feedback on their reports from a minimum of two assessors. The final report marking is a large undertaking involving co-ordination of around 200 academic assessors.

How has the impact of this project been evaluated?

We ran the first 14-week MD Project in Jan-Apr 2022. Research skills attained by students were assessed by survey at the beginning and then again at the end of the block, using the Research Spider [3]. The data show the students perceive a significant improvement in their research skills across all domains, except applying for research funding, which is outside the scope of the MD Project (Figure 1).

Figure 1. Students’ self-reported research skills in ten research skill areas pre and post the MD Project. Likert Scale of  0 (no experience) to 5 (highly experienced), n=182, P<0.0001, paired t-test.

Student reflections

There were about 30 different research and generic project-related skills cited by students as important skills they learnt by completing the MD Project. Students report that the top three skills they learnt during the MD Project were searching and reviewing the literature (15%), time management skills (9%) and scientific writing skills (8%). They report having a better understanding of the overall research process, statistical and other data analyses, and are upskilled in the use of software that supports research.  

A selection of quotes on the best things learnt during the MD Project:

• “Developing familiarity with many aspects of the research process as a whole”
• “The opportunity to do a project that involved so much hands-on experience”
• “The importance of effective time-management and planning when conducting a research project”
• “The importance of effective communication and collaboration with other members of a research team to ensure that goals are met”.

Staff reflection

The supervisor reports on their students at the conclusion of the 14-week project block were resoundingly positive, many reporting their students were outstanding, highly engaged, dedicated to the project and progressed well with the project and learning research skills. When asked to give general feedback on their student’s overall performance in the MD Project, Supervisors reported:

• “My student was engaged with the research project and showed an eagerness to learn and develop their research skills.”
• “My student posed thoughtful research questions and listened carefully to advice provided by myself and others. It has been a pleasure working with them on this project.”
• “My student has shown outstanding levels of commitment, professionalism and tenacity during this project. They clearly became invested in the relevance and importance of the work.”

Project outputs

A number of students have published or intend to submit their project findings for peer-reviewed publication, or present their results at national or international conferences. These are very desirable outcomes but are not required for the MD Project.

Conclusion

Overall, the compressed timeline of the new MD Project was beneficial, as we retained the learning gains and avoided the stop-start progress of the previous MD Project Program. Both students and project supervisors reported a positive experience and that students attained both research and project management skills.

References

[1] Kaur, R. et al., Perceived research skills development and student satisfaction with a mandatory research project: A survey of five cohorts from 2017-2020, presented at ANZAHPE, online conference 2022.

[2] Hart, J. et al., Research supervisors’ views of barriers and enablers for research projects undertaken by medical students; a mixed methods evaluation of a post-graduate medical degree research project program. BMC Medical Education, 2022. 22(1): p. 370-370.

[3] Smith, H. et al., The ‘Research Spider’: a simple method of assessing research experience. Primary Health Care Research & Development, 2002. 3(3): p. 139-140.