Effective in-service training design and delivery: evidence from an integrative literature review

Background In-service training represents a significant financial investment for supporting continued competence of the health care workforce. An integrative review of the education and training literature was conducted to identify effective training approaches for health worker continuing professional education (CPE) and what evidence exists of outcomes derived from CPE. Methods A literature review was conducted from multiple databases including PubMed, the Cochrane Library and Cumulative Index to Nursing and Allied Health Literature (CINAHL) between May and June 2011. The initial review of titles and abstracts produced 244 results. Articles selected for analysis after two quality reviews consisted of systematic reviews, randomized controlled trials (RCTs) and programme evaluations published in peer-reviewed journals from 2000 to 2011 in the English language. The articles analysed included 37 systematic reviews and 32 RCTs. The research questions focused on the evidence supporting educational techniques, frequency, setting and media used to deliver instruction for continuing health professional education. Results The evidence suggests the use of multiple techniques that allow for interaction and enable learners to process and apply information. Case-based learning, clinical simulations, practice and feedback are identified as effective educational techniques. Didactic techniques that involve passive instruction, such as reading or lecture, have been found to have little or no impact on learning outcomes. Repetitive interventions, rather than single interventions, were shown to be superior for learning outcomes. Settings similar to the workplace improved skill acquisition and performance. Computer-based learning can be equally or more effective than live instruction and more cost efficient if effective techniques are used. Effective techniques can lead to improvements in knowledge and skill outcomes and clinical practice behaviours, but there is less evidence directly linking CPE to improved clinical outcomes. Very limited quality data are available from low- to middle-income countries. Conclusions Educational techniques are critical to learning outcomes. Targeted, repetitive interventions can result in better learning outcomes. Setting should be selected to support relevant and realistic practice and increase efficiency. Media should be selected based on the potential to support effective educational techniques and efficiency of instruction. CPE can lead to improved learning outcomes if effective techniques are used. Limited data indicate that there may also be an effect on improving clinical practice behaviours. The research agenda calls for well-constructed evaluations of culturally appropriate combinations of technique, setting, frequency and media, developed for and tested among all levels of health workers in low- and middle-income countries.


Background
The need to increase the effectiveness and efficiency of both pre-service education and continuing professional education (CPE) (in-service training) for the health workforce has never been greater. Decreasing global resources and a pervasive critical shortage of skilled health workers are paralleled by an explosion in the increase of and access to information. Universities and educational institutions are rapidly integrating different approaches for learning that move beyond the classroom [1]. The opportunities exist both in initial health professional education and CPE to expand education and training approaches beyond classroom-based settings.
An integrative review was designed to identify and review the evidence addressing best practices in the design and delivery of in-service training interventions. The use of an integrative review expands the variety of research designs that can be incorporated within a review's inclusion criteria and allows the incorporation of both qualitative and quantitative information [2]. Five questions were formulated based on a conceptual model of CPE developed by the Johns Hopkins University Evidence-Based Practice Center (JHU EPC) for an earlier systematic review of continuing medical education (CME) [3]. We asked whether: 1. particular educational techniques, 2. frequency of instruction (single or repetitive), 3. setting where instruction occurs, or 4. media used to deliver the instruction make a difference in learning outcomes; and, 5. if there was any evidence regarding the desired outcomes, such as improvements in knowledge, skills or changes in clinical practice behaviours, which could be derived from CPE, using any mixture of technique, media or frequency.

Inclusion/exclusion criteria
Articles were included in this review if they addressed any type of health worker pre-service or CPE event, and included an analysis of the short-term evaluation and/or assessment of the longer-term outcomes of the training. We included only those articles published in English language literature. These criteria gave priority to articles that used higher-order research methods, specifically meta-analyses or systematic reviews and evaluations that employed experimental designs. Articles excluded from analysis were observational studies, qualitative studies, editorial commentary, letters and book chapters.

Search strategy
A research assistant searched the electronic, peer-reviewed literature between May and June 2011. The search was conducted on studies published in the English language from 2000 to 2011. Multiple databases including PubMed, the Cochrane Library and Cumulative Index to Nursing and Allied Health Literature (CINAHL) were utilized in the search. Medical subject headings (MeSH) and key search terms are presented below in Table 1.

Study type, quality assessment and grade
An initial review of titles and abstracts produced 244 results. We identified the strongest studies available, using a range of criteria tailored to the review methodology. Initial selection criteria were developed by a panel of experts. Grading and inclusion criteria are presented in Table 2. The grading criteria were adapted from the Oxford Centre for Evidence-Based Medicine (OCEMB) levels of evidence model [4]. Grading of studies included within systematic reviews was reported by authors of those reviews and was not further assessed in this integrative review. Therefore, reference to quality of studies in our report refers to those a priori judgments. Only tier 1 articles (grades 1 and 2) were included in our analysis.
After prioritization of the articles, 163 tier 1 articles were assessed by a senior public health professional to determine topical relevance, study type and grade. A total of 61 tier 1 studies were selected to be included in the analysis following this second review. An additional hand search of the reference lists cited in published studies was conducted for topics that were underrepresented, specifically on the frequency and setting of educational activities. This search added eight articles for a total of 69 studies, including 37 systematic reviews and 32 randomized controlled trials (RCTs), see inclusion process for articles included in analysis, Figure 1.
A data extraction spreadsheet was developed, following the model offered in the Best Evidence in Medical Education (BEME) group series [5] and the conceptual model and definition of terms offered by Marinopoulos et al. in the JHU EPC earlier review of CME [3]. Categorization decisions were necessary in cases when the use of terminology was inconsistent with the Marinopoulos et al. definitions of terms for CPE [3]. For example, an article that analysed 'distance learning' as a technique and used the computer as the medium to deliver an interactive elearning course was coded and categorized as an 'interactive' technique delivered via 'computer' as the medium of instruction. See illustration of categorization terminology in panels A, B, and C, Figure 2, for an illustration of how terminology was used to categorize and organize articles for analysis.

Results
Selected articles that best represent common findings and outcomes (effects) of CPE are discussed in the results and discussion sections; the related tables present all the articles analysed and categorized for that topic, and each article is included only once. Relevant information obtained from educational psychology literature is referenced in the discussion.

Techniques
The articles or studies that specifically addressed educational techniques are summarized in Table 3. Technique refers to the educational methods used in the instruction. Technique descriptions are based on the Marinopoulos et al. definitions of terms [6] and reflect the approaches defined in the articles analysed.

Case-based: use of created or actual clinical cases that present materials and questions
Though case-based learning was not specifically compared with other techniques in the literature reviewed, it was often noted as a method in articles that discussed interactive techniques. Case-based learning was also noted as a technique used for computer-delivered CPE courses. Triola et al. compared types of media utilized for case-based learning and found positive learning outcomes both with the use of a live standardized patient and a computer-based virtual patient [7].
Didactic/lecture: presenting knowledge content; facilitator determines content, organization and pace Lecture was often referred to in the literature as traditional instruction, lecture-based or didactic teaching. Didactic instruction was not found to be an effective educational technique compared with other methods. Two studies [8,9] found no statistical difference in learning outcomes, and three studies found didactic to be less effective than other techniques [10][11][12]. Reynolds et al. compared didactic instruction with simulation. The study was limited by small sample size (n = 50), but still demonstrated that the simulation group had a significantly higher mean post-test score (P <0.01) and overall higher learner satisfaction [12].
Several systemic reviews that compared didactic instruction to a wide variety of teaching approaches also identified didactic instruction as a less effective educational technique [13][14][15].

Feedback: providing information to the learner about performance
Multiple articles identified feedback as important for outcomes [16][17][18]. Herbert et al. compared individualized feedback in the form of a graphic (a prescribing portrait based on personal history of drug-prescribing practices) to small group discussion of the same material and found that both the feedback and the live, interactive session were somewhat effective at changing physician's prescribing behaviours [16]. The Issenberg et al. systematic review of simulation identified practice and feedback as key for effective skill development [17]. A Cochrane review of the evidence to support CPE suggested the importance of feedback and instructor interaction in improving learning outcomes [18].

Games: competitive game with preset rules
The use of games as an instructional technology was addressed in one rigorous systematic review. The authors found only a limited number of studies, which were of low  to moderate methodological quality and offered inconsistent results. Three of the five RCTs included in the review suggested that educational games could have a positive effect on increasing medical student knowledge and that they include interaction and allow for feedback [19].

Interactive: provide for interaction between the learner and facilitator
Five articles specifically compared interactive CPE to other educational techniques. De Lorenzo and Abbot found interactive techniques to be moderately superior for knowledge outcomes than didactic lecture [10]. Two other studies found interactive techniques were more effective when feedback from chart audits was added to the intervention [16,20]. Three systematic reviews and one meta-analysis specifically noted the importance of learner interactivity or engagement in learning in achieving positive learning outcomes [21][22][23][24] (refer to summary of articles focused on outcomes).
Point-of-care (POC): information provided as needed, at the point of clinical care Two articles and one systematic review specifically addressed point-of-care (POC) as a technique. The systematic review included three studies and concluded that while the findings were weak, they did indicate that POC led to improved knowledge and confidence [25]. In an examination of media, Leung et al. determined that handheld devices were more effective than print-based, POC support, although outcome measures were selfreported behaviours [26]. You et al. found improved performance on a procedure among surgical residents who       received POC mentoring via a video using a mobile device, compared with those who received only didactic instruction [27].
Problem-based learning (PBL): present a case, assign information-seeking tasks and answer questions about the case; can be facilitated or non-facilitated Four articles specifically compared problem-based learning (PBL) to other methods. One study identified PBL as slightly better [11], and two studies indicated it to be relatively equal to didactic instruction [8,9]. A systematic review of 10 studies on PBL reported inconclusive evidence to support the approach, although several studies reported increased critical thinking skills and confidence in making decisions [28].

Reminders: provision of reminders
The Zurovac et al. study conducted in Kenya found that using mobile devices for repetitive reminders resulted in significant improvement in health care provider's case management of paediatric malaria, and these gains were retained over a 6-month period [29]. Intention-to-treat analysis showed that correct management improved by 23.7% (95% confidence interval (CI) 7.6 to 40.0, P <0.01) immediately after intervention and by 24.5% (95% CI 8.1 to 41.0, P <0.01) 6 months later, compared with the control group [29]. Reminders were also noted as an effective technique by two of the systematic reviews [13,14].

Self-directed: completed independently by the learner based on learning needs
This term was difficult to extract for analysis due to widely varying terminology. Some authors used the term 'distance learning', and some used it to define the medium of delivery, rather than technique. This analysis specifically discusses articles that were consistent with the description for self-directed learning, even if the authors used different terminology.
A recent systematic review identified that moderatequality evidence suggests a slight increase in knowledge domain compared with traditional teaching, but notes that this may be due to the increased exposure to content [30]. One RCT found modest improvements in knowledge using a self-directed approach, but noted it was less effective at impacting attitudes or readiness to change [31].
Multiple studies focused on use of the computer as the medium to deliver instruction and noted that self-directed instruction was equally (or more) effective as instructorled didactic or interactive instruction and potentially more efficient.
Simulation may include models, devices, standardized patients, virtual environments, social or clinical situations that simulate problems, events or conditions experienced in professional encounters [17]. Simulation was noted as an effective technique for promotion of learning outcomes across the systematic reviews, particularly for the development of psychomotor and clinical decision-making skills. The systematic reviews all highlighted inconclusive and weak methodology in the studies reviewed, but noted sufficient evidence existed to support simulation as useful for psychomotor and communication skill development [32][33][34] and to facilitate learning [35]. The systematic review by Lamb suggests that patient simulators, whether computer or anatomic models, are one of the more effective forms of simulations [36].
Outcomes of the four separate RCTs indicated simulation was better than the techniques to which they were compared, including interactive [37,38], didactic [12] and problem-based approaches [35]. A study by Daniels et al. found that although knowledge outcomes were similar between the interactive and simulation groups, the simulation team performance in a labour and delivery clinical drill was significantly higher for both shoulder dystocia (11.75 versus 6.88, P <0.01) and eclampsia (13.25 versus 11.38, P = 0.032) at 1 month post-intervention [38].
Simulation was also found to be useful for identifying additional learning gaps, such as a drill on the task of mixing magnesium sulfate for administration [39]. A systematic review focused on resuscitation training identified simulation as an effective technique, regardless of media or setting used to deliver it [40].
Team-based: providing interventions for teams that provide care together Articles discussed here focused on the technique of providing training to co-workers engaged as learning teams. One systematic review of eight studies found that there is limited and inconclusive evidence to support team-based training [41]. Two of the articles reporting on the same CPE study did not identify any improvements in performance or knowledge acquisition with the addition of using a team-based approach [39,42].

Frequency
This review included consideration of frequency, comparing single versus repetitive exposure. The findings regarding frequency are summarized in Table 4.
The three articles focused on frequency all support the use of repetitive interventions. These studies evaluated repetition using the Spaced Education platform (now called Qstream), an Internet-based medium that uses repeated questions and targeted feedback. The evidence from these three articles demonstrated that repetitive, time-spaced education exposures resulted in better knowledge outcomes, better retention and better clinical decisions compared with single interventions and live instruction [43][44][45].
The use of repetitive or multiple exposures is supported in other systematic reviews of the literature, as well as one  RCT conducted in Kenya that used repeated text reminders and resulted in a significant improvement in adherence to malaria treatment protocols [29].

Setting
Setting is the physical location within which the instruction occurs. We identified three articles that looked specifically at the training setting. The findings regarding setting are summarized in Table 5. Two of them stemmed from the same intervention. Crofts et al. specifically addressed the impact of setting and technique (team-based training) on knowledge acquisition and found no significant difference in the post-score based on the setting [42]. A systematic review of eight articles evaluating the effectiveness of team-based training for obstetric care did not find significant differences in learning outcomes between a simulation centre and a clinical setting [41]. Coomarasamy and Khan conducted a systematic review and compared classroom or stand-alone versus clinically integrated teaching for evidence-based medicine (EBM). Their review identified that classroom teaching improved knowledge, but not skills, attitudes or behaviour outcomes; whereas clinically integrated teaching improved all outcomes [46]. This finding was supported by the Hamilton systematic review of CPE, which suggests that teaching in a clinical setting or simulation setting is more effective (Table 1), as well as the Raza et al. systematic review of 23 studies to evaluate stand-alone versus clinically integrated teaching. This review suggested that clinically integrated teaching improved skills, attitudes and behaviour, not just knowledge [18].

Media
Media refers to the means used to deliver the curriculum. The majority of RCTs compared self-paced or individual instruction delivered via computer versus live, group-based instruction. The findings regarding media are summarized in Table 6.

Live versus computer-based
Live instruction was found to be somewhat effective at improving knowledge, but less so for changing clinical practice behaviours. When comparing live to computerbased instruction, a frequent finding was that computerbased instruction led to either equal or slightly better knowledge performance on post-tests than live instruction. One of the few to identify a significant difference in outcomes, Harrington and Walker found the computer-based group outperformed the instructor-led group on the knowledge post-test and that participants in the computer-based group, on average, spent less time completing the training than participants in the instructor-led group [47].
Systematic reviews indicate that the evidence supports the use of computer-delivered instruction for knowledge and attitudes; however, insufficient evidence exists to support its use in the attempt to change practice behaviours. The Raza Cochrane systematic review identified 16 randomized trials that evaluated the effectiveness of Internet-based education used to deliver CPE to practicing health care professionals. Six studies showed a positive change in participants' knowledge, and three studies showed a change in practice in comparison with traditional formats [18]. One systematic review noted the importance of interactivity, independent of media, in achieving an impact on clinical practice behaviours [48].

Mobile
One article assessed the use of animations against audio instructions in cardiopulmonary resuscitation (CPR) using a mobile phone and found the group that had audiovisual animations performed better than the group that received live instruction over the phone in performing CPR; however, neither group was able to perform the psychomotor skill correctly [49]. Leung et al. found providing POC decision support via a mobile device resulted in slightly better self-reporting on outcome measures compared with print-based job aids, but that both the print and mobile groups showed improvements in use of evidence-based decision-making [26].

Print
The systematic review of print-based materials conducted by Farmer et al. did not find sufficient evidence to support the use of print media to change clinical practice behaviours [50]. A comparison of the use of print-based guidelines to a live, interactive workshop indicated that those who completed live instruction were slightly better able to identify patients at high risk of an asthma attack. However, neither intervention resulted in changed practice behaviours related to treatment plans [51].
Multiple systematic reviews caution against the use of print only media, concluding that live instruction is preferable to print only. Another consistent theme was support for the use of multimedia in CPE interventions.

Outcomes
Outcomes are the consequences of a training intervention. This literature review focuses on changes in knowledge, attitudes, psychomotor, clinical decision-making or communication skills, and effects on practice behaviours and clinical outcomes. All of the articles that focused on outcomes were systematic reviews of the literature and are summarized in Table 7.
The weight of the evidence across several studies indicated that CPE could effectively address knowledge outcomes, although several studies used weaker methodological approaches. Specifically, computer-based instruction was found to be equally or more effective than Same intervention as Crofts et al. 2007 Statistical evidence to support the use of live, eclampsia training to increase providers' performance rate for completion of basic tasks. Neither the location (simulation centre or in local hospitals), nor the inclusion of teamwork training made any significant difference to the performance results for basic task completion.
EBM Evidence-based medicine, I Intervention.    Both intervention groups had a didactic session, performed a thoracentesis on a manikin while using video on a mobile phone and received feedback from a live instructor; control group did not receive any video-aided guidance.
C Control, CME Continuing medical education, CPR Cardiopulmonary resuscitation, EBM Evidence-based medicine, I Intervention, NR Not reported, PDA Personal digital assistant. POC Point-of-care, RCT Randomized controlled trial.  Firm conclusions are not possible due to the overall low quality of the literature. Despite this, the literature supported the concept that CME was effective at the acquisition and retention of knowledge, attitudes, skills, behaviours and clinical outcomes. Common themes included that live media was more effective than print, multimedia was more effective than single media interventions, multiple exposures were more effective than a single exposure, and simulation methods are effective in the dissemination of psychomotor and procedural skills.   live instruction for addressing knowledge, while multiple repetitive exposures leads to better knowledge gains than a single exposure. Games can also contribute to knowledge if designed as interactive learning experiences that stimulate higher thinking through analysis, synthesis or evaluation.
No studies or systematic reviews looked only at attitudes, but CPE that includes clinical integration, simulations and feedback may help address attitudes. The JHU EPC group systematic review evaluation of the short-and long-term effects of CPE on physician attitudes reviewed 26 studies and, despite the heterogeneity of the studies, identified trends supporting the use of multimedia and multiple exposures for addressing attitudes [6].
Several systematic reviews looked specifically at skills, concluding that there is weak but sufficient evidence to suggest that psychomotor skills can be addressed with CPE interventions that include simulations, practice with feedback and/or clinical integration. 'Dose-response' or providing sufficient practice and feedback was identified as important for skill-related outcomes. Other RCTs suggest clinically integrated education for supporting skill development. Choa et al. found that neither the audio mentoring via mobile nor animated graphics via mobile resulted in the desired psychomotor skills, reinforcing the need for practice and feedback for psychomotor skill development identified in other studies [49].
Two systematic reviews focused on communication skills and found techniques that include behaviour modeling, practice and feedback, longer duration or more practice opportunities were more effective [52,53]. Evidence suggests that development of communication skills requires interactive techniques that include practice-oriented strategies and feedback, and limit lecture and print-based materials to supportive strategies only.
Findings also suggest that simulation, PBL, multiple exposures and clinically integrated CPE can improve critical thinking skills. Mobile-based POC support was found to be more useful in the development of critical thinking than print-based job aids.
Several systematic reviews specifically looked at CPE, practice behaviours and the behaviours of the provider. These studies found, despite reportedly weak evidence, that interactive techniques that involved feedback, interaction with the educator, longer durations, multiple exposures, multimedia, multiple techniques and reminders may influence practice behaviours.
A targeted review of 37 articles from the JHU EPC review on the impact of CPE on clinical practice outcomes drew no firm conclusions, but multiple exposures, multimedia and multiple techniques were recommended to improve potential outcomes [6]. Interaction and feedback were found to be more useful than print or educational meetings (systematic review of nine articles) [24], but print-based unsolicited materials were not found to be effective [50].
The systematic review of live, classroom-based, multiprofessional training conducted by Rabal et al. found 'the impact on clinical outcomes is limited' [54].

Discussion
The heterogeneity of study designs included in this review limits the interpretations that can be drawn. However, there is remarkable similarity between the information from studies included in this review and similar discussions published in the educational psychology literature. We believe that there is sufficient evidence to support efforts to implement and evaluate the combinations of training techniques, frequency, settings and media included in this discussion.
Avoid educational techniques that provide a passive transfer of information, such as lecture and reading, and select techniques that engage the learner in mental processing, for example, case studies, simulation and other interactive strategies. This recommendation is reinforced in educational psychology literature [55]. There is sufficient evidence to endorse the use of simulation as a preferred educational technique, notably for psychomotor, communication or critical thinking skills. Given the lack of evidence for didactic methods, selecting interactive, effective educational techniques remains the critical point to consider when designing CPE interventions.
Self-directed learning was also found to be an effective strategy, but requires the use of interactive techniques that engage the learner. Self-directed learning has the additional advantage of allowing learners to study at their own pace, select times convenient for them and tailor learning to their specific needs.
Limited evidence was found to support team-based learning or the provision of training in work teams. There is a need for further study in this area, given the value of engaging teams that are in the same place at the same time in an in-service training intervention. This finding is especially relevant for emergency skills that require the collaboration and cooperation of a team.
Repetitive exposure is supported in the literature. When possible, replace single-event frequency with targeted, repetitive training that provides reinforcement of important messages, opportunities to practice skills and mechanisms for fostering interaction. Recommendations drawn from the educational psychology literature that address the issue of cognitive overload [56] suggest targeting information to essentials and repetition.
Select the setting based on its ability to deliver effective educational techniques, be similar to the work environment and allow for practice and feedback. In this time of crisis, workplace learning that reduces absenteeism and supports individualized learning is critical. Conclusions from literature in educational psychology reinforce the importance of 'situating' learning to make the experience as similar to the workplace as possible [57].
Certain common themes emerged from the many articles that commented on the role of media in CPE effectiveness. A number of systematic reviews suggest the use of multimedia in CPE. It is important to note that the studies that found similar knowledge outcomes between computerbased and live instruction stated that both utilized interactive techniques, possibly indicating the effectiveness was due to the technique rather than the media through which it was delivered. While the data on use of mobile technology to deliver CPE were limited, the study by Zurovac et al. indicated the potential power of mobile technology to improve provider adherence to clinical protocols [29]. Currently, there is unprecedented access to basic mobile technology and increasing access to lower-cost tablets and computers. The use of these devices to deliver effective techniques warrants exploration and evaluation, particularly in low-and middle-income countries.
CPE can positively impact desired learning outcomes if effective techniques are used. There are, however, very limited and weak data that directly link CPE to improved clinical practice outcomes. There are also limited data that link CPE to improved clinical practice behaviours, which may influence the strength of the linkage to outcomes.

Limitations
The following limitations apply to the methodology that we selected for this study. An integrative review of the literature was selected because the majority of published studies of education and training in low-and middleresource countries did not meet the parameters required of a more rigorous systematic review or meta-analysis. The major limitation of integrative reviews is the potential for bias from their inclusion of non-peer-reviewed information or lower-quality studies. The inclusion of articles representing a range of rigor in their research design restricts the degree of confidence that can be placed on interpretations drawn by the authors of those articles, with the exception of original articles that explicitly discussed quality (such as systematic reviews). This review did not make an additional attempt to reanalyse or combine primary data.
Therefore, for purpose of this article, we also graded all articles and included only tier 1 articles in the analysis. This resulted in restriction of information on certain topics for this report, although a wider range of information is available.
We faced an additional limitation in that many articles included in the review were neither fully transparent nor consistent with terminology definitions used in other reports. This is due in part to the fact that we went beyond the bio-medical literature, to include studies conducted in the education and educational psychology literature, as was appropriate to the integrative review methodology. Certain topics were underdeveloped in the literature, which limits the interpretation that can be drawn on these topics. Other topics are addressed in studies conducted using lower-tier research methodologies (for example observational and/or qualitative studies) that were not included in this article. In addition, the overwhelming majority of studies focused on health professionals in developed or middle-income countries. There were very few articles of sufficient rigor conducted in low-and middle-income countries. This limits what we can say regarding the application of these findings among health workers of a lower educational level and in lower-resourced communities.

Conclusions
In-service training has been and will remain a significant investment in developing and maintaining essential competencies required for optimal public health in all global service settings. Regrettably, in spite of major investments, we have limited evidence about the effectiveness of the techniques commonly applied across countries, regardless of level of resource.
Nevertheless, all in-service training, wherever delivered, must be evidence-based. As stated in Bloom's systematic review, 'Didactic techniques and providing printed materials alone clustered in the range of no to low effects, whereas all interactive programmes exhibited mostly moderate to high beneficial effect. … The most commonly used techniques, thus, generally were found to have the least benefit' [14]. The profusion of mobile technology and increased access to technology present an opportunity to deliver in-service training in many new ways. Given current gaps in high-quality evidence from low-and middleincome countries, the future educational research agenda must include well-constructed evaluations of effective, cost-effective and culturally appropriate combinations of technique, setting, frequency and media, developed for and tested among all levels of health workers in low-and middle-income countries. review and coding, and contributed to selected sections of the manuscript. All authors read and approved the final manuscript.