Setting and participants
The study took place in Mukono, a district in central Uganda, from 15 November 2016 to 15 December 2016 [16]. Mukono has a population of just under 600 000, with an estimated under-five population of 30% [17]. Regarding health indicators, Mukono has extremely poor neonatal mortality and malnutrition indicators [18]. In 2011, pneumonia was the leading cause of death amongst children under the age of five in the district, with a case fatality rate of 5 % [18]. This study was implemented by Omni Med, an NGO based in the United States and Uganda that has been training CHWs in Mukono District since 2009 and adapting the iCCM strategy with Ministry of Health approval since 2012 (http://www.omnimed.org/).
CHWs were recruited from three sub-counties in the district: Mpatta, Nakisunga and Mpunge. These areas were selected as locations since CHWs in all three sub-counties had received initial training from Omni Med staff but had not received any iCCM refresher training in the past 2 years. CHW supervisors in individual parishes were informed of the study by a formal letter delivered by the staff of Omni Med. CHW supervisors then had 14 days to notify the CHWs in their parish about the training.
In order to be eligible, CHWs had to have completed basic iCCM training but not have received any refresher training on the pneumonia component of iCCM in the preceding 2 years. There were no exclusion criteria based on age, gender, sex or tribe. Participants were assured of the right to accept or refuse to take part in the study without consequences. Those who agreed to take part signed an informed consent form, which was verbally translated into Luganda from English.
Materials
The tablet used in the study was a $50 USD Amazon Kindle Fire 7 (8 GB). The tablets have a 7-in. screen and operate using an Android platform. Each tablet was housed in a “ruggedised” rubber carry case ($16 USD) to help prevent damage and issued with a solar charger ($15 USD), allowing individual CHWs the ability to charge the battery on their tablet device independent of having access to a source of mains electricity. This model was selected due to its relatively low cost, its expandable memory capacity of 256 GB which enabled us to store and host multiple high-quality instructional videos independent of internet access and high consumer reliability ratings.
Four instructional videos were created by Omni Med staff in July 2016 using up to date iCCM guidelines [19]. The videos were filmed in Mukono in the local dialect (Luganda) with English subtitles and featured local CHWs and a Ugandan physician (KK). The videos focused on (i) What is pneumonia? (ii) How to recognise pneumonia? (iii) How to treat pneumonia? and (iv) How to prevent pneumonia? [19]. The video tutorials were reviewed and approved by the Commissioner for Child Health at the MoH and two independent medical doctors for accuracy, before being preloaded onto the tablets.
Study design
This was a pilot randomised controlled trial carried out over the period of 1-month. We adhered to the UNICEF “Principles for Innovation and Technology in Development” [20]. The unit of randomisation was the sub-county level, and the unit of analysis was at the level of the individual CHW. CHWs were randomly allocated to one of two groups (intervention or control) at a sub-county level. Sub-counties were assigned a number between one and three and a random number generator with the parameters of one to three was used. It was decided the first number to be drawn would be allocated as the control group and the next two as the intervention. Mpatta was assigned as the control arm and Mpunge and Nakisunga as the intervention arms. For further information regarding the control and intervention groups, please refer to the CONSORT diagram (see Additional file 1). Due to the nature of the intervention, CHWs could not be blinded to their group assignment.
Standard iCCM training covering all aspects of diarrhoea, malaria and pneumonia is traditionally carried out by trained instructors in didactic training modules over a period of 5 days. In this study, CHWs in the control group received didactic training delivered by Omni Med staff on the recognition, treatment and prevention of pneumonia in 1 day. Prior to commencing training, all CHWs completed a multiple choice questionnaire (MCQ) assessment. The training session was held in a classroom, and CHWs returned 6 days later to complete a post-training MCQ and practical assessment.
The CHWs allocated to the intervention group received training using the low-cost tablet devices with pre-loaded instructional videos. They attended a half-day workshop on a Monday, where an Omni Med staff member provided basic training on how to use the tablet and CHWs completed the same pre-training MCQ assessment as the control group. Upon returning the tablet after 5 days, CHWs completed the post-training MCQ and practical assessment, as well as a questionnaire regarding their experience of the program.
Both intervention and control groups received a reimbursement of approximately $5 USD for travel costs.
Sample size calculation
We sought to detect an effect size of approximately 0.6 standard deviations (2 points on a test with a possible maximum score of 24 (8.3%), using a pooled standard deviation of changes in scores on a similar test used in a small, unpublished feasibility study and which was also marked out of 24 was 3.4). We sought to detect a larger effect size than the 0.4 standard deviations considered the minimum important difference for educational interventions in Hattie’s seminal work because of the resource limitations for any form of on-going training for CHWs and, indeed, for CHW programmes in general [21, 22].
The sample size calculation was generated using SAS software, Version 9.1.3 of the SAS System for Unix [23]. It was determined we would require 39 CHWs in each arm, to detect a difference of this magnitude, assuming a one-sided test with 80% power and a type I error rate of 5%. To allow for an estimated 25% dropout rate, we initially aimed to recruit 50 CHWs per arm in this pilot trial.
Use of structured knowledge and clinical examination mark schemes
CHWs were assessed using a dual assessment. To assess theoretical knowledge before and after training, a MCQ with 24 questions was used (see Additional file 2). The questions in the MCQ test were developed using the iCCM syllabus and translated into Luganda. The questions on the post-training MCQ were the same as those asked in the pre-training MCQ to facilitate comparison of performance. CHWs were given 60 min to complete each MCQ.
To assess levels of practical and clinical knowledge acquisition after receiving training, we developed a clinical testing tool, utilising seven clinical cases (see Additional file 3). The cases consisted of a short clinical vignette, accompanied by a video, which contained a child under the age of five who either had normal or fast breathing. CHWs worked through the cases under the supervision of an independent assessor who was blinded to group assignment. Based on their assessment of the child’s respiratory rate and other clinical information contained within the cases, they answered between two and four questions per case regarding how they would manage the child. The answers given by the CHWs were recorded on a mark scheme by the independent assessor. CHWs were only assessed using the clinical assessment post-training.
MCQs and clinical case answer sheets were marked by two members of Omni Med staff who were blinded to group assignment using an answer key. This was double-checked for accuracy by an independent marker to verify the score. A score of 1 was given for each correct answer, and 0 for incorrect answers. There was no negative marking in either test. The responses for the theoretical knowledge MCQ tests and post training practical tests were uploaded to an Excel spread sheet and stored electronically for subsequent analysis.
End user feedback
To elicit end user feedback from the CHWs regarding their views and experiences of tablet-based training, we administered a post-intervention questionnaire containing open-ended questions (see Additional file 4).
Outcomes
The primary feasibility assessment outcome was knowledge acquisition and retention as defined by changes in MCQ scores (post-training score to pre-training score) and post-training clinical test scores for both the intervention and control groups. The secondary outcome measures assessed the experiences of CHWs using the tablet-based form of training.
Statistical analysis
The primary analysis was per protocol. Statistical analysis was conducted using RStudio [24] and Stata v14 [25]. To compare the characteristics of the control and intervention groups, we used a chi-squared test with Yates’ continuity correction and two-sided two-sample t tests depending on the nature of the data.
Changes in MCQ scores were approximately normally distributed in both the control (W = 0.99, p = 0.74) and intervention groups (W = 0.99, p = 0.96) as determined by using a Shapiro-Wilk test. Differences in these outcomes between groups were analysed using one-sided independent samples t test. A one-sided test was used because we did not expect the intervention group to have inferior outcomes. Similarly, post-training clinical case test scores were also approximately normally distributed within the control (W = 0.97, p = 0.17) and intervention (W = 0.98, p = 0.35) groups, and thus the difference between groups was also analysed using a one-sided independent samples t test. To estimate the effect of participant demographics (age, gender, years as a CHW, number of children under the age of five, and years of education) on the change in MCQ scores, we undertook a univariate analysis for each demographic for each group using an appropriate statistical test. Any demographic variables statistically significant at p < 0.05 in either group were included in a single multiple linear regression, including group assignment and an interaction term between group assignment and the demographic variable to determine if the effect of the variable was consistent across groups. We also calculated the Pearson’s correlation between pre-training MCQ scores and MCQ change scores to determine if pre-training scores influenced propensity to improve. A p value of < 0.05 was considered statistically significant.
Analysis of end-user feedback
All of the post-intervention questionnaires were collated, and frequency counts were used for closed questions, while open-ended question responses were analysed using thematic analysis. The analysis was performed through coding each individual sentence written by the CHW to look for meaningful patterns across the data. This was done by two authors (JOD and MC) who worked together to generate initial codes and subsequent themes. The phases of analysis were familiarisation with the data, generation of initial codes, searching for themes amongst codes, reviewing of themes, appropriately defining and naming themes, and writing up findings.