Setting
The setting for the impact assessment was the HNECC PHN and general practices across the Hunter, New England, and Central Coast regions in the Australian state of New South Wales who had participated in the MPA Program in 2018 and 2019. The HNECC regions cover an area of approximately 130 000 square kilometres and are home to 1.2 million residents, a quarter of whom live in rural, regional, and remote locations. The combined area has a total of 410 general practices and nine Aboriginal Medical Services; 65 of them participated in the MPA Program in 2018 and 2019 [22].
Evaluation design
FAIT was developed specifically to improve research translation and to optimise and assess the impact from research investments [23]. FAIT has been demonstrated to be an effective impact assessment tool in a range of Health and Medical Research (HMR) projects in Australia and internationally [24, 25]. It combines three validated methods of impact assessment (Payback, Economic analysis, and Narratives) to present a multidimensional, comprehensive approach to assessing the impact of research projects and programs. A detailed description of the FAIT method can be found in Additional file 1. This study represents the first application of FAIT to a non-research program of work.
Program logic model and payback
A detailed program logic model (PLM) was developed to map the pathway between the need for the MPA Program and the eventual impact of the intervention (see Fig. 1). Within FAIT, the PLM underpins all three methods.
Next, a modification of the Payback Framework, first developed by Buxton and Hanney [26], was applied. The Payback Framework captures impact within broad domains of benefit. For the MPA project the relevant domains were selected by the evaluators (SR, RL, AS & SD) in collaboration with PHN staff (AT, NI) to reflect the anticipated benefits of the MPA program from the various perspectives. The selected domains were: knowledge advancement, capacity building, practice change, community benefits and economic impacts. Existing metrics were selected or specifically developed to capture the intended impacts of the MPA Program such as skills development, job satisfaction and workforce productivity.
Economic evaluation
A cost-consequence analysis (CCA) was used to compare costs of the MPA program to its potential consequences for general practices and for MPA students. A CCA does not limit itself to standard metrics like cost–benefit ratios. Rather it lays out the costs and consequences for decision makers to make their own subjective judgements on program cost-efficacy and whether the investment choice represents value for money.
Costs
All resources used for the MPA program were reported in monetary terms (economic costs). This included actual cash costs and the ‘opportunity costs’ of directing resources to the MPA program rather than another activity. A bottom-up approach was used to capture all resources from the perspectives of HNECC PHN, participating practices and MPA students. All resources were identified, valued, and aggregated separately as detailed in Additional file 2. Costs are presented in monetary units so that the value of different resources can be aggregated and compared. Unit costs and cost assumptions are captured in Additional files 2 and 3, and in Table 3.
Consequences
Consequences were focussed on realised and potential benefits for general practices and MPA graduates. These would also indirectly benefit the PHN whose main aim and function is to improve the delivery and quality of primary care, including general practice, within its jurisdiction. The consequences included were limited to those that were monetisable. Consequences not readily monetisable (e.g., job satisfaction and increased confidence in performing clinical tasks) are presented in their natural units and discussed within the Payback metrics (Table 2).
General practices
The main monetisable consequences were efficiencies gained through utilisation of MPA graduates in their upskilled role for half a day, every day; and opportunities for the practice to gain from increases in billable services. The administrative duties performed by the MPA graduate prior to the completion of their course (i.e. undertaking general reception and administrative duties) would be picked up by the Practice Manager or a casual administration assistant. Financial and administrative data to inform on actual level of billable services was unavailable, so modelling was informed by survey data and supplemented with expert input from PHN staff.
MPA graduates
Monetizable consequences for graduates were their potential higher remuneration following course completion. Additional wages were projected over a full year and based on survey responses and an aspirational goal of MPAs being paid $3.00 extra per hour (mid-point between the receptionist and enrolled nurse wage scale). Sensitivity analyses was used to provide the range of potential returns to both general practices and MPA graduates.
Narrative
The narrative was built from the program logic model and impacts and responses from study participants, all of whom provided consent for their qualitative responses to be included in the analysis. Comments were grouped by key themes and captured the various perspectives around impact. The results for the application of FAIT to the MPA Program are summarised and presented in a scorecard format by each method.
Data collection
Data collection methods for each component of FAIT are summarised here:
Payback: Online surveys and interviews with participants and administrative records from the HNECC PHN.
Economics: Online surveys and interviews with participants, administrative records from the HNECC PHN, and secondary data and expert input provided by the MPA Coordinator and Primary Care Improvement Officer from the PHN.
Narrative: Online surveys and interviews with participants.
Participants
Participants represented five groups of general practice staff: (1) practice managers; (2) current MPA students; (3) PN’s; (4) GP’s and (5) MPA graduates. Eligible general practices were those that were listed as having at least one MPA student enrolled between January 2018 and December 2019. Eligible practice managers, PNs and GPs were those who were working at those practices and eligible MPA students were those who were currently enrolled in the MPA Program, at the time of the survey. Eligible MPA graduates were those who had graduated by June 2020. All eligible practices were invited to participate. Of the 62 eligible practices, 36 had one current MPA student (58%) and 26 (42%) had a returned graduate. Of these 26 practices, 4 had their MPA graduate leave their practice by the time of the survey, leaving only 22 eligible MPA graduates. To supplement general practice staff views, the MPA Coordinator and Primary Care Improvement Officer from the PHN, and the Program Manager for Health Programs at UNE Partnerships were consulted due to their experience with the program and capacity to provide contextual clarity.
Surveys and interviews
Online surveys were programmed and deployed using REDCap (Research Electronic Data Capture) a secure web application for building and managing online surveys and databases [28]. Five surveys were developed, one for each participant group. All groups, except for MPA graduates, self-administered the survey. Graduate surveys were administered via telephone interview with a member of the HMRI team who entered the graduate’s answers for them. This was to enable the capture of more detailed responses with regards to the impact of the program on graduates personally. A pragmatic pilot involving two practices from the cohort was undertaken. Suggested changes were incorporated prior to deployment.