A conceptual framework for measuring community health workforce performance within primary health care systems

Abstract

Background

With the 40th anniversary of the Declaration of Alma-Ata, a global effort is underway to re-focus on strengthening primary health care systems, with emphasis on leveraging community health workers (CHWs) towards the goal of achieving universal health coverage for all. Institutionalizing effective, sustainable community health systems is currently limited by a lack of standard metrics for measuring CHW performance and the systems they work within. Developed through iterative consultations, supported by the Bill & Melinda Gates Foundation and in partnership with USAID and UNICEF, this paper details a framework, list of indicators, and measurement considerations for monitoring CHW performance in low- and middle-income countries.

Methods

A review of peer-reviewed articles, reports, and global data collection tools was conducted to identify key measurement domains in monitoring CHW performance. Three consultations were successively convened with global stakeholders, community health implementers, advocates, measurement experts, and Ministry of Health representatives using a modified Delphi approach to build consensus on priority indicators. During this process, a structured, web-based survey was administered to identify the importance and value of specific measurement domains, sub-domains, and indicators determined through the literature reviews and initial stakeholder consultations. Indicators with more than 75% support from participants were further refined with expert qualitative input.

Results

Twenty-one sub-domains for measurement were identified including measurement of incentives for CHWs, supervision and performance appraisal, data use, data reporting, service delivery, quality of services, CHW absenteeism and attrition, community use of services, experience of services, referral/counter-referral, credibility/trust, and programmatic costs. Forty-six indicators were agreed upon to measure the sub-domains. In the absence of complete population enumeration and digitized health information systems, the quality of metrics to monitor CHW programs is limited.

Conclusions

Better data collection approaches at the community level are needed to strengthen management of CHW programs and community health systems. The proposed list of metrics balances exhaustive and pragmatic measurement of CHW performance within primary healthcare systems. Adoption of the proposed framework and associated indicators by CHW program implementors may improve programmatic effectiveness, strengthen their accountability to national community health systems, drive programmatic quality improvement, and plausibly improve the impact of these programs.

Background

The landmark 1978 Declaration of Alma-Ata formally entrenched for the first time a political declaration that enshrined health as a human right within the global health agenda and emphasized primary health care (PHC) as a key mechanism for achieving health for all [1]. The 40th anniversary of the Declaration of Alma-Ata was a global call to re-invigorate efforts to achieve health for all through strengthening PHC systems [2, 3]. Mounting evidence since 1978 in low-and-middle-income countries (LMICs) has shown that focusing on provision of health services at the community level not only leads to more efficient and equitable use of health resources and better health outcomes [4], but also is a consistent component of strong, effective health systems. Since Alma-Ata, community health worker (CHW) programs in LMICs have been promoted to strengthen broad-based health efforts within community settings. CHWs, who for the purposes of this paper, refer to frontline health workers with up to 6 months of initial training, serve as the first point of contact for community members, especially for individuals living in low-income or rural communities whose access to facility-based health care may be limited. Often as community members themselves, CHWs possess a unique understanding of the local context, including barriers and facilitators to accessing timely and quality PHC, and can facilitate the most effective linkages to care.

Despite decades of interest and renewed commitment to expand Universal Health Coverage (UHC) by scaling up CHW programs, a universal, standardized system for empirically measuring the effectiveness of CHW programs does not yet exist [5,6,7]. Measurement is limited by the lack of an accepted and pragmatic set of theoretically grounded and validated indicators [8]. The integrated Community Case Management (iCCM) Framework was an attempt to provide guidance on measurement and metrics of iCCM programs; however, it has few metrics measured at the community level [9]. There is a need for consensus on measuring CHWs’ performance as a part of the larger community health system, broadly defined as “set of local actors, relationships, and processes engaged in producing, advocating for, and supporting health in communities and households outside of, but existing in relationship to, formal health structures” [10,11,12,13]. In practice, community health systems may include the enabling environment in which CHWs work, including household-level caregivers, other formal and informal healthcare providers, organizational intermediaries including non-governmental and faith-based organizations, other government sectors such as housing and education, among others [11]. As the link between individuals, communities, and health facilities, CHWs must be fluent in navigating the formal, facility-based health system, while maintaining their relationships within the community where they work [11, 14]. Consequently, the measurement of CHW performance needs to account for not only the activities of the CHW, but also those of the community health system that support the CHW. Experts have called for the recognition of the community health system as its own, unique sub-system of the health system, articulating a need to standardize the way performance and success of this sub-system is measured [11].

Addressing this gap in standardized metrics for assessing performance of community health systems is one of the main goals of Population Council’s Frontline Health (FLH) project, supported by the Bill & Melinda Gates Foundation and implemented in partnership with Last Mile Health. USAID, UNICEF, and the Bill & Melinda Gates Foundation are working together around a set of jointly defined investment priorities that advance frontline delivery of and community engagement in primary health care. The Integrating Community Health (ICH) collaboration focuses on catalytic partnerships for system strengthening; measurement, learning, evaluation, and accountability; and, advocacy and pathways to scale. FLH is a partnership of the Population Council and Last Mile Health with USAID, UNICEF, and ICH partners in Bangladesh, Democratic Republic of the Congo, Haiti, Kenya, Liberia, Mali, and Uganda.

This partnership builds on the momentum set forth by the Kampala Declaration and the Agenda for Global Action [15] for higher commitment by governments and development partners to strengthen health workforce, and supports the recently released WHO guideline on health policy and system support to optimize CHW programs through the generation of tools and best practices to improve the design, implementation, performance, and evaluation of CHW programs [16]. The Frontline Health project interacts with country and global stakeholders to contribute to the objective of advancing metrics and evidence for community health. As part of this effort, key learning and research priorities for countries considering greater institutionalization and professionalization of CHW programs have already been identified in another manuscript [17].

Here, we distill lessons from the literature and expert consultations to propose the Community Health Worker Performance Measurement Framework. The goal of this framework is to guide governments and implementing agencies in the development of priority standardized metrics for measuring the performance of CHW programs within the context of the broader system within which they operate. In this article, we describe the process of developing the framework and associated metrics, identify considerations for measurement of CHW program performance, and articulate future considerations for developing a robust agenda for monitoring community health systems at the country level.

Methods

Review of evidence

Community health literature from peer-reviewed databases and online knowledge management centers (CHW Central and Human Resources for Health (HRH) Global Resource Center), as well as publications from multilateral organizations, were reviewed to identify broad measurement domains in CHW program monitoring. A total of 85 reports from the CHW Central and 300 from the HRH Global Resource Center were screened for relevance. Peer-reviewed articles were identified using a convenience snowball sampling and included if they identified critical measurement gaps to improve CHW programs. This information was abstracted and consolidated to inform the measurement domains for the framework. At first, the review was aimed at identifying a framework in the existing literature that would comprehensively identify priority metrics for measuring CHW programs, instead of developing a new framework. We identified 34 frameworks (Box 1) that peripherally addressed this objective, including but not limited to the CHW Assessment and Improvement Matrix (CHW AIM) [18], CHW logic model proposed by Naimoli et al [6], USAID Community Health Framework [19], and the Primary Health Care Performance Initiative (PHCPI) conceptual framework [20]. While these frameworks were useful in identifying areas of measurement appropriate to community health, they were not explicitly developed for operationalizing measurement of CHW programs [6, 18,19,20]. Given this, a new framework was proposed that leverages these existing frameworks and is pragmatically geared towards monitoring community health worker programs.

The review of literature and existing frameworks helped identify and define sub-domains of a draft Community Health Worker Performance Measurement Framework. This was refined further based on a series of consultations as described below (Fig. 1).

Fig. 1

Development of the Community Health Worker Performance Measurement Framework

Concept development

Four critical considerations guided the development of the framework:

• 1. Measurement of CHW performance should consider system-level processes including the role of governance, policy, internal and external investments, and other supportive mechanisms, and account for the broader community health system that influences CHW program performance.

• 2. In countries without centralized coordination, NGOs typically play an outsized role in implementation of CHW programs, with divergent CHW roles, responsibilities, and timelines. To account for this variation, we recommend the use of existing standardized metrics (e.g., for iCCM, HIV, TB, family planning, maternal health) to measure downstream impacts of services on a specific outcome area.

• 3. While several factors contribute to the success of CHW programs, the proposed indicators around the framework focus on aspects that are amenable to and critical for the purposes of measurement of CHW performance. For multidimensional and critical concepts such as community empowerment and job satisfaction, formative work is currently being undertaken by the Population Council within the Frontline Health project to develop scale-based metrics.

• 4. Balance short-term demand for data to inform programmatic activities with the long-term data needed to understand program effectiveness.

Operationalizing measurement constructs as indicators

To operationalize the framework sub-domains, the monitoring, evaluation, and learning (MEL) plans of the seven ICH NGO partners were reviewed to identify pragmatic indicators across a range of contexts [21,22,23,24,25,26,27]. These indicators had been operationalized by the implementing NGOs, with varying levels of success across a wide range of contexts. The ICH NGO partners’ proposed indicators were mapped on the frameworks’ sub-domains. Where appropriate, the language was revised to generalize the indicators, and new measurement sub-domains were added or combined. Next, indicators in the peer-reviewed literature were identified from 383 reports in the Health Care Provider Performance Review database using the filters of “lay health worker” and processes of care outcomes [28]. A desk review of global data collection tools was conducted to identify existing global indicators that evaluate CHW performance. Reviewed sources are as follows: WHO National Health Workforce Accounts [29], Woman’s and Man’s Questionnaires from Demographic and Health Surveys (DHS) [30], DHS Service Provision Assessments (SPA) [31], WHO Service Availability and Readiness Assessment (SARA) [32], World Bank Service Delivery Indicators (SDI) [33], CHW Assessment and Improvement Matrix (CHW AIM) [18], Countdown to 2030 [34], WHO European Health for All [35], OECD Health Care Quality Indicators [36], SDG Goal 3 Indicators [37], WHO Global Reference List of 100 Core Health Indicators [38], and Integrated Community Case Management (iCCM) indicators [9]. Each of these sources was systematically searched for its inclusion of indicators that explicitly measured CHW performance at the community level.

Consultations and prioritization of metrics

Three key consultations were held to align findings from the literature with expert opinion and stakeholder perspectives. The first two consultations followed a modified Delphi approach [39]. In May 2018, representatives of the seven ICH NGO partners and the Ministry of Health of each ICH country met in Johannesburg, South Africa. This group (n = 29) worked to prioritize measurement domain areas for the framework, clarify definition of each domain and identify a minimum set of feasible and efficient indicators as well as to document challenges in measurement from the perspective of implementers and policymakers [40]. Feedback solicited during the workshop was incorporated in a revised version of the framework. A Technical Advisory Group (TAG) of 25 distinct participants was then convened in Washington, DC, United States of America, to identify priority areas for measurement from the perspective of global stakeholders, community health system advocates, and measurement experts. During the TAG meeting, a structured web-based survey was administered to TAG members, as well as select members of the Frontline Health Workers Coalition who volunteered to participate. For each of the measurement sub-domains, the respondents were asked to determine its importance (response categories—yes, no, maybe). For each of the indicators under the sub-domains, respondents were asked whether the indicator was valuable (response categories—yes, no, unclear, unsure). Sub-domains that had greater than 75% support as important areas of measurement, were further discussed in plenary, and the associated indicators further refined. For sub-domains with less than 75% agreement on their level of importance, changes were made based on expert feedback alone. Results from the survey are presented in Additional file 1.

As a final step, in October 2018, a consultation was held in Liverpool, United Kingdom, during a side session of the Fifth Global Symposium on Health Systems Research. A meeting of a third set of 32 researchers, program implementers, and policy stakeholders, focused on community health systems, met to further validate the framework and indicators by confirming their priority areas for measurement and the relevant metrics through a nominal group technique process [41, 42]. Following this process, participants were asked in a group discussion format to adopt the lenses of (a) donors/international policymakers, (b) national/sub-national policymakers and managers, (c) program/service implementers, and (d) monitoring/evaluation specialists and researchers and prioritize metrics within the framework based on their perspective. Qualitative notes taken during this discussion, including priority areas and indicators documented during nominal group technique process and perspective-based presentations, were used to further clarify the indicators, refine the definitions, and contextualize them.

Results

Community Health Worker Performance Measurement Framework

The Community Health Worker Performance Measurement Framework (Fig. 2), derived from iterative framework and indicator review and consultation, identifies critical areas for measuring the performance of CHW programs within their community health systems [43, 44]. While community health systems are inherently non-linear and complex, the framework structure uses the common input-process-output-outcome logic model approach and has four areas: inputs, programmatic processes, community health performance outputs (measured at the CHW level and at the community level), and outcomes [45]. Specific measurement domains and sub-domains are defined under each of these categories, with operational definitions in Table 1.

Fig. 2

Community Health Worker Performance Measurement Framework

Table 1 Operational definitions of measurement constructs in the Community Health Worker Performance Measurement Framework

The sections below detail measurement considerations, approaches for disaggregation, and data challenges for each of the programmatic process and performance output sub-domains under process and outputs in the framework, as well as recommendations for framing each measure from a gender and equity lens derived from the consultative and review process.

Table 2 identifies indicators for each of the measurement sub-domains and classifies them by their utility (high or moderate) for managers at the facility level, district level, and regional/national level, as well as suggests likely data sources. Inputs at the level of policy, governance, logistics, funding, and information management systems tend to have a large amount of variability in what might be appropriate to measure, given the country context, the maturity, the diversity of service delivery programs, and the degree of health system integration of the CHW program—therefore, specific indicators at this level should be reflective of what is most appropriate to understand the fidelity of program inputs. Given that CHW programs may focus on a wide range of context-appropriate health interventions, measurement of health outcomes attributable to the CHW program should also align with nationally or internationally recommended standardized coverage and impact health indicators.

Table 2 Illustrative indicators for community health worker performance measurement

Based on feedback from TAG members, the following areas had greater than 75% agreement as the most critical areas of measurement: attrition (100%), quality of services (95%), service delivery (95%), supervision (95%), experience of services (90%), use of services (90%), referral/counter-referral (85%), CHW absenteeism (80%), incentives (80%), performance appraisal (80%), credibility/trust (75%), data reporting (75%), data use (75%), and costs. While the measurement of costs, cost-benefit, and cost-effectiveness of CHW was unanimously considered vital, no specific indicators were recommended due to the variability in the functions of such programs. Measures around supervision and performance appraisal were combined due to the overlap in the associated metrics.

Measurement considerations for core framework constructs

Supportive systems

• A. Supervision and Performance Appraisal: Provision of routine support to CHWs is important for problem solving, skills development, motivation, and quality service delivery. An important existing gap is the frequent lack of contact with supervisors or former mentors at training institutions once the CHW graduates and is posted, especially in rural areas [53]. The measurement of the assignment and provision of supervisory visits is important and associated data might be routinely collected in training logs, CHW, and supervision reports. CHW performance appraisal and feedback from supervisors are key components of quality supervision and require assessment as well, possibly in a non-routine study. Alternately, digital and mobile job aids with algorithms and behavioral analytics could help operationalize quality assessments and provide remote supervision. The Perceived Supervision Scale (PSS) is a six-item scale that captures regular contact, two-way communication as well as joint-problem-solving, and has been validated in six countries [54]. We recommend the use and refinement of this measure across different contexts or, alternately, testing a modified PSS such that it captures elements of perceived value of CHW supervision, and the content and alignment of the supervisory visit with protocols. Alternatively, the quality of supervision may also be assessed through a spot check, where supervisors are accompanied by their managers on their supervisory visits, and feedback on the quality of supervision is provided after the visit based on observation.

• B. Data use: The use of data by health workers at all levels of the health system can help with responsive feedback and quality improvement efforts. If data are being routinely collected and reported, it does not automatically imply that they are being used for decision-making. Assessment of data use may require special cross-sectional and ethnographic studies, to understand whether data are being used for decision-making and assess reasons for data use/disuse behaviors. Inclusion of this metric in the framework also highlights the need to have management structures in place such that those that are collecting the data, especially client data, also have access to that information [46].

CHW development

• A. Recruitment: Several aspects of measurement of CHW recruitment are important, including understanding how many CHWs are recruited, their density per geographic area (e.g., by district, region), and disaggregating recruitment by the representation of CHWs from different types of communities (e.g., diverse ethnicities, socioeconomic backgrounds, and originating from that community, among others). At the national and sub-national levels, the number of CHWs that have been recruited need to be monitored for fiscal planning and ensuring adequate coverage. From a measurement perspective, it is important to assess the alignment between national CHW selection policies and the actual recruitment and to recognize information systems’ capacity to capture workforce turnover data. In some cases, education, language, or other requirements may limit recruitment of CHWs from marginalized communities [47]. In addition to the actual number of CHWs who have been recruited, it is also important to understand coverage, and quality and transparency of the recruitment process. Community participation in the recruitment process can facilitate community ownership and validation of CHWs within the communities where they work.

• B. Training: Initial and ongoing knowledge and skills-based training is critical to prepare CHWs for their role in service delivery. Records on the numbers of CHWs trained are typically maintained at the program, or district level at which training occurs, and might be further aggregated at a sub-national level to assess alignment with national targets. Maintaining training records at the program or district level can be a challenge as these trainings are often led by external NGOs who may be equally challenged by information systems limitations to feasibly transfer or maintain training data. Identifying mechanisms to consolidate the number of CHWs trained and maintaining rosters of the topic areas on which training has been received is important to scaling programs and may require additional district-level oversight of data management. If a national certification program exists for CHWs, tracking the number of existing and new CHWs who have completed certification is important. Finally, ensuring that measurement of training comprises not just the technical content, but also the procedures around data entry and reporting is crucial. While the number of trainings held is somewhat routinely tracked, the actual competencies of CHWs are more complex to assess. Trainings may capture pre- and post-test knowledge scores, but efforts to assess the extent to which CHWs effectively apply their newly acquired skills are less frequently undertaken [50].

• C. Incentives: While measurement of financial and non-financial incentives provided to CHWs is important, it is also a difficult area to routinely monitor. Where CHWs receive a pre-determined stipend, salary, or performance-based incentives, the receipt of these stipends might be routinely monitored through monthly/quarterly CHW or supervisor reports. Often, if CHW stipends are provided through supervisors or other personnel in the healthcare system, having routine measurement of whether the stipend was received can aid in curbing corrupt practices or mismanagement of funds. Non-financial incentives may come in many forms—they may be institutionalized by governments in the form of educational or promotional opportunities, or by communities in the form of social recognition of the CHW [17, 53, 55]. If an institutionalized non-financial incentive exists, it might be feasible to routinely measure it. If not, programs may consider identifying and disaggregating non-financial incentives that exist at the community level and assess them cross-sectionally.

Support from community-based groups

• A. CHWs operate within the complex interplay of community and health systems. Engagement with and support from the communities in which they operate is critical to optimal functioning of CHWs, as also highlighted by the U.S. Government Evidence Summit [5, 16, 51, 56, 57]. It is important for communities to engage through a feedback mechanism, such as a community scorecard, with which they can assess challenges the community faces that can be used to provide feedback and targeted support to CHWs. Community-based groups such as village health committees, facility management committees, or other contextually appropriate local governance structures, including non-health sector groups, can play a role in identifying solutions to challenges. These community mechanisms are complex to measure routinely or in a binary format, and may be best measured through cross-sectional, intermittent, qualitative assessments.

CHW competency

• A. Knowledge on specific technical subject matter underpins quality of service provision. Measurement of CHW knowledge may be considered at the end of each training cycle, or cross-sectionally, where new knowledge is being imparted and skills developed. In lieu of metrics to capture CHW knowledge, assessment of the actual performance of the CHW is more important, as knowledge does not always translate to practice, known as the “know-do” gap [58]. If the performance of CHWs does not meet required standards, an assessment of knowledge might be relevant if it is suspected that poor knowledge is the cause of poor performance.

• B. Service delivery: The delivery of health services refers to the quantity of promotional, preventive, curative, and rehabilitative services CHWs provide to community members. Indicators under this domain pertain to the activity of the CHW, measured at the level of the individual CHW. CHW activities might be consolidated from CHW service delivery registers and routine reports. Indicators to assess CHW service delivery should be tailored to the individual CHW program context and the types of services provided by the CHW. The illustrative indicators in Table 2 may be further broken down by type of services for which the home visit was made. Note that measures of service coverage at the community level typically involve population-based surveys and are listed under “use of services”.

• C. Service quality: Measurement of the quality of services provided by the CHW is perhaps one of the most critical measures to assess effectiveness of a CHW program, as historically, these programs have lost financial support when service quality does not meet required standards or varies substantially [59]. Service quality should be measured both from the technical or clinical perspective and the client’s perspective (captured under “experience of care”). Unless routine quality assessment and control measures are built into supervisory activities, the measurement of service quality may require non-routine studies. When measured routinely, assessment of quality may be done through spot checks by supervisors to ascertain quality of service delivery using a checklist, or by examining client ledgers or registers. Simple checklists implemented by supervisors to observe and assess existing quality of services or community follow-up tools utilized by community members themselves may provide immediate and comprehensive feedback to the CHW.

  If conducted as a non-routine study, assessment of quality at the community level poses logistical challenges that are not typically experienced when assessing quality at the facility level, where direct observation of client-provider interaction and exit interviews are the normative methods for quality assessment. CHW services vary widely by content area including routine counseling, linkages to health facilities, provision of certain interventions, and early detection and identification of health problems. Measures of CHW service quality should be tailored to specific CHW programs developed around a menu of potential service delivery priority areas. One indicator that is a common measure of quality is the timeliness of service receipt (from the first onset of symptoms).

• D. Data reporting: Refers to the regularity and completeness, including data validity and verification, of reporting on the services CHWs provide at the community or household level [60]. The level of data reporting can skew our understanding of all other progress measures—plausibly CHWs who are more active in their communities are also reporting more regularly and accurately. Differences in types of reporting mechanisms could lead to data that is not easily comparable across settings.

• E. Absenteeism: Refers to the frequency with which CHWs are absent from their routine responsibilities. In settings where CHWs are volunteers or part-time employees, a regular schedule of activities may not exist [61]. This makes assessment of absenteeism challenging. If CHWs are reporting on their activities with some level of regularity, that might be a proxy measure for their regularity in their roles as CHWs. In cases where CHWs are expected to collect and report digitally, such data could be routinely made available on data dashboards.

CHW well-being

• A. Motivation: Refers to intrinsic and extrinsic factors that influence CHWs’ interest in and willingness to perform their jobs. The challenge of measuring motivation through specific indicators stems from its latency and multidimensionality as a construct and demands a scale-based approach. Motivation dimensions previously measured in community and primary health care settings include inertia towards one’s job; external financial rewards; self-worth contingencies including recognition from communities, supervisors, and colleagues; accepting the value of the job; internalizing the job’s value into one’s sense of self; and performing an activity for its own sake [62,63,64]. Motivation is best ascertained by CHW self-report through special surveys. Further investigation of adapted scales can potentially lead to the development of proxy metrics that could be included in routine community-integrated monitoring mechanisms.

• B. Job satisfaction: Refers to a latent construct that captures a CHW’s sense of his or her ability to perform his or her job in a particular work climate [65]. Similar to motivation, job satisfaction is best suited to measurement using a scale-based approach that aggregates the attributes of a CHW’s job, including the job design and shared/participatory decision-making, empowerment, timeliness of decision implementation, availability of material supportive aides, support by communities in which they work, adequate feedback, recognition, management by supervisors, and mutual trust and cooperation among peers. While scale development approaches allow for refining measurement of CHW job satisfaction, and it is recognizably best measured through adapted scales in special surveys, two indicators that emerged as potential proxy metrics are presented in Table 2.

• C. Attrition/retention: In comparison to absenteeism, measurement of attrition aims to capture the proportion of practicing CHWs who resign, retire, or abandon their positions. Given that CHWs may be volunteers, and their “employment” may often not be captured in official rosters, prolonged inactivity may be considered a proxy for attrition, where “prolonged” is a standard length of time determined at the country level, e.g., 1 month, 3 months [47]. The indicators presented in Table 2 are limited by the fact that some CHWs may not report regularly but may continue delivering services and/or intend to continue in their roles. As such, CHWs identified through monitoring this metric should be followed up in person.

Community access

• A. Use of services: Community-level use of services is typically measured through population-based surveys. To the extent possible, these indicators should be aligned with extant national surveys that already capture indicators specific to CHW services. For example, the DHS measures the percentage of pregnant/recently delivered women who were visited by a CHW in the last 12 months, as well as several other health-area-specific indicators focused on use of services offered by CHWs. Table 3 presents a list of these indicators based on extant surveys. To ensure that programmatic CHW indicators are comparable with globally accepted measures, attention should be paid to alignment with these extant metrics.

• B. Knowledge of service availability: Understanding the degree to which the community is aware of the presence and availability of a local health worker and the services they provide is critical to respond to community needs and priorities [5, 66]. Measuring this may require a population-based study.

• C. Referral/counter-referral: Timely and appropriate referral from the community to the health facility is often considered one of the key functions of CHWs. Counter-referral from the health facility back to the CHW facilitates continuity of care in the home context of the client. One of the challenges with measuring referral is identifying the appropriate denominator (i.e., no. of clients eligible for referral per protocol). Such an assessment is not feasible to undertake routinely; therefore, referral may be measured per 100 or “X” clients seen by the CHW, disaggregated by reason for referral and used for comparisons across settings or over time [67, 68]. Measurement of counter-referral may only be pertinent in cases where continuity of care is recommended, as in the care of several chronic diseases. For example, if a child was referred to the facility for treatment of diarrhea, unless it is severe, there may not be a reason for counter-referral. However, if a client is referred to the health facility for suspected hypertension, counter-referral may be more appropriate. Measurement of counter-referral may also not be feasible in the absence of advanced information technology and digital systems that link facility records back to the community.

Table 3 Standard CHW service delivery metrics that are currently measured at the community level in national surveys

Community-centered care

• A. Empowerment: Community members’ agency—awareness of and access to—community health services as well as the participation or engagement of communities in shaping and maintaining community health services (including CHW programs) is a critical potential outcome of CHW programs. Given the multidimensional and highly contextual nature of empowerment, it is a difficult concept to capture through a standardized metric [49, 69]. Though proxies such as numbers of community meetings held, numbers of community members involved in CHW and facility feedback, and community health system-level contributions emerged in our discussions, these fall short given the varied definitions of empowerment and community-integrated governance structures across countries. Rather, cross-sectional special surveys alongside qualitative methods that describe the process, content, and relational dynamics (e.g., of community meetings) are required to assess empowerment. Adapting scales of individual, organizational, and community empowerment that have been developed in the context of health promotion can serve as a starting point to develop composite measures of empowerment as related to specific community health system components [70].

• B. Experience of care: Understanding and measuring the experience of care from the client’s perspective is critical, but complex to standardize in the context of community health systems. This construct often overlaps with service use and technical service quality for a particular health area, though offers the subjective understanding of the interaction between client’s interactions with a CHW. Table 2 recommends that experience of care is measured through two indicators: levels of satisfaction with services received by a CHW in the last 3 months and the client perception of respectful care. Experience of care can be captured through self-report and, to some extent, observation by other community members or a trained data collector, close in timeframe to the service delivery (e.g., within 3 months) to avoid recall bias [71]. Composite measures to capture aspects of experience of care should be developed, validated, and incorporated into supervisory visits.

• C. Credibility/trust of CHW: The degree to which clients consider the CHWs, and ultimately the services provided by CHWs, to be credible and reliable impacts the use of CHW services. While client trust in health information and services provided by the CHW are presented as proxy metrics in Table 2, these are not validated nor reflect the multidimensionality of trust within CHW-client relationships (includes dimensions of honesty, confidentiality, competency, and mutual respect, and partnership in health care decision-making processes) [72]. Quantitative measures of trust in CHWs should be adapted from PHC facility settings, contextualized to health areas and socio-cultural contexts through consensus-building methods, and evaluated through special surveys and scale-based approaches that are best suited to capturing latent constructs like trust [72]. Based on these studies, more realistic and relevant proxies can be investigated and recommended.

Measurement considerations for equity, gender, and accountability (see Table 2)

Equity, gender, and accountability are critical aspects to consider across most measurement domains [66, 73, 74] that require various qualitative and quantitative measurement methods from an array of stakeholder perspectives. Two primary concepts around equity of community health systems warrant discussion—first, the equitable selection and access to training and growth opportunities for CHWs themselves; second, the activities and inherent biases of the CHW and the health system differentially affect use and quality of services received by different community groups. When feasible, metrics should be disaggregated by place of residence, socioeconomic status—including education and wealth, sex, age distribution, occupation, social capital, language, religion, ethnicity, tribe, family type, health risk-level, village-size, marital status, immigration/migrant status, and refugee/asylee status (Table 2 footnotes). Disaggregation by equity-promoting variables requires a substantial commitment to advancing equity and utilizing accompanying resources. As a starting point, three levels of disaggregation, in alignment with the DHS might be considered—by wealth quintile, by level of education, and by place of residence (i.e., urban and rural) [75]. For allocation of resources at a national or sub-national level to the districts, performance metrics should be available, disaggregated by district to facilitate appropriate allocation of resources. We note here that equity should not be misunderstood for equality—while equality deals with fairness through equal distribution of resources, equity is concerned with need-based resource allocation even if that means unequal distribution [76].

Gender (in) equity may broadly refer to gender-related barriers to CHW performance, any unique needs and protections expressed by CHWs as a result of their gender (e.g., female CHWs may have certain safety concerns in some locations), minority identity in a particular context, and power relations facing CHWs within the health system hierarchy or community governance structures [74, 77]. These power relations relate to the attitudes around and interactions between CHWs, their supervisors, and other health workers or community workers. Measurement considerations might entail understanding the gender, ethnic, or age profiles of the cadre of CHWs and how that might affect perceptions of safety, interaction with the community and other health personnel in the health system, and overall motivation to continue working as a CHW. Beyond the need for capturing disaggregated metrics and needs of CHWs, who occupy lower positions in the health system hierarchy, it is important to consider policy-relevant metrics that support female health workers in particular, such as equitable hiring practices, uncompensated leave, sexual harassment, and discriminatory training policies [78]. Given that CHWs are typically responsible for all members of the community and given the expanding scope of their responsibilities (i.e., beyond traditional RMNCH care to include aging and non-communicable disease management), gender balance of recipients of care from the CHW at the community level is also important to consider.

Accountability in the context of community health systems is multifold, challenging to measure, but highly relevant in understanding how to institutionalize CHW programs and community health systems as agreed on by the TAG and literature [66]. It refers to the dual accountability of the CHW to the community for timely delivery of services and to the health system to perform required tasks, and to the responsiveness of the health system and communities to CHWs, namely by providing a supportive work environment, remuneration for service, and systematic feedback. Two perspectives might be considered around the measurement of accountability: as an outcome, which could be captured by policy-level inputs (e.g., funding integration in national and sub-national budgets and protective regulation around CHW scope) and local administrative government status reports, and as a governance and health system process that includes community participation in planning and review of CHW activities. Across the framework, indicators capture the types of support, remuneration (incentives), and recognition/systematic feedback that a CHW may receive from the government, health systems, and community, as well as the benefits the community receives from the activities of the CHW.

Economic evaluation

Emerging health system needs involve studies on the cost of scale up, particularly returns on investment in community health systems; economic evaluation metrics specific to community health systems better enable justification of health investments at the national level. However, no extant routine or recommended indicators around economic evaluation of CHW programs were identified [50]. Economic-cost-related metrics pose a challenge and require consideration of deaths or complications averted by CHW programs and comparative out-of-pocket cost savings models with respect to preventative care seeking from CHW and facilities. Ideally, economic evaluation metrics would involve various perspectives—health systems, societal, etc.—to reflect savings for households, facilities, or administrative/governments [17, 79].

Discussion

The recommended domains (Fig. 2) and indicators (Table 2) are the result of extensive input from community health systems experts and practitioners in the field. The experts ranged from academics to officials from ministries of health across seven countries. A final consultation with practitioners in Liverpool, United Kingdom, validated the results; in Liverpool, participants debated the strength and value of indicators but did not identify new, actionable metrics. The experts also agreed that these recommendations strike a balance between a pragmatic minimum set of indicators and a more exhaustive set of metrics that would be costly and impractical to measure with regularity. The proposed list of metrics is not comprehensive; rather, it is presented as a point of reference to assist in standardizing metrics for CHWs.

CHWs work in a wide variety of contexts: from institutionalized, salaried roles well-integrated in the public health sector [13], to informal, volunteer-based work acting as important community educators and linkages between communities and the health system. CHWs may also deliver multiple interrelated and interdependent interventions simultaneously. Their services may involve activities across different levels of the health system, including referrals made to the health facilities [54]. This variability in their roles and range of services contributes to the challenge of standardizing metrics to measure the performance of CHW programs and attribute their impact on health outcomes [48].

The appropriate selection of indicators depends on the maturity of the community health system, as well as the types of data that are considered most critical to advocate for such community-based interventions nationally. The maturity of a CHW program can be observed in the degree to which the community health system is integrated with the formal health system (at national, regional, district, and facility levels). In some contexts, CHW programs operate as disparate stand-alone, NGO or private sector-led programs that may run in parallel to the public health system. In more mature settings, CHW programs are formally aligned with well-established government policy, with a formal governance structure, funding support, training agenda, job description, appropriate support from public health facilities, and efforts to integrate care from the community to the health facility. Within the proposed framework, some performance indicators will be more relevant to early CHW initiatives, while others will only become useful as the CHW service delivery layer becomes fully integrated into the health system.

The framework in Fig. 2 articulates measurement domains that should be captured by CHW programs, and the illustrative indicators in Table 2 represent a more pragmatic approach reflective of the limitations in standardized indicators, capacity for data collection and aggregation at the community level, and limited availability of community-based census or “denominator” data. The framework is limited to CHW programs targeted at health outcomes. However, the authors recognize the need for a multisectoral approach at the community level and the potential contributions of CHWs in education, agriculture, and other sectors. Some sub-domains have missing illustrative indicators (e.g., economic evaluation); others have indicators (e.g., referral/counter-referral) where methods of collection are challenging and under-developed resulting in a large amount of variability in the feasibility of measurement; still others have composite metrics (e.g., community empowerment) that are not validated or are multidimensional and highly contextual. Indicators for inputs (existing governance, policy, funding, and information systems) were not included largely due to their high variability across CHW program contexts. The framework and approach does not provide country-specific guidance for actionable use of particular indicators. Country-level adaptation and testing of indicators in practice can overcome this limitation.

These limitations highlight opportunities to strengthen the measurement of community health systems. To facilitate standardization, development and validation of comprehensive indicators is important. Advances on this front are ongoing, as in the case of the recently proposed six-item perceived supervision scale that has been validated in seven languages [54]. In practice, a more limited subset of the proposed metrics that have been validated and found most valuable to global and national decision-making should be identified and disseminated. An example of metrics prioritization is seen in Liberia, where a focus on quality has led to continued optimization of a nationally implemented CHW program. It should be noted that in emergency and fragile settings, CHW programs often emerge organically from the needs of the community. The most pertinent areas of measurement for such settings need to be identified and tested. While we engaged a number of national and global stakeholders in the development of this framework, its ongoing development would benefit from wider perspectives. The framework does not cover the full range of the community health system space, which involves more multisectoral players.

Of critical importance is the need to strengthen community health information systems (CHIS). Where previously, much of the community-level data collection and aggregation occurred manually on paper, the rapid digitization of health service data and data collection using mobile devices has opened new frontiers for enhancing CHIS performance [60]. In advancing primary health care, the 2018 Astana Declaration emphasizes the importance of building systems to collect “disaggregated, high quality data to improve information continuity, disease surveillance, transparency, accountability and monitoring health system performance”, and emphasizes investments into appropriate technology to facilitate this [51]. Several parallel investments are already ongoing. For example, the Health Data Collaborative community data sub-group aims to harmonize and endorse standards for CHIS to maximize integration with national health information systems. At the global level, the National Health Workforce Accounts were proposed at the 69th World Health Assembly and present a set of 78 indicators that can be collected nationally to improve the availability, quality, and use of data on human resources for health [29]. The Primary Health Care Performance Initiative supports countries to measure the most critical indicators to advance primary care [52]. As each of these tools and resources mature, intersections across them need to be explored. For example, the proposed framework and indicators herein can potentially support the assessment of the 10 programmatic components proposed by the most recent version of the CHW AIM Program Functionality Matrix [80]. Given the recent reinvigoration of investments in primary health care and community health systems, coordination across different initiatives to leverage existing work and avoid duplication will also be prudent. Efforts to support the standardization and collection of data must be accompanied by training and support to develop numeracy and skills to use data for decision-making.

Conclusions

The proposed framework and indicators are a critical first step to addressing a long-acknowledged gap in identifying relevant, pragmatic, and contextually appropriate indicators to monitor the performance of CHW programs. Indicators are presented with practical insights and recommendations for routine and special study methods for data collection as well as reflections on integrating CHW performance indicators into routine health information systems. Adoption of the proposed indicators can guide the development of a robust monitoring system for CHW programs, help improve day-to-day programmatic performance, and in the long run have an impact on improved health outcomes. However, the authors emphasize that systems and resources to capture and utilize data at the community level face practical challenges far greater than those experienced in data capture at the level of the facility. We present this framework and indicators to generate a conversation and iteratively develop stronger systems to monitor CHW programs.