Monitoring and evaluation of human resources for health: an international perspective

The importance of sound empirical evidence for informed policy decision-making and the monitoring of progress towards achieving health system goals and human resources for health (HRH) development in particular is widely recognized. Human resources for health are central to managing and delivering health services, and in most countries account for a high proportion of national budgets assigned to the health sector. Defining precisely HRH can help to identify opportunities and constraints and the potential impact of HRH on population health. HRH analysis can also be used as a tool to sensitize political and social stakeholders to the importance of addressing workforce issues, contributing to the consideration of HRH issues on the public and political agenda [1].

Despite the undoubted importance of human resources to the functions of health systems, there is little consistency between countries in how HRH strategies are monitored and evaluated. In terms of health system inputs, international assessments of health personnel or other non-monetary resources remain less widespread than comparisons of health care expenditures [2]. The global evidence on policy options for health systems tends to be weak. It has been acknowledged that no country has discovered an ideal model, and appropriate policies differ widely across country settings [3].

While each country's situation is unique, a comprehensive framework for examining the contribution of HRH to the achievement of health system goals can be a useful tool towards developing evidence-based policy options. The World Health Organization's analytical framework for health systems performance assessment describes the role of HRH in each of four main functions of health systems [4]. While the approaches used to implement this framework have been the subject of much scientific and political controversy [5–7], a general point of consensus is the critical role of HRH in ensuring that services are delivered effectively.

At the same time, despite the high profile accorded to HRH within both the developed and developing worlds, there has tended to be less discourse on broader health policies and the more specific HRH interventions required to achieve them. Often HRH policy and planning have been simply (and mistakenly) equated with training [8]. However, recruitment and retention of HRH may be affected by a number of different factors; for one, poor working conditions may be contributing to both numerical and distributional imbalances in many countries [9]. Assessing HRH impacts on the delivery of health care services should thus also take into account factors exogenous to the health system, including the heterogeneity of labour markets and the mechanisms for adjustment between supply and demand for health personnel.

This paper presents an integrated approach for developing an evidence base on HRH to support decision-making. Drawing on the WHO framework for health systems performance assessment, we focus on the methods and materials for monitoring HRH inputs. First we examine conceptual and methodological issues for selecting relevant indicators. Then we discuss means for enhancing cross-national comparability. We then critically review the main data sources that might be used to generate indicators. Lastly, we touch on statistical applications and present some empirical findings for illustrative purposes.

The WHO framework on health system performance assessment is based on the concept of health action, and encapsulates any set of activities whose primary intent is to maintain or improve population health, enhance the system's responsiveness to the expectations of the population, and assure fairness of financial contributions to the system. To achieve these main goals, four functions are performed by health systems: financing, stewardship, service provision and resource generation (see Fig. 1).

In particular, health system financing refers to the process by which revenues are collected from primary and secondary sources, accumulated in fund pools and allocated to provider activities. Stewardship involves the aspects of setting, implementing and monitoring the rules for the health system; assuring a level playing field for all actors in the system (particularly purchasers, providers and patients); and defining strategic directions for health systems as a whole. Service provision is the combination of inputs into a production process that takes place in a particular organizational setting and that leads to the delivery of interventions. Resource generation refers to the production of inputs – particularly human resources, knowledge, and physical resources such as facilities, equipment and consumables – for the provision of services. While HRH are directly or indirectly related to each of these four functions and influenced by them, in this paper we will concentrate on assessing HRH inputs as they pertain to the latter two.

Different elements should be borne in mind when selecting and calculating appropriate indicators for HRH monitoring and evaluation. From the perspective of human resources as an input to health action, HRH can be broadly defined as "the stock of all individuals engaged in the promotion, protection or improvement of population health" [4]. This encompasses those working across different domains of health systems: public and private sectors; clinical, research and public health interventions; preventive and curative personal care; etc.

In general terms, a number of criteria have been identified as crucial for selecting indicators, including policy relevance, reliability, validity, simplicity and ability to aggregate/disaggregate information [10, 11]. Indicators on HRH are used for various purposes, such as measures of planning, implementing, monitoring and evaluating policies. To facilitate both the data collection and analysis processes, it is important to focus on a limited and essential number of indicators that are comparable and measurable regularly using standard data sources.

A series of components for selecting HRH indicators for the performance of the services provision and resources generation functions can be found in Fig. 2. The proposed list follows the general framework of health system performance assessment, classified in terms of the level of achievement, distribution (equity) and efficiency (productivity) of HRH [12]. This list is not meant to be exhaustive, nor is it necessarily expected that all these indicators will be used in any given HRH assessment. This will depend on the user's particular circumstances, with the eventual number reduced to an essential minimum in accordance with data availability and quality. This basic basket of indicators is also not intended to be restrictive, and focuses on indicators appropriate for quantitative analyses. Other indicators may be used as appropriate in the context of national and local operations or different types of study designs.

Detailed descriptions of the key indicators, including formulas and potential sources for measurement, can be found in the Additional file. Depending on the data sources and classifications used, many of these indicators can be disaggregated by occupation (physicians, nursing and midwifery professionals, etc.), by administrative units (districts, provinces or states, etc.) or other sociodemographic characteristics (age, sex, etc.). At the same time, indicators for some topics highly relevant to HRH analysis must be further investigated and tested. This includes measures of labour relations or dissatisfaction (such as work stoppages or other absences), regulation of health professions (such as rules to ensure standards of training and practice), financing of HRH production (such as fairness of financing for training programmes), risk factors to health workers (such as susceptibility to violence or to contracting infectious diseases including HIV/AIDS) and physical conditions (such as lack of materials or state of the workplace).

The range of indicators necessary to describe the profile and monitor the progress of HRH is extensive, integrating many aspects such as occupation, industry, training and others. A number of challenges must be faced in enhancing cross-national comparability. The roles of health care workers vary from country to country, and the professions also have different national histories and cultures. It can be difficult to categorize data when professional boundaries are not well defined.

The use of internationally standardized classifications – such as the International Standard Classification of Occupations (ISCO), the International Standard Industrial Classification of all Economic Activities (ISIC), and the International Standard Classification of Education (ISCED) [20–22] – can provide the backbone for comparing HRH between countries and over time. Classifications provide a framework for the description and comparison of statistics by categorizing items according to shared characteristics. But international classifications are not widely used in health research, and their potential as tools for defining the scope of the health system including human resources and for conducting HRH analyses has generally not been met. Different user areas may have different degrees of interest in the various elements. Studies of health systems and HRH in many countries still apply classifications that are not always comparable with standardized categorizations, although use of or mapping to ISCO and other international standards is becoming more widespread. Even when international classifications are used, they are often not used to full advantage.

The International Labour Office's 1988 revision of the ISCO classification system (ISCO-88) pools occupational titles into a hierarchical four-digit system that can be aggregated to progressively broader groups, representing a value set with respect to the type of work performed or to be performed. Occupations are specified according to the precision needed in major (one-digit level), sub-major (two-digit level), minor (three-digit level) and unit (four-digit level) groups. The basic criteria used to define the system of major, sub-major, minor and unit groups are the "skill level" and "skill specialization" required to carry out the tasks and duties of the occupations [20].

The main occupations of interest with a health care-related specialization fall within two of the ten major groups: group 2 "professionals" (generally well-trained workers in jobs that normally require a university or advanced-level degree for recruitment) and group 3 "technicians and associate professionals" (generally requiring skills at a tertiary non-university educational qualification level). Health practitioners in the professional group include physicians, nursing and midwifery professionals and other health professionals, such as dentists, pharmacists and veterinarians. Among associate professionals are modern health associate professionals (such as medical assistants, dental assistants, physiotherapists, opticians and sanitarians), nursing and midwifery associate professionals and traditional medicine practitioners.

For HRH analysis purposes, it is essential to have jobs classified at least according to the three-digit ISCO-88 level or equivalent. Information at the two-digit level alone does not allow distinction of health occupations from other life sciences occupations. Moreover, different health occupations can be separately identified only at the four-digit level, such as physicians (code 2221) from other health professionals. And certain related occupations aside from medical and nursing practitioners are likewise identifiable only at the four-digit level, in particular medical equipment operators (code 3133), health and safety inspectors (code 3152) and institution-based personal care workers (code 5132).

Even when occupations are classified according to ISCO-88, comparability issues may arise. For example, while physicians are unambiguously identified among the professionals major group (code 2221), the classification of nurses and midwives is less clear, crossing two major groups where they could be recorded: "nursing and midwifery professionals" (code 223) or "nursing and midwifery associate professionals" (code 323). This distinction was designed to reflect differences in tasks and duties that may be a consequence of differences in work organization as well as in education and training. In some countries, the possibility of distinguishing between the two typologies of nurses and midwives remains of limited relevance; inadequacies in the reporting system or incomparability of the education systems and measures of technical capacity may mean that some nursing and midwifery jobs do not fit easily into these two categories.

In addition, health systems employ a large number of workers with non-health occupational backgrounds, such as administrators, accountants, drivers and other support staff. Capturing this wider range of workers entails consideration of occupations across all of the major ISCO-88 groups, so further information on industry may be required. The ISIC classification can form a basis for such analysis. Using ISIC's latest revision requires information compiled at the three-digit level, in order to be able to distinguish the health sector from social work activities [21].

Other relevant classifications include ISCED, a cross-nationally comparable instrument for definitions of levels and fields of education and training [22]; the Central Product Classification, covering all goods and services that are produced or traded, including health services [23]; and the International Classification of Status in Employment, designed to facilitate cross-national comparability in the production and presentation of statistics on jobs by types of economic risk and levels of authority [24]. Using combinations of these classifications when describing HRH it will be possible to identify, for example, employment in non-health activities among those with a health-related education, and employment in health activities with jobs that do not require medical or nursing skills.

In practice, the precision of coding to such standard classification systems will depend largely on the level of specification in the raw data. Accurate occupational, industry and training information is vital in assessing HRH, starting with the most basic of indicators on size and composition of the health workforce. Many countries use national coding systems for occupation and other pertinent areas. However, data on health workforces are more directly comparable once differences in the organization of national health systems and the educational systems for health occupations are attenuated through harmonized classifications. Use among all countries of ISCO-88 to the greatest detail possible, or mapping of national classifications to this standard, would greatly improve cross-national comparability and facilitate use of health workforce information for informed decision-making.

Effective planning and management of HRH depend vitally on the availability of high quality and timely statistics. Despite a prevailing view that statistics on HRH are scarce, the sources that potentially can produce information relevant to this issue are quite diverse. Routine administrative records, censuses and household and establishment sample surveys are among the sources that allow researchers and decision-makers to calculate different indicators according to their goals and needs. Some of the published empirical studies on HRH have drawn on information in health-related professional registries [25], national accounts gathered by agencies of the United Nations system [26], or records of medical schools and faculties [27]. Other research agendas have focused on using demographic census data [28, 29], labour force survey findings [30], or specialized surveys of health human resources [31].

Two fundamental challenges for data compilation at the international level are to identify appropriate sources and to gain timely access to the data [32]. Frameworks have been formulated that can support efforts to coherently combine statistics from different sources and for different units [33]. However, in many countries, information on the health workforce is fragmented, and the statistics generated by these various sources have received limited public dissemination and generally been underused. Even in countries where updated data are available, it may be difficult to establish the size and composition of the health workforce because the collection and dissemination of representative occupational data using the ISCO-88 standard at the three- or four-digit level (or other comparable national classification) remains less widespread.

The empirical approaches that can be used for HRH analysis are numerous, and the selection of the appropriate technique (or techniques) for a given study will depend on many factors, notably the objectives of the study to be conducted, the type of information needed, and the data availability and quality. Such approaches can be broadly categorized as either quantitative or qualitative in nature.

Quantitative applications can range from descriptive techniques, such as univariate and bivariate frequency distributions of HRH by occupation and other background variables, to more advanced statistical methods. Summary descriptives are highly useful for visualizing complex data structures and integrating data items in graphical or tabular form for presentation. Advanced methods can be used for analysing the determinants of a specific HRH matter, involving techniques for testing the effects and interactions of certain variables while controlling for the effects of other variables (covariates). A wide range of models can be applied, depending on the variables and outcomes of interest. This can mean models that yield a single estimated equation with unique correspondence between coefficients and variables, or multiple equations and subsequent coefficients. They may involve time-series analysis or other types of smoothing techniques for projecting data in time/space based on observed trends. Among the advanced estimation methods for HRH analysis found in the published literature are multiple regression (to explain factors affecting the geographical distribution of physicians) [39] and proportional hazard models (to deal with issues of retention of nurses in the health system) [25].

Qualitative methods are techniques for examining non-numeric characteristics of the area of interest. They include the analysis of content from texts, focus group discussions, in-depth interviews and participant observations. Applications to HRH analysis may be especially pertinent for topics such as regulation of health professions, labour relations, and worker satisfaction and motivation (see, for example, [40]).

A strong HRH analysis should ideally be grounded in the basket of indicators previously discussed. For illustrative purposes, Figs. 3 and 4 present examples of how descriptive quantitative techniques can be applied for monitoring HRH within the framework for health system performance assessment. Drawing on one type of source, routinely reported data – as covered by national reporting systems and compiled in the WHO databases on the numbers of health personnel and medical schools – were aggregated by region and then organized in graphical form for presentation. Such data can be used in monitoring the service provision and resource generation functions, respectively.

As suggested by these findings, major variations in the supply of health personnel and training opportunities occur by region. For example, some 90% of countries in Africa and South-east Asia are characterized with a density of physicians of less than 50 per 100,000 inhabitants, and about 50% of countries have a similar density of nurses and midwives. In contrast, none of the European countries are marked with a comparably low density. Moreover, some 30% or more of countries in Africa and South-east Asia are ranked in the poorest category in terms of density of medical schools (one per 10 million inhabitants or more), whereas none are found among this group in Europe or the Americas.

Such analyses, while informative on HRH monitoring and evaluation from an international perspective, are subject to certain limitations. The classification of health personnel compiled through routine reports does not follow cross-nationally comparable standards. For one, the data fail to distinguish between professional nurses and midwives and their associate professional counterparts. They also make no obvious distinction between those who are actively working in health care services and those who are not. An important shortcoming of the data on numbers of medical schools is the lack of details on numbers and characteristics of entrants or graduates within these institutions.

No single data source can reflect the growing complexity of HRH issues. Rather, a variety of instruments – each with its own strengths and limitations – can be exploited to produce different types of statistics. The combination of complementary information from various sources can provide useful and rich information on the profile and other characteristics of the health workforce. Moreover, availability of different types of data sources can also serve as a control for the common information they collect, offering means for triangulation [41].

Figure 5 offers an illustration of the uses of triangulation for data quality control. The same indicator, in this case the ratio of physicians to nursing and midwifery personnel (indicator of skills mix), was measured for different countries through two different types of data sources: household-based labour force survey data (as compiled through the LIS project, see [30]) versus the routinely reported data. Simple linear regression was then used to characterize the relationship between the measures obtained from the two sources. Some of the observed discrepancies may be related to the occupational classification or to the sources' representation. The resulting parameters could eventually be used as a basis for adjusting reported numbers to compensate for data discrepancies (although before any adjustment is made, the assumptions underlying the evaluation method must be carefully considered).

The purpose of this paper was to improve knowledge about the information on HRH that researchers and institutions collect and why they do it. A number of indicators and data sources were discussed here, focusing on those being used for HRH monitoring and evaluation following the WHO framework on Health Systems Performance Assessment. The indicators were broadly categorized according to the service provision and resource generation functions of health systems. Issues of harmonization of data collection and processing methods for enhancing cross-national and time-trend comparability were addressed. Given that the roles and tasks performed by the same categories of health occupations might vary across countries, the occupational, industrial and educational classifications associated with the definition of HRH should be taken into account when conducting comparative analyses. In turn, the selection of an appropriate analytical approach will depend on the policy questions to be addressed, the resources available and the context of the study. Moreover, regardless of the methodology used, the ability to track time trends will generally be constrained by the periodicity of the data sources available.

Despite the existence of a number of reporting systems (censuses, surveys, routine administrative records, etc.) that can be used to examine HRH issues and their contributory factors, many of these sources remain underused and the information on current efforts for analysis is frequently incomplete. Few available sources have been designed with the sole intention of producing information on HRH. WHO is undertaking work to refine and expand knowledge on HRH issues, including compiling and analysing statistics from existing sources as well as statistics based on specially designed surveys. This strategy includes partnerships with ministries, central statistical agencies and other data providers, research centres, libraries and public health schools as part of a process of capacity building in countries. In particular, four main collaborative projects are currently being implemented for the compilation of an evidence base on HRH from an international perspective.

The World Health Survey (WHS) is being conducted in collaboration with Member States in more than 70 countries from all regions and levels of development, aiming to provide a wide range of quantitative information that can be used to assess population health and health systems performance [42]. A module on health occupations was developed for inclusion in the household-based survey in order to collect comparable baseline information on the current health workforce for primarily monitoring the services provision function. A standard questionnaire was designed to identify all adults in the sampled households who had ever been trained in a health-related field or engaged in the health workforce, with follow-up questions on occupation and training, main work activities, workplace industry and sector, income and methods of remuneration, migration and other topics. The survey instruments were pre-tested in 12 countries in 2002, and fieldwork for the main survey phase will have begun in participating countries by early- to mid-2003.

Another project is the development of an electronic database called the Global Directory of Health Training Institutions, which will provide information on the status and trends of health education worldwide. This is an expansion of the former listing of medical schools [38], building a wider database that also will include training institutions for other health professions such as nursing, dentistry and pharmacy. It will capture selected characteristics of these institutions, notably the numbers and characteristics of entrants and graduates, in order to be able to measure a range of indictors for monitoring the resource generation function.

In-depth assessments of human resources for health are being conducted in 2002–2003 in six developing countries (Chad, Côte d'Ivoire, Jamaica, Mozambique, Sri Lanka and Zimbabwe). The study involves collecting quantitative and qualitative data at the national, institutional and individual levels. Four areas are included in this assessment: regulation of health occupations, health training institutions, health care facilities and health care providers. In order to ensure national representation of facilities and providers, the guidelines for implementation recommend the use of complementary list- and area-sampling frames for coverage of the range of facility types, especially from the private sector, for which available lists are frequently incomplete [43]. The survey questionnaires are similar to those developed for the WHS and Global Directory of Health Training Institutions, but were designed to gather more detailed information on professional licensing and registration, labour conditions, supply and demand for health personnel, and other relevant topics in order to develop short-term and long-term recommendations for HRH policy and planning.

A fourth project is the creation of a meta-database of data sources on HRH. This involves the comprehensive review for all countries of sources of information on human resources in health systems, in order to maximize the stock of methods and materials for international analyses on HRH. The task entails an extensive search process using electronic library sources and World Wide Web-based searches to identify data sources, including general sources such as censuses, labour force surveys and other reporting systems as mentioned above, with special attention to availability of statistics on health occupations and labour market characteristics. In some cases special tabulations on HRH may be directly requested from countries or from other international agencies (such as ILO or EUROSTAT). The development of such a database will allow for distinguishing variables to support consistent cross-country comparisons and within-country in-depth assessments.

As previously mentioned, understanding of the growing complexity of HRH issues requires a variety of methods and materials. Improved coordination of data-gathering activities could act as a catalyst for improving the availability, quality and comparability of data for monitoring and reporting HRH initiatives. HRH assessments should therefore be constructed as a result of a collaborative process between all the health system stakeholders, including health care providers, local and government agencies, nongovernmental organizations and international agencies. It is desirable to further assess to what extent research efforts are concurring, what overlapping may exist, and how research may be improved by filling gaps. Regarding implementation, ways should be found to ensure that the data received and used by international organizations are the most recent and accurate, and are processed through the use of internationally standardized classifications at the greatest level of detail possible.